Just some thoughs from Xooglers:
Let's get a real database
Posted by Ron at 3:09 PM Friday, December 09, 2005
This story is a bit of a tangent, but I think it's worth telling because there are some useful lessons to be learned.
AdWords was built using the MySQL database, which is open-source and therefore available for free. It is by now also nearly as full-featured as the best commercial databases, but back in 2000 this was not the case. MySQL was quite a capable system, but missing a few (what some would consider basic) features. These missing features were obviously not a show-stopper, as we managed to get AdWords to work without them, but in a few cases it did take some extra programming to work around one of these missing features. On the plus side, MySQL was fast and reliable and, as I have already noted, free.
After AdWords launched, Jane, the ads group manager, decided that now would be a good time to switch over to a "real" database. "Real" is one of those words that Doug ought to add to his list of words. It means "expensive". Many managers seem to have this idea that it is invariably true that you get what you pay for, and that therefore nothing that is available for free can possibly be any good. Using MySQL was acceptable as an expedient to get things up and running quickly and with a minimal of capital outlay, but now that things were settling down it was time to recognize that this was really, fundamentally, a mistake, and it should be fixed sooner rather than later.
The flip side of this philosophy is the one more commonly espoused by engineers, which is nicely summed up by the old aphorism, "If it ain't broke, don't fix it." Yes, MySQL was missing some features, but it wasn't broke
. We had spirited debates in ads group meetings over what to do.
We finally decided to go with a commercial database (I won't say which one) over the objections of a number of engineers, including myself. To ease the transition it was decided to convert AdWords over to the new system first, and to do the main ads system later. It was a project on a par with the internationalzation effort in terms of the tedious work required to comb over nearly all of the AdWords code and change all of the database queries. (Databases are supposed to all be compatible with one another, but in reality they pretty much aren't.)
To make a long story short, it was an unmitigated disaster. The new system was slower than molasses in February. Some heroic optimization efforts eventually produced acceptable performance, but it was never as good as the old MySQL-based system had been. For a long time we were stuck with the worst of all possible worlds, with the two ads systems running on two different databases. It was still that way when I left Google in October of 2001, but I have heard through the grapevine that they eventually went back to MySQL. (Since then, MySQL has added many of the features that had been missing at the time.)
The moral of the story is that sometimes, and in particular with free software, you get more than what you pay for. There are a lot of companies out there paying dearly for commercial databases (and operating systems for that matter). As far as I'm concerned they might as well be flushing that money down the toilet. Actually, they might be better off. We certainly would have been.
As an aside, there is a raging debate in the hacker community about the overall economic merit of the open source model. (Making money producing free software is quite a challenge.) I am not taking sides in that debate here. All I am saying is that from the end user's point of view free software is often much better than the producers of commercial software would like people to think.
Posted by Ron at 3:09 PM Friday, December 09, 2005
This story is a bit of a tangent, but I think it's worth telling because there are some useful lessons to be learned.
AdWords was built using the MySQL database, which is open-source and therefore available for free. It is by now also nearly as full-featured as the best commercial databases, but back in 2000 this was not the case. MySQL was quite a capable system, but missing a few (what some would consider basic) features. These missing features were obviously not a show-stopper, as we managed to get AdWords to work without them, but in a few cases it did take some extra programming to work around one of these missing features. On the plus side, MySQL was fast and reliable and, as I have already noted, free.
After AdWords launched, Jane, the ads group manager, decided that now would be a good time to switch over to a "real" database. "Real" is one of those words that Doug ought to add to his list of words. It means "expensive". Many managers seem to have this idea that it is invariably true that you get what you pay for, and that therefore nothing that is available for free can possibly be any good. Using MySQL was acceptable as an expedient to get things up and running quickly and with a minimal of capital outlay, but now that things were settling down it was time to recognize that this was really, fundamentally, a mistake, and it should be fixed sooner rather than later.
The flip side of this philosophy is the one more commonly espoused by engineers, which is nicely summed up by the old aphorism, "If it ain't broke, don't fix it." Yes, MySQL was missing some features, but it wasn't broke
We finally decided to go with a commercial database (I won't say which one) over the objections of a number of engineers, including myself. To ease the transition it was decided to convert AdWords over to the new system first, and to do the main ads system later. It was a project on a par with the internationalzation effort in terms of the tedious work required to comb over nearly all of the AdWords code and change all of the database queries. (Databases are supposed to all be compatible with one another, but in reality they pretty much aren't.)
To make a long story short, it was an unmitigated disaster. The new system was slower than molasses in February. Some heroic optimization efforts eventually produced acceptable performance, but it was never as good as the old MySQL-based system had been. For a long time we were stuck with the worst of all possible worlds, with the two ads systems running on two different databases. It was still that way when I left Google in October of 2001, but I have heard through the grapevine that they eventually went back to MySQL. (Since then, MySQL has added many of the features that had been missing at the time.)
The moral of the story is that sometimes, and in particular with free software, you get more than what you pay for. There are a lot of companies out there paying dearly for commercial databases (and operating systems for that matter). As far as I'm concerned they might as well be flushing that money down the toilet. Actually, they might be better off. We certainly would have been.
As an aside, there is a raging debate in the hacker community about the overall economic merit of the open source model. (Making money producing free software is quite a challenge.) I am not taking sides in that debate here. All I am saying is that from the end user's point of view free software is often much better than the producers of commercial software would like people to think.