Lessons eBay Learned In The Data Center

Check out the LInuxWorld image gallery.

If IT managers ever wanted a model for how to transition a data center, they should look no further than eBay.

Paul Strong, a research scientist at eBay Research Labs, related the fall and rise of the online auction site's back-end infrastructure between the years 1999 and 2000 during his keynote at the LinuxWorld and Next Generation Data Center conference in San Francisco this week.

Strong's main takeaway: Relationships are everything and changing one thing causes ripples. Massive ones.

"We had a typical three-tier structure with a second database for failover," Strong said. "By November 1999, however, the database servers approached their limits of physical growth. ... Failure to keep up is not an option." The overload helped contribute to the celebrated crash of eBay that year, which sent pundits and news organizations into a frenzy suggesting that the end of the World Wide Web was at hand.

So in 2000, Strong said eBay made a database split with the rest of the technology. The company partitioned the database into separate instances with the separate application notion of a database from its physical implementation.

In this way, Strong said the databases may be combined and separated with no code changes and applications can continue to function without noncritical data by using several layers of cached memory.

"This was very hard with lots of people running around," he said. "We created a data abstraction layer because we had to get ourselves to take the database and take it out and hide it."

The other shift for eBay was to migrate from large mainframes to a series of smaller commodity servers. The shift may look simple on the outset, but Strong said eBay had a lot at stake.

"One of the most humbling things in working for eBay is that you realize that 1.3 million people claim eBay as their primary or secondary income. People depend on us for all or part of their living income," Strong said.

By shifting to a more virtualized environment, Strong said eBay has been able to handle the 241 million registered users who post 6 million new items per day with about 100 million items for auction at any point in time.

"We're able to roll code to the site every two weeks onto 16,000 application servers and push through changes," Strong said, noting that the changes could be as simple as a color change on a widget to something as complex as the roll out of eBay Express.

Of eBay's three properties (eBay, PayPal, Skype), Strong said eBay is the most mature but does not serve as a distinct model for the other two.

"PayPal is structured very much like eBay, but they are kept separate from each other for legal reporting reasons," Strong said. "Skype is more of a peer-to-peer model and we're applying new techniques to always improve the system."

Unlike Amazon.com, which is offering up its data center resources to other businesses, Strong said that eBay is evaluating the market but has no plans to sell off its excess computing power.