The Brawn Behind EBay's Always-On Auctions

EBay Technologies Inc. president Maynard Webb sums up life at the online auction site: "It's not for the faint of heart."

Such stoicism is reflected in the new IT infrastructure created by eBay Technologies, the IT arm of eBay Inc. The architecture, two years in the making, is flexible enough to handle instant recoding but sturdy enough to process more than 800,000 transactions every minute.

That hasn't always been the case at the online auction site. Back when Webb joined the company, eBay's IT infrastructure wasn't quite so flexible, scalable, or reliable. In fact, Webb--who was CIO at Bay Networks and Gateway Inc. before joining eBay--witnessed just how precarious eBay's IT systems were when he interviewed for his current position two years ago. The Sun Microsystems E10000 64-processor servers had crashed for three consecutive days, just a short time before financial analysts were due for a visit. President and CEO Meg Whitman "was talking to CNN, telling the world about the outage problems, when I walked in," Webb recalls. The high-profile outages gave Webb pause. "Maybe this was bigger than I thought, " he found himself wondering.

During the three-day interview, Webb ended up getting heavily involved in solving eBay's problems. By 5 a.m. on the third day, just a few hours before the analysts were to arrive, the auction site was back up. Needless to say, Webb was offered the job.

The first project that Webb tackled after taking the post at eBay was establishing backup and recovery systems that would keep the site running in the event of another outage. Within weeks, he had in place backup that was designed to handle a recovery effort within four hours. Three months later, Webb had installed an even more-aggressive backup infrastructure that can recover systems in five minutes to a half hour. That backup system still serves the auction site.

Webb also knew something had to be done to strengthen eBay's server infrastructure. He decided to do away with the E10000s that had crashed because, he says, they have limited scalability. Webb considered an infrastructure from IBM based on a mainframe and AIX Unix servers. "Few folks could deal with our volume," Webb says, "and I knew IBM could." But he wasn't convinced that the IBM systems could support eBay's need for speed: Webb wanted systems that could deliver capacity on demand and handle server swaps on the fly. Further, Webb didn't think eBay's top management could live with IBM's price, which would easily have topped $20 million.

One of Webb's directors at the time, Mike Wilson, came up with the idea of using cheaper, smaller servers in a cluster that could easily grow with eBay's needs. Webb liked the idea and settled on a cluster of 12 Sun E4500 servers.

Around the time Webb was stabilizing eBay's IT infrastructure, he also began assembling a crack IT team that would carry eBay into international markets and new business opportunities, including acquisitions. Together, the IT employees and updated technology now support one of the world's busiest Web sites.

The data is staggering. The 5-year-old company is on track to have gross merchandise sales of $9 billion this year. In the second quarter of 2001, it had 34.1 million registered users, and each day it acquires about 40,000 new registrants. EBay lists more than 98 million items, including baseball cards, cars, and airplanes. Bids can be quite pricey, too: The oldest known pair of Levi's jeans sold for $46,532, much more than the few thousand dollars Levi Strauss & Co. had offered the seller.

So far, the IT environment and Webb and his team are serving eBay well. On Sept. 11, Webb kept to his schedule. Meetings that had been on the calendar for weeks went on as planned, though Webb kept in constant contact with his management team. The sturdy-but-flexible IT architecture was ready, too, when, at one point that day, Webb stepped out of a meeting and issued a new policy: EBay would not play host to any sales of memorabilia from the terrorist attacks.

The policy required immediate action to the systems. "We don't get months to master policy decisions, and we change lines of code many times over." Because of eBay's use of Clear Case, a software-configuration management tool from Rational Software Corp., the auction site can make changes to multiple applications simultaneously without sacrificing integrity.

EBay is also revamping its application development architecture to support the Java 2 Enterprise Edition framework. Currently in its first phase of implementation, the architecture, which is code-named V3, is overseen by Chuck Geiger, eBay's VP of technology strategy. It's replacing a C++ object framework that requires a lot of structural programming. Though the C++ environment is more flexible than Cobol, it can't compete with J2EE, which is becoming the de facto application development framework. J2EE's objects can handle a much higher level of abstraction than C++.

This holiday season will likely serve as yet another test for eBay. IT advisory firm Gartner says it expects retailers to make nearly $11.9 billion in online revenue in North America this quarter, a 30% increase compared with last year.

EBay's customers--the buyers and sellers known as the Community--dictate much of eBay's IT strategy. And the company's attitude toward the Community borders on reverence. That's what propelled eBay to build an IT infrastructure that was stable, always available, and flexible.

The company has a rigorous development and design process to support the Community. Ideas for new Web-site features often come from individual buyers and sellers, as well as customer groups who regularly meet with senior VP Marty Cagan, who's in charge of product management and design. Cagan and his staff collaborate with Lynn Reedy, VP of product development, and her team of engineers.

Once those two teams create a new feature or product, they present it to Marty Abbott, VP of operations. Abbott and his team determine what impact the new feature will have on the infrastructure, and they make the necessary upgrades and changes, steps that can mean the purchase of new servers or adding capacity.

EBay is in the process of adding a new feature that's a result of listening to customers and interdepartmental collaboration. The "Sell Your Item" feature is designed to simplify the sales process, which had required sellers to wade through multiple site pages and track down multiple documents.

Another piece of the company's revamped IT infrastructure includes an Oracle database. "We're processing millions of transactions per day, and that means velocity and volume," says Bob Sanguedolce, the company's CIO. "Things happen fast and a lot at eBay."

Data-mining tools and an extensive data warehouse--which won the Leadership Data Warehousing award last summer from the Data Warehousing Institute, a consulting firm and consortium of professionals in the data warehousing field--help eBay determine its next steps. "We want to know the value of a customer," Sanguedolce says, "and the effectiveness of their dollars."

It's been a long two years at eBay, and the IT team continues to put in extra time to ensure that the company is ready for anything. It's well worth the effort, Abbott says. "Days are still long now, but it's not because the Community can't get on the site for 30 minutes," he says. Instead, the overtime leads to site enhancements and new products that will keep customers coming back for more.