IW500: 6 Lessons Learned From FedEx's Private Cloud

FedEx in the fall of 2010 opened a new data center in Colorado Springs, Colo., that is intended to function as its "general purpose, workload agnostic" computing environment--in short, as its private cloud, said Rob Carter, executive VP and CIO.

He said he implemented what he views as the "dominant design" for future data centers, taking his cue from what he's seen Salesforce.com, Twitter, and Facebook build for themselves. Here are six conclusions on how to do that, based on his talk Monday at the InformationWeek 500 conference in Dana Point, Calif.

1. Bet on Moore's Law and x86.

We've reached a stage with today's x86 architecture where large-scale servers can be easily produced, but it's still early in the application of Moore's law, which calls for a doubling of computing power at a minimum every two years. In 30 days, the amount achieved starting with a penny and doubling each day comes to $5 million.

With x86, we're only at day 19 of that process, Carter said. Big gains are soon to come. Stick with x86, even if you think you need the power of a legacy system to handle your workload. The Colorado Springs data center "looks like rows and rows of scale-out x86 servers," he said. "We had a big relational database environment. We had put millions into hardware to support our billing and revenue processes. Over the last few years, we had spent $10 million," he said. The same environment, now on x86, represents $30,000 in hardware running Linux.

2. Apply the lessons of SOA.

You don't hear the phrase "services-oriented architecture" much anymore but Carter used it twice. Organize your applications as x86-based services, where one service can serve many application needs. The applications that run in the new data center rely on what Carter called "commonized services" in a shared environment, with standardized approaches to user access and authentication, messaging, security, and database access.

The approach saves IT from needing to do many of the same things over and over again. The environment, both server and networking, has been simplified and applications run more predictably in it. Troubleshooting, often a convoluted process, gets simplified as well. "The ability to scale and the commonality among applications (of thoroughly tested services) provides very few points of failure," he said.

3. Virtualize, then virtualize some more.

The Colorado Springs data center is heavily virtualized, with 70% of FedEx's servers running as virtual machines. That means it's been "LEED gold certified as a green data center that uses less power, has less of a footprint," said Carter. Power consumption now represents 25% of the typical data center budget. By using virtualization, FedEx finally got a grip on its climbing power consumption and realized another "compelling" savings.

4. Enlist reluctant top management.

To persuade top management to make the move, Carter didn't do five-year plans or 10-year projected returns on investment. Instead, he drew a map of what the existing data center environment looked like versus the simplified one he wanted to operate. His mappings of the existing application infrastructure and dependencies were so complex, intense, and confusing that his CEO dubbed them scenes from "Hurricane Rob" in his honor. "I had a lot of ugly pictures," he said.

5. You don't have to wait for just the right software.

Carter didn't get caught in analysis paralysis. Carter wasn't looking to bring in an external supplier's private cloud system to get started or waiting for a cloud computing standard, such as OpenStack, to firm up. As a Salesforce.com customer, he and his IT staff "compared notes" with Salesforce on its data center operations and FedEx knew how Twitter and Facebook had built their new data centers. His staff worked with the LAMP stack--Linux with a proven set of integrated open source code--to build a shared x86 infrastructure, then created pilot services on top of it.

There were cultural issues. Some IT staffers felt they were doing a good job with the way legacy systems were running. Carter illustrated that FedEx was using 200 different applications to manage addresses, a key component of its business, when one address service could do the same thing more efficiently and at a lower cost. Another new service supplied currency conversions throughout the company. Carter told his staff, you're doing a good job of managing the details of the existing infrastructure. "It was the macro picture (of future data center operation) that looked unsustainable."

6. What about the public cloud?

Carter acknowledged his approach doesn't include a blueprint for working with the public cloud. But FedEx is a close partner with Salesforce.com--"I've known Marc Benioff for a long time," said Carter--and initially, FedEx will "live in a hybrid cloud world with Salesforce" CRM applications. But Carter also noted the seasonal nature of FedEx's business and how it doubles with the approach of the holidays. "Our business is unique in its peaking factor ... Amazon Web Services' capacity to handle intense workloads, like Netflix streaming, means it might be quite suitable for some of our work," he said.