Inside Microsoft's $550 Million Mega Data Centers

Though the building alone covers a whopping 11 acres, you can't even see Microsoft's new $550 million data center in the hills west of San Antonio until you're practically on top of it. But by that point, you can hardly see anything else.

These days, the massive data center is a bustling construction zone where visitors have to wear hardhats, helmets, orange safety vests, goggles and gloves. By September, it'll be the newest star in Microsoft's rapidly expanding collection of massive data centers, powering Microsoft's forays into cloud computing like Live Mesh and Exchange Online, among plenty of other as-yet-unannounced services. Pulling in, visitors are stopped by Securitas guards who check IDs and ask if they work for Microsoft. An incomplete gate marks the way. Microsoft's general manager of data center services, Mike Manos, won't say exactly what security measures will be in place when the data center opens, but won't rule anything out. "Will the gates be able to stop a speeding Mack truck?" I ask. "Or more," he responds. "Will you have biometrics?" "We have just about everything."

As the car rounds the bend beyond the gate, the building sweeps into full view. The San Antonio data center building itself is 475,000 square feet, or about 11 acres. It's a 1.3 mile walk to circumnavigate the building. To get a perspective on that, it's one building that's the size of almost 10 football fields laid out side-by-side, or 1/10th the floor space of the entire Sears Tower, covered with servers and electrical equipment. "I thought I understood what scale looked like," Manos says.

When the San Antonio data center was under peak construction, 965 people were working full time to build it, with more than 15 trucks of material coming and going each day in order to get the job done in 18 months from scouting the site to opening up. The facilities were built with continuous workflow of materials in mind, even after the site's completion.

As one walks toward the data center's main entrance, a feature that stands out is a row of several truck bays much like would be seen in an industrial park. Trucks pull up and leave servers or other materials inside the bays or "truck tracks," to be picked up and inventoried in the next room and then moved to storage or deployment.

Most everything in the data center is functional. On the small scale, wainscoting-like pieces of plywood cover the bottom of hallway walls to protect both the walls and servers and other equipment moving back and forth. On the large scale, San Antonio is actually two data centers side by side to separate business risk. "One side could burn down and the other one could continue to operate," Manos says.

The components inside are just as gargantuan as those on the outside. Seven massive battery rooms contain hundreds of batteries and 2.7 mW of back-up power apiece. Very few industrial sites, among them aluminum smelters, silicon manufacturers and automobile factories, consume as much energy as mega data centers of the order Microsoft is building.

There are two huge water towers on site for storing chilled water, one for each half of the data center. Each is so big -- roughly 30 feet in diameter by 30 feet tall -- that it could easily serve as a water tower for a small town. Microsoft's San Antonio data center receives a large supply of nuclear and wind energy as well as a large recycled water supply. After energy, Manos believes, water will be the next raw material to affect data center expense calculations.

The scale of the San Antonio facility is so large that Microsoft has had to develop software to manage it, since no independent vendors can easily do so. For example, the software can run a full simulation of the data center running on all cylinders and run tests for metrics and measure loads and other things. The company can also measure and calculate down to the smallest elements, such as how many watts the data center uses and how much revenue is gained per search query, for example.

Manos has a group called Research and Engineering that's operates like data center design R&D arm, with 40 to 50 active projects underway at any one time.

Some of the company's other work rests on things like sensor research and emerging monitoring capabilities. "There are some design elements we are working on that will definitely reshape the way we think about data centers in general," he says. Some of those might even eventually turn into products.

Microsoft won't disclose the exact specs on the water towers or indeed many of its huge machines and pieces of other infrastructure that make the data center tick, but says it will likely do so in the future. Unlike Google, which has been mostly closed-mouth on its data centers, Microsoft has begun taking not only journalists, but also enterprise customers and software and hardware partners on tours in order to share best practices, technical detail and even carry out peer reviews. Manos has also been making the rounds speaking at data center shows.

"Historically, we've taken the approach that data centers are our competitive advantage, but 12 to 18 months after putting cool new stuff into place, it comes onto the market anyway," Manos says. It's Microsoft's business to sell more server software and more services, and that means divulging best practices and being transparent.

Microsoft customers have also told Microsoft that they need more information on what to measure and how, as well as what to implement and how. Manos believes there are far too many data center metrics floating about, and too little understanding of what to do with them. "The industry as a whole, we've got to get our act together," he says, adding that there's an increasing possibility of Federal legislation that may someday regulate data center efficiency in some way. He estimates that only 10% of companies have adopted any sort of capability to measure energy efficiency today, and that confusion reigns.

Take CADE, an energy efficiency metric McKinsey & Company recently unveiled. Manos says it’s a good metric, but it gives data center operators a number between one and 100 that's tough to decipher. Microsoft is working on a way to make CADE more meaningful by turning the raw numbers into a diagram of what it means and what a company needs to do to improve it.

Power is understandably a big issue in such huge data centers. Manos says power is increasingly costing more over the life of a server than the initial acquisition cost of that server. He believes there are many more revolutionary gains to be had in energy efficiency in just about every facet of the data center, including in electrical distribution, cooling, and patterns and practices.

In one sign Microsoft is serious about going green, the company instructed Turner Construction, which is building the data center in San Antonio, to save all of the old growth trees on the property. Only one, which is now nicknamed "the Manos tree" was within the building's original footprint, and stands alone in the parking lot.

Microsoft breaks its data centers into "classes" much like aircraft carriers. San Antonio is of the Quincy class, Microsoft's older and more traditional data centers. The Chicago class will be Microsoft's biggest data centers and will use self-contained shipping containers filled with servers as their unit of scale, while the Dublin class focuses entirely on energy efficiency. In the near future, the company plans to announce its newest data center and a new class of data centers while retiring another class.

The San Antonio data center will be ready to go by September, but right now Microsoft is just testing out its systems. The day I was there, it was doing heat testing with heat banks to make sure the air conditioning system could handle high heat output. In every case, Microsoft pushes its tests beyond the breaking point. And if you thought their data centers couldn't possibly get any bigger, they can: the new facility going up in Chicago is twice the size of San Antonio's data center.