Data Center Consolidation Gets Under Way 2

he first two phases of the federal government's long-term data center consolidation effort--an agency-by-agency inventory of data center assets, followed by submission of draft consolidation plans--have been completed. Now begins the hard work: Implementing those plans, with the goal of reducing the number of federal data centers by 40% by 2015.

It's an aggressive goal, but one that has the backing of President Obama. In June, Obama issued a memo instructing agencies to adopt a zero-growth strategy for data center space over the near term, to be followed by a reduction in data center floor space over the next five years.

Four months before that presidential memo, federal CIO Vivek Kundra introduced the government's overarching plan, the Federal Data Center Consolidation Initiative (FDCCI). At the time, Kundra laid out a series of deliverables with deadlines culminating on Dec. 31, 2010, when the final data center consolidation plans are to be approved by the Office of Management and Budget.

While many private sector companies have been scaling back the number of data centers they operate for several years, the trend among federal agencies has been the opposite--skyrocketing growth. The number of federal data centers has quadrupled over the past 12 years, from 432 in 1998 to 2,094 in 2010. Agencies with the most data centers are the departments of Defense (with 772), State (361), Interior (210), and Health and Human Services (185).

For the first nine months of the FDCCI, the feds moved down the path of data consolidation in concept, but without a firm target for reducing the overall count. In November, Kundra stopped the waffling and said the goal is to reduce the number of federal data centers by 40% by 2015. That means eliminating some 800 data centers over the next five years.

Agencies are now working with OMB to ensure that their targets are realistic. The approved plans will be released along with the fiscal 2012 budget, in the first quarter of 2011.

Although agencies have completed inventories of their data centers--a tally of the number and types of servers, terabytes of storage, number of virtualized systems, and square footage--OMB hasn't released the results of that exercise. Getting a firm handle on the government's data center assets hasn't been easy. Last year, in a preliminary count, OMB determined that there were 1,100 federal data centers; a more comprehensive review this year resulted in the much higher assessment of 2,094 data centers.

Kundra attributes the initial, incomplete tally to inconsistent definitions of what constitutes a data center and generally poor IT asset monitoring within federal agencies. For its more encompassing count, the government included all data centers that are at least 500 square feet in size and used for data processing.

As agency CIOs begin ratcheting down their data center operations, they face some significant hurdles. In many agencies, for example, IT assets aren't all managed by a central IT department, which means CIOs don't always have the authority needed to mandate consolidation.

Homeland Security CIO Richard Spires, the designated cross-agency leader of the FDCCI, acknowledges the organizational challenges but says a methodical approach to consolidation will help overcome any resistance. Homeland Security is in the process of consolidating 24 data centers into two primary data centers by the end of fiscal 2014. "I'm also living this," says Spires.

Spires advises agency CIOs to pursue "quick wins" in data center consolidation, then build on those early successes. Departments will be "much more willing to come on board if they see the price points and the success you've had," he says. "We're at the tipping point at DHS, but it doesn't happen in a month or even six months."

Kundra often points to DHS's consolidation efforts as an example for others to follow. The agency has shuttered six data centers so far, and a dedicated team in the CIO's office is working to bring some of the department's big "component" agencies--the Federal Emergency Management Agency, Immigration and Customs Enforcement, Transportation Security Administration, and Citizenship and Immigration Services, among them--into its two enterprise data centers.

To draw in those agencies, DHS is offering enterprise services from those facilities including e-mail, SharePoint, and a development and test environment. It has published a service catalog and hired customer-service reps to support its expanding portfolio of services.

In determining which applications to offer as services from Homeland Security's data centers, Spires is comparing the internally hosted model to what's available from commercial service providers. "For those types of apps that could be potentially in the cloud, we are certainly looking at the cost-benefit of whether or not we should move those applications to our own data centers or into cloud-based services," Spires says.

Server occupancy in Homeland Security's two big, central data centers has been steadily increasing over the past two years. Next, the agency plans to install a new high-voltage power line and power station at the data center located at Stennis Space Center in Mississippi. And it wants to establish an operations center to manage its data centers.

While some agencies like DHS are in the early stages of moving forward with their data center consolidation plans, others are waiting for the green light from OMB to get started. The plans submitted in accordance with FDCCI include summaries of agency approach and rationale, cost-benefit analysis, specifics on which data centers will be cut, timelines for consolidating, virtualization goals, and more.

Since this article is being published in advance of OMB's Dec. 31 deadline for approving FDCCI plans, agency CIOs were reluctant to share all of the details of their consolidation plans, citing the possibility that those plans may still change. "There are still a lot of things being worked through," says Spires.

With that caveat, here are the data center consolidation plans for three government agencies.

The Army's Server Moratorium

The Army operates 250 data centers worldwide, making it one of the largest data center operators in federal government. That sprawl led in 2006 to a plan to consolidate many of those data centers into 10 new "area processing centers." While APCs have opened in Kuwait, Germany, North Carolina, and Oklahoma, the costs were more than anticipated, which has slowed the project and resulted in a shift in plans.

The Army has decided to use the Defense Information Systems Agency's Defense Enterprise Computing Centers as a core part of its new strategy. In the process, the Army plans to reduce the number of data centers it operates by 75%, from 250 to 65, in five years.

Col. Kris Kramarich, the Army's network integration and data center consolidation lead, calls the original plan to consolidate primarily into 10 APCs "unrealistic" in the near term. "We want to leverage as much existing stuff in DOD as possible," Kramarich says.

Among the first systems to be migrated to DISA facilities will be e-mail. The Army expects to save about $100 million annually by moving from disparate e-mail systems and multiple versions of Microsoft Exchange and Active Directory to one integrated system hosted by DISA. The migration is scheduled for completion by the end of September 2011.

To expedite consolidation, the Army earlier this year issued a moratorium on server and switch purchases. Departments must get a waiver to buy new computer hardware. The process has helped the Army rein in server sprawl, and it has even identified data centers it didn't know about. Going forward, the moratorium will deter investment in data centers that are scheduled to close. "Case-by-case approvals can help us balance mission and technology requirements," Kramarich says.

Under the Department of Defense's Base Realignment and Closure program, the Army is moving or closing 16 data centers. Separately, it has issued a request for proposals for a $249 million, five-year cloud computing initiative called the Army Private Cloud, from which it will deliver software, platforms, and infrastructure as services. It's also implementing technology to meter its data centers to measure power use, and it's cataloging applications with an eye on consolidating them.

Getting the Army's many divisions fully on board the effort will be an ongoing challenge. "We have 250 different data centers and 250 different justifications for those data centers," Kramarich says. "The best thing we can do to address that is demonstrate success with initial sites and, as we come across issues, work through them on the spot as we move forward."

NASA Postpones Construction

NASA has operations far and wide--and a management structure to match. Each of its 10 space centers has its own CIO, and it's not unusual for IT infrastructure to spring up in support of NASA's different missions and research projects. With so many IT fiefdoms, data center consolidation will happen gradually.

NASA has about 15,000 servers in approximately 79 data centers, including three enterprise data centers, large data centers at each space center, and "a bunch of little, project-oriented" centers, says Chris Kemp, the agency's CTO for IT.

Two years ago, the agency started down the path of a five-year data center consolidation effort. But it reassessed that plan earlier this year--postponing construction of a $1.5 billion data center--to reevaluate its needs. OMB's data center consolidation directive was a factor in NASA's decision to go back to the drawing board.

NASA has made progress. It consolidated all of its mainframes into one facility at Marshall Space Flight Center in Huntsville, Ala., collapsed 237 e-mail systems and dozens of ID management systems into one enterprise-wide e-mail system and directory, and whittled down to a single accounting system.

With its Nebula cloud computing environment, which originated in a containerized data center at NASA's Ames Research Center in California, the space agency is one of the government's early leaders in cloud computing. Kemp sees Nebula as a way of potentially consolidating the agency's far-flung scientific computing resources, and NASA is in the process of building a private cloud to run enterprise applications like e-mail and finance. "Going forward, we have a cloud-first way of looking at things," Kemp says. "We intend to solve problems with as little infrastructure and custom software as possible."

Another goal is to save 20% on data center energy costs by decommissioning underused applications and servers, instrumenting systems to gauge power consumption, and making increased use of virtualization.

But Kemp warns against moving too fast. "Federal CIOs need to understand why organizations deep inside their agencies are doing their own thing and what they can provide to turn those noncustomers into customers," he says. "There's a lot of work to be done to see where there are true opportunities to consolidate, and where, if you consolidated, you'd totally break everything."

Yes, there's unnecessary redundancy in federal IT operations, Kemp says. On the other hand, "there's also a lot of necessary redundancy" for programs like space flight that have no margin for error.

Veterans Affairs' Ambitious Goals

The Department of Veterans Affairs has been consolidating its data centers since 2004, a process that slowed down in 2007 and 2008, but that's now back on track. Where it once had 175 data centers, including one for each of its 153 medical centers, it's down to 87. The VA's ultimate goal is to have six or eight central data centers, including existing facilities in Philadelphia; Austin, Texas; and Hines, Ill.

In 2006, the agency planned to collapse its medical center data centers into four primary centers and four backups, but there were roadblocks. "Data center consolidation was not in vogue at the C-level in government at that point, so it was a political battle," says Charles De Sanno, the VA's executive director of enterprise systems engineering.

That 2006 plan included a pilot project to establish "demonstration data centers" in New York and Sacramento, with replicated operations in Philadelphia and Denver for disaster recovery. Those regional processing centers were to serve the Veterans Health Administration and its hospitals, but a nine-hour system outage, the result of a configuration mishap, caused De Sanno and team to pull back on the plan.

Since then, CIO Roger Baker, who joined the VA in 2009, has provided funding for data center consolidation in two additional regions around the country. The performance problems that sidetracked consolidation in 2006 have been worked out. "You can't say anymore that you don't want to go to a data center because it doesn't perform," says De Sanno. He expects the migration of the next two regions of medical centers into consolidated data centers to be completed over the next 16 months. E-mail consolidation will also take place during this phase.

Next, the VA will move into an agency-wide phase, called its National Data Center Program, during which all of its data centers--not just those of the Veterans Health Administration, its largest division--get consolidated. That's expected to take six years, depending on funding. In the end, the agency will probably have six to eight data centers, though the final number remains in flux.

The VA's major data centers are a mix of owned and leased space. Moving ahead, the agency is leaning toward increased use of leased facilities because they can be implemented faster than those requiring new construction.

De Sanno is even contemplating use of containerized data centers as a faster, cheaper alternative to brick-and-mortar buildings. He visited Microsoft's data center in Chicago, which is constructed of containers that come loaded with blade servers and networking and cooling gear. In theory, such containers wouldn't even need to be inside a building. Says De Sanno, "We don't necessarily have to have a physical data center where somebody walks into a room."

It remains to be seen whether Uncle Sam can actually eliminate 800 data centers over the next five years, even as the IT requirements of virtually all agencies continue to grow. But the FDCCI plans are proof of their intentions--and will serve as a point of reference for holding them accountable.