How Not To Plan A Data Center

Federal Data Center Consolidation Makes Progress

An entire floor at the National Geospatial-Intelligence Agency's new Technology Center has sat vacant for the past three years, the result of poor planning and communications and the challenge of forecasting an intelligence agency’s data storage needs. The project is getting rebooted with $54.6 million in defense funding, but it's anyone’s guess how it will end up.

The saga of NGA's Technology Center, located at Fort Belvoir, Va., is the subject of a Government Accountability Office report tied to the Pentagon’s ongoing Base Realignment and Closure (BRAC) efforts. As part of that consolidation, the NGA in 2006 began constructing several new buildings--its "Campus East," including a sprawling, 2.4 million-square-foot headquarters and the four-story, $155 million Technology Center.

The tech center blueprints originally called for 108,000 square feet of data center space, split between the third and fourth floors of the building. Within 12 months, that space requirement was cut in half, to 55,000 square feet, taking into account (belatedly) the efficiencies of consolidating systems on more densely configured servers. The idea was to establish 22,000 square feet of data center space on the fourth floor, and do the same on the third floor.

[ How efficiently are the feds managing IT investments? Read What's The ROI On Federal IT Spending? ]

Construction started in October 2007, but within a few months the IT contractor on the job, General Dynamics, determined that the third floor wouldn’t be needed to accommodate the sought-after 10 petabytes of storage capacity. General Dynamics explained that advances in server and storage technology made it possible to squeeze data into a smaller footprint.

With construction already underway, however, it was too late to omit an entire floor, and the facility was completed as planned in 2009. An NGA photo (below) taken in January of this year shows the squeaky clean space without a server rack or storage appliance in sight.

The project has been plagued with problems from the start. The NGA originally determined that it would need 8 petabytes of storage, and then revised that estimate to 10 petabytes. The agency has since upped its requirement to "hundreds of petabytes" over 10 years, which is the business case for the $54.6 million in funding that will now go toward outfitting the third-floor with raised floors, power, cooling, and lighting. The Army Corps of Engineers has issued a request for proposals for that work.

Along the way, the Department of Defense's right hand didn’t know what its left hand was doing, according to the GAO report. When the NGA, whose intelligence work is used by the U.S. military, determined that it no longer needed the third floor of the Technology Center, it failed to notify the U.S. Army, the official project manager. The Secretary of Defense and Congress were left out of the loop, too.

The project's shoddy documentation didn't help. DOD departments must provide a "Form 1391" in support of construction work, but the NGA didn't mention the Technology Center in its February 2006 filing for the Fort Belvoir campus, according to the GAO report. That part of the project was still in the "conceptual design phase," the agency later explained.

The origins of the project go back to 2005, when the NGA set out to build its Campus East to serve as a new home for 8,500 employees, as its other offices were being closed. Each year, the NGA drafted a business plan for the project, but it failed to make any mention of its decision to mothball the third floor of the tech center. It wasn’t until last year that officials with the Army's BRAC office, on a tour of the campus complex, realized that office space the size of a football field was empty.

The Army should have—and could have--been a more hands-on project manager, but Army officials assumed that the DOD's Office of the Deputy Under Secretary of Defense for Installations and Environment was reviewing and approving all changes in scope. For its part, the Office of the Deputy Under Secretary of Defense for Installations and Environment thought the Army was monitoring those things.

The silver lining in this debacle is that the NGA stood to save up to $37.2 million by not finishing the third floor. But any such savings were gobbled up by cost overruns elsewhere in the campus project. The BRAC Commission had pegged the NGA's consolidation and relocation costs at $1.1 billion over six years. The actual costs came in at more than $2.5 billion.

But all is not lost. What was on the verge of becoming yet another federal IT boondoggle now looks like a brilliant strategic plan. An internal study by the NGA, conducted from 2008 to 2010, identified a long-term need for hundreds of petabytes of data storage. Half of that capacity will be for the agency's internal needs, including its use of intelligence-collecting sensors in space and on the ground, and by a shift to cloud computing. The other half will be generated by other intelligence agencies, under a plan by the Office of the Director of National Intelligence (DNI) to share IT resources.

The third floor of the NGA's Technology Center is the perfect solution. Not only will it give the agency data storage headroom through 2020, but it will also let the NGA take back some of the IT workload now handled by a leased "interim transition capability" facility. The original contract for that facility was due to expire in fiscal 2013. But the agency, facing the prospect of growing data volumes with no place to put them, recently extended the contact another five years.

"The decision to build out the third floor of the tech center is an outgrowth of the vast increase in data handling, storage and dissemination capabilities that the agency needs to effectively support the workforce and our geospatial intelligence mission partners," an NGA spokesman said in response to my questions about the data center restart.

Documents on the Defense Comptroller's website show that the NGA was also looking to invest $9.3 million in its other primary data center, located in Arnold, Mo. That funding will go into power and cooling upgrades that are projected to support a 35% increase in data storage.

There are two big lessons here, aside from the obvious need for federal agencies to do a better job of project management. One is that pristine data center space can almost always be put to good use--if not now, then later. The second, related lesson: When estimating data storage requirements, take your first estimate and multiply by 10.

The Office of Management and Budget demands that federal agencies tap into a more efficient IT delivery model. The new Shared Services Mandate issue of InformationWeek Government explains how they're doing it. Also in this issue: Uncle Sam should develop an IT savings dashboard that shows the returns on its multibillion-dollar IT investment. (Free registration required.)