Software // Information Management
Commentary
7/11/2012
11:28 AM
John Foley
John Foley
Commentary
Connect Directly
LinkedIn
Twitter
Google+
RSS
E-Mail
50%
50%

How Not To Plan A Data Center

The National Geospatial-Intelligence Agency built a bigger data center than it needed. Now the potential federal IT boondoggle is being turned into a storage center for data generated by remote sensors and cloud computing.

Federal Data Center Consolidation Makes Progres
Federal Data Center Consolidation Makes Progress
(click image for larger view and for slideshow)
An entire floor at the National Geospatial-Intelligence Agency's new Technology Center has sat vacant for the past three years, the result of poor planning and communications and the challenge of forecasting an intelligence agency’s data storage needs. The project is getting rebooted with $54.6 million in defense funding, but it's anyone’s guess how it will end up.

The saga of NGA's Technology Center, located at Fort Belvoir, Va., is the subject of a Government Accountability Office report tied to the Pentagon’s ongoing Base Realignment and Closure (BRAC) efforts. As part of that consolidation, the NGA in 2006 began constructing several new buildings--its "Campus East," including a sprawling, 2.4 million-square-foot headquarters and the four-story, $155 million Technology Center.

The tech center blueprints originally called for 108,000 square feet of data center space, split between the third and fourth floors of the building. Within 12 months, that space requirement was cut in half, to 55,000 square feet, taking into account (belatedly) the efficiencies of consolidating systems on more densely configured servers. The idea was to establish 22,000 square feet of data center space on the fourth floor, and do the same on the third floor.

[ How efficiently are the feds managing IT investments? Read What's The ROI On Federal IT Spending? ]

Construction started in October 2007, but within a few months the IT contractor on the job, General Dynamics, determined that the third floor wouldn’t be needed to accommodate the sought-after 10 petabytes of storage capacity. General Dynamics explained that advances in server and storage technology made it possible to squeeze data into a smaller footprint.

With construction already underway, however, it was too late to omit an entire floor, and the facility was completed as planned in 2009. An NGA photo (below) taken in January of this year shows the squeaky clean space without a server rack or storage appliance in sight.

The project has been plagued with problems from the start. The NGA originally determined that it would need 8 petabytes of storage, and then revised that estimate to 10 petabytes. The agency has since upped its requirement to "hundreds of petabytes" over 10 years, which is the business case for the $54.6 million in funding that will now go toward outfitting the third-floor with raised floors, power, cooling, and lighting. The Army Corps of Engineers has issued a request for proposals for that work.

Along the way, the Department of Defense's right hand didn’t know what its left hand was doing, according to the GAO report. When the NGA, whose intelligence work is used by the U.S. military, determined that it no longer needed the third floor of the Technology Center, it failed to notify the U.S. Army, the official project manager. The Secretary of Defense and Congress were left out of the loop, too.

The project's shoddy documentation didn't help. DOD departments must provide a "Form 1391" in support of construction work, but the NGA didn't mention the Technology Center in its February 2006 filing for the Fort Belvoir campus, according to the GAO report. That part of the project was still in the "conceptual design phase," the agency later explained.

The origins of the project go back to 2005, when the NGA set out to build its Campus East to serve as a new home for 8,500 employees, as its other offices were being closed. Each year, the NGA drafted a business plan for the project, but it failed to make any mention of its decision to mothball the third floor of the tech center. It wasn’t until last year that officials with the Army's BRAC office, on a tour of the campus complex, realized that office space the size of a football field was empty.

source: NGA

The Army should have—and could have--been a more hands-on project manager, but Army officials assumed that the DOD's Office of the Deputy Under Secretary of Defense for Installations and Environment was reviewing and approving all changes in scope. For its part, the Office of the Deputy Under Secretary of Defense for Installations and Environment thought the Army was monitoring those things.

The silver lining in this debacle is that the NGA stood to save up to $37.2 million by not finishing the third floor. But any such savings were gobbled up by cost overruns elsewhere in the campus project. The BRAC Commission had pegged the NGA's consolidation and relocation costs at $1.1 billion over six years. The actual costs came in at more than $2.5 billion.

But all is not lost. What was on the verge of becoming yet another federal IT boondoggle now looks like a brilliant strategic plan. An internal study by the NGA, conducted from 2008 to 2010, identified a long-term need for hundreds of petabytes of data storage. Half of that capacity will be for the agency's internal needs, including its use of intelligence-collecting sensors in space and on the ground, and by a shift to cloud computing. The other half will be generated by other intelligence agencies, under a plan by the Office of the Director of National Intelligence (DNI) to share IT resources.

The third floor of the NGA's Technology Center is the perfect solution. Not only will it give the agency data storage headroom through 2020, but it will also let the NGA take back some of the IT workload now handled by a leased "interim transition capability" facility. The original contract for that facility was due to expire in fiscal 2013. But the agency, facing the prospect of growing data volumes with no place to put them, recently extended the contact another five years.

"The decision to build out the third floor of the tech center is an outgrowth of the vast increase in data handling, storage and dissemination capabilities that the agency needs to effectively support the workforce and our geospatial intelligence mission partners," an NGA spokesman said in response to my questions about the data center restart.

Documents on the Defense Comptroller's website show that the NGA was also looking to invest $9.3 million in its other primary data center, located in Arnold, Mo. That funding will go into power and cooling upgrades that are projected to support a 35% increase in data storage.

There are two big lessons here, aside from the obvious need for federal agencies to do a better job of project management. One is that pristine data center space can almost always be put to good use--if not now, then later. The second, related lesson: When estimating data storage requirements, take your first estimate and multiply by 10.

The Office of Management and Budget demands that federal agencies tap into a more efficient IT delivery model. The new Shared Services Mandate issue of InformationWeek Government explains how they're doing it. Also in this issue: Uncle Sam should develop an IT savings dashboard that shows the returns on its multibillion-dollar IT investment. (Free registration required.)

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Andrew Hornback
50%
50%
Andrew Hornback,
User Rank: Apprentice
7/26/2012 | 2:26:23 AM
re: How Not To Plan A Data Center
Last time I looked, government contracts went to the lowest bidder... whether it's a nuclear warhead, data center or carton of toilet paper.
Sam Iam
50%
50%
Sam Iam,
User Rank: Apprentice
7/20/2012 | 8:37:00 AM
re: How Not To Plan A Data Center
No one can be as committed to building ships or planes as they are to building data centers and implementing IT systems. I know that the defense contractors have the capability, but why would anyone use them for something so far from their core competency. Does anyone want to make the case that this agency fairly and unbiasedly looked at IBM, HP, and General Dynamics... and then decided General Dynamics' IT and data center credentials are superior to the previous two? If this were about the defense contractors wanting to seriously expand into IT because they are committed to IT, as opposed to leveraging their contacts to sell the Feds whatever they can sell them, we would see all of these firms in the commercial, non-Federal sector, IT market as we do with IBM and HP.
Andrew Hornback
50%
50%
Andrew Hornback,
User Rank: Apprentice
7/20/2012 | 3:11:26 AM
re: How Not To Plan A Data Center
I think one of the major things that this story illustrates (aside from the obvious need for a better project management process - lack of communication/documentation in project management is unforgiveable), is that technology marches on, period.

True, there were some major balls dropped here, but storage technology keeps marching forward. Compare the floor space required for 10 Petabytes of Enterprise-class storage at the start of this project with the end date. By the same respect, the amount of data storage required by the organization grows at a rate, whether similar or dissimilar.

This isn't a situation that can't be dealt with - and who knows, had the project been built smaller, would the American taxpayer be later building another facility for this entity to expand into at a higher cost?

Andrew Hornback
InformationWeek Contributor
Andrew Hornback
50%
50%
Andrew Hornback,
User Rank: Apprentice
7/20/2012 | 3:04:24 AM
re: How Not To Plan A Data Center
Sam - pretty much every major defense contractor also has IT contracting capabilities. General Dynamics, Northrup Grumman, Lockheed Martin, Boeing - all have very capable IT divisions that support both internal and external entities.

Andrew Hornback
InformationWeek Contributor
MyW0r1d
50%
50%
MyW0r1d,
User Rank: Strategist
7/16/2012 | 7:56:43 PM
re: How Not To Plan A Data Center
You would be surprised then Sam of their IT presence by visiting their website. They are the general defense contractor, what they don't already have or provide, they are willing to buy, build, or subcontract to seal a government contract (a document they know intimately). Even their acronym (General Dynamics or general defense) hints at their deep commitment to the cause.

If ever there was a place to build such a megacomplex under such "flexible" controls, where better than the greater DC/Virginia/Maryland area to find government agencies that could be suitable tenants and cover poor (absent) planning. Change the destination to office or warehouse space while modernization programs are undertaken or continue the buildout for a consolidated, government cloud computing host in support of the Shared Services mandate.
Sam Iam
50%
50%
Sam Iam,
User Rank: Apprentice
7/16/2012 | 4:43:05 AM
re: How Not To Plan A Data Center
A bunch of problems, but the one that sticks out is: Why is General Dynamics working on this project instead of an IT firm? They are a shipbuilder and defense contractor.
The Agile Archive
The Agile Archive
When it comes to managing data, donít look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest - September 10, 2014
A high-scale relational database? NoSQL database? Hadoop? Event-processing technology? When it comes to big data, one size doesn't fit all. Here's how to decide.
Flash Poll
Video
Slideshows
Twitter Feed
InformationWeek Radio
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.