Government agencies on the national, state and local levels want to build a means of effectively sharing and analyzing homeland security data. The full difficulty of that task will rival anything the government has ever accomplished.
Three years ago I talked to an IRS official about an ambitious project that would make it possible for people in the agency to instantly retrieve individual and business tax records. It was a conversation that taught me a lot about what a hugely difficult task data integration can be. Now, scores of government agencies on the national, state and local levels are trying to find ways to share and analyze data that relates to homeland security. If data integration at the IRS was a hard task, comprehensive integration and analysis of security data across all levels of government is going to be absolutely Herculean.
The database standardization project at the IRS, the official told me, involved pulling together information that resided in virtually every storage format known to mankind -- including mainframes, reel-to-reel tape, digital audio tapes and optical storage devices, among many others. The repositories used different means of data security, they were physically far-flung, and their ability to communicate with applications and each other was uneven or nonexistent. At the time, the agency projected it would take 10 years and $10 billion to complete its planned IT modernization initiatives.
Three years later, every time I see something on the news about efforts underway to make various security agencies work together and share information, I reflect back on that IRS interview. Homeland security involves many government departments and agencies, not just one. It involves many levels of government, not just the federal level. If the IRS's legacy systems were varied, the systems in place at the many entities that contribute to homeland security give new meaning to the word "diverse." I haven't even mentioned the varied formats of the data itself. And that's just the integration part: If the IRS's project involved actual analytics, it was trifling compared to what U.S. security imperatives will demand.
Some agencies, at least, are working to meet those demands. We hear about it piecemeal, but real efforts are underway. Law enforcement agencies around Washington are using wireless technology to integrate data from various jurisdictions. The FBI is deploying analysis software in its anti-terrorism efforts. And more money has been earmarked to secure the nation's ports. Not all the news is good: The Government Accountability Office said last month that the FBI lacks a coordinated plan to modernize its IT systems, and a report this month from the Department of Homeland Security's Inspector General office criticized the department for "not playing a lead role" in consolidating terrorist watchlist information.
The things we learn from the news media will inevitably make up only a tiny fraction of the whole story. That's as much a comment on the sheer scale of the job at hand as it is on the limitations of the media. As anyone familiar with data integration knows, there's a lot more to it than meets the eye. As anyone familiar with security knows, it's usually not a good idea to show your hand to your adversaries.
For all its infuriating inefficiency, arbitrary protocols and catty interagency rivalries, the U.S. government still managed to successfully partner with the private sector to build the Hoover Dam, the Liberty Ship fleet, the national highway system and the Apollo 11 lunar module. It absolutely is no exaggeration whatsoever to say that constructing a comprehensive means of accumulating, sharing and analyzing its myriad security data will be a task that rivals those. It will be carried out more quietly than the accomplishments that have preceded it. But it's got to be done.
The Agile ArchiveWhen it comes to managing data, donít look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
2014 Analytics, BI, and Information Management SurveyITís tried for years to simplify data analytics and business intelligence efforts. Have visual analysis tools and Hadoop and NoSQL databases helped? Respondents to our 2014 InformationWeek Analytics, Business Intelligence, and Information Management Survey have a mixed outlook.
Join us for a roundup of the top stories on InformationWeek.com for the week of December 7, 2014. Be here for the show and for the incredible Friday Afternoon Conversation that runs beside the program!