Government // Big Data Analytics
Commentary
12/10/2013
10:01 AM
Joel Westphal
Joel Westphal
Commentary
100%
0%

The War On Military Records

The era of big data has arrived on the battlefield and we need to find new ways to deal with it.

When I took over the responsibilities as chief of records management for US Central Command in May 2009, one of my primary responsibilities was to oversee the records for the Joint Task Force Headquarters, US Forces-Iraq and US Forces-Afghanistan. For the next four years, my life became devoted to capturing, preserving, and then organizing what I believe is one of the most important document collections from the early twenty-first century, the Operation Iraqi Freedom Collection.

In the days following the end of combat operations in Iraq in September 2010, the media focused on a variety of important topics: What will Iraq look like in the post-war? How will we handle the troops coming home? Determining the costs of the war both in human and economic terms.

Little attention was paid, though, to the documents that will be crucial to future historians and others who will debate the conduct of the conflict, looking through an historical lens rather than media's present-day view on current events. Military analysts and historians require primary source material, from communiques (which are often emails), to reports drafted in the field, to the orders issued to units to accomplish a given task. Without these records a true history of Second Iraq War, a conflict lasting longer than the Second World War, would not be possible.

[What's your big data challenge? Read: How NASA Manages Big Data]

Since the end of the first Gulf War in 1991, modern historians have struggled to write about that conflict. In fact, a review of historical texts available on Amazon turned up only a few first-hand accounts and other oral histories of this conflict, and the question is "why?" The answer lies into what occurred shortly after the war ended.

In a rush to get home, our troops and their leadership orchestrated what was possibly the single largest destruction of records in our nation's history. Millions of records were either burned or simply left in the desert to literally be buried by the sands of time. This action (which is a matter of public record) is little known or spoken about outside of the National Archives and the Department of Defense (DoD).

It was never covered by the news media and if it weren't for post-war efforts to deal with Gulf-War Syndrome by the Veterans Administration, even more records may well have been destroyed. As tragic as this was, the lessons learned from this shortsightedness allowed us to press leaders in Iraq and Afghanistan into action to avoiding making the same mistake.

The continuity and preservation of our modern democracy in large part relies on records, with the Constitution and Declaration of Independence serving as examples of the ultimate primary source documents. Without preserving records, the elements of our historical memory are lost.

The loss of a primary source record is especially alarming. A primary source record has no bias. It is not an opinion, but is an artifact of what has taken place. The document that orders a unit to take a hill, a piece of the desert or a town, or even how to treat wounded prisoners may prove to be more valuable than 5 to 10 oral histories on the same topic because verbal testimony can be construed as hearsay, while a record can be a smoking-gun.

The Second Iraq War (or the "Second Gulf War" or "Bush's War" -- whichever name future historians will choose) may also be known as the first Digital War in our nation's history. More than 99% of all the records created during the conflict were born digitally and resided only in digital form, thus no paper copies were available.

This was a dramatic paradigm shift from our last major conflict where the vast majority of the records were created in paper form. This shift has made records far more vulnerable than they were in the past as the destruction of paper is far more difficult than simply hitting the delete key.

Previous
1 of 2
Next
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
<<   <   Page 2 / 2
Joel5171
50%
50%
Joel5171,
User Rank: Apprentice
12/10/2013 | 3:14:51 PM
Re: Terabyte Limit Doesn't Compute
I cannot speak for rules and regulations from the National Archives. There are permanent records and temporary records. NARA usually does not want temporary records.

However, I think for the next conflict (and lets hope that day never comes) records issue there should be some discussion on keeping it all. But understand this opens up some serious issues for FOIA and eDiscovery.

Also, the 54 terabytes does not include the records from the Army, Navy and Air Force who collect their own service records. It also does not include other classified records.
Laurianne
50%
50%
Laurianne,
User Rank: Author
12/10/2013 | 2:55:11 PM
Re: Terabyte Limit Doesn't Compute
Doug makes a good point. Also, how did plentiful cloud storage options factor into the decision process?
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
12/10/2013 | 2:30:05 PM
Terabyte Limit Doesn't Compute
In the context of big data, 54 terabytes is no big deal. I don't get why the National Archives has cut off what they will save at 14 terabytes. Why not save it. You never know when some future revelation or future form of analysis renders that information relevant and useful. Geez, you can find 2 TB and 3 TB external hard drives for as little as $100. Why do we have to make any hard choices here. Save everything! History won't forgive you for repeting the same mistakes you described after the first Gulf War.
Joel5171
50%
50%
Joel5171,
User Rank: Apprentice
12/10/2013 | 12:15:20 PM
Re: Preserving Public Records
This is a great question! Given that the article had to be of a certain length a lot of what I wanted to say could not be included. This was one issue that I could not fit it.

A File Analysis tool is really just one answer to a question that requires multiple answers. For example, you still need an Electronic Records Management Application (ERMA) to ensure records are kept for as long as their disposition requires.

These tools help identify (and quickly) what and where the information is so you can then apply retention on them. They quickly assist the archivist or records manager in cleaning up those cluttered shared drives and SharePoint repositories that are out there.

I still hope to publish a more expansive article in either the American Archivist or Archivaria which really details how the entire operation to recover, preserve and then identify the records took place.
WKash
100%
0%
WKash,
User Rank: Author
12/10/2013 | 10:37:04 AM
Preserving Public Records
Thanks for sharing your experiences -- and the dilemma facing government records archivists.  Given the immense volumes of records generating daily, it is impossible to comprehend how individuals could sort through and manage all these records anymore.  My question is, are these content classification tools potentially breeding a false sense of security that they are identifying all the records that, by statute, must be preserved?

 
<<   <   Page 2 / 2
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest, Nov. 10, 2014
Just 30% of respondents to our new survey say their companies are very or extremely effective at identifying critical data and analyzing it to make decisions, down from 42% in 2013. What gives?
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on InformationWeek.com for the week of November 16, 2014.
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.