How Government Can Make Open Data Work

Will 2014 finally become the year of open data? We're certainly seeing evidence that open data is moving from the margins into the mainstream, with new uses for data that governments and other sources are making freely available to the public.

But if we're going to see open data's promise fulfilled, it will be important for governments, and the federal government in particular, to make it easier for the public to access and use their open data.

I've described open data as "accessible public data that people, companies, and organizations can use to launch new ventures, analyze patterns and trends, make data-driven decisions, and solve complex problems." As Bethann Pepoli recently wrote in InformationWeek, cities around the US are starting to embrace open data as a tool for good government and civic innovation.

The federal government is doing the same. The Obama administration has begun implementing its Open Data Policy, announced last May, which calls for government agencies to make their data open by default and work with companies and nonprofits to put the data to use. The most recent example occurred last week with the White House hosting two days of "datapaloozas" to encourage entrepreneurs, developers, and nonprofit organizations to apply government safety and education data.

[Find out more about what data agencies have to offer. Read Data.gov Makes Agency Data More Accessible.]

I've participated in and studied the open data movement for the last three years. I've drawn on that experience and dozens of interviews for a new book I've written, Open Data Now. The book is meant to reach beyond a tech audience by illustrating how open data is impacting government policies and practices, innovation, consumer advocacy, and more. And as I learned in my research, open data's broadest impact may be as a kind of natural resource for business.

McKinsey and Company recently estimated the annual value of open data on the world economy could top $3 trillion. While McKinsey included open data from both public and private sources, government data is the largest source of open data. At the GovLab at New York University, where I am senior adviser, we're taking a different approach than McKinsey's to understand the evolving value of government open data: We're studying open data companies from the ground up. I'm now leading the GovLab's Open Data 500 project, funded by the John S. and James L. Knight Foundation, to identify and examine 500 American companies that use government open data as a key business resource.

Our preliminary results show that government open data is fueling companies both large and small, across the country, and in many sectors of the economy, including health, finance, education, energy, and more. But it's not always easy to use this resource. Companies that use government open data tell us it is often incomplete, inaccurate, or trapped in hard-to-use systems and formats.

It will take a thorough and extended effort to make government data truly useful. Based on what we are hearing and the research I did for my book, here are some of the most important steps the federal government can take, starting now, to make it easier for companies to add economic value to the government's data.

1. Improve data quality
The Open Data Policy not only directs federal agencies to release more open data; it also requires them to release information about data quality. Agencies will have to begin improving the quality of their data simply to avoid public embarrassment. We can hope and expect that they will do some data cleanup themselves, demand better data from the businesses they regulate, or use creative solutions like turning to crowdsourcing for help, as USAID did to improve geospatial data on its grantees.

2. Keep improving open data resources
The government has steadily made Data.gov, the central repository of federal open data, more accessible and useful, including a significant relaunch last week. To the agency's credit, the GSA, which administers Data.gov, plans to keep working to make this key website still better. As part of implementing the Open Data Policy, the administration has also set up Project Open Data on GitHub, the world's largest community for open-source software. These resources will be helpful for anyone working with open data either inside or outside of government. They need to be maintained and continually improved.

3. Pass DATA
The Digital Accountability and Transparency Act would bring transparency to federal government spending at an unprecedented level of detail. The Act has strong bipartisan support. It passed the House with only one dissenting vote and was unanimously approved by a Senate committee, but still needs full Senate approval and the President's signature to become law. DATA is also supported by technology companies who see it as a source of new open data they can use in their businesses. Congress should move forward and pass DATA as the logical next step in the work that the Obama administration's Open Data Policy has begun.

4. Reform the Freedom of Information Act
Since it was passed in 1966, the federal Freedom of Information Act has gone through two major revisions, both of which strengthened citizens' ability to access many kinds of government data. It's time for another step forward. Current legislative proposals would establish a centralized web portal for all federal FOIA requests, strengthen the FOIA ombudsman's office, and require agencies to post more high-interest information online before they receive formal requests for it. These changes could make more information from FOIA requests available as open data.

5. Engage stakeholders in a genuine way
Up to now, the government's release of open data has largely been a one-way affair: Agencies publish datasets that they hope will be useful without consulting the organizations and companies that want to use it. Other countries, including the UK, France, and Mexico, are building in feedback loops from data users to government data providers, and the US should, too. The Open Data Policy calls for agencies to establish points of contact for public feedback. At the GovLab, we hope that the Open Data 500 will help move that process forward. Our research will provide a basis for new, productive dialogue between government agencies and the businesses that rely on them.

6. Keep using federal challenges to encourage innovation
The federal Challenge.gov website applies the best principles of crowdsourcing and collective intelligence. Agencies should use this approach extensively, and should pose challenges using the government's open data resources to solve business, social, or scientific problems. Other approaches to citizen engagement, including federally sponsored hackathons and the White House Champions of Change program, can play a similar role.

Through the Open Data Policy and other initiatives, the Obama administration has set the right goals. Now it's time to implement and move toward what US CTO Todd Park calls "data liberation." Thousands of companies, organizations, and individuals will benefit.

Joel Gurin is the author of the book Open Data Now, editor of OpenDataNow.com, and senior adviser at The GovLab at NYU.

You can use distributed databases without putting your company's crown jewels at risk. Here's how. Also in the Data Scatter issue of InformationWeek: A wild-card team member with a different skill set can help provide an outside perspective that might turn big data into business innovation. (Free registration required.)