How To Fix The GIS Data Mess

Used to be that only local governments and shipping companies cared about geographic information systems. But now, the market is telling us that GIS is a "can't live without" feature of not only place-based applications like Google Maps, but a mainstay of federal economic stimulus programs.

There's only one problem: GIS is fundamentally broken.

Here's an example: 30% of the trails in the woods behind my house were recently destroyed by a school construction project. If GIS had kept pace with other information technologies and practices, trail data would have been available to local government agency planners, and allowances could have been made prior to construction. In this case, 40 feet would have made a pretty significant difference. Make no mistake: Effective, usable GIS has the ability to affect you, literally right in your backyard.

It’s too bad GIS has been a train wreck for years. To understand why that is, imagine that IANA and ICANN don’t exist to provide policy guidance and domain name system implementation. Instead, every organization, large and small, keeps tables of their own IP addresses and Web servers, in varying formats. Governments and universities just suffer with this situation. Finally, a large company, in the spirit of innovation, gloms on to what the authors of modern DNS figured out: The Internet's not real usable if there's not a federated system for host-to-name lookups. But this corporation decides that everyone must sign on to their terms of use if they want access to this new federated address system.

Frankly, this would make for a pretty bad Internet, because we’d all have to abide by one private organization's profit-driven licensing and usage guidelines. Many would choose simply not to buy into the service. Significant advances in the field of federated name-to-address lookup that served the public good would be stifled, because there's a bottom line to consider. It's not private industry's role to provide for the public good; it's private industry's role to innovate to the point where they attract customers and maximize profit.

I don't care how "not evil" this corporation is; it’s not going to release its proprietary algorithms and reference code into the public domain without a clear strategic (read: financial) motive.

That's the situation with GIS. There are thousands of data sources for geographic information systems. Look no further than your local municipality or county. The problem is, there's no universal standard, and even if there were, there's no widespread, non-proprietary way to federate that data. Silo systems are an egregious 1980’s era enterprise architecture issue, and we're all paying for it. It's easy to blame ESRI or Google for this state of affairs, and they certainly have played a part, but this is a case where we all—federal, state, and local government; higher education; and private industry—need to step up and ask how to fix the problem. Here’s what each of us must do:

Federal Government’s Role: According to an article by Dr. Christopher Tucker of the US Geospatial Intelligence Foundation, the problem started in President Nixon's era, when Nixon failed to approve a proposed agency to centrally coordinate mapping and other tasks that we'd classify as GIS nowadays. This was an "aha!" moment. I've worked with some pretty awesome GIS professionals, and the secret sauce seems to be to treat GIS as a multi-agency, collaborative sport in need of strong governance.

I've always wondered why the feds didn't get that. Now I know. To quote the good doctor, "The federal government lacks a functioning governance or operational structure to coordinate, deploy or utilize spatial data in a manner that could make ‘place-based’ decision making effective."

The federal government needs to prioritize governance, but it also must fund research that will provide for a federated GIS model.

State & Local Governments’ Role: This can be summed up in two words: technocratic isolationism. Over the years, I've observed that many local government GIS professionals don't prioritize process over technology, earning well-deserved labels as technocrats. In this multi-agency world of ours, everybody's got a piece of location-based data.

Enterprise architects know that you need governance when you have stewardship over data that multiple folks create and own. As a result of struggles brought on due to lack of process or governance, local governments spend more time in mortal combat than they need to. Smaller governments also have few economies of scale, while state governments tend to have stakeholders with massively inflated expectations.

All of this adds up to isolationism--the belief that any substantial collaboration with agencies that don't immediately border theirs is useless and a waste. To be fair, there are some relatively successful statewide GIS clearinghouse efforts. But they can't succeed on their own.

State and local governments need to prioritize process, governance, and collaboration.

Higher Education's Role: I'm not an expert on higher education, but I do know that lots of primary IT research and reference implementations used to come out of universities. Nowadays, however, universities seem to focus on "incubators" and other vehicles to serve the private sector.

A well-known university president went on NPR about a year ago to beg bright people to come work for universities instead of taking gold-plated private research jobs. I get his point about needing the best and the brightest to be able to produce the best serendipitous finds, which tend to result from studies that don't always necessarily have a poin--research unconstrained by immediate usefulness. And you absolutely need good and open research to solve GIS enterprise architecture issues.

Universities need to get over their lack of self esteem, figure out how to attract the best and brightest, and fund primary research without intellectual property being tied up or locked away by the private sector.

The Private Sector’s Role: As that university president bemoaned, the private sector has been able to lure incredibly bright and talented innovators. But, again, there's a profit motive. I've praised Google in the past for being enlightened about letting customers access their own data via various APIs. But that doesn't mean that Google wants to release its proprietary algorithms or reference code to the world, nor should it. It's ridiculous to blame the private sector for chasing profit. If private industry has a problem, it's self-deception: "Sure, anybody can use our services."

Right, except for those terms of use that say, "You acknowledge and agree that Google (or Google's licensors and their suppliers, as applicable) own all legal right, title and interest in and to the Service and Content."

Local governments are not going to sign off on those terms. Their taxpayers bought and paid for the content that you've included in your service. It wouldn't be responsible to acknowledge and agree that the content belongs to Google.

Private industry needs to be more upfront about just how proprietary their GIS services are, and quit pretending that universities don't have a role in primary research.

Ideally, the private sector should run innovative place-based applications using GIS data. Universities should produce key public innovations that benefit all. Maybe those innovations need a little private sector magic to make them user friendly. And government should continue to be the steady force behind the scenes, collecting and maintaining the place-based data that is so critical to good decision-making. The ultimate solution to the GIS train wreck requires both policy and technology. Policy's hard, but sometimes, good technical solutions make policy easier to formulate. There's a significant opportunity for a smart enterprise architect with GIS leanings to figure out how states, cities, and counties can easily federate their data then provide a reference implementation, ask others to pound on it, make it better, and share the federated love.

Here's a shortlist of needed features:

Make it distributed and fault tolerant. One of the beauties of DNS is that one dead DNS server doesn't break the Internet.

Include ETL/transformation capabilities. It's unrealistic to think that people will give up their various data formats or convert everything. You've got to be able to transform the data as you federate it.

Make it extensible. People will want to store data that you have not even considered. Nobody stored LDAP data in DNS back when I first worked with it, because LDAP didn't exist.

Make it easy, and if you can't make it easy, at least make it open. Open will mean that the private sector can fill gaps, create portals, and so on.

Above all, remember that just as IT professionals would have never believed that the Internet would grow to the point where your grandmother relied on it for stock tips, geospatial services will continue to blossom in ways we can't imagine.

In the future, not all useful location data will be entered by GIS professionals with $5,000 accurate-to-within-an-inch GPS units. Some of it will come from people with skin in the game, including those who want to avoid seeing more trails get plowed over.

Jonathan Feldman is an InformationWeek Analytics contributor. He's incredibly grateful to whichever agency or neighbors reblazed the trails near his house. Write to him at [email protected]

InformationWeek Analytics has published a guide to the Open Government Directive and what it means for federal CIOs. Download the report here (registration required).