5 Big Data Trends Government IT Can't Ignore - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Government // Big Data Analytics
12:43 PM
Connect Directly

5 Big Data Trends Government IT Can't Ignore

Government IT leaders must stop sitting on the big data sidelines, waiting to make a play.

Whether or not big data has yet reached its inflection point, it is clearly transformational for IT organizations and the enterprises they support. In this environment of rapid proliferation of big data, however, governments at levels from local to federal are still sitting on the sidelines, waiting to see what will happen next before making a play.

Instead, they should do what leading businesses are doing: become thoroughly knowledgeable about the five major trends that must be addressed by their IT organizations as they plan for the future.

[The next wave? Read Yahoo Talks Apache Storm: Real-Time Appeal.]

These rapidly advancing trends are challenging many fundamental assumptions about IT strategy and planning, and therefore have the potential to revolutionize how business is conducted and managed. In the same way, they offer governments opportunities to significantly improve their efficiency and performance as well as better serve their constituencies. So understanding these big-data trends is essential for governments wanting to prepare for the changes ahead.

1. Open-source is the future of big data. Enterprises are increasingly adopting open-source technologies as they see open-source software as a competitive differentiator that enables agility, mitigates risk, and lowers costs. In big data, open-source, led by Hadoop, is driving the most significant innovations. With low implementation costs and high adoption levels -- including direct support from trendsetting technology organizations such as Facebook, Twitter, Amazon, and LinkedIn -- open-source software is spreading. Emerging open-source frameworks and technologies of particular interest in big data include Storm, Kafka, and S4 for stream processing, Drill and Dremmel for large-scale querying, and R for statistical computing.

2. Hadoop is set to replace EDWs. Traditional enterprise data warehouses (EDWs), typically designed to house an enterprise's core data, are expensive and ill-equipped for solving big data problems. Data flows from operational systems (such as ERP, financial, and HR) into EDWs that in turn provide consistent and structured data for reporting and business uses.

Hadoop, on the other hand, is an open-source framework built around a high-volume distributed architecture that runs well on low-cost commodity hardware. This architecture and its associated languages and tools allow for solving complex analytics problems relatively quickly.

Hadoop is the ideal platform for analyzing ERP data in conjunction with disparate data sources. For example, a company could combine ERP information with sensor data, weather information, and transportation rates -- data sources with different structures or no structure at all -- to optimize the most cost-effective time and place to ship perishable products. All the loading, structuring, analysis, and reporting can be done directly and rapidly with Hadoop without moving data into or out of the EDW.

Some companies are already augmenting EDWs with Hadoop, offloading traditional ETL (extract, transform, load) functions and making use of the distributed processing capability. Others are using Hadoop to replace EDWs altogether. EMC and Teradata, among other major vendors, have already made bold moves into the Hadoop space.

3. Big data and analytics are increasingly embedded into devices. Hadoop's flexibility and open architecture make it a natural fit for embedding directly into devices from medical equipment to drones. Future "smart" devices will process data and conduct advanced analytics at the source, similar to the way mobile phones have transformed from simple handsets to minicomputers. Embedding Hadoop into devices will accelerate and streamline the collection and processing of high-volume data such as video and audio.

4. Software giants play big data catch-up. While they are unequivocally leaders in analytics, the world's leading software providers are laggards in adopting solid big data strategies. Even database leaders and visualization and business intelligence companies are challenged to bring products to market fast enough to keep pace with the changes.

With so much big data innovation occurring in the open-source space, commercial software companies must not be reactive. Traditional analytics software companies will still play a role, especially in large enterprises, but their leadership will take on a new meaning, and licensing costs could decrease dramatically as customers discover many low-cost options.

Universities, longtime training grounds for specialized skills in data and analytics software (especially SAS and SPSS), are embracing open-source tools, which are freely available and generally a better fit for the academic environment. This trend could radically change the nature of the entry-level market by providing candidates with skills that more closely match employers' increasing demand for open-source experts.

5. Siri is just the beginning. Science fiction has long captivated audiences with sketches of the ideal computing interface -- think Dave and Hal in 2001: A Space Odyssey, and Jarvis in Iron Man. Similarly for big data and real-time analytics, the tipping point -- both in a corporate sense and for society in general -- will be the day a non-technical, non-mathematical, non-engineering user can ask questions of the data without typing or using a mouse. This day is not far off. Siri, the iPhone's interactive personal assistant, was the first commercial success in this area. Experts think that by 2017, nearly two thirds of analytics vendors will incorporate voice recognition into their software.

These trends are truly disruptive and call into question the basic tenants of IT strategy and planning. To grasp the opportunities, forward-looking businesses are revisiting their big data plans with an eye toward this rapidly changing landscape.

Governments should be doing the same.

What will you use for your big data platform? A high-scale relational database? NoSQL database? Hadoop? Event-processing technology? One size doesn't fit all. Here's how to decide. Get the new Pick Your Platform For Big Data issue of InformationWeek Tech Digest today. (Free registration required.)

Christian P. Hagen is a Partner with A.T. Kearney's Strategic Information Technology Practice and is based in Chicago. He specializes in helping clients leverage information technology to increase efficiencies, improve customer relationships, and gain competitive advantage. ... View Full Bio
We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Ninja
10/27/2014 | 7:22:11 PM
Love the trends
The key takeaway for me is that data and our understanding of its utilization are going through a positive stage of maturity. The growth of Hadoop could definitely help here. According to a SAS survey, those using Hadoop were significantly more inclined to list low-cost hardware and storage as an actual benefit of Hadoop than were those who have yet to deploy it. At the same time 2 in 5 non-Hadoop users expect no benefits from Hadoop.

Peter Fretty
User Rank: Apprentice
10/24/2014 | 2:04:22 PM
Who Performs Well with Big Data?
You've mentioned some promising trends. Who do you think is performing well in taking advantage of these trends in government IT in the US or elsewhere?

Here are two examples I found from GovLoop:

  • Smarter Healthcare: Predicting the likelihood of hospitalization or death within 90 days, the Patient Care Assessment System (PCAS) calculates the Care Assessment Needs (CAN) Score. This score allows the Veterans Health Administration (VHA) to focus care teams and proactively care for their patients. The system collects 120 unique elements for 5.25 million patients and is supported by an 80-terabyte corporate data warehouse.
User Rank: Strategist
10/23/2014 | 7:35:13 AM
Big data is a game changer and has matured to a great extent
Great post Christian, big data is a game changer and has matured to a great extent, analytics-derived insights to key business operations including risk management and and a well designed data analytics strategy can strengthen internal controls and proactively manage risks which is one area governments have found hard to cover. I work for McGladrey and there's a whitepaper on our website that offers good information on the above topic that readers will find very informative. bit.ly/mcgldryanlytcs
10 Trends Accelerating Edge Computing
Cynthia Harvey, Freelance Journalist, InformationWeek,  10/8/2020
Is Cloud Migration a Path to Carbon Footprint Reduction?
Joao-Pierre S. Ruth, Senior Writer,  10/5/2020
IT Spending, Priorities, Projects: What's Ahead in 2021
Jessica Davis, Senior Editor, Enterprise Apps,  10/2/2020
White Papers
Register for InformationWeek Newsletters
Current Issue
[Special Report] Edge Computing: An IT Platform for the New Enterprise
Edge computing is poised to make a major splash within the next generation of corporate IT architectures. Here's what you need to know!
Flash Poll