Big Data Talent War: 7 Ways To Win

IBM and Oncor, Texas' largest electric distribution company, are using big data technology to manage an intelligent power grid that will run more than 3.2 million smart electrical meters in the state by the end of the year.

The advanced metering system gives Oncor near-real-time information on billions of data measurement points, including machine-to-machine (M2M) sensors, advanced meters, and power lines. The project also provides a Web-based interface where Oncor customers can monitor their energy usage.

In a phone interview with InformationWeek, IBM director of database software and systems Bernie Spang said that Big Blue became involved with Oncor's smart grid project two years ago. "Oncor was looking to expand its early pilot program with smart meters, and to build out a smart energy solution that gives them the ability to better optimize supply and demand, and to provide information back to their customers through the Web," Spang said.

[ Related: How Big Data Can Help Utilities Avoid Blackouts. ]

When IBM came on board, Oncor was expanding its project to more than 3 million advanced meters, each of which sends information every 15 minutes to Oncor's offices -- a dramatic increase over a traditional electric meter with its monthly readings. The logistics posed a classic big-data challenge. How could Oncor process and store all of that data in a cost-effective way?

"You're talking about a nearly three thousand times increase of data transactions when you go from once a month to every 15 minutes," said Spang.

Oncor needed to upgrade from its relational database management system, which was efficient at managing modest volumes and velocities of data, but was too inefficient for the new smart grid. "Relational database management is rows and columns, and tables of data that are interconnected," said Spang. "It's not very efficient for gathering and managing smart meter and sensor data, which is time-stamped and comes in intervals," he said.

So Oncor installed IBM's Informix database software to run its smart grid. The new Informix system loaded meter data 20 times faster, and performed queries up to 30 times faster than the system it replaced, IBM claims. In addition, it reduced data storage needs by 70%, an improvement that remained linear over time and therefore allowed Oncor to accurately predict its storage requirements.

Informix is a structured data management system that uses a time-series data model rather than a conventional relational model. This results in dramatic storage savings, IBM says. "If the system is designed to think in terms of time-series data, you only have to store the new data points when the data changes," Spang said. For example, if a smart meter delivers data updates every 15 minutes, but gives the same reading for 10 hours straight, there's no reason to store the same reading 40 times. If you need to run an analysis of the data, the system is smart enough to fill in the information -- which hasn't changed over the 10-hour period -- even though it didn't store the data initially.

The smart grid market is expected to grow significantly in the next decade, up from $4.9 billion in 2011 to $43.3 billion in 2020, according to a 2011 study by IBM. By 2015, an estimated 300 million smart meters will be in use worldwide, the study says.

Obviously, the world is becoming increasingly instrumented with meters and sensors, and that's one reason why IBM is bullish on Informix's time-series data model. "The technology will become increasingly important as folks start using data that's coming from all of these instruments," Spang said.

At this hands-on Wall Street & Technology Virtual Event, Big Data On Wall Street, experts and solution providers will offer detailed insight into how risk management, financial reporting, trading analytics and financial modeling, along with a host of other opportunities, can all benefit from applying big data techniques and technologies to business processes. When you register, you will gain access to live and on-demand webcast presentations, as well as virtual booths packed with free resources. It happens Nov. 8.