Big Data Tip: Don't Save Everything

Big data is all about gaining insights from very large and diverse volumes of information. But often organizations make the mistake of trying to collect every bit of data that's available to them, no matter how inconsequential.

This record-it-all approach is a waste of resources and money. A smarter solution is to decide beforehand what data is essential to your operation, and then take the necessary steps to collect, process, filter, and analyze it, says Joel Young, senior VP of research and development and CTO of Digi International, a machine-to-machine (M2M) solutions provider.

Founded in 1985, Digi International has evolved from a supplier of multi-port serial adapter cards for servers to a cloud platform for connecting devices. The company's cloud-based iDigi product, for instance, allows organizations to connect and manage device networks.

[ Big data has value. Accounting rules should reflect that. Read more at What's Your Big Data Worth? ]

In a phone interview with InformationWeek, Young said that companies are often overwhelmed by big data, particularly if they lack a clear definition of how they want to use it. Machine-to-machine communications, which may involve hundreds, if not thousands, of devices spread across a wide geographic area, can exacerbate this problem. "When you have a device that's sending information every second or minute, and you have a hundred thousand of these, you get a lot of data very quickly," said Young.

Some companies aren't confused by big data because they have a clear idea of what they want to do, and how they want to do it. "Others are lost," Young said.

To avoid the problem of having too much data -- much of which an organization may never analyze -- some big data soul searching is in order. "What problem are you trying to solve? You've got all this data, what do you want to do?" asked Young rhetorically. "A lot of times there's a whole lot of data you may not even need."

Once a company identifies the business problem it wants to solve, it can decide which data it needs, and establish rules for gathering that information. "One of the biggest problems I've found with big data is that people record way, way more than they need to," said Young.

A vending company that Digi International worked with recently had a big data problem with its old coin-operated vending machines. The firm had two major issues with its vending hardware, which totaled about 50,000 machines, many deployed in remote locations.

The first problem was that workers who collected coins from the machines would often keep some of the change for themselves. The second was that when the machines failed -- perhaps due to a tripped circuit breaker or a clogged hose -- the company didn't hear about the problem unless someone called in to report it.

"There are all kinds of data you could collect on these machines," Young said, but the company really needed to know just two things: the number of quarters inserted into the vending devices, and whether or not the machines were running.

Digi International set up a cellular system for the company's machines, which now report back to the cloud-based iDigi platform. Since cellular data can get expensive, particularly when each machine generates only about $12 a day in revenue, the system logs data to iDigi only once a day. "We charge $5 per month per machine, including the cellular service," said Young.

A new study by research firm IDC shows that only 3% of data today is tagged, and a scant 0.5% is analyzed. In addition, the volume of big data will nearly double every two years between today and 2020, reaching 40,000 exabytes, or 40 trillion gigabytes, in just 7 years, IDC predicts.

Predictive analysis is getting faster, more accurate and more accessible. Combined with big data, it's driving a new age of experiments. Also in the new, all-digital Advanced Analytics issue of InformationWeek: Are project management offices a waste of money? (Free registration required.)