Big Data Tip: Don't Save Everything - InformationWeek
Data Management // Big Data Analytics
10:53 AM
Connect Directly
Ransomware: Latest Developments & How to Defend Against Them
Nov 01, 2017
Ransomware is one of the fastest growing types of malware, and new breeds that escalate quickly ar ...Read More>>

Big Data Tip: Don't Save Everything

No wonder you're overloaded by ever-growing volumes of data. You're saving too much, advises Digi International exec.

13 Big Data Vendors To Watch In 2013
13 Big Data Vendors To Watch In 2013
(click image for larger view and for slideshow)
Big data is all about gaining insights from very large and diverse volumes of information. But often organizations make the mistake of trying to collect every bit of data that's available to them, no matter how inconsequential.

This record-it-all approach is a waste of resources and money. A smarter solution is to decide beforehand what data is essential to your operation, and then take the necessary steps to collect, process, filter, and analyze it, says Joel Young, senior VP of research and development and CTO of Digi International, a machine-to-machine (M2M) solutions provider.

Founded in 1985, Digi International has evolved from a supplier of multi-port serial adapter cards for servers to a cloud platform for connecting devices. The company's cloud-based iDigi product, for instance, allows organizations to connect and manage device networks.

[ Big data has value. Accounting rules should reflect that. Read more at What's Your Big Data Worth? ]

In a phone interview with InformationWeek, Young said that companies are often overwhelmed by big data, particularly if they lack a clear definition of how they want to use it. Machine-to-machine communications, which may involve hundreds, if not thousands, of devices spread across a wide geographic area, can exacerbate this problem. "When you have a device that's sending information every second or minute, and you have a hundred thousand of these, you get a lot of data very quickly," said Young.

Some companies aren't confused by big data because they have a clear idea of what they want to do, and how they want to do it. "Others are lost," Young said.

To avoid the problem of having too much data -- much of which an organization may never analyze -- some big data soul searching is in order. "What problem are you trying to solve? You've got all this data, what do you want to do?" asked Young rhetorically. "A lot of times there's a whole lot of data you may not even need."

Once a company identifies the business problem it wants to solve, it can decide which data it needs, and establish rules for gathering that information. "One of the biggest problems I've found with big data is that people record way, way more than they need to," said Young.

A vending company that Digi International worked with recently had a big data problem with its old coin-operated vending machines. The firm had two major issues with its vending hardware, which totaled about 50,000 machines, many deployed in remote locations.

The first problem was that workers who collected coins from the machines would often keep some of the change for themselves. The second was that when the machines failed -- perhaps due to a tripped circuit breaker or a clogged hose -- the company didn't hear about the problem unless someone called in to report it.

"There are all kinds of data you could collect on these machines," Young said, but the company really needed to know just two things: the number of quarters inserted into the vending devices, and whether or not the machines were running.

Digi International set up a cellular system for the company's machines, which now report back to the cloud-based iDigi platform. Since cellular data can get expensive, particularly when each machine generates only about $12 a day in revenue, the system logs data to iDigi only once a day. "We charge $5 per month per machine, including the cellular service," said Young.

A new study by research firm IDC shows that only 3% of data today is tagged, and a scant 0.5% is analyzed. In addition, the volume of big data will nearly double every two years between today and 2020, reaching 40,000 exabytes, or 40 trillion gigabytes, in just 7 years, IDC predicts.

Predictive analysis is getting faster, more accurate and more accessible. Combined with big data, it's driving a new age of experiments. Also in the new, all-digital Advanced Analytics issue of InformationWeek: Are project management offices a waste of money? (Free registration required.)

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
2017 State of IT Report
In today's technology-driven world, "innovation" has become a basic expectation. IT leaders are tasked with making technical magic, improving customer experience, and boosting the bottom line -- yet often without any increase to the IT budget. How are organizations striking the balance between new initiatives and cost control? Download our report to learn about the biggest challenges and how savvy IT executives are overcoming them.
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll