Big Data: Dead By Definition, Alive In Practice

Big data is at a crossroads. On one hand, big data is dead, the term having been used so often that it's been stripped of tangible value.

On the other hand, big data has never been so alive, as more companies than ever are trying to improve so-called big data analytics. How can such a dichotomy exist? The answer can be found in the enormous gap between what big data means by definition and what it really means in the important practice of data management.

Big data by definition
The term big data -- by the most commonly-used definition -- refers to data sets that are too large and complex to manage within traditional systems. This data is generally unstructured or semi-structured and require investment in new tools, technologies, skillsets, and team members to manage it.

[Data analysis is a do-or-die requirement for today's businesses. Read 16 Top Big Data Analytics Platforms.]

This is big data as it exists on paper, the end product of a meteoric hype cycle. This is not big data as it exists in practice, however. Analyst research and customer experiences suggest that in practice, what organizations and IT personnel refer to as big data is actually just data.

Big data in practice
In a recent TDWI research report, 88% of organizations cited structured, relational data as their primary big data type. This is traditional transactional and analytical data living in relational databases. Additional recent research from Enterprise Management Associates (EMA) shows that nearly half of organizations' big data projects are based on Oracle and/or SQL Server -- the traditional systems supposedly incapable of managing big data.

The same research shows that only 28% of organizations are concerned that their current systems cannot scale to meet the demands of their big data projects. 

My own experience in working with customers tells a similar story. Traditional relational databases and the data volumes housed in them serve as the bedrock of a surprisingly large number of big data projects. So despite the hype, in practice, many projects tagged with the big data label (and therefore supposedly too large and too complex for traditional systems) are actually built around data that is most often neither bigger nor more complex than what we've been working with for years, and that can most often be managed within the same systems we've been using for years.

Pitfalls of the hype
That's not to say the continued hype surrounding big data is all bad. It's certainly a bad thing for those companies that get too caught up in it. The pitfall of any hype cycle is that it drives companies to do things that don't actually solve their fundamental business problems. For example, I often hear from customers who invest in Hadoop, load their data, and then say "now what?" They've invested in a new technology -- because the hype led them to believe that's what they're supposed to do -- without knowing if it can actually help them.

EMA's recent survey found that stakeholder support and business strategy issues are the top two barriers preventing organizations from succeeding with their big data projects. Technology and infrastructure concerns were significantly smaller issues, according to the survey. This suggests that big data projects often start from the bottom up tied to a desire for IT innovation, rather than from the top down, tied to a desire to tackle significant business challenges.

When big data projects run into roadblocks, it's usually because business objectives aren't clear or the right people haven't been granted access to the right data. In other words, projects are not derailed because IT doesn't have the right technology, they're derailed because the company isn't aligned on what it's trying to accomplish in the first place, and that can only be rectified from the top down.

In reality, big data is generally not a technical challenge. Maybe Yahoo and Google needed to reinvent their infrastructure well beyond traditional capabilities, but most companies do not. Not yet, anyway.

Refocusing on a fundamental need
For companies that don't get misdirected by big data, however, the hype is a great thing. The big data trend has reawakened many organizations to the longstanding an fundamental need to become more data driven.

When the end goals are to solve a business problem and make better use of data inside and outside the organization, companies can often do so without investing in expensive new platforms or costly data scientists. Make no mistake: Many things have to happen for organizations to get the most out of their data. You must outline clear business objectives, IT and business leaders must be committed to collaborating, and the right people need to be granted access to the right data. It's just that most of those things have little or nothing to do with the mainstream definition of big data.

Awakened by the hype, but not caught up in it, smart companies are making sure data is in the right place at the right time, is shared by systems, and is available in reports that can be analyzed in ad-hoc fashion by business teams.    

The hyped-up definition of big data is dead, but the pursuit of making data the lifeblood of a business is more alive than ever.

WebRTC, wireless, video, unified communications, contact centers, SIP trunking, the cloud: All of these topics and more make up the focus for Enterprise Connect 2014, the leading conference and expo on enterprise communications and collaboration. Across four days, you'll meet thought- and market-leaders from across the industry and access the information you need to implement the right communications and collaboration products, services, software, and architecture for your enterprise. Find out more about Enterprise Connect and register now. It happens March 17-20.