Big Data Debate: End Near For Data Warehousing? - InformationWeek
Data Management // Big Data Analytics
10:41 AM
Connect Directly
Building Security for the IoT
Nov 09, 2017
In this webcast, experts discuss the most effective approaches to securing Internet-enabled system ...Read More>>

Big Data Debate: End Near For Data Warehousing?

Some Hadoop advocates say this new platform will unseat the relational data warehouse from its dominant role in BI. Database champions say 'not so fast!' Share your opinion.

The enterprise data warehouse (EDW) is the backbone of analytics and business intelligence for most large organizations and many midsize firms. The tools and techniques are proven, the SQL query language is well known, and there's plenty of expertise available to keep EDWs humming.

The downside of many relational data warehousing approaches is that they're rigid and hard to change. You start by modeling the data and creating a schema, but this assumes you know all the questions you'll need to answer. When new data sources and new questions arise, the schema and related ETL and BI applications have to be updated, which usually requires an expensive, time-consuming effort.

Enter Hadoop, which lets you store data on a massive scale at low cost (compared with similarly scaled commercial databases). What's more it easily handles variety, complexity and change because you don't have to conform all the data to a predefined schema.

That sounds great, but where do you find qualified people who know how to use Pig, Hive, Scoop and other tools needed to run Hadoop? More importantly, how do you get fast answers out of a batch-oriented platform that depends on slow and iterative MapReduce data processing?

Will Hadoop supplant the enterprise data warehouse and relegate relational databases to data mart roles? Or is Hadoop far too green and too slow to change the way most people work? In our debate, Scott Gnau of Teradata and Ben Werther of Platfora square off. Share your opinion using the comment tool at the end of the article.

For The Motion

Ben Werther
Ben Werther
Founder & CEO, Platfora

The EDW Is A Relic

The proposition of the enterprise data warehouse seems tantalizing -- unifying all the data in your enterprise into one perfect database.

So you start an 18-month journey to find important data sources, agree on the important business questions, map the business processes, and architect and implement it into the one database to rule them all.

And when you are done, if you ever finish, you have a calcified relic of the world 18 months prior. If your world hasn't changed much in 18 months, then that might be ok. But that isn't the reality in any large business I've encountered.

Why is Hadoop was gaining so much momentum? Clearly it's cost-effective and scalable, and it's intimately linked in people's minds to companies like Google, Yahoo and Facebook. But there's more to it. Everywhere I looked, companies are generating more and more data -- interactions, logs, views, purchases, clicks, etc. These were being linked with increasing numbers of new and interesting datasets -- location data, purchased user demographics, Twitter sentiment, etc. The questions that these swirling data sets could one day support can't be known. And yet to build a data warehouse, I'd be expected to perfectly predict what data would be important and how I'd want to question it, years in advance, or spend months rearchitecting every time I was wrong. This is actually considered "best practice."

The brilliance of what Hadoop does differently is that it doesn't ask for any of these decisions up front. You can land raw data, in any format and at any size, in Hadoop with virtually no friction. You don't have to think twice about how you are going to use the data when you write it. No more throwing away data because of cost, friction or politics.

And yet, in the view of the status-quo players, Hadoop is just another data source. It is a dumping ground, and from there you can pull chunks into their carefully architected data warehouses -- their system of record." They'll even provide you a ‘connector' to make the medicine go down sweet. Sure, you are back in the land of consultants and 12-18 month IT projects.

But let's go through the looking glass. The database isn't the "system of record" -- it is just a shadow of the data in Hadoop. In fact there is nothing more authentic than all of that raw data sitting in Hadoop. But machinery has been missing to complete the story, namely a way to do interactive business intelligence, exploration and analysis against the data in Hadoop. Platfora is among the vendors working on this need.

Imagine what this means. Raw data of any kind or type lands in Hadoop with no friction. And without building a data warehouse, without the pain of ETL integration, and without any other IT project, everyday business users can put that data to work immediately. The machinery to support this is now appearing, and users' ability to harness data is undergoing a generational shift.

There is no longer a need for a traditional data warehouse. It is an inflexible, expensive relic of a bygone age. It is time to leave the dark ages.

Ben Werther is the Founder & CEO of Platfora, the company behind the first in-memory business intelligence platform for Hadoop. He is an industry veteran and big data thought leader and was head of products at Greenplum through the EMC acquisition.

Against The Motion

Scott Gnau
Scott Gnau
President, Teradata Labs

EDWs Will Thrive

Some people suggest that relational database management systems (RDBMS), and data warehouse built on top of them, are no longer needed. In fact, some argue that new technologies like Hadoop can do the job of the Data Warehouse at a fraction of the time and cost -- and, by the way, Hadoop is "free."

We can't blame some for wanting to believe the argument.

Before hitting the arguments, let me say that Hadoop has an important part in the future analytics environment because it provides a big data refinery, which can bring in massive amounts of raw material (data) -- and more importantly the corresponding analytics. One of the great features of Hadoop is that you can pile information into it without deciding in advance what you need to save or how you intend to use it. As businesses require more precise analytics, Hadoop as a source of new fuel is critical.

The core argument really comes down to a couple of points: 1. Data Warehouses are too "rigid and inflexible," and 2. The "community" will fix all of the limitations of Hadoop.

On the surface, these points sound very compelling. But with a deeper look they are misleading and self-contradictory.

Starting with the point about inflexibility of data warehouses, it's important to distinguish the technology, RDBMS, from the practice, data warehousing. Rigid schemas attributed to EDWs -- where the users have to define what they are looking for before starting the search, and where some of the misconceptions stem -- are often the result of rigid IT policy, and sometimes the result of dated or inadequate data warehouse architecture. Rigid structures are not an inherent problem in today's best data warehouse architectures that are designed for analytics.

Is structure bad in analytic environments? No! Imagine what would happen if you ran a public company and every quarter an analyst had to go through piles of un-modeled data, whether in Hadoop or otherwise, to come up with your financial quarterly results. The chance that something would go wrong in this process is too high to allow that uncertainty -- sometimes structure is really good to have!

So, do all these successful enterprises use structure and data models because it is the only way to go in an RDBMS or a Data Warehouse? Of course not. This is not about what a data warehouse can do; this is about what the business needs. Claiming that customers will stop requiring data quality and accurate data models across all their data infrastructure is misleading.

Let's move to the second question. Why would you need a data warehouse if Hadoop is going to support everything from SQL to BI in a year or two?

This claim ignores a simple fact: it took decades of work from some of the most brilliant computer scientists to build databases. Can Hadoop provide and implement the same functionality in a couple of years?

The answer is obviously, no, and it would be a real shame to waste the community's efforts to rebuild existing functionality vs. inventing newer and more extraordinary use cases. And some of the early deliverables in the Hadoop world that purport to eliminate RDBMS's require schemas and have physical design constraints that go against the "flexibility" argument of Hadoop. What's more, these claims leave out the fact that Hadoop was originally not developed for BI or SQL execution. It's like using a hammer when you really want a screwdriver -- let's free Hadoop to be the great tool it was designed to be!

History teaches us that the impact of new technologies is over-estimated in the short-term and underestimated in the long run. Hadoop is not and will not become a data warehouse. RDBMs and data warehouses will thrive, not die, because of Hadoop. We think Hadoop will be an integral part of future analytic data infrastructure solutions, but not the only part!

Scott Gnau is president of Teradata Labs, where he directs all research, development and sales support activities related to data warehousing, big data analytics, and associated solutions.

Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Apprentice
1/23/2016 | 10:37:43 AM
Polyglot Persistence
Their will always be a need for both. SQL will be better at some things and NoSQL better at others. Use them both, just pick the correct one for the correct job. Thinking the data is going to change all of the time with little thought is a silly argument. With professional agile teams making development decisions, there is a good deal of planning and thought that goes into system changes whether the database requires it or not. I have created structured data in NoSQL and virtually non-structured data in relational databases (simply tagging back to the id). They both worked great. We will continue to see large advancements in both technologies as SQL works to scale horizontally better and NoSQL continues to develop better querying technologies. I predict that SQL and NoSQL will merge in the middle as these two technologies continue to work toward each other.
User Rank: Strategist
4/12/2014 | 9:52:03 AM
Interesting debate
Interesting debate. Of course most discussed topic at C level...   It is true that big data has created a question on data warehouse solutions though it is for time being and  picture will clear soon.  The term "big data" will evolve slowly and from my perspective it will be categorized in two ways- In house big data and data available to all (example Social media). Not all the organization will be able to leverage big data as it is being projected now that it can transform business strategy making. CIOs will prefer to have their data warehouse separate till technology implementation challenges gets to some clarity. Enterprises will slowly commence big data technology in place and start leveraging it by use cases more relevant to them. 4-5 years down the line when their system will stabilize they will think that Hadoop can process data warehouse data also for their reporting needs or not. My personal opinion .. lets wait to see how it goes  .... <a href=""> mahender   </a>
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
2017 State of IT Report
In today's technology-driven world, "innovation" has become a basic expectation. IT leaders are tasked with making technical magic, improving customer experience, and boosting the bottom line -- yet often without any increase to the IT budget. How are organizations striking the balance between new initiatives and cost control? Download our report to learn about the biggest challenges and how savvy IT executives are overcoming them.
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll