Comments
Teradata QueryGrid: Beyond Enterprise Data Warehouse
Newest First  |  Oldest First  |  Threaded View
masifabbasi
100%
0%
masifabbasi,
User Rank: Apprentice
5/24/2014 | 4:13:11 PM
Re: Propagating a myth about Hadoop
Very nice article... Thanks for expanding on QueryGrid. Companies like Cisco's composite have built some top-class optimizers that do the similar stuff, but have the edge of 8 years of RnD on Federated queries. Top class rule based optimizers, Intelligent data ships, and an overlay of cost based optimizer makes it really useful for today's myriad of data sets.  Just wanted to respond to some of your comments above:

"DBMS vendors like to say that Hadoop is all about "unstructured" or "semi-structured" information."


I don't think that is what they are saying. Companies like Teradata (IMHO) are saying that Hadoop acting like a data lake is a good place to store structure, unstructured and semi-structured data, which is cold, and near-real-time access isn't a requirement. Hadoop vendors like Hortonworks also agree with the statement.

"Check out my recent profile of Merck & Co., which combined 16 different types of very structured data to figure out why some batches of a vaccine had high yield rates and other batches had low yield rates. The advantage was being able to "dump everything in a lake" without time-consuming data modeling and ETL work."

That is exactly what these companies are proposing. Modelling and ETL aren't required for all use cases in the data management world. We have done POCs where we had done worked with 10-100's of Terabytes of data using a combination of Hadoop & Aster, and acheived results within a couple of weeks.  Essentially these companies are saying that you need a cheap platform to store the data (like Hadoop), but when you want to do number crunching and complex analytics, you need powerful and intelligent SQL, MapReduce and Graph Engines (unless you are Facebook,Google, LinkedIn having access to top-tiers of talent), and use your skillset to acheive the result set.



Having said all that, majority of the problems out there could be solved using a multitude of technologies. The decision to use a particular technology depends/should depend on the following:

1. Urgency of problem (SLA)

2. Your skillset/Strength

3. Skillset available in the market

4. ROI

Thanks for sharing the article BTW :)

 

 
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
4/7/2014 | 5:50:10 PM
Propagating a myth about Hadoop
RDBMS vendors like to say that Hadoop is all about "unstructured" or "semi-structured" information. And their favorite example is Twitter and Facebook comments. That's not the full truth and it's a bit of a left-handed compliment meant to diminish the value of Hadoop. The fact is, Hadoop can handle very structured information that happens to be varied, voluminuous, inconsistent (a.k.a. sparse) or all of the above.

Check out my recent profile of Merck & Co., which combined 16 different types of very structured data to figure out why some batches of a vaccine had high yield rates and other batches had low yield rates. The advantage was being able to "dump everything in a lake" without time-consuming data modeling and ETL work. Within three months Merck was able to cluster and visualize batch yield rates and spot "smoking guns" within 10 years worth of product and manufacturing plant data. The data wasn't actually all that big -- only 1.5 terabytes -- but the ability to bring together a variety of data quickly made all the difference.


The Business of Going Digital
The Business of Going Digital
Digital business isn't about changing code; it's about changing what legacy sales, distribution, customer service, and product groups do in the new digital age. It's about bringing big data analytics, mobile, social, marketing automation, cloud computing, and the app economy together to launch new products and services. We're seeing new titles in this digital revolution, new responsibilities, new business models, and major shifts in technology spending.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest - August 20, 2014
CIOs need people who know the ins and outs of cloud software stacks and security, and, most of all, can break through cultural resistance.
Flash Poll
Video
Slideshows
Twitter Feed
InformationWeek Radio
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.