Teradata QueryGrid: Beyond Enterprise Data Warehouse - InformationWeek
IoT
IoT
Comments
Teradata QueryGrid: Beyond Enterprise Data Warehouse
Newest First  |  Oldest First  |  Threaded View
masifabbasi
100%
0%
masifabbasi,
User Rank: Apprentice
5/24/2014 | 4:13:11 PM
Re: Propagating a myth about Hadoop
Very nice article... Thanks for expanding on QueryGrid. Companies like Cisco's composite have built some top-class optimizers that do the similar stuff, but have the edge of 8 years of RnD on Federated queries. Top class rule based optimizers, Intelligent data ships, and an overlay of cost based optimizer makes it really useful for today's myriad of data sets.  Just wanted to respond to some of your comments above:

"DBMS vendors like to say that Hadoop is all about "unstructured" or "semi-structured" information."


I don't think that is what they are saying. Companies like Teradata (IMHO) are saying that Hadoop acting like a data lake is a good place to store structure, unstructured and semi-structured data, which is cold, and near-real-time access isn't a requirement. Hadoop vendors like Hortonworks also agree with the statement.

"Check out my recent profile of Merck & Co., which combined 16 different types of very structured data to figure out why some batches of a vaccine had high yield rates and other batches had low yield rates. The advantage was being able to "dump everything in a lake" without time-consuming data modeling and ETL work."

That is exactly what these companies are proposing. Modelling and ETL aren't required for all use cases in the data management world. We have done POCs where we had done worked with 10-100's of Terabytes of data using a combination of Hadoop & Aster, and acheived results within a couple of weeks.  Essentially these companies are saying that you need a cheap platform to store the data (like Hadoop), but when you want to do number crunching and complex analytics, you need powerful and intelligent SQL, MapReduce and Graph Engines (unless you are Facebook,Google, LinkedIn having access to top-tiers of talent), and use your skillset to acheive the result set.



Having said all that, majority of the problems out there could be solved using a multitude of technologies. The decision to use a particular technology depends/should depend on the following:

1. Urgency of problem (SLA)

2. Your skillset/Strength

3. Skillset available in the market

4. ROI

Thanks for sharing the article BTW :)

 

 
D. Henschen
50%
50%
D. Henschen,
User Rank: Author
4/7/2014 | 5:50:10 PM
Propagating a myth about Hadoop
RDBMS vendors like to say that Hadoop is all about "unstructured" or "semi-structured" information. And their favorite example is Twitter and Facebook comments. That's not the full truth and it's a bit of a left-handed compliment meant to diminish the value of Hadoop. The fact is, Hadoop can handle very structured information that happens to be varied, voluminuous, inconsistent (a.k.a. sparse) or all of the above.

Check out my recent profile of Merck & Co., which combined 16 different types of very structured data to figure out why some batches of a vaccine had high yield rates and other batches had low yield rates. The advantage was being able to "dump everything in a lake" without time-consuming data modeling and ETL work. Within three months Merck was able to cluster and visualize batch yield rates and spot "smoking guns" within 10 years worth of product and manufacturing plant data. The data wasn't actually all that big -- only 1.5 terabytes -- but the ability to bring together a variety of data quickly made all the difference.


How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
2017 State of IT Report
In today's technology-driven world, "innovation" has become a basic expectation. IT leaders are tasked with making technical magic, improving customer experience, and boosting the bottom line -- yet often without any increase to the IT budget. How are organizations striking the balance between new initiatives and cost control? Download our report to learn about the biggest challenges and how savvy IT executives are overcoming them.
Video
Slideshows
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll