Software // Information Management
Commentary
7/3/2012
12:26 PM
Rajan Chandras
Rajan Chandras
Commentary
Connect Directly
RSS
E-Mail
50%
50%

How To Manage Big Data

There's an argument for integrating master data management with big data, but much work needs to be done before that happens.

As the discussion around big data gains momentum and substance, questions are being raised about how master data management (MDM) fits into the picture. From what I've seen, the amalgam of master data management with big data is a lot like the Sherlock Holmes story about the dog that didn't bark--what's noteworthy is that there's nothing of note.

I heard (or rather, didn't hear) something similar at the recent MDM and Data Governance Summit in San Francisco, where I presented as well as was part of a panel on master data management and data governance. The message in a nutshell, is that there's definitely a use case for integrating master data management with big data, but there hasn't been much distance covered yet.

The value proposition for bringing master data management into big data analytics is essentially no different from the standard MDM use case: providing identity (i.e. uniqueness) to entities.

Take the product marketing team that's interested in collecting and collating comments made by consumers across the Internet, in discussion forums, personal blogs and other hard-to-decipher places. The data it collects follows a typical big data pattern:

1. Large in size and can quickly go into hundreds of terabytes and beyond.

2. Semi-structured. (Take the phrase "unstructured data" with a grain of salt, because the contents may be free form, but there's almost always some sort of interesting structure around it that can be leveraged for analysis.)

3. Velocity of data is usually high, coming in fast and furious, which poses a challenge to conventional data extract, transform, and load (ETL) architectures and relational databases.

The product is the central point of intersection of master data management and big data. There's potentially important information in all the consumer feedback, but unless we tie in the free-form comments to our product catalog, what use is all that information?

This requires a classic both-ends-to-the-middle approach--technologies and techniques (like natural language processing) to help make sense of the free-form comments at one end, and an accurate and consistent product hierarchy (the product master) at the other end. MDM is clearly an enabler here.

I've heard similar stories elsewhere. At a recent user group conference for a large data warehousing appliance vendor, a pervasive message was the need for mastering identity to support consumer analytics.

Back at the MDM-DG conference, I had an interesting chat with the founder and CEO of Reltio, a venture capital-funded big data initiative, (don't bother visiting the website since it's in stealth mode right now). He had a sound understanding of the challenges of integrating MDM with big data, having previously led product marketing and strategy at a leading MDM vendor. His thoughts around using MDM to support big data analytics resonated with the emerging theme: MDM helps lay the foundation for big data analytics, but we're in the early stages of defining this integration framework.

Beyond big data, the conference featured the usual (and successful) mix of decision-makers, practitioners, and vendors. For example, an interesting presentation on a data governance approach formulated on the basis of Malcolm Gladwell's tipping point theory (loosely defined as a quick and unexpected change that dramatically transforms a situation) can be leveraged to drive data governance deeper and wider in the organization.

The challenges of data governance are fairly ubiquitous. The most common is a shortage of budget and resources, though not, in a curious twist, particularly driven by a lack of executive-level interest in data governance. The mantra for data governance seems to be "doing more with less".

In the products section, two offerings caught my eye, neither of which are new, but seem to have evolved into mature products worthy of attention.

Orchestra Networks' MDM/DG product offers a pleasant change from the integration-heavy architectures of the dominant MDM platforms out there, e.g. IBM, Informatica, Oracle, etc. Orchestra Networks starts with a facility to model your data and take it from there, which presents a much cleaner and more manageable approach, particularly, I suspect, for departmental and smaller MDM implementations.

Talend continues its foray into open source and the cloud. It's a product range that's expanding and maturing fast, and seems to be at or close to the point (call it the "tipping point") where it merits serious consideration in your MDM strategy, especially if you aren't already locked into the IBM, Oracle, or SAP stack. Keep in mind, of course, that "open source" doesn't always equate to "free."

Rajan Chandras has more than 20 years of experience advising and leading business technology initiatives, with a focus on strategy and information management. Write him at rchandras at gmail dot com.

At this year's InformationWeek 500 Conference, C-level execs will gather to discuss how they're rewriting the old IT rulebook and accelerating business execution. At the St. Regis Monarch Beach, Dana Point, Calif., Sept. 9-11.

Comment  | 
Print  | 
More Insights
Comments
Threaded  |  Newest First  |  Oldest First
dchasselshp5
50%
50%
dchasselshp5,
User Rank: Apprentice
7/6/2012 | 4:28:06 PM
re: How To Manage Big Data
This is interesting there was a point raised here http://www.ebizq.net/blogs/ebi... In summary MDM is seen as vital to see the BPM people first mindset be taken to enterprise level? Not only that as relevant data is identified and used at the front end so by default the other Gǣversions of the truthGǥ lurking in the legacy mess can be identified and archived? This is the use of MDM on the outside in approach which is now the focus? Interesting to see IBM now push this message? Whilst I am sure the use of MDM to support big data analytics has its place the ROI for improving operational efficiency may prove to be the better option? This aided by significant lower costs to build such front solutions? However this requires a good build tool that can deliver flexible application that represents the future. Naomi Blooms blog hits the spot http://infullbloom.us/?p=3222.
GǣGǪ.how those models can become applications without any code being written or even generatedGǥ.
GǣIf IGm right, youGll want to be on the agile, models-driven, definitional development side of the moat thus createdGǪ..Gǥ
Naomi went on to say later Gǣvendors are in trouble if not there (best) or moving quickly to object-based, metadata-driven, definitional development.Gǥ
This has already been created and proven in the UK and it really does work but that MDM capability will be important for larger companies try to sort out the legacy mess.
Stefan_Bernbo
50%
50%
Stefan_Bernbo,
User Rank: Apprentice
9/19/2012 | 3:40:42 PM
re: How To Manage Big Data
Interesting angle- It is true that a common challenge of data governance is a shortage of resources. We have identified a solution to decrease energy costs while at the same time increasing reliability and efficiency. I would be interested to see what you think: http://www.youtube.com/watch?v...
The Agile Archive
The Agile Archive
When it comes to managing data, dont look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Government Tech Digest Oct. 27, 2014
To meet obligations -- and avoid accusations of cover-up and incompetence -- federal agencies must get serious about digitizing records.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on InformationWeek.com for the week of October 26, 2014 and for the incredible Friday Afternoon Conversation that runs beside the program.
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.