4 min read

How to Get Your Big Data Journey Back on Track

Moving your big data initiative from proof of concept to production calls for mapping out some key steps.

Big data has been in the hype cycle for a while and Gartner last year identified that the topic is approaching the trough of disillusionment. Factually, many companies are still early on in their adoption of the related architecture patterns and technologies.

Many are building proof of concept systems or have unplanned emerging technological adoption. The challenge they face is the confusing technology landscape and the lack of long standing patterns. Everything appears to be in flux. Nevertheless, some technology savvy companies like Twitter and LinkedIn succeeded in extracting tremendous value and even setting standards like the Lambda Architecture or developing technologies like Kafka.

The challenge for the current adopters is to minimize risk and maximize value. Understanding how many successful big data journeys are structured and how to converge towards such a path best does that. From my experience at Big Data Partnership, the former is distinctively easier than the latter unfortunately. This is especially true once change management and internal organizational dynamics have to be taken into account.

In a greenfield situation the challenge is merely to take a business first and technology second approachand execute along a set of sensible steps:

  • Business goals. They should be up to date and clear.
  • Discover your strategy. How can big data support the business goals?
  • Discover your roadmap and use cases. Plan for the long term and identify short term use cases that are achievable, design an accompanying architecture and products.
  • Develop your use cases. Build proof of concepts to demonstrate value, viability, and solicit buy-in.
  • Deploy your architecture. Productionize your use cases with low risk, employing the learning from developing the proof of concepts.
  • Support and train. Focus on your business by employing professional services, vendors, and support where feasible, and train your staff along the above mentioned path to bring them with you on the big data journey.

The real life is messy and not many have the opportunity to start with a blank slate or resources and time necessary to do so. More often than not some steps are missed. For example:

  1. Business goals are years old, implicit, and expected to not have changed despite the competitive landscape having moved along.
  2. Strategy, surprisingly, for some is an afterthought once someone broke with the accepted technology stack to achieve something unachievable with the traditional stack.
  3. Roadmaps and use cases emerge ad hoc on a need basis and are not aligned with a vision risking investments’ future value.
  4. Architecture is bolted on legacy soft and hardware leading to brittle, inefficient, and unreliable outcomes.

All of these bring the risk to misleadingly conclude big data may be immature or pure hype when often the project was set up for failure. This can reset internal buy-in for years, in the meanwhile competitors take on the lead on new products and insights.

The solution to projects that did not take an ideal planned sequential approach to exploring and adopting big data is to identify the blind spots and address them by priority. Generally this is a top-down approach ensuring goals and strategies are up to date, clear, and aligned first. Next an ideal roadmap with immediate practical use cases and long-term vision should be established.

Then the current work, architecture, or proof of concept has to be put into perspective in this roadmap. At this point painful decisions may have to be made if it turns out that the current work needs realignment. Or it may be that it does fit, and the exercise reveals a clear short-term plan and vision of how to continue the big data journey. At the very least it provides a tool to communicate internally the journey and its benefits to stakeholders for resource allocation and buy-in.

Where are you in your big data journey, and what roadblocks have you encountered?

Editor's Choice
Sara Peters, Editor-in-Chief, InformationWeek / Network Computing
Jessica Davis, Senior Editor
Richard Pallardy, Freelance Writer
Carrie Pallardy, Contributing Reporter
John Edwards, Technology Journalist & Author
Carlo Massimo, Contributing Writer
Salvatore Salamone, Managing Editor, Network Computing