How to Get Your Big Data Journey Back on TrackHow to Get Your Big Data Journey Back on Track

Big data has been in the hype cycle for a while and Gartner last year identified that the topic is approaching the trough of disillusionment. Factually, many companies are still early on in their adoption of the related architecture patterns and technologies.

Many are building proof of concept systems or have unplanned emerging technological adoption. The challenge they face is the confusing technology landscape and the lack of long standing patterns. Everything appears to be in flux. Nevertheless, some technology savvy companies like Twitter and LinkedIn succeeded in extracting tremendous value and even setting standards like the Lambda Architecture or developing technologies like Kafka.

The challenge for the current adopters is to minimize risk and maximize value. Understanding how many successful big data journeys are structured and how to converge towards such a path best does that. From my experience at Big Data Partnership, the former is distinctively easier than the latter unfortunately. This is especially true once change management and internal organizational dynamics have to be taken into account.

In a greenfield situation the challenge is merely to take a business first and technology second approachand execute along a set of sensible steps:

The real life is messy and not many have the opportunity to start with a blank slate or resources and time necessary to do so. More often than not some steps are missed. For example:

All of these bring the risk to misleadingly conclude big data may be immature or pure hype when often the project was set up for failure. This can reset internal buy-in for years, in the meanwhile competitors take on the lead on new products and insights.

The solution to projects that did not take an ideal planned sequential approach to exploring and adopting big data is to identify the blind spots and address them by priority. Generally this is a top-down approach ensuring goals and strategies are up to date, clear, and aligned first. Next an ideal roadmap with immediate practical use cases and long-term vision should be established.

Then the current work, architecture, or proof of concept has to be put into perspective in this roadmap. At this point painful decisions may have to be made if it turns out that the current work needs realignment. Or it may be that it does fit, and the exercise reveals a clear short-term plan and vision of how to continue the big data journey. At the very least it provides a tool to communicate internally the journey and its benefits to stakeholders for resource allocation and buy-in.

Where are you in your big data journey, and what roadblocks have you encountered?