Big Data: Are We Really Ready for the Final Frontier?

I've just read an excellent post from Rich Murdane entitled "Big Data" and the Failure to Communicate. I'm glad Rich penned this piece as I was beginning to think I was either alone with my position and thoughts or, worse still, just missing the point! Rich has managed to nicely put into words a feeling that has been gradually building with me. For a few months now I've had a growing sense of disquiet that "big data" was becoming a buzz word - a phrase that was on everybody's lips but often with little substance behind it. That is, folks are talking about it because they feel they should be talking about it, but in reality they (and we as the broader industry) have not yet taken the time (or perhaps more kindly, had the time) to really think through what big data is and more importantly what it means to each of us and what  value it might deliver to our organisations.

I fear there is a real chance that much money will be spent too early as folks jump on the big data bandwagon before they they really have a good handle on which part of big data they should be focusing on. If we look at what "Big Data" is then a number of things spring to mind. For me the the obvious one is condition monitoring data - generally structured time series data. It's not just the size of this data that makes it big, it's how fast it arrives - what's beginning to be more commonly talked about as the velocity of data.

For me, and the big data problems I'm likely to face in my industry, I think this may well be the key issue at least in the short term. Operational technology is prevalent and in some cases this could mean thousands of data points arriving per second - huge waves of data washing in. One of the questions that I think needs to be considered is how high velocity big data should be used, or perhaps how the best value can be derived. Quite a bit of the discussion I'm hearing and reading suggests that this question is not being asked, but rather there's a leap of faith being taken resulting in a default position that this data needs to be stored and analyzed. I agree that over time this might well be the case, but for many right now I wonder if this value proposition is understood well enough to be effective. Perhaps it might be wiser to wait and look at other ways to gain value from this style of big data, one of which might be a form of complex event processing to monitor the stream of data and pluck out those events or short term trends which allow an action to be taken to materially impact operations. Those of us without a compelling business case to store, and later interpret across, huge data volumes might be best to wait and watch, taking action not because we feel that we should be, but because we understand what the value is and how we will unlock it.

This watch, monitor and wait brief should be wide enough to make sure that all forms of big data that might deliver value are considered. This means that it's not just the structured data area that is important, unstructured data is perhaps even bigger. For many of us with long pedigrees in the data space this may be somewhat of a new and unfamiliar area. I think we have no choice but to step outside of our comfort zones and become more familiar with this space. Much of the short to medium term value to be derived from big data, in my opinion, will come from this area for many industries. Sentiment analysis and mining of data from social networks is a great example of this which is already becoming important for many organizations. It's also an example where a stream of data can be monitored for data if interest as well as gaining value from analysis across larger pieces of historical data.

Big data is here to stay. It's not a matter of if it will impact your company, but now only a matter of when. You'll have to jump in and ride the wave at some time, just be sure you don't jump into too soon. Make sure you haven't worn out your funding and senior management support, will and trust before there is a real opportunity for you to harvest value from big data. Big Data may or may not be the final frontier. I could borrow from Star Trek, but I'd rather borrow from 1980s one hit wonder Star Trekkin'. It's Data Jim, but not as we know it. Good luck in your explorations, but be careful. There may not be Klingons on your starboard bow, but there is almost certain to be an executive or two asking the hard questions if his money disappears into a black hole with no value return.