Big Data: Are We Really Ready for the Final Frontier?

I've just read an excellent post from Rich Murdane entitled "Big Data" and the Failure to Communicate. I'm glad Rich penned this piece as I was beginning to think I was either alone with my position and thoughts or, worse still, just missing the point! Rich has managed to nicely put into words a feeling that has been gradually building with me. For a few months now I've had a growing sense of disquiet that "big data" was becoming a buzz word - a phrase that was on everybody's lips but often with little substance behind it. That is, folks are talking about it because they feel they should be talking about it, but in reality they (and we as the broader industry) have not yet taken the time (or perhaps more kindly, had the time) to really think through what big data is and more importantly what it means to each of us and what  value it might deliver to our organisations.

I fear there is a real chance that much money will be spent too early as folks jump on the big data bandwagon before they they really have a good handle on which part of big data they should be focusing on. If we look at what "Big Data" is then a number of things spring to mind. For me the the obvious one is condition monitoring data - generally structured time series data. It's not just the size of this data that makes it big, it's how fast it arrives - what's beginning to be more commonly talked about as the velocity of data.

For me, and the big data problems I'm likely to face in my industry, I think this may well be the key issue at least in the short term. Operational technology is prevalent and in some cases this could mean thousands of data points arriving per second - huge waves of data washing in. One of the questions that I think needs to be considered is how high velocity big data should be used, or perhaps how the best value can be derived. Quite a bit of the discussion I'm hearing and reading suggests that this question is not being asked, but rather there's a leap of faith being taken resulting in a default position that this data needs to be stored and analyzed. I agree that over time this might well be the case, but for many right now I wonder if this value proposition is understood well enough to be effective. Perhaps it might be wiser to wait and look at other ways to gain value from this style of big data, one of which might be a form of complex event processing to monitor the stream of data and pluck out those events or short term trends which allow an action to be taken to materially impact operations. Those of us without a compelling business case to store, and later interpret across, huge data volumes might be best to wait and watch, taking action not because we feel that we should be, but because we understand what the value is and how we will unlock it.

This watch, monitor and wait brief should be wide enough to make sure that all forms of big data that might deliver value are considered. This means that it's not just the structured data area that is important, unstructured data is perhaps even bigger. For many of us with long pedigrees in the data space this may be somewhat of a new and unfamiliar area. I think we have no choice but to step outside of our comfort zones and become more familiar with this space. Much of the short to medium term value to be derived from big data, in my opinion, will come from this area for many industries. Sentiment analysis and mining of data from social networks is a great example of this which is already becoming important for many organizations. It's also an example where a stream of data can be monitored for data if interest as well as gaining value from analysis across larger pieces of historical data.

Big data is here to stay. It's not a matter of if it will impact your company, but now only a matter of when. You'll have to jump in and ride the wave at some time, just be sure you don't jump into too soon. Make sure you haven't worn out your funding and senior management support, will and trust before there is a real opportunity for you to harvest value from big data. Big Data may or may not be the final frontier. I could borrow from Star Trek, but I'd rather borrow from 1980s one hit wonder Star Trekkin'. It's Data Jim, but not as we know it. Good luck in your explorations, but be careful. There may not be Klingons on your starboard bow, but there is almost certain to be an executive or two asking the hard questions if his money disappears into a black hole with no value return.

Findable, trustable and believable. How FaceBook Slipped Up

If I had to sum up the core of what's important for me in the unstructured data area right now I'd use three words: findability, believability, and trustability. Given that my word processing software has just drawn squiggly red lines under each of them, I'm not even sure they are official words. But they should be - without these three things users won't be totally comfortable using whatever piece of software, system or web portal we give them to work with and will probably disengage with it, at least in some fashion.

Don't believe me? Even the big players miss the mark sometimes. Look at FaceBook as an example. I suspect that the recent rumblings and user backlash against the new interface wasn't just because the look and feel changed. The experience changed as well. People didn't know where to look to find things and even if certain things were still possible. For me I wasn't bothered by the change to the look and feel, but I was still thrown by the change. Little changes make the difference. I no longer saw links I posted from external
sites showing up in the my general news feed. Did that mean that the post had failed, or could other people see them and my own posts where just filtered out of what I saw, or had I done something wrong - selected the wrong privacy settings, perhaps? My level of trust had dropped, and dropped quickly. So much so that I took to using other means to share links in some areas, resorting to emailing links around or using specialist sites like Garmin Connect and Strava to publish my running and cycling information. Changes to FaceBook's iPhone interface also left me wondering where to look to find certain posts. I now had a taxonomy of sorts which I could use to refine and filter what displayed on screen. I could choose to view all stories, just see photos, status updates, links, pages, posts from close friends, posts from people in my area or post from people who were associated with the University I attended, etc. Refiners! Great, less noise to wade through, give me just what I want to see. That's a good thing right? I thought so too, but again I didn't always see what I thought I should see, leading to a lingering doubt. Again a trust issue - was I seeing all of the data available or did I have to look somewhere else or do something else to bring it up? Findability had suffered as well (or at least I perceived it had).

Translate this into the corporate world and this might mean that if users lose trust in how their unstructured data is being stored or made available to thrm that they'll stop using a document management system or content management system and resort to emailing documents around the organization and / or putting content on file shares or in other information silos. Even in a well used and hitherto trusted system small changes can quickly result in users working around the system, taking us back toward the unstructured data anarchy that many of us are slowly working to bring order to. Slip behind on that challenge and suddenly your information is fragmenting and findability suffers as well.

What lessons are there to be taken from this? Firstly, small changes matter. Something seemingly insignificant can send ripples out which can set your information management goals back months. Secondly, user interface changes can and will impact on information management and information integrity and quality. Don't make design choices without considering the impact on the wider issues and your information management goals. Choices here may not seem important but can at times be just as critical as deeper information architecture questions or taxonomy design. And finally, communication is key. Make it clear what changes mean to how information can be found and what information is and isn't available and perhaps problems like these, and the negative perceptions that come with them, can be avoided.

Have you lived through similar issues? My medium term goals for information inside our organisation rely heavily on getting findability, believability and trustability right so I'd love to hear about what you faced and how you tackled it.