The Next Data Rock Star

I miss the 1980s, there was lots to like about that time: great music, a lifestyle less cluttered by technology, crazy hair, the fact that I could still grow enough hair to have crazy hair. For most of my adulthood if you had asked me which time in my life I'd like to go back to I'd have picked the 1980s. But now I'm not so sure, I'm starting to pine for the late 1990s through to the mid 2000s. Why? Because, that was a time when quite a few data folks working in the business intelligence arena were the rock stars of IT.

I'm lucky enough to have worked on my fair share of interesting and high profile data warehousing and BI gigs during that period. The problems I had the privilege of working on were often difficult, few (if any) commodity solutions or services had manifested, and the solutions we provided were often anything but cheap. However in solving those problems we added real value to the business, at times doing things that their traditional IT staff or software vendors had said couldn't be done or had tried (and failed) to solve with more application centric approaches. Perhaps most importantly during that period few BI projects were undertaken unless the business could really see the benefit. Maybe this was because of the high cost, maybe not. Whatever the reason, those of us lucky enough to be leading the charge and playing key roles in these projects often got the accolades of not just our IT and data colleagues but also  of key business stakeholders as well. I was fortunate to parlay this into a successful consulting career  with word of mouth securing a series of back to back BI engagements stretching for over a decade.

But I fear that the BI Rock Star is a thing of the past. It's becoming a crowded field. Any number of people now "do BI" and what was once bordering on black magic is commodity thanks to advances in (BI) software products and hardware processing power. Even worse, the phrase "reporting" is commonly be used almost interchangeably with "business intelligence". We've all too often moved away using BI undertakings to support complex decisions and solve wicked problems to one where often all too many reports are produced for banal reasons or reasons unknown. It's no wonder then that business stakeholders seem to be seeing less value from their business intelligence dollar.

So, if BI is indeed on the wane, I wonder where the next Data Rock Star will come from. If I was to encourage my kids into the field where would I point them? Will the Data Scientists be the next wave to be lauded by the business, or will those jumping on the Big Data bandwagon get that privilege? Maybe those playing in the NoSQL space are about to be thrust to glory, or perhaps the social media tsunami will mean that's it the intersection of all three which will be the sweet spot. I actually think it won't be any of these. Both Big Data and Data Scientists to me still feel like buzz words - hot topics - that the industry has thrown up and bandied about to the point where airline magazine syndrome has kicked in. I don't see the clear path to the wide stream killer need that means business stakeholders will clamour for solutions in these areas enmasse any time soon.

So, if you can see more clearly than I, I'd love to know what the next big data thing will be. I miss being a rock star!

Big Data - Is it Really New?

Despite the fact that Big Data is fastest becoming one of the buzz words in the data industry I still wonder if we yet know exactly what it is. Is it anything more than just a concept, an umbrella topic if you like, to attempt to signal that something new is beginning to emerge in our industry? But, is big data really something new? Is the buzz justified, do we really need to reinvent our approaches, our practices and the skills we engender in those coming up through data careers?

I’d argue that some, if not many, data professionals already know a thing or two about big data. Well over a decade ago I was the architect leading a project to build a data warehouse to handle call traffic and billing data for one of Australia’s biggest telecommunications companies. Back then the volume of data we were working with was hardly small and I’d suggest that those that the people who work with that warehouse, and the BI solutions in enables, today are facing even larger data volumes thanks to the proliferation of mobile phones and other devices. We found ways of handling the challenges that the volume of data posed and more than a few of these approaches would still work today, even without the additional leg up we now get from increased computing power.

But what of the other characteristics of big data? It’s not just volume that’s the challenge with big data, but the velocity too. I’ve heard some commentators discussing that it is the rate at which data arrives which is in fact the biggest issue that comes with big data. But we’ve been dealing with high velocity data for a while now as well. Complex event processing has been available in major database products for at least a few years and people working with any form of operational technology will be used to data flowing in from sensors and other protection or control devices at millisecond intervals.

So I believe that we can leverage much of what we already know and practice as data professionals to start to address the volume and velocity aspects of [structured] big data. It’s the complexity and variety aspects of big data that I think will give us the real problems we need to deal with. The problem of making, or taking, the correct meaning from unstructured data, especially that coming from outside our organisations, such as sentiment analysis from social media, and then somehow find a way to effectively integrate it with our structured data sets is where I believe we’ll find our headaches. It’s here that we’ll need innovative new tools and techniques, but even then I think they’ll rely on key areas and practices which at least some of us have worked with for a while. Metadata will play a big role here to establish the right context and data mining may well make an appearance as well.

So, in my opinion at least, we already know a thing or two about big data. Let’s not re-invent the wheel, but rather try to build on what we’ve learned in the past. It’s a novel idea I know, and perhaps not one that IT professionals have much of a proven track record with, but hopefully this type of approach might shorten the time to value for those making early investments in and around big data.

When I Grow Up I Want to Be a Data Scientist

Some time in the early 1970s I formed my first career aspiration. When I grew up I wanted to be a bus. Yes, you read that right - not a bus driver, but a bus. It seems ridiculous now, some 40-odd years later, but I certainly wanted it bad back then! The reason it didn't seem crazy all those years ago is that I really had no idea what being a bus involved and what's more a bus seemed like a pretty cool thing to be - lots of the books that I had read to me featured buses zooming all over the countryside having all manner of exciting adventures, there was at least one cartoon where one of the main characters was a bus, and buses even had songs sung about them. Why wouldn't a three year old boy want to be a bus?

Fast forward 40 years and focus on the data industry: big data is being talked about in server rooms, board rooms and everywhere in between, and the term Data Scientist is becoming more and more common. It's a position I'd never heard of twelve months ago and now it seems hardly a day goes by where I don't see it mentioned in a blog post, industry email, or white paper, etc. A quick look at a number of job posting sites across the web confirms that organisations are indeed looking to hire people into roles with the Data Scientist title. But does anyone really know what the job entails, is there an agreed job description, even in a general sense? Judging by the diverse descriptions of the job ads I looked at it certainly doesn't seem so. I've talked before about my concerns that the Big Data concept is very much an overloaded and poorly understood term and I worry that the new "must have" role of Data Scientist is facing the same problems.

But, so what? Does this situation even matter? I think so. I applaud anything that raises the profile of those of us that work in the data and information space. I still remember what it was like to ride the wave in the early days of business intelligence and data warehousing and if others can have similar experiences and success today due to the next big thing being data centric then that's a good thing. However, I worry if the  shining beacon of the Data Scientist role starts to lure people from other facets of IT, or potentially worse still, attracts students entering University whilst the speciality is still in its infancy. With the role a Data Scientist will play, and how they will deliver value, yet to become understood in a consistent way then how can we expect our higher education institutions to prepare courses of study that will equip students with the skills that they will need when they reach the work force? I wonder if we will see such courses morphing, emerging and disappearing over coming years as industry and the universities try to better find what is expected of Data Scientists and how to build the knowledge and skills base they need to be successful. What of these early students, the guinea pigs or crash test dummies of the coming wave of graduate data scientists? Are we leading them astray, promising them areas of work, challenges and career paths which no-one yet knows will actually be there in the same form (or at all) five or ten years from now? Let's not promise too much too early. Better people come to the profession with their eyes open and with realistic views of what to expect, than simply because of hype.

Enough doom and gloom. If anyone can really tell me what a Data Scientist does, I'd love to know. I actually might want to be one when I grow up.

Big Data: Are We Really Ready for the Final Frontier?

I've just read an excellent post from Rich Murdane entitled "Big Data" and the Failure to Communicate. I'm glad Rich penned this piece as I was beginning to think I was either alone with my position and thoughts or, worse still, just missing the point! Rich has managed to nicely put into words a feeling that has been gradually building with me. For a few months now I've had a growing sense of disquiet that "big data" was becoming a buzz word - a phrase that was on everybody's lips but often with little substance behind it. That is, folks are talking about it because they feel they should be talking about it, but in reality they (and we as the broader industry) have not yet taken the time (or perhaps more kindly, had the time) to really think through what big data is and more importantly what it means to each of us and what  value it might deliver to our organisations.

I fear there is a real chance that much money will be spent too early as folks jump on the big data bandwagon before they they really have a good handle on which part of big data they should be focusing on. If we look at what "Big Data" is then a number of things spring to mind. For me the the obvious one is condition monitoring data - generally structured time series data. It's not just the size of this data that makes it big, it's how fast it arrives - what's beginning to be more commonly talked about as the velocity of data.

For me, and the big data problems I'm likely to face in my industry, I think this may well be the key issue at least in the short term. Operational technology is prevalent and in some cases this could mean thousands of data points arriving per second - huge waves of data washing in. One of the questions that I think needs to be considered is how high velocity big data should be used, or perhaps how the best value can be derived. Quite a bit of the discussion I'm hearing and reading suggests that this question is not being asked, but rather there's a leap of faith being taken resulting in a default position that this data needs to be stored and analyzed. I agree that over time this might well be the case, but for many right now I wonder if this value proposition is understood well enough to be effective. Perhaps it might be wiser to wait and look at other ways to gain value from this style of big data, one of which might be a form of complex event processing to monitor the stream of data and pluck out those events or short term trends which allow an action to be taken to materially impact operations. Those of us without a compelling business case to store, and later interpret across, huge data volumes might be best to wait and watch, taking action not because we feel that we should be, but because we understand what the value is and how we will unlock it.

This watch, monitor and wait brief should be wide enough to make sure that all forms of big data that might deliver value are considered. This means that it's not just the structured data area that is important, unstructured data is perhaps even bigger. For many of us with long pedigrees in the data space this may be somewhat of a new and unfamiliar area. I think we have no choice but to step outside of our comfort zones and become more familiar with this space. Much of the short to medium term value to be derived from big data, in my opinion, will come from this area for many industries. Sentiment analysis and mining of data from social networks is a great example of this which is already becoming important for many organizations. It's also an example where a stream of data can be monitored for data if interest as well as gaining value from analysis across larger pieces of historical data.

Big data is here to stay. It's not a matter of if it will impact your company, but now only a matter of when. You'll have to jump in and ride the wave at some time, just be sure you don't jump into too soon. Make sure you haven't worn out your funding and senior management support, will and trust before there is a real opportunity for you to harvest value from big data. Big Data may or may not be the final frontier. I could borrow from Star Trek, but I'd rather borrow from 1980s one hit wonder Star Trekkin'. It's Data Jim, but not as we know it. Good luck in your explorations, but be careful. There may not be Klingons on your starboard bow, but there is almost certain to be an executive or two asking the hard questions if his money disappears into a black hole with no value return.