Big Data - Is it Really New?

Despite the fact that Big Data is fastest becoming one of the buzz words in the data industry I still wonder if we yet know exactly what it is. Is it anything more than just a concept, an umbrella topic if you like, to attempt to signal that something new is beginning to emerge in our industry? But, is big data really something new? Is the buzz justified, do we really need to reinvent our approaches, our practices and the skills we engender in those coming up through data careers?

I’d argue that some, if not many, data professionals already know a thing or two about big data. Well over a decade ago I was the architect leading a project to build a data warehouse to handle call traffic and billing data for one of Australia’s biggest telecommunications companies. Back then the volume of data we were working with was hardly small and I’d suggest that those that the people who work with that warehouse, and the BI solutions in enables, today are facing even larger data volumes thanks to the proliferation of mobile phones and other devices. We found ways of handling the challenges that the volume of data posed and more than a few of these approaches would still work today, even without the additional leg up we now get from increased computing power.

But what of the other characteristics of big data? It’s not just volume that’s the challenge with big data, but the velocity too. I’ve heard some commentators discussing that it is the rate at which data arrives which is in fact the biggest issue that comes with big data. But we’ve been dealing with high velocity data for a while now as well. Complex event processing has been available in major database products for at least a few years and people working with any form of operational technology will be used to data flowing in from sensors and other protection or control devices at millisecond intervals.

So I believe that we can leverage much of what we already know and practice as data professionals to start to address the volume and velocity aspects of [structured] big data. It’s the complexity and variety aspects of big data that I think will give us the real problems we need to deal with. The problem of making, or taking, the correct meaning from unstructured data, especially that coming from outside our organisations, such as sentiment analysis from social media, and then somehow find a way to effectively integrate it with our structured data sets is where I believe we’ll find our headaches. It’s here that we’ll need innovative new tools and techniques, but even then I think they’ll rely on key areas and practices which at least some of us have worked with for a while. Metadata will play a big role here to establish the right context and data mining may well make an appearance as well.

So, in my opinion at least, we already know a thing or two about big data. Let’s not re-invent the wheel, but rather try to build on what we’ve learned in the past. It’s a novel idea I know, and perhaps not one that IT professionals have much of a proven track record with, but hopefully this type of approach might shorten the time to value for those making early investments in and around big data.

What Do You Mean You Don't Need a Data Model?

On more than one occasion across my career I've found myself having to justify why we should bother undertaking data modeling exercises. Thankfully I'm not facing that problem currently, but I am in the midst of refreshing a set of strategy, standards and guidelines documents and recently found myself writing a "Why Model Data?" section as a preface to one document. It got me thinking that, despite all of the prior times I'd had the discussions, I'd never really succinctly put down in one place the why it is that I think we should undertake a data modeling process - how it adds value, if you like. So, this blog post is the result.

Modeling of information systems without modeling of the data they operate with can, and does, lead to bad outcomes. Modeling at only a systems or application level often fails to consider important characteristics of the underlying data, instead painting a picture which is only representative of a particular method of use of that data. Such modeling frequently impedes reuse of data across organisations, lowers the quality and speed of decision making and can lead to the unnecessary IT spend and the development of siloed IT applications with unwanted and sometimes unrecognised redundancy. Modeling of data assists with building understanding not only of data content but also of data relationships, with an effective data model being a key enabler of successful integration and well aligned and efficient application architectures and portfolios.

The explicit act of data modeling also encourages discussion about the meaning of data and the appropriate (or otherwise) use of various data elements in different functions of the organisation. It may well expose differing definitions and understanding of the same elements of data that, prior to the modeling exercise, those in the organisation had been unaware of.

Higher level data models also serve as a useful communication tool, helping to bridge the IT – Business divide and provide a mechanism to forge a common understanding of particular areas of the organisation’s data. Data which is easily and well understood is more likely to be reused as not only is the barrier to reuse lowered, but the urge to collect and store the same data elsewhere for another purpose less likely to come to the fore. This important aspect can be a mechanism for cost avoidance in that it lessens the risk of the design and implementation of systems which make ineffective or inappropriate use of the data assets.

Finally, when coupled with other [types of] models, data models allow an understanding of an organisation’s current technology landscape and the way it interacts with business processes. This understanding forms the foundation for efficient and low cost impact analysis around process and system change, in turn enabling business agility.

So there you have it. Part elevator pitch, part business case executive summary. I hope it helps someone the next time they're arguing for funding to develop or maintain the enterprise data model or trying to convince a bunch of rogue developers that simply using Visio to draw an ERD after the application is delivered isn't really the best approach!

We Don't All Need World's Best Practice

In anything but the smallest of IT departments chances are that at sometime you will need to rely on the efforts of others to design and deliver some project or other, be they internal staff, contract resources brought in-house or a full blown systems integration firm. There’s also a reasonable chance that you may not have managerial authority over those undertaking the work, or perhaps at best a dotted line report to you.  When faced with this scenario most of us will want to exert some degree of control or influence over how the work gets done or, at the very least, we’ll have an approach we’d prefer those charged with performing the work to follow.

One common element I’ve noted amongst many of the people I’ve managed or mentored over my career is an hesitation, or even unwillingness, to commit to paper a set of rules or guidelines which will govern the way in which others conduct their design, development and implementation activities. Often these folks have claimed that this sort of guiding documentation isn’t required, citing reasons including direct supervisory authority over the project team, the ability to exert technical influence in a one on one scenario with key project team members, or that the project team is experienced and skilled enough that such governance is not required. To my way of thinking none of these reasons really holds water. Even if you are able to direct or influence the behavior of the project team, to tackle this in an ad-hoc fashion is ineffective, not fair on the project team (it’s hard to work within given constraints when those constraints are only trickle fed to you and often only arrive at a review stage) and often this direction falls by the wayside as project pressures heat up (if it ever actually occurs at all). Leaving things ungoverned is also a risky proposition – there are many ways to skin a cat – and there is no guarantee that the approach taken by the project team (no matter how good or effective it may be) will result activities or outputs which mesh well with your existing team, processes, procedures or landscape. Failure to govern, guide and set some ground rules can, and usually will, result in project outcomes which are not as good as they could otherwise have been.

I have another theory as to why people are reluctant to commit such ground rules to paper – fear! It’s a natural reaction when you’re dealing with the unknown. Other than a paper CV and perhaps a few preliminary meetings you have little real idea of the depth of experience and skills of the “outsiders” you will be working with. Concern creeps in: “will they know so much more than me” and “will I look stupid alongside them or in their eyes” are just two of the things those little voices in your head might start to whisper to you. I can recall these feelings in the past even the project lay squarely in an area in which I had deep expertise. Imagine how uncomfortable a person already a little uncertain of his or her skill and experience in an area might feel! Alongside the fear comes overwhelm: the feeling that there are just too many things to think about, that it would not be possible to get them all documented without missing at least one or two items. This thought process leads right back to feeding the fear with concerns that omissions from the document will only make the author look even worse in the eyes of his or her management or the new people arriving for the project.

Let me put an idea out there. You don’t need to know more than the people coming into your company, you don’t even need to know the best way to tackle a certain technology problem or the ins and outs of the latest development techniques or what domain thought leaders are debating amongst themselves. There is no need to commit to documenting world’s best practice, nor to hold your project resources to that standard. Good practice will be fine for the vast majority of situations. But how do you even know what good practice is? And how do you make sure that you cover all of the big-ticket items? Knowing everything that is important to think about can be daunting.

I favour tackling this with something I call the Bad Outcomes approach. Rather than trying to think of everything that needs to be done in a certain way or toward a certain approach, simply make a list of the bad outcomes that could result from the upcoming project. Start with the really big ones; the things that might cost your company at the high end of the scale, whether it be financially or in other less direct ways such as reputation or brand damage, or even worse could cost you your job. Once you have those down move on to the layer below, those bad outcomes which may not be as catastrophic but are still likely to cause a prolonger period of discomfort. Mull on these items for a few days; discuss them with colleagues, both from within the IT function and from the business, adding any new items that might come up. Revisit your list and drop any items which would only cause minimal impact or short term pain and with luck you’ll have a relatively short list of the outcomes that you need to govern against. As an example, the last time I went though this exercise I ended up with only eleven items for a project likely to be worth multiple tens of millions of dollars. Now you’ll have focus – you’ll know what to work on that’s really important. It won’t matter if you don’t use the World’s Best Practice approach to govern and guide each item, so long as you find a way which is likely to avoid the outcome then you’ll have what you need and you’ll likely have saved your company a pretty penny in avoided costs and perhaps even your job along the way.

The next time you’re faced with the need to craft a strategy or pen a set of standards or guidelines don’t worry about what you don’t know. Remember, no-one will know all there is to know about a subject, so accept that you won’t always know the best way to solve a problem or everything there is to think about, but I’m pretty sure you, like me, have your scars and war stories so you’ll know what you want to avoid, so start there. Good luck!

Technology Leadership - A Misnomer?

If you've had a career of any reasonable length in the IT industry and have a track record of success behind you then chances are you're now in, or at some future point will be offered, a role which is considered a Technology Leadership role. In the data and information space these types of roles include Architects, Business Intelligence Managers, Competency Centre Leaders, Data Quality Managers and the like.

I've worked in many of these roles at various times and the one thing that they all have in common is that often this so called technology leadership is actually dealing with issues that have little to directly do with the technology at all. Rather the Technology Leader spends his days strategy setting, looking for ways to deliver business value through future change or the current actions of his people, engaged in stakeholder management, selling concepts to senior management, and so on. Most of these issues are actually more closely aligned with managing functions, people, shaping programs of work, etc and require some ability to control and direct and have access to, and authority over, budget and resourcing decisions. Without this the leadership may become divorced from the ability to deliver which can have serious ramifications on credibility, influence and ability to show effect and value from the leadership role. Perhaps it could even be argued that people in such roles are shouldering, and in some cases having to own, some of the concerns traditionally associated with line management without having enough say in solving those problems - i.e. their plans could be quashed, derailed or rerouted by others who do have control over budget, resourcing or strategy.

In order to allow our technology leaders to add the most value we need to realise and acknowledge their leadership is in fact only in part technology based. Our technology leaders need a seat at the management table to be truly effective. Taking this step delivers another benefit. Technology Leaders are in the decision loop early, ensuring better alignment with other initiatives and allowing others in the management structure better visibility of, consideration of, and therefore use of the area from which the "technology" leader comes.

The alternative to this arrangement is the introduction of Thought Leadership roles. These may well be ideal positions for the person formerly known as Technology Leader if the organization can stretch to that. However most can't due to lack of size, true need, commoditisation of many facets of the IT function, or shrinking budgets. The unkind (or is that better phrased as jealous?) spin on these roles might well be lots of thinking, musing and sprouting of opinion without the need for implementation efforts hampered by real world constraints and politics. Hey Gartner - if you're ever looking, I'm over here :)

BI - It's Not Just Reporting

Many times across my career I've been frustrated by what is, in my opinion, a gross under utilisation of business intelligence tools, platforms and even functions and teams such as Business Intelligence Competency Centres. All too often significant investment has been made in purchasing software, building data warehouses and other solutions, training staff and hiring external consultants, only to have this considerable horsepower and potential being set to work on (just) "the reporting".

Building BI solutions often comes at considerable cost in both time and money and if the chief item provided is purely a set of static reports that tell us what has happened in our organisations then someone, somewhere (and probably somewhere higher up in the organisational chart) will sooner or later start to question the wisdom of the investment. And that's not just a problem for the here and now of the current project, but for the future too. If he has been tainted as to the general value of business intelligence then he may well push back against future initiatives in the area, potentially robbing the organisation of future competitive advantage or effectiveness gains. The problem is made worse still when the default approach to setting project scope is to examine the set of existing reports available from in and around the incumbent solution. In this case the company will have a shiny new toy and (maybe) a prettier set of reports, but may not have gained much in the way of additional business value or actionable insight.

We need to remember that BI offers the potential for much more than just reporting. This was true well over a decade ago and is even more true today with the technology and tool advances we've seen between then and now. Just look at the name: BI, that's Business intelligence folks - intelligence for the business. Simply knowing what happened (via a report), for the sake of compliance, audit or because the procedures say we need to have a piece of paper punched and filed on a shelf, really does little for the running of the business, let alone its advancement. BI should be about helping make decisions that better the business, that help it carry out its operations in the best way possible.

The term "decision support" may seem a bit old school, but perhaps it does capture what BI can offer to the business other than just "the reports". Recently I heard the phrase "the three tenses of BI" and it resonated with me. The first phase is historical - this is the looking back, telling us what happened, function of BI - these are the reports that so often get developed as the majority, or even the entirety, of a BI project. The second tense is the present tense. BI instruments in this area tell us what is happening right now. Reports can serve this function too, but often they miss a key component that stops them playing an effective role. Whatever is used here, be it reports, OLAP components or some type of complex event processing, the information conveyed must be provided in a form and in a time frame that allows it to be used by the recipients in a manner and a timeliness which causes a change which benefits the business that would not otherwise have occurred. The third tense of BI looks to the future - using not only internal information but also looking at wider information sets to try and predict what may happen in the future in and around the organization, thereby providing food for thought for those setting the strategic direction for the organization. What's important to note there is that the second and third tenses of BI require information and intelligence to be presented in order not just to enable decisions to be made, but perhaps also go so far as to highlight that decisions should, or even must, be made.

Simply having a great set of tools available isn't enough. If we want to move on from being perceived as just report writers then it's up to us, as BI professionals, to educate others as to the potential of BI, challenging and assisting them to find ways in which it can be better used across our organisations. So go on, take the first step - share this blog post :)

When I Grow Up I Want to Be a Data Scientist

Some time in the early 1970s I formed my first career aspiration. When I grew up I wanted to be a bus. Yes, you read that right - not a bus driver, but a bus. It seems ridiculous now, some 40-odd years later, but I certainly wanted it bad back then! The reason it didn't seem crazy all those years ago is that I really had no idea what being a bus involved and what's more a bus seemed like a pretty cool thing to be - lots of the books that I had read to me featured buses zooming all over the countryside having all manner of exciting adventures, there was at least one cartoon where one of the main characters was a bus, and buses even had songs sung about them. Why wouldn't a three year old boy want to be a bus?

Fast forward 40 years and focus on the data industry: big data is being talked about in server rooms, board rooms and everywhere in between, and the term Data Scientist is becoming more and more common. It's a position I'd never heard of twelve months ago and now it seems hardly a day goes by where I don't see it mentioned in a blog post, industry email, or white paper, etc. A quick look at a number of job posting sites across the web confirms that organisations are indeed looking to hire people into roles with the Data Scientist title. But does anyone really know what the job entails, is there an agreed job description, even in a general sense? Judging by the diverse descriptions of the job ads I looked at it certainly doesn't seem so. I've talked before about my concerns that the Big Data concept is very much an overloaded and poorly understood term and I worry that the new "must have" role of Data Scientist is facing the same problems.

But, so what? Does this situation even matter? I think so. I applaud anything that raises the profile of those of us that work in the data and information space. I still remember what it was like to ride the wave in the early days of business intelligence and data warehousing and if others can have similar experiences and success today due to the next big thing being data centric then that's a good thing. However, I worry if the  shining beacon of the Data Scientist role starts to lure people from other facets of IT, or potentially worse still, attracts students entering University whilst the speciality is still in its infancy. With the role a Data Scientist will play, and how they will deliver value, yet to become understood in a consistent way then how can we expect our higher education institutions to prepare courses of study that will equip students with the skills that they will need when they reach the work force? I wonder if we will see such courses morphing, emerging and disappearing over coming years as industry and the universities try to better find what is expected of Data Scientists and how to build the knowledge and skills base they need to be successful. What of these early students, the guinea pigs or crash test dummies of the coming wave of graduate data scientists? Are we leading them astray, promising them areas of work, challenges and career paths which no-one yet knows will actually be there in the same form (or at all) five or ten years from now? Let's not promise too much too early. Better people come to the profession with their eyes open and with realistic views of what to expect, than simply because of hype.

Enough doom and gloom. If anyone can really tell me what a Data Scientist does, I'd love to know. I actually might want to be one when I grow up.

Data Ownership and Data Responsibility Should Go Hand in Hand

Chances are that all of us who have tried to advance a Data Governance initiative will have had the data ownership discussion and socialized the idea that IT shouldn't own the data. In the earlier years of my data governance efforts getting acceptance of this concept was most often the first stumbling block. Common remarks from both the business and IT fronts reflected long established culture and practices in which data was seen as an IT focus - they managed it, wrangled it, invested and got to the root cause of data problems, and they held the responsibility of fixing data problems. This resistance, although frustrating, at least had a common theme which allowed me (and I suspect others like me) to build up a toolkit of arguments, presentations, stories and case studies to help move stakeholders toward an understanding that data is best owned by the business and not the IT function.

I've written about the need to keep working away toward implementing data governance before in my post on Thinking Like a Vogon. Having a bag of tricks built up and finessed over time is, in my opinion, key to successfully importing data governance at an organization. It allows those of us charged with championing data governance to keep working toward advancing stakeholder understanding and buy-in over weeks, months and years. Recently I've noticed a trend which has the potential to poke a hole in this bag of tricks, reducing its effectiveness and necessitating a rethink of how I engage with stakeholders. These days I encounter less push back when I propose the idea that IT doesn't own the data. The concept that the business owns the data now often seems to be easily accepted, with some business users going as far as saying "of course we own the data!"

Acceptance! It's all easy from here, right? Maybe not. While there seems to be a lot more widespread agreement that business users own the data this doesn't always translate into the best outcomes for data governance and improved data quality. Statements from stakeholders along the lines of "we own the data, but IT let us down because they can't get the data quality right and they don't understand our data" ring alarm bells for me. That one step forward may just have been followed by two quick steps back. What's missing here is responsibility. Ownership without responsibility is hollow and reflects an understanding which still has quite some way to go to reach maturity. I liken this sort of statement to a father boasting about the achievements of his children before reaching for the phone to arrange a nanny to look after them whilst they are home from boarding school. Just like we can't expect school teachers and others to take sole responsibility for shaping our childrens' lives and helping them through their problem times, business users can not claim to be data owners and then take a passive role only getting involved in data issues long enough to point the finger of blame at IT.

So how can we this new phenomenon be dealt with? Sure, a part of what needs to be done is updating the message and tailoring the tools and communication methods we use to address this new position. But, it's more than that, as data governance professionals we need to look for ways of showing those in the business with this view "what's in it for them". We should look to find success stories - show where there was demonstrable benefit gained when another business user started to take an active interest in the health and well being of their data. Even better, we should spend time with those business users who have already embraced not just data ownership but data responsibility as well, helping to make them into champions for the cause and assisting them to sell the benefits to their colleagues. The broader IT function also has a role to play here. We need to ensure that IT doesn't reinforce the existing culture, and as such there needs to be a little gentle push back when the business shows signs of pushing all data responsibility to IT. This need not mean a flat out "no", nor an abdication of any and all responsibility, but rather taking these opportunities to engage with business users, working with them as enablers rather just than in isolation as technical troubleshooters blindly trying to solve problems often without enough context or knowledge. By working together - making changes on both sides - IT will boost data context familiarity and those on the business side will get involved more often and earlier. From a management perspective, look to change IT policies if they talk about IT owning and managing the data, and think about establishing data reference groups.

No matter what approaches we take, there is one very important thing that can not be forgotten. We must help business people in their new roles as data stewards. We can not leave them to feel their way along. Build documentation and tools to help and guide them, recognise new responsibilities by modifying Position Descriptions and the way these people are measured and rewarded for performance. Not only will this signal that the organisation is serious about data governance and data quality, but it will also guide and encourage the stewards to perform better.