Why IT Can't See 2 Plus 2 Doesn't Equal 5 (And Why It's Not Their Fault)

How do a loaf of bread and a small bottle of orange juice lead to a blog post about data? Here's how. Last week I bought (or at least tried to buy) a $4 loaf of bread and a $2 orange juice. The young lady (a trainee, according to her badge) behind the counter cheerfully informed me that it would be $27.60 and seemed a little taken aback when I asked if she was sure about the price. She assured me it was right and even went so far as to turn her monitor around to show me - "See, there it is!". Risking the chance of becoming the customer from hell, I pushed my claim that the price was wrong. Another staff member (herself probably not old enough to have left school) was called to assist and tried her best to educate the problem customer - "Look, the computer says that's what it is, so that's what it costs". It took me leaving the goods on the counter and turning to walk out to have the duty manager come over and straight away see the problem and charge me the correct price.

I bet we've all had at least one similar experience and laughed about it, putting it down to "kids today" or asking "what are the schools teaching now?". But step back for a second and think - should we really be so quick to judge? Take a less critical view and ask what is really happening. The shop assistants are most likely casual employees, which means they spend only a few hours each week in that environment, they don't live and breathe the goods they sell and couldn't tell you the price of a loaf of bread without referring to a price list. While they can do their job and function largely without mistakes if enough boundaries are set and systems provided, when something goes wrong or when the pressure is on they lack the familiarity and deeper understanding which lets them sense when something (e.g. my price) is wrong. Chances are that the math skills of the two casual shop assistants were just fine, it was the lack of context of the individual prices and not yet having a confidence gained from situational familiarity which prevented them from recognizing a problem and speaking out with a different answer than that given by the system.

Is there a parallel between this example and how we work with data in our organizations? I think so! I've certainly heard (on more than one occasion) people from the business bemoaning that IT just don't get it, they don't understand the data. If the IT folks don't work with that data much, only a few hours per week, or in blocks of time many months or years apart, then that's not surprising. Even if they do work on the data more frequently they don't have the context gained from working with the data in the same way the business users do. What's apparent (at least to me) is that this gap in understanding is not the fault of the IT staff. If people don't work with a dataset closely and regularly enough to gain familiarity and understanding then even what are [to others] glaringly obvious mistakes may well go unnoticed. In the same way as the shop manager needs to find a way to better equip her staff to deal with situations like mine, we (as Business and IT Managers) need to find ways to better deal with data centric issues in our organizations. This may happen in a number of ways, for example

• embedding IT staff in the business to have them work with the data and alongside the business users 
• having business people take an active and accountable role in data migration projects and not leaving IT staff to make snap judgement calls about the data at the pointy end of a project to hit a go-live date, or 
• bringing the idea of data governance into the organisation and getting the business people involved with decisions around the data.

No matter which of these approaches you adopt I think two things should always be present. Firstly don't blame the folks at the coal face, and secondly introduce and promote the idea that data ownership belongs with the business not the IT department. Miss these opportunities and chances are that, just like me with my bread and OJ,  at some stage your company will be asked to pay a much higher price than seems reasonable.

Data Migration: Don't Ask What Should I Take, Ask What Can I Leave Behind?

I've written before about the importance of Selling the Data Migration Strategy and getting senior management buy-in. Recently I've been engaged in that sales job within my own organisation and just the other day met some resistance. On past projects disagreements around my strategic intent often took the form of objecting to my position that migration is an activity that should be front and centre for the business with them, rather than the IT or IS function, having accountability for its success. However, that wasn't the case this time, instead the contentious issue was that the strategy I'd written didn't explicitly call out the we should only migrate the minimum amount of data.

From my point of view this isn't something that belongs in an Enterprise Data Migration Strategy. It's not that I don't think that keeping the volume of data to be migrated down has merit. I don't need convincing that involving more data increases the cost and heightens the risks of overruns or impacts outside of the organisation's walls but, even more importantly, I don't think it's my place to be mandating a minimum data approach in a blanket fashion. For me, each of our data migrations will have different needs, different profiles and different levels of data to be migrated. And let's not forget that it's the business, not the technical staff, who should be making the calls about how much data to migrate!

This discussion got me thinking. There is value in keeping migration data volumes low so how could I, through the strategy, encourage this behaviour without presupposing it was the right course of action for each migration project? After some thinking I've landed on updating the data migration strategy to include a new principle:

Migrate the appropriate level of data. 

Expanding this a little, I see two key points that should be considered in finding this appropriate level of data.

• Firstly, a default position of migrating all available data is not where we should start. 
• Secondly, and following on from this, don't just ask the question "how much data do we need to migrate", but instead try the question "how much data can we afford not to migrate - what data can we leave behind?" 

With this question as the starting point I think there is a way of ticking all the boxes: the appropriate amount of data is migrated, the data migration volume is kept as low as possible, and IT doesn't drive how much data is migrated - the business still gets to make this decision.

The unanswered question here is how does the business decide on the appropriate level of data? I've used various techniques to help in the past and I've some new ideas as well, but that's a discussion I'll leave for another blog post.

Big Data: Are We Really Ready for the Final Frontier?

I've just read an excellent post from Rich Murdane entitled "Big Data" and the Failure to Communicate. I'm glad Rich penned this piece as I was beginning to think I was either alone with my position and thoughts or, worse still, just missing the point! Rich has managed to nicely put into words a feeling that has been gradually building with me. For a few months now I've had a growing sense of disquiet that "big data" was becoming a buzz word - a phrase that was on everybody's lips but often with little substance behind it. That is, folks are talking about it because they feel they should be talking about it, but in reality they (and we as the broader industry) have not yet taken the time (or perhaps more kindly, had the time) to really think through what big data is and more importantly what it means to each of us and what  value it might deliver to our organisations.

I fear there is a real chance that much money will be spent too early as folks jump on the big data bandwagon before they they really have a good handle on which part of big data they should be focusing on. If we look at what "Big Data" is then a number of things spring to mind. For me the the obvious one is condition monitoring data - generally structured time series data. It's not just the size of this data that makes it big, it's how fast it arrives - what's beginning to be more commonly talked about as the velocity of data.

For me, and the big data problems I'm likely to face in my industry, I think this may well be the key issue at least in the short term. Operational technology is prevalent and in some cases this could mean thousands of data points arriving per second - huge waves of data washing in. One of the questions that I think needs to be considered is how high velocity big data should be used, or perhaps how the best value can be derived. Quite a bit of the discussion I'm hearing and reading suggests that this question is not being asked, but rather there's a leap of faith being taken resulting in a default position that this data needs to be stored and analyzed. I agree that over time this might well be the case, but for many right now I wonder if this value proposition is understood well enough to be effective. Perhaps it might be wiser to wait and look at other ways to gain value from this style of big data, one of which might be a form of complex event processing to monitor the stream of data and pluck out those events or short term trends which allow an action to be taken to materially impact operations. Those of us without a compelling business case to store, and later interpret across, huge data volumes might be best to wait and watch, taking action not because we feel that we should be, but because we understand what the value is and how we will unlock it.

This watch, monitor and wait brief should be wide enough to make sure that all forms of big data that might deliver value are considered. This means that it's not just the structured data area that is important, unstructured data is perhaps even bigger. For many of us with long pedigrees in the data space this may be somewhat of a new and unfamiliar area. I think we have no choice but to step outside of our comfort zones and become more familiar with this space. Much of the short to medium term value to be derived from big data, in my opinion, will come from this area for many industries. Sentiment analysis and mining of data from social networks is a great example of this which is already becoming important for many organizations. It's also an example where a stream of data can be monitored for data if interest as well as gaining value from analysis across larger pieces of historical data.

Big data is here to stay. It's not a matter of if it will impact your company, but now only a matter of when. You'll have to jump in and ride the wave at some time, just be sure you don't jump into too soon. Make sure you haven't worn out your funding and senior management support, will and trust before there is a real opportunity for you to harvest value from big data. Big Data may or may not be the final frontier. I could borrow from Star Trek, but I'd rather borrow from 1980s one hit wonder Star Trekkin'. It's Data Jim, but not as we know it. Good luck in your explorations, but be careful. There may not be Klingons on your starboard bow, but there is almost certain to be an executive or two asking the hard questions if his money disappears into a black hole with no value return.

Data Migration: Pushing Risk to the Vendor is not the Answer

Data migration exercises are risky. I've previously blogged about steps that can be taken to reduce risk in my posts on Selling the Data Migration Strategy and Data Migration: Track Record or Process Maturity. One area that I've not touched on in either of these posts, but one that has come up as an idea in pretty much every data migration project I've ever been involved with, is the use of the commercial negotiations process to handle data migration risks. Here I'm talking about payment milestones being closely tied to successful migration of the data. This must be a good thing, right? If our vendor doesn't get our data across to the new system then we don't pay them, or we withhold a sizable chunk of cash. I'm not so sure - for me this approach is at best a blunt instrument, something to beat a vendor over the head with, to cajole them into getting the job done.  

Step back and analyse the position you'd be in if you exercised the right to withhold some part of payment and you may start to share my point of view. OK, so your data migration project has hit problems, it's either running over time or the vendor has flat out just failed to deliver. That's not good, but at least you've got a good chunk of the project budget left, you've not wasted your company's money, or have you? Take a look at the projected benefits for the data migration project and / or the larger project the migration is part of and consider which of those benefits has either been forgone or deferred to some future date due to the migration problems. How do the financial outcomes look now? Sure, you may still have $200k in the project budget courtesy of the non delivery clause you negotiated, but you're going to have to spend that (and then some) to attempt the migration again, or fix what you have, with another vendor. Also some or all of the (financial savings) benefits the company was counting on in this year's budget now won't come through until next year (or later). Not the best situation to be in!

Of course, the counter argument can be made that by having financial penalties built in to the contract, or payment milestones tied to delivery milestones it's more likely that the vendor will be motivated to ensure that the job gets done, that your data gets migrated no matter what it takes. It's those five words which bother me - "no matter what it takes". What this may well mean is that vendor is sacrificing the true desired outcomes of your project in order to protect their financial interests. There will always be a degree of this apparent in any vendor- client relationship, but this is much easier to manage when there aren't extra pressures looming in the shape of potential financial forfeits. Shortcuts, a focus on short term fixes to specific problems rather than techniques in line with the best overall outcomes, and circumventing process and testing in order to hit deadlines are all behaviours you may need to deal with if you insist on these sorts of clauses. 

But what about the scenario where the vendor is the party suggesting this type of arrangement? In the past I've seen vendors offer arrangements where the lion's share of their payment (as high as 90% of the project total) is not due until the migration project is complete and accepted. You could argue that in such cases there is no harm in using these terms, after the vendor is offering the option freely, rather than being forced to this arrangement. In reality, the vendor may be being forced to these terms, not by you (their client) but rather by market forces. They may have to offer such arrangements in order to win work. Often this will be the case when they are inexperienced / new to the market space or have failed to build a portfolio of past successful projects which they can use as reference sites. There will be exceptions, but often the vendors who need to offer these types of terms are perhaps the vendors you least want to have on your data migration project. 



So, what's the alternative? In my opinion it's to be proactive. Don't push the risk to the vendor, but step up and own that risk yourself. Recognise and publicise the data migration risks early and take steps to mitigate them. It's your data so take accountability for the success of the data migration process. You can't afford to be simply a passive by-stander through the migration. If you are, when things fail (and they probably will at least to to some degree) yes, you'll be able to point the finger of blame at the vendor and say they failed to deliver.  But you'll still have a failed project on your resume, your company will have missed some or all of its projected benefits, and you may well have an extended miserable time as you attempt to mop up behind the project and live with work-arounds for years to come.  A data migration strategy, upfront planning, carefully selecting a vendor who is aligned to your strategy and likely to be able to deliver because they work effectively to an established and proven process, and staying actively involved throughout the migration are far more likely to keep  your risks from becoming realised issues. 


So, be brave, step up and own that risk. Good luck!

Implementing Data Governance: Think Like A Vogon

Recently I received a note from a prior colleague. He was in the midst of trying to implement data governance at an organization but was getting severe pushback, basically he saw he was headed for project failure. I had a couple of pieces of advice for him which I'll share here as I'm certain he's not the only person struggling to get data governance off the ground.

Establishing data governance at an organization is not easy, it will take time and chances are that the first approach you try may not work. In my opinion if you're working on something with a name along the lines of "Data Governance Project" then you've got a problem. For me, establishing data governance isn't a project. It's not something that can be up and running in a few short weeks or months, but rather I think of it as an initiative - something to be worked on over time, building toward an objective with the flexibility to change tack or even retreat, regroup and try again depending upon the appetite of the organization and the receptiveness of the executive and key stakeholders.

Which brings me to my first piece of advice: Think Like A Vogon. If you weren't lucky enough to be a teenager in the 1970s and 1980s then you may not know what a Vogon is so a little back story may be in order. The Vogons appeared in Douglas Adams' book / radio play / TV series The Hitchhikers' Guide to the Galaxy. Their battle cry was "resistance is useless!". If you're going to embark on a data governance initiative then you need to adopt this as your mantra. You need to keep telling yourself not to give up, you will strike resistance, but you can't stop at the first hurdle (or the second or third for that matter) . Keep telling yourself that resistance is useless because you will establish data governance in the end!

The Vogons used their battle cry as the answer to any dissenting argument or position. Their common approach was to simply keep shouting "RESISTANCE IS USELESS!" at whomever didn't agree with them.

This brings me to my second piece of advice: Think Like a Vogon, Don't Act Like a Vogon. Use the mantra as a means to keep you going when you face pushback or you struggle to get buy in for your ideas, not as something to beat people over the head with. If your only trick is the compliance line, e.g. telling / forcing people to comply with data governance rules because they are the rules then chances are you've missed the value pitch for data governance and you're going about it for all of the wrong reasons anyway. Stop telling people they have to comply and start telling them what's in it for them if they do come along for the ride. Show what past problems might have been avoided had some data governance been in place or find how some small form of data governance will drive some demonstrable short term savings and go and implement that small piece. Sell the value!

The Vogons were not just a race of bureaucrats, they also wrote poetry . Unfortunately this poetry was commonly thought to be the third worst in the universe. Essentially their poetry was  nonsense verse and decidedly unpleasant to be subjected to.

Which brings me to my third piece of advice: Make Your Artifacts More Appealing Than Vogon Poetry. If you have a ponderous Charter, a rambling Terms of Reference or information that the Data Stewards will find it hard to make sense of, then invest some time in making it more useful. Make your communications with your stakeholders clear and consistent, make your value pitch concise and tangible and construct the material for your stewards so that it's functional and easy to use.  My final piece of advice is to make the data governance initiative visible. Let people know about it and what it can do for them. Spend some time and effort raising the profile. If you can get others excited by its potential then you may get support, and even pull, for data governance from within the business. Don't be afraid to let others know about your data governance initiative. Believe me, it's not something to be hidden away in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying "Beware of the Leopard."

I'll leave you to your Vogon Googling. I'm off to find a Dentrassi to make me a nice cup of tea!

Data Migration: Track Record or Process Maturity?

What's more important when it comes to engaging a vendor in the data migration space - a track record of prior success or a demonstrable maturity of both company and process? I suspect the answer will differ from person to person and project to project. Having recently been involved in a few vendor selection exercises I've spent quite some time pondering this question.

The obvious response might be to favour the vendor with an enviable track record of successfully delivering their past projects. Surely putting your faith in an organization that has consistently and regularly delivered is a good thing - right? Maybe not! I'd advocate considering a few other things before signing that contract. For one, how long is that track record? A 100% success rate is impressive, but what does it really tell you if the company is a new player or is entering a new market space? It may be an indication of things to come, but it could equally well be a reflection of the firm's (and her people's) early stage enthusiasm and a "do whatever it takes" attitude that has got those projects delivered.   If this is the case, is this what you want for your project? A successful delivery may be the end result,  and maybe what people see when the look at the project I years to come, but what might it look like on the inside whilst the project is in motion? Throwing people and dollars at a problem to get a project over the line can bring undesirable side effects for those involved. Increased working hours, last minute ad hoc requests, a scattergun approach and a feeling that true outcomes may have been sacrificed to the Gods of the Gantt Chart all take their toll on the morale of those on the project - both on the vendor and customer sides. People can and do burn out. Your best internal people may jump ship, recognizing what's coming, while others on your team may tough it out but carry the scars of their experiences from the project back into their normal roles, potentially causing distrust or reluctance toward adoption of the new system as a by product of their attitude. Factors like these can and do hit project costs but, perhaps more importantly, also can have a larger impact on realization of the projected benefits of the project. The unfortunate truth here, though, is that often success will be recognized and rewarded when a challenged project finally gets delivered, with no one noticing that the really important outcomes were missed. Be careful what you measure! But I’ll leave the discussion of selecting appropriate KPIs and their impact on driving behavior to another blog post.

What’s the alternative? I’d advocate looking for evidence of process, and not just for the existence of a few pages talking academically about the process in a tender response, but some indication that that process has actually been used in an organization’s prior data migration engagements. When this is apparent (and verifiable) it’s usually indicative of a level of maturity.  Chances are that in reaching that level of maturity the vendor has learned a thing or two. Among the ones that are likely to materially matter to you are how to ensure their staff make good use of the process rather than just blindly follow it, and (hopefully) the fact that the process has been born of experience, that the scars and successes of past projects have all had some bearing on what will happen on contemporary data migration projects (yours included). Having such a process will mean the success of your upcoming data migration project is less dependent on the knowledge or experience of one or two key people but instead allows a broader group to be able to make choices and take effective actions. Actions which are likely to be well aligned with the desired end result and not just knee jerk responses to a current problem. This is especially important as time and delivery pressures come to the fore as the go live date looms closer. Going down this path should also mean that there is far less need for heroic efforts to deliver the project. Those long hours that many of us recognize from past migration projects should be the exceptions rather than a daily norm. Your people will be less likely to burn out and / or become jaded with the new system and you’ll avoid the creep of toxic feelings that may otherwise have undermined the system adoption. And who knows, with all that good stuff going on you might even successfully migrate the data AND get good benefits realization!