Data Migration: Low Hanging Fruit Can Be Bad For Your Health!

 If you’ve been through a few data migration exercises then you’ll probably recognise the scene.  You’re in the midst of a busy data migration, the pressure is on, there’s too much to do and too little time to do it in. You’ve been back and forward with the business users trying to resolve data quality issues that are preventing data from successfully loading in the new system. One, two, three, or more trial data loads have come and gone and still there are pieces of data which stubbornly refuse to play ball and load in to the new system.

With deadlines looming you start looking for ways to solve the problem, to migrate the data, to just get it in there. Or perhaps, even worse, those above you begin to apply the pressure for you to do the same. The questions / demands start to come. “Look for the low hanging fruit, what can you do to just make it work? Just get the data loaded!”

One common idea is that a whole parcel of load errors associated with free text fields can be easily and quickly solved by simply truncating the data to match the maximum allowable length of the fields in the target system.  Load the data in; produce an exception report for the text fields that were too long and were truncated and, hey presto, problem solved! You’ve got a bundle of errors out of the way and proved that the data can load. There’s now confidence that, come the next cutover trial or the real cutover and go live, that the migration in this area will be fine. Now it’s been done successfully once it can be repeated successfully and can only get better. After all the business users will get your exception report and immediately and diligently get to work on cleansing those fields which needed attention.

They will, won’t they? It’s not like they got anything else competing for their time. Their line manager doesn’t need them to do anything else. They’re ready and waiting to cleanse data for you as their top priority.

OK, wake up and head back over to the real world! Once (or if) they finish doing the day to day tasks of their own job, not to mention the tasks of those of their colleagues who have been seconded to that big project that is underway they may have time to add the data cleansing tasks to their to do lists, but I bet those tasks don’t get added to the top of the list.  If that data is going to get cleansed then you need to present a reason to prioritise its cleansing.  A hard failure is that reason, and a much more compelling and immediate reason than an exception report for data which (after all) has already loaded.  If the data doesn’t load then it needs to be cleansed if it is to be available I the new system.

Give the business all the help you can. Point out exactly and ambiguously what the problem is, show them exactly which field(s) in which record(s) need to be cleansed and what good looks like (i.e. what rules and conditions must be met before the data can migrate) and use data profiling tools to give them some indications ahead of the next trial load that the cleansing they are doing is effective and having the desired effect. But, don’t own their problem nor try to solve it for them in isolation. Most times, the business folds will know the data much better than the data migration folks. As soon as the data folks start making assumptions about what is safe to do with the data in the name of getting it to load, be it truncating text fields or some other technique, then the organisation takes on a long tail risk and liability in that the meaning of the data may have been lost or changed which may manifest in negative impacts to process or operations at some point post go-live. Bad for the organisation and bad for you as often times it is the data migration folks that will be perceived as having been the cause of the problem.

But, in the end, pragmatism must rule the day. Your data migration must run. There will be those (hopefully few) areas where you’ll just have to make some calls for the sake of allowing the data to get into the system at final cutover. If you don’t you’ll find your name quickly in the same sentence as the words “…we didn’t go live because of…” So go ahead, pick that low hanging fruit, but wait until you absolutely must, it still may be poisonous, but less immediately deadly than the alternative! 

Could External Quality Assurance Hamper Your Chance of Data Migration Success?

I’m not sure of the exact rate for failures in data migration projects. Along the way I’ve seen Gartner report that somewhere around 83% of migrations either fail or overrun budgets and schedules and if memory serves I believe I’ve read that Forrester reported the success rate at around 16%.  The exact number probably depends upon who is doing the reporting, whom they survey and how candid the responses they receive are. Whatever the case, the number is big, scarily big.

To my way of thinking any area of a major project where the weight of historical evidence suggests that somewhere between 8 and 9 of every 10 attempts will be significantly challenged should be subject to two things:

•  External quality assurance processes in an attempt to make sure that the chance of success isn’t derailed by things not being done as they should. Adding another voice of experience or another set of eyes if you will; and

•  Some form of contribution to the wider data migration community of practice to help understand where things go wrong and over time (as a collective drawing from the positive and negative experience across many projects) look to evolve the methodologies used to undertake data migrations and lift the success rate.

Unfortunately, at least in my experience at least, the two items often work at cross-purposes. All too often I’ve seen the first endeavour block or even derail the second.  Quality assurance efforts will often be established as a risk mitigation exercise. That same aversion to risk often results in lack of comfort and confidence in anything which can’t be shown to have been done many times before. An established methodology is preferred over that which might be construed as cutting or bleeding edge.  That’s all well and good but, chances are, if you are following an established approach then that approach has been followed by a fair number of those 83% of projects that failed (to some degree) before yours.

This resistance to any attempt to stray from the well-worn path hinders the adoption and evolution of new concepts in so doing prevents them from gaining wider acceptance, development and enhancement over time by the wider crowd of data migration practitioners.

So we, as those practitioners, have two choices. We can accept that we can do little to change accepted practice, keep our heads down, collect our pay cheques and hope that luck or our best efforts place us in the lucky 17%, or we can look to find ways to not only increases the chances of success for our project but also contribute to the longer term average success rate of data migration projects in general. If we do want change then we must also recognise that radical shifts in methodology just won’t be possible; governance and quality assurance processes simply won’t allow that. Instead I think we must look for chance to use new techniques to build upon more accepted methodologies, filling the gaps or shoring up the problem areas that pose the biggest problems in our particular current projects.  This could take any number of forms from using lead indicators alongside lag indicators to build gradually build confidence across a project or the gradual introduction of new and improved approaches to the techniques and timing of reconciliation. 

Whatever, and however, we may go about this I hope that over time as a community of practitioners we can slowly build acceptance for new techniques, new methodologies and new measurement paradigms and over time slowly shift what is deemed to be acceptable and common practice. Who knows, maybe sometime before the end of my career we may actually see a failure rate that doesn’t send cold shivers down the collective spines of project managers everywhere. 

Dead Men Pay No Bills

There should be more data quality horror stories out there. No, really there should. When we data quality practitioners struggle to get stakeholder buy-in for our initiatives it's good to have a few in the kit bag to pull and illustrate the problems poor quality data can have. So, to that end, let me share a recent experience.

Mid-afternoon late last week my phone buzzed with the arrival of a new text message. It came from a number I didn't recognise: "Steven, we refer to our demand for payment of last week sent to your address in Midway Point. If this amount is not paid by Wednesday at 5PM legal action will be started. Please contact us on xxxxxxxx and quote reference number xxxxxxxx to arrange payment." My first thought was wrong number or phishing attempt. Either way I wasn't going to action the request. Odd though - my Father is Steven and he had a Midway Point address. Odder still in that he died almost 5 years ago.

Curiosity got the better of me. A phone call to the provided number revealed that it was in fact a debt supposedly owed by my Father to a local electricity retailer. That bill had been dealt with years before when the lawyers were wrapping up Dad's affairs, why was a debt collection service chasing payment now and why / how on earth were they chasing me?

This is squarely in brand damage territory. Not the stuff of front page headlines, but certainly fodder for those trashy tabloid "current affairs" TV shows, and certainly capable of causing some degree of negative sentiment, brand avoidance and customer churn. Putting the commercial implications aside, it also has potential for emotional distress. Five years since Dad's passing gave me the space need to see the absurdity of the situation, but I suspect a more recent death would cause unnecessary upset for the person receiving the payment demand.

Looking from the outside in I can only make educated guesses at what is likely to have happened to get to this point. Here goes:

1. I know the company replaced their customer care and billing system between Dad's death and the payment demand. I assume there was a data migration exercise at the same time. Could it be that some part of the data associated with Dad's account wasn't migrated successfully, or even attempted to be migrated at all? Perhaps only the structured transactional data was migrated and the non-structured data attached to the account, such as notes advising of his death, wasn't migrated, or lost its association during migration.

2. There has been a concerted push from the major shareholder of the utility to recover debts recently. What if some form of data matching had been applied in order to find my mobile phone number (previously not associated with Dad's account) and use it as another communications channel to attempt to recover the debt. Dad and I share the same first initial and last name, we have at various times in the last forty odd years shared the same address and we were born in the same month of the year. Did we somehow become fuzzily matched as the same person? I'm all for finding ways to get value from your data in new ways, but it's got to be based on solid foundations - if the data quality is not understood, acknowledged and good enough then the insights gained may well be flawed.

Who knows what really happened - I'm guessing as to the chain of events that transpired. But I am sure of one thing - this is one bill that won't be paid.

Congratulations, Your Data Migration Project Was a Success! Or Was It?

In recent years I've begun to notice something amongst data migration providers that I'd seen little of before. Many of them (or at least many of those that I've had dealings with) are beginning to be far more proactive in taking steps to ensure that their migration projects are successful. That's a good thing, isn't it? Well I guess that depends how you define your success metrics. Does a successful migration always translate into benefit (or at least lack of significant pain) for the enterprise?

Perhaps the most noticeable shift has been towards the "less is more" philosophy. That is, rather than trying to take all of the data that's available, or that the business users say they need, there is a push back to reduce the amount of data migrated between systems. In my opinion this is a good thing - not migrating data that isn't required should remove complexity, decrease cost and time lines and, perhaps most significantly, reduce the risk of unexpected problems leading to delays and cost blowouts. For me there are a number of scenarios when this approach makes a lot of sense:

• Some of the data can be shown to add no value to business operations (that is it's only required for compliance or records management requirements) AND the business users agree that this is the case;
• There have been several past restructures to business operations resulting in data which is very hard to map between different charts of account or the like;
• The cost of migrating the data is significantly higher than the value that having it available in the new system would deliver AND not migrating it does not expose the company to any significant additional risk.

However, I've seen other arguments for this approach, among them:

• Only open balances and other current data are required for the business to operate;
• Migrating anything over and above current data introduces more cost and risk of project time overruns;
• It will be more cost effective to use alternate means (other than migrating it) to get to old data, including:
• Leaving the old system in place in a read only state;
• Leaving the old data available via a an existing data warehouse or BI solution;
• Leaving (just) the database from the legacy solution in place and having IT staff query it to service user requests.

There's a common thread amongst these items - they all focus on a successful project outcome, i.e. getting the project delivered on time, whilst making some major assumptions about life post migration. So, if you're faced with a migration project where your vendor is proposing this style of approach, or even if you yourself are considering this option, take some time to validate those assumptions. Ask the business users if they really can operate on that minimal data set. If they say yes then ask them some more questions, paint a picture of how the world well be and test their theory - how will they do trend reporting, how will they respond to an e-discovery request, how will they bring a new product to market quickly or design future sales and marketing campaigns in a timely fashion?

Don't just talk to the business users, spend some time with the IT folks as well. What's the impact to them of having to keep old systems around for some (potentially significant) amount of time? Chances are IT Management won't have budgeted for having to continue to pay licence fees for the old software on top of the new fees, let alone the operational support costs of keeping the computing hardware that it runs upon up and running. This can be made worse still if a third party supports this hardware and requires that it be kept in warranty as then there may well be new hardware purchases, product installations and potentially even a migration of the old system (to the old system) as well. And then there's the need to maintain support and help desk skills with the old products. At best this means having to keep staff cross skilled, potentially reducing their ability to perform higher order work, but it may also mean the need to bring on new staff, adding head count and increasing wage costs. A tricky situation becomes even worse if the project business case doesn't recognise the need for increased IT operational spend or projects IT savings from the increased efficiency the new system will bring. 

Only when you've satisfied yourself that the assumptions are understood and hold should you proceed with reducing the data migration volume. Chances are there will be a middle ground where the data required is more than the bare minimum set and both the business and IT can operate without significant compromise - aim here! Hit this and there's every chance that your migration project will be seen as successful not just at go-live but also in the weeks, months and years that follow. Aim for the minimum migration set, without first validating the approach, then enjoy the accolades at go-live, but hit the road in a hurry because it won't be long before both business and IT managers are queuing up at your door to ask some hard questions. Good luck!

Your Data Migration Project is Not a Data Quality Exercise

If I look back across all of the data migration exercises I’ve been involved with one aspect has come up during every single one of them – at some point someone involved with the project will try and seize the opportunity to use the migration as the means by which all past data quality sins will be dealt with and solved.  There was a time when I’d have embraced this idea and tried to incorporate it in to the project plan, on more than one occasion I may have even been the person advancing the idea.

But not any more. That was then, this is now. Looking back over some of the projects which struck problems, one recurring theme seems to have been a push for wide ranging data quality improvements as one of the key deliverables, either as a mid stream addition to, or change of, focus or a key aim from day one.  I’ve reached a conclusion – looking at it now it seems stupidly obvious, self evident if you like, but I suspect it may seen a tad controversial to more than a few people out there:

Data migration exercises are about migrating data. This should be the first and only aim. If you don’t get your data from the old to the new system then you have failed – it’s a simple as that. All of those forecast business benefits that rely upon having data migrated will be reduced, deferred or missed entirely. Don’t even think about using the migration to do anything else, data migration is hard enough without making a conscious choice to try and increase the complexity, cost, timeframe and risk. Don’t do it to yourself!!

I’d suggest only entertaining the concept of improving data quality when it can be shown that the new system can’t and won’t operate successfully with the level of quality of data currently available in the incumbent system. Then, and only then, look to incorporate data quality improvements, but keep them as light as possible. Only improve the data as much as is required to allow the new system to operate. This may seem counterintuitive, even perhaps bordering on delinquency of our duty as data professionals, but step back and look at the bigger picture, the interconnectivity of the data in (and across) systems. The changes you may make to improve a particular aspect of the data in one area may have unforeseen and impacts elsewhere. Sure, data profiling can go some way to help understand the impacts here, but do you really want to be following convoluted lineage trails just to be sure that no nasty surprise side effects crop up? Worse still, much improved data quality in one area of the system will soon be forgotten if whole other areas can’t even get the base data they need to operate due to migration woes.

Now, before all of you data quality folks delete my blog address from your favourites folder, let me reassure you that I’m not saying that we shouldn’t improve data quality. In fact I think that the arrival of a new system and the data migration project that comes with it is a great opportunity to push for funding to improve data quality. We just need to play the long game and look a little further ahead - designing projects to make specific data quality improvements before the data migration even begins. Start selling the benefits of improved data quality well ahead of the migration (months, if not a year or more) and angle the pitch in such a way as to show how these projects will reduce the risk of your company becoming the next data migration horror story headline with huge cost overruns, customer churn and brand damage.

Why IT Can't See 2 Plus 2 Doesn't Equal 5 (And Why It's Not Their Fault)

How do a loaf of bread and a small bottle of orange juice lead to a blog post about data? Here's how. Last week I bought (or at least tried to buy) a $4 loaf of bread and a $2 orange juice. The young lady (a trainee, according to her badge) behind the counter cheerfully informed me that it would be $27.60 and seemed a little taken aback when I asked if she was sure about the price. She assured me it was right and even went so far as to turn her monitor around to show me - "See, there it is!". Risking the chance of becoming the customer from hell, I pushed my claim that the price was wrong. Another staff member (herself probably not old enough to have left school) was called to assist and tried her best to educate the problem customer - "Look, the computer says that's what it is, so that's what it costs". It took me leaving the goods on the counter and turning to walk out to have the duty manager come over and straight away see the problem and charge me the correct price.

I bet we've all had at least one similar experience and laughed about it, putting it down to "kids today" or asking "what are the schools teaching now?". But step back for a second and think - should we really be so quick to judge? Take a less critical view and ask what is really happening. The shop assistants are most likely casual employees, which means they spend only a few hours each week in that environment, they don't live and breathe the goods they sell and couldn't tell you the price of a loaf of bread without referring to a price list. While they can do their job and function largely without mistakes if enough boundaries are set and systems provided, when something goes wrong or when the pressure is on they lack the familiarity and deeper understanding which lets them sense when something (e.g. my price) is wrong. Chances are that the math skills of the two casual shop assistants were just fine, it was the lack of context of the individual prices and not yet having a confidence gained from situational familiarity which prevented them from recognizing a problem and speaking out with a different answer than that given by the system.

Is there a parallel between this example and how we work with data in our organizations? I think so! I've certainly heard (on more than one occasion) people from the business bemoaning that IT just don't get it, they don't understand the data. If the IT folks don't work with that data much, only a few hours per week, or in blocks of time many months or years apart, then that's not surprising. Even if they do work on the data more frequently they don't have the context gained from working with the data in the same way the business users do. What's apparent (at least to me) is that this gap in understanding is not the fault of the IT staff. If people don't work with a dataset closely and regularly enough to gain familiarity and understanding then even what are [to others] glaringly obvious mistakes may well go unnoticed. In the same way as the shop manager needs to find a way to better equip her staff to deal with situations like mine, we (as Business and IT Managers) need to find ways to better deal with data centric issues in our organizations. This may happen in a number of ways, for example

• embedding IT staff in the business to have them work with the data and alongside the business users 
• having business people take an active and accountable role in data migration projects and not leaving IT staff to make snap judgement calls about the data at the pointy end of a project to hit a go-live date, or 
• bringing the idea of data governance into the organisation and getting the business people involved with decisions around the data.

No matter which of these approaches you adopt I think two things should always be present. Firstly don't blame the folks at the coal face, and secondly introduce and promote the idea that data ownership belongs with the business not the IT department. Miss these opportunities and chances are that, just like me with my bread and OJ,  at some stage your company will be asked to pay a much higher price than seems reasonable.