Data Migration: Low Hanging Fruit Can Be Bad For Your Health!

 If you’ve been through a few data migration exercises then you’ll probably recognise the scene.  You’re in the midst of a busy data migration, the pressure is on, there’s too much to do and too little time to do it in. You’ve been back and forward with the business users trying to resolve data quality issues that are preventing data from successfully loading in the new system. One, two, three, or more trial data loads have come and gone and still there are pieces of data which stubbornly refuse to play ball and load in to the new system.

With deadlines looming you start looking for ways to solve the problem, to migrate the data, to just get it in there. Or perhaps, even worse, those above you begin to apply the pressure for you to do the same. The questions / demands start to come. “Look for the low hanging fruit, what can you do to just make it work? Just get the data loaded!”

One common idea is that a whole parcel of load errors associated with free text fields can be easily and quickly solved by simply truncating the data to match the maximum allowable length of the fields in the target system.  Load the data in; produce an exception report for the text fields that were too long and were truncated and, hey presto, problem solved! You’ve got a bundle of errors out of the way and proved that the data can load. There’s now confidence that, come the next cutover trial or the real cutover and go live, that the migration in this area will be fine. Now it’s been done successfully once it can be repeated successfully and can only get better. After all the business users will get your exception report and immediately and diligently get to work on cleansing those fields which needed attention.

They will, won’t they? It’s not like they got anything else competing for their time. Their line manager doesn’t need them to do anything else. They’re ready and waiting to cleanse data for you as their top priority.

OK, wake up and head back over to the real world! Once (or if) they finish doing the day to day tasks of their own job, not to mention the tasks of those of their colleagues who have been seconded to that big project that is underway they may have time to add the data cleansing tasks to their to do lists, but I bet those tasks don’t get added to the top of the list.  If that data is going to get cleansed then you need to present a reason to prioritise its cleansing.  A hard failure is that reason, and a much more compelling and immediate reason than an exception report for data which (after all) has already loaded.  If the data doesn’t load then it needs to be cleansed if it is to be available I the new system.

Give the business all the help you can. Point out exactly and ambiguously what the problem is, show them exactly which field(s) in which record(s) need to be cleansed and what good looks like (i.e. what rules and conditions must be met before the data can migrate) and use data profiling tools to give them some indications ahead of the next trial load that the cleansing they are doing is effective and having the desired effect. But, don’t own their problem nor try to solve it for them in isolation. Most times, the business folds will know the data much better than the data migration folks. As soon as the data folks start making assumptions about what is safe to do with the data in the name of getting it to load, be it truncating text fields or some other technique, then the organisation takes on a long tail risk and liability in that the meaning of the data may have been lost or changed which may manifest in negative impacts to process or operations at some point post go-live. Bad for the organisation and bad for you as often times it is the data migration folks that will be perceived as having been the cause of the problem.

But, in the end, pragmatism must rule the day. Your data migration must run. There will be those (hopefully few) areas where you’ll just have to make some calls for the sake of allowing the data to get into the system at final cutover. If you don’t you’ll find your name quickly in the same sentence as the words “…we didn’t go live because of…” So go ahead, pick that low hanging fruit, but wait until you absolutely must, it still may be poisonous, but less immediately deadly than the alternative! 

Data as an Asset – Walk the Talk

Establishing data governance programs, data quality initiatives or other projects aimed at looking after or enhancing the well being of a company’s data can be difficult to get underway and sometimes even more challenging to keep alive.  Such initiatives (especially if your company’s funding approach forces you to tackle them as a series of individual projects) cannot always deliver the concrete and tangible “things” that you might get from a more traditional IT project. More often than not there is no new asset created, no new system to implement and enable across a go-live weekend, nor are benefits always obtained and measurable in the short term. Unless your business is facing real pain (and honestly attributes that pain to not just problems with the data itself, but also with how the data came to be in the state it is) then you may well struggle building support for, and keeping data initiatives running.

Often this seems surprising to data folks. It seems that this isn’t logical. The business says all of the right things about valuing data quality, about wanting a single source of truth from which to make better, faster and more cost effective decisions and maybe someone has even sprouted the “data as an asset” phrase, but this hasn’t resulted in them beating down the door with bags of money to fund data improvement initiatives. At first glance it seems key business stakeholders are showing moral support for the concepts but no commitment to the actions required to really solve the problems in the longer term.  They are the chicken to the data team’s pig.

But perhaps it’s not all that surprising. Are you, as a data leader, walking the walk or simply talking the talk? Do you treat your company’s data as an asset yourself? And does your data strategy reflect that? Do you even have a data strategy or are you one of the many, described by Joyce Norris-Montanari in her recent blog post  (http://www.dataroundtable.com/?p=12661), who fall into the group lacking a vision for their data and a strategy to help them get there?

If you’re not walking the talk then chances are that you’ll need some work up front to sell a congruent picture to the business, and particularly to those stakeholders holding the purse strings.   Make sure that you can show the basics around the company’s data that you would have around any other asset.

• Knowing what you have – an inventory of your data;
• Knowing where this data is – which systems house which data and (ideally) how it moves around between them;

• Knowing which stakeholders care about which data (you don’t need to get to business data ownership just yet, but at least you should know which data helps achieve, or threaten, which stakeholder’s bonus package);
• Knowing what preventive maintenance is required to keep your assets running at the required level and optimize the investment in them- some idea (and documentation) of where the problem areas lie. The big data quality issues (and importantly their real impact on the business);
• An understanding of how you can protect your asset – which may include classifications for data along with associated handling rules and other security aspects; and
• A vision for continuous improvement and a better future state of asset management – thought out and documented ideas for better use of your asset (which might include identifying and reducing redundancy, identifying inventory gaps or working better at the extreme ends of the asset life cycle).

With enough of the above in place you have your house in order and can say that you are indeed making efforts to treat data as you would any other asset. Now that you have a congruous story to present to the business you may find your next funding request is met more favourably. Even better, chances are you may well have identified a number of projects to deliver some concrete shorter-term benefits along the way as well.   

Could External Quality Assurance Hamper Your Chance of Data Migration Success?

I’m not sure of the exact rate for failures in data migration projects. Along the way I’ve seen Gartner report that somewhere around 83% of migrations either fail or overrun budgets and schedules and if memory serves I believe I’ve read that Forrester reported the success rate at around 16%.  The exact number probably depends upon who is doing the reporting, whom they survey and how candid the responses they receive are. Whatever the case, the number is big, scarily big.

To my way of thinking any area of a major project where the weight of historical evidence suggests that somewhere between 8 and 9 of every 10 attempts will be significantly challenged should be subject to two things:

•  External quality assurance processes in an attempt to make sure that the chance of success isn’t derailed by things not being done as they should. Adding another voice of experience or another set of eyes if you will; and

•  Some form of contribution to the wider data migration community of practice to help understand where things go wrong and over time (as a collective drawing from the positive and negative experience across many projects) look to evolve the methodologies used to undertake data migrations and lift the success rate.

Unfortunately, at least in my experience at least, the two items often work at cross-purposes. All too often I’ve seen the first endeavour block or even derail the second.  Quality assurance efforts will often be established as a risk mitigation exercise. That same aversion to risk often results in lack of comfort and confidence in anything which can’t be shown to have been done many times before. An established methodology is preferred over that which might be construed as cutting or bleeding edge.  That’s all well and good but, chances are, if you are following an established approach then that approach has been followed by a fair number of those 83% of projects that failed (to some degree) before yours.

This resistance to any attempt to stray from the well-worn path hinders the adoption and evolution of new concepts in so doing prevents them from gaining wider acceptance, development and enhancement over time by the wider crowd of data migration practitioners.

So we, as those practitioners, have two choices. We can accept that we can do little to change accepted practice, keep our heads down, collect our pay cheques and hope that luck or our best efforts place us in the lucky 17%, or we can look to find ways to not only increases the chances of success for our project but also contribute to the longer term average success rate of data migration projects in general. If we do want change then we must also recognise that radical shifts in methodology just won’t be possible; governance and quality assurance processes simply won’t allow that. Instead I think we must look for chance to use new techniques to build upon more accepted methodologies, filling the gaps or shoring up the problem areas that pose the biggest problems in our particular current projects.  This could take any number of forms from using lead indicators alongside lag indicators to build gradually build confidence across a project or the gradual introduction of new and improved approaches to the techniques and timing of reconciliation. 

Whatever, and however, we may go about this I hope that over time as a community of practitioners we can slowly build acceptance for new techniques, new methodologies and new measurement paradigms and over time slowly shift what is deemed to be acceptable and common practice. Who knows, maybe sometime before the end of my career we may actually see a failure rate that doesn’t send cold shivers down the collective spines of project managers everywhere. 

Dead Men Pay No Bills

There should be more data quality horror stories out there. No, really there should. When we data quality practitioners struggle to get stakeholder buy-in for our initiatives it's good to have a few in the kit bag to pull and illustrate the problems poor quality data can have. So, to that end, let me share a recent experience.

Mid-afternoon late last week my phone buzzed with the arrival of a new text message. It came from a number I didn't recognise: "Steven, we refer to our demand for payment of last week sent to your address in Midway Point. If this amount is not paid by Wednesday at 5PM legal action will be started. Please contact us on xxxxxxxx and quote reference number xxxxxxxx to arrange payment." My first thought was wrong number or phishing attempt. Either way I wasn't going to action the request. Odd though - my Father is Steven and he had a Midway Point address. Odder still in that he died almost 5 years ago.

Curiosity got the better of me. A phone call to the provided number revealed that it was in fact a debt supposedly owed by my Father to a local electricity retailer. That bill had been dealt with years before when the lawyers were wrapping up Dad's affairs, why was a debt collection service chasing payment now and why / how on earth were they chasing me?

This is squarely in brand damage territory. Not the stuff of front page headlines, but certainly fodder for those trashy tabloid "current affairs" TV shows, and certainly capable of causing some degree of negative sentiment, brand avoidance and customer churn. Putting the commercial implications aside, it also has potential for emotional distress. Five years since Dad's passing gave me the space need to see the absurdity of the situation, but I suspect a more recent death would cause unnecessary upset for the person receiving the payment demand.

Looking from the outside in I can only make educated guesses at what is likely to have happened to get to this point. Here goes:

1. I know the company replaced their customer care and billing system between Dad's death and the payment demand. I assume there was a data migration exercise at the same time. Could it be that some part of the data associated with Dad's account wasn't migrated successfully, or even attempted to be migrated at all? Perhaps only the structured transactional data was migrated and the non-structured data attached to the account, such as notes advising of his death, wasn't migrated, or lost its association during migration.

2. There has been a concerted push from the major shareholder of the utility to recover debts recently. What if some form of data matching had been applied in order to find my mobile phone number (previously not associated with Dad's account) and use it as another communications channel to attempt to recover the debt. Dad and I share the same first initial and last name, we have at various times in the last forty odd years shared the same address and we were born in the same month of the year. Did we somehow become fuzzily matched as the same person? I'm all for finding ways to get value from your data in new ways, but it's got to be based on solid foundations - if the data quality is not understood, acknowledged and good enough then the insights gained may well be flawed.

Who knows what really happened - I'm guessing as to the chain of events that transpired. But I am sure of one thing - this is one bill that won't be paid.

Why Data Quality is Like Shaving

Maintaining data quality is like shaving. That thought hit me early yesterday morning whilst I was doing it - shaving, that is, not maintaining data quality! Here's why.

A few things that most of us know about shaving and probably also know about data quality:

1. It's not much use if you just do it once. You've got to keep at it every day.
2. If you don't do it for a while you can still get back to a good state, it just might take a bit longer and be a bit uncomfortable for a while.
3. Sometimes there's blood!
4. Convincing someone who's never thought of data quality as important that they should do something about it is as hard as convincing someone who has had a beard for 20 years that they should shave it off.

And here are a few things that might not be quite so obvious about shaving (and about data quality undertakings):

1. If you don't do it most people won't notice nor care. But a few people who are really important to you might treat you differently and not give you what you want. Just ask my wife!
2. A little is good. Do a bit more and many people around you might look at you strangely but chances are the extra you did will pay you back many times over one day when you least expect it. Just ask a cyclist who has had to apply sticky bandages to his legs after a fall if it's easier with or without hair on his legs!
3. Doing it everywhere you possibly can is probably a wasted effort (and just a little bit creepy!) And if you must do it everywhere, then some areas will definitely need more caution during the exercise (refer to item 3 in the first list!)
4. The longer you've been around the more places you'll have to pay attention to. At 25 we all laughed at people with ear and nose hair, but it comes to us all in time. That data you've got hanging around from the time it was "faster" to work in spreadsheets will start showing eventually!
5. Shaving gives you the chance to take a good look at yourself in the mirror everyday, often without any of your imperfections covered up. Maintaining data quality calls for the same regular scrutiny and openness.

And here's the biggest thing most people will immediately accept about shaving and should accept about data quality too:

Chances are you're not going to ask someone from your internal hairdressing department to keep you clean-shaven. It will work far better if you take care of that yourself. Data quality's no different. It's your beard and it's your data!

So That's What IT Recruiters Are Good For!

The BI Market seems to be fairly hot right now, at least in Australia, if the number of calls I'm getting from headhunters and IT Recruiters is any sort of indicator. However an experience I had this week makes me wonder.

Right now I have a need to ramp up the number of BI folks in our company to handle some project demands. It's a short term gig for the time being so we directly approached a few service firms as well as posting in one or two LinkedIn groups. Two days later I received an email from an IT Recruiter offering me the chance to interview for a great BI position with a first class company there were working very closely with. The position description felt very familiar and no wonder, I'd written most of it only a few days earlier. It had been augmented a bit to give the impression that what was in reality a short term engagement with a slim chance of extension was instead just the initial piece of a long engagement. A hunch had me checking Australia's main jobs website. Sure enough, two more recruiters were also advertising jobs which were also obviously the roles I was trying to fill. Both had extra or erroneous information, I suspect to make their version of the job more attractive than the one being advertised by their competitors.

Recruitment "consultants" taking initiative and creating opportunities to make income for themselves and their companies is one thing, but it causes problems for those of us working in, or managing those working in, the BI arena. With multiple parties now advertising my two roles now seemed like eight, so when those two are filled it will seem as if many more roles have been quickly filled. I suspect this contributes to the general feeling that demand is high and might even be artificially forcing rates up. Worse still will it cause a scenario where this perception of great demand draws people from other IT areas to re-train in the BI field, potentially causing a future glut of supply?

So let's be generous and say that recruitment consultants generate a buzz for us and drive potential candidates our way, but those of us with our own Human Resource Departments could do that anyway. What else do recruiters bring to the table? In my experience the majority of those I've dealt with lack any depth of understanding - matching skills and expertise to roles is haphazard at best, and even basic screening of applicants seems limited to "if they can spell BI then they're qualified to get a second interview".

So, for me I struggle to see the value..... But then I had an experience that made me realize that recruitment consultants do serve a valuable purpose - they're here to make us laugh. I was approached by a recruiter interested in my data warehousing expertise and keen to learn if I had a forklift license and what my typing speed was. I suspect data warehousing may have been a new term for him! Another, during a preliminary chat when I was trying to find a candidate to fill a new data warehousing role, asked me if I felt rising real estate costs would make warehouse space more expensive and cause my firm to lift our rates.

So, guys, keep the laughs coming, but you'll excuse me if I pass on your other offerings!

The Next Data Rock Star

I miss the 1980s, there was lots to like about that time: great music, a lifestyle less cluttered by technology, crazy hair, the fact that I could still grow enough hair to have crazy hair. For most of my adulthood if you had asked me which time in my life I'd like to go back to I'd have picked the 1980s. But now I'm not so sure, I'm starting to pine for the late 1990s through to the mid 2000s. Why? Because, that was a time when quite a few data folks working in the business intelligence arena were the rock stars of IT.

I'm lucky enough to have worked on my fair share of interesting and high profile data warehousing and BI gigs during that period. The problems I had the privilege of working on were often difficult, few (if any) commodity solutions or services had manifested, and the solutions we provided were often anything but cheap. However in solving those problems we added real value to the business, at times doing things that their traditional IT staff or software vendors had said couldn't be done or had tried (and failed) to solve with more application centric approaches. Perhaps most importantly during that period few BI projects were undertaken unless the business could really see the benefit. Maybe this was because of the high cost, maybe not. Whatever the reason, those of us lucky enough to be leading the charge and playing key roles in these projects often got the accolades of not just our IT and data colleagues but also  of key business stakeholders as well. I was fortunate to parlay this into a successful consulting career  with word of mouth securing a series of back to back BI engagements stretching for over a decade.

But I fear that the BI Rock Star is a thing of the past. It's becoming a crowded field. Any number of people now "do BI" and what was once bordering on black magic is commodity thanks to advances in (BI) software products and hardware processing power. Even worse, the phrase "reporting" is commonly be used almost interchangeably with "business intelligence". We've all too often moved away using BI undertakings to support complex decisions and solve wicked problems to one where often all too many reports are produced for banal reasons or reasons unknown. It's no wonder then that business stakeholders seem to be seeing less value from their business intelligence dollar.

So, if BI is indeed on the wane, I wonder where the next Data Rock Star will come from. If I was to encourage my kids into the field where would I point them? Will the Data Scientists be the next wave to be lauded by the business, or will those jumping on the Big Data bandwagon get that privilege? Maybe those playing in the NoSQL space are about to be thrust to glory, or perhaps the social media tsunami will mean that's it the intersection of all three which will be the sweet spot. I actually think it won't be any of these. Both Big Data and Data Scientists to me still feel like buzz words - hot topics - that the industry has thrown up and bandied about to the point where airline magazine syndrome has kicked in. I don't see the clear path to the wide stream killer need that means business stakeholders will clamour for solutions in these areas enmasse any time soon.

So, if you can see more clearly than I, I'd love to know what the next big data thing will be. I miss being a rock star!