Data Migration: Low Hanging Fruit Can Be Bad For Your Health!

 If you’ve been through a few data migration exercises then you’ll probably recognise the scene.  You’re in the midst of a busy data migration, the pressure is on, there’s too much to do and too little time to do it in. You’ve been back and forward with the business users trying to resolve data quality issues that are preventing data from successfully loading in the new system. One, two, three, or more trial data loads have come and gone and still there are pieces of data which stubbornly refuse to play ball and load in to the new system.

With deadlines looming you start looking for ways to solve the problem, to migrate the data, to just get it in there. Or perhaps, even worse, those above you begin to apply the pressure for you to do the same. The questions / demands start to come. “Look for the low hanging fruit, what can you do to just make it work? Just get the data loaded!”

One common idea is that a whole parcel of load errors associated with free text fields can be easily and quickly solved by simply truncating the data to match the maximum allowable length of the fields in the target system.  Load the data in; produce an exception report for the text fields that were too long and were truncated and, hey presto, problem solved! You’ve got a bundle of errors out of the way and proved that the data can load. There’s now confidence that, come the next cutover trial or the real cutover and go live, that the migration in this area will be fine. Now it’s been done successfully once it can be repeated successfully and can only get better. After all the business users will get your exception report and immediately and diligently get to work on cleansing those fields which needed attention.

They will, won’t they? It’s not like they got anything else competing for their time. Their line manager doesn’t need them to do anything else. They’re ready and waiting to cleanse data for you as their top priority.

OK, wake up and head back over to the real world! Once (or if) they finish doing the day to day tasks of their own job, not to mention the tasks of those of their colleagues who have been seconded to that big project that is underway they may have time to add the data cleansing tasks to their to do lists, but I bet those tasks don’t get added to the top of the list.  If that data is going to get cleansed then you need to present a reason to prioritise its cleansing.  A hard failure is that reason, and a much more compelling and immediate reason than an exception report for data which (after all) has already loaded.  If the data doesn’t load then it needs to be cleansed if it is to be available I the new system.

Give the business all the help you can. Point out exactly and ambiguously what the problem is, show them exactly which field(s) in which record(s) need to be cleansed and what good looks like (i.e. what rules and conditions must be met before the data can migrate) and use data profiling tools to give them some indications ahead of the next trial load that the cleansing they are doing is effective and having the desired effect. But, don’t own their problem nor try to solve it for them in isolation. Most times, the business folds will know the data much better than the data migration folks. As soon as the data folks start making assumptions about what is safe to do with the data in the name of getting it to load, be it truncating text fields or some other technique, then the organisation takes on a long tail risk and liability in that the meaning of the data may have been lost or changed which may manifest in negative impacts to process or operations at some point post go-live. Bad for the organisation and bad for you as often times it is the data migration folks that will be perceived as having been the cause of the problem.

But, in the end, pragmatism must rule the day. Your data migration must run. There will be those (hopefully few) areas where you’ll just have to make some calls for the sake of allowing the data to get into the system at final cutover. If you don’t you’ll find your name quickly in the same sentence as the words “…we didn’t go live because of…” So go ahead, pick that low hanging fruit, but wait until you absolutely must, it still may be poisonous, but less immediately deadly than the alternative! 

Data as an Asset – Walk the Talk

Establishing data governance programs, data quality initiatives or other projects aimed at looking after or enhancing the well being of a company’s data can be difficult to get underway and sometimes even more challenging to keep alive.  Such initiatives (especially if your company’s funding approach forces you to tackle them as a series of individual projects) cannot always deliver the concrete and tangible “things” that you might get from a more traditional IT project. More often than not there is no new asset created, no new system to implement and enable across a go-live weekend, nor are benefits always obtained and measurable in the short term. Unless your business is facing real pain (and honestly attributes that pain to not just problems with the data itself, but also with how the data came to be in the state it is) then you may well struggle building support for, and keeping data initiatives running.

Often this seems surprising to data folks. It seems that this isn’t logical. The business says all of the right things about valuing data quality, about wanting a single source of truth from which to make better, faster and more cost effective decisions and maybe someone has even sprouted the “data as an asset” phrase, but this hasn’t resulted in them beating down the door with bags of money to fund data improvement initiatives. At first glance it seems key business stakeholders are showing moral support for the concepts but no commitment to the actions required to really solve the problems in the longer term.  They are the chicken to the data team’s pig.

But perhaps it’s not all that surprising. Are you, as a data leader, walking the walk or simply talking the talk? Do you treat your company’s data as an asset yourself? And does your data strategy reflect that? Do you even have a data strategy or are you one of the many, described by Joyce Norris-Montanari in her recent blog post  (http://www.dataroundtable.com/?p=12661), who fall into the group lacking a vision for their data and a strategy to help them get there?

If you’re not walking the talk then chances are that you’ll need some work up front to sell a congruent picture to the business, and particularly to those stakeholders holding the purse strings.   Make sure that you can show the basics around the company’s data that you would have around any other asset.

• Knowing what you have – an inventory of your data;
• Knowing where this data is – which systems house which data and (ideally) how it moves around between them;

• Knowing which stakeholders care about which data (you don’t need to get to business data ownership just yet, but at least you should know which data helps achieve, or threaten, which stakeholder’s bonus package);
• Knowing what preventive maintenance is required to keep your assets running at the required level and optimize the investment in them- some idea (and documentation) of where the problem areas lie. The big data quality issues (and importantly their real impact on the business);
• An understanding of how you can protect your asset – which may include classifications for data along with associated handling rules and other security aspects; and
• A vision for continuous improvement and a better future state of asset management – thought out and documented ideas for better use of your asset (which might include identifying and reducing redundancy, identifying inventory gaps or working better at the extreme ends of the asset life cycle).

With enough of the above in place you have your house in order and can say that you are indeed making efforts to treat data as you would any other asset. Now that you have a congruous story to present to the business you may find your next funding request is met more favourably. Even better, chances are you may well have identified a number of projects to deliver some concrete shorter-term benefits along the way as well.   

Why Data Quality is Like Shaving

Maintaining data quality is like shaving. That thought hit me early yesterday morning whilst I was doing it - shaving, that is, not maintaining data quality! Here's why.

A few things that most of us know about shaving and probably also know about data quality:

1. It's not much use if you just do it once. You've got to keep at it every day.
2. If you don't do it for a while you can still get back to a good state, it just might take a bit longer and be a bit uncomfortable for a while.
3. Sometimes there's blood!
4. Convincing someone who's never thought of data quality as important that they should do something about it is as hard as convincing someone who has had a beard for 20 years that they should shave it off.

And here are a few things that might not be quite so obvious about shaving (and about data quality undertakings):

1. If you don't do it most people won't notice nor care. But a few people who are really important to you might treat you differently and not give you what you want. Just ask my wife!
2. A little is good. Do a bit more and many people around you might look at you strangely but chances are the extra you did will pay you back many times over one day when you least expect it. Just ask a cyclist who has had to apply sticky bandages to his legs after a fall if it's easier with or without hair on his legs!
3. Doing it everywhere you possibly can is probably a wasted effort (and just a little bit creepy!) And if you must do it everywhere, then some areas will definitely need more caution during the exercise (refer to item 3 in the first list!)
4. The longer you've been around the more places you'll have to pay attention to. At 25 we all laughed at people with ear and nose hair, but it comes to us all in time. That data you've got hanging around from the time it was "faster" to work in spreadsheets will start showing eventually!
5. Shaving gives you the chance to take a good look at yourself in the mirror everyday, often without any of your imperfections covered up. Maintaining data quality calls for the same regular scrutiny and openness.

And here's the biggest thing most people will immediately accept about shaving and should accept about data quality too:

Chances are you're not going to ask someone from your internal hairdressing department to keep you clean-shaven. It will work far better if you take care of that yourself. Data quality's no different. It's your beard and it's your data!

Data Quality - Running in the Badlands

I like to run a little. Actually, I like to run a lot. Back in the days when my work had me travelling most weeks one of the upsides was the opportunity to regularly find new running routes in new cities. Most times this resulted in an enjoyable run and the chance for a bit of sightseeing. However on occasions things didn't go quite so well. Several times I've felt very uncomfortable and one on occasion I had genuine fears for my safety.

Three such times stand out for their similarities although they occurred in different cities around the world - Las Vegas, Milan and Philadelphia. On each occasion I was speaking at a conference and so had a little time to kill, but not enough time so that I could run too far from my hotel. All of these cities promote themselves as either tourist or conference destinations and to the casual observer seem to fit this brief well - safe, busy, functional and (in two of the three cases) prosperous. At first glance it would seem that nothing is amiss and that the "service" they offer is well provided; they are "fit for purpose", if you like. However in each city less than six blocks from my hotel I found myself running through either seriously rundown areas, areas with (in one case) chain mesh fences and burning cars or areas where I was harassed and threatened.

Now, I admit that probably 99% or more of tourists or other visitors to these cities will probably never see the areas I did nor find themselves in a threatening situation. It strikes me that there are parallels with data quality in many business systems: the places where everyone goes, those areas that support the most common transactions and queries have had most obvious data quality issues sorted out. In most cases the people using these systems largely have no problems which stem from data quality issues so over time the perception develops that data quality is not an issue. Even when the odd data quality problem does crop up often its dismissed as an outlier or the pain of it forgotten before it can be used as a driver for a data quality improvement initiative. Much like the cities I visited all seems to be working smoothly as it should.

However on my problem runs I definitely didn't get the result I was after - I didn't enjoy sightseeing and the training effect I had planned for didn't eventuate as I either had to cut my run short or ran (away) at a faster pace than I had planned. Without doubt there was at least a loss of effectiveness and probably a failure to get to the desired outcome. The same occurs in our systems when data quality issues lurk just out of plain sight. When users venture into those areas, be it with ad-hoc requests or with infrequently run analytics, the outcome they seek may well either be missed entirely or hampered by inefficiencies. These issues have knock on effects too - perhaps someone reading this blog won't run next time they attend a conference in one of those cities or may even choose another conference entirely, in much the same way users may choose not to again wade through an analytic process in a certain area of data, preferring instead to rely on gut feel or their own sets of numbers from desktop spreadsheets.

So should we as data quality or data governance practitioners act to rectify these less mainstream data quality issues. The answer is probably "it depends". For me, it's a cost benefit question, if the impact of the problem is bigger than the cost to fix it then there could be a strong business case to be built for a fix. The bigger issue though is not knowing that the problems exist in the first case, so perhaps it is beholden on us to attempt to understand where these problems lie so we, and our stakeholders, can best choose where to focus data quality improvement efforts. Interestingly this very style effort actually is in play at the hotel I stayed at in Philadelphia. Seeing me walk back into the lobby in my running gear the hotel concierge asked how my run was and then offered me a pre-prepared map showing run routes and a very clearly marked border indicating the outer limits of where it was safe to run. I only wish they had advertised the existence of this data steward before I'd gone out running!

Data Ownership and Data Responsibility Should Go Hand in Hand

Chances are that all of us who have tried to advance a Data Governance initiative will have had the data ownership discussion and socialized the idea that IT shouldn't own the data. In the earlier years of my data governance efforts getting acceptance of this concept was most often the first stumbling block. Common remarks from both the business and IT fronts reflected long established culture and practices in which data was seen as an IT focus - they managed it, wrangled it, invested and got to the root cause of data problems, and they held the responsibility of fixing data problems. This resistance, although frustrating, at least had a common theme which allowed me (and I suspect others like me) to build up a toolkit of arguments, presentations, stories and case studies to help move stakeholders toward an understanding that data is best owned by the business and not the IT function.

I've written about the need to keep working away toward implementing data governance before in my post on Thinking Like a Vogon. Having a bag of tricks built up and finessed over time is, in my opinion, key to successfully importing data governance at an organization. It allows those of us charged with championing data governance to keep working toward advancing stakeholder understanding and buy-in over weeks, months and years. Recently I've noticed a trend which has the potential to poke a hole in this bag of tricks, reducing its effectiveness and necessitating a rethink of how I engage with stakeholders. These days I encounter less push back when I propose the idea that IT doesn't own the data. The concept that the business owns the data now often seems to be easily accepted, with some business users going as far as saying "of course we own the data!"

Acceptance! It's all easy from here, right? Maybe not. While there seems to be a lot more widespread agreement that business users own the data this doesn't always translate into the best outcomes for data governance and improved data quality. Statements from stakeholders along the lines of "we own the data, but IT let us down because they can't get the data quality right and they don't understand our data" ring alarm bells for me. That one step forward may just have been followed by two quick steps back. What's missing here is responsibility. Ownership without responsibility is hollow and reflects an understanding which still has quite some way to go to reach maturity. I liken this sort of statement to a father boasting about the achievements of his children before reaching for the phone to arrange a nanny to look after them whilst they are home from boarding school. Just like we can't expect school teachers and others to take sole responsibility for shaping our childrens' lives and helping them through their problem times, business users can not claim to be data owners and then take a passive role only getting involved in data issues long enough to point the finger of blame at IT.

So how can we this new phenomenon be dealt with? Sure, a part of what needs to be done is updating the message and tailoring the tools and communication methods we use to address this new position. But, it's more than that, as data governance professionals we need to look for ways of showing those in the business with this view "what's in it for them". We should look to find success stories - show where there was demonstrable benefit gained when another business user started to take an active interest in the health and well being of their data. Even better, we should spend time with those business users who have already embraced not just data ownership but data responsibility as well, helping to make them into champions for the cause and assisting them to sell the benefits to their colleagues. The broader IT function also has a role to play here. We need to ensure that IT doesn't reinforce the existing culture, and as such there needs to be a little gentle push back when the business shows signs of pushing all data responsibility to IT. This need not mean a flat out "no", nor an abdication of any and all responsibility, but rather taking these opportunities to engage with business users, working with them as enablers rather just than in isolation as technical troubleshooters blindly trying to solve problems often without enough context or knowledge. By working together - making changes on both sides - IT will boost data context familiarity and those on the business side will get involved more often and earlier. From a management perspective, look to change IT policies if they talk about IT owning and managing the data, and think about establishing data reference groups.

No matter what approaches we take, there is one very important thing that can not be forgotten. We must help business people in their new roles as data stewards. We can not leave them to feel their way along. Build documentation and tools to help and guide them, recognise new responsibilities by modifying Position Descriptions and the way these people are measured and rewarded for performance. Not only will this signal that the organisation is serious about data governance and data quality, but it will also guide and encourage the stewards to perform better.

Why IT Can't See 2 Plus 2 Doesn't Equal 5 (And Why It's Not Their Fault)

How do a loaf of bread and a small bottle of orange juice lead to a blog post about data? Here's how. Last week I bought (or at least tried to buy) a $4 loaf of bread and a $2 orange juice. The young lady (a trainee, according to her badge) behind the counter cheerfully informed me that it would be $27.60 and seemed a little taken aback when I asked if she was sure about the price. She assured me it was right and even went so far as to turn her monitor around to show me - "See, there it is!". Risking the chance of becoming the customer from hell, I pushed my claim that the price was wrong. Another staff member (herself probably not old enough to have left school) was called to assist and tried her best to educate the problem customer - "Look, the computer says that's what it is, so that's what it costs". It took me leaving the goods on the counter and turning to walk out to have the duty manager come over and straight away see the problem and charge me the correct price.

I bet we've all had at least one similar experience and laughed about it, putting it down to "kids today" or asking "what are the schools teaching now?". But step back for a second and think - should we really be so quick to judge? Take a less critical view and ask what is really happening. The shop assistants are most likely casual employees, which means they spend only a few hours each week in that environment, they don't live and breathe the goods they sell and couldn't tell you the price of a loaf of bread without referring to a price list. While they can do their job and function largely without mistakes if enough boundaries are set and systems provided, when something goes wrong or when the pressure is on they lack the familiarity and deeper understanding which lets them sense when something (e.g. my price) is wrong. Chances are that the math skills of the two casual shop assistants were just fine, it was the lack of context of the individual prices and not yet having a confidence gained from situational familiarity which prevented them from recognizing a problem and speaking out with a different answer than that given by the system.

Is there a parallel between this example and how we work with data in our organizations? I think so! I've certainly heard (on more than one occasion) people from the business bemoaning that IT just don't get it, they don't understand the data. If the IT folks don't work with that data much, only a few hours per week, or in blocks of time many months or years apart, then that's not surprising. Even if they do work on the data more frequently they don't have the context gained from working with the data in the same way the business users do. What's apparent (at least to me) is that this gap in understanding is not the fault of the IT staff. If people don't work with a dataset closely and regularly enough to gain familiarity and understanding then even what are [to others] glaringly obvious mistakes may well go unnoticed. In the same way as the shop manager needs to find a way to better equip her staff to deal with situations like mine, we (as Business and IT Managers) need to find ways to better deal with data centric issues in our organizations. This may happen in a number of ways, for example

• embedding IT staff in the business to have them work with the data and alongside the business users 
• having business people take an active and accountable role in data migration projects and not leaving IT staff to make snap judgement calls about the data at the pointy end of a project to hit a go-live date, or 
• bringing the idea of data governance into the organisation and getting the business people involved with decisions around the data.

No matter which of these approaches you adopt I think two things should always be present. Firstly don't blame the folks at the coal face, and secondly introduce and promote the idea that data ownership belongs with the business not the IT department. Miss these opportunities and chances are that, just like me with my bread and OJ,  at some stage your company will be asked to pay a much higher price than seems reasonable.