The Desire Paths in Your Data

On the days that I drive to work I have around a one mile walk from my car park to my office. There's one particular spot where I can take a shortcut by going off the paved path and cutting across some open ground. It saves me a minute or so on the way to work. Despite the fact that there is a perfectly good sidewalk built by the local council it's obvious that a lot of people must do the same as me, and opt for the faster route as well, because there's now a clearly worn path across the open ground, established by people choosing this path every day. Urban planners call these desire paths - the paths that people choose to take between two points regardless of other options that may be available to them.

If you look hard enough you'll find desire paths through your data as well. Your business intelligence solution offers many conventional paths (like the sidewalks built by my local council). These are the pre-built reports and other instruments that you present to your users. But I bet many of your users have built desire paths - the tips and tricks they use to get and analyse the information they need to do their everyday jobs. Or perhaps the extra things they have to do, those downloads and manipulations in Excel or the sneaky Access database sitting on a spare PC under someone's desk,  because your BI solution doesn't let them do their jobs or perform the analysis they need in its entirety. Now you could argue that desire paths in a BI solution actually represent a form of self-service BI. However, unless you've engineered things this way then you're probably only kidding yourself. Yes, desire paths are examples of your users finding ways to help themselves, but chances are that it's due to shortcomings, not features, within your BI solution.

Desire paths have a dark side in the natural world - they often result in trampled vegetation, can contribute to erosion and (not being as safe as constructed sidewalks) can also lead to injury for those using them. Similarly, in the IT world they are actually likely to cause harm to the reputation of your BI solution or of your BI team. Just like people in the natural world wandering a direct route desire path, shaking their head(s) and wondering why their local planning authority spent all that money building a meandering sidewalk twice as long as it needed to be, your users may well wonder where all of that IT spend on BI went, especially when they "can do the same thing (but better) in Excel". Perhaps worse still desire paths can cause actual harm to your organisation, be it financial or other type of loss, due to decisions based upon flawed analysis or data as desire paths can lack the checks and balances that come from underlying planning.

Our challenge as practitioners, managers and custodians of our organisations' BI assets is to find these desire paths (and in so doing also to find which of our pre-built paths are no longer as relevant as they could be). Can we build established footpaths where the users' desire paths have been worn in over months and years of usage? It works in the natural world and it can work in the technology world as well. There are stories of park planners in some countries visiting areas after fresh snowfall to get a sense of where those using the area are choosing to walk. I've heard of at least one university which built no structured sidewalks on a new campus until such time as the students established desire paths between buildings, simply choosing to pave these paths one year on.

In today's reality of shrinking budgets we all get busy just keeping our BI environments running and meeting the needs for new reports and analytics, let alone spending time worrying about how we'll deal with the next big (data) thing looming on the horizon. But I'd advocate there may be good value to be had in taking the opportunity to look for and act on desire paths when we can. The trick will be finding, or engineering, those occasions, like those times of fresh snowfall, when the circumstances are right to see the desire paths and incorporate key ones amongst them into your BI solution.

Happy (re)building!

Pragmatic Data Governance - It's a Matter of Trust.

This post had its genesis in a comment I wrote in response to a great piece about data governance written by Jim Harris (Twitter: @ocdqblog) over at Obsessive Compulsive Data Quality (http://www.ocdqblog.com/home/aristotle-data-governance-and-lead-rulers.html). I hope Jim won’t mind if I expand on those comments here. This post has grown organically and I'm not totally sure it captures all of the key issues bouncing around in my head, but getting it our there might help my thoughts develop and gel. So look out for a follow up post in the not too distant future.

Among other things Jim talks about the need for data governance to be applied with a degree of flexibility rather than following a set of rigid rules. This is a great perspective which exposes one of the key reasons why data governance initiatives often fail. In my opinion too often it seems that those involved (or those sponsoring) expect that hard and fast rules are what's needed.

I wonder how much of this stems from organisational cultures where staff time is closely budgeted and monitored. A common pushback I've struck in the past is managers of potential data stewards (almost) insisting on an exact breakdown of what data governance activities their people will be working on and how long each activity will take. Add to that the desire to help out new data stewards by providing them with tools to use as they learn what their new role involves and a situation where the rules and their strict application come to the fore can quickly emerge! The problem is compounded if the stewards feel that they may be blamed in some way if others are not happy with their choices or the downstream effects of them.

In my opinion, when setting up data governance programs we need to make sure that data stewards have top coverage – that is senior management endorsement and support when it matters- to give them time to act and the freedom to make considered and pragmatic choices that might involve bending or extending data governance “rules”. We must trust and allow the stewards to interpret rules according to the circumstances at hand whilst still being mindful of past precedent and any negative impacts their choices might have.

Without this support I’d question the value that a data governance program is going to deliver. Sure, maybe some cosmetic short-term data quality wins might occur, but chances are they’re the low hanging fruit and would probably have happened eventually anyway.  In the longer term all that is likely to happen is that the data stewards will come to be seen as blockers to progress - the holders and enforcers of a set of petty rules, putting up hoops for others to jump through. The hard issues (the ones likely to be standing in the way of unlocking real value) may never get tackled, the program will slowly wither and die as complaints mount against it and future data governance efforts will have a much harder time gaining any traction.

When a steward feels the need to build in a degree of butt covering into what he or she does there is a problem. If he or she can claim to have exactly followed the rules as a means of avoiding any blame storming about a bad situation that may have manifested from a data governance decision chances are the best outcome for the business isn’t going to manifest. We need to give data stewards the room to think, reason, weigh up the contributing points of view and then make the appropriate choice in the circumstances. A little trust in human nature and faith that people will want to do the right thing is required. If the organizational culture doesn’t support this then a successful data governance initiative might be possible, but it may mean that the data stewards need to sit higher in the food chain. But, ideally you can seek out and find stewards who already have the trust and respect of senior management and then will have the flexibility to act pragmatically when needed.