So That's What IT Recruiters Are Good For!

The BI Market seems to be fairly hot right now, at least in Australia, if the number of calls I'm getting from headhunters and IT Recruiters is any sort of indicator. However an experience I had this week makes me wonder.

Right now I have a need to ramp up the number of BI folks in our company to handle some project demands. It's a short term gig for the time being so we directly approached a few service firms as well as posting in one or two LinkedIn groups. Two days later I received an email from an IT Recruiter offering me the chance to interview for a great BI position with a first class company there were working very closely with. The position description felt very familiar and no wonder, I'd written most of it only a few days earlier. It had been augmented a bit to give the impression that what was in reality a short term engagement with a slim chance of extension was instead just the initial piece of a long engagement. A hunch had me checking Australia's main jobs website. Sure enough, two more recruiters were also advertising jobs which were also obviously the roles I was trying to fill. Both had extra or erroneous information, I suspect to make their version of the job more attractive than the one being advertised by their competitors.

Recruitment "consultants" taking initiative and creating opportunities to make income for themselves and their companies is one thing, but it causes problems for those of us working in, or managing those working in, the BI arena. With multiple parties now advertising my two roles now seemed like eight, so when those two are filled it will seem as if many more roles have been quickly filled. I suspect this contributes to the general feeling that demand is high and might even be artificially forcing rates up. Worse still will it cause a scenario where this perception of great demand draws people from other IT areas to re-train in the BI field, potentially causing a future glut of supply?

So let's be generous and say that recruitment consultants generate a buzz for us and drive potential candidates our way, but those of us with our own Human Resource Departments could do that anyway. What else do recruiters bring to the table? In my experience the majority of those I've dealt with lack any depth of understanding - matching skills and expertise to roles is haphazard at best, and even basic screening of applicants seems limited to "if they can spell BI then they're qualified to get a second interview".

So, for me I struggle to see the value..... But then I had an experience that made me realize that recruitment consultants do serve a valuable purpose - they're here to make us laugh. I was approached by a recruiter interested in my data warehousing expertise and keen to learn if I had a forklift license and what my typing speed was. I suspect data warehousing may have been a new term for him! Another, during a preliminary chat when I was trying to find a candidate to fill a new data warehousing role, asked me if I felt rising real estate costs would make warehouse space more expensive and cause my firm to lift our rates.

So, guys, keep the laughs coming, but you'll excuse me if I pass on your other offerings!

The Next Data Rock Star

I miss the 1980s, there was lots to like about that time: great music, a lifestyle less cluttered by technology, crazy hair, the fact that I could still grow enough hair to have crazy hair. For most of my adulthood if you had asked me which time in my life I'd like to go back to I'd have picked the 1980s. But now I'm not so sure, I'm starting to pine for the late 1990s through to the mid 2000s. Why? Because, that was a time when quite a few data folks working in the business intelligence arena were the rock stars of IT.

I'm lucky enough to have worked on my fair share of interesting and high profile data warehousing and BI gigs during that period. The problems I had the privilege of working on were often difficult, few (if any) commodity solutions or services had manifested, and the solutions we provided were often anything but cheap. However in solving those problems we added real value to the business, at times doing things that their traditional IT staff or software vendors had said couldn't be done or had tried (and failed) to solve with more application centric approaches. Perhaps most importantly during that period few BI projects were undertaken unless the business could really see the benefit. Maybe this was because of the high cost, maybe not. Whatever the reason, those of us lucky enough to be leading the charge and playing key roles in these projects often got the accolades of not just our IT and data colleagues but also  of key business stakeholders as well. I was fortunate to parlay this into a successful consulting career  with word of mouth securing a series of back to back BI engagements stretching for over a decade.

But I fear that the BI Rock Star is a thing of the past. It's becoming a crowded field. Any number of people now "do BI" and what was once bordering on black magic is commodity thanks to advances in (BI) software products and hardware processing power. Even worse, the phrase "reporting" is commonly be used almost interchangeably with "business intelligence". We've all too often moved away using BI undertakings to support complex decisions and solve wicked problems to one where often all too many reports are produced for banal reasons or reasons unknown. It's no wonder then that business stakeholders seem to be seeing less value from their business intelligence dollar.

So, if BI is indeed on the wane, I wonder where the next Data Rock Star will come from. If I was to encourage my kids into the field where would I point them? Will the Data Scientists be the next wave to be lauded by the business, or will those jumping on the Big Data bandwagon get that privilege? Maybe those playing in the NoSQL space are about to be thrust to glory, or perhaps the social media tsunami will mean that's it the intersection of all three which will be the sweet spot. I actually think it won't be any of these. Both Big Data and Data Scientists to me still feel like buzz words - hot topics - that the industry has thrown up and bandied about to the point where airline magazine syndrome has kicked in. I don't see the clear path to the wide stream killer need that means business stakeholders will clamour for solutions in these areas enmasse any time soon.

So, if you can see more clearly than I, I'd love to know what the next big data thing will be. I miss being a rock star!

Data Governance and the Not So Thirsty Horse

What's that old joke? The one about the deeply religious man caught in his house as flood waters rise around him.  Early in the piece as the waters are only a few feet deep a dingy arrives to take him to safety and he refuses, saying "I'll be fine, the Lord will look after me". Later, as the water rises and forces him to the house's upper story, a bigger boat arrives and those on board implore him to leave his house and come with them. Once again he answers "I'll be fine the Lord will look after me". Some hours later as he is perched dangerously on his roof with water lapping at his feet a helicopter descends and those on board urge him to leave before it is too late. But the man is insistent - "The Lord will look after me, I don't need you to save me". Not long later the waters rise again and the man is swept away to his death. Arriving in Heaven he is confused and, on being granted an audience with God, asks Him "Lord, I put my faith in you. I trusted you. Why didn't you save me?". God looks at him somewhat incredulously and asks "what more did you want? I sent you a dingy, a boat and a helicopter!"

On more than one occasion in my career I've marvelled at how much that joke paralleled what was going on around me. You may have even experienced this yourself at some time - demands coming from all sides, requests for clarity around data meanings, master data rulings, demanding to know why you don't know the ins and outs and business rules of obscure pieces of data that is about to be touched by some project or rather, and complaining because there's not enough documentation and other collateral in place - forcing them to have to spend more effort and dollars inside their project.  If this was indeed all missing then I could understand the consternation, but when a data governance framework exists and the business users understand and acknowledge that they (and not the IT group) steward the data and are the ones with the true knowledge of that data, then I start to bristle a bit. Maybe not the first time, or even the second or third time you walk people through what's available and how to make best use of, and leverage it, but sometime thereafter it's sure to come. I've noticed similar behaviour in other industries across my career, but it does seem to be remarkably pronounced amongst those working in IT.

Often I've put this down to some form of Not Invented Here Syndrome and accepted that some individuals will just not use what is put before them, be it through lack of acceptance, understanding or whatever. I'm sure that the phrase "you can lead a horse to water, but you can't make him drink" has passed my lips on numerous occasions. But lately I've started to think that perhaps this is not the answer. The horse really needs a drink, he just doesn't know the water is good yet! I've written before  about the need for persistence and tenacity when implementing data governance programs. Perhaps this situation offers an opportunity to create greater acceptance of data governance, data stewardship and the like. Meeting those IT folks placing demands on you half way may well work - give them a little of what they are asking for, do a bit of "their" work for them.  Sure, it's re-inventing the wheel to some degree, but nudge them toward what's already in place, shepherd them into the data governance framework and show them (via a few small wins) how they can leverage the data stewards working within it. My initial findings suggest that where those people begin to see that it can make their jobs easier they start to be more accepting of it and begin to use it to augment their other activity and in some cases make it first port of call. Given enough time I'm convinced we can make data governance advocates and champions of at least  some of these people.

I'm keen to hear from you if you've experienced (and leveraged) similar issues as I'm sure I'll encounter many stables full of horses who don't know they're thirsty across the rest of my working life!

Data Governance and the Self Licking Ice Cream

A friend and colleague regularly uses the self licking ice cream phrase to illustrate to the project management team he mentors that projects need to provide value to the enterprise and not just operate in a vacuum. No matter how well those in the project build a system, or how well that project is managed, it's unlikely to be perceived as a useful and successful exercise unless the various stakeholder groups agree that the work was needed and worthwhile. From their point of view all that the project will have achieved is keeping those on the project busy and employed for its duration.

But that can't be said for data governance, can it? Every organization has at least some data quality problems, conflicts around data ownership, data silos preventing leveraging the data asset across the business, so therefore every organization could benefit from a data governance program.

That may be true, but I'd still like to bet that many, if not most, data governance programs are seen by many in the organization as self licking ice creams. Ask an exec or senior manager from one of those firms and chances are they'll not see the value of the data governance initiative. If you're heading a data governance program and your execs don't recognise the need for it then, even if you think what you're doing is worthwhile, I've got bad news for you - to others you're licking your own ice cream! It won't matter if you've been making great advances in improving data quality, finally getting the data stewards to work together to understand master data issues, or building understanding across your data landscape and producing mountains of documentation along the way. From the outside you'll just be seen as doing little more than using scarce funds and making a call on their peoples' time for no other reason than to keep yourself in a job.

Of course securing an executive sponsor for the program and working to win wider executive awareness and buy-in will help avoid this scenario. However, this will be difficult to achieve and retain unless you continue to find ways to make your data governance work relevant and demonstrate value that can be seen by key stakeholders in the organization. For me, to do this it's critical to have demand placed on a data governance program. If people in the enterprise don't recognise they have data problems and aren't coming to you to help solve them, then you need to find something tangible for your program to tackle. If you're lucky enough to be facing upcoming system changes with associated data migration then that may be a great driver. If not, then try and find the areas where non trivial amounts of money can be saved due to data problems or look for the issues that keep your execs awake at night because they lack confidence in the data. If you believe that those situations can be helped through data governance then start to work on them, if not keep looking - trying to force them to be data governance issues won't do you much good.

If there's nothing in your organization that you can apply data governance to (and be seen to create value) right now then don't be afraid to let things lie for a while. Pushing ahead on things others don't perceive as valuable may actually torpedo the longer term success of a data governance program. Better to lie low and bide your time rather than have others shut the program (and perhaps your career prospects) down. But, keep your ear to the ground because there will come a time when there will be a big problem facing the organization which data governance could help solve. Be ready to seize the opportunity. If you don't and one of your execs later reads about data governance then being a self licking ice cream will be the least of your worries - you may fall victim to airline magazine syndrome!

Big Data - Is it Really New?

Despite the fact that Big Data is fastest becoming one of the buzz words in the data industry I still wonder if we yet know exactly what it is. Is it anything more than just a concept, an umbrella topic if you like, to attempt to signal that something new is beginning to emerge in our industry? But, is big data really something new? Is the buzz justified, do we really need to reinvent our approaches, our practices and the skills we engender in those coming up through data careers?

I’d argue that some, if not many, data professionals already know a thing or two about big data. Well over a decade ago I was the architect leading a project to build a data warehouse to handle call traffic and billing data for one of Australia’s biggest telecommunications companies. Back then the volume of data we were working with was hardly small and I’d suggest that those that the people who work with that warehouse, and the BI solutions in enables, today are facing even larger data volumes thanks to the proliferation of mobile phones and other devices. We found ways of handling the challenges that the volume of data posed and more than a few of these approaches would still work today, even without the additional leg up we now get from increased computing power.

But what of the other characteristics of big data? It’s not just volume that’s the challenge with big data, but the velocity too. I’ve heard some commentators discussing that it is the rate at which data arrives which is in fact the biggest issue that comes with big data. But we’ve been dealing with high velocity data for a while now as well. Complex event processing has been available in major database products for at least a few years and people working with any form of operational technology will be used to data flowing in from sensors and other protection or control devices at millisecond intervals.

So I believe that we can leverage much of what we already know and practice as data professionals to start to address the volume and velocity aspects of [structured] big data. It’s the complexity and variety aspects of big data that I think will give us the real problems we need to deal with. The problem of making, or taking, the correct meaning from unstructured data, especially that coming from outside our organisations, such as sentiment analysis from social media, and then somehow find a way to effectively integrate it with our structured data sets is where I believe we’ll find our headaches. It’s here that we’ll need innovative new tools and techniques, but even then I think they’ll rely on key areas and practices which at least some of us have worked with for a while. Metadata will play a big role here to establish the right context and data mining may well make an appearance as well.

So, in my opinion at least, we already know a thing or two about big data. Let’s not re-invent the wheel, but rather try to build on what we’ve learned in the past. It’s a novel idea I know, and perhaps not one that IT professionals have much of a proven track record with, but hopefully this type of approach might shorten the time to value for those making early investments in and around big data.

Technology Leadership - A Misnomer?

If you've had a career of any reasonable length in the IT industry and have a track record of success behind you then chances are you're now in, or at some future point will be offered, a role which is considered a Technology Leadership role. In the data and information space these types of roles include Architects, Business Intelligence Managers, Competency Centre Leaders, Data Quality Managers and the like.

I've worked in many of these roles at various times and the one thing that they all have in common is that often this so called technology leadership is actually dealing with issues that have little to directly do with the technology at all. Rather the Technology Leader spends his days strategy setting, looking for ways to deliver business value through future change or the current actions of his people, engaged in stakeholder management, selling concepts to senior management, and so on. Most of these issues are actually more closely aligned with managing functions, people, shaping programs of work, etc and require some ability to control and direct and have access to, and authority over, budget and resourcing decisions. Without this the leadership may become divorced from the ability to deliver which can have serious ramifications on credibility, influence and ability to show effect and value from the leadership role. Perhaps it could even be argued that people in such roles are shouldering, and in some cases having to own, some of the concerns traditionally associated with line management without having enough say in solving those problems - i.e. their plans could be quashed, derailed or rerouted by others who do have control over budget, resourcing or strategy.

In order to allow our technology leaders to add the most value we need to realise and acknowledge their leadership is in fact only in part technology based. Our technology leaders need a seat at the management table to be truly effective. Taking this step delivers another benefit. Technology Leaders are in the decision loop early, ensuring better alignment with other initiatives and allowing others in the management structure better visibility of, consideration of, and therefore use of the area from which the "technology" leader comes.

The alternative to this arrangement is the introduction of Thought Leadership roles. These may well be ideal positions for the person formerly known as Technology Leader if the organization can stretch to that. However most can't due to lack of size, true need, commoditisation of many facets of the IT function, or shrinking budgets. The unkind (or is that better phrased as jealous?) spin on these roles might well be lots of thinking, musing and sprouting of opinion without the need for implementation efforts hampered by real world constraints and politics. Hey Gartner - if you're ever looking, I'm over here :)

When I Grow Up I Want to Be a Data Scientist

Some time in the early 1970s I formed my first career aspiration. When I grew up I wanted to be a bus. Yes, you read that right - not a bus driver, but a bus. It seems ridiculous now, some 40-odd years later, but I certainly wanted it bad back then! The reason it didn't seem crazy all those years ago is that I really had no idea what being a bus involved and what's more a bus seemed like a pretty cool thing to be - lots of the books that I had read to me featured buses zooming all over the countryside having all manner of exciting adventures, there was at least one cartoon where one of the main characters was a bus, and buses even had songs sung about them. Why wouldn't a three year old boy want to be a bus?

Fast forward 40 years and focus on the data industry: big data is being talked about in server rooms, board rooms and everywhere in between, and the term Data Scientist is becoming more and more common. It's a position I'd never heard of twelve months ago and now it seems hardly a day goes by where I don't see it mentioned in a blog post, industry email, or white paper, etc. A quick look at a number of job posting sites across the web confirms that organisations are indeed looking to hire people into roles with the Data Scientist title. But does anyone really know what the job entails, is there an agreed job description, even in a general sense? Judging by the diverse descriptions of the job ads I looked at it certainly doesn't seem so. I've talked before about my concerns that the Big Data concept is very much an overloaded and poorly understood term and I worry that the new "must have" role of Data Scientist is facing the same problems.

But, so what? Does this situation even matter? I think so. I applaud anything that raises the profile of those of us that work in the data and information space. I still remember what it was like to ride the wave in the early days of business intelligence and data warehousing and if others can have similar experiences and success today due to the next big thing being data centric then that's a good thing. However, I worry if the  shining beacon of the Data Scientist role starts to lure people from other facets of IT, or potentially worse still, attracts students entering University whilst the speciality is still in its infancy. With the role a Data Scientist will play, and how they will deliver value, yet to become understood in a consistent way then how can we expect our higher education institutions to prepare courses of study that will equip students with the skills that they will need when they reach the work force? I wonder if we will see such courses morphing, emerging and disappearing over coming years as industry and the universities try to better find what is expected of Data Scientists and how to build the knowledge and skills base they need to be successful. What of these early students, the guinea pigs or crash test dummies of the coming wave of graduate data scientists? Are we leading them astray, promising them areas of work, challenges and career paths which no-one yet knows will actually be there in the same form (or at all) five or ten years from now? Let's not promise too much too early. Better people come to the profession with their eyes open and with realistic views of what to expect, than simply because of hype.

Enough doom and gloom. If anyone can really tell me what a Data Scientist does, I'd love to know. I actually might want to be one when I grow up.