Self-Service BI: A Case of Trust Working Both Ways?

In the breakneck world of data, which I have been privy to since the mid 1990s, business intelligence remains one of the most enduring terms. So I was surprised to learn from my colleague Myles Suer’s blog piece about Self-Service vs. Traditional BI that it was first referenced in 1865. The writer Richard Millar Devens used “business intelligence” to describe how a banker had the foresight to gather and act on information thus getting the jump on his competition. To this day, business intelligence describes the ability to gather information and act on it quickly.

For me and perhaps my romantic British outlook, this anecdote evoked images of data pipeline ingestions via clandestine meetings in the shadowy corners of taverns, with silver crossing the palms of those delivering data secrets — not too dissimilar to a data engineer’s modern day role (I jest, of course: Data engineers would deal only in bitcoin or similar these days.)

A simple fact remains: Information is power, and the ability to act on it quickly separates the leaders from the laggards. And whilst the means to deliver BI have evolved greatly, the motivation behind it has not.

Access remains a core challenge: Who benefits? Who loses out? BI has long been plagued by an audience divided, the data haves and have nots. Democratising access to all users is essential to the next chapter of BI: self-service at scale.

The Evolution of Business Intelligence

Early Days

Modern BI began when “data” became what we consider it to be today. New reports were initially based on a landscape of mainframes and required a formal request to the few who could access the data and steer the titanic machines to produce the results.

The 1980s gave us power dressing (big hair! bigger shoulder pads!) plus personal computers and distributed databases. This technological shift placed computing power into the hands of the individual consumer — yet access to corporate data still resided with the “techies”.

The Rise of the Data Warehouse

The birth of the enterprise data warehouse was heralded as the solution to limited access. Proponents promised it would address shortfalls, including the expense and time required to create reports from transactional data sources and the impact that running those reports had upon the source application. The focus of design shifted to the business questions being asked and not the processes being supported. New BI toolsets, such as BusinessObjects and Cognos, started to emerge; these allowed ad hoc queries to be composed without the need to write SQL. (I was Team BObjects — I am sure Cognos would have cheekily labelled us Team BO.)

Yet even with the new toolsets in play, fear persisted. The data-haves feared granting access to the have-not masses, and BI use still tended to lie with super users whom the IT department trusted. IT, after all, was usually in control of access, and many IT teams restricted access to the few, those who would “use it and not lose it” as data became the focus of their role.

Modern Business Intelligence

More recent years have witnessed the trend of digital transformation. The term dashboard has become more commonly associated with analytics than a review on Top Gear, and lakes are a place where fearless data scientists can free-dive into data, as murky as it may appear to the mere mortals amongst us. Information democracy has become an aspiration for any organisation with the related (and burning) desire to be truly data-driven.

In each of these ages of BI technology, self-service has been something out of reach for much of an organisation. There has still been a reliance upon technical teams or super users to produce results — but this causes bottlenecks and delivery delays to those in need. The super users may have access to self-service BI but to be truly data driven, do we not need everyone to have ready access to the data that will guide them?

Blockers to Wider Adoption of Self-Service BI

So how do you transform data “have nots” into data “haves”? The road to adoption is not always clear. The word adoption can assume a population has the ability to access a provision in the first place. But this is rarely the case.

And fears around empowering the data have-nots persist. As mentioned earlier, there can be reluctance to provide access to BI toolsets, as IT may reason that toolset use would be sporadic and therefore the skills would wane over time (resulting in requests diverting back to the centre). This can indeed take place! However, if we provide a more intuitive and informative environment in the first place, might this not reduce the complexities faced — and therefore the likelihood of it falling to disuse?

Human prejudice can hinder progress. A concerning trait I have witnessed on more than one occasion is the belief that the business users are simply not data-savvy enough to be trusted to create their own reports. Best leave such matters to the experts, the people who “know and understand” the data, the detractors reason.

In my own experience it is the people in the business, who live and breathe what is behind it, that 9 times out of 10 will spot a data issue before an isolated techie will. So let’s give the business users the opportunity to become more data savvy. Let’s give them the tools and guidance they need to become more data literate.

And guidance goes both ways. During my work as a consultant I was often found either teaching the design of BI environments (and the underlying marts/warehouses) or auditing the customers’ end result for best practice.

All too often I confronted approaches and systems that assumed the consumer understood the intricacies of the data. Such systems assumed users would know not to perform a certain query, or that the underlying data behaved a different way. In anything self-service we cannot assume the consumer understands everything. Designers must place themselves firmly in the user’s shoes and design an environment with the appropriate guide rails.

From the Shadows Emerges True “Self-Service” BI

What should self-service BI look like? Like many things, beauty is in the eye of the beholder. Depending on who you are and what you want to do with data, the details likely vary, but how about this for a rule of thumb:

Self-service BI is an approach to data and analytics that enables users of varying roles and skill levels to independently access the correct data to support their needs.

Self-service BI is like a chameleon that adapts itself to the environment and the personas it serves. Such personas include:

• Data scientists looking for their source data to train their models

• Operations analysts tracking down a report output for their quarterly review

• An account executive walking a customer through their annual review

• A marketing analyst seeking a list of prospective customers who fit certain criteria

The interface adapts to the unique needs of the persona. And with interfaces potentially ranging from SQL editors to natural language processing, self-service for all may require a chameleon or suite of solutions to suit the individual. Regardless, all variations require the foundational data to be:

Discoverable

Data sources are readily found and if they exist in multiple domains then the choice is made clear.

Understood

Data is clearly labelled with full descriptions and little room for ambiguity or misinterpretation.

Trustworthy

Provenance of the data is known and any shortfalls in quality are surfaced so the consumer may judge its suitability to the purpose at hand.

Accessible

Users know how to access the data without fear of query errors. Data can be readily extracted and presented in a manner suited to the skills of the consumer and purpose of that data. This is where the latest toolsets, combined with well judged implementation, designs come into their own.

So based on the above, what self-service business intelligence relies on is self-service data intelligence: our knowledge of the data.

Data intelligence provides the basis for the reciprocal trust we need consumers to have in using the data environments that we in turn are trusting them to use. We must ensure that users trust in the data, and they know what they will be getting when they use it. And surely, by giving them the benefit of this, are we not able to trust them in their use of that data? Trust goes both ways.

But there is an elephant in the room — a third group of people whom I fear still walk among us. They are not the “haves” or the”‘have nots” but the “want nots.” To this day I recall the senior manager who told my team he “did not need data to run his business.”

The fact is, as with any system, the simple act of building it does not mean “they will come.” You can build the most innovative self-service data environment catering to all manner of skill level and understanding with training fully explaining the “how to.”’ But if your potential user population does not believe that data can help them, or do not feel motivated to use data in their day to day work, they will not use it.

Do not assume that the benefits of being data driven are self-evident! You need to show them the “why to.” But that’s a whole other blog starting right there….

Final Thoughts

As I have discussed, self-service can manifest in numerous different ways — and it must, if it’s to service a broad class of user., But at the root of it all is the need to make data discoverable, understood, trustworthy, and accessible to those who need it when they need it. There lies the promised land of increased productivity and the ability to outpace both demand for data and the competition, perhaps leaving time for slightly less clandestine meetings in taverns — this time to toast success.