The Catch 22 of Traditional Business Intelligence

So much has already been said about how much of a pain business intelligence is. The complexity, the constant IT bottlenecks, the crazy cost of hardware, software, consultants and whatnot. Gil Dibner of Gemini Venture Funds (formerly of Genesis Partners) described it very eloquently and in great detail in his blog post about the SiSense investment round.

Since business intelligence imposes so many limitations and challenges, every existing BI vendor picks its favorite ones and positions itself as the best at addressing those specific ones. Some focus on providing easy-to-use front-end tools for the business user, some on handling complex ETL scenarios and large data sets, others on open source software to remove software licensing costs and so on.

Business intelligence vendors have been constantly rolling out new functionality and technology through the years. But still, it seems like business intelligence has been standing still. No progress has been made in expanding it to the wider market that can't afford long and costly development/customization cycles. In fact, most of the BI vendors which do not sell enterprise-class solutions (e.g., SAP, IBM and Microsoft) haven't been able to grow much and remain focused on niche markets.

Well, my friends, it's time somebody told it to you straight.

Business intelligence can deliver on its promise, but the entire idea needs a complete overhaul. As long as vendors keep improving specific functions within the traditional BI paradigm, no progress will be made. The traditional business intelligence paradigm needs to be scrapped and replaced by something that is much easier, much faster, much cheaper and much more manageable.

What is the core of the traditional BI problems? The traditional paradigm contains an inherent flaw that prevents it from taking BI to the next level where ROI is indisputable and business users get another powerful tool added their arsenal - in companies of all (or most) sizes.

The Inherent Flaw in the Traditional BI Paradigm

If you search "why business intelligence projects fail" in Google you will find an abundance of white papers and articles (mostly written by BI vendors themselves) giving their two cents' worth. When BI vendors pick their top reasons, they usually pick issues dealt with by their offerings and not by the competitions'. Marketing 101. Fair enough.

But one top reason on which they all seem to agree for a BI project's failure is the lack of up-front planning. That is to say, in order for a business intelligence project to succeed, you must compile your requirements ahead of time, coordinate with all the relevant parties (IT, business departments and executives) and plan the project in accordance to those requirements. Otherwise, you are destined to fail.

In other words, they blame you - the consumer - for a failed BI project. Had you planned ahead, the project would have been a success and you wouldn't have flushed hundreds of thousands of dollars worth of software licenses, hardware and personnel time down the toilet.

Sadly, they have a point. Since traditional BI solutions aren't very sympathetic to unplanned changes from an architectural point of view, anything you don't think of in advance is difficult and expensive to introduce later. So you better think long and hard about what you need, because otherwise any requirements you miss could mean the difference between a successful project and a complete mess.

But herein lies the catch.

It doesn't matter who you are or how much experience you have, it is utterly impossible to know in advance what your requirements are when it comes to BI. BI is highly dynamic and requirements change all the time because the business changes all the time and because business users come up with new ideas all the time. A report you need now is not the report you need later, an analysis you do now may only be relevant for a short period of time and meaningless shortly thereafter.

Most importantly - if you are a department or company seeking a BI solution but without any BI development experience, you have no way of knowing how a particular requirement will affect the architecture of your solution. Thus, you could easily find yourself disregarding the immediate testing of some particular capability because it seems trivial to you, just to discover later that the entire solution comes tumbling down when you actually try to use it, and that without it the system is useless.

You cannot imagine how often this happens, especially when a solution calls for OLAP cubes built over a data warehouse (blah).

It's the traditional BI vendors who made up the rules for this game over 10 years ago. They are the ones who've been aggressively promoting a paradigm where everything needs to be thought of in advance - otherwise you are sure to fail. It makes sense because these vendors focus on enterprise-wide BI for fortune 500 companies where the complexity of a BI project is masked by the complexity of the corporation's own business processes. These organizations are used to things taking years to reach perfection because every other process they have pretty much takes the same amount of time.

But trying to implement the same concepts on slightly smaller corporations is the exact reason why most BI projects fail.

Don't get me wrong. It's always good to plan ahead. But know this - business intelligence requirements are impossible to predict and nearly impossible to measure until the end users use it on real data - in real-life scenarios - over time.

You cannot do this with traditional BI without investing a TON beforehand, and even then you have no guarantees. When you go for BI as advocated by the traditional platform players, you are basically throwing hundred dollar bills down a wishing well and hoping for the best.

Learn from the thousands and thousands of companies who have already learned this harsh lesson with blood and tears. Don't do it. There are ways to change the rules of the game while still getting the same class of business intelligence, without compromising on functionality, speed or capacity. But you cannot expect to find it by turning to the traditional BI players who have an over-sized BI developer eco-system for which they need to provide work. This can only be done by younger, innovative BI companies armed with new technologies, fresh ideas and sensible pricing models.

---

By: Elad Israeli | The ElastiCube Chronicles - Business Intelligence Blog