Thursday, August 5, 2010

The Catch 22 of Traditional Business Intelligence

So much has already been said about how much of a pain business intelligence is. The complexity, the constant IT bottlenecks, the crazy cost of hardware, software, consultants and whatnot. Gil Dibner of Gemini Venture Funds (formerly of Genesis Partners) described it very eloquently and in great detail in his blog post about the SiSense investment round.

Since business intelligence imposes so many limitations and challenges, every existing BI vendor picks its favorite ones and positions itself as the best at addressing those specific ones. Some focus on providing easy-to-use front-end tools for the business user, some on handling complex ETL scenarios and large data sets, others on open source software to remove software licensing costs and so on.

Business intelligence vendors have been constantly rolling out new functionality and technology through the years. But still, it seems like business intelligence has been standing still. No progress has been made in expanding it to the wider market that can't afford long and costly development/customization cycles. In fact, most of the BI vendors which do not sell enterprise-class solutions (e.g., SAP, IBM and Microsoft) haven't been able to grow much and remain focused on niche markets.

Well, my friends, it's time somebody told it to you straight.

Business intelligence can deliver on its promise, but the entire idea needs a complete overhaul. As long as vendors keep improving specific functions within the traditional BI paradigm, no progress will be made. The traditional business intelligence paradigm needs to be scrapped and replaced by something that is much easier, much faster, much cheaper and much more manageable.

What is the core of the traditional BI problems? The traditional paradigm contains an inherent flaw that prevents it from taking BI to the next level where ROI is indisputable and business users get another powerful tool added their arsenal - in companies of all (or most) sizes.

The Inherent Flaw in the Traditional BI Paradigm

If you search "why business intelligence projects fail" in Google you will find an abundance of white papers and articles (mostly written by BI vendors themselves) giving their two cents' worth. When BI vendors pick their top reasons, they usually pick issues dealt with by their offerings and not by the competitions'. Marketing 101. Fair enough.

But one top reason on which they all seem to agree for a BI project's failure is the lack of up-front planning. That is to say, in order for a business intelligence project to succeed, you must compile your requirements ahead of time, coordinate with all the relevant parties (IT, business departments and executives) and plan the project in accordance to those requirements. Otherwise, you are destined to fail.

In other words, they blame you - the consumer - for a failed BI project. Had you planned ahead, the project would have been a success and you wouldn't have flushed hundreds of thousands of dollars worth of software licenses, hardware and personnel time down the toilet.

Sadly, they have a point. Since traditional BI solutions aren't very sympathetic to unplanned changes from an architectural point of view, anything you don't think of in advance is difficult and expensive to introduce later. So you better think long and hard about what you need, because otherwise any requirements you miss could mean the difference between a successful project and a complete mess.

But herein lies the catch.

It doesn't matter who you are or how much experience you have, it is utterly impossible to know in advance what your requirements are when it comes to BI. BI is highly dynamic and requirements change all the time because the business changes all the time and because business users come up with new ideas all the time. A report you need now is not the report you need later, an analysis you do now may only be relevant for a short period of time and meaningless shortly thereafter.

Most importantly - if you are a department or company seeking a BI solution but without any BI development experience, you have no way of knowing how a particular requirement will affect the architecture of your solution. Thus, you could easily find yourself disregarding the immediate testing of some particular capability because it seems trivial to you, just to discover later that the entire solution comes tumbling down when you actually try to use it, and that without it the system is useless.

You cannot imagine how often this happens, especially when a solution calls for OLAP cubes built over a data warehouse (blah).

It's the traditional BI vendors who made up the rules for this game over 10 years ago. They are the ones who've been aggressively promoting a paradigm where everything needs to be thought of in advance - otherwise you are sure to fail. It makes sense because these vendors focus on enterprise-wide BI for fortune 500 companies where the complexity of a BI project is masked by the complexity of the corporation's own business processes. These organizations are used to things taking years to reach perfection because every other process they have pretty much takes the same amount of time.

But trying to implement the same concepts on slightly smaller corporations is the exact reason why most BI projects fail.

Don't get me wrong. It's always good to plan ahead. But know this - business intelligence requirements are impossible to predict and nearly impossible to measure until the end users use it on real data - in real-life scenarios - over time.

You cannot do this with traditional BI without investing a TON beforehand, and even then you have no guarantees. When you go for BI as advocated by the traditional platform players, you are basically throwing hundred dollar bills down a wishing well and hoping for the best.

Learn from the thousands and thousands of companies who have already learned this harsh lesson with blood and tears. Don't do it. There are ways to change the rules of the game while still getting the same class of business intelligence, without compromising on functionality, speed or capacity. But you cannot expect to find it by turning to the traditional BI players who have an over-sized BI developer eco-system for which they need to provide work. This can only be done by younger, innovative BI companies armed with new technologies, fresh ideas and sensible pricing models.

By: Elad Israeli | The ElastiCube Chronicles - Business Intelligence Blog


  1. Elad,
    You have correctly pin-pointed the reasons for ‘too often’ failed BI projects but there is another reason:
    The traditional BI vendors and their deployment partners behave like bean-counters i.e. caring for their own project revenues rather than their customers’ expenditure; the more hours a BI project consumes, the better for them.
    Therefore they don’t really care how ‘big’ the BI project is. On the contrary, they encourage company-wide, deep and thorough analysis (of course led by them) followed by a heavy deployment that is in most cases, as you mentioned, worthless by the time it is concluded.
    Your conclusion that successful BI project can only be done by “younger, innovative BI companies armed with new technologies, fresh ideas and sensible pricing models”, though correct is not enough.
    A successful BI project can be achieved by turning the concept upside down: instead of starting big, start small and fast and grow as you go.
    In other words, companies should encourage users to start small, easy and fast to deploy, IT independent BI projects that can grow with the necessities of the particular users.
    This approach will yield faster and better results, as well as happier users at a significant lower cost.

  2. Thanks for your comment and I agree with every word.

  3. Sounds lie you agree with using Powerpivot to me.

  4. Elad,

    Hate to disappoint you but the real life is that requirements for practically any project (probably except for choosing and installing an electric kettle) are impossible to predict so your proposition boils down to nonapplicability of project management.


  5. Thanks, Alexander.

    I don't believe it is necessarily true that requirements are always hard to predict, but it is indeed the case in BI. That has always been true, yet the traditional BI vendors have design their products to ignore this fact completely. That is why introducing changes into an existing solution is nearly impossible and requires re-doing a lot of work.

    The point I am trying to make here is that the fact BI requirements change so often is something that needs to be addressed by design at the BI solution's core, otherwise the project will very likely fail in the sense that no business user would use it, simply because it's not what they need anymore.

    So I am not against project management, I am for defining a series of smaller scale projects with shorter timelines that can be enhanced and customized on the go. That is something that cannot be done with traditional tools, but can be done with others.

  6. Hi Elad,
    can you give some examples of the BI tools you consider as suitable then? I'm really curious to know which ones these "others" are.


  7. Rose,

    Out of the known commercial software, I am only familiar with SiSense who is addressing this issue specifically.

    Alternatively, you could develop an in-house system.


Total Blog Directory Technology Blog Directory Business Intelligence Directory