tag:blogger.com,1999:blog-8489991802309187017.post367416269089517821..comments2023-07-12T23:51:00.904-07:00Comments on The ElastiCube Chronicles: In-memory BI is not the future. It’s the past.Elad Israelihttp://www.blogger.com/profile/07558330790219988349noreply@blogger.comBlogger23125tag:blogger.com,1999:blog-8489991802309187017.post-18265226139399505852013-10-30T08:47:49.656-07:002013-10-30T08:47:49.656-07:00Hi Mike,
Thanks for your comments.
Not only have...Hi Mike,<br /><br />Thanks for your comments.<br /><br />Not only have I not changed my mind, I am now even more confident that In-Memory technology is just an interim technology. <br /><br />QlikView has been struggling continuously with the limitations their own technology impose - in particular in the ability to penetrate the enterprise. This is a known fact. Their growth has significantly slowed down.<br /><br />Tableau's success has nothing to do with In-Memory, and in fact their technology is not really in-memory technology, at least not in the sense of QlikView or SAP Hana.<br /><br />SAP HANA claims to crunch 0.5TB for about $500K on hardware and software. The same benchmarks can be achieved for 10% that cost with alternative technology (like In-Chip). The other MISO giants are just jumping on the hype bandwagon (too late, as usual).<br /><br />If you've researched the internet, you'll notice that we're pretty much betting the company on the fact that In-Memory has a very bleak future. RAM storage will not be able to catch up with the speed at which data grows and there's really no reason to use an In-Memory approach when you can achieve exactly the same, without imposing a RAM limit.<br /><br />Thanks again,<br />EladElad Israelihttps://www.blogger.com/profile/14039349385755633029noreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-34221146801675510602013-10-29T20:06:14.297-07:002013-10-29T20:06:14.297-07:00it's been over 3 years since your observations...it's been over 3 years since your observations with regard to the longevity and usefulness of in-memory technologies. Curious if your opinion has changed given the continued success of Qlik, Tableau, Tibco Spotfire, Gartner's creation of a new BI genre & Magic quadrant for data discovery and the continued investment the likes of Microsoft, SAP, SAS and Microstrategy are making to close the "business data discovery" chasm btw their traditional BI offerings and that of the "new genre of data discovery" leader product capabilities.<br /><br />lowering RAM costs, more effective data compression routines, more effective chipsets and architectures opening the parallel pipes btw CPU and RAM with increasing cache sizes and transfer speeds along with the leverage of Non-volatile RAM stores in hybrid configs, all are compelling. <br /><br />Thoughts?Mike Lampahttp://www.linkedin.com/in/mikelampanoreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-68007512890500855432012-11-15T04:52:34.121-08:002012-11-15T04:52:34.121-08:00Qlik use the term 'associative' to describ...Qlik use the term 'associative' to describe how they do joins and how their UI works. The database they use is not 'associative' in the 'associative database' meaning of the term. In fact, the database they use is a simple table store.<br /><br />HTH,<br />EladElad Israelihttps://www.blogger.com/profile/14039349385755633029noreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-59879348195620713852012-11-14T15:32:28.696-08:002012-11-14T15:32:28.696-08:00hi
nice article
but i have question if you don’t...hi <br />nice article <br />but i have question if you don’t mind , you said " there is nothing “associative” about QlikView’s in-memory technology " <br />and I have read in QlikView concept and introduction whitepaper that :<br /> " QlikView then extracts the data and builds a powerful, associative database that enhances the way users work with the information"<br />so does QlikView transfer the data to an associative database in the RAM and handle the data model as associative database design or what ?<br />Regards <br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-50458848271244167572011-12-17T14:45:51.446-08:002011-12-17T14:45:51.446-08:00In the late nineties and early 2000s, I did a lot ...In the late nineties and early 2000s, I did a lot of work tuning Oracle databases. One of my principles was similar to the concepts here: to read the disks once, and do the rest of the work in memory. Results were consistently speeding response time up from hours to minutes, and even hours down to seconds. And that was on much slower hardware. See the long presentation I made at the Oracle User's Group on the subject:<br />http://rodgersnotes.wordpress.com/2010/09/14/oracle-performance-tuning/<br /><br />Another principle is to avoid loops. Even if you already have a lot of data in memory, doing row by row loop de loops on a dataset is a great way to go slow. Much of the secret is in how you code. <br /><br />It's better to first, change bad data models, and fix convoluted code (see also my work on cartesian products in SQL). <br /><br />Adding new hardware and software onto poor data models and code is a really expensive fix.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-43958119829664089152011-07-05T15:54:09.529-07:002011-07-05T15:54:09.529-07:00Ever since I published this post, I've been ta...Ever since I published this post, I've been talking to several people about how problematic the RAM restriction is in pure in-memory technology.<br /><br />It depends on who you ask, I suppose, but I did find something interesting.<br /><br />Google estimates 210 monthly searches for the exact term "out of memory qlikview", using the Google Adwords Keyword Tool.<br /><br />The number itself may be misleading as Google's estimates are not very accurate. But the fact Google even has this term recorded says a lot about people's search patterns.<br /><br />Another interesting thing is that this number was 140, a couple of months ago.<br /><br />As time marches on and data volumes grow, I believe more and more people are going to hit the wall - not just in QlikView, but in any BI technology that utilizes RAM like it's free.Elad Israelihttps://www.blogger.com/profile/14039349385755633029noreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-30699491329205676962011-07-02T04:27:43.690-07:002011-07-02T04:27:43.690-07:00PC: We've been doing surveys of BI users for t...PC: We've been doing surveys of BI users for ten years and our results show that the need for large data sets tends to be exaggerated, and that fast performance is determinant of project success.<br /><br />Eynav: Data compression is closely related to cognition. The ability to predict how well data can be compressed (without actually compressing it) would be AI in my book. ;-)<br /><br />Anon: Interesting point, but solid state disks are fast but not the same as in-memory because the data still has to be loaded from the "disk" into memory for the CPU to access it.<br /><br />Elad: Great post. As I mention in my blog columnar architecture often gets confused with in-memory. <br />http://bi-trends.blogspot.com/2011/06/column-oriented-databases-are-not-same.htmlAnonymoushttps://www.blogger.com/profile/11682285726633289779noreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-17177862633284944332011-01-14T06:20:49.216-08:002011-01-14T06:20:49.216-08:00I am planning articles both on the BI aspects of N...I am planning articles both on the BI aspects of NoSQL and SSD storage. Stay tuned.<br /><br />In a nutshell:<br /><br />The traditional origins of NoSQL do not come from BI and focus on mass scalability for applications that must handle ridiculously extensive read/writes. Because of its original purpose, NoSQL is often synonymous to distributed database architecture (Cassandra, MongoDB, etc).<br /><br />In BI, a distributed architecture introduces more problems than solutions. But because BI is focuses more on 'read' rather than 'read/write' and due to multi-core 64-bit computers, it is possible to apply certain aspects of NoSQL on BI solutions, without forcing a significant learning curve from a DBA whose familiarity is with SQL-based databases. More on that in the article.<br /><br />As far as SSD is concerned - we've actually been doing some testing of our own software on SSD and you do get performance gains. But, I think that it's impact on RDBMS would be limited, at least in the BI space. Once you read a certain point, I/O is not the main bottle neck anymore but rather the memory and CPU cache.<br /><br />EladElad Israelihttps://www.blogger.com/profile/14039349385755633029noreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-81892794355169044752011-01-14T05:49:47.296-08:002011-01-14T05:49:47.296-08:00Hey, and what about the NoSQL movement on this day...Hey, and what about the NoSQL movement on this days with Cassandra, or Hadoop, combined with the new SSD storage.<br /><br />We, on our company are using QlikView, but I like to know new alternatives (like Jaspersoft, Pentaho, Oracle alternatives, and yours, Prism)Anonymoushttps://www.blogger.com/profile/07026574476032596748noreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-84263189239102233352010-10-24T13:45:53.997-07:002010-10-24T13:45:53.997-07:00Sure.
However, that is limited to customers who a...Sure.<br /><br />However, that is limited to customers who already have SharePoint (and SQL 2008 R2). For those who don't, there are much easier, more accessible ways of getting the same thing. Using SharePoint with PowerPivot is overkill for those who don't already have them installed and/or the expertise to use them.Elad Israelihttps://www.blogger.com/profile/14039349385755633029noreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-81068219656230805892010-10-24T13:38:35.846-07:002010-10-24T13:38:35.846-07:00Another advantage of PowerPivot is the integration...Another advantage of PowerPivot is the integration with SharePoint, end users like it.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-54148387376928830562010-10-20T09:07:59.896-07:002010-10-20T09:07:59.896-07:00tse, not sure what you're trying to say. Care...tse, not sure what you're trying to say. Care to elaborate?Elad Israelihttps://www.blogger.com/profile/14039349385755633029noreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-20173559564770338832010-10-20T02:48:54.456-07:002010-10-20T02:48:54.456-07:00Called it what you want, this is old milk on new b...Called it what you want, this is old milk on new bottles!Unknownhttps://www.blogger.com/profile/10436223812642163537noreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-35301482613750261332010-10-07T12:30:54.510-07:002010-10-07T12:30:54.510-07:00That's an interesting idea. Thanks.That's an interesting idea. Thanks.Elad Israelihttps://www.blogger.com/profile/14039349385755633029noreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-41190651007234176772010-10-06T23:16:22.508-07:002010-10-06T23:16:22.508-07:00I have a Solid State Disk myself and it is so extr...I have a Solid State Disk myself and it is so extremely fast that I don't need in-memory BI anymore.<br />Maybe it is a nice idea to blog about "The need to analyze in memory disappears with the advent of SSD's"?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-8730839543622242822010-10-03T12:28:00.224-07:002010-10-03T12:28:00.224-07:00Einav,
Thank you for the kind words.
There is no...Einav,<br /><br />Thank you for the kind words.<br /><br />There is no magic formula you can use to calculate how much RAM you would need for intermediate queries. It depends on a lot of things. Since QlikView usually prefer to do most calculations during the actual data import stage, the 20% value is what I hear them (numeous times) recommend to their customers in order to accommodate the simple click-to-filter functionality of their UI.<br /><br />The way QlikView works is somewhat different than what PowerPivot does so you will have to test and see. I agree that PowerPivot's technology is better engineered for multi-user support (if you consider in-memory technology a good multi-user solution in the first place) but Microsoft hasn't done much yet to capitalize on this. Right now QlikView's technology is better for multi user simply because they've been doing it longer and have worked out many issues Microsoft have yet to understand.<br /><br />EladElad Israelihttps://www.blogger.com/profile/14039349385755633029noreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-42253358832645671822010-09-30T02:48:36.561-07:002010-09-30T02:48:36.561-07:00Thanks, PC.
Not sure about it being a niche solut...Thanks, PC.<br /><br />Not sure about it being a niche solution ;-)<br /><br />EladElad Israelihttps://www.blogger.com/profile/14039349385755633029noreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-85233537893398260112010-09-29T07:31:10.401-07:002010-09-29T07:31:10.401-07:00Good article.
I'm an extensive user of Qlik
th...Good article.<br />I'm an extensive user of Qlik<br />the battle is between amount (and not speed) of data management and h/w evolution.<br /><br />You have probably a good niche solution.<br /> <br />regardsPChttp://www.linkedin.com/in/paolocattaneonoreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-76852096750625693112010-09-27T10:06:20.190-07:002010-09-27T10:06:20.190-07:00Elad,
What an interesting blog. Thank you for sh...Elad, <br /><br />What an interesting blog. Thank you for sharing. I would like to understand a bit better how do you calculate the 20% overload in RAM for each user who query the In-memory data concurrently. <br />Do you have some formula to calculate it? the reason I am asking is actually the fact that we are now deploying Panorama on PowerPivot and we are being asked to size the implementation. <br />As you know, our offering is usually a combination of OLAP (disk) and In-memory (RAM), hence the problem is smaller than the traditional in-memory vendors such as Qlikview. <br />Also, to the best of our knowledge, PowerPivot is a bit better in handling large datasets, which supposed to decrease the overload per concurrent user. <br /><br />Also, we would love to see more interesting technologies opening their platforms for different BI suites to run on, same as I suggested to Qliktech goes also to SiSense. But again, it is a business decision, not always a technical one. We helped different companies become a better platform for BI, why wont we help you. good luck!<br /><br />Eynav Azaria, www.Panorama.comAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-89855743873086350392010-09-24T12:49:29.435-07:002010-09-24T12:49:29.435-07:00Absolutely true, KS.
Performance is just one of t...Absolutely true, KS.<br /><br />Performance is just one of the recurring issues in typical BI implementations. But it's an example of how important the underlying technology is to enable simplifying the BI process. New technology that is not designed within real life practical guidelines is better left within the confines of academic studies.<br /><br />The exciting thing is that there is a lot of BI experience out there, and coupled with the amazing new hardware that is available on any run of the mill computers it is very possible to create fast and adaptive BI without compromising on scalability or endure heavy costs.<br /><br />We have plenty of material about Prism on our website (http://www.sisense.com). You can also download it there. I am also available at elad (-a-t-) sisense.com for any inquiry.<br /><br />Regards,<br />EladElad Israelihttps://www.blogger.com/profile/14039349385755633029noreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-39602766720581555712010-09-24T12:37:58.853-07:002010-09-24T12:37:58.853-07:00Good article, I am a huge QlikView fan but only be...Good article, I am a huge QlikView fan but only became one because I was open to learning about (and trying) new technology. It will be interesting to learn more about Prism. The real key, though, is not just performance, but each step in the development and implementation cycle. To quote a recent CITO Research article, "to have maximum impact, BI (solutions) must be fast. They must be created fast, perform fast, and adapt fast..."<br /><br />-KSAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-9155359000072069952010-09-24T11:52:56.016-07:002010-09-24T11:52:56.016-07:00Thanks for your comment, TB.
The goal of this pos...Thanks for your comment, TB.<br /><br />The goal of this post is not about a particular product, but rather about the future of backbone BI technology.<br /><br />QlikView's success is not debatable and I am by no means saying QlikView is a bad product. The point I am trying to make, in regards to QlikView in particular, is that you do not need in-memory technology to achieve the same results and that holds great benefit for future BI solutions.<br />Thanks again for your comment.Elad Israelihttps://www.blogger.com/profile/14039349385755633029noreply@blogger.comtag:blogger.com,1999:blog-8489991802309187017.post-83532527146549485352010-09-24T10:32:59.390-07:002010-09-24T10:32:59.390-07:00The goal of BI isn't to see how much data you ...The goal of BI isn't to see how much data you can stuff into a box. The measure of usefulness involves how easily and quickly you can ask, refine, and reframe questions of the data. By that measure QlikView is an excellent tool.<br /><br />Thanks for the blog!-TBhttps://www.blogger.com/profile/08386812862574514289noreply@blogger.com