'The BI Survey 8' Finds Customers Want Faster Performance, Better Support'The BI Survey 8' Finds Customers Want Faster Performance, Better Support

Too many business intelligence vendors are drinking their own Kool-Aid. That's one of the big-picture findings of the just-released BI Survey 8. This year's poll, overseen and analyzed by survey founder Nigel Pendse, reveals BI deployments aren't nearly as pervasive as vendors and consultants suggest. What's more, customers aren't nearly as happy about product performance and support as vendors believe. The report also reveals independent vendors fare better, overall, in the survey's prized product performance ratings.

"Generally speaking, the customers of the big vendors are not happy with the way things have turned out, and I think many are turning back to the small specialists, who seem to be doing much better," Pendse says.

Not-So-Pervasive BI

Upholding its reputation as the BI industry's most exhaustive and in-depth report, "The BI Survey 8" is based on detailed feedback from 2,622 users, consultants and vendors in 60 countries, with the United States, Germany and the United Kingdom accounting for the largest number of respondents. Comparisons of user, vendor and consultant perceptions yielded some of the survey's starkest contrasts. For example, although the vendor community has championed the years-long drive to get BI into the hands of more users, Pendse describes that push as "something dreamed up by vendors that want to sell more licenses." In contrast, he points to "The BI Survey 8" finding that users report far more limited penetration of BI tools and technologies than do vendors.

"Users report a typical (median) deployment rate of 8 percent [of employees], but consultants report a rate of 9.5 percent and vendors [say] 14 percent," the survey states. "It seems that vendors believe too much of their own propaganda about mass deployment of their products."

The report defines a BI user as someone who interacts directly with a BI tool. "Being a passive recipient of a PDF report doesn't qualify," Pendse explains. "They don't have to be defining reports or building cubes, but they must have some form of hands-on access, such as access to a dashboard." Pendse says deployment figures skew higher in North America and among larger companies, but he adds that so-called pervasive deployments -- those with more than 50 percent of employees using BI -- account for only 7.8 percent of respondents.

Performance Is Relative

As in past surveys, the most widespread technical problem reported by practitioners was slow query performance. But complaints are down from 21 percent in last year's survey to 18 percent in "The BI Survey 8." The survey finds projects that use query performance as an evaluation criterion tended to be more successful than those that do not. It also finds the faster the query response, "the more business benefits are reported and the more likely it is that business goals will be achieved."

Performance continues to be a top complaint in part because consumer technologies have raised user expectations. "BI query performance has not been getting worse; in fact it improved a little bit in this year's survey -- typically being 5 to 10 seconds," Pendse explains. "In the meantime, however, Google has come along and will query vastly more data than the world's largest data warehouse in .22 seconds, so people's expectations for performance have changed."

Encouragingly, "The BI Survey 8" finds software reliability is a major problem for only 10.1 percent of deployments, slightly better than in last year's survey. What's more, the overall level of satisfaction with BI deployments remains high, with more than 73 percent of respondents reporting their projects "largely or completely met their business goals." Reasons for Choosing Specific Vendors

Reasons Given For Choosing Nominated Products
(click image for larger view)

One of the most interesting and helpful aspects of the annual BI survey is its detailed quantitative analysis of why customers buy certain BI tools, what they use them for and how successful they are in deployment. For example, the "Reasons Given for Choosing Nominated Products Chart" (click on the chart at right) shows why respondents chose the product they described as their primary BI product when completing the survey. (Note: Figures include the sample sizes in parentheses after the product name. Averages for all products are shown at the top of the chart. Better-than-average results are shown in green and below-average ratings are in red.)

As the chart shows, Business Objects was chosen more often than average for its "ability to support large numbers of concurrent users" and its "product reputation," while it was chosen less often than the average for its "large data handling capacity" and its "fast performance." QlikView was selected more often than average for its "ease of use for end users" and "ease of use for application builders"; it was selected less often than average for its "ability to support large numbers of end users" and its "Web architecture."

Industry Sector Analysis By Product
(click image for larger view)

Another revealing selection dimension is the "Industry Sector Analysis by Product" (click on the chart at left). Higher-than-average representation in an industry shows up in green; below-average representation shows up in red. Information Builders' WebFocus, as an example, has a higher-than-average presence in insurance, health-care and government and a lower-than-average representation in consumer package goods manufacturing and media and publishing. MicroStrategy has a higher-than-average presence in retail and a lower-than-average presence in the automotive industry (lucky, one might add, given the state of that industry).

Respondent Roles By Product and Vendor
(click image for larger view)

Examining "Respondent Roles by Product and Vendor" (see chart at right), respondents were asked whether they play a technical or business role. The survey reports end-user-oriented products "like MIK, Inform PM OLAP, Excel PivotTables and Cubeware had high proportions of business users participating, with relatively few purely technical users." In contrast, Actuate has the most technical respondents, followed by Business Objects and MicroStrategy. Cognos Analysis has a much more business-oriented profile among its respondents than does Cognos Reporting.

Among multi-product suites, "Hyperion predictably had more business-oriented users, while Oracle and SQL Server had the most technically oriented," the report states. "SAP respondents, though still relatively technical, were nevertheless more business-oriented than the other large, multi-product vendors." Which Customers Are Happiest?

The 489-page "The BI Survey 8" covers 23 products in total. The most prized survey findings -- those that prompt would-be software buyers to shell out $4,995 for the report -- are the eight aggregated ratings and 26 product-by-product measures of attributes including selections wins, deployment depth, success rates, implementation times, product reliability and customer loyalty, among other measures. Survey publisher Business Application Research Center (BARC) won't allow journalists to share any of the more than 50 dashboards detailing these results, but Pendse says customer sentiment in recent years has favored single-product independents while customers of the acquired mega-vendors have expressed frustration.

"Most people have been pretty disappointed by the results of those acquisitions of a year or two ago," Pendse says. "In most cases the next big releases [following those acquisitions] either came very late, didn't happen at all or didn't meet up with expectations." He cites a Hyperion upgrade touted before Oracle's acquisition for which "the main focus turned into integration with Oracle tools that the average Hyperion customer doesn't even have."

Similarly, the gap between customer perceptions and vendor perceptions of product-support quality was most acute among the mega-vendors. "The companies that are doing the acquiring or getting acquired delivered way worse product support than the one-product independent vendors," Pendse says. "Business Objects customers in particular complained about poor-quality support." Indeed, Business Objects (followed by Hyperion) had the lowest product-support-quality score among the 23 products and vendors ranked.

In contrast, independent products from Panorama, Information Builders, QlikTech and MicroStrategy had the highest overall scores across the 26 product quality, success, support and loyalty attributes measured.

The report notes many independent vendors cooperated with and supported the survey by encouraging customers to participate in the Web-based surveys. It also acknowledges that SAP, Oracle and SAS did not support the survey drive. Did this affect the outcome of the survey results?

"We didn't get as many SAS customers as we would have liked, but we did get good [statistically valid] samples of SAP, Hyperion and Oracle customers," Pendse says.

And given that the surveys were conducted from June to September 2008, has the value of the research been diminished by drastic changes in the global economy following the research period? "The Survey is about what has happened over a long period, not just the most recent purchases and implementations," Pendse responds. "If we are to speculate, I think it's a safe bet that new BI projects are likely to be smaller and more pragmatic, rather than grandiose -- which is a development I welcome. The Survey results confirm smaller, faster projects are much more successful than attempted strategic BI projects, which take too long, cost too much, deliver too-little and often fail."