Special Report: BI Megatrends 2008Special Report: BI Megatrends 2008

BI, we hardly knew ye. Just when everyone thought there was general agreement on what business intelligence (BI) was – methods, tools and systems for manipulating quantitative data to build views and perform analysis for decision-making – the industry changes so much that we may again have competing definitions.

Is BI part of an organization’s information management infrastructure? Is it middleware? Do casual users need it, or does the biggest ROI come from providing richer analytics to power analysts? Should it be separate from or tightly integrated with applications, rules and process management systems? Perhaps paradoxically, the BI industry’s maturation could bring more diversity in the answers to these questions, not less.

As the New Year unfolds, it would be easy to reduce what happened to the BI industry in 2007 to one colossal megatrend: consolidation. Oracle bought Hyperion Software. SAP acquired Business Objects (and a few others). And Cognos, which itself acquired Celequest in late 2006 and Applix in 2007, is in the process of being snapped up by IBM. To be sure, several important BI tool and platform players are still independent, including Actuate, Datawatch, Information Builders, MicroStrategy, Panorama and SAS. Also unchanging is the growing presence of Microsoft, not to mention that of open source BI, the domain of Pentaho and Jaspersoft. Performance management and specialized data- and text-analytics vendors still abound. Nonetheless, it feels like the industry as a whole has turned in a new direction and will never again resemble its pre-2007 self.

Could there be more consolidation in 2008? Entirely possible: If nothing else, IBM, Microsoft, Oracle and SAP are now markets unto themselves and will undoubtedly stay on the hunt to acquire software products and services that fill holes and create competitive advantages. This article explores how the consolidation trend might play out, but more importantly it presents six important megatrends in the context of questions organizations and BI practitioners must answer as they evaluate software and work to deliver on key strategies with BI.

Consolidation Megatrend: Delayed Impact

Companies acquire other companies for a number of reasons, many of which are not prominently mentioned at the smiles-all-around press conferences heralding new synergies and competitive edges. Big investors play a role, with one side wanting to see quick revenue and market share growth and the other a sale at peak valuation. Of course, what matters going forward is how well the cultures of these larger software providers meld together to meet new strategic, technology and market objectives. This process rarely happens overnight; thus, for BI customers and the industry as a whole, the true significance of these acquisitions could take months if not years to come clear. In 2008, everyone will be watching and most likely waiting to know more before making any moves: Will the BI software offered by these larger vendors meet customers’ requirements more effectively – or less?

A longstanding part of BI software’s value proposition has been its independence from the applications and databases underneath. Living outside these systems, BI can focus on meeting users’ data access, analysis and reporting needs; and being free of any one system, BI can work with multiple and heterogeneous data sources. Of course, making these promises a reality has typically required hard work by both vendors and IT staff.

What will happen now? The hope is that some of the difficult plumbing will begin to ease – or at least become more of the vendor’s responsibility. Of course, the direction that improvements in data integration and access take will likely depend on the vendor combination.

All three of the major acquired BI vendors have gone to great lengths to point out that their products will work as well as ever with what are now competitors’ applications and information management platforms. In other words, it should be just like old times: in SAP shops, Cognos software will remain very competitive with SAP’s Business Objects portfolio and Oracle's Hyperion offerings; these latter two will continue to have important partnerships with IBM; and Oracle will still be a big factor even in predominantly IBM and SAP installations. Time can only tell how soon competitive juices overwhelm the chivalrous atmosphere. And, needless to say, if the big vendors’ attentions turn too far inward toward sorting out sales territories and the like, it will create openings for independents, including Microsoft, to make their case.

Many CIOs will be pleased to have “fewer throats to choke” and bring consolidation to their own organizations, possibly witling their own BI portfolios down to a chosen big vendor. Much will depend on how the organization views BI: as information management infrastructure, as part of applications or, in its traditionally delineated role, above it all. Importantly, however, organizations have to guard against letting consolidation lead to neglect in BI innovation and complacency in how they use information for greater human and process efficiency and effectiveness.

Accurate, timely and comprehensive data access, analysis and reporting are urgent priorities for businesses seeking to establish informed decision-making at all levels. Innovative companies want to run operations and strategic activities such as pricing and demand management using analytic applications, not just spreadsheets and hunches. The most sophisticated and demanding of these are pushing the BI and data warehousing envelope. New technology approaches and architectures are required.

With the consolidation megatrend in the background, let’s take a look at six questions organizations must answer in shaping future BI implementations. Perhaps it is the former journalist in me, but I chose to break them out as the “who, what, when, where, why and how” of BI.

Who Needs BI?

Organizations overwhelmingly see the value of making information accessible to all relevant functions in operations, according to a recently completed operational BI benchmark study by Ventana Research (Intelligent Enterprise served as a media sponsor, so we thank members of the IE community for sharing their insights). The problem is making it happen. The research found that the majority of operational BI deployments currently serve 100 or fewer users, not thousands. Dashboards have been a huge boon, but many operational workers still collaborate by emailing or sharing printouts of spreadsheets and reports.

Before settling on what technology will be cost effective for expanding the reach of BI, organizations must determine who needs what sort of data access, reporting and analysis capabilities. Unsatisfactory attention to user requirements, particularly regarding reporting, is a major factor slowing down BI’s growth and a frequent reason why deployments have to be done over. The “build it, and they will come” era in BI and data warehousing is over. The megatrend here is about people: We will see a rise in the establishment and influence of BI competency centers and advisory councils, involving business and IT executive sponsors, to hammer out the mission and user requirements of large BI implementations.

What Information Do Users Need?

To improve customer service, many organizations are pursuing the elusive single view of the customer. Access to multiple data sources is generally necessary for this and similar efforts focused on products, suppliers, prices, performance metrics and more. Information integration, involving both structured data and unstructured content sources, is increasingly seen as a business imperative. With the volume of data growing and new forms, including sensor data and multimedia coming online, information architects have to know more about the information itself; otherwise, routine tasks of moving, loading and providing access to the data will become overwhelmingly difficult and error prone.

Master data management (MDM) was a hot topic in 2007 and will continue to be in 2008. From Microsoft to Teradata, BI and data warehousing vendors are finding that they have to have MDM in their portfolios. MDM is the process of defining and managing master data, often to establish a reference source about customers or other objects of interest across the organization. Some of the most interesting smaller vendors, including Exeros, Initiate Systems, Siperian and Trillium, are focused on MDM activities such as data relationship discovery, profiling, data quality and customer data integration. Along with data governance, MDM will be a megatrend as organizations seek to know more about their information, the quality of sources and the relationships between pieces of information. MDM will be important to how organizations speed access to the right information and present it in the right context.

When Do Users Need to Know?

The move to provide operational BI is a driving reason why organizations are shifting their information management infrastructures to support at least intraday updating of data. Nearly 80 percent of participants in the Ventana Research operational BI study regard one or more updates a day as essential and 32 percent say data must be updated at least once an hour. The study revealed a significant shift toward greater update frequency since Ventana’s 2005 report. Of course, delivering data to operational users isn’t the only issue; companies in retail, gaming, manufacturing and other industries want to apply real-time analytics to timely data so their response to customer behavior, for example, can be immediate.

Coming out of a batch-oriented history, “real time” represents a huge, ongoing shift for BI and data warehousing. Systems that are loading and transforming terabytes now have to perform these feats while meeting OLTP-like availability, reliability and concurrency requirements. This megatrend is drawing attention to appliance technology from DATAllegro, Greenplum, Ingres and Netezza for certain workloads; specialized databases and accelerators from ParAccel, Hyperroll, Sybase IQ and Vertica that promise faster performance; and new configurations built for speed from HP (NeoView) and Teradata (Release 12). These vendors plus configurations from IBM and Oracle are also employing messaging and transaction middleware to improve the speed of the information infrastructure. Clearly, in 2008 speed will be a BI and analytics obsession.

Where Should BI Live?

As discussed previously, everyone will be watching the direction in which BI is pulled by IBM (toward information management), Oracle (similar, plus middleware), Microsoft (personal productivity applications and collaborative portals) and SAP (applications, processes and middleware). In organizations in which service-oriented architecture (SOA) is maturing, we will see growing interest in BI and data services that can be plugged into composite applications. This will spur a secondary trend toward providing software development kits that focus on rapid creation of Web-based BI applications and services.

Will data virtualization mean that BI no longer needs a home and can be pulled and reshaped as necessary? Thus far, the virtualization shift has been a slow work in progress; enterprise information integration (EII) is far from the mainstream. As well, statelessness may be a challenge if data warehouses must act like OLTP systems. Nonetheless, SOA and other advances, including MDM, will make data virtualization a megatrend in BI for 2008.

Why Is This Information Important?

One of the great things about Google and other search engines is that they always take your information needs seriously. Type in just about any word combination and the engine will roar off and bring you back results. Whether the answers are useful is a different question, but no search is denied. We are all equal in the eyes of Google.

Inside an enterprise, however, some queries and searches are more important than others. If allowed to proceed, some could bring back the wrong results, expose sensitive data or bring systems to their knees. Regulatory compliance increasingly involves watching the flow of information and observing who wants to change it, and why. Thus, the “why” megatrend will be about an increased management effort to understand the priority of workloads and why particular user communities need particular pieces of information. Tools from companies such as Appfluent and Teleran are helpful; we will likely see a cottage market develop to improve not only performance tuning but also the general monitoring of information flow and usage according a person’s role in processes and the organization.

This is also where search, taxonomies and the development of semantic understanding of information could create business advantages. BI only provides part of what’s needed for decision making; in many organizations, unstructured data is a huge and largely untapped resource for analysis. Putting the two together will raise the level of intelligence about why certain information is important.

How Can BI Provide Higher Value?

For some business operations, improving customer service and raising process efficiency are critical benefits organizations want to receive from BI deployments. Others need timely and complete data to improve forecasting, understand costs or provide information services for business collaboration. BI’s importance should always be closely tied to the business benefits it can provide. While some of these benefits are tried and true, others are more dynamic given market changes and new leadership priorities. The BI infrastructure has to balance stability with flexibility.

To achieve this balance and expand its value, BI needs an influx of new technology. One space to watch is complex event processing (CEP) and the development of operational intelligence focused on monitoring and analysis of events. Rather than wait until the system can provide a view of historical data, leading organizations are shifting to enable rapid understanding of real-time event patterns, causation and correlation. Fraud detection, securities trading and online commerce are prominent areas for CEP applications. The development of operational intelligence, with CEP as the key technology, will be a megatrend in 2008. How much this new approach coexists with or supplants BI and data warehousing, not to mention business process and rules management, will be one of the most interesting stories of New Year.

The who, what, when, where, why and how megatrends described in this article open up the possibility of many new definitions of BI, analytics and their role for users and business processes. So much for the same-old BI: there’s no place for complacency in the pursuit of business intelligence.

David Stodder is vice president and research director - information management and IT performance management at Ventana Research. He was previously editor-in-chief and editorial director of Intelligent Enterprise.