The Making of a Real-Time Hero 2The Making of a Real-Time Hero 2

Born in a batch world, data warehousing has exploited technological advances to become a mission-critical resource for deciphering trends, segmenting markets, and gaining a deeper understanding of past business performance. Thanks to data warehousing, strategic decision-making is no longer confined by human limits. Automated reporting and business intelligence (BI) analysis software delivers information culled from superhuman data stores that will only get bigger and more muscular over time.

Time: There lies the rub — and the greatest challenge to the future of data warehousing. "Information latency" has become Lex Luthor to this data superhero. Time is the nemesis that most threatens its greater success and glory. Business cycles are shrinking; customer demand is fickle; and in some industries, margins are so thin that only knowledge and its immediate application can drive profitability. Meanwhile, BI is expanding beyond its brainy early adopters and heading for the enterprise — and must therefore meet the needs of a widening variety of users, some of whom need a continuous influx of fresh, "real-time" information. The popularity of enterprise dashboards and portals is raising expectations about an "always on" world in which important changes are noted by the BI system and reflected in the interface continuously. Latency must be driven out.

Metrics and monitoring to serve performance management and corporate governance objectives are also changing the nature of what's expected. Ventana Research, in a 2003 Operational Performance Management End User Report, stated that 62 percent of respondents thought measuring and monitoring of business activities and processes was "very important," with "improving efficiency" cited as the chief reason. Quite often, improving business efficiency is the primary reason for building a data warehouse. Thus, it's clear that the data warehouse will be critical to excellence in performance and process management.

As we head deeper into a new century, these influences are sending the entire concept of a data warehouse into transition. Rising alongside traditional "strategic" needs is a new set of demands for data warehouse support for operational, event-driven, and "tactical" activity that involves rapid decision-making to serve a specific process or action.

For some organizations, the urgency goes beyond serving employees and business partners. Customers are now part of the information supply chain. "With strong demand coming directly from customers that ask for fast answers to meet their needs, not a single company can afford to treat customers in a time other than 'immediately,'" says Mauricio Novaes, Loyalty Manager at Claro (SP1). "Our company has to be prepared to answer the customer's questions. It doesn't matter if the customer is on the phone with a call center, searching for information on the Web, or walking through a store. The information has to be the same: timely and correct."

To Noel-Levitz, a service provider that helps more than 1,600 colleges and universities manage enrollment, student recruitment, and retention, real-time BI is about speeding up the delivery of analytics. "Institutions make real-time decisions about the most effective ways to recruit individuals, to intervene with students at risk of dropping out, and to avert student loan default," says Tim Thein, senior vice president of the Littleton, Colorado-based company. "We use SAS analytics to score different populations — and the results must be delivered in real time. Anything less can lead to administrative inefficiencies and to less-than-optimal service for the student — the education institution's number one customer."

Answering the Call

Novaes reports that Claro is meeting real-time BI demand by integrating information about each customer's contact point. "We are developing what we all Touch Point Server to integrate our database marketing infrastructure with that of the data warehouse. This synchronicity is what we need to support 'touch' CRM campaigns that are designed to respond appropriately to each customer's needs in every contact with the company. We are improving our hardware capacity to enable real-time answers to customers at these touch points. We are finding that business demand is causing exponential growth in our transaction data."

Using Teradata database software from NCR's Teradata Division, Claro employs an operational data store (ODS) to integrate information about customer transaction behavior, especially for use in predictive analytics. An ODS frequently helps organizations by first facilitating data cleansing and other quality activities, and second and most importantly, providing a nearer to real-time supply of accurate, structured data to users and their BI systems. "We don't have to store this data forever," Novaes, explains. "We can define what we need to store from each customer. Or, we can store different periods for different customer segments."

Claro is planning for "mixed workload" data warehousing; that is, where the system supports both strategic and tactical BI and analytics. "At the same time that the data warehouse is being asked to work with actual transaction data," Novaes says, "it is being asked to respond to several automatic reports based on data from strategic and tactical areas, including finance and marketing."

Analytics as Services

Noel-Levitz employs a services-based approach. "We meet the need for real-time BI by using Microsoft Visual Basic .Net and SAS Integration Technologies, as well as SAS/Access Interface to ODBC," says Thein. "Developers created custom Web services that enable clients to input their data, manipulate it, and then submit it for scoring. SAS Integration Technologies lays a foundation for standardized communications and effective information distribution, and SAS/Access enables SAS solutions to read, write, and update data to SQL tables."

Thein continues: "Our statisticians save client data on a SQL server and build their recruitment models with SAS Enterprise Miner. Analysts create a scoring algorithm based on variables unique to a specific institution. The algorithm is deployed via Web services, and the university, using .Net Web services created by Noel-Levitz, submits the data about prospective students. Another Web services routine writes the data to SQL tables. On the back side, integration technology picks up the data via SAS/Access, runs the scoring algorithm created in SAS Enterprise Miner, and returns the scores to the SQL tables. Then, Web services return the score — and the probability of whether a student will choose to attend — back to the end user."

Web services are a key part of evolving middleware approaches to both application and data integration, where the term "enterprise information integration" (EII) is gaining favor to describe a next-generation approach to the amalgamation of data to answer a query. If organizations can streamline (or bypass) the typically time-consuming replication and loading steps — not to mention the creation of downstream data marts and OLAP cubes — they can reduce information latency and come closer to real-time BI and data warehousing goals.

Certainly, exposing users to and involving users with as little of the data integration and query service infrastructure as possible goes a long way toward creating a real-time BI perception. Noel-Levitz offers a good example of how Web services help to meet this requirement.

Passivity is Passe

Within the data warehouse edifice itself, however, changes are afoot to stamp out the latency nemesis. Availability must increase for updating and query activity. Again, middleware, federation, and EII are shaping up as answers — ones championed by IBM, among others. Oracle has traditionally focused on greater centralization to reduce the number of sources and the technology required to make distributed queries work. The balance desired between strategic and tactical data warehouse activity will likely dictate whether a federated or centralized approach is appropriate.

Richard Winter, president of Winter Corp., along with Stephen Brobst, now chief technology officer with NCR's Teradata Division, developed the concept of "active" data warehousing, which has become an important part of Teradata's vision. "The active data warehouse addresses a type of automated decision-making that has rarely been accomplished in the past," Winter wrote in the Fall 2000 Teradata Review. "It is too complex and far reaching in its data needs to fit within the picture for operational systems; it is too time sensitive to have been attempted in most data warehouses. And yet it is emerging as the critical factor in enterprise performance as e-business takes hold and pervasive computing promises to speed the pace of many businesses even further."

"As an engineer by trade, I can't stand this 'real-time' expression, even though I get forced to use it," Brobst says now. "By engineering standards of flight control systems and so forth, we're not really talking about 'real time.' I prefer the term 'right time' because what we're talking about is aligning business processes with data availability, freshness, performance, and delivery. The concept of an active data warehouse is broader than just 'real time.' As opposed to passive reporting, an active data warehouse is actively involved in business decision-making. That might mean the user needs data within two seconds; or if 20 minutes is appropriate, that's fine, too. We shouldn't be driving what we do by technology, but by business processes."

"Plus, when we talk about real time, there are really two types," Brobst continued. "The first is where you need information provided to tactical decision-support applications in real time, but the data doesn't have to be updated in real time. A good example would be a call center such as BCP, one of the largest communications providers in Brazil. BCP's system will retrieve call detail records and perform analysis so that customer representatives can offer the best plan. The data doesn't have to be calls you made in the last 13 minutes; the real-time response is about getting data through the scoring algorithms and other analysis."

Brobst said that the second type of real time data warehousing is where the data itself is as up-to-date as possible. "But obviously, it doesn't make any difference if I can't deliver the data fast enough. If the data is only two seconds old but it takes me longer than two seconds to deliver the data, that's not very interesting. The big issue becomes cost justification. Is it worth the technology that you need to get that last two minutes of data?"

Middleware technology is changing how companies assess whether they should try to achieve real-time data warehousing for tactical users. "But middleware is an enabler, not a driver," Brobst said. "During the early adopter phase of active data warehousing, unless you were Wal-Mart, or some other gargantuan company, you couldn't afford to build the middleware yourself. Now, we're moving beyond this early phase because tools are coming on the market that will enable a greater number of companies to achieve active data warehousing goals. When middleware was largely file-based, it was totally inappropriate for active data warehousing. Now, we're seeing companies such as Ab Initio produce stream-based tools, which are highly useful."

Strategic and Tactical

Transaction databases and data warehouses — whether for strategic or tactical decision-making — will always be tough to support within one system because they serve different purposes. "The data warehouse will never replace transactional bookkeeping — that's the golden copy," said Brobst. However, the technology goal of most data warehouse solution providers is to support a mixed workload of strategic and tactical decision-making.

By developing an intelligent I/O subsystem and query optimizer that can determine whether the nature of the query is strategic or tactical — and then adjust how the database responds internally — Teradata has produced "the only commercial databases that can handle a mixed data warehouse workload," Brobst asserts. The company is working on enhancing the current release with the ability to do data mining and run scoring models that reveal "propensity to buy," for example, and send the results to assist in tactical decisions.

Discussions with user organizations and leading technology providers reveal the sense that an expanding universe of BI and data warehousing applications is about to unfold. Tactical data warehousing applications will encounter "green field" opportunities in many organizations, especially those frustrated by the limitations of current call center and CRM applications. "Real time" may remain in the eye of the beholder: but if the data warehouse can save the day by delivering higher efficiency and more profitable customer relationships, it will indeed be a superhero.

David Stodder [[email protected]] is the Editorial Director and Editor-in-Chief of Intelligent Enterprise.