Are CIOs Ready For 'Active Data Warehousing'?Are CIOs Ready For 'Active Data Warehousing'?

Speaking to a packed room of CIOs at the information 500 conference in Tucson, Ariz., this morning, Stephen Brobst, the CTO for Teradata, made a compelling case that the future of data warehousing lies in "activating" the data in storage to enable automated, near-real-time decision-making to increase business' revenue.Making the point that new data loads are outpacing the ability of Moore's Law to keep up, Brobst outlined five stages of "maturity" for organizations grappling with this flood: reporting, analyzing, predicting, "operationalizing," and activating.

Most companies with traditional data storage and access systems, Brobst argued, are stuck in the first three stages and are faced with a "chasm" to cross to move into the final two stages, where data become integrated into daily, hourly business processes, accessible not just to corporate managers but to in-the-field decision makers, and finally "activated," that is, used to set off automatic triggers (when to make a specific offer to a specific customer speaking to a call-center rep, for instance).

Brobst cited a Gartner studay that shows a quarter of existing businesses leading the way in those final two stages, while fully half of all U.S. businesses will be there in 12 months. Needless to say, new data-warehouse architectures, including Teradata's enterprise data warehouse products, will be required to achieve this singular, integrated, "activated" information vision.

I have three responses to this vision. One is that Brobst's diagnosis is much the same thing that Sun Microsystems CTO Greg Papadopoulos is describing when he talks about the "red shift": in the new world of exploding data quantities, the companies that are best able to marshal, integrate, and make use of data across departments and functions will enjoy double-digit growth in the coming years.

Second, I think both Brobst and Gartner are overly optimistic about the maturing process around data warehousing. The availability of effective, relatively inexpensive storage and networking tools notwithstanding, most companies I talk with are still stuck in Phase 1, reporting and collecting data from all their operations, and are only now moving into effective and relevant analytics around that information. As Brobst himself noted, "More than half cost of data warehousing is in data integration and cleansing -- it's an n-squared problem." Small and medium-sized businesses in particular are a long way from making near-real-time business decisions based on the volumes of data flooding their servers.

Third, the red shift or data explosion is going to have a profound effect on the position and perception of IT departments within organizations. Essentially, since the dot-com crash IT has been seen as a cost center, and one that is continually asked to do more with less. Innovation and revenue creation have given way to the necessary evils of cost-cutting and basic maintenance. Once Brobst's concept of the "active data warehouse" becomes a reality (and gets understood by CEOs), whole new landscapes of top-line growth and big new efficiencies will open up -- but only for CIOs who can conquer the data mountains now looming before them.