CEP for ETL: Next-Generation Tech for Low-Latency Data WarehousingCEP for ETL: Next-Generation Tech for Low-Latency Data Warehousing

Complex event processing (CEP) is the next big thing in data integration. At least that's the game plan at Microsoft and Informatica. Given that IBM and Oracle also have CEP available on their technology toolbelts, there's little doubt that success will breed more adaptations of CEP for low-latency data integration.

In case you're not familiar with CEP (also known as stream processing), it's a technology that has matured out of rarified use in financial trading and government intelligence gathering scenarios. Today CEP is being employed for real-time network threat detection, transportation optimization, online commerce and smart grid power management. For CEP, "real time" means processing capacities and speeds ranging anywhere from thousands to millions of events (or patterns) detected within sub-seconds or even milliseconds.

Here's how Tom Casey, General Manager, SQL Server Business Intelligence, describes how Microsoft intends to exploit CEP technologies set to debut in next year's planned SQL Server 2008 R2 launch (and set to debut this month in a community technology preview release): "You can build all kinds of applications with this ingredient technology -- from business process optimization to RFID aggregation. We will process standing queries at very high scale and with high throughput. There are many possible scenarios, but one that we'll focus on is real-time updates to the data warehouse. We're talking about the ability to pipe things through sensor networks, Web analytics, Weblogs, clickstreams or other data stream types. You'll then be able to determine thresholds or categorizations, update your core business activity monitoring environments and write to your data warehouse."

Real-time monitoring of sensor networks, for example, could mean optimized logistics facilitated by RFID sensors or real-time threat/risk detection through information or communications networks. Real-time Web analytic and clickstream analysis would yield upsell- and cross-sell offers while customers are still on your Web site. And in this case, it would be real "real-time," as in sub-seconds, not the minutes or hours we often hear about in today's "low-latency" warehousing scenarios. It's not clear to me whether the technology will turbocharge existing techniques such as change data capture and trickle feeding or whether it will replace these approaches outright. But suffice it to say that it will change the game.

With its September acquisition of CEP vendor Agent Logic, Informatica also sees the opportunity for CEP in low-latency integration scenarios. According to a press release on that deal..."The combination of Agent Logic and Informatica will advance Informatica's technology leadership in two fundamental ways. First, the combination of Agent Logic Complex Event Processing and the Informatica Platform enables another type of broader data integration project -- event-driven active data integration. Second, the combination of Agent Logic Complex Event Processing and Informatica Identity Resolution supports identity-aware event processing. [These two approaches] will enable a variety of applications in financial services, healthcare and, particularly, the public sector.Of course, plenty of companies are looking at blending CEP and BI, from Aleri to Truviso and many points on the alphabet in between. As Seth Grimes explores in this post, the combination has been pondered for years, but things are finally starting to perk. With this latest news from Microsoft and Informatica, perhaps 2010 will be a breakout year for CEP as applied to data warehousing and business intelligence gathering.Complex event processing (CEP) is the next big thing in data integration. At least that's the game plan at Microsoft and Informatica. And given that IBM and Oracle also have CEP available on their technology toolbelts, there's little doubt that success will breed more adaptations of CEP for low-latency data integration.