Real-Time Ambition: Reaching the Potential of Event ProcessingReal-Time Ambition: Reaching the Potential of Event Processing

"Real-time" is now one of the buzz phrases that shows up all over the place, but if there's any technology that can truly deliver on these words, it's the emerging category of complex event processing (CEP). With so many organizations now using conventional business intelligence technologies and approaches, CEP offers a speed-to-insight advantage that may be the next competitive differentiator, but is the technology ready for mainstream adoption?

Developed in the computer science labs of Stanford, Caltech, M.I.T., Cambridge and other universities, CEP was first put into use on Wall Street and in super-secret intelligence applications. The technology spots patterns in complex, high-volume data while it's streaming through business systems and applications, rather than after it's stored and a matter of history. Conventional data warehouse technologies have come a long way in reducing time to analysis, thanks to techniques such as trickle feeding, but CEP latency is measured in milliseconds and the volumes of data processed can exceed 50,000 messages per second.

It's not every application that requires this kind of speed and processing power, but CEP is quickly moving out of limited, high-end applications. The technology is far from mature and major vendors are still waiting on the sidelines, but as you'll discover in this article, CEP is being productized, embedded in other applications and pioneered in more mainstream industries such as shipping. Open-source and "community edition" offerings have even made the technology affordable to small and midsized business seeking rapid detection of changes in customer behavior or system performance.

CEP's Starting Point

The sweet spot for CEP is applications involving high volumes of data, complex analyses and time sensitivity (as in millisecond or sub-milliseconds). Algorithmic trading in the financial services industry is a classic example in which all three requirements are present. Algorithms are basically used to search patterns in data indicating buying or selling opportunities. Wall Street firms have used trading algorithms for decades, but CEP puts the approach on steroids, applying thousands of algorithms simultaneously against tens of thousands of trading data points per second. With CEP, a single broker monitoring a dashboard can now achieve what used to require armies of traders and platoons of PhDs.

CEP applications were once rare and always custom-built, but the technology is now common on Wall Street, and it more often starts with off-the-shelf products. "Best practices are falling into place and costs are starting to come down," says Mary Knox, a Gartner analyst who serves the investment services industry. "We're also seeing CEP bundled with applications such as order management systems and market data distribution systems."

CEP has also been a boon to intelligence agencies, so much so that In-Q-Tel, the technology investment firm set up by the Central Intelligence Agency, has backed CEP vendor StreamBase. Intelligence and military applications include surveillance and alerting, monitoring network traffic and screening message queues or e-mail.

"Two big challenges are coping with the volume and velocity of information as it's streaming," says Troy M. Pearsall, executive vice president of technology initiatives at In-Q-Tel. "It's important to be able to examine the data and see the patterns as they're coming toward you rather than after the fact."

CEP Spreads its Wings

Where CEP once was a custom project, it has been productized in recent years by vendors including Apama (acquired by Progress Software in 2004), Vhayu, StreamBase, Coral8, RiverGlass Inc. , TIBCO, Aleri, Syndera and SeeWhy.

The most recent entrant into the CEP market is BEA, which last month introduced the WebLogic Event Server. IT industry giants are also closely watching this technology. IBM, for one, hosted the First Symposium on Event Processing in March 2006 and it has partnered with Coral8, creating an integration to DB2 9 so XML-based CEP results can be stored in the database. Oracle hosted the Second Event Processing Symposium last November, and it has been blogging about its internal development efforts.

As CEP products and implementations have multiplied, the technology has spread from the pioneers in financial services, intelligence and telecommunications to manufacturers, casinos and retailers, to name just a few industries.

Event-Architecture and CEP

Shipper Con-Way Freight is embracing CEP as part of a years-long push into services-oriented and event-based architecture. To compete with the likes of FedEx and UPS, which have acquired their way into the less-than-truckload (LTL) shipping business, Con-Way knew it had to become a genuine real-time business, so it embarked on its SOA push way back in 1999. That effort laid the groundwork for an event-based architecture initiative in 2003.

"We justified the move into event-driven architecture in the scope of a single project," says Maja Tibbling, lead enterprise architect. "It had to do with integrating billing data with the collection system to reduce days sales outstanding."

Con-Way used TIBCO adaptors and an enterprise service bus (ESB) to propagate all billing-related events from transactional systems to the collections system. This real-time, publish-and-subscribe approach, built on SOA, sped the flow of information and enabled Con-Way to automate system interaction with a third-party collections package, reducing days sales-revenue outstanding. The initiative "saved millions" and ensured that event-based architecture "became the systematic backbone for every project that followed," says Tibbling. But the complex event processing had yet to begin.

In the wake of the billing project, Con-Way gradually extended the architecture to cover all shipment-related events – everything from the 6,000 to 10,000 daily appointments for shipments to the corresponding movement, loading and unloading of trucks. In 2006, TIBCO's BusinessEvents CEP software was added to analyze those events and ensure successful completion of automated business processes tied to mission-critical operations including planning, dispatching and coordination of inbound and outbound trucks.

"We now have millions of business events that are being published and processed in a zero-latency, event-driven manner, and all of the events that happen over time are known to the complex event processor," says Tibbling. "With that technology in the mix, we have visibility into the sequence of events and the relationship between events, so we can start looking across a number of constraints to see threats and opportunities and make broader, better decisions."

Managers will use an event-driven dashboard to guide their planning, and Tibbling says the combination of event-based architecture and CEP has reduced misloads, streamlined planning and increased the performance of the entire freight network. The success hasn't gone unnoticed, as Con-Way was recently ranked number one in Transportation and Logistics on Fortune’s 2007 List of “America’s Most Admired Companies.”

Not For Big Companies Only

With the ranks of CEP practitioners including big government, big finance and big telco, you might think the technology is accessible only to giants (even Con-Way, which is "small" compared with FedEx and UPS, is a $4 billion company). That's not the case, however, as there's an active open-source CEP project called Esper that offers both Java and .NET components. Commercial vendors are also responding with low-cost and even no-cost CEP alternatives. SeeWhy, for one, offers a Community Edition of its CEP product that can be used at no charge in limited deployments. eCourier, a small, London-based same-day courier firm, is using SeeWhy Community Edition in a real-time CRM application to monitor changes in customer activity.

"SeeWhy monitors our bookings and builds a profile of each customer's booking patterns," explains Jay Bregman, CTO and co-founder of the firm, which now has 55 full-time employees and 130 couriers. "The system alerts us if booking levels are dropping so we can call the customer and resolve any problems before they defect."

eCourier equips couriers with GPS-enabled handheld computers, and it differentiates its services with Web-based booking, tracking, billing and logistics information. The company had been using Crystal to create a weekly customer defection report comparing bookings from week to week, but as the customer base grew, "the report became too intense for anyone to analyze and we were just reacting to complaints," says Bregman.

Implemented early this year, it took the SeeWhy deployment about two months to build up profiles of each of the company's 2,000 customers. Alerts are now delivered in real time as the system spots disparities between current bookings and historical booking patterns for each individual customer.

"Some clients book five deliveries a week and others book five deliveries an hour," Bregman explains. "When we get an alert, we know it represents a break in the trend for that particular client."

At least 10 accounts have been saved in the four months the system has been operational, says Bregman, and adds that the company is contemplating using the technology for supply-and-demand planning.

"We have to have the right number of couriers available every day for fast service, but not too many, because that hurts our margins," he explains. "We'd like to use SeeWhy to look at demand patterns and automate the entire planning process. That would really pay dividends as we scale the business to over 1,000 couriers."

Is CEP Ready For Prime Time?

As compelling as CEP is to those who are familiar with the technology, it has a long way to go in terms of broad recognition. "One of the big problems with the technology is that it can be difficult for people to understand it and contextualize it to their business problems because it's not information that you're used to being able to get," says Bregman.

There's also a lack of standardization in the industry. Some vendors, including Progress/Apama and others, support Event Processing Language (EPL). Other vendors, including Coral8 and StreamBase, have query languages that are loosely based on SQL, but CEP presents very different query challenges.

"You have to think about real-time data differently than SQL," says John Leslie, CTO at Wall Street On Demand, an investment research firm that uses StreamBase's software to continuously update more than 400,000 Web-based investment portfolios. "SQL is about dealing with sets of data, but you don't have that concept with real-time data streams."

StreamBase's StreamSQL language, which the company has offered as an open standard, "provides a level of abstraction that keeps your developers out of the weeds of worrying about physical storage and logical data mapping," says Leslie, "but real-time data is different in that there is no beginning and there is no end."

It's unlikely standards debates will get in the way of CEP progress, submits John Morrell, marketing director at Coral8. "There have been multiple meetings among vendors including Coral8, StreamBase, Oracle, IBM and others, and we're coming to grips with issues like making sure we're all using the same terms, raising awareness about what these systems can do and making reference architectures available that everyone can agree upon," he says. "Once that's done, I think we'll work toward language standards."

Morell says it's also unlikely CEP will displace conventional BI technology except in analyses that truly call for real-time insight. "We were recently working with an RFID solution provider to look at how CEP could be used within a system that handled 20 different analyses," he explains. "We checked off about eight of those 20 that could benefit from real-time insight, but the other 12 don't require the technology."

One broad application that stands to benefit from CEP is clickstream analysis and the emerging customer experience management domain. For example, Morrell says Coral8 customer Sallie Mae is driving real-time retention offers by correlating clickstream, voice-response and call-center data with historical transaction and demographic information.

Of course, the final key to unlocking the power of CEP is developing systems and applications that can take advantage of real-time insight. "It's all very well and good to put in an ultra-low-latency CEP solution, but if your infrastructure can't handle that or the end recipient of the alert is a person, you may just create bottlenecks," says Knox of Gartner. "You have to design architectures and infrastructures that can take advantage of real-time insight."