Real-Time Analytics is Spreading and Getting EasierReal-Time Analytics is Spreading and Getting Easier

Everything today is moving faster than it used to, from Amazon delivery times (2-day shipping to same-day shipping) to software development cycles (Agile instead of waterfall). We can look things up in an instant on Google, or via voice on Amazon Alexa, instead of looking in a book or going to the library. So it's hardly surprising that more organizations are looking to get faster with their analytic insights, and even point them at event stream data as well.

Gartner VP Distinguished Analyst W. Roy Schulte is a specialist in the area of event stream processing and real time analytics, and he provided an update on their growing adoption rates at the Gartner Data and Analytics Summit in Grapevine, Texas this month.

He said that while this type of work used to be typically processed in racks at the data center, today this processing has spread out to other areas, too, including at the edge, in branch offices, in factories, and sometimes even in trucks and on trains. information reported on QVC's efforts to incorporate real-time analytics into its own operations last year.

(In a show of hands at Schulte's session at the Gartner event this month, about a quarter of attendees indicated that they have real-time event stream processing in production, and another few indicated they have it in pilot.)

Schulte said that this type of work has spread to regional cloud and corporate data centers and is proliferating in open source and hybrid open source/commercial deployments.

"The sophistication of the applications are richer and more powerful," he said. "We are seeing things like machine learning operating on top of event streams."

For organizations that are just getting started with analytics on event streams, Schulte said, "It's important to know that you already have a lot of data you can use." For instance, there is data coming from business systems, order records, streams of insurance claims, ATM information, transactions, and searches. Data can come from IoT devices, RFID, geolocation devices, and other instruments.

"Maybe I have days or weeks' worth of data. I correlate that with the fact that the machine finally broke. So I work to create a model that looks for factors that impact failure." -- Gartner's Roy Schulte

For those new to the concept, Schulte noted that an "event" is defined as a state change or something that happens, and "event streams" are a sequence of event data objects in order. An "event" is defined as anything that happens, or a state change. An "event stream" is a sequence of event data objects in order. Event data is not state data like a bank account balance or a truck location, and it is not reference data like a price list or a customer name.

When organizations think about real-time analytics, they are thinking about getting data in motion (as opposed to data at rest), Schulte said. There are a few common use cases for this kind of data, he said.

The classic use case for this is situational awareness. For instance, your real-time dashboard is updated. Maybe it monitors your trucking fleet or contact center or Twitter feed or stock portfolio.

Another use case is decision automation for sense-and-respond applications. For instance, the analysis of the data triggers an alert to possible fraud, or to a market opportunity, or to a weather change. That alert can be part of an automated system that then adjusts machine settings. This kind of decision automation can also be put into play in IT systems that generate alerts and automate actions.

A third use case applies analytics to historical event streams, and Schulte said that anything that is older than 15 minutes is considered a historical event stream.

"Maybe I have days or weeks' worth of data. I correlate that with the fact that the machine finally broke," he said. "So I work to create a model that looks for factors that impact failure."

Schulte said that open source software is a growing influence in the event stream processing space. There are 10 open source ESP platforms, and eight open source streaming data integration platforms. In addition, vendors embed open source technology in hybrid open/closed products. Among the big names are Apache Storm, Apache Spark and Spark Streaming, Kafka, R, Python, and Spark ML and MLib.

But organizations can expect the whole effort to get easier in years to come as more vendors incorporate ESP into existing tools. 

Virtually all IoT platforms incorporate some kind of event stream processor, and many other products from technology vendors now also include event stream processors, too. Schulte further noted that a growing number of BI and analytics platforms, including Microsoft Power BI,  now incorporate event stream processors.

"Expect to see more of these traditional data discovery and data reporting tools adding event stream processing," he said.