Esper - what and why in Complex Event Processing

I have been generating this definition out of the word cloud image generated from Esper documentation:

An engine to query streams of events using select statements over time windows and pattern expressions.
(by streams of events you need to think continuous queries vs traditional store and later query analytics)

Esper is actually much more capable - especially in the just released Esper 3.2, that for example brings support for a subset of nascent not finalized ANSI SQL standard match_recognize.
The Esper 3.2 documentation has a nice summary on how match_recognize compares to Esper well established select from patttern[ ... ] and followed-by EPL language features.

It is also great to see EsperTech on the Forrester Wave Complex Event Processing Platforms, Q3 2009: "Forrester evaluated nine complex event processing (CEP) platforms using 114 criteria (...). EsperTech, the only open source option, is also a Strong Performer."

Esper, OSGi and Complex Event Processing

There is an interesting article that discusses the pros and cons of integrating OSGi with Esper to achieve modularity and dynamicity beyond what Esper already provides.

The article goes over comparing 3 scenarios and comes with sample code (and complete source, with Maven and Eclipse artifacts)

1. Exposing Esper runtime and administrative interfaces as OSGi services
2. Layering Esper interfaces as OSGi services through some service provider abstraction
3. Using Esper and the OSGi EventAdmin service specification
   

This third approach although a bit more complex for readers without OSGi background is the recommended one by the author - Valery Abu-Eid.

It is nice to see such exploration going on. I know we already have users running Esper in OSGi environment, although the vast majority is still running on bare JSE JVM or on JEE / servlet container stacks.

Readers familiar with the matter will also notice that WebLogic Event Server (now Oracle CEP) - is also leveraging OSGi. A main difference is that Esper does not mandate OSGi at all while the BEA/Oracle product does mandate end users to author OSGi modules - with pros and cons.

If you have experimented with similar stacks I'd be pleased to hear about your experience.
There is also good material available from the DynamicJava.org site to get started with OSGi based runtimes - including their own OSGi stack.

Read more about Esper and OSGi experiments at DynamicJava.org

CEP engine or CEP platform? Or both?

There has been recurrent debate and opposition between the value proposition of a CEP engine and a CEP platform.

The archetype of a CEP engine would be something like Esper. You can fully embed it in an existing runtime (Java or .NET/C# for NEsper), ranging from standalone JVMs with bare metal thread pools to JEE application servers or SOA stacks.
The archetype of a CEP platform would be something more like Streambase, or the Oracle CEP server or well you name it - where the product is a server - no choice - sometime even worse with a whole bunch of dependencies (for example a relational database and/or a mandatory JEE application server - see IBM Business Events for example).

I am firm believer that there are numerous technical advantages of getting started with Complex Event Processing using a CEP engine.
Check out the Esper examples (one integrates with JMS f.e. TibcoEMS, another with JBoss, etc) or the growing list of ISVs embedding Esper into their own solutions - from asset tracking, network security and trade platforms of all kinds.
More control and flexibility and more reuse of your existing investments.
This also fits well if IT, speed and flexibility is a competitive advantage - which for example was echoed in a recent WallStreet & Technology article:

Wall Street Opens Doors to Open Source Technologies
"Esper is already built. Instead of being a standalone server product that runs as a separate product, it's a library that you link in with your own proprietary stuff. That way you don't have to worry about another additional process." (an industry source who is involved in the development of trading systems at a major investment bank)

Needless to say, things are not black or white. I am also a firm believer that there are numerous long-run advantages of scaling up your CEP use through a CEP platform approach. Enforce shared practice for monitoring, high availability, deployment unit management and consistency across projects.
Interestingly, Esper tackles this through its (commercial) Enterprise Edition - with bells and whistles around management, tooling and high availability.
More enterprise *ilities and more wisdom long run - likely.

Middleware as a service

Although my last posts have mostly been related to Complex Event Processing and the various vendors and solutions out there, I have spent some time recently exploring the IaaS and PaaS markets.
I won't provide my analysis but instead wrap up on the few hands on experience I had thus far - all promising although with vastly different perspectives for enterprise adoption.

Amazon EC2
I have been a long time user of Amazon EC2 and did a few experiment with it, like running Esper performance benchmark and ensuring it scales linearly as I kickoff Amazon AMI with more CPU and RAM capacity. Very useful although you're send back to school manual on OS-level handling, package install and being root user...

VMWare / BEA ex LiquidVM / ex VirtualEdition
I had of course been quiet familiar with VMWare over the last few years, heading BEA' WebLogic Server Virtual Edition adoption in my geography. Things happens to have slow down a bit on this chapter, although JRockit Virtual Edition is very much on the Oracle agenda as it was explained at Oracle OpenWorld 2008. I expect more this year unleashed with a wider perspective around the cloud topic as some expects.
For prior (although outdated) work on this - check out this slide deck (pdf, slow site).
Very promissing convergence between Java runtime and virtualized stack - patented by the way... (20090070776, "SYSTEM AND METHOD TO IMPROVE MEMORY USAGE IN VIRTUAL MACHINES RUNNING AS HYPERVISOR GUESTS")

Google AppEngine
Recently I have been playing with the Google AppEngine Java - very neat - build upon a few open source bits like Jetty (which also happens to be used by the BEA WebLogic Event Server despite BEA has been a long time maker of its own servlet stack in WebLogic), although the Eclipse IDE support is limited to a bare minimum and the security restriction (class white list) is time wasting in the end-developer hands to validate compatibility of existing stacks.

Is Google AppEngine scaring for IBM?
I recently came accross an IBM lab paper titled "Porting Google App Engine Applications to IBM Middleware".
It provides some API-level compatibility to port GAE java apps to a good old WebSphere / Tivoli stack but is unfortunately missing the whole point.

Middleware as a Service
The clear value proposition of GAE is middleware as a service namely:
- no software install
- out of the box load balancing and clustering
- scale up/down and out on demand
- pay per use

WebLogic on Amazon EC2 and more
From that perspective, the WebLogic Server on EC2 technical white paper and the WebLogic Server in the cloud Oracle' datasheet are far better aligned to the value proposition - although incomplete in a comparison to GAE - as one would have to add Coherence (where GAE provides memcache / JSR-107 JCache) and very likely Enterprise Manager or ex-BEA WebLogic Liquid Operation Control to deal with self-provisionning and scale on demand.

CEP and JDBC/ODBC for ad-hoc queries on real-time data

Sounds like EsperJDBC and Aleri JDBC/ODBC capabilities have found some echo. Coral8 just announced interoperability with ODBC with its 5.6 minor release - 3 months after EsperJDBC introduction.
The Coral8 documentation states that Coral8 now can expose public windows (equivalent of Esper named windows) over ODBC and not just only with a proprietary SDK.

If you want to explore this concept with Esper - this slide deck is a must read:

EsperJDBC

View more presentations from espertech. (tags: esper cep)

A demo also demonstrates how you can perform ad-hoc queries on your CEP real-time data and computations using any JDBC compliant tool - namely Actuate / Eclipse BIRT and create dashboards in minutes.

(Note: EsperJDBC is not open source and only available in EsperTech' commercial offering although you can access free trials)

(Note: Post updated as per Jeff comment on Aleri)

IBM WebSphere Business Events - going Esper? And WebSphere eXtreme Scale going on his own? And so does ILOG?

As you may already know, IBM complemented its CEP/BEP offering with the AptSoft acquisition back in January 2008.
And they got SMILE continuous query processing research going on, and integration into their data grid WebSphere eXtreme Scale / ObjectGrid offering.
And now they got ILOG.
So what's the strategy around?

As with all IBM products, software downloads for trial and comprehensive documentation is available online, and you will quickly notice that WebSphere Business Events (v6) requires a good old database. This is where intermediate state will be kept and where correlation will occur.
Interestingly, CEP emerged out a wider trend that seeks to revolutionize the way we manage data (remember we sometime refered to Active Database in the academic work that precedes current CEP product offering) with a shift from "store then query" to "query as it comes" - or in other words, store the queries and flow data thru aka "continuous queries".

This quickly makes me claim that IBM WebSphere Business Events is lacking a big piece of the CEP platform puzzle, and likely why IBM is willing to complement this with another solution for more troughput/latency sensitive use cases with its InfoSphere Streams vision. Indeed, for that one, you'll have to wait late 2010 as explained.

So now should we assume that Esper will take part in an IBM offering sometime down the road? Or should we assume that Esper got it so right that Esper materials end up in an IBM "what is WebSphere Business Events" slide?

Here are some surprising hints. The nice "Advanced Event Processing" one is Esper, as described in the EsperTech site for more than 2 years. It appears in a slide deck from IBM Pulse 2008 partner conference.




(disclaimer: Up until now, WebSphere Business Events v6 is not built upon Esper)

To add to IBM offering description around CEP or business event as they call it, you'll notice that WebSphere eXtreme Scale (former IBM ObjectGrid) has now integrated the IBM Smile research in it thus exposing some IBM made SQL like event processing language (similar to Esper and Coral8 approach). This approach is different from the WebSphere Business Event product although it largely overlaps in what it does.

CREATE VIEW stockView AS SELECT tickerSymbol, avg(price) as avgPrice FROM
(SELECT * FROM stockStream FETCH LATEST 5 MINUTES) group by tickerSymbol

There are some interesting tutorials available, and I think this part of IBM offering would deserve a lot more exposure in the CEP community.

Instead of that, IBM disparate vision around event processing just added ILOG to the cake as per this just released press coverage:
"IBM Completes ILOG Acquisition, Opens Door To Event Processing"
But I am not learning you anything since on IBM owns terms:
"None of these products amounted to a real strategy. We didn't have tight engineering, focused on business event management".

It will be interesting to see where that ends...