CEP and JDBC/ODBC for ad-hoc queries on real-time data

Sounds like EsperJDBC and Aleri JDBC/ODBC capabilities have found some echo. Coral8 just announced interoperability with ODBC with its 5.6 minor release - 3 months after EsperJDBC introduction.
The Coral8 documentation states that Coral8 now can expose public windows (equivalent of Esper named windows) over ODBC and not just only with a proprietary SDK.

If you want to explore this concept with Esper - this slide deck is a must read:

EsperJDBC

View more presentations from espertech. (tags: esper cep)

A demo also demonstrates how you can perform ad-hoc queries on your CEP real-time data and computations using any JDBC compliant tool - namely Actuate / Eclipse BIRT and create dashboards in minutes.

(Note: EsperJDBC is not open source and only available in EsperTech' commercial offering although you can access free trials)

(Note: Post updated as per Jeff comment on Aleri)

IBM WebSphere Business Events - going Esper? And WebSphere eXtreme Scale going on his own? And so does ILOG?

As you may already know, IBM complemented its CEP/BEP offering with the AptSoft acquisition back in January 2008.
And they got SMILE continuous query processing research going on, and integration into their data grid WebSphere eXtreme Scale / ObjectGrid offering.
And now they got ILOG.
So what's the strategy around?

As with all IBM products, software downloads for trial and comprehensive documentation is available online, and you will quickly notice that WebSphere Business Events (v6) requires a good old database. This is where intermediate state will be kept and where correlation will occur.
Interestingly, CEP emerged out a wider trend that seeks to revolutionize the way we manage data (remember we sometime refered to Active Database in the academic work that precedes current CEP product offering) with a shift from "store then query" to "query as it comes" - or in other words, store the queries and flow data thru aka "continuous queries".

This quickly makes me claim that IBM WebSphere Business Events is lacking a big piece of the CEP platform puzzle, and likely why IBM is willing to complement this with another solution for more troughput/latency sensitive use cases with its InfoSphere Streams vision. Indeed, for that one, you'll have to wait late 2010 as explained.

So now should we assume that Esper will take part in an IBM offering sometime down the road? Or should we assume that Esper got it so right that Esper materials end up in an IBM "what is WebSphere Business Events" slide?

Here are some surprising hints. The nice "Advanced Event Processing" one is Esper, as described in the EsperTech site for more than 2 years. It appears in a slide deck from IBM Pulse 2008 partner conference.




(disclaimer: Up until now, WebSphere Business Events v6 is not built upon Esper)

To add to IBM offering description around CEP or business event as they call it, you'll notice that WebSphere eXtreme Scale (former IBM ObjectGrid) has now integrated the IBM Smile research in it thus exposing some IBM made SQL like event processing language (similar to Esper and Coral8 approach). This approach is different from the WebSphere Business Event product although it largely overlaps in what it does.

CREATE VIEW stockView AS SELECT tickerSymbol, avg(price) as avgPrice FROM
(SELECT * FROM stockStream FETCH LATEST 5 MINUTES) group by tickerSymbol

There are some interesting tutorials available, and I think this part of IBM offering would deserve a lot more exposure in the CEP community.

Instead of that, IBM disparate vision around event processing just added ILOG to the cake as per this just released press coverage:
"IBM Completes ILOG Acquisition, Opens Door To Event Processing"
But I am not learning you anything since on IBM owns terms:
"None of these products amounted to a real strategy. We didn't have tight engineering, focused on business event management".

It will be interesting to see where that ends...

CEP and patents

I was recently updating my knowledge base around event processing, CEP and DSMS to have a better view on the current state of the art regarding patent applications around those topics, when I found a fairly recent pending application from Oracle named:

"Processing xml data stream(s) using continuous queries in a data stream management system".
The patent application is available online

It was filed in Nov. 17, 2006, published in May 22, 2008, and also claims to be related to a DBMS SQL/XML related patent filed in 2005 (is that an attempt to claim for anteriority?).

As far as I can sum it up, it organizes its claim around three fundamental findings:
(usual disclaimer here - this is my own view etc)

1. CQL or SQL-alike continuous event processing languages fall short on dealing with structured data streams
   
2. especially XML
3. both as input event stream and as output event streams (out of the continuous query).

f.e. support for
SELECT RStream(count(*)) FROM StockTradeXMLStream AS
sx [RANGE 1 Hour SLIDES 5 minutes] WHERE XMLExists( ‘/TradeRecord[TradeSymbol = “ORCL” and
TradePrice >= 14.00 and TradePrice <= 16.00]’ PASSING VALUE(sx))

I was a bit surprised by this patent because there is an interesting positionning dilemna. Oracle has publicly advertised its wishes to standardize CQL in some ways, and this patent is now an advertised intent to lock a subset of it.

• Is it a good way toward standardization consensus?: I don't think so - it blurs your motivation for standardization ("leave XML out, it's patented - more value in my own product and more FUD to yours")
• Is it a good way to quickly empower the user? I don't think so - more locking deep down to the language (remember all the messy ANSI SQL derivatives right?)
• Is it a good way to innovate? I don't think so - bring it to end users and call for participation

As to know if there is anteriority or not - let's list some of XML support in different DSMS / CEP products:

• Coral8, back in Sept. 2006, support the same XMLAgg, XMLTable and alike - see also their XML cookbook
• Esper, back in August 2006, supports for XML events. The Esper EPL actually does a very nice job at not exposing XPath or XML in the EPL and automates the XPath plumbing - so that the EPL is actually event representation agnostic. I think that is a superior design
  
• As a generalization, Esper, since day 1 (somewhere in 2005), supports Object structured event streams - which is a generalization of xml structured streams obviously.
• iSpheres (now at Avaya) - can't even have a date but likely 2003 or so

And I could keep the list growing...
I am not saying patents are good or evil or a bit of both - that's no the point here. But this one just strikes me for what it is. I'd be curious to hear from Coral8 folks as well as others.

SOA, EDA and CEP

There has been some coverage in the blogosphere on how SOA, EDA and CEP relate to each others.

"So CEP is not EDA, EDA is more than CEP. Promoting CEP as being EDA is far too simple. And yet that is what is happening in the current IT space."
"Especially the vendors of event processors focus too much on CEP as being EDA".

Mark Palmer from StreamBase answered some

"CEP, in fact, is a really important element of an effective EDA. It's not a required element, but it sure makes it better"

That's how I presented it at JavaOne back summer 2007, and this still happens to be true - no matter how vendors are mashing up various concepts about it to try to differentiate one from each other (CEP and Grid, CEP and Web2.0, CEP and DDS, CEP and edge processing, CEP and SOA, CEP and BRMS, CEP and XTP etc.).
Here are two excerpts from the presentation I delivered with Thomas Bernhardt from Esper/EsperTech fame:




If you talk to folks using CEP (build or buy - does not matter) in some industries such as investment banking, they'll tell you this has nothing to do with SOA but has to do with XTP.

It's time to learn from past history.

A similar debates occurred not that long time ago around the SOA and ESB terms (2005 article on ComputerWorld). For the record, back in 2001 ESB and SOA did not even existed. Yet, some visionaries such as Sonic folks introduced ESB as a COTS product, despite established or maturing MOMs presence such as TibcoRV, MQ, or JMS-based/J2EE MOMs and a whole set of in house built solutions.

With no surprise, in 2008, I'd be curious to hear someone that would disagree with the statement below, which is in fact a copy/paste/replace from comments happening right now in the blogosphere rel. CEP and EDA as stated at the beginning of this post:

An ESB is not SOA, SOA is more than ESB. Promoting ESB as being SOA is far too simple. And yet that is what happened in the early days (back in 2001).

Got it now? Here is mine bold claim:

A CEP engine is to EDA what an ESB is to SOA.
(And it's also more than just that)

Esper - course on DSMS at University of Oslo

The University of Oslo, Norway, is running for 2008 some courses on Data Stream Management Systems (DSMS) - some of it based on Esper.
DSMS was the term coined by academic work as opposition to DBMS (database) in the "early days" before the more marketed Complex Event Processing CEP acronym was introduced. CEP is now more widely used by vendors, press and analysts as productization and real world deployment is taking place.

Some course materials are available online, and it is worth the read because it recaps on the key concepts, fit for purpose, and mechanics of DSMS/CEP engines such as TelegraphCQ, Stream, Borealis and Esper.

• Introduction
• DSMS part 1 - Applications Concepts and Systems
• DSMS part 2 - Sensor Networks (less interesting unless you are in this area)
• Course catalog

There are also 2 project assignments that plan to leverage Esper - which is an excellent illustration of how open source can be leveraged in the academic world.

• Extending a DSMS benchmark
• Online analysis of medical sensor data with the Esper Event Stream Processing System

There is obviously great confidence that the Esper project will not be closed down anytime soon for monetization purpose - unlike what happened with other such DSMS projects (if not all...). The direction around Esper and enterprise open source is already crystal clear and that's definitely a win-win for both academics, community, and enterprise practitioners.