The 1 million events per second CEP

BEA has finally published some paper on WebLogic Event Server performance.
It is available here (registration required).

They finally jump in the vendor dance of the 1 million events per second ESP/CEP claim that I had blogged about before here, thus catching up with EsperTech' Esper and all others - unfortunately only as far as these performance claims mean.

What is nice is that BEA provides details on the actual scenario, continuous queries and hardware used although the kit is not available for one to run on his own. To date Esper is still the only one to be completely open in the CEP performance area with full disclosure of results but also actual performance kit source code (see here).

Giving it a quick read, here is what I can sum up:




(both use a one thread per connection model to avoid context switching)

So obviously the BEA Event Server performs well, no doubt on that. This said there are a few immediate comments:

• The Esper benchmark somewhat set the stage, with a platform more than 4 times smaller (all ratio preserved, Esper appears almost 2.5 times faster though we cannot really compare things).
• The Esper benchmark EPL statements are more complex, especially regarding memory profile: they keep sliding windows of 1000 events where the BEA benchmark keeps 2 or 3 events.
• The Esper benchmark runs with more statements (1000 instead of 400). This is interesting as running more statement likely means less statement-level contention - so I am wondering why BEA is not running more statements here.
• The BEA benchmark does not disclose CPU usage, and for some reason the average latency is high compared to the 99.99% latency (ie the worst case latency must be high). There are probably differences in the way the two benchmarks actually measure latency (total run, post warm up, snapshots etc).
  

Unfortunately, I haven't a quad CPU quad core Intel Caneland box in my lab to play further with this. This being said, the two benchmarks are completely meaningless as the use cases are highly specifics and narrowed down, so you should really read all this as vendor standard claims, or run your own benchmark for your own business meaningful scenario.

Welcome BEA in the 1 million events per second CEP category, and kudos to them for disclosing (almost) all the details.

I also encourage you to read what Opher Etzion from IBM say on this, or also read what StreamBase published recently on their own benchmark to understand why all those claims are... really just claims up until the point you do your own evaluation!