The 1 million events per second CEP

BEA has finally published some paper on WebLogic Event Server performance.
It is available here (registration required).

They finally jump in the vendor dance of the 1 million events per second ESP/CEP claim that I had blogged about before here, thus catching up with EsperTech' Esper and all others - unfortunately only as far as these performance claims mean.

What is nice is that BEA provides details on the actual scenario, continuous queries and hardware used although the kit is not available for one to run on his own. To date Esper is still the only one to be completely open in the CEP performance area with full disclosure of results but also actual performance kit source code (see here).

Giving it a quick read, here is what I can sum up:




(both use a one thread per connection model to avoid context switching)

So obviously the BEA Event Server performs well, no doubt on that. This said there are a few immediate comments:

• The Esper benchmark somewhat set the stage, with a platform more than 4 times smaller (all ratio preserved, Esper appears almost 2.5 times faster though we cannot really compare things).
• The Esper benchmark EPL statements are more complex, especially regarding memory profile: they keep sliding windows of 1000 events where the BEA benchmark keeps 2 or 3 events.
• The Esper benchmark runs with more statements (1000 instead of 400). This is interesting as running more statement likely means less statement-level contention - so I am wondering why BEA is not running more statements here.
• The BEA benchmark does not disclose CPU usage, and for some reason the average latency is high compared to the 99.99% latency (ie the worst case latency must be high). There are probably differences in the way the two benchmarks actually measure latency (total run, post warm up, snapshots etc).
  

Unfortunately, I haven't a quad CPU quad core Intel Caneland box in my lab to play further with this. This being said, the two benchmarks are completely meaningless as the use cases are highly specifics and narrowed down, so you should really read all this as vendor standard claims, or run your own benchmark for your own business meaningful scenario.

Welcome BEA in the 1 million events per second CEP category, and kudos to them for disclosing (almost) all the details.

I also encourage you to read what Opher Etzion from IBM say on this, or also read what StreamBase published recently on their own benchmark to understand why all those claims are... really just claims up until the point you do your own evaluation!

on GPL and dual licensing

db4o has just published a paper written by a law firm Greenberg Traurig on GPL and dual licensing. This is core of their community and commercial business model, just as it is for Esper and EsperTech.

That is one of the best and most of to date resources on GPL and dual licensing I have ever read, and I 'd encourage anyone working and using GPL code to read and understand it. It also does cover GPL and hosted projects (Application Service Provider), work for hire and distributor (of course) situation.

You may read there: You may hear people argue about whether the GPL is a contract or not. The important thing to understand is that whether it is or not, if you accept code under GPL terms you must abide by the GPL. If you breach the GPL, your rights under GPL license terminate, and you have no right to use the code at all. If this happens, you will be infringing the licensor’s copyright, and copyright infringement puts you at risk of civil damages or even criminal culpability with the possibility of heavy fines.


Read it from there or PDF here directly from db4o website.

If you want to read further on GPL and dual licensing, the following law blog is quite interesting as well.

Price per CPU is foolish and price per core reaches its end

I have been scratching my head recently on software vendors policies regarding pricing and multi core pricing. The game is completely foolish and hopefully will reach its end soon, as more and more vendors will show up pricing per user to best serve the SaaS model of their customers, or pricing per instance (read JVM instance, or software server instance in a generic fashion).

Lets give it some background first and take the example of a small software vendor selling a distributed in-memory cache - aka Tangosol now owned by Oracle. Up until very recently before the Oracle acquisition, Tangosol had public price on its website, on a "per CPU" (in clear text) basis.
In August 2007 they shipped a minor version of their software (now owned by Oracle) and announced a new price of about twice more expensive. I dare to raise the issue and Cameron (Tangosol former CEO now at Oracle) kindly dare to answer it: Tangosol was priced per core and not per CPU despite advertised as per CPU on their web site... Due to Oracle policies regarding CPU and core pricing, they had to go thru this price change.
Read it here.
(no offense to Tangosol, it's great stuff and I am glad they clarified this).

Conclusion 1: Don't trust advertised per CPU prices. Anyone advertising per CPU price without disclosing complete details regarding CPU and core policies is lying to defer the price debate, and brings complexity in a TCO comparison if they are in a competitive situation.

Next step, let's recap on vendor policies regarding CPU and cores...

You already know that you cannot buy single core x86 (Intel/AMD) chips anymore, that the norm is dual core, and that quad core per chip (read per socket on the mother board, with 1 socket = 1 CPU) will be the default in a 6 months timeframe (go check on dell.com, you can already buy quad CPU quad core servers).

(disclaimer: this data is from searching in public sources):
For Oracle, you can read that each core accounts for either 0.5 or 0.75 CPU depending if it is a total dual core or quad core. So a double core costs 1.5 CPU units and a quad core costs 2 CPU units.
Read it here and here.
A quad CPU quad core costs 8 CPU units at Oracle.

For BEA, you can read that each core accounts for 0.25 CPUstarting at the 3rd core in a single CPU. So a dual core costs 1 CPU unit and a quad core costs 1.5 CPU units.
Read details here.
A quad CPU quad core costs 6 CPU units at BEA.

For IBM, they switched to a proprietary defined Processor Value Unit that is assumed to better assess various processor architectures. In fact this is quite similar to what Oracle does if you look here and on IBM site here.
A quad CPU quad core costs 8 CPU units at IBM.

For RedHat / JBoss (this is a subscription based business model but still), you can read half about it on RedHat site here and look at the JBoss TCO calculator here.
It works thru 4 CPU packs, but I could not find any information on what their CPU means... so let's exclude it for now...

All in all, you can get a 25% price difference just by considering this CPU unit. Oracle can tell you its stuff is 25% cheaper compared to BEA, you'll give them exactly the same amount of money in the end on a quad CPU quad core hardware.

Fair enough you said ??

Lets consider a few points:
- do the business project people and admins know how many cores total they will need to run on, currently run on?
- do they instead perfectly know how many server instances and apps they have to start stop operate and monitor?
- should your software license price be directly impacted if an application increases or decreases in overall throughput because business project owners asked this and that new feature to be added?
- did you already over-provisionned your hardware platforms to hide this fact?
- did you took time to ask the vendor policies regarding virtualisation? What if you turn the 4 CPU quad core box in 8 virtual 4 (virtual) CPU ""quad core""(they are virtualized now right?) system to maximize its utilization?
- do you have a virtualization or resource brokering project (read dynamic intelligent real time throw in on-demand adaptive capacity) and you are close to the day you'll click a button to move an app instance running on this old 2 CPU dual core to your new 4 CPU quad core hardware when it is on peak load (heard about VMotion and XenMotion already right?) and with no downtime. Did you thought on how you'll be able to optimize or even plan your per CPU license costs in such a scheme?

It's obvious a per instance model (per JVM instance, per server software instance) brings simple answers to all these questions and is perfectly aligned with what IT and project people are held accountable for running real world apps.

The problem for switchers (BEA, IBM, Oracle and all /CPU vendor) is that it will lead them to fairly complex discussions with their switching customers, and the best exit is likely to start selling a (time limited) enterprise wide (project limited) licenses that does not account anything anymore (until you realize it - hey it's a sell/buy game!). The pain the vendor put on you to manage and account all these licenses even becomes part of its value proposition.
... up until a billback system is put in place to enable a pay for what you used model fully aligned on your own business running costs and ROI targets.

Esper performance: more than 500 000 event/second

The Esper team has published performance results for its Esper event processing engine.

"Esper exceeds over 500 000 event/s on a dual CPU 2GHz Intel based hardware, with engine latency below 3 microseconds average (below 10us with more than 99% predictability) on a VWAP benchmark with 1000 statements registered in the system - this tops at 70 Mbit/s at 85% CPU usage."
Read more starting from the doc and the Esper wiki.

I have been working on this as I basically wrote the performance kit that is available to anyone under a GPL license, and whose client side (which is not Esper specific) can also be used to test other solutions - and I am glad to see those results.

ESP/CEP has been a common place for vague information around performance in a mine is bigger than yours attitude from all vendors to date (no one ever published publicly a detailled benchmark). As almost everyone claims hundreds of thousands of event per second, this just shows Esper is enterprise grade ready and capable of the same kind of performance results.

In the meantime there are a few extra comments to draw from this
- the benchmark app includes a NIO/TCP based Esper event server that can also make use of a staged (read highly concurrent and multithreaded) architecture. This shows how easy it is to set up something like that with Esper. The code is about 5 classes.
- for some really odd reasons, BEA WebLogic Event Server, that does make use of Esper as announced here, was said to top at only 50 000 event/second ie 10x slower (read here). I wish someone from BEA would shed some more light on this.

Coral8 and IBM - how does it compares to Esper (Tech) and BEA agreement?

Tim has commented about the Coral8 announcement in his blog.
Beehind the announcement, the fact that Coral8 and IBM have an agreement in place in the Event Processing arena, is not really comparable to what exists between Esper(Tech) and BEA for the WebLogic Event Server (I had blogged about it here).

The fundamental differences are:
- IBM WebSphere RFID includes an adapter to the Coral8 Event Processing engine. It is not an OEM agreement such as between EsperTech and BEA.
- The agreement is limited to one product in one vertical marke - while the EsperTech and BEA agreement is not tied to any vertical market in particular.
- It's unclear what does "Coral8 provides a free deployment license to all IBM WebSphere RFID customer" means ... you likely need to consider support, size of target deployment platform, pre-production environnement etc. Where free ends?

This said, congrats to Coral8 team for that. The Event Processing market has much to win from alliances like that one that help brings Event Processing to mainstream.
In the meantime I am wondering how hard would it be to have an Esper adapter for IBM WebSphere RFID ;-)

This is my new blog

I finally decided to move by blog here on blogger.
My old blog at http://blogs.codehaus.org/people/avasseur/ was not fun enough to edit and customize.

I have imported all my old posts in this blog although for the oldest ones, the comments are closed.

BEA WebLogic (Esper?) Event Server

There has been quite a lot of rumors these last weeks regarding the new BEA WebLogic Event Server which set the date of the entry of BEA into the Complex Event Processing arena.

As official facts and news can now be read by everyone, I thought it was a good time for a recap and for a few comments and thoughts. You may find this post quite long to read, but well ... this is my industry viewpoint!

(1) Facts

Rumors were all at rage trying to figure out if BEA "ripped off" the Esper ESP/CEP open-source GPL licensed engine, which has been in the arena for more than two years, release after release, commercially supported by EsperTech - hence the title of an hypothetical BEA WebLogic (Esper?) Event Server of this post.

Now we know, BEA is using Esper to bring to the market its WebLogic Event Server and has an agreement in place with EsperTech, the dual licensing company behind Esper and NEsper.

Now is time to shed some light on that topic. But first, let's recap...

(1 continued) Facts

So it first started on InfoQ with the public beta announcement of BEA product. Guillaume from Groovy fame immediately wondered if Esper is used under the hood.
Read on on InfoQ

Some weeks after, Coral8 did the same analysis and it seriously tried to lower the BEA value proposition - which is sort of expected from a BEA competitor and Esper competitor:

"all BEA did was OEM an open-source product called Esper and put it inside their product"

This forced BEA marketing folks to answer in the same article with an interesting quote, shedding some light on the BEA and Esper / EsperTech agreement.

Read it on Intelligent Enterprise

The same news was discussed again by Philip Howard, from Bloor Research, who has been following the Complex Event Processing space for a while, in an article where he describes some more the BEA Time and Event Driven (TED) product family that includes this rumored WebLogic Event Server.

"There has been some speculation that TED is based around the open source Esper event processing engine in part (...)"
Read it on IT Director

Finally, right after the BEA product went out from beta to be generally available truth as finally been spread by EsperTech - the company that provides support, services and OEM licenses around Esper and NEsper (its .Net counterpart) which was founded by Esper' founder:

Read the EsperTech press release.

(2) So how far Esper and BEA WebLogic Event Server are different?

At that stage all those rumors and this finally official press release don't shed light on how far Esper is modified by BEA as it seems to be the case, to make it his own. I'll certainly give a shot to this. For now here are some findings:

Esper: select avg(price) from Stock:win.time(1 min)
BEA : select avg(price) from Stock retain 1 min
Esper: every (A -> B)
BEA : every (A followed by B)

Part of the documentation is the same f.e. for explaining the core capabilities:
Esper doc
WebLogic Event Server doc
Or for providing a number of use cases:
Market data Feed Monitor Case Study
Use cases

(3) Open questions

At that stage, this raises a few interesting points and speculations going further:

• Esper is at version 1.10. There are tremendous improvements brought versions after versions such as deterministic behavior with statement pipelines, performance improvement, bug fixes, extended APIs and so on.


• How the BEA product will catch up if they modify their own version in various ways thru refactoring and such while Esper keeps being improved?


• How the two products will distinguish each others going further, what will make each one attractive to who? Low price point, a la carte, higher price point , same ESP/CEP capabilities, off the shelf experience ... all interact. Will we see open-source stacking up an Esper Event Server as well?


• Esper has the very same client API and event processing language that its counterpart NEsper. This means Esper / NEsper / EsperTech is the only one to give choice with same semantics across platforms. Will BEA move to .Net as well?


• Esper GPL license explains why BEA had to go over a dedicated agreement with EsperTech so as to redistribute the product. This proof points the dual licensing model and shows that a GPL/LGPL based open source/dual license is one among the many viable open source model, especially when geared toward both OEMs and end-users.


• Will open source purists argue this means BEA is using open source and contributing back on some areas (those very one would argue they are already commoditized by far), but that BEA is also ready to buy open source to get the right to tailor it without giving much back so as to build a competitive advantage? On the other hand, other will argue GPL or LGPL is not business friendly (if you think you can get for free into a bar and also get free beer then you will). From there this starts to make sense to buy a foundation from which you can build a competitive advantage in your journey - does it?


• But does this means Oracle or IBM or RedHat or IONA or SUN could do the same and enter an agreement with Esper? Those big ones have yet to enter the Complex Event Processing space beyond announcements or lab projects. New startups are poping up all around at the same time so decision is far from simple, especially as BEA moved first around such a blend.


• Does this means that BEA decided to leverage Esper and acquire a license instead of spending 20 to 100 M$ in acquiring one of the other startup in this field (or entering an OEM agreement with one of them)? We can assume BEA has done carreful due diligence, both on the IP, the technical excellence and the extensibility options of Esper before taking such a decision.

No doubt everyone will have various rumbling about all this!

(4) Esper provides choice, BEA provides off the shelf

Most important to recap, the BEA WebLogic Event Server is not good or bad in its way it does (or don't) interacts / extends the Esper project and the open source community around it depending who you ask. What is striking is that:

• The ESP/CEP innovation came from Esper and open source in the first place (as far as Esper goes, of course ESP/CEP come a long way in academic works before Esper and at other startups). The full Java ESP/CEP engine around a POJO centric model and open API, plus the real convergence between ESP and CEP capabilities was introduced by Esper and truly driven by the community around it. And now it is also available for .Net with NEsper.



• BEA WebLogic Event Server adds a server model, with deployment capabilities, Spring wiring, security, real time JRockit JVM integration, all based on a very innovative OSGi based lightweight kernel build upon other open source blocks (Eclipse Equinox f.e. for the OSGi core). The open source and commercial blend goes far beyond Esper (Jetty and Spring are in there). You may likely find more open source LOCs than BEA LOCs in this product if you were to count everything - simply because this is how modern software is build. Would you rather build it, buy it or license things and blend it instead?


• BEA offering shines because it is an off the shelf platform with its own server component which is not a JEE app server but a completely new one. How would it compare f.e. to a classic WebLogic Server (the JEE one) with Esper hooked a JMS consumer/producer, both in terms of features but also investment and learning curve?


• Esper is obviously the best option to integrate into other BEA technologies such as WebLogic Server (thru JMS as I said), but also if you are not a BEA shop, running application server / ESB from open source or big vendors. It also brings affordable ESP/CPE compared to other startup-high-license-cost in this rather new market. It also has a significantly increased flexibility as there is no target server model. You can also get support from the source through EsperTech (professional open source a la JBoss). And if you want real-time capabilities, you can run it on a real time JVM, from BEA, but also from Sun or IBM, or even inside an Azul box. This is a la carte event processing as Tom explains in the press release. A number of analysts have already started to argue around the fact that a la carte event processing may make more sense for the industry so I am sure we'll see more of that blend.

Congrats to Esper and EsperTech for their success with BEA, and congrats to BEA for this new product offering that truly shakes the complex event processing arena - perhaps not in what it does but more in how it was assembled and engineered - with a best of breed approach all the way up.

As I was just about to publish this, another news came out:
SL Corporation Helps BEA Extend Real-time Monitoring and Visualization of WebLogic Event Server. This increases the feature set of WebLogic Event Server, by a new blend. Interestingly, Progress Apama (another Esper and now BEA competitor), is using just the same as announced here and already mentioned by Tim Bass, from Tibco (did I said Tibco also plays in the ESP/CEP space and has a very close relationship with SL Corporation (see here). That said this does not surface yet in WebLogic Event Server 2.0 final release.

Interesting and exciting times in the software industry isn't it? Who said it is build vs buy? You need to think build vs buy vs license vs assemble.

Esper brings Complex Event Processing (CEP) to mainstream

There has been two interesting posts recently regarding Esper.

One from Tim Bass, from Tibco, reporting from the InformationSecurityAsia2007.
Tim reports in his blog that complex event processing solutions like Esper can be used for extrusion detection (that is inverse of intrusion detection in a network where you want to track malicious users and software acting from inside, zombi machines etc.).
Tim reports Esper is already known from several experts in the field and is beeing considered for such use cases. Glad to hear and thanks Tim for the info.

The second interesting news appears in Intelligent Enterprise where Esper is beeing quoted as the only active open-source project for CEP - both for Java and .Net platforms (what, you never heard about NEsper? Bet you will !).

Esper is quoted separately from the so called more commercial solutions like Apama, StreamBase, Coral8, Tibco etc. Especially because Esper is a great initiative to bring CEP to mainstream thru innovative, affordable and a la carte approach as quoted here:


Not For Big Companies Only.
With the ranks of CEP practitioners including big government, big finance and big telco, you might think the technology is accessible only to giants . That's not the case, however, as there's an active open-source CEP project called Esper that offers both Java and .NET components.


It 's nice to see Esper quoted, although I am not sure I would do such a difference between Esper and the others solutions available for the following reasons:

• Esper is commercially backed and supported - checkout Esper' founder company EsperTech


• Esper users include major investment banks already and other big shops - checkout the mailing list if you want to get a taste

Of course I can't say if those guys are evaluating Esper or if they run some of their production system with it already - so you never know who is a user evaluating software and who is ready / has already entered the commercial relationship with commercially backed open source software unless you ask and they dare to reveal it.

There's trully no difference in fact when you look at that from a community point of view.
When you are building open source software (which I have been doing for years now) there is only one thing that matters: the community.
The community rules the use cases and drives the product direction in unanticipated ways accross industries of all kind and size.

This is a typical mistake regularly done when trying to compare open source software and commercial software as two fundamentally different things. Would you start saying Linux is for small companies and Windows for the big ones...

Event Driven Application Servers - BEA is in

BEA has started to spread the news about the fact they'll enter the Event Driven Application Server area - which I covered during my JavaOne 2007 session (with a bias on the first and only Java and .Net container for event-driven applications aka (N)Esper). The product name is WebLogic Event Server 2.0 (go figure what happened to 1.0!?), slatted for a public beta in the next few days.

Tim from Tibco wraps up the news in his blog.

Stay tuned.

Esper at JavaOne, on Event Driven Application Server and XTP

I am pleased to be back at this year JavaOne 2007 as a speaker on "Event-Driven Application Servers". I'll showcase ESP/CEP technology to the Java community and industry, San Francisco, on Friday May 11th, 12:10.

The session that I co-lead with Tom, Esper and EsperTech founder, will introduce the audience with Event Driven Architectures (EDA), Event Stream Processing (ESP) and Complex Event Processing (CEP). We will showcase the why, what and how on event-driven application servers, going from simple integration over JEE platforms to tailored off-the-shelf leading edge solutions playing in the eXtreme Transaction Processing (XTP) field.

This has implication in various areas such as investment banking, RFID, monitoring, compliance management and many more (up to this good old SOA...).

If you attend the event or are in the bay area and want to meet up, ping me or leave me a comment!

The session id is TS-1911 and if you happen to attend JavaOne this year you can already add it to your session schedule or click below to connect with me for the event.



I hope to see you there and see a whole bunch of long time friends there.

Esper
JavaOne 2007

Esper tutorial on OReilly ONJava

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

It's live!

Event-driven architectures turn a traditional data-driven application's architecture upside-down. Instead of storing the data and running queries against stored data, Esper allows applications to store queries and run the data through.

This article introduces Esper, a lightweight ESP/CEP engine written in Java.

This Esper tutorial is now available online at OReilly ' ONJava. The tutorial illustrates an airport self-service terminal that we monitor and improve with real-time customer relationship management by using both ESP and CEP capabilities of Esper.

It includes a fully functional yet simple to run bundled application and also includes Eclipse configuration files so that you can start practicing right away (if you are an Eclipse user...).

I hope this will help get those blooming ESP/CEP concepts to mainstream. The tutorial should give you enough ground to understand where we come from and enough details to get started in your own project right away.

If the EPL snips below does not sound familiar to you

select avg(price) from StockTick.win:time(30 sec)
where symbol='GOOG'

select type, count(*)
from BaseTerminalEvent.win:time(10 minutes)
group by type
output all every 1 minutes


Read it online at ONJava

or read more about Esper

Terracotta' Jonas & Eugene on AOP at AOSD 2007

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Jonas and Eugene from Terracotta - the Naturally Clustered Java provider and maker of OpenTerracotta - have published a very interesting paper on industry use of AOP and point out some limitations present in current AOP frameworks: "Clustering the Java Virtual Machine using Aspect-Oriented Programming".

Excerpt:
"application startup time with AspectJ load-time weaving [which I partly authored] was ... slower and memory overhead was ... bigger ... when comparing with similar transformations done with either the Terracotta runtime or the AspectWerkz AOP engine [which I authored with my friend and former coworker Jonas]."

This comes to no surprise to me as AspectJ is born for build time weaving and Eclipse integration and AspectWerkz was born for load time and runtime weaving. As author of both (as we joint forces from AspectWerkz to AspectJ back mhh.. 2 years ago) I was unable to get the best of both world in one engine as we initially focussed on features integration - such as thru the @AspectJ style that is annotation-defined and driven AOP. Since that time my open source bandwith has been drawned due to work engagements. That is unfortunate as I believe there are no technical reasons for this multi-weaving-optimized engine to not happen. Fortunately AspectJ is strong on runtime performance which is something you really have to consider as well.

Their paper is available online on AOSD 2007