vFabric (ex SpringSource) on VMware vCloud for a cloud scale app

Since I joined VMware almost 1y ago, I had little time to update my blog on what 's going on. I have a whole lot more for this year and I hope you'll find it of interest.

For this come back, I'll start a whole series of posts all tagged USI2011.

Cloud... let's get concrete please!

We just helped launched a cloud scale application challenge with 2 partners around VMware: Steria for the hosting part and Octo driving the challenge - a major consulting boutique of high level yet real solid architecture skills especially when it comes to the topics at stake - applications and clouds.

The challenge is opened to dev/devops/architect skilled teams in France and is already oversubscribed and got good buzz (here, here, and twitter #USI2011).

See the complete flyer.

In few days and for about 4 months you'll have access to VMware vCloud Director over the internet to run and test drive your platform, running atop a nice set of blades operated by Steria (we'll disclose details later), and prefilled with empty linux or better the entire portfolio of vFabric cloud application platform (it 's all your choice to use it).

If we leave Zimbra and Desktop virtualization solutions we have a great combination of what VMware cooked in 2010 - following the SpringSource, GemStone, and RabbitMQ acquisitions - now called vFabric - and combining this with the vCloud Director who went publicly generally available in second half of 2010. Awesome combination of technologies, proven for years in the field from virtualization to Java runtimes and coming to you over the cloud through vCloud!

The whole goal is to write a scalable application - and we mean large scale.

We want to observe how you deal with few issues especially when running "in a cloud" - in the 2011+ years. We want to see the solution you bring to the table - use vCloud, use vFabric (or whatever you'd like if you can sustain a discussion to a CIO on why it's the best platform for his apps), and get back to us with real code and results.

There has been a number of discussion in the twitter and blogosphere (again in France) of small tweaks enabling key differences in raw performance (indeed there might be, going back to tuning an OS TCP stack) but be aware we'll also carefully look at your cloud capabilities:

• technologies being used are rather mainstream, digestible or plain darkness?
• how do you deal with provisioning and management at scale?
• are you elastic, from a compute and data perspective
• how fast can you scale, does your scale out mechanism integrates with "add VM" in vCloud and also reduces load from existing nodes and not simply add new capacity waiting for new requests
• are you HA and no-SPOF (it is an overkilled term but if you have elascticity you may have scale-down hence requirement for nodes being droped fairly often)
• and the list goes on...

(disclaimer: I am not the judge - those are some thoughts)

If you want to read more (and read French) - go have a look at

• Octo' challenge announcement "Et si vous codiez une application qui supporte 1 milliard d’utilisateurs ?"
• Octo blog on the challenge and thoughts on how you will be judged
• Université du SI 2011 Octo' conference where we'll debrief and showcase about this challenge
• and of course the vFabric and vCloud Director from VMware directly

For the next post, I'll uncover a bit more with pics and videos what it looks like to run vFabric in the cloud with vCloud Director

Moving on ... to the Cloud

As wrapped up yesterday, I have left BEA now Oracle after 6+ great years there.
I am joining VMware to help adoption of VMware Platform-as-a-Service product line - especially across EMEA.
I'll be part of the SpringSource division, acquired by VMware back August 2009 for 400+M$.

This is a fantastic time to join the team - and I am glad that I can build upon ALL what I have been working on those past years - both technical and sales side - as well as learn and drive a massive amount of new things.
Here is the way I see it:

VMware is at the forefront of fundamental yet evolutionary shifts in the way we build, run and manage enterprise applications and deal with computing resources and services.

I'll hopefully be able to write more on that journey in the next few posts.

Moving on

Those last 6 years that I have been working at BEA now Oracle have been fantastic. I did had a great run there, in various positions both engineering and sales side.

It's time to recap a bit.

I have been a bit of:
- technical-crazy enough to discuss open source AOP gory details with then JBoss CTO on ServerSide, and to also compile a JRockit JVM on my own for our AOP-in-the-JVM prototype derived from our AspectWerkz work (2004-2006),
- lucky enough to evangelize emerging trends such as AOP, POJO and lightweight containers across the world back when Java(TM) was still a Sun trademark,
- unlucky yet professional enough to work with fantastic people who ended up being successful faster than me onto there I21/SpringSource/VMware journey
- and business-crazy enough to end up in the category of software sales, in fact getting our largest customers to adopt our products - from WLS to emerging SIP and SipServlet, real time Java, CEP containers, up to bare metal Java running straight onto hypervisors such as VMware (a number of those RIP or rescheduled as part of BEA now Oracle - as you can figure out) and last but not least data grids.

Along that path I made sure to revitalized my open source hat by giving time and knowledge to bootstrap a niche open source project to a sustainable pure-play open core CEP Java leader aka Esper & EsperTech. A fantastic geek+BizDev mix with great talented individuals all the way along. If you wonder, Esper / EsperTech is playing the open-core business model in a fairly interesting way.

I am now onto something next - and I already feel home like always been there - more on that next one soon.

Esper - what and why in Complex Event Processing

I have been generating this definition out of the word cloud image generated from Esper documentation:

An engine to query streams of events using select statements over time windows and pattern expressions.
(by streams of events you need to think continuous queries vs traditional store and later query analytics)

Esper is actually much more capable - especially in the just released Esper 3.2, that for example brings support for a subset of nascent not finalized ANSI SQL standard match_recognize.
The Esper 3.2 documentation has a nice summary on how match_recognize compares to Esper well established select from patttern[ ... ] and followed-by EPL language features.

It is also great to see EsperTech on the Forrester Wave Complex Event Processing Platforms, Q3 2009: "Forrester evaluated nine complex event processing (CEP) platforms using 114 criteria (...). EsperTech, the only open source option, is also a Strong Performer."

Esper, OSGi and Complex Event Processing

There is an interesting article that discusses the pros and cons of integrating OSGi with Esper to achieve modularity and dynamicity beyond what Esper already provides.

The article goes over comparing 3 scenarios and comes with sample code (and complete source, with Maven and Eclipse artifacts)

1. Exposing Esper runtime and administrative interfaces as OSGi services
2. Layering Esper interfaces as OSGi services through some service provider abstraction
3. Using Esper and the OSGi EventAdmin service specification
   

This third approach although a bit more complex for readers without OSGi background is the recommended one by the author - Valery Abu-Eid.

It is nice to see such exploration going on. I know we already have users running Esper in OSGi environment, although the vast majority is still running on bare JSE JVM or on JEE / servlet container stacks.

Readers familiar with the matter will also notice that WebLogic Event Server (now Oracle CEP) - is also leveraging OSGi. A main difference is that Esper does not mandate OSGi at all while the BEA/Oracle product does mandate end users to author OSGi modules - with pros and cons.

If you have experimented with similar stacks I'd be pleased to hear about your experience.
There is also good material available from the DynamicJava.org site to get started with OSGi based runtimes - including their own OSGi stack.

Read more about Esper and OSGi experiments at DynamicJava.org