Facts and Thoughts

vFabric on vCloud - meet the vFabric cloud application platform (episode 2)

2011-02-08T22:23:00.003+01:00

In this second episode of "vFabric on vCloud" I'll introduce you to vFabric. Our end user here is an architect/developper/devops for the cloud scale application challenge that Octo runs with VMware and Steria - but the demonstration is fairly generic.

The user is having access to the VMware powered platform through vCloud Director, over the internet using its web browser.

He has a vFabric 4 VM cluster running and has access to the built-in Hyperic instance.

Using Hyperic, the user has a centralized, secured and top to bottom view on his application platform - for the entire vFabric stack. Even if the cluster is elastic with new VM coming in, Hyperic built-in self-discovery kicks in and keeps consistency in a dynamic environment.

The vFabric cloud application platform features a number of web related platform services:

tc Server - lightweight and scalable enterprise version of Tomcat for Java apps and Spring apps
ERS - an Apache based httpd that can frontend the stack or deal with non Java apps (PHP, Perl etc)
RabbitMQ - an AMQP compliant messaging broker with many languages binding and additional protocols, that can also be connected straight to browsers for web messaging
GemFire - a distributed data grid that combines replicated / partionned caching and eviction with map reduce and no-data loss / shared nothing parralel persistence capabilities
Hyperic - an agent based distributed system for performance monitoring and control of the entire stack - from bare operating system to inside-application - that provides fundamental capabilities to manage elastic environments at scale, including non-vFabric if you want.

vFabric can be used as a whole platform "better together" - with for example tc Server offloading state (session, Hibernate L2 or java caching) to GemFire, and async processing to RabbitMQ - while ERS can load balance tc Server instances.

Hyperic glue the entire distributed environment together from an operational perspective - for deployment, configuration, remote control and performance management - but you can use command line just as well.

In a next episode we will look at part 3 of the user experience - as seen from a cloud operator adding new vFabric VM to the cluster to increase compute and data capacity at deploy time or at runtime with a "clone to scale" built-in design.

vFabric on vCloud - meet the vFabric cluster vApp (episode 1)

2011-02-03T12:25:00.004+01:00

In this first episode of "vFabric on vCloud" I'll introduce you to the end user experience. Our end user here is an architect/developper/devops for the cloud scale application challenge that Octo runs with VMware and Steria - but the demonstration is fairly generic.

The user is having access to the VMware powered platform through vCloud Director, over the internet using its web browser.

He has access to a pre-provided vApp - a set of 4 vFabric VM and one utility VM that are wired on a private network and can be managed as a whole entity by the user (start, stop, suspend etc) but also by the underlying cloud provider or organization manager if needed.

Once started, the user is having access to its 5 VM over a secured SSH and can then create a tunnel to access for example the vFabric Hyperic web UI in a secured fashion even with use of default password.

In our example, the vApp has been pre-provided through a vApp template by the organization manager. It encapsulates all best practices and ensure isolation yet flexibility for a multi tier application environment.

Depending on roles we could have given the user more freedom to create his own vApp(s) based on quotas limits and catalog access where our VM and vApp templates are listed.

In a next episode we will look at part 2 of the user experience - few seconds after a vFabric VM cluster boot where all the vFabric cloud application platform gets accessible, manageeable, and dynamic and elastic.

vFabric (ex SpringSource) on VMware vCloud for a cloud scale app

2011-01-25T00:01:00.004+01:00

Since I joined VMware almost 1y ago, I had little time to update my blog on what 's going on. I have a whole lot more for this year and I hope you'll find it of interest.

For this come back, I'll start a whole series of posts all tagged USI2011.

Cloud... let's get concrete please!

We just helped launched a cloud scale application challenge with 2 partners around VMware: Steria for the hosting part and Octo driving the challenge - a major consulting boutique of high level yet real solid architecture skills especially when it comes to the topics at stake - applications and clouds.

The challenge is opened to dev/devops/architect skilled teams in France and is already oversubscribed and got good buzz (here, here, and twitter #USI2011).

See the complete flyer.

In few days and for about 4 months you'll have access to VMware vCloud Director over the internet to run and test drive your platform, running atop a nice set of blades operated by Steria (we'll disclose details later), and prefilled with empty linux or better the entire portfolio of vFabric cloud application platform (it 's all your choice to use it).

If we leave Zimbra and Desktop virtualization solutions we have a great combination of what VMware cooked in 2010 - following the SpringSource, GemStone, and RabbitMQ acquisitions - now called vFabric - and combining this with the vCloud Director who went publicly generally available in second half of 2010. Awesome combination of technologies, proven for years in the field from virtualization to Java runtimes and coming to you over the cloud through vCloud!

The whole goal is to write a scalable application - and we mean large scale.

We want to observe how you deal with few issues especially when running "in a cloud" - in the 2011+ years. We want to see the solution you bring to the table - use vCloud, use vFabric (or whatever you'd like if you can sustain a discussion to a CIO on why it's the best platform for his apps), and get back to us with real code and results.

There has been a number of discussion in the twitter and blogosphere (again in France) of small tweaks enabling key differences in raw performance (indeed there might be, going back to tuning an OS TCP stack) but be aware we'll also carefully look at your cloud capabilities:

technologies being used are rather mainstream, digestible or plain darkness?
how do you deal with provisioning and management at scale?
are you elastic, from a compute and data perspective
how fast can you scale, does your scale out mechanism integrates with "add VM" in vCloud and also reduces load from existing nodes and not simply add new capacity waiting for new requests
are you HA and no-SPOF (it is an overkilled term but if you have elascticity you may have scale-down hence requirement for nodes being droped fairly often)
and the list goes on...

(disclaimer: I am not the judge - those are some thoughts)

If you want to read more (and read French) - go have a look at

Octo' challenge announcement "Et si vous codiez une application qui supporte 1 milliard d’utilisateurs ?"
Octo blog on the challenge and thoughts on how you will be judged
Université du SI 2011 Octo' conference where we'll debrief and showcase about this challenge
and of course the vFabric and vCloud Director from VMware directly

For the next post, I'll uncover a bit more with pics and videos what it looks like to run vFabric in the cloud with vCloud Director

Moving on ... to the Cloud

2010-04-20T11:49:00.003+02:00

As wrapped up yesterday, I have left BEA now Oracle after 6+ great years there.
I am joining VMware to help adoption of VMware Platform-as-a-Service product line - especially across EMEA.
I'll be part of the SpringSource division, acquired by VMware back August 2009 for 400+M$.

This is a fantastic time to join the team - and I am glad that I can build upon ALL what I have been working on those past years - both technical and sales side - as well as learn and drive a massive amount of new things.
Here is the way I see it:

VMware is at the forefront of fundamental yet evolutionary shifts in the way we build, run and manage enterprise applications and deal with computing resources and services.

I'll hopefully be able to write more on that journey in the next few posts.

Moving on

2010-04-19T18:00:00.004+02:00

Those last 6 years that I have been working at BEA now Oracle have been fantastic. I did had a great run there, in various positions both engineering and sales side.

It's time to recap a bit.

I have been a bit of:
- technical-crazy enough to discuss open source AOP gory details with then JBoss CTO on ServerSide, and to also compile a JRockit JVM on my own for our AOP-in-the-JVM prototype derived from our AspectWerkz work (2004-2006),
- lucky enough to evangelize emerging trends such as AOP, POJO and lightweight containers across the world back when Java(TM) was still a Sun trademark,
- unlucky yet professional enough to work with fantastic people who ended up being successful faster than me onto there I21/SpringSource/VMware journey
- and business-crazy enough to end up in the category of software sales, in fact getting our largest customers to adopt our products - from WLS to emerging SIP and SipServlet, real time Java, CEP containers, up to bare metal Java running straight onto hypervisors such as VMware (a number of those RIP or rescheduled as part of BEA now Oracle - as you can figure out) and last but not least data grids.

Along that path I made sure to revitalized my open source hat by giving time and knowledge to bootstrap a niche open source project to a sustainable pure-play open core CEP Java leader aka Esper & EsperTech. A fantastic geek+BizDev mix with great talented individuals all the way along. If you wonder, Esper / EsperTech is playing the open-core business model in a fairly interesting way.

I am now onto something next - and I already feel home like always been there - more on that next one soon.

Esper - what and why in Complex Event Processing

2009-09-07T13:00:00.000+02:00

I have been generating this definition out of the word cloud image generated from Esper documentation:

An engine to query streams of events using select statements over time windows and pattern expressions.
(by streams of events you need to think continuous queries vs traditional store and later query analytics)

Esper is actually much more capable - especially in the just released Esper 3.2, that for example brings support for a subset of nascent not finalized ANSI SQL standard match_recognize.
The Esper 3.2 documentation has a nice summary on how match_recognize compares to Esper well established select from patttern[ ... ] and followed-by EPL language features.

It is also great to see EsperTech on the Forrester Wave Complex Event Processing Platforms, Q3 2009: "Forrester evaluated nine complex event processing (CEP) platforms using 114 criteria (...). EsperTech, the only open source option, is also a Strong Performer."

Esper, OSGi and Complex Event Processing

2009-06-06T10:00:00.002+02:00

There is an interesting article that discusses the pros and cons of integrating OSGi with Esper to achieve modularity and dynamicity beyond what Esper already provides.

The article goes over comparing 3 scenarios and comes with sample code (and complete source, with Maven and Eclipse artifacts)

Exposing Esper runtime and administrative interfaces as OSGi services
Layering Esper interfaces as OSGi services through some service provider abstraction
Using Esper and the OSGi EventAdmin service specification

This third approach although a bit more complex for readers without OSGi background is the recommended one by the author - Valery Abu-Eid.

It is nice to see such exploration going on. I know we already have users running Esper in OSGi environment, although the vast majority is still running on bare JSE JVM or on JEE / servlet container stacks.

Readers familiar with the matter will also notice that WebLogic Event Server (now Oracle CEP) - is also leveraging OSGi. A main difference is that Esper does not mandate OSGi at all while the BEA/Oracle product does mandate end users to author OSGi modules - with pros and cons.

If you have experimented with similar stacks I'd be pleased to hear about your experience.
There is also good material available from the DynamicJava.org site to get started with OSGi based runtimes - including their own OSGi stack.

Read more about Esper and OSGi experiments at DynamicJava.org

CEP engine or CEP platform? Or both?

2009-05-15T21:27:00.010+02:00

There has been recurrent debate and opposition between the value proposition of a CEP engine and a CEP platform.

The archetype of a CEP engine would be something like Esper. You can fully embed it in an existing runtime (Java or .NET/C# for NEsper), ranging from standalone JVMs with bare metal thread pools to JEE application servers or SOA stacks.
The archetype of a CEP platform would be something more like Streambase, or the Oracle CEP server or well you name it - where the product is a server - no choice - sometime even worse with a whole bunch of dependencies (for example a relational database and/or a mandatory JEE application server - see IBM Business Events for example).

I am firm believer that there are numerous technical advantages of getting started with Complex Event Processing using a CEP engine.
Check out the Esper examples (one integrates with JMS f.e. TibcoEMS, another with JBoss, etc) or the growing list of ISVs embedding Esper into their own solutions - from asset tracking, network security and trade platforms of all kinds.
More control and flexibility and more reuse of your existing investments.
This also fits well if IT, speed and flexibility is a competitive advantage - which for example was echoed in a recent WallStreet & Technology article:

Wall Street Opens Doors to Open Source Technologies
"Esper is already built. Instead of being a standalone server product that runs as a separate product, it's a library that you link in with your own proprietary stuff. That way you don't have to worry about another additional process." (an industry source who is involved in the development of trading systems at a major investment bank)

Needless to say, things are not black or white. I am also a firm believer that there are numerous long-run advantages of scaling up your CEP use through a CEP platform approach. Enforce shared practice for monitoring, high availability, deployment unit management and consistency across projects.
Interestingly, Esper tackles this through its (commercial) Enterprise Edition - with bells and whistles around management, tooling and high availability.
More enterprise *ilities and more wisdom long run - likely.

Middleware as a service

2009-04-30T10:34:00.006+02:00

Although my last posts have mostly been related to Complex Event Processing and the various vendors and solutions out there, I have spent some time recently exploring the IaaS and PaaS markets.
I won't provide my analysis but instead wrap up on the few hands on experience I had thus far - all promising although with vastly different perspectives for enterprise adoption.

Amazon EC2
I have been a long time user of Amazon EC2 and did a few experiment with it, like running Esper performance benchmark and ensuring it scales linearly as I kickoff Amazon AMI with more CPU and RAM capacity. Very useful although you're send back to school manual on OS-level handling, package install and being root user...

VMWare / BEA ex LiquidVM / ex VirtualEdition
I had of course been quiet familiar with VMWare over the last few years, heading BEA' WebLogic Server Virtual Edition adoption in my geography. Things happens to have slow down a bit on this chapter, although JRockit Virtual Edition is very much on the Oracle agenda as it was explained at Oracle OpenWorld 2008. I expect more this year unleashed with a wider perspective around the cloud topic as some expects.
For prior (although outdated) work on this - check out this slide deck (pdf, slow site).
Very promissing convergence between Java runtime and virtualized stack - patented by the way... (20090070776, "SYSTEM AND METHOD TO IMPROVE MEMORY USAGE IN VIRTUAL MACHINES RUNNING AS HYPERVISOR GUESTS")

Google AppEngine
Recently I have been playing with the Google AppEngine Java - very neat - build upon a few open source bits like Jetty (which also happens to be used by the BEA WebLogic Event Server despite BEA has been a long time maker of its own servlet stack in WebLogic), although the Eclipse IDE support is limited to a bare minimum and the security restriction (class white list) is time wasting in the end-developer hands to validate compatibility of existing stacks.

Is Google AppEngine scaring for IBM?
I recently came accross an IBM lab paper titled "Porting Google App Engine Applications to IBM Middleware".
It provides some API-level compatibility to port GAE java apps to a good old WebSphere / Tivoli stack but is unfortunately missing the whole point.

Middleware as a Service
The clear value proposition of GAE is middleware as a service namely:
- no software install
- out of the box load balancing and clustering
- scale up/down and out on demand
- pay per use

WebLogic on Amazon EC2 and more
From that perspective, the WebLogic Server on EC2 technical white paper and the WebLogic Server in the cloud Oracle' datasheet are far better aligned to the value proposition - although incomplete in a comparison to GAE - as one would have to add Coherence (where GAE provides memcache / JSR-107 JCache) and very likely Enterprise Manager or ex-BEA WebLogic Liquid Operation Control to deal with self-provisionning and scale on demand.

CEP and JDBC/ODBC for ad-hoc queries on real-time data

2009-02-11T13:50:00.003+01:00

Sounds like EsperJDBC and Aleri JDBC/ODBC capabilities have found some echo. Coral8 just announced interoperability with ODBC with its 5.6 minor release - 3 months after EsperJDBC introduction.
The Coral8 documentation states that Coral8 now can expose public windows (equivalent of Esper named windows) over ODBC and not just only with a proprietary SDK.

If you want to explore this concept with Esper - this slide deck is a must read:

EsperJDBC

View more presentations from espertech. (tags: esper cep)

A demo also demonstrates how you can perform ad-hoc queries on your CEP real-time data and computations using any JDBC compliant tool - namely Actuate / Eclipse BIRT and create dashboards in minutes.

(Note: EsperJDBC is not open source and only available in EsperTech' commercial offering although you can access free trials)

(Note: Post updated as per Jeff comment on Aleri)

IBM WebSphere Business Events - going Esper? And WebSphere eXtreme Scale going on his own? And so does ILOG?

2009-01-07T12:15:00.001+01:00

As you may already know, IBM complemented its CEP/BEP offering with the AptSoft acquisition back in January 2008.
And they got SMILE continuous query processing research going on, and integration into their data grid WebSphere eXtreme Scale / ObjectGrid offering.
And now they got ILOG.
So what's the strategy around?

As with all IBM products, software downloads for trial and comprehensive documentation is available online, and you will quickly notice that WebSphere Business Events (v6) requires a good old database. This is where intermediate state will be kept and where correlation will occur.
Interestingly, CEP emerged out a wider trend that seeks to revolutionize the way we manage data (remember we sometime refered to Active Database in the academic work that precedes current CEP product offering) with a shift from "store then query" to "query as it comes" - or in other words, store the queries and flow data thru aka "continuous queries".

This quickly makes me claim that IBM WebSphere Business Events is lacking a big piece of the CEP platform puzzle, and likely why IBM is willing to complement this with another solution for more troughput/latency sensitive use cases with its InfoSphere Streams vision. Indeed, for that one, you'll have to wait late 2010 as explained.

So now should we assume that Esper will take part in an IBM offering sometime down the road? Or should we assume that Esper got it so right that Esper materials end up in an IBM "what is WebSphere Business Events" slide?

Here are some surprising hints. The nice "Advanced Event Processing" one is Esper, as described in the EsperTech site for more than 2 years. It appears in a slide deck from IBM Pulse 2008 partner conference.

(disclaimer: Up until now, WebSphere Business Events v6 is not built upon Esper)

To add to IBM offering description around CEP or business event as they call it, you'll notice that WebSphere eXtreme Scale (former IBM ObjectGrid) has now integrated the IBM Smile research in it thus exposing some IBM made SQL like event processing language (similar to Esper and Coral8 approach). This approach is different from the WebSphere Business Event product although it largely overlaps in what it does.

CREATE VIEW stockView AS SELECT tickerSymbol, avg(price) as avgPrice FROM
(SELECT * FROM stockStream FETCH LATEST 5 MINUTES) group by tickerSymbol

There are some interesting tutorials available, and I think this part of IBM offering would deserve a lot more exposure in the CEP community.

Instead of that, IBM disparate vision around event processing just added ILOG to the cake as per this just released press coverage:
"IBM Completes ILOG Acquisition, Opens Door To Event Processing"
But I am not learning you anything since on IBM owns terms:
"None of these products amounted to a real strategy. We didn't have tight engineering, focused on business event management".

It will be interesting to see where that ends...

CEP and patents

2008-11-24T16:37:00.002+01:00

I was recently updating my knowledge base around event processing, CEP and DSMS to have a better view on the current state of the art regarding patent applications around those topics, when I found a fairly recent pending application from Oracle named:

"Processing xml data stream(s) using continuous queries in a data stream management system".
The patent application is available online

It was filed in Nov. 17, 2006, published in May 22, 2008, and also claims to be related to a DBMS SQL/XML related patent filed in 2005 (is that an attempt to claim for anteriority?).

As far as I can sum it up, it organizes its claim around three fundamental findings:
(usual disclaimer here - this is my own view etc)

CQL or SQL-alike continuous event processing languages fall short on dealing with structured data streams
especially XML
both as input event stream and as output event streams (out of the continuous query).

f.e. support for
SELECT RStream(count(*)) FROM StockTradeXMLStream AS
sx [RANGE 1 Hour SLIDES 5 minutes] WHERE XMLExists( ‘/TradeRecord[TradeSymbol = “ORCL” and
TradePrice >= 14.00 and TradePrice <= 16.00]’ PASSING VALUE(sx))

I was a bit surprised by this patent because there is an interesting positionning dilemna. Oracle has publicly advertised its wishes to standardize CQL in some ways, and this patent is now an advertised intent to lock a subset of it.

Is it a good way toward standardization consensus?: I don't think so - it blurs your motivation for standardization ("leave XML out, it's patented - more value in my own product and more FUD to yours")
Is it a good way to quickly empower the user? I don't think so - more locking deep down to the language (remember all the messy ANSI SQL derivatives right?)
Is it a good way to innovate? I don't think so - bring it to end users and call for participation

As to know if there is anteriority or not - let's list some of XML support in different DSMS / CEP products:

Coral8, back in Sept. 2006, support the same XMLAgg, XMLTable and alike - see also their XML cookbook
Esper, back in August 2006, supports for XML events. The Esper EPL actually does a very nice job at not exposing XPath or XML in the EPL and automates the XPath plumbing - so that the EPL is actually event representation agnostic. I think that is a superior design
As a generalization, Esper, since day 1 (somewhere in 2005), supports Object structured event streams - which is a generalization of xml structured streams obviously.
iSpheres (now at Avaya) - can't even have a date but likely 2003 or so

And I could keep the list growing...
I am not saying patents are good or evil or a bit of both - that's no the point here. But this one just strikes me for what it is. I'd be curious to hear from Coral8 folks as well as others.

SOA, EDA and CEP

2008-10-29T15:31:00.006+01:00

There has been some coverage in the blogosphere on how SOA, EDA and CEP relate to each others.

"So CEP is not EDA, EDA is more than CEP. Promoting CEP as being EDA is far too simple. And yet that is what is happening in the current IT space."
"Especially the vendors of event processors focus too much on CEP as being EDA".

Mark Palmer from StreamBase answered some

"CEP, in fact, is a really important element of an effective EDA. It's not a required element, but it sure makes it better"

That's how I presented it at JavaOne back summer 2007, and this still happens to be true - no matter how vendors are mashing up various concepts about it to try to differentiate one from each other (CEP and Grid, CEP and Web2.0, CEP and DDS, CEP and edge processing, CEP and SOA, CEP and BRMS, CEP and XTP etc.).
Here are two excerpts from the presentation I delivered with Thomas Bernhardt from Esper/EsperTech fame:

If you talk to folks using CEP (build or buy - does not matter) in some industries such as investment banking, they'll tell you this has nothing to do with SOA but has to do with XTP.

It's time to learn from past history.

A similar debates occurred not that long time ago around the SOA and ESB terms (2005 article on ComputerWorld). For the record, back in 2001 ESB and SOA did not even existed. Yet, some visionaries such as Sonic folks introduced ESB as a COTS product, despite established or maturing MOMs presence such as TibcoRV, MQ, or JMS-based/J2EE MOMs and a whole set of in house built solutions.

With no surprise, in 2008, I'd be curious to hear someone that would disagree with the statement below, which is in fact a copy/paste/replace from comments happening right now in the blogosphere rel. CEP and EDA as stated at the beginning of this post:

An ESB is not SOA, SOA is more than ESB. Promoting ESB as being SOA is far too simple. And yet that is what happened in the early days (back in 2001).

Got it now? Here is mine bold claim:

A CEP engine is to EDA what an ESB is to SOA.
(And it's also more than just that)

Esper - course on DSMS at University of Oslo

2008-10-29T11:06:00.002+01:00

The University of Oslo, Norway, is running for 2008 some courses on Data Stream Management Systems (DSMS) - some of it based on Esper.
DSMS was the term coined by academic work as opposition to DBMS (database) in the "early days" before the more marketed Complex Event Processing CEP acronym was introduced. CEP is now more widely used by vendors, press and analysts as productization and real world deployment is taking place.

Some course materials are available online, and it is worth the read because it recaps on the key concepts, fit for purpose, and mechanics of DSMS/CEP engines such as TelegraphCQ, Stream, Borealis and Esper.

Introduction
DSMS part 1 - Applications Concepts and Systems
DSMS part 2 - Sensor Networks (less interesting unless you are in this area)
Course catalog

There are also 2 project assignments that plan to leverage Esper - which is an excellent illustration of how open source can be leveraged in the academic world.

Extending a DSMS benchmark
Online analysis of medical sensor data with the Esper Event Stream Processing System

There is obviously great confidence that the Esper project will not be closed down anytime soon for monetization purpose - unlike what happened with other such DSMS projects (if not all...). The direction around Esper and enterprise open source is already crystal clear and that's definitely a win-win for both academics, community, and enterprise practitioners.

Complex Event Processing and Data Fusion - Esper demo

2008-10-16T16:25:00.008+02:00

I have put together a live demo of a CEP track and trace application based on Esper Complex Event Processing (CEP).

The entire scenario has been designed and developed by Data Fusion Research Center AG, a Switzerland based company "that is quickly becoming a leading center of knowledge, research, and development in the field of geospatial data fusion and analysis". DFRC has chosen Esper for their CEP solution, and has kindly made preliminary results of their work available. I have rewrapped their application bits into a single Java WebStart package so that you can run it securely in a sandboxed environment without installing anything.

The Scenario

The scenario illustrates:

Edge computing, with raw events coming from radio sensors or ground based radars at the boundaries of a classical IT infrastructure
Complex Event Processing, an automated way for deriving coarse events out of real-time event streams, using advanced concepts such as time driven computations and causality concepts. A CEP solution typically comes with an abstracted programming model or event processing language so as to empowers the application developers with a continuous query paradigm.
Data Fusion, a "set of techniques that combine data from multiple sources and gather that information in order to achieve inferences, which will be more efficient and potentially more accurate than if they were achieved by means of a single source" (wikipedia). Data Fusion can sometime appear as a conceptual superset of CEP. In the DFRC application, the CEP algorithm can be tuned in several ways - which directly maps to the "potentially more accurate" goal of the Data Fusion approach.
Rich client application, to empower the business user and represent CEP / Data Fusion derived information in the most efficient way (in this case a satellite map with real time moving icons for identified aircrafts). CEP + BAM or CEP + BI - it all comes down to materializing real-time information out of the CEP / Data Fusion engine.

The Problem

Assume ground radars are pushing position events from disparate non-identified sources flying all around (friends or foo, UFO, noise, doesn't matter):

PointEvent {
latitude
longitude
}

For the example app, the events are simulated from a raw flat file:

46.5 7.2
47 7.2
46 7.2
46.5 7.2
47 7.2
46.3 7.2
...

The challenge is to identify all the flight paths in real time and eliminate noise, so as to figure out where are the aircrafts, what are their flight path, and determine if further investigation has to be performed by humans or downstream systems.

It is all about turning real-time raw event streams into situational awareness.

A flight path is a directed sequence of position events that is extremely likely to represent a real aircraft trajectory. It will be displayed on an interactive map, and specific tresholds of the data fusion detection algorithm can be tuned in the client side application.
Relying on CEP and Data Fusion concepts ensures we can scale to a large number of aircrafts, a high troughput of position events, and truely empowers the business users, turning a raw stream of latitute/longitude tuples into a rich system.

The Solution

The solution is implemented using Esper. Esper combines

A full featured EPL - event processing language. It can be for simplicity considered as an SQL-look-alike language augmented with time and causality. Main point here: this is a continuous query paradigm, and not a repeatedly executed query, and there is no database.
An efficient, feature rich CEP engine, implemented in Java (also available in .Net/C#). Refer to the docs, presentations and website for more details and usage scenarios.
An open middleware platform with open APIs, leveraging existing standards, that can be integrated into an existing infrastructure.

The algorithm designed by DFRC can be summarized as below (from their case study). Key capabilities, such as reusing existing geodesic distance and azimuth delta computation libraries straight into the Esper EPL language are also key capabilities being leveraged.

"Basically the algorithm was written to correlate events that are close enough together in distance and direction during a specific time frame.
Those events are considered as a potential flight path. Once it correlates events, it builds a flight path between connected events.

The algorithm compares flight paths, if any two paths share start or end points, which would mean that we have a longer flight path containing 3 points.

It then checks any 3 point flight path measuring the azimuth difference from the 1st to 2nd and the 2nd to 3rd. If difference is less than a predetermined number of degrees we consider it an identified aircraft."

Demonstration and Conclusion

Run the live demo (Java WebStart)
Quick howto:
Accept the Java Web Start security dialog
The rich client application launches and display a satellite map
Click the Start Button
Blue dots on the map are the raw position events. Red lines are the flight paths identified out of the raw data. Green arrows are the identied aircrafts

Read more from DFRC AG case study
Read more about Esper

The demo is entirely databaseless and serverless and fits in just 3 MB binaries. The very same concepts implemented here with Esper can of course be pushed out to the real world, with entire control of the architecture, its scalability and integration with fully fledged sensors/server/client setup.
Congratulations to DFRC for putting it together.

Complex Event Processing with Esper making its way to Java community

2008-10-13T16:09:00.007+02:00

There has been some nice coverage around Esper and Complex Event Processing (CEP) in the Java community recently:

Complex Event Processing with Esper, on DZone and published by OCI Inc.' Paul Jensen. A nice introductory article that covers the basics of Esper 2.x.
Open Source SOA upcoming book at Manning, due for availability in March 2009 by Jeff Davis - see interview on DZone. The book will cover CEP with Esper, alongside ESB, SCA, BRMS, and BPM all with open source solutions.

"Jeff: Complex Event Processing is a somewhat emerging technology, at least in the open source space. With CEP, real-time business events are sent to the engine, which can then use correlation and pattern matching rules to determine whether any anomalies are occurring within your enterprise. The real-time analytical engine is what differentiates CEP from other more traditional BI vendors, which tend to evaluate events after-the-fact. CEP is very exciting, and can be used for anything from compliance, monitoring service levels, to real-time trending."

It's great to see that after the core Esper team - including myself - spent time doing some evangelization for the last 2 years with articles all around on OReilly, The Server Side, InfoQ and Java One, it is now being taken to the next level thru autonomic contributions. That is one of the great outcome of making things available under an open source model.

CEP standard, really?

2008-09-08T22:30:00.009+02:00

The recently published paper by Oracle and StreamBase has triggered some deep echo in the CEP blogosphere.
The paper "Towards A Streaming SQL Standard" which was promoted in a StreamBase sponsored press release first appeared from the VLDB / Very Large Data Base Conference which took place on August 28, 2008 and is a joint work from Oracle, StreamBase and Stanford/Cornell universities (full paper).

Marc Adler from the user camp posted twice here and there on that matter, also asking Coral8 vendor to comment, which they did, and few days later, Apama also commented here although trying to move the debate "to mine (event processing language) is better than yours".

Overall, the comments are a mix of

"how that can be call standardization effort ?"
"how much an end user does care about the differences between X and Y engines as long as each engine documentation is clear about its behavior"
"how a new SPREAD operator would be more of a solution than an output clause" - a so called "big switch" approach that is available in Coral8, Esper, and likely others.

It all sounds like the title of the paper ("... standards") and its context (publication in a mostly academic conference (not a user conference)) got disconnected - on purpose or not.

There are some facts I would like to remind everyone:

Standards are not driven by having 2 actors work or study together (whatever there respective market caps or marketing loudness). Standards are driven by acceptance - which is in itself a de-jure or a de-facto acceptance. To date, neither StreamBase nor Oracle advertise a product that implements what they advocate.
This is an academic work ie mostly R&D from Oracle. This is not claimed as going toward product direction change for Oracle nor StreamBase. I think this is far from whatever road a company like Oracle would follow for a standardization effort and this is more close to StreamBase hooking on the elephant to issue a press release that ranks #4 on Google when searching "Oracle CEP".
Oracle itself has more than one event processing language and implementation. The one from WebLogic Event Server (which comes from BEA acquisition hence inherits from Esper capabilities given the relationship between EsperTech and BEA) does have the so called "big switch" approach that is illustrated in the paper. See docs from Oracle site about the Oracle Complex Event Processing 2.0 (ie rebranded WebLogic Event Server 2.0)

The Oracle product doc itself precisely illustrates this big switch like this:
SELECT * FROM StockTickEvent RETAIN 5 EVENTS OUTPUT EVERY 90 SECONDS
which in Esper would be
select * from StockTickEvent.win:length(5) output every 1.5 minutes

A better title in that regards would have been "Unifying tuple based and time based processing in event streaming engines" (far less attractive), or simply the one used by complexevents.com to announce the paper "StreamBase and Oracle team on CEP research" (although not a suitable title for an academic publication) - nothing more, nothing less - and blogosphere should take some wisdom reacting to such an initiative. Collaboration is always a good thing, and obviously always starts with 2 actors.

Complex Event Processing meets Business Intelligence: CEP BI

2008-08-06T11:15:00.002+02:00

EsperTech has announced a technology agreement with Business Objects, an SAP company. The news describes::
"EsperTech (...) unveiled that Business Objects, an SAP company and the world's leading provider of business performance optimization solutions, will incorporate EsperTech's Esper product, for evaluation, into Business Objects Labs' "Event Driven BI" prototype"

Beyond the headlines, you can actually download and run the real world prototype from Business Objects Labs that integrates BusinessObjects XI 3.0 ( XCelsius 2008, InfoView, BI Widgets) with Esper, and also access a recorded demo.

This is a really great combination, and it's great to see how BO worked on abstracting the Esper CEP into Business Intelligence semantics that is known by their user base and target audience rather than creating yet another CEP engine or claiming "we do CEP too" without true innovation.

Also interesting, Coral8 has also announced a "Marketing and Technology" agreement with Actuate. Actuate is a fairly smaller BI player, but well know for beeing the maker of the open source Eclipse hosted BIRT platform.

I am really a big fan of CEP + BI/real time dashboard integration, and it is great to see those two announcements, that further bring CEP into the hands of business users through various level of visual and semantics abstractions.

There are also some great side comments to draw from those 2 news showing up at roughly the same time:

Business Objects, Actuate (who are competitors), are probably right in their approach, and so are EsperTech and Coral8 (who are competitors): rising tide raises all boats.
It (again) demonstrates EsperTech has grown well beyond its initial BEA OEM relationship (Esper powers WebLogic Event Server / now Oracle), up to the point of beeing used by an Oracle top competitor: SAP.
If you haven't done so, checkout EsperTech EsperJDBC - a bridge between Esper CEP and any JDBC compliant tool. It shipped in May 2008 and includes a nice demo that is using ... BIRT!

EsperJDBC Sample code and recorded demo (follow the "EsperJDBC and Business Intelligence reporting and charting with Eclipse BIRT" demo link) available.

It seems that while EsperTech worked early in that area, beyond technology, Coral8 secured some more marketing ;-) Well done folks.

On CEP and SaaS

2008-04-06T11:48:00.000+02:00

There has been a few announcements recently around CEP solutions delivered as SaaS.
The first one to my knowledge is DataComplex, which is itself build upon Esper (read announcement here).
The second one seems to be RuleCore, which as far as I recall from previous posts from his lead is build upon the Mule open source ESB and Oracle BerkeleyDB.

DataComplex is pretty open on its capabilities and you can sign up for a free trial. It is also interesting that the focus is "business users", with f.e. google doc integration (ie a line in a google doc sheet (excel like) can be an event in itself), and other interesting things to come out of it.
The pricing is well aligned on any SaaS business. It is a mix of per user and technical capabilities:

account number (access to management console)
number of event stream types (not event count)
unlimited statements and events count (up to the point you overload your instance)
given memory size for the server instance

The price range from $10/month to $400/month
(details here)

RuleCore on the opposite is really vague on its exact solution despite rants like "for situation detection I actually think we are one of the leaders among the CEP vendors". You cannot access any docs, screenshot, architecture diagram, customer reference, or any trial at all. It is thus difficult to figure out who is the target user, but it seems the only supported transport is JMS - so a fairly technical requirement.
Despite low advertised prices at 0.0001 euro/event there is actually a 500 euro/month fixed subscription.
Otherwise you can stick to a 50 times more expensive pricing per event for a 50 euro/month fixed subscription.
The model does not provides any details about actual technical capabilities (shared platform or dedicated instance like DataComplex, etc)
(details here)

In both cases I think such offerings are worth tracking but without clear business user value and clear abstraction of the entire technical details, the value proposition is seriously impacted.

DataComplex is definitely well ahead RuleCore on building out a value proposition. There are certainly benefits in putting CEP based solution as SaaS offering, but I believe it makes sense when you empower your business users almost immediately. It is not about being affordable as RuleCore seems to argue around, it is about IT agility and business alignment. This is what successful SaaS vendors do: SalesForces, Postini (acquired by Google), etc. SaaS is not just about externalizing hosting.

You might want to read here for some basics

If you are like RuleCore a technology provider more than a SaaS provider, you certainly should consider do what Microsoft just announced: provide new licenses on a per CPU/month or per Subscriber/month basis for the real SaaS business focused vendors themselves.
(details here)

Esper maturity and open source maturity models

2008-03-31T15:35:00.004+02:00

Esper has definitely reached an exciting maturity already. There are many things happening, both controlled and uncontrolled, and this is what makes open source so powerful to widespread new paradigms such as CEP.

There has been some thoughts and criteria on what makes an open source project or product mature or not. The common criterias are:
On the community side (ie widespread interest and growing number of knowledgeable people around it)

coverage in leading conference, by the core team (somewhat sponsored then) or by virtually any volunteer speaker ("raise your hand if you have some smart thing to say")

use by/integration with/ other open source projects or initiatives in a loosely controlled/sponsored way ("innovation happens elsewhere" kind of effect)

coverage in good old technical books

On the enterprise side (ie more and more real-world deployments)

existence in a supported form, with full IP liability and possibly enterprise level certifications and features

actual real world use in such a form

apparition in technical requirements for job descriptions

actual roadmap and feature delivery

Here are some links of maturity models with such criterias:
QSOS
OSMM
industry analyst views
HP content

All those maturity criterias are matched today by Esper, and this is very exciting to watch and act upon this.
Here is what I can already wrap up:

dedicated coverage in JavaOne, TheServerSide, NoFluffJustStuff, JAX conference, and more coming. Half of the coverage was from volunteer speakers: Brian Sletten (Zepheira) and Papick G. Taboada. Links are:
Esper slideware
NoFluffJustStuff
JAX 2007

integration prototype with Apache WS02 by Paul Fremantle (Co-Founder and VP of Technical Sales at WSO2), and Apache/IONA Camel/ActiveMQ by James Strachan (Technical Director at IONA), David Greco and team. There is way more to do on that area of course.
See WSO2 and Apache Camel

dedicated chapter in "OSWorkflow" by Diego Adrian Naya Lazo, available on Amazon bookstore and any good bookstore. It covers Event Driven SOA, Event Driven BPM and Esper enabled BAM in a dedicated chapter!
Here at Packt or on Amazon

growth of EsperTech as a company with a dual licensing and professional support model, including a wider product portfolio with f.e. EsperHA

more OEM agreements, more supported customers (see a few referencable names)

apparition in job descriptions f.e. here by SunGard / Finetix
Here on MySpace or from Finetix directly
Esper is now in version 2.0, with 2.1 at the corner, with a version about every 3 months with a chunk of features that is most often here to match real-world demand

On the common things I hear is that NEsper, the Esper .Net/C# version is "lagging a bit behind". For one reason this is because the NEsper team is somewhat iterating on it as a port initiative thus always slightly with a version behind, and for another reason is perhaps that the .Net community is not as large and disparate as the Java community to digest new concepts somewhat that fast (and so is it with NHibernate, Spring Net, and a bunch of other products initially started in the Java world)

I am really looking forward to 2008 and beyond!
and I am already pleased to announce that Esper will get coverage at JAX India 2008 (nice job Papick) - abstract available here and that Tom' TheServerSide Symposium 2008 Las Vegas slides are available online.

Esper adds HA, local cache and resiliency to CEP with EsperHA

2008-03-13T11:34:00.002+01:00

EsperTech has released its first non open source product EsperHA.
EsperHA is an extension to Esper the leading open source ESP/CEP event processing engine for Java. It essentially adds key features to match common enterprise deployments requirements:
- continuous query results can be made resilient and recovered if a crash occurs
- they can deal with large data window with an overflow / page out to disk option (f.e. keep a 20 days moving average VWAP, or find causality between events over long periods of time - countless of use cases)
- can simply make EPL statement durable so that they are re instantiated upon recovery / restart

From the announcement there is much more to look at, and the fact that all this comes at no code change, with fine level statement-level and stream-level configuration means that one can scale up or down its Esper applications with minimal effort.
By contrast, Esper keeps everything in live memory which provides excellent throughput and low latency (see some figures here) but is 1/ bounded (few G or more likely 32G to 128G in real deployments) and 2/ is dropped upon shutdown / crash.
EsperHA ensures one can trade-off low latency, high throughput and full resiliency in one system.

This is an interesting crossroad for EsperTech as a company as it further strengthen its positioning and readiness for real world deployments, and nicely complements its so far professional open source / dual license model with a classic software vendor one.
There was already great evidence of Esper fit for purpose in real world situations, as OEM and actual deployments demonstrated so far, but having this HA / resiliency option will certainly reinforce this.

There is common FUD from classic / elephant vendors thrown at open source software - essentially saying that it is not enterprise ready, not operable, as it is made by a bunch of volunteers that don't care about production reliability. This seems EsperHA and EsperTech proof point how much of FUD are that kind of claims.

You can sign up for a free trial and gain access to the download, docs and samples here.

Lab49 - have you looked at Esper?

2008-02-26T22:41:00.002+01:00

Lab49 folks have been quite verbose in the Complex Event Processing (CEP) arena for financial services. They basically ended up partnering with pretty much anyone and almost every single month, skip to another CEP vendor to partner with.
Nice but certainly tough to build up and scale up product expertise especially given there is no much standard in the CEP arena so far - but that definitely gives them a nice view on each products.

To complete their 360° they should definitely look at Esper and EsperTech offering, including also NEsper as they seem to be quite involved in .Net as well. They probably know where to reach out for that, and will certainly hear about it anyway as the number of deployments of (N)Esper in financial services is growing nicely.

Few history on Lab49 ' somewhat fuzzy logic about partners:
- February 2008 - Aleri and Lab49 Partner on Tool to Visualize Market Liquidity
- January 2008 - BEA-Intel-Lab49 Whitepaper on CEP in Capital Markets (you can also find a more recent much vendor influenced podcast)
- November 2007 - Real-Time Trading and Industry Networking Event with Lab49 at StreamBase
- June 2007 - Coral8 and Lab49 Partner to Provide Algorithmic Trading Framework on the Microsoft Software Platform

Now you know what partner means. We are all partners to make the point that CEP brings business value, and Lab49 does it well in financial services I think.

Open Source ESBs and Esper CEP

2008-01-28T16:54:00.000+01:00

Stefan Ried, a former SoftwareAG now at Forrester has given an interesting take on possible strategies for Open Source ESBs during a recent Expert Meeting on BPM/BAM/CEP/SOA/EDA.

You can read it from there.
It is interesting to see CEP in the ESB capabilities list for 2008 (as a reminder 2008 is this year). This trend is definitely an interesting one, and folks like WSO2/Synapse have actually already done some prototype work with Esper - as described here on Paul' blog.

You can definitely bet you'll hear much more about those ESBs / CEP integration with Esper throughout this year, despite it is already doable without federating capabilities in one server but using an ESB and an Esper event server bridged together with any kind of transport(JMS f.e.). This has the benefit of keeping the ESB stateless, rather than making it stateful (I consider CEP to be of a stateful kind) but certainly makes the architecture slightly more advanced so I am sure either ways will find their fans.

Open Source Event-Driven SOA is right on Esper' track and WSO2/Synapse has definitely made some good steps forward already despite productization is not yet there right now. Let's wish Mule and JBossESB catch up so as to provide users choices yet consistent CEP semantics - thru Esper of course!

Backup your data with MirrorFolder

2008-01-07T12:00:00.000+01:00

I thought I'd blog about a great affordable piece of Windows Software: MirrorFolder. It provides a software level real-time RAID-1 mode that can backup files in real time, and I am also largely using the option to auto-synchronize my backups upon connection of my USB hard drives. There is also a whole bunch of options for fine grained configuration.

The 4.x version now includes a centralized GUI that is a great complement to the Windows shell integration (right click on a folder to configure its backup strategy).
Get it from Techsoft. It's 39 US$ so close to just 20 euros these days. Lucky me.

CEP vendors: RIP Kaskad, new Pion, and same IBM

2008-01-04T10:19:00.000+01:00

End of 2007 has been an interesting time for the Complex Event Processing (CEP) ecosystem.
First, Kaskad adds itself to the RIP list and closes business as reported by Marc and confirmed by Kaskad former CTO Colin Clark. This was 16 employees. So for those who would argue that a company with 16 employees is going to sustain better than one with less than 5 in the same market, well, not that obvious.
I am not surprised by the news. Back in February 2007, Kaskad had announced it would open source part of it offering. I had asked them for more details both privately and publicly several time as I found the news interesting especially from an Esper and EsperTech perspective but never got any answer from Kaskad team. I wonder if they weren't already dead back that time...

Other news I could spot thanks to Marco: some Californians have started to chase the CEP market with an open source approach ala Esper - Pion from AtomicLabs: Our goal is to build the world's #1 open source platform for complex event processing.
It is actually quite far to compare to Esper, might be implemented in C++ and not Java / .Net, and has yet to be proven and most importantly has yet to expose some real CEP features especially some Event Processing Language (EPL). Data or event processing is not CEP. Welcome aboard guys and good license choice by the way (Affero GPL will ensure SaaS customers will have to buy something from them as well).

Finally, IBM just woke up with WebSphere CTO saying CEP is SOA next big thing. Once again few notes here... What Jerry Cuomo says is actually ...I really believe it's the next big thing in SOA, and that's event processing. So he jumps on the CEP wagon but in fact extends its scope to event processing to make the point. Well, who would disagree with that then?
I actually talked about the CEP - SOA - Event Processing relationship in my JavaOne Esper presentation back early 2007.
I am really looking forward to see what IBM will really deliver in that space, beyond research, research, research, forward looking announcements like SystemS or tiny limited features additions here and there (see ObjectGrid CEP f.e.).
I bet the next CTO will finally acknowledge that XTP meets SOA precisely through CEP and this one is really the next big thing - ie CEP is far from being just in SOA.
For some reason Jerry forgot to add XTP to his mix ;-)

2008 is going to be an interesting transition year for the CEP space as everyone is basically working on it growth, be it with open source, evangelizing, use case solving, or SOA bandwagon.

The 1 million events per second CEP

2007-11-21T14:59:00.000+01:00

BEA has finally published some paper on WebLogic Event Server performance.
It is available here (registration required).

They finally jump in the vendor dance of the 1 million events per second ESP/CEP claim that I had blogged about before here, thus catching up with EsperTech' Esper and all others - unfortunately only as far as these performance claims mean.

What is nice is that BEA provides details on the actual scenario, continuous queries and hardware used although the kit is not available for one to run on his own. To date Esper is still the only one to be completely open in the CEP performance area with full disclosure of results but also actual performance kit source code (see here).

Giving it a quick read, here is what I can sum up:

(both use a one thread per connection model to avoid context switching)

So obviously the BEA Event Server performs well, no doubt on that. This said there are a few immediate comments:

The Esper benchmark somewhat set the stage, with a platform more than 4 times smaller (all ratio preserved, Esper appears almost 2.5 times faster though we cannot really compare things).
The Esper benchmark EPL statements are more complex, especially regarding memory profile: they keep sliding windows of 1000 events where the BEA benchmark keeps 2 or 3 events.
The Esper benchmark runs with more statements (1000 instead of 400). This is interesting as running more statement likely means less statement-level contention - so I am wondering why BEA is not running more statements here.
The BEA benchmark does not disclose CPU usage, and for some reason the average latency is high compared to the 99.99% latency (ie the worst case latency must be high). There are probably differences in the way the two benchmarks actually measure latency (total run, post warm up, snapshots etc).

Unfortunately, I haven't a quad CPU quad core Intel Caneland box in my lab to play further with this. This being said, the two benchmarks are completely meaningless as the use cases are highly specifics and narrowed down, so you should really read all this as vendor standard claims, or run your own benchmark for your own business meaningful scenario.

Welcome BEA in the 1 million events per second CEP category, and kudos to them for disclosing (almost) all the details.

I also encourage you to read what Opher Etzion from IBM say on this, or also read what StreamBase published recently on their own benchmark to understand why all those claims are... really just claims up until the point you do your own evaluation!

on GPL and dual licensing

2007-11-06T17:08:00.001+01:00

db4o has just published a paper written by a law firm Greenberg Traurig on GPL and dual licensing. This is core of their community and commercial business model, just as it is for Esper and EsperTech.

That is one of the best and most of to date resources on GPL and dual licensing I have ever read, and I 'd encourage anyone working and using GPL code to read and understand it. It also does cover GPL and hosted projects (Application Service Provider), work for hire and distributor (of course) situation.

You may read there: You may hear people argue about whether the GPL is a contract or not. The important thing to understand is that whether it is or not, if you accept code under GPL terms you must abide by the GPL. If you breach the GPL, your rights under GPL license terminate, and you have no right to use the code at all. If this happens, you will be infringing the licensor’s copyright, and copyright infringement puts you at risk of civil damages or even criminal culpability with the possibility of heavy fines.

Read it from there or PDF here directly from db4o website.

If you want to read further on GPL and dual licensing, the following law blog is quite interesting as well.

Price per CPU is foolish and price per core reaches its end

2007-09-26T18:15:00.000+02:00

I have been scratching my head recently on software vendors policies regarding pricing and multi core pricing. The game is completely foolish and hopefully will reach its end soon, as more and more vendors will show up pricing per user to best serve the SaaS model of their customers, or pricing per instance (read JVM instance, or software server instance in a generic fashion).

Lets give it some background first and take the example of a small software vendor selling a distributed in-memory cache - aka Tangosol now owned by Oracle. Up until very recently before the Oracle acquisition, Tangosol had public price on its website, on a "per CPU" (in clear text) basis.
In August 2007 they shipped a minor version of their software (now owned by Oracle) and announced a new price of about twice more expensive. I dare to raise the issue and Cameron (Tangosol former CEO now at Oracle) kindly dare to answer it: Tangosol was priced per core and not per CPU despite advertised as per CPU on their web site... Due to Oracle policies regarding CPU and core pricing, they had to go thru this price change.
Read it here.
(no offense to Tangosol, it's great stuff and I am glad they clarified this).

Conclusion 1: Don't trust advertised per CPU prices. Anyone advertising per CPU price without disclosing complete details regarding CPU and core policies is lying to defer the price debate, and brings complexity in a TCO comparison if they are in a competitive situation.

Next step, let's recap on vendor policies regarding CPU and cores...

You already know that you cannot buy single core x86 (Intel/AMD) chips anymore, that the norm is dual core, and that quad core per chip (read per socket on the mother board, with 1 socket = 1 CPU) will be the default in a 6 months timeframe (go check on dell.com, you can already buy quad CPU quad core servers).

(disclaimer: this data is from searching in public sources):
For Oracle, you can read that each core accounts for either 0.5 or 0.75 CPU depending if it is a total dual core or quad core. So a double core costs 1.5 CPU units and a quad core costs 2 CPU units.
Read it here and here.
A quad CPU quad core costs 8 CPU units at Oracle.

For BEA, you can read that each core accounts for 0.25 CPUstarting at the 3rd core in a single CPU. So a dual core costs 1 CPU unit and a quad core costs 1.5 CPU units.
Read details here.
A quad CPU quad core costs 6 CPU units at BEA.

For IBM, they switched to a proprietary defined Processor Value Unit that is assumed to better assess various processor architectures. In fact this is quite similar to what Oracle does if you look here and on IBM site here.
A quad CPU quad core costs 8 CPU units at IBM.

For RedHat / JBoss (this is a subscription based business model but still), you can read half about it on RedHat site here and look at the JBoss TCO calculator here.
It works thru 4 CPU packs, but I could not find any information on what their CPU means... so let's exclude it for now...

All in all, you can get a 25% price difference just by considering this CPU unit. Oracle can tell you its stuff is 25% cheaper compared to BEA, you'll give them exactly the same amount of money in the end on a quad CPU quad core hardware.

Fair enough you said ??

Lets consider a few points:
- do the business project people and admins know how many cores total they will need to run on, currently run on?
- do they instead perfectly know how many server instances and apps they have to start stop operate and monitor?
- should your software license price be directly impacted if an application increases or decreases in overall throughput because business project owners asked this and that new feature to be added?
- did you already over-provisionned your hardware platforms to hide this fact?
- did you took time to ask the vendor policies regarding virtualisation? What if you turn the 4 CPU quad core box in 8 virtual 4 (virtual) CPU ""quad core""(they are virtualized now right?) system to maximize its utilization?
- do you have a virtualization or resource brokering project (read dynamic intelligent real time throw in on-demand adaptive capacity) and you are close to the day you'll click a button to move an app instance running on this old 2 CPU dual core to your new 4 CPU quad core hardware when it is on peak load (heard about VMotion and XenMotion already right?) and with no downtime. Did you thought on how you'll be able to optimize or even plan your per CPU license costs in such a scheme?

It's obvious a per instance model (per JVM instance, per server software instance) brings simple answers to all these questions and is perfectly aligned with what IT and project people are held accountable for running real world apps.

The problem for switchers (BEA, IBM, Oracle and all /CPU vendor) is that it will lead them to fairly complex discussions with their switching customers, and the best exit is likely to start selling a (time limited) enterprise wide (project limited) licenses that does not account anything anymore (until you realize it - hey it's a sell/buy game!). The pain the vendor put on you to manage and account all these licenses even becomes part of its value proposition.
... up until a billback system is put in place to enable a pay for what you used model fully aligned on your own business running costs and ROI targets.

Esper performance: more than 500 000 event/second

2007-08-14T14:30:00.000+02:00

The Esper team has published performance results for its Esper event processing engine.

"Esper exceeds over 500 000 event/s on a dual CPU 2GHz Intel based hardware, with engine latency below 3 microseconds average (below 10us with more than 99% predictability) on a VWAP benchmark with 1000 statements registered in the system - this tops at 70 Mbit/s at 85% CPU usage."
Read more starting from the doc and the Esper wiki.

I have been working on this as I basically wrote the performance kit that is available to anyone under a GPL license, and whose client side (which is not Esper specific) can also be used to test other solutions - and I am glad to see those results.

ESP/CEP has been a common place for vague information around performance in a mine is bigger than yours attitude from all vendors to date (no one ever published publicly a detailled benchmark). As almost everyone claims hundreds of thousands of event per second, this just shows Esper is enterprise grade ready and capable of the same kind of performance results.

In the meantime there are a few extra comments to draw from this
- the benchmark app includes a NIO/TCP based Esper event server that can also make use of a staged (read highly concurrent and multithreaded) architecture. This shows how easy it is to set up something like that with Esper. The code is about 5 classes.
- for some really odd reasons, BEA WebLogic Event Server, that does make use of Esper as announced here, was said to top at only 50 000 event/second ie 10x slower (read here). I wish someone from BEA would shed some more light on this.

Coral8 and IBM - how does it compares to Esper (Tech) and BEA agreement?

2007-08-12T14:28:00.001+02:00

Tim has commented about the Coral8 announcement in his blog.
Beehind the announcement, the fact that Coral8 and IBM have an agreement in place in the Event Processing arena, is not really comparable to what exists between Esper(Tech) and BEA for the WebLogic Event Server (I had blogged about it here).

The fundamental differences are:
- IBM WebSphere RFID includes an adapter to the Coral8 Event Processing engine. It is not an OEM agreement such as between EsperTech and BEA.
- The agreement is limited to one product in one vertical marke - while the EsperTech and BEA agreement is not tied to any vertical market in particular.
- It's unclear what does "Coral8 provides a free deployment license to all IBM WebSphere RFID customer" means ... you likely need to consider support, size of target deployment platform, pre-production environnement etc. Where free ends?

This said, congrats to Coral8 team for that. The Event Processing market has much to win from alliances like that one that help brings Event Processing to mainstream.
In the meantime I am wondering how hard would it be to have an Esper adapter for IBM WebSphere RFID ;-)

This is my new blog

2007-08-12T14:24:00.000+02:00

I finally decided to move by blog here on blogger.
My old blog at http://blogs.codehaus.org/people/avasseur/ was not fun enough to edit and customize.

I have imported all my old posts in this blog although for the oldest ones, the comments are closed.

BEA WebLogic (Esper?) Event Server

2007-07-20T23:42:00.000+02:00

There has been quite a lot of rumors these last weeks regarding the new BEA WebLogic Event Server which set the date of the entry of BEA into the Complex Event Processing arena.

As official facts and news can now be read by everyone, I thought it was a good time for a recap and for a few comments and thoughts. You may find this post quite long to read, but well ... this is my industry viewpoint!

(1) Facts

Rumors were all at rage trying to figure out if BEA "ripped off" the Esper ESP/CEP open-source GPL licensed engine, which has been in the arena for more than two years, release after release, commercially supported by EsperTech - hence the title of an hypothetical BEA WebLogic (Esper?) Event Server of this post.

Now we know, BEA is using Esper to bring to the market its WebLogic Event Server and has an agreement in place with EsperTech, the dual licensing company behind Esper and NEsper.

Now is time to shed some light on that topic. But first, let's recap...

(1 continued) Facts

So it first started on InfoQ with the public beta announcement of BEA product. Guillaume from Groovy fame immediately wondered if Esper is used under the hood.
Read on on InfoQ

Some weeks after, Coral8 did the same analysis and it seriously tried to lower the BEA value proposition - which is sort of expected from a BEA competitor and Esper competitor:

"all BEA did was OEM an open-source product called Esper and put it inside their product"

This forced BEA marketing folks to answer in the same article with an interesting quote, shedding some light on the BEA and Esper / EsperTech agreement.

Read it on Intelligent Enterprise

The same news was discussed again by Philip Howard, from Bloor Research, who has been following the Complex Event Processing space for a while, in an article where he describes some more the BEA Time and Event Driven (TED) product family that includes this rumored WebLogic Event Server.

"There has been some speculation that TED is based around the open source Esper event processing engine in part (...)"
Read it on IT Director

Finally, right after the BEA product went out from beta to be generally available truth as finally been spread by EsperTech - the company that provides support, services and OEM licenses around Esper and NEsper (its .Net counterpart) which was founded by Esper' founder:

Read the EsperTech press release.

(2) So how far Esper and BEA WebLogic Event Server are different?

At that stage all those rumors and this finally official press release don't shed light on how far Esper is modified by BEA as it seems to be the case, to make it his own. I'll certainly give a shot to this. For now here are some findings:
Esper: select avg(price) from Stock:win.time(1 min) BEA : select avg(price) from Stock retain 1 min Esper: every (A -> B) BEA : every (A followed by B)
Part of the documentation is the same f.e. for explaining the core capabilities:
Esper doc
WebLogic Event Server doc
Or for providing a number of use cases:
Market data Feed Monitor Case Study
Use cases

(3) Open questions

At that stage, this raises a few interesting points and speculations going further:

Esper is at version 1.10. There are tremendous improvements brought versions after versions such as deterministic behavior with statement pipelines, performance improvement, bug fixes, extended APIs and so on.

How the BEA product will catch up if they modify their own version in various ways thru refactoring and such while Esper keeps being improved?

How the two products will distinguish each others going further, what will make each one attractive to who? Low price point, a la carte, higher price point , same ESP/CEP capabilities, off the shelf experience ... all interact. Will we see open-source stacking up an Esper Event Server as well?

Esper has the very same client API and event processing language that its counterpart NEsper. This means Esper / NEsper / EsperTech is the only one to give choice with same semantics across platforms. Will BEA move to .Net as well?

Esper GPL license explains why BEA had to go over a dedicated agreement with EsperTech so as to redistribute the product. This proof points the dual licensing model and shows that a GPL/LGPL based open source/dual license is one among the many viable open source model, especially when geared toward both OEMs and end-users.

Will open source purists argue this means BEA is using open source and contributing back on some areas (those very one would argue they are already commoditized by far), but that BEA is also ready to buy open source to get the right to tailor it without giving much back so as to build a competitive advantage? On the other hand, other will argue GPL or LGPL is not business friendly (if you think you can get for free into a bar and also get free beer then you will). From there this starts to make sense to buy a foundation from which you can build a competitive advantage in your journey - does it?

But does this means Oracle or IBM or RedHat or IONA or SUN could do the same and enter an agreement with Esper? Those big ones have yet to enter the Complex Event Processing space beyond announcements or lab projects. New startups are poping up all around at the same time so decision is far from simple, especially as BEA moved first around such a blend.

Does this means that BEA decided to leverage Esper and acquire a license instead of spending 20 to 100 M$ in acquiring one of the other startup in this field (or entering an OEM agreement with one of them)? We can assume BEA has done carreful due diligence, both on the IP, the technical excellence and the extensibility options of Esper before taking such a decision.

No doubt everyone will have various rumbling about all this!

(4) Esper provides choice, BEA provides off the shelf

Most important to recap, the BEA WebLogic Event Server is not good or bad in its way it does (or don't) interacts / extends the Esper project and the open source community around it depending who you ask. What is striking is that:

The ESP/CEP innovation came from Esper and open source in the first place (as far as Esper goes, of course ESP/CEP come a long way in academic works before Esper and at other startups). The full Java ESP/CEP engine around a POJO centric model and open API, plus the real convergence between ESP and CEP capabilities was introduced by Esper and truly driven by the community around it. And now it is also available for .Net with NEsper.

BEA WebLogic Event Server adds a server model, with deployment capabilities, Spring wiring, security, real time JRockit JVM integration, all based on a very innovative OSGi based lightweight kernel build upon other open source blocks (Eclipse Equinox f.e. for the OSGi core). The open source and commercial blend goes far beyond Esper (Jetty and Spring are in there). You may likely find more open source LOCs than BEA LOCs in this product if you were to count everything - simply because this is how modern software is build. Would you rather build it, buy it or license things and blend it instead?

BEA offering shines because it is an off the shelf platform with its own server component which is not a JEE app server but a completely new one. How would it compare f.e. to a classic WebLogic Server (the JEE one) with Esper hooked a JMS consumer/producer, both in terms of features but also investment and learning curve?

Esper is obviously the best option to integrate into other BEA technologies such as WebLogic Server (thru JMS as I said), but also if you are not a BEA shop, running application server / ESB from open source or big vendors. It also brings affordable ESP/CPE compared to other startup-high-license-cost in this rather new market. It also has a significantly increased flexibility as there is no target server model. You can also get support from the source through EsperTech (professional open source a la JBoss). And if you want real-time capabilities, you can run it on a real time JVM, from BEA, but also from Sun or IBM, or even inside an Azul box. This is a la carte event processing as Tom explains in the press release. A number of analysts have already started to argue around the fact that a la carte event processing may make more sense for the industry so I am sure we'll see more of that blend.

Congrats to Esper and EsperTech for their success with BEA, and congrats to BEA for this new product offering that truly shakes the complex event processing arena - perhaps not in what it does but more in how it was assembled and engineered - with a best of breed approach all the way up.

As I was just about to publish this, another news came out:
SL Corporation Helps BEA Extend Real-time Monitoring and Visualization of WebLogic Event Server. This increases the feature set of WebLogic Event Server, by a new blend. Interestingly, Progress Apama (another Esper and now BEA competitor), is using just the same as announced here and already mentioned by Tim Bass, from Tibco (did I said Tibco also plays in the ESP/CEP space and has a very close relationship with SL Corporation (see here). That said this does not surface yet in WebLogic Event Server 2.0 final release.

Interesting and exciting times in the software industry isn't it? Who said it is build vs buy? You need to think build vs buy vs license vs assemble.

Esper brings Complex Event Processing (CEP) to mainstream

2007-07-13T18:06:00.000+02:00

There has been two interesting posts recently regarding Esper.

One from Tim Bass, from Tibco, reporting from the InformationSecurityAsia2007.
Tim reports in his blog that complex event processing solutions like Esper can be used for extrusion detection (that is inverse of intrusion detection in a network where you want to track malicious users and software acting from inside, zombi machines etc.).
Tim reports Esper is already known from several experts in the field and is beeing considered for such use cases. Glad to hear and thanks Tim for the info.

The second interesting news appears in Intelligent Enterprise where Esper is beeing quoted as the only active open-source project for CEP - both for Java and .Net platforms (what, you never heard about NEsper? Bet you will !).

Esper is quoted separately from the so called more commercial solutions like Apama, StreamBase, Coral8, Tibco etc. Especially because Esper is a great initiative to bring CEP to mainstream thru innovative, affordable and a la carte approach as quoted here:

Not For Big Companies Only.
With the ranks of CEP practitioners including big government, big finance and big telco, you might think the technology is accessible only to giants . That's not the case, however, as there's an active open-source CEP project called Esper that offers both Java and .NET components.

It 's nice to see Esper quoted, although I am not sure I would do such a difference between Esper and the others solutions available for the following reasons:

Esper is commercially backed and supported - checkout Esper' founder company EsperTech

Esper users include major investment banks already and other big shops - checkout the mailing list if you want to get a taste

Of course I can't say if those guys are evaluating Esper or if they run some of their production system with it already - so you never know who is a user evaluating software and who is ready / has already entered the commercial relationship with commercially backed open source software unless you ask and they dare to reveal it.

There's trully no difference in fact when you look at that from a community point of view.
When you are building open source software (which I have been doing for years now) there is only one thing that matters: the community.
The community rules the use cases and drives the product direction in unanticipated ways accross industries of all kind and size.

This is a typical mistake regularly done when trying to compare open source software and commercial software as two fundamentally different things. Would you start saying Linux is for small companies and Windows for the big ones...

Event Driven Application Servers - BEA is in

2007-05-29T17:21:00.000+02:00

BEA has started to spread the news about the fact they'll enter the Event Driven Application Server area - which I covered during my JavaOne 2007 session (with a bias on the first and only Java and .Net container for event-driven applications aka (N)Esper). The product name is WebLogic Event Server 2.0 (go figure what happened to 1.0!?), slatted for a public beta in the next few days.

Tim from Tibco wraps up the news in his blog.

Stay tuned.

Esper at JavaOne, on Event Driven Application Server and XTP

2007-05-02T22:51:00.000+02:00

I am pleased to be back at this year JavaOne 2007 as a speaker on "Event-Driven Application Servers". I'll showcase ESP/CEP technology to the Java community and industry, San Francisco, on Friday May 11th, 12:10.

The session that I co-lead with Tom, Esper and EsperTech founder, will introduce the audience with Event Driven Architectures (EDA), Event Stream Processing (ESP) and Complex Event Processing (CEP). We will showcase the why, what and how on event-driven application servers, going from simple integration over JEE platforms to tailored off-the-shelf leading edge solutions playing in the eXtreme Transaction Processing (XTP) field.

This has implication in various areas such as investment banking, RFID, monitoring, compliance management and many more (up to this good old SOA...).

If you attend the event or are in the bay area and want to meet up, ping me or leave me a comment!

The session id is TS-1911 and if you happen to attend JavaOne this year you can already add it to your session schedule or click below to connect with me for the event.

I hope to see you there and see a whole bunch of long time friends there.

Esper
JavaOne 2007

Esper tutorial on OReilly ONJava

2007-03-12T12:01:00.000+01:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

It's live!

Event-driven architectures turn a traditional data-driven application's architecture upside-down. Instead of storing the data and running queries against stored data, Esper allows applications to store queries and run the data through.

This article introduces Esper, a lightweight ESP/CEP engine written in Java.

This Esper tutorial is now available online at OReilly ' ONJava. The tutorial illustrates an airport self-service terminal that we monitor and improve with real-time customer relationship management by using both ESP and CEP capabilities of Esper.

It includes a fully functional yet simple to run bundled application and also includes Eclipse configuration files so that you can start practicing right away (if you are an Eclipse user...).

I hope this will help get those blooming ESP/CEP concepts to mainstream. The tutorial should give you enough ground to understand where we come from and enough details to get started in your own project right away.

If the EPL snips below does not sound familiar to you


select avg(price) from StockTick.win:time(30 sec)
where symbol='GOOG'

select type, count(*)
from BaseTerminalEvent.win:time(10 minutes)
group by type
output all every 1 minutes

Read it online at ONJava

or read more about Esper

Terracotta' Jonas & Eugene on AOP at AOSD 2007

2007-03-10T13:08:00.000+01:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Jonas and Eugene from Terracotta - the Naturally Clustered Java provider and maker of OpenTerracotta - have published a very interesting paper on industry use of AOP and point out some limitations present in current AOP frameworks: "Clustering the Java Virtual Machine using Aspect-Oriented Programming".

Excerpt:
"application startup time with AspectJ load-time weaving [which I partly authored] was ... slower and memory overhead was ... bigger ... when comparing with similar transformations done with either the Terracotta runtime or the AspectWerkz AOP engine [which I authored with my friend and former coworker Jonas]."

This comes to no surprise to me as AspectJ is born for build time weaving and Eclipse integration and AspectWerkz was born for load time and runtime weaving. As author of both (as we joint forces from AspectWerkz to AspectJ back mhh.. 2 years ago) I was unable to get the best of both world in one engine as we initially focussed on features integration - such as thru the @AspectJ style that is annotation-defined and driven AOP. Since that time my open source bandwith has been drawned due to work engagements. That is unfortunate as I believe there are no technical reasons for this multi-weaving-optimized engine to not happen. Fortunately AspectJ is strong on runtime performance which is something you really have to consider as well.

Their paper is available online on AOSD 2007

Free as Open Source they said?

2006-01-13T16:26:00.000+01:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

It's interseting to follow the different strategies around Open Source these days.
Can Open Source be FOSS at all?

I have been myself leading a project for some time with Jonas Bonér in the AOP field, and we more than once had potential corporate users willing to be helped or supported (thus charged) with some more formal ways than classic user mailing lists and "fix it yourself / wait for the community to fix it" attitude.

BEA's blended strategy of certifying and supporting 24/7 major open source components (like Spring) is well echoed by several facts these last weeks.

First some post on TheServerSide "Developping J2EE without xxx? Why?", xxx beeing your perhaps favorite open source framework, (xxx and yyy abstracted for the purpose of this blog article).

Some various comments from this thread:

"xxx is good"
"Don' t look from the developer angle only"
"Couple of years down the track, if xxx goes bust or I can't find developers who want to work with old technology like xxx as everyone has moved on to yyy_standard or something similar, and my application breaks for certain reason, where will I get support or resource to maintain my application?"

More interesting, a recent quote from HP Software head Todd DeLaughter (translated from French):
"We could do something with Open Source. It will depend on the market evolution. [..] Today the problem is that sooner or later customers are asking their software provider to certify and support the stack, and provide services around. That won't change any time soon."

Obviously it would then be intersting to look at the cost of Open Source, beyond the FOSS zero acquisition cost.

It's not secret that JBoss is now selling software (not free, with trial copy and commercial license) and doing marketing around it - see the last JBoss newsletter and the messages around JBoss ON "JBoss Operating Network", the what sounds to be administration solution of JBoss app servers (see f.e. here).
Quoting the newsletter:
"JBoss is holding a series of workshops [...] will include hands-on training on JBoss ON (trial copies of the software will be provided)"

Deploying stuff on an Open Source stack is end to end is definitely not free at the end. FOSS is just one view on one single phase in a project: development.

Left aside the training cost for the young developers themselves.
In a recent post, Yakov Fain calculates that if you follow introduction trainings on Spring, Hibernate and JBoss, which seems today' minimum to get started on a so called FOSS stack, you'll have to travel some accross the globe, and afford a total of $9000.

Obviously, we then should take into account support prices, and possible performance gaps between each stacks, meaning all in all you'll need more CPU and hosting power here and there to run your beast. And CPU is never free.

I'd be pleased to see the debate around the Java community shift from programming practices and tech buzzwords (IoC AOP ligthweight whatever is best implemented here things) to some from the trench real project end to end considerations.
Obviously lightweight approaches and Open Source do play a key role in the whole picture, both with pros and cons.

So FOSS - free as in free beer? Really? The world is obviously more complex than that.

Microsoft goes AOP?

2005-12-29T17:48:00.000+01:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

While AOP has reached a decent maturity level in the Java platform, thanks to several iniatives from different players with different priorities and goals (IBM, BEA, JBoss, Spring, and individuals like Bob Lee to name a few), Microsoft seems to be willing to move beyond that.

Last year at AOSD they were hosting a session on Phoenix, an interesting building block in the .Net CLR labs to help enable AOP frameworks. The community in .Net around AOP is fairly active as well, and Microsoft did awarded some of the projects, like Alan Cyment' SetPoint hosted on CodeHaus, the home of AspectWerkz.

Recently (Nov. 2005) Microsoft has set up a research event to emulate discussions around the best way to have good AOP solutions in the .Net platform. Several key .Net AOP project leads and folks from the AOP research academia were invited for a full day event at Redmond.

As far as I know Sun never set up such an event to try to understand AOP from its roots, and most of the discussion on the field has been driven by individual to individual networking and the AOSD annual conference.

I am eager to see where .Net ends up in that field in 2 years from now. There are certainly interesting architectures and concepts that we have implemented in Java based AOP that could be ported rightaway to .Net, though .Net luminaries have certainly several new ideas that 'd be worth exploring from Java luminaries point of view - and possibly standardizing on - obviously outside the JCP - unless Sun suddenly cares more about AOP.

Spring 2.x - innovation or maturity?

2005-12-23T10:57:00.000+01:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Last week I had the pleasure to discuss some more with the Spring team at JavaPolis, and I was thus given a quick informal update on what they are heading at in Spring 2.x, especially on the AOP side, discussing with Rod and Rob.

Spring AOP in Spring 1.x has been easy to use - though missing a good pointcut language. This simplicity is well described in a recent article on dev2dev by Eugene, and I had debated this issue and tried to solve it with Spring pointcuts on steroids with AspectWerkz.

In Spring 2.x, with the help of newly appointed excellent Adrian Coyler as chief scientist, my AspectJ lead peer, Spring AOP will unleash AspectJ and @AspectJ that I spent quite some time on this year as per our engagement to merge best of breed AspectWerkz with AspectJ.

But going beyond, Spring 2.x also introduces a way to define aspects in your spring bean xml files directly f.e.:

This is actually a really nice feature that I started playing with 2 years ago when implementing AspectWerkz with Jonas. So there is not much innovation there.

There might actually be some issues and hard work going forward. By having aspects defined this way, the AspectJ runtime that is used when your are using the excellent AJDT Eclipse Plugin for AspectJ will not take those into account (whilst regular @AspectJ ones are fully supported).

This means there 'll be definitely a need to open-up some more AJDT and have some extension point in it so that the Spring IDE can leverage it and have the AJDT crosscutting view display the Spring xml defined aspect.

Wups - xml defined aspect I have said - just like this June 2003' Jonas article introduces, back in AspectWerkz 0.6.3

Maturity - finally!

JRockit powered AOP prototype available

2005-10-20T11:22:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

It's there !

I am very pleased to announce the prototype version of JRockit that includes our AOP API is available. It will drive you far far away from current bytecode instrumentation techniques to a world of plain Java API, complete runtime control for hot deployment and undeployment, agnostic from the programming model you like for your aspects.

If you are involved in the AOP field, bytecode instrumentation field, APM field, and did not heard from us recently, please drop me an email and I 'll consider your participation in evaluating this technology to help us drive it to reality.

You can read the official announcement and some background information in our BEA dev2dev dedicated forum.

Synchronized block join points (v2)

2005-10-10T13:30:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Back in July Jonas suggested some semantics for a synchronized block join point. The discussion was interesting but I am wondering if we actually came up with the right question.

What if we simply think in terms of "is this type listening to locking events ?" instead of trying to come up with a pointcut expression that defines the semantics of a synchronized block - as done in Jonas discusion?

Lets draft some code:

Consider the program:
class Bar { synchronized void foo() { .. } static synchronized void statfoo() { .. } void stuff() { .. synchronized(bar) { .. } } }

So the deal is to listen to lock events - mainly:

waiting the lock

just acquired the lock

just released the lock (or perhaps about to release it - but this does not matter much for this discussion)

I argue that instead of defining semantics, the issue should be narrowed to a type pattern ie something as simple as Bar (or more fancy variations based on type hierarchy or annotation - you bet it).

Given that we can define this interface:
interface Lockable { void waiting(); void acquired(); void released(); }

As a user you can then implement this interface on your own types, use some sort of AOP or proxy to have it introduced where you want, etc.

So what should happen to make that work?

First based on the type pattern, we simply introduce Lockable to Bar. This can be done manually, by the mean of AOP if f.e. I was writing @Distributed class Bar { .. } and so on - depends on the use case.

Second the interface must then be implemented by Bar.
The user can provide one if implemented manually, or the system (AOP f.e.) can provide an empty one ie something like (depends on the AOP system but you get the idea)
@Introduce("Bar") class LockableNOOP implements Lockable { void waiting() {} void acquired() {} void released() {} }

Last we need to transform the program so that the synchronized block or the Bar' synchronized method are triggering event to that.
This is not that hard since we just need to look at the type hierarchy for non static synchronized method and given that an instance synchronized method can be rewritten (by the system) as a synchronized(this) block we end up to something like that:

Replace all synchronized blocks like that:

Given
synchronized(stuff) { .. }
to
if (stuff instanceof Lockable) { Lockable lockable = (Lockable)stuff; lockable.waiting(); synchronized(stuff) { lockable.acquired(); ..// unchanged body } lockable.released(); } else { synchronized(stuff) { ..// unchanged body } }
Fairly easy.

So far, nothing will happen, as we have a LockableNOOP implementation, unless the user (you) explicitly provided some implementation.

Last part: configure the system so that it advises Lockable' methods, so that you can add your behavior there (f.e. distribute lock, or trace them for some application performance system). There, usual AOP stuff can be used.

You can now access the enclosing static join point information if you need - which gives you if you really need it, the rich semantics Jonas was looking for:
call(Lockable.waiting()) && withincode(....) && target(t) && cflow(...) ..... // t is the locked object

There are two interesting things to solve now:

What if I want to kick out the synchronized() block ?(f.e. for distributed lock). I think a small variation of Lockable can solve that - f.e. boolean shouldUseJavaLocks(), and a tiny change for the transformed program.

What about the static synchronized statfoo() {..} in Bar? For that one, the transformation would have to be a bit more compex so that we f.e. lock on an introduced static field. That said, the Lockable interface is not handy anymore. Any suggestion for that one? Should we define three methods like static void lockable_waiting() that ones would implement or introduce to adress this use case?

On a more high level perspective, this send us back to a recurrent set of problems in the AOP / bytecode transformation space:

to achieve API transparency, we have a very intrusive transformation engine (though the one described here can be quite fast)

by changing the bytecode we break the visibility another system may require to work properly (as f.e. we add if blocks, change synchronized method into synchronized blocks etc).

There is thus a set of open questions:

So should that belongs to the JVM directly, and in which form?
(as f.e. JVMTI already provides monitor entry / exit events, that should be fairly easy).

Should those 3 Lockable' methods (or 6 if we include the static versions) belong to java.lang.Object direclty?

Would an hybrid system that changes the synchronize blocks using bytecode transformation but then relies on JVM level support for AOP such as we do in JRockit be enough to listen for lock events?

Is the cost of the introduced instanceof acceptable?

Happy thinking!

How will nextgen AOP looks like?

2005-09-28T11:08:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

This last days I have been thinking some about how next gen AOP implementations may look like in say 4 years from now in the Java landscape.

As you may known, 5 years ago AspectJ was not based on bytecode manipulation but was based on source code transformation.
As of today, full blown AOP like AspectJ and JBoss AOP are usually build on top of really heavy bytecode manipulation, while more lightweight approach like Spring AOP are build on top of proxy (which under the hood can use bytecode generation technology ie JDK or CGlib proxy).

In the next few days we will start spreading the prototype implementation of AOP support in JRockit JVM which allows to implement all of the AOP interceptions features without even touching the bytecode but with a more abstracted subscription based API. If you missed the dev2dev series you can read more about it there.

This technology is definitely a breakthrough in the field, but does not address the introduction/mixin features ie capability to add method / field / annotation / change the object hierarchy of the object model in a crosscutting manner.
This means that f.e. to implement AspectJ as it is on top of the new JVM based technology we need to wether

remove the feature from AspectJ - this is dumb as introduction are an important part of AOP

do that in the VM - well, perhaps but unlikely from my perspective as it would mean changing the Java language itself else ones would not be able to call the introduced members from regular code

use an hybrid bytecode / VM based AOP approach - why not but it looks a lot like going back to the old bytecode based implementation complexity

do it in another way!

My thoughts on that is that we will likely do it in another way - and perhaps going back to source code transformation. This year Sun' folks presented the Jackpot project at JavaOne - which is sort of a rule driven source transformation engine. If that ones goes mainstream somewhere with Java 7, ones will have a fairly great technology set to rethink the way we implement AOP in Java, mixing nextgen APT (that will expose the full AST), Jackpot and JVM weaving.

AOP weavers - are we doing it wrong?

2005-08-10T12:18:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

These last days I have been prototyping around an interesting idea.

As you know we have been working on an API to add JVM support for AOP in JRockit. The prototype will be available any time soon.

The nice thing about it is that you don't manipulate the bytecode anymore and that you are using only well known java.lang.reflect.* API to tell the JVM if your pointcut is matching or not. This is somehow similar to what ones can do with Spring AOP - as this one is proxy based (see f.e. MethodMatcher API in Spring).

The immediate benefit of it is that first there is no need to have another in-memory representation of classes beeing weaved (before they get loaded) that is backed by some expensive (both memory and CPU) bytecode analysis.
This advantage is detailled in our JVM support for AOP part1 article.

As a consequence it is really easy to query the method annotation, generics properties, and such - something that is extremely complex to achieve with acceptable overhead in regular bytecode based weaver such as AspectJ or AspectWerkz (actually more complex than changing some bytecode instruction).

So what is that idea I had ?

Having AOP support in JRockit is nice, but it will take some time before that gets mainstream (with eventually a JSR etc). In this transition period, there must be a way to implement a better bytecode based weaver that will perform way better than current weavers (both AspectJ and AspectWerkz), that will be easier to implement, and whose only requirement is Java 5 (well off course, it won't be as good as JRockit JVM support for AOP - so it is still a transition technology).

I have thus been sketching on an hybrid system that makes extensive use of the hotswap API, and whose actual weaver relies only on pointcut matching backed by the java.lang.reflect.* API ie does not build any kind of equivalent structure backed by bytecode analysis.

As such the memory overhead is zero, and the CPU overhead is way less than current AspectJ and AspectWerkz.

The overall idea is quite simple and consists in 2 phases:

Phase 1

A first weaver is changing the bytecode in some stable way - such that all classes are transformed (lets say prepared) the same way - while not introducing any dependancies on any kind of AOP, and while not adding any kind of performance overhead (ie no changes in the execution flow such as introduced by wrappers method and such usually used when implementing instrumentation needed for around advice).

All classes thus get loaded as expected with a very limited time overhead and no memory overhead at all (thanks to the excellent ASM performance).

Phase 2

Then when an actual class gets loaded (as per regular application behavior) I get a small callback invoked when this class has just been loaded and just before anything else happens (ie the class static initializer invoked by the JVM). Current load time weavers do things "just before the class gets loaded" and as such cannot access the java.lang.reflect representation of what they weave.

This callback can then perform the actual weaving by relying entirely on the java.lang.reflect.* API to match the pointcuts. It then constructs the instrumented version of the class and hotswap it thanks to the Java 5 API. The JVM eats this one and goes on.

The first prepare phase is needed as the hotswap API does forbids change in the schema (ie cannot add or remove methods or fields).

A nice extra side effect is that at any point in time any class can be exposed to the AOP layer thru a simple API. This can be handy for some cases where there are some circular references in the dependancies (f.e. the instrumentation needs the java.lang.reflect.Method class to match against the pointcuts so if I want to add aspects to the Method class, I cannot do it until I have a representation of this class ie there 's no way to have it working by simply invoking the callback from the prepared Method class itself).

This might seems like gory details, especially when compared to what we do in the JRockit JVM support for AOP, but I am sure there are some concepts worth digging there - as memory usage and weaving overhead as been reported more than once as a major problem (see f.e. use case with AspectJ reported by Ron Bodkin where the system takes 4 minutes to start instead of less than a minute without any weaver here)

This makes it also very easy to add aspects even to java.* classes (though it's not generally a good idea unless you know what you are doing).

This sample is an actual code snip of the actual AOP transformation (part of phase 2). As you can see it relies on the java.lang.reflect API.

public MethodVisitor visitMethod(int access, String name, String desc, String signature, String[] exceptions) { if (!filter(access, name)) {//fast filter for f.e. clinit method as we look for method execution join points // see if we get a match for this join point // only java.lang.reflect.* API is used here Member method = ReflectQuery.getMethod(m_klass, name, desc); Class thisClass = m_klass; Class targetClass = null; Member withinCode = null; //do matching if (match(method, thisClass, targetClass, withinCode)) { //change the bytecode but don't change the schema for hotswap purpose } else { return super.visitMethod(access, name, desc, signature, exceptions); } } }

Finally I must say this idea is not 100% new as I wrote a paper on it for AOSD 2004 (read my paper here). At that time though I was not doing the pointcut matching based on the java.lang.reflect API - as the purpose was to enable dynamic AOP in AspectWerkz 1.0) ie I was not solving the problem of the memory and CPU overhead of the actual weaver.

What do you think? Are those ideas worth digging more?

JVM support for AOP in BEA JRockit

2005-08-02T09:34:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

We have plublished a follow-up to our JavaOne 2005 session that describes with some more details the JVM support for AOP that is beeing designed within the BEA JRockit JVM.

This first article introduces the problems that usually happen with current weavers (in the Java land) and briefly described the proposed solution.

The next part to appear in the following weeks will give more details and code samples.

Read the article

JRockit JVM Support For AOP, Part 1 by Jonas Bonér and Alexandre Vasseur, Joakim Dahlstedt -- AOP is all the rage, but how do you implement it? In this article, Jonas, Alexandre, and Joakim show that the current approaches to implementing AOP suffer from many problems, making scalability an issue. Moreover, they indicate that the traditional approach to aspect weaving duplicates efforts that the JVM already performs.

We are currently working on making the prototype implementation available for further evaluation.

If you could not attend JavaOne 2005, the slides are available from the JavaOne web site, and they are also available here

The full webcast of the JavaOne session is also available on dev2dev.

AspectJ 5 load time weaving with Java 1.3 using ... AspectWerkz

2005-08-01T14:48:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)


**** Introduction

As development of AspectJ 5 and the merger with AspectWerkz making good progress, several users have started to wonder how the load time weaving under Java 1.3 / 1.4 VM be enabled.

Despite the name, AspectJ 5 is not at all tied to Java 5. I have already explained in this post how to write plain Java aspects using JavaDoc annotation and Backport175.

In this post I 'll explain how to enable load time weaving for Java 1.3/1.4 for AspectJ... by using AspectWerkz ! ie that AspectJ will sit on top of the very low level layer of AspectWerkz.

**** Some background

In AspectWerkz we enable load time weaving to happen thru a wide range of options that user can choose:

Java 5 agent

JRockit specific integration with JRockit agent

bootclasspath family

hotswap family

I have integrated the most important ones in the AspectJ 5 code base already - ie the first two - but some of them won't be integrated - hence this post.

When running Java 5 or JRockit (java 1.3 / 1.4 / 1.5) you don't need to read further and can stick to what's provided out of the box in AspectJ 5. Else... continue reading.

The AspectWerkz low level layer named aspectwerkz-core is actually a generic load time weaving layer. It comes with one interface that one has to implement - very similar to the JSR-163 instrumentation agent: the org.codehaus.aspectwerkz.hook.ClassPreProcessor.

This core layer comes with a set of tools that allows to turn this one on into the environment. The most easiest to use is what we called the "prepared bootclasspath" where you basically run a little script that will patch the java.lang.ClassLoader to hook in the agent, and will give you back a jar. This jar file can then be used first in the classpath when you start your JVM so that the agent gets called.

You can read more about all the options in the AspectWerkz doc here.


**** Running AspectJ by using AspectWerkz for load time weaving

I describe here the implementation of the agent, but you can use directly the one I ship with this post if you are not interested in the details. You can assume this one is LGPL, as AspectWerkz is.

The implementation of the ClassPreProcessor to use AspectJ on top of AspectWerkz core is straightforward and looks like that:
package org.aspectj.ext.ltw13; public class ClassPreProcessorAdapter implements // implements the AspectWerkz core interface org.codehaus.aspectwerkz.hook.ClassPreProcessor { /** * Concrete preprocessor we delegate to * This one sits in org.aspectj.weaver.loadtime.* */ private static ClassPreProcessor s_preProcessor; static { try { s_preProcessor = new Aj(); s_preProcessor.initialize(); } catch (Exception e) { throw new ExceptionInInitializerError("could not initialize preprocessor due to: " + e.toString()); } } public void initialize() { ; } public byte[] preProcess(String className, byte[] bytes, ClassLoader classLoader) { // skip bootCL if (classLoader == null) { return bytes; } // skip AJ weaver well know stuff to avoid circularity if (className != null) { String slashed = className.replace('.', '/'); if (slashed.startsWith("org/aspectj/weaver/") || slashed.startsWith("org/aspectj/bridge/") || slashed.startsWith("org/aspectj/util/") || slashed.startsWith("org/aspectj/apache/bcel") || slashed.startsWith("org/aspectj/lang/") ) { //System.out.println("SKIP " + className); return bytes; } } // do the weaving using AspectJ weaver return s_preProcessor.preProcess(className, bytes, classLoader); } }

By using the AspectWerkz Plug utility we generate our load time weaving enabled java.lang.ClassLoader in awaj-boot.jar (see AspectWerkz doc for other options, that might be more license friendly if you care). This can be done thru Ant:

The last step consists in starting the JVM with this one, passing in a specific option to say what is the ClassPreProcessor implementation we want to use (the default one beeing AspectWerkz).
Starting a sample with Ant will thus looks like this - the important details are int he jvmarg option
.....

**** Sample project

I am attaching the complete source code, along with a sample Aspect in a zip.

This sample further use Backport175 so that the project is 100% Java 1.3 - without any of the AspectJ specific extra keyword to defines aspects that would disturb your regular javac compiler.

Off course, this load time weaving can be used for any kind of aspects, so adapt it for using AJC if you have some .aj files to compile.

Note: this zip does not include AspectWerkz jars, Backport175 jars, and AspectJ jars needed. You will have to get those and fix the Ant build.xml for your environment. See the build.xml file in the zip. (the sample is also refering to AspectWerkz 2.1RC1 - so change it to AspectWerkz 2.0 / sorry about those minor details).

It may also happen that you will need a SAX XML parser has Java 1.3 does not ships one. You may read more about that in the AspectWerkz FAQ for example
here (read "Does AspectWerkz support Java 1.3?"). This is not contained in the sample project neither.

Get AspectJ load time weaving on Java 1.3 enabled by AspectWerkz !

You will need AspectWerkz 2.0 for it. Get it there.

To run the sample, you will also need AspectJ and Backport and the Backport175 AspectJ extension. See build.xml on how to get those.

Opt-out AOP: good or evil ?

2005-07-13T13:02:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

There is an interesting option in AspectJ that is named -Xreweavable.

When turned on using any kind of compilation/weaving mode (compiling with ajc, within Eclipse AJDT, doing bynary weaving, or using loadtime weaving), each weaved class keeps track of

the aspects that are affecting it

its state before the weaving happened (ie its bytecode without aspects)

This option is very handy when you plan to weave more than once your application (because f.e. this is a library that you distribute), and when you want to make sure everyone will be able to add some more aspects in there.

Actually, discussions are taking place if this should be the default.

An interesting consequence is that it is fairly easy to implement an opt-out AOP engine that simply restores the state prior to weaving, and thus kicks out all the aspects from the application !

There are of course realistic use-cases for it :

sanity check if production is very scared about use of AspectJ, knowing that the developpers team is using it for their own needs (declare error / warning f.e. and tracing in QA stage etc)

obtain some more info about which aspect is in, and is affecting the application, to increase the trust everyone gets in using the technology

introspect some third parties libraries and see if they actually use aspects

add some more reporting features on the go, to be able to have at any time the list of aspects that are in the system f.e. thru some sort of web console

enforcing that some key section of the system are not allowed to be advised by anything, or only by some sort of well know and certified aspect(s)

A small amout of code is required for it, the trick is mainly to add another Java 5 agent or loadtime weaver that will check for this reweavable state and do the reporting.

This is actually very easy to do when using f.e. AspectWerkz-core as the backend to hook this one in and have it work as well on Java 1.3, and using ASM for the bytecode details.

Here is a sample output (I am using AspectJ loadtime weaving to do the weaving first, but that could be done using post compilation with AJC, or compilation from within Eclispe AJDT etc)
// in this sample, start the app with AspectJ loadtime weaving (could have been compiled with ajc instead) // and pipe with the UndoAspectJ opt-out engine using the AspectWerkz core // as a backend #java -javaagent:aspectjweaver.jar -Daj5.def=test/aop.xml -javaagent:aspectwerkz-jdk5-2.0.jar -Daspectwerkz.classloader.preprocessor= org.aspectj.ext.undoaspectj.ClassPreProcessor test.undoaspectj.Sample weaveinfo Type 'test.undoaspectj.Sample' (Sample.java:28) advised by around advice from 'test.undoaspectj.Sample$TestAspect' (Sample.java) // here is the opt-out message UndoAspectJ - test/undoaspectj/Sample was affected by test.undoaspectj.Sample$TestAspect // execution without any aspect takes place: Sample.target

// without the opt-out we would see the aspect executing: weaveinfo Type 'test.undoaspectj.Sample' (Sample.java:28) advised by around advice from 'test.undoaspectj.Sample$TestAspect' (Sample.java) // execution with aspect takes place: Sample$TestAspect.around Sample.target

Though the first idea is quite odd and counter productive, this illustrates that ones could easily implement a security layer on top of AspectJ without even going deep in the AspectJ code base, to enforce security constraints about the use of AOP in the system and in the deployments at large.

So is opt-out AOP good or evil ?

@AspectJ in AJDT - a world premiere !

2005-07-05T13:32:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

I am happy to announce that the @AspectJ support within Eclipse AJDT will appear in the next build snapshots, ie very soon !

That is an important step that brings to the AspectWerkz and AspectJ merger all its meaning, by providing a plain Java syntax (based on annotations) to write AspectJ aspects (aka @AspectJ) while still providing excellent support for it in Eclipse AJDT plugin - just as it exists for the well know AspectJ style (code style).

In the list of things to already expect, that will officially ship as of AspectJ 1.5 M3 in the following days:

error checking for pointcuts (even though those looks like strings within an annotation)

warnings when @AspectJ annotations are misused (f.e. used in a class that is not annotated with an @Aspect annotation etc)

advised by view

advises view and cross-reference view

and off course AJC integration (ie weaving happens behind the scene as compilation is done)

As a world premiere, here is a snapshot of an @AspectJ aspect in AJDT !

For those of you using AspectWerkz, you may wish to give it a try very soon, as it is already much better that the Eclipse plugin I had written for AspectWerkz, and as it will help you to get started with the @AspectJ syntax in your favorite IDE.

Start your download engines to get the next build snapshots and the upcoming AspectJ 1.5 M3 with your fresh Eclipse 3.1 final !

(Many thanks to Andrew and Mik for answering my questions to get that bits working)

AspectJ, @Aspect and Java 1.3

2005-06-23T11:31:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Following the Backport175 release that Jonas and I have done some days ago, and since AspectJ 1.5 upcoming M3 and especially the @Aspect style is starting to take shape after some month of hard work on my side, I think it is a good timing to also explain that this @Aspect sytle, despite its name tied to these Java 5 annotations, is not at all tied to Java 5, and will bring the plain Java aspect concept to all of you using Java 1.3 and 1.4 (from the real world).

The @Aspect style is also referred as @AJ, or annotation style, or this way to write aspect with annotation that we came up with thru AspectWerkz.
It mainly means that AspectJ allows you to write an aspects using wether the well known code style or the AspectWerkz like annotation style.

// code style public aspect SomeAspect { void before() : execution(* Foo.bar()) { // do some stuff // use thisJoinPoint etc } }

// annotation style, here with Java 5 @Aspect public class SomeAspect { @Before("execution(* Foo.bar())") void before(JoinPoint thisJoinPoint) { // do some stuff // use thisJoinPoint etc } }

And since Backport175 brings annotations to Java 1.3 while preserving the bytecode format that the Java 5 specification describes, it seamlessly allows you to use the very same AspectJ version to write @Aspect on Java 1.3 as well (but please don't call it the doclet style!)

// annotation style, backported to Java 1.3 or 1.4 /** * @Aspect */ public class SomeAspect { /** * @Before("execution(* Foo.bar())") */ void before(JoinPoint thisJoinPoint) { // do some stuff // use thisJoinPoint etc } }

For the Java 1.3 to work, you just need an single little jar that contains the AspectJ defined annotations (like @interface Aspect) backported to Java 1.3 in the form of regular interfaces (ie interface Aspect) and then make sure you have those classes first when doing the weaving.
Aside, since AJC (the AspectJ compiler) is not yet integrated in the doclet parsing pipeline, you need to follow a binary weaving strategy ie:

develop the aspect and the application with Java 1.3 (remember it's plain Java so you can use IntelliJ if you want)

compile the sources with regular javac

post-compile with Backport175 to handle the annotations

binary weave with AJC

That sounds a bit cumbersome but some simple Ant script is of great help. Aside Backport175 comes with plugins for Eclipse and IntelliJ so you end up with just the binary weaving step - that is rather common.

I have made this little AspectJ extension available here (that can be used with Java 1.3)

You can also grab the source code for it, that includes a small demo here.

Just set aj.root in the build.properties to the folder that contains aspectjrt.jar and aspectjtools.jar from a recent nightly build (will work for M3, does not for M2, so use a recent one!).
Then run ant clean test.

Check the build.xml file to understand how the different build steps are performed.

In the IDE it will also popup pretty well. See for example the little green
@ signs in the margin in this IntelliJ screenshot.

So now we have AspectJ @Aspect straight in IntelliJ.
Off course, you'd better use Eclipse AJDT to benefit from the rich user experience that it provides when dealing with software modularity thru AOP and AspectJ.

Using Backport175 also allows to use custom annotations in the pointcuts under Java 1.3 to define stronger contract between the aspect and the application, as Adrian describes here as the Holy Trinity (when you further add dependency injection)
(the demo don't include that but I let you try it).

In a follow-up post I will also explain how the new AspectJ enhanced load time weaving ala AspectWerkz can work with Java 1.3 and 1.4 as well - since it has been a recurrent question.

JavaOne 2005 - AOP, distributed Java, and more

2005-06-22T18:06:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

I thought it would be interesting to post what kind of session and BOF I am planning to attend during JavaOne 2005 next week.

There are indeed interesting trends going on, and needless to say I don't really want to hear that the EJB (3) XML deployment descriptors are back or that kind of things. Instead, AOP, Annotations and distributed architectures will get my interest.

BOF-9888, Chasing 9s: Modeling and Measuring Availability of J2EE™ Applications (Sun, monday 7h30 PM)

TS-7659, Runtime Aspects With JVM™ Support (BEA JRockit off course, Tuesday 11h AM - don't miss it ! and meet us at the BEA and Eclipse booth)

TS-7695, The Spring Framework: Introduction to Lightweight J2EE™ Architecture (Interface21, Tuesday 11h AM)

TS-7212, Clustering, Consistency, and Caching: An Implementor's View of JSR 107 [JCache] (BEA, Tuesday 01h30 PM)

TS-7159, Java™ Platform Clustering: Present and Future (Sun, Tuesday 2h45 PM) (don't miss it, it is a sneak from Sun research labs, about transparent clustering)

BOF-9385, Apt Usage of APT: When and How to Use the Annotation Processing Tool (Chariot Solutions, Tuesday 7h30 PM) (I know they will give a quote to AspectJ)

BOF-9161, Exploring Annotation-Based Programming Through the APT and Mirror APIs (BEA/Sun, Tuesday 9h30 PM) (ha ! Annotations that drove AspectWerkz for a while)

BOF-9441, Practical Application of Aspects in Everyday Development (ALTERthought, Tuesday 10h30 PM) (AOP from the users - an evidence of adoption phases as regards AOP)

TS-5471, Jini™ and JavaSpaces™ Technologies on Wall Street (JPMorgan, GigaSpaces,others, Wednesday 11h AM)

TS-7429, Speculative Locking: Breaking the Scale Barrier (Azul, Wednesday 4h PM)

TS-7410, Network Attached Processing: Tapping 384-Way SMP, 256GB Java™ Technology Compute Bricks (Azul, Thursday 1h15 PM)

TS-7339, Data Grids for J2EE™ Platform Cluster Deployments (GemStone, Thursday 3h45 PM)

See you there !

JRockit AOP and AspectJ 5 and JavaOne 2005

2005-06-07T18:34:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

JavaOne 2005 will be a good opportunity for you to have an update on the
AOP field. Last year we presented AspectWerkz plain Java AOP and its
J2EE integration facilities. This year, several sessions will include
AOP related sections and give testimony of interesting use cases, and
as Jonas recently announce we will for the first time present
the JRockit JVM support for AOP.

Make sure you attend this technical session on Tuesday June 28 (TS-7659, "Runtime Aspects With JVM Support") and have a
sight at what's next in the AOP galaxy. If you can't make it, visit us
at the BEA booth where we will be eager to show you the JRockit JVM
AOP support thru live demos.

You can also visit us at the Eclipse booth where we will show latest
AspectJ 5 and AJDT versions. This includes plain Java AOP and
load-time weaving ala AspectWerkz that I have been working on for some
month since we merged with AspectJ, and many new things regarding Java
5 support in AspectJ 5 (annotation matching, generics etc)

Meet you there !

Java Open Source playing with sharks

2005-06-01T12:14:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

As a person largely involved in the Java Open Source landscape, I have been observing radical changes during the last 5 years in this area, and the 5 years birthday of Struts makes me comment it some.

Five years ago, I was starting my own Struts clone for PHP to try to bring an MVC model to the rather new PHP OO stack I was using on a customer site. This project ended up pretty much nowhere (e-php on SF) since, well..., I could luckily do something way more interesting.
But back at that time it was rather simple to start a small Open Source project and try to have it grow. Some were actually even considering you as a strange beast when you happened to be part of that world, one of my former employer beeing the first.

At that time, I also remember sales folks arguing that a Java developer had to have "fluent in Struts" in their resume to actually be a good Java Joe developer (before that you had to have "fluent in Javascript" in their mind, so we had made a big progress for the IT consultants thanks to Struts ;-) )

Most of the (good) Java Joe developers where using Apache projects from the Jakarta line mainly as their sole Open Source use in corporate projects, and Tomcat was pretty much the sole Open Source container that you could find in production.

At that time JBoss started to change the game by grabbing several Open Source fames and projects on board (Tomcat, and then Hibernate to name the most famous).
Back in 2002, it became fairly common to see "JBoss knowledge is a plus" in job descriptions for IT consultants. Open Source had started to play with sharks, and you had to play with it, following the "fluent in Struts" move.

Next in the pie, Eclipse went bigger and bigger, and Open Source projects that were used and presented in conferences were more and more backed by someone that could afford it.
At that time I boosted AspectWerkz by joining Jonas' first iterations, all that on my spare time, unpaid, for the fun of it, as most of the OSS committers, bringing my little Open Source load time weaving framework that I had started some month before (beSee on SF).

AOP and IoC started to gain in popularity, and Spring was doing its first moves. It became common to see "fluent in Hibernate" in job description, and I remember the first time I actually noticed it was in february 2003.

I have been pushing the AOP ball since that time as part of my BEA job, and Open Source was more and more playing with sharks. AspectWerkz was backed by BEA, AspectJ revealed to be largely backed by IBM as an Eclipse technology project, and sucessfull projects that you could hear about in conferences where almost all backed by a shark - be it a big open source community like Apache, Eclipse, ObjectWeb that can afford its own events, or by a commercial vendor like BEA, IBM or JBoss.
Even more, some projects happens to be backed by more than one shark. AspectWerkz and AspectJ is an evidence of it thru the AspectJ 5 merger.

Back in 2004 the Spring fame became a shark by its own thanks to a well shapped service model around Rod' Interface21, that obviously adressed (and still adresses) Java Joe developer needs.

Last year I was wondering when I would see a "fluent in Spring" requirement in an IT job description, and it happened this morning, on the same line as "Tomcat, JBoss, Hibernate" for a "challenging J2EE architect opportunity" (interested?).

I have the feeling that Open Source will not be anymore this hectic group of people working for free for the fun of it as that sound to be back in 2000. Open Source is becoming more and more a distribution channel or a big thing to take into account when thinking about return on investment. If you look closely at some open source projects, you 'll also realize that it is rather common that 80% if not more of the workload is handled by a group of people sitting on the same open space.

The recent deal (backed by concrete M$) done around Geronimo (project that was already backed by Apache) is yet another evidence. Open Source is playing with sharks, so you'd better love that. And if you do want to kickstart an Open Source project, you'd better think which shark will play with you if you want your jewel idea to sustain.

As a last evidence, almost every single IT service company in France finally spinned of an official department dedicated to Open Source, bridging the gap with "SSLL" models (french acronym for open source service provider, doing consulting only on OSS stacks) . I have noticed this trend those last few months (Devoteam, Atos to name a few).
This not to comment on the Eclipse phenomenon, with almost all software vendors joining it and shaping their product line in a new more or less anticipated direction.

Open Source is not anymore the thing you use to maximize revenues, or to make sure you recruit uptodate brains. It's the thing you have to play with straight within your strategy, no matter if you are a software vendor or an IT service company.

So what will be the next Open Source moves in the Java are(n)a ?
Happy Open Source thoughts !

(note: dates in that post are those I remember and might be approximations. Aside, the job opportunities I am talking about are all for the French market, so you may expect some delta)

EJB 3 and AOP: the EJB interceptor dilemna

2005-04-14T16:50:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

In this post I present an implementation a subset of the EJB 3 specification (JSR-220) : the EJB interceptors. As of today no EJB 3 preview brings an implementation that seamlessly integrates with AOP, despite the concept similarities.

The proposed implementation is fully runnable out of any container, and introduce a specific extension to allow use of pointcut in EJB interceptors, thanks to AspectWerkz extensible AOP container.

Wants to know more about EJB 3 interceptors, and how they are closed to AOP or different from AOP ? Read more.

Should we consider EJB interceptor as AOP ?

As you may know, the EJB 3 specification (JSR-220) defines

Interceptors for EJBs (stateless, statefull and message-driven). If it might be a good idea to spread an answer

for the need of addressing cross-cutting concerns in J2EE and EJBs in particular, it might be a bad idea to do it for

those of us (more and more numerous) familiar with AOP implementations like AspectJ and

AspectWerkz, especially because huge limitations of what the specification

allows us to do with those interceptors - at first sight and excluding any vendor specific extension.

The most anti-AOP concept in the specification is that the EJB that wants interceptor have to declare it explicitly

using a @javax.ejb.Interceptor or @javax.ejb.Interceptors class level annotation - which breaks the obliviousness of

aspects. That in favor of making things explicit, which might be good for J2EE users.

Further on, the EJB bean itself can have a method that intercept its own business method ie the bean is the

aspect (that sounds like marketing isn'it ?). Well. That's a specification and is interesting in the sense that it

democratize a technology.
So what can we do to democratize AOP as part of the EJB 3 specification ?


@javax.ejb.Stateless
@javax.ejb.Interceptor("test.ejb3.MyInterceptor")
public class MyEJBIsTheAspect {

    // business method
    public int businessSum(int i, int j) {
        return i + j;
    }

    // interceptor method within the bean (the bean is the aspect)
    @javax.ejb.AroundInvoke
    public Object interceptMySelf(InvocationContext ctx) throws Exception {
        System.out.println("--> MyEJBIsTheAspect.interceptMySelf");
        System.out.println("  method: " + ctx.getMethod());
        for (int i = 0; i < ctx.getParameters().length; i++) {
            Object o = ctx.getParameters()[i];
            System.out.println("  args["+i+"]: " + o);
        }
        return ctx.proceed();
    }

}

State of the art (not that much..) in JBoss and Oracle AS EJB 3 preview

After having a look at JBoss EJB 3 and
href="http://www.oracle.com/technology/tech/java/ejb30.html">OracleAS EJB 3 previews, I was even more disapointed.

Both of them are using a reflective based approach to invoke the interceptors. This means that

the performance of the interceptor will be bad, and that a
href="http://www.aip.org/history/heisenberg/p08.htm">Heisenberg effect will be inevitable and actually fairly big

(no wonder that ones will use interceptor to gather performance metrics and thus as soon as you observe the bean, you

are observing a different things than what actually happens).

Ones may say this is a microscopic view. It is. But when thinking about EJB 2 stories in the past a sound idea would be

to make sure we don't waste resources where we can avoid it. And thinking about JBoss history around AOP, that's rather

suprising that the EJB 3 interceptors are not cleanly integrated in their AOP framework. I personnaly consider that we

have enough technology around to make it far better, and far more consistent with AOP. Given the impact that AOP will

continue to have, ones would better figuring out how to do that now with EJB 3 - assuming that EJB 3 succeeds.

AspectWerkz extensible container value proposal to EJB 3 interceptors

I decided to give it a try with the AspectWerkz extensible container. As Jonas Bonér described it in a
href="http://www.theserverside.com/articles/article.tss?l=AspectWerkzP1">TSS article, AspectWerkz can be considered

as a generic AOP runtime platform, in which ones can hook in any kind of AOP/AOP like programming model. AspectWerkz

aspects are one, AspectJ aspects are another, and we have implementations for Spring AOP and AOP Alliance aspects. An

AspectModel answers three essential properties: what is the life cycle of the aspect, how to invoke

it and how the programming model exposes the closure with wich the user proceed.

It happens that the EJB 3 interceptor can be seen as a very simple AOP programming model in two ways:

ones can define interceptor class. An interceptor (advice) is thus a @javax.ejb.AroundInvoke annotated method with a

specific signature (no interface, no mandatory method name) within a class - the interceptor class (aspect).

ones can define a @javax.ejb.AroundInvoke annotated method (advice) in the EJB itself (aspect): the bean is the

aspect !

the closure is defined with the interface javax.ejb.InvocationContext


public interface javax.ejb.InvocationContext {
    public Object getBean();
    public Method getMethod();
    public Object[] getParameters();
    public void setParameters(Object[]);
    public Context getEJBContext();
    public java.util.Map getContextData();

    public Object proceed() throws Exception;
}

The value proposal I am bringing here with the AspectWerkz extensible container is the following:

no reflection at all. See performance figure f.e at our
href="http://docs.codehaus.org/display/AW/AOP+Benchmark">benchmark site

seamless integration of the interceptors with the aspects. They are made "aspect" thanks to AspectWerkz and the

other aspect model (AspectWerkz aspect, AspectJ aspect, whatever aspect model registered in the runtime) coexists

nicely

easy points for vendor specific extension:
- real pointcut to narrow the matching of an interceptor method to a very precise set of bean methods
- life cycle control for the interceptor class
- supporting cflow(), args() and alike AOP semantics
- introducing high performant hot deployment and undeployment of EJB interceptors

"out of container" runtime available

rich integration patterns: application preparation (like ejbc / appc for faster deployment) or seamless

deployment

AspectWerkz extensible container details

To better understand how things will looks like, you need some background in AspectWerkz:

The first key part is "org.codehaus.aspectwerkz.transform.inlining.spi.AspectModel". We will provide two

implementations of it. One for interceptor class (the interceptor class is the aspect), and one for bean as

interceptor (the bean is the aspect).
The second key part is in integrating the EJB interceptor closure "InvocationContext" that defines the "proceed()"

method with the more general idea of "Joinpoint.proceed()" of the runtime. In short, this means that the interceptor

will be entirely part of the aspect chain, and if there is a transaction aspect to handle EJB transaction boundaries,

they will share the very same chain - as ones expects - thus making it easy for the implementation to organize

precedence rules (as define in section 3.5.1 of JSR 220 for example).
As we already wrote this transaction aspect for EJB 3 in a previous
href="http://www.theserverside.com/articles/article.tss?l=AspectWerkzP2">AspectWerkz TSS tutorial, that's rather a

simple idea and basic requirement.

The runtime will take care of aggregating the models depending on which aspect apply to which join point, each aspect

having its specific AspectModel and each AspectModel implementation being responsible for generating what it needs:


// a closure to deal with the join points
// ie models org.codehaus.aspectwerkz.joinpoint.JoinPoint interface
// to fit with the transaction aspect, or any other aspect
// and javax.ejb.InvocationContext to fit EJB3 interceptors
 class jitEJB3Closure implements JoinPoint, InvocationContex 
 {
   
    // the closure hosts state, optimization makes it more complex than that

    // the target of the join point ie the intercepted ejb
    // which is also third aspect: remember the bean is the aspect !
    private MyEJBIsTheAspect target;

    // first aspect in the chain, statically compiled
    private TXAspect aspect_0;

    // second aspect: the interceptor class
    private MyInterceptor aspect_1;          
 
    // details skipped that makes those fields initalized as they should.

    // as defined in javax.ejb.InvocationContext
    public Method getMethod() {...}            

    // as defined in JoinPoint
    public Class getTargetClass {...}          

    // other methods as per each aspect model affecting the join point
    ...

    // generic proceed() method as defined in JoinPoint and InvocationContext
    public Object proceed() throws Throwable   
    {
        // sort of a loop to proceed with the advice chain
       
        // case first round: TX aspect
        // TX aspect
        // as defined in the "AspectWerkzAspectModel implements AspectModel"
    aspect_0.manageTX(this)
        // "this" is considered as the JoinPoint instance

        ...

        // case second round: interceptor class
        // as defined in the yet to be written
        // "EJBInterceptorModel  implements AspectModel"
    aspect_1.someNameOfYourChoice(this)       
        // "this" is considered as the InvocationContext instance
       
        ...

        // case third round: the bean is the aspect
        // as defined in the yet to be written
        //  "EJBIsTheAspectModel implements AspectModel"
    target.interceptMyself(this)
        // "this" is considered as the InvocationContext instance
        ...
     }

   }

That may sound a bit of gory details, but that's actually simple once you get the idea of this closure acting for

multiple AspectModel in mind. The code is actually 15O lines for the interceptor class model (EJBInterceptorModel)

and 25 lines for the bean is the aspect model (EJBIsTheAspectModel) thanks to some inheritance. It mainly deals with

generating the InvocationContext methods (like getMethod(), getParameters())

dealing with the EJB 3 aspect life cycle ie
- push the EJB instance on stack for the EJBIsTheAspectModel
- create an interceptor class instance, bookeep it and push it on stack for the EJBInterceptorModel (as the spec
  
  does not specifies the life cycle, I will use a singleton model).

What about the deployment ?

I presented the runtime, but there is an interesting topic as well: the deployment. AspectWerkz pioneered the aop.xml to

define which
aspects are affecting the system. Obviously, we don't want that for EJB interceptor. We need to use the AspectWerkz API

to register what we need in the system. The interesting concept here is that we will transform what the EJB

specification defines thanks to annotation (@javax.ejb.Interceptor, @javax.ejb.Interceptors, @javax.ejb.AroundInvoke) in

real AOP pointcut that the AOP container understand.
That is where a vendor may easily hook in an extension (ie introduce a new annotation for example) to refine the

interceptor class life cycle, or to introduce a pointcut.

I am defining a class that will expose an API that will hide the AOP registration from the user or from the other parts

of our spec. implementation.


public class EJBInterceptorDeployer {

   public static void deploy(String ejbClassName, ClassLoader loader) {
      // step 1 - interceptor class
     
      // read the ejbClass @Interceptor and @Interceptors class level annotation
      // for each interceptor class name found as value of those annotation
      //     get the @AroundInvoke method information
      //     deploy an aspect
      //         using the EJBInterceptorModel
      //         to the pointcut : "execution(!@javax.ejb.AroundInvoke !static * " + ejbClassName + ".*(..))"
      //
     
      // step 2 - @AroundInvoke method of the bean itself
      // find the @AroundInvoke method if any
      // deploy an aspect
      //     using the EJBIsTheAspectModel
      //    to the same pointcut
     
   }

 // method making use of AspectWerkz aspect deployment API

}

The thing to note about the deployer is that it is not using reflection. It is f.e. using our
href="http://backport175.codehaus.org/">BackPort175 implementation to read the EJB annotations from the bytecode. If

we were not doing so, we would trigger the EJB class loading while our system is not yet defined,
and the weaver would not be able to do the job when using load-time weaving approaches.

What about running the system ?

In the sample application, I am deploying the EJB using the EJBInterceptorDeployer presented in the previous section.
In a full blown container I would hook the call to it when the application or EJB deployer would register the EJB in the

system.
The nice thing is that I can run my EJB and still have the interceptors when running as a standalone application by

using AspectWerkz load time weaving, and a simple static block to declare which class is an EJB.

The sample application can be run with (for ones familiar with AspectWerkz, there is no aop.xml..)


java -javaagent:lib\aspectwerkz-jdk5-2.0.jar test.ejb3.Sample


public class Sample {

    static {
        EJBInterceptorDeployer.deploy("test.ejb3.MyEJBIsTheAspect", Sample.class.getClassLoader());
    }

    public static void main(String args[]) throws Throwable {
        // some one would do a lookup or inject for that when in the container
        MyEJBIsTheAspect myEjb = new MyEJBIsTheAspect();

        System.out.println("Calling the EJB");
        int i = myEjb.businessSum(1, 2);
        System.out.println("got : " + i);
    }
}


AW EJB3 - Deploying test.ejb3.MyInterceptor for test.ejb3.MyEJBIsTheAspect
AW EJB3 - Deploying test.ejb3.MyEJBIsTheAspect for test.ejb3.MyEJBIsTheAspect
Calling the EJB
--> MyInterceptor.interceptStandalone
  method: public int test.ejb3.MyEJBIsTheAspect.businessSum(int,int)
  args[0]: 1
  args[1]: 2
--> MyEJBIsTheAspect.interceptMySelf
  method: public int test.ejb3.MyEJBIsTheAspect.businessSum(int,int)
  args[0]: 1
  args[1]: 2
got : 3

Conclusion

Though at first glance I found the interceptor part of the EJB 3 specification fairly odd, and was a bit sad that some

important AOP concepts like precedence and aspect life cycle, as well as expressiveness of the pointcut language are

completely sacrified, I must admit that it might help ones familiarize with AOP concepts.

It took 1h to implement the EJB 3 interceptor spec and have it perfectly integrated with AOP - unlike so far exposed EJB

3 previews by JBoss and Oracle (though those are actual preview, while this article is more a technology focus and

positionning).
It took 15 minutes more to implement a pointcut extension thru a new @org.codehaus.aspectwerkz.ejb3.AroundInvokeAOP

annotation, that allows me to reuse the expressiveness of the pointcuts ie a vendor extension:


public class MyInterceptor {

    @AroundInvoke
    @AroundInvokeAOP("execution(* *.businessSum(..))")
    public Object interceptStandalone(InvocationContext ctx) throws Exception {
        System.out.println("--> MyInterceptor.interceptStandalone");
        System.out.println("  method: " + ctx.getMethod());
        for (int i = 0; i < ctx.getParameters().length; i++) {
            Object o = ctx.getParameters()[i];
            System.out.println("  args["+i+"]: " + o);
        }
        return ctx.proceed();
    }
}

Some more time would allow me to have further features, like f.e. supporting cflow and args pointcut, so that an

interceptor may look like an advice as it looks like in AspectWerkz aspects, with static access to intercepted method

arguments etc (ie no boxing in an Object[] array).

A next iteration would be to integrate with the AspectWerkz hot deployment feature, and this would bring in hot

deployment of EJB interceptors at no cost - while still having everything statically compiled for the runtime to perform

at its best.

This implementation may seems at first glance full of details, and perhaps like a hammer to address a simple part of the

spec. I don't think so. I think it address the very crucial point that so far everyone has skept - including

JBoss folks (that have a foot in the AOP trench): it integrates seamlessly with the aspects and AOP concepts, and

actually allow advanced user to apply more advanced AOP concepts as well ie bridging the gap between a commercial

concept and needs (EJB interception) and a sound concept backed by years of research (AOP and cross-cutting).

So which vendor will be the first to have its EJB play well with AOP : like applying an interceptor thru a real

pointcut, defining programmatic precedence etc, dealing with cflow and hotdeployment of interceptors, and all that with

an implementation that can scale ?

Want the source code ?
This one is part of AspectWerkz CVS.
It can be browsed from
href="http://cvs.aspectwerkz.codehaus.org/viewrep/aspectwerkz/aspectwerkz4/src/compiler-extensions/ejb3">here.

Side note: my feedback on the specication:

There are some odd things in the 3.5 section of the JSR-220 that I have spot:

- (sect. 3.5.1) an interceptor class or EJB can only have one single @AroundInvoke method (advice) in it, and its

signature is (sect. 3.5.4) "@AroundInvoke Object someNameOfYourChoice(InvocationContext ctx) throws Exception". Why

limitating that to having one single method in the interceptor class or EJB (sect. 3.5.1). My guess is that it is tied

to the fact that precedence between advice would then be harder to define in the spec without new semantics.
Then one might wonder why an interceptor is not defined as an interface with one single method "intercept(...)"

that the EJB could implement as well. Having this interface would suppress the need for @AroundInvoke which sounds like

an annotation overuse (unless it is a door open for vendor extension as I will do below..). I'll be interested in EG

feedback on that topic, especially in regards of the first limitation.
- interceptor components life cycle is unspecified but stateless. That sounds like a very important and powerfull

concept of AOP (especially AspectJ) that has been left aside. It is probably an interesting door to provide vendor

specific extension ie thread safe interceptor, per bean class interceptor, per bean instance interceptor, per

application interceptor etc (but then interceptor component may not be stateless anymore.)
- (3.5) an interceptor intercepts all business method (or MessageListener methods for MDB). This means that every

interceptor will be poluted with a code snip like "if (invocationContext.getMethod().getName().equals("doSomething"))

..." ie ones will have to write sort of a pointcut in a very loosy way while AOP allow us to write that in a neat way

(and further, a way that tools can easy understand to spot which method is intercepted by what).
- javax.ejb.InvocationContext is tied to EJB. What will happen when interceptors will end up somewhere else ?

Fortunately, the implementation exposed in this article addresses those issues nicely from a vendor specific extension

perspective.

Note: These are my own thoughts and not of my employer ..

BEA joins Eclipse

2005-02-23T17:14:00.000+01:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

That's an exciting time for BEA.

We now have announced we are joining Eclipse with a number of interesting things in the pipe, ranging from AOP with the AspectWerkz/AspectJ effort leading to AspectJ 5, to the IDE part that will shape up BEA WorkShop, and leading positions in the Web Tools Platform, and in the JDT compiler area.

Things were a bit odd when we shipped an AspectWerkz Eclispe plugin some month ago, since everyone out there was thinking that there was a death match between BEA WorkShop and Eclipse... Didn't you thought that way ?

And don't say we have started to sneak in randomly just to catch up. Things are now ranging from the JVM with AOP work going on within JRockit up to the IDE...

Read the news
http://biz.yahoo.com/prnews/050222/sftu152_1.html

JUnit Test cases and number of tests ?

2005-02-22T15:26:00.000+01:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Last time I was reading about a project, folks were arguing they had many many tests.

But what is exactly this measurement ? Do we have to count the number of classes that extend JUnit TestCase, or the number of test*(..) methods, or the number of calls to JUnit assert*(..) ?

The current figures for AspectWerkz are pretty much those one:
- 92 extends TestCase
- 544 test*() methods
- 1380 calls to assert*(..)

Off course, what would matter would be the coverage (that has several definitions as well), but still, the number of test in an interesting thing to define.

So how many tests do I have ?

Writting IDEA plugins

2005-02-15T18:02:00.000+01:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Today Jonas and I shipped Backport175, a back port of the JSR-175, the Java 5 Annotations spec.

This project comes with an Eclipse plugin and an IDEA plugin, so its time to say something about IDEA...

I just like the architecture of the IDEA plugins. There is very few docs on the topic, some very bad javadoc, and some samples around but no books yet etc.
Anyway, you can find some help on the IDEA forums, and guys like the Groovy ones have written plugins as well, so you can find a bunch of code around.

The nice thing in IDEA plugins is that it s just code, a rather clear API, and a bit of IOC (IDEA is using PicoContainer). A plugin as just one XML descriptor.
For an Eclipse plugin, you need a lot of XML, find what are the extension point, and dig in the docs and books, and get familiar with the Eclipse plugin wizardry. The game is very different, while for IDEA, it s just an API that you have to get familiar with.

What is missing the most in IDEA v4 is a good plugin project wizard, but if you have the chance to give a try to IDEA IRIDA, the EAP of the upcoming v5, then you have it, and it's as easy to debug your plugin as it is for a regular app.
The great thing with IDEA is that you can write a very small Ant script to build the plugin. I would not even try to go there for Eclipse (perhaps I am wrong ...).

You can probably learn some in that space by comparing both Backport175 plugins. It will leads you thru
- how to have a menu of your own to add / remove the feature to a given project (nature / actions and menu)
- how to hook in in the build system (builder / compilation listener)
- how to deal with markers

Off course, it is probably not the correct design neither for Eclipse nor for IDEA (I am far from beeing efficient in that plugin area) but it seems to do the job.

IDEA plugin CVS
Eclipse plugin CVS

Thanks to JetBrains to allow free use of IDEA for OSS projects !

Spring pointcut on steroids

2005-01-24T18:32:00.000+01:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Three months ago when we wrote the AWBench AOP benchmark suite, I had to hack some little Spring AOP aspect and pointcuts.

If you are familiar with Spring and know about AOP (AspectJ, AspectWerkz, JBoss etc), you know that Spring pointcuts are rather ... well sadly un-intuitives.
They might be for new users, but not anymore when you have very basic AOP knowlegde, and you are more and more !

I could not resist and integrating Spring with AspectWerkz pointcut (yes, once again for the third time, after the Spring container Jonas did and the extensible container we shipped.)

A Spring pointcut before the wedding with AspectWerkz - almost impossible to combine with other patterns unless you have your "Mastering Spring XML" in the pocket...


    <bean id="theMethodExecutionGetTargetAndArgsAroundAdvisor1"
        class="org.springframework.aop.support.RegexpMethodPointcutAdvisor">
        <property name="advice">
            <ref local="theMethodExecutionGetTargetAndArgsAroundAdvice"/>
        </property>
        <property name="pattern">
            <value>.*aroundStackedWithArgAndTarget.*</value>
        </property>
    </bean>

Spring pointcut as a real pointcut thanks to the wedding with AspectWerkzPointcutAdvisor


    <bean id="theMethodExecutionAfterThrowingAdvisor"
        class="awbench.spring.AspectWerkzPointcutAdvisor">
        <property name="advice">
            <ref local="theMethodExecutionAfterThrowingAdvice"/>
        </property>
        <property name="expression">
            <value>execution(* *..*.afterThrowingRTE(..))</value>
        </property>
    </bean>

Such a basic code source is there and should perhaps be reviewed by Spring guys for some of the API details.

The great news is that this will be part of a next release of Spring, and details will be handled by the AspectJ 5 engine instead. That was about time ;-)

AspectJ and AspectWerkz to Join Forces

2005-01-19T10:09:00.000+01:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Today we have finally announced that we are joining AspectJ to work on AspectJ 5.

We will focus on what has been the success of AspectWerkz
- plain Java AOP, relying on an annotation defined aspect style
- load time weaving, and integration in J2EE environment

The new team is backed by both IBM and BEA and the new project will be delivered in the coming months. One of the priority is to make sure that a smooth migration path exists from AspectWerkz to AspectJ 5.

That' s a great day for AspectWerkz and AspectWerkz users.
Read more about it

YAPB AOP 1.2.3.4 released

2004-12-22T09:42:00.000+01:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

I am pleased to announce the first release of YAPB AOP.

For those who don't like this April Fool like YAPB AOP, please read this post on The Server Side - read
Note: YAPBAOP is a real project with real source code and runnable demo !

Features
YAPB AOP 1.2.3.4 is Yet Another Proxy Based AOP for Java.
It is
- proxy-based (AspectWerkz proxy)
- pure java, no language extensions, no external tools needed
- simple and intuitive
- direct, programmatic configuration and un-configuration
- j2ee friendly, no need to mess with class loading
- supports AOP Alliance API for MethodInterception
- JDK 1.5.0 annotations support to match on annotation
- per instance programmatic interception
- full speed

Sample code
here is how YAPB AOP 1.2.3.4 use is simple:


// no aspect
System.out.println(" ( no aspect )");
YapbaopDemo me0 = new YapbaopDemo();
me0.method();

System.out.println(" ( bind a new aspect )");
Yapbaop.Handle handle = Yapbaop.bindAspect(DemoAspect.class, "* yapbaop.demo.YapbaopDemo.*(..)");
YapbaopDemo me1 = (YapbaopDemo) Proxy.newInstance(YapbaopDemo.class);
me1.method();

handle.unbind();

// get a new one but not using the proxy cache then..
System.out.println(" ( unbind it and get a new proxy YapbaopDemo-2)");
YapbaopDemo me2 = (YapbaopDemo) Proxy.newInstance(YapbaopDemo.class, false, false/*not advisable*/);
me1.method();// still has advice
me2.method();// no advice

Extra features
It was very boring to do and does not bring any value to the table, so please, stop inventing (is is the right word ?) YAPB AOP clones based on your favorite bytecode library / proxy library.

It took 40min to hack (time for a train trip, YAPB AOP are that easy today).

If you want to do something usefull, write an email to your favorite AOP framework representative (AspectJ, AspectWerkz, JBoss AOP, Spring AOP, dynAOP, joyAOP [RIP ?], jeetAOP [RIP ?], YAPB AOP [RIP]) and wonder how to
- write some usefull reusable aspect
- write some good article
- provide a new feature, a performance improvement, a bug report or fix
- go an apply to speak about it to the next conference you can find
- imagine new things !

Download Now !
- Get it there (3 Mo jar since YAPBAOP is actually 1 class of 200 lines messed with a lot of dependancies so that you think it is a killing framework)
- Run the demo with:


java -cp yapbaop-1.2.3.4.jar yapbaop.demo.YapbaopDemo

Access the source code
The framework - here
The demo to play with - here
The AOP alliance aspect for the demo - here

Write your own !
Ever wanted to be an Open Source star [*] ? Write your own. Enroll now !
[*] A wrong assumption seems to be that to be an Open Source star, ones have to write at least one YABP AOP. If you only contribute to the next big thing, you ll never be a star, so you 'd better start your own YA RIP framework instead of working with others - that's what mankind is about according to this wrong assumption.

Here are some ideas :
- change the Yapbaop.bindAspect to support binding to a java.lang.reflect.Method
- do the same with an array of java.lang.reflect.Method
- do the same with 1+ Method Annotation(s)
- add some filtering based on 1+ Class Annotation(s)

If you have read till there
Thanks to read this post.
If you dig YAPB AOP architecture, you will find that it is based on
- AspectWerkz proxy
- AspectWerkz extensible AOP container to support AOP Alliance aspect
- AspectWerkz runtime as regards binding of aspects
Quite interesting actuallly isn'it ?

AspectWerkz AOP Eclipse Plugin

2004-12-13T19:25:00.000+01:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Well, we for long have said that plain Java AOP and annotation defined aspects don't needs fancy GUI support, but I have to admit, this makes everyones life easier.

So I am pleased to announce the availability of the AspectWerkz Eclipse Plugin based on the 2.x releases.

Get it now, see the doc and the screenshots !

The plugin contains two almost disjunct features:

an embedded compiler (a builder in the Eclipse plugin world) for our Java 1.4 strongly typed annotations. Just write the annotations as you would do with Java 5 but in JavaDoc (arrays, nested annotations etc are supported), and write one interface per annotation (where you would write an @interface with Java 5), and the plugin will do the trick when Eclipse will compile your code.
The annotations are then accessible at runtime (using AspectWerkz Annotations API)and you can use them in your pointcuts (nothing new there).

a glue layer that shows the cross cutting view that is add little markers in the gutter to remind you that this piece of code at that line (join point) is advised by this and that before advice in this and that aspect.

(Each feature is a builder so you can deactivate them on a a per project basis if you don't like them / use them.)

Aside since your app will gets weaved, you can just click run and the aspects are in - nothing more to do.
If you need to expose your dependencies jar files to the weaver, use the dedicated AspectWerkz launch configuration that will configure load time weaving for you.

Feedback welcome.
Any volunteer for further evolution ?

The plugin ships for Eclipse 3 and Java 1.4. More to come on Java 5 later.

Alex

Oh forgot to say before others make a story about that: yes we are the last one to write an Eclipse plugin, way (decades?) after AspectJ and some month after JBoss.

JAOO 2004 AOP coverage - day 1 and 2

2004-09-22T11:36:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

I am back from the JAOO conference that is happening this week in Aarhus, a city in the west part of the Denmark.
Jonas and I had the opportunity to speak there, and since I feel that there is only a little coverage of it in the blog
community, I have decided to take care of that DK duty.

This post won't give you the right view on this conference, since I attented to my own fields related session aka AOP related, but anyway, you will have a minimum.

The conference

The conference is this year largely sponsored by Microsoft, among others of the .Net or Java ground. Great guys are speaking there, like Rod Johnson (Spring / Interface21 - you know him), Rickard Oberg (former JBoss ejb guy, closed source AOP/CMS lead at Senselogic) , Martin Fowler, Arno Schmidmeier (early AspectJ user, AOSD consultant), Peter Von der Ahé (javac Sun guy), Jonas Bonér and myself to name a few and a bunch of .Net guys.

Though the conference location was a bit far from the airport, the accomodation was good, lunches were fine, with DK stuff but not too much for a FR guy, and the monday night social party was pretty good.

Day 1 - Keynote by Microsoft

I have attended the monday keynote given by a Microsoft guy. It was a good insight to see what MS is attempting to achieve, thinking ahead the C# things while people are just excited about the upcoming release of a new Visual C# things.

it was interesting to hear about how clever is Microsoft when it leads to the latest Java 5 Tiger new features, which are some sort of compiler trick for most of them.
In .Net, when you have an int, this one is an object, and there is boxing / unboxing features somehow, but the int himself is reflectively accessible. In the Java world we have blessed (or cursed as Jonas said...) primitive types (int ...) and boxed types (Integer etc), and Tiger provides boxing and unboxing but thru a compiler trick - that is the bytecode will make use of yourInteger.intValue() and Integer.valueOf(yourint) under the hood.

Microsoft gave a bunch of other samples, and admitted that it was somehow easier to start a good language desgin from scratch that sketching on an existing one.

Day 1 - Sessions

I did not attended more on monday morning since I had to prepare some for our own session and I hope to see someone providing feedback on it since it is not fair to write it myself but still...

Annotation driven AOP with AspectWerkz

Jonas and I were giving a talk in the J2SE track on Annotation and AOP.
We started with an AOP crash course just to get everyone on the train.
Then we explained how Tiger annotations are used in AspectWerkz to have annotation defined aspects (something that JBoss has recently lazily followed).

Annotations are also used in AspectWerkz as a matching mechanism, and we went thru a sample where it is perfectely safe to refactor your methods without taking care about the underlying AOP without any whatever IDE plugings.

You can check our slides there, but for the impatient, here is how our strongly typed matching aspect looks like for annotation driven AOP. As you can see, there is no string based, wildcard based magic there.


// Annotation driven AOP
// Sample of strongly typed matching
// in AspectWerkz AOP with annotations

@Service
public class Math {

  @Async
  public Result complexComputation(Arguments args) {
    ...// this may take a while - we should do it in another thread
  }
} 

public class AsyncAspect {

  @Around
  @Within(Service.class)
  @Execution(Async.class)
  Object doAsynchronously(JoinPoint joinpoint) {
      ...
  }
}

The session went fine, and the room was pretty much full, with around 400 attendees and after the session we met and chatted some with Rickard Oberg.

After that we had a sort of J2SE panel, which turned out to be a BOF since the JAOO crew had not provided us with a moderator. It ended up in discussing some details and pitfalls in the new Tiger features, like generics and static import.

Night 1 - All conferences are about social events

In the afternoon we met Arno Schmidmeier, an early AspectJ user (meaning late 90's), who is now doing AOSD consulting in Germany (give him a call if you need some AspectJ on-site knowledge), and we then head up to the social event / party with Rod Johnson. Time to share a bunch of beers paid by JAOO sponsors (Microsoft, Borland, Quest, and then I don't really remember) while discussing ideas about Proxy based AOP as in Spring, Geronimo integration, Spring AOP container for AspectWerkz, and some other things.

Day 2 - Sessions

On day 2 I had to leave early and get back at work so I could only make it for Arno and Rickard'sessions and missed Rod' session unfortunately.

AOP - let the code looks like the design

Arno Schmidmeier gave a talk in the Domain Driven Development track, where the main point was to explain how AOP could allow you to let the code looks like the design.

He went thru some use-cases he had seen during his consultancy jobs, and the most interesting part of the session was when he explained how an expression for a domain requirements like "for every successfull bank account operation the accont balance must be checked after the operation and in case of failure, an error should be triggered by the system and the system notified that it should rollabck etc." could be mapped to an AspectJ after returning aspect and thus making his points that AOP enables you to have the code look like the design while avoiding code tangling issues.

Attendees raised questions about code readability, modelling issues and similarities to a predicate language approach. I would have liked to see a running samples in AJDT and alike, and be able to navigate thru the aspects, and may be emphasis more on the domain level pointcuts idea.

Rickard experiences on AOP

I attended Rickard talk about experiences on AOP.
Rickard ellaborated a talk around many of the complex use-case he has for AOP in his company (Senselogic) CMS platform. He started with some classis technical domain concerns, and then jumped on his idea of interface driven system, where an object is actually decomposed in an assembly of entities, each one having a defined stack of advices attached to it.

This approach, which seems quite close to quantum AOP concepts, opens for a bunch of interesting things when it comes to GUI update propagation and server to server replication of sub-graph of persistable objects as well as undo/redo management which again should propagate...

Rickard introduced the idea of a "property=development|production|whatever" pointcut element, that allows for sort of global on/off on the pointcuts. Something you can achieve with a "if" pointcut in AspectJ - although he probably did implemented it at weave time and not at runtime unlike the AspectJ "if".

Rickard then explored the advantages and drawbacks of his "home made" AOP framework, and explained the idea of the abstract schema, where the aspect is abstract and then implements interface of classes where it is actually weaved to, providing sort of natural syntax for accessing the advice target instance methods from within the advice body.

He then went thru a small demo wich showed a nice swinguish things, and then went thru some ajdoc equivalent, where generated javadoc for a class/member is exposing which advices are bounded to it and thru which pointcut.

The next session was Rod Johnson session on Spring and Spring AOP. Unfortunately Jonas and I had to leave since mh, we are pretty busy these days.

Wrap up

Those 2 days were a good investement. I enjoyed the talk we gave there, and it was nice to meet and discuss more with people like Rod, Arno and Rickard.
JAOO has probably a lot more to offer, even if you are on the .net side or on both side (ThoughtWorks had a booth there by the way).
I had some time to think about Rickard abstract schema and experiment some more on Tiger annotations - something I will try to blog about soon.

Get to see the schedule and the JAOO slides there.

Below are some picts of Jonas, Arno, Rickard.

Jonas speaking

Arno session

Rickard explaining his Abstract Schema

Annotation-driven AOP in Java at JAOO

2004-09-19T12:19:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Jonas and I are going to speak at JAOO about Annotation driven AOP in Java (Tiger / J2SE 5).
The session will cover aspects defined as annotated java classes, as well as matching on annotations and defining strongly typed pointcut thru annotations.

This is a great opportunity to speak there, especially since those ideas have been implemented since the early time of AspectWerkz. Something that some of the other AOP framework are now following.

Plain Old Java Aspect

2004-08-19T17:24:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Back from hollydays, and back to work.

It's time to work on those Plain Old Java Aspects (POJA).
Java 5 Annotations are now so old, in the place since 1.5...


public class Poja {

  @Around("execution(* *..*(String message)) AND within(silly.dot.com.at.work.*)")
  public Object intercept(JoinPoint jp, String message) {
    ...
  }

}

AspectWerkz in WebLogic - domain wizard

2004-07-20T15:37:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

I have sketeched a BEA WebLogic domain wizard, so that it is now possible to install a WebLogic server with AspectWerkz AOP in a minute.

Requirements:
- WebLogic 8.1 SP2 (8.1 whatever should work).
- Use of JRockit is mandatory for now (included in WebLogic 8.1 or other version if you want).

How to:
- Download the jar (7 Mo, includes the whole AspectWerkz 1.0beta1 distribution with some extra fix).
- Drop the jar file in your BEA_HOME/weblogic81/common/templates/domains.
- Start the BEA Configuration Wizard from the start menu.
- Select the AspectWerkz node and the "WebLogic WorkShop domain".
- Select JRockit
(for other screens, refer to the usual BEA installation documentation)

Limitations:
- It is a WebLogic WorkShop domain wizard. You can probably adapt it for WebLogic Server only, or WebLogic Integration and WebLogic Portal
- Use of JRockit is mandatory, although it is possible with some minimal change to adapt the generated "startWebLogic" script for Sun HotSpot VM (I would do it with the AspectWerkz Plug utility and -Xbootclasspath option - refer to the documentation).
- If you are not using 8.1 SP2, you might need to adapt the "startWebLogic" script to set your JAVA_HOME to the correct JRockit version.

Extra bonus:
- There is draft version of a WebLogic AOP console extension included, that allows you to browse the META-INF/aop.xml definitions accross the class loader hierarchy of your deployed applications.
Just start the weblogic admin console as usual and the extension will appear on the left navigation pane.

AOP in J2EE, Tomcat tutorial

2004-07-09T11:38:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Since we released AspectWerkz 1.0 beta 1 some days ago, with a real cool feature we had in our roadmap since the very begining, I have written a small tutorial for you to practice.

AspectWerkz now supports the concept of an AOP container. No other AOP framework out there supports this idea in the way we do it : class loader aware and cross platform.
It is now possible to have an AOP deployment descriptor within your application, and interesting point in an AOP world: have several of them, all along your application.

The standard way of doing so is to use META-INF/aop.xml files, and for some container like Tomcat it is possible to use WEB-INF/aop.xml as well.
If you add a jar file in your application and this jar file contains a META-INF/aop.xml descriptor, it will also be taken into account. It is thus easy to distribute aspect jar file with predefined behavior.

You can drop Aspect classes at your system classpath and those will affect all deployed application, AND, you can have Aspect classes within your web application that will only affect this specific deployed application.

Read the tutorial here.

Note that if an Aspect is deployed within an application, the pointcut it defines will be limited by the class loader isolation principle. Even if the pointcut assumes to match all method in all classes, it will only "see" all methods in classes loaded by this classlaoder and the child classloader.

Try it now !

Aspect Oriented IT fits BEA Liquid Computing

2004-05-30T18:11:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Last week I attended eWorld, mainly as a speaker. It was a unique view on BEA's Liquid Computing vision kick off. Where AOP fits in Liquid Computing vision ? Why should we go on simplifying the AOP stack as our motto in AspectWerkz : full Java, J2EE integration, Java 1.5 readyness, value added Aspects, tool ecosystem... ?

Liquid Computing is the vision for a Simplified Enterprise Computing. Don't think it is SOA. SOA is what BEA provides today with Platform 8.1, but tomorrow, it will be something more.
"The time has come for a comprehensive vision for IT. One that improves business responsiveness by simplifying enterprise computing."
- Alfred S. Chaung
Founder, Chairman & CEO
BEA Systems, Inc.

Reading more in BEA's Liquid Computing pdf brochure, I found it is good exercise to emphasys on specific topics where AOP is a key way to enable Liquid Computing.

Note: the following is my own personnal vision and does not illustrates any official statement from BEA.

Active Adaptability: "Dynamic instrumentation and monitoring will track, display and manage system capabilities. At the core, the infrastructure
is constantly tuning itself in the service of the business."

Self healing policies can be easily builded on Dynamic Aspects.

Dynamic weaving capabilities of BEA JRockit and AspectWerkz has been demonstrated at our eWorld session, activating JMX reporting at runtime, in a web service / pojo based application running on 8.1 and a not yet released version of JRockit.

We can extend the demo to dynamically weave a caching aspect at our web service level to provide a business specific self healing policy. All this stack could be builded in 1 day based on AspectWerkz dynamic AOP stack, no matter the application complexity itself.

This is active adaptabilty now, on Platform 8.1. The same key idea can be enabled today on your older WebLogic Server 6.1, or on your Tomcat stack - whatever is running Java 1.3+ can be handled by AspectWerkz Dynamic AOP.

Enterprise compatibility: "the application just plugs into a
compatibility layer [Compatibility Fabric], rapidly delivering and sharing new
capability."

Cross cutting requirements can be addressed rapidly by AOP technologies.

The AOP stack is the compatibility fabric of the cross cutting concerns inherent to all software applications. Thinking AO enables to decouple this concern and address it as a single requirement, allowing change at this level only, to provide the same functionnality with a different underlying implementation but without any change in the application that serves your business.

At our session we have demonstrated an authentication / authorization service implemented in AO. The underlying implementation was using JAAS. It can now be easily enhanced to provide an implementation specific to any IT environment, be it JAAS compatible or not, be it LDAP based, or be it with very specific policies. This software complexity is reduced to the Aspect, and does not impact the functionnal requirements of the application.

It is not WebLogic specific. The same AO stack can be used to secure a Swing based application as well.

The AO implementation brings cross application security live in a minute, no matter the complexity of the application(s) and the IT environment. For more complex requirements, this AO implementation can be enhanced with business specific rules (application flow, business rules, dynamic activation of policies) by adding aspects on the stack, where those are needed.

Breakthrough Productivity: A powerfull engine that extends self-service "into meaningful application functionality. It gives business owners
the ability to make changes to processes and capabilities on their own.[..] There’s no downtime, no need for traditional re-programming and, because IT maintains governance, no loss of IT control."

Self servicing can be enabled thru a dynamic AO stack. Deploy a new aspect that encapsulates a new functionnal requirement, and have it inserted in your running application with dynamic AO, and metadata driven programming.

Metadata driven AOP brings self-servicing to a new IT control era, where model driven architecture can be unleashed by an AO based service stack. Add a cache on service methods, no matter the application was designed for a cache requirements. Change this cache to be distributed, single user based, or for the flow of this specific use-case. Add a JMX reporting layer on it to control cache hit count. Remove this cache when system is not under load.

At our AOP: SOA for the application eWorld session, we have demonstrated hot acitvation of a caching aspect while application was live. We have extended the cache scope to some other business method at a later point, just by using a tool and 3 mouse clicks.

AO solves a traditionnal multi-step IT project iterations in a disruptive way. Pick vendor certified aspects, hotdeploy them with tools and have them bringing your IT to the next generation of your requirements without any downtime. This is Breakthrough Productivity. I would have wrote Disruptive Productivity.

Conclusion: Go live with AO !

The three key concepts of Liquid Computing are part of Aspect Oriented Technologies and especially full java dynamic AOP stacks like AspectWerkz. A good aspect set, tools ecosystem and manageability of the AO layer itself disrupts the classical vision of IT projects. "Design, develop, QA, deploy" is turned into an on demand self servicing layer allowing you to concentrate on your assets. Dynamic AO brings Active Adaptability, address Enterprise Compatibilty and provides Disruptive Productivity.
The AO ways decorrelates your IT complexity from the complexity to address new requirements. Changes are made easy.

I hope this high level view on AO, the issues we aim to address everyday as an AOP stack builder, and the demo we can do today convince you that Aspect Oriented Technology is a key piece of the Liquid Computing puzzle.

Up to us to bring all this value to you now.

Make it live today with AspectWerkz. Ask for docs, demos, support, articles, tutorials. Join us in this aspect and tool ecosystem building.

Go live with AO. Go live with Liquid Computing. Go live with AspectWerkz.

AOP integration in Tomcat 5

2004-05-27T10:20:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

On several recent ML post AspectWerkz users complained about getting into troubles when hooking in AOP in Tomcat 5. Steve provided good feedback but ... there is off course a damned better way to handle that !

Here is a little hands on that gives the idea for a 5 minutes shot.
Pretty much straigthforward based on

cvs head ie the next 1.0 beta and not 0.10

no integration detail, just some bin/aspectwerkz.bat magic

aop.xml deployed wihin the target application itself

a little fix in a strange Manifest.mf file in qdox-1-3.jar

I was concerned about this issue and had an hectic face to face burger discussion with Jonas about "my" so called seamless integration of AOP... I needed to sit down on that and hack, too bad for the jet lag in this crazy GMT 9h far from home SF BEA eWorld clock.

Since the Codehaus had to migrate some CVS details today, the little qdox.jar fix that is needed cannot be commited, but just do it yourself if you want to go on with those explanations. Or wait tomorrow since Bob the despot is around there as well... He'll fix that for me if he is not too busy on the new Codehaus / BEA BeeHive stuff there.

So I took a Tomcat 5 from Apache, unzipped the stuff on my windows box.
I checked that it was fine with the bin/startup.bat and shutted it down to spread AOP in.

Then I made sure that AW *cvs head* was builded, and ASPECTWERKZ_HOME and JAVA_HOME was set.

I decided to give a try to this integration issue and decided to start with a silly tracing aspect that is sitting in AW samples (examples.logging.JavaLoggingAspect). I then builded the samples with a maven target (aspectwerkz:samples:compile)and had a new target/aspectwerkz-samples.jar ready to use.

Then I just opened this bin/catalina.bat that is actually starting things and added this just before the comment:


        rem Execute Java with the applicable properties
        ...

to have AOP seamless integration:


        set CLASSPATH=%CLASSPATH%;%ASPECTWERKZ_HOME%\target\aspectwerkz-samples.jar
        set _EXECJAVA=%ASPECTWERKZ_HOME%\bin\aspectwerkz.bat
        set JAVA_OPTS=%JAVA_OPTS% -Daspectwerkz.transform.verbose=false

So as you see I go with the bin\aspectwerkz.bat way, which is the easiest kick off stuff we provide, with adaptive behavior regarding Java version detection etc, even if when it comes to production usage I would rather use some JRockit VM.

You will have noticed that I just added my silly tracing sample aspect in the CLASSPATH and replaced bin/java call by bin/aspectwerkz.bat - not taking care about the details.

From there, Tomcat started up *almost* fine. The AW/lib/qdox-1.3.jar has a strange Manifest.mf file that Tomcat complains about... Just dropped this Manifest.mf from the qdox.jar and we are up and running with AOP online mode / class load time weaving / AOP container from cvs head.

Time to bind my system level tracing Aspect to some application.
I did a try by writting a 5 lines aop.xml file (actually copy pasted it from AW samples/hotdeployed.xml) and dropping it in Tomcat samples in webapps\servlets-examples\META-INF\aop.xml, since this is what we now provided in cvs head (and not in 0.10).

Once that done, nothing happened. I added some tracing stuff (-Daspectwerkz.transform.verbose=true) and figured out (with a bit of pain since toString() on Tomcat classloader is damned too much verbose..) that Tomcat classloader is not taking this META-INF topmost stuff into account.

I decided to move that to the WEB-INF/classes path, so ending up in webapps\servlets-examples\WEB-INF\classes\META-INF\aop.xml [file below]. I ll double check that later since this WEB-INF/.../META-INF is a bit strange there.

and here it is (once I browsed to http://localhost:8080/servlets-examples/)


METHOD_EXECUTION--> filters.ExampleFilter::doFilter
  METHOD_EXECUTION--> HelloWorldExample::doGet
  METHOD_EXECUTION<-- HelloWorldExample::doGet
  METHOD_EXECUTION--> filters.ExampleFilter::toString
  METHOD_EXECUTION<-- filters.ExampleFilter::toString
METHOD_EXECUTION<-- filters.ExampleFilter::doFilter
METHOD_EXECUTION--> filters.ExampleFilter::doFilter
  METHOD_EXECUTION--> SessionExample::doGet
    METHOD_EXECUTION--> listeners.SessionListener::sessionCreated
      METHOD_EXECUTION--> listeners.SessionListener::log
      METHOD_EXECUTION<-- listeners.SessionListener::log
    METHOD_EXECUTION<-- listeners.SessionListener::sessionCreated
  METHOD_EXECUTION<-- SessionExample::doGet
  METHOD_EXECUTION--> filters.ExampleFilter::toString
  METHOD_EXECUTION<-- filters.ExampleFilter::toString
METHOD_EXECUTION<-- filters.ExampleFilter::doFilter

aop.xml


<!DOCTYPE aspectwerkz PUBLIC
    "-//AspectWerkz//DTD//EN"
    "http://aspectwerkz.codehaus.org/dtd/aspectwerkz.dtd">
<aspectwerkz>
    <system id="sample">
        <aspect class="examples.logging.JavaLoggingAspect">
            <pointcut name="pc" expression="execution(* *..*.*(..))"/>
            <advice name="logMethod" type="around" bind-to="pc"/>
        </aspect>
    </system>
</aspectwerkz>
]]>

Conclusion
We are almost there and 1.0 will be damned good don't you think ?

See you at eWorld

2004-05-20T13:56:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

I am moving to San Francisco for BEA eWorld.

Jonas and I will give several sessions / demos about AOP in BEA JRockit and the BEA WebLogic Platform, including WebLogic Server and WebLogic WorkShop.

In case you are in the area but do not attend to eWorld, just join the Silicon Valley BEA User Group since we will be there on wedsnesday as well.

We have been under the radar since AOSD, so be ready for something disruptive !
I will post a viewlet of some parts of the demo in some weeks.

dynamic AOP and HotSwap

2004-02-25T23:15:00.000+01:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

In a recent post Bill Burke describes his preliminary research as regards HotSwap usage for dynamic AOP.

He first defines in a few words what is dynamic AOP and what is the HotSwap or class redefinition in Java. Then he briefly describes how JBossAOP handles
dynamic AOP thru a "prepare" phase, and gives some measurements about the overhead he obtained when using the -Xdebug mode during a JBoss startup sequence, an option required to activate the HotSwap API under Java 1.4.

I would like to give some feedback on his post, since I have been working in the dynamic AOP field for some time now thru my involvment in AspectWerkz and
thru a research paper and a runtime weaving prototype I did for the Dynamic Aspect Workshop for AOSD 2004.

I think dynamic AOP has to be splitted in two categories:

rearranging AOP constructs at existing join points

redefining join points (we could generalize with redefining pointcuts, or even redefining the whole AOP system)

The first category is fully supported by AspectWerkz. It is possible to rearrange (add/remove/reorder) aspects and advices at runtime providing that the join point(s) where the construct(s) is(are) bounded is(are) already existing as a result of the transformation phase, no matter it is a post compilation phase (ala AspectJ) or a class load time weaving.

As regards introductions, AspectWerkz allows to change the introduction implementation (thus the implementation of the added methods) at runtime (swap mixin implementation), again providing that the binding (which mixin(s) applies to which class(es)) is already existing.

This first category relies on the framework capabilities and several means allow to minimize the overhead when f.e. no advice is bounded at a particular join point.

The overhead even if minimal cannot be avoided here. Even reduced to a method call that does a boolean check, this is not the original bytecode instruction sequence which is running when the join point is reached.

This category is the one supported by JBoss as well.

The second category is the most interesting one. Redefining join points (thus pointcuts) allows - in theory - to add new AOP construct at new locations (join points) in the running application.
This should come with the guarantee that there is

absolutely no runtime overhead prior the "activation" of the new join points, and no overhead after "deactivation"

no assumption that the join points are known before activation (especially at class load time / application deployment time)

I think that this second category requires runtime weaving capabilities from the AOP framework (as long as the AOP frameworks rely on bytecode instrumentation).

A use-case for join point redefinition at runtime is some on-demand profiling tool that needs to activate profiling only when necessary, for a limited time, and without any overhead prior to activation, and without specific knowledge on what will be profiled at application deployment time.

As Bill explains, JBossAOP "addresses" the second category by providing a way to declare pointcuts with nothing bounded. This definition is used at class post compilation / class load time and is what Bill calls the "prepare phase".

The overhead involved by such an approach is not measured by Bill, but is conceptually limited to a boolean check and a method level indirection, this for each pointcuts declared for future activation.

I don't think this approach is really adressing dynamic AOP. The overhead at a single join point level is indeed minimal, but we might have a need to declare very invasive pointcuts to allow further on-demand profiling (or other AOP based constructs), and thus this minimal overhead might occurs trillions times in the running application (potentially at each method call, each method execution and each field access).

So what might be a better solution for true runtime join point redefinition ?

As Bill quotes, HotSwap API allows to change the implementation of method / constructor bodies of a class at runtime, and inject them in the running JVM.

There are some interesting things to consider, that Bill forgot to mention and led him to some approximative statements (I forgot to told him ;-) )

HotSwap is an optional JVM capability. Some JVMs do not have HotSwap support (f.e IBM JRE)

HotSwap might forbid class schema change. It might not be possible to add new methods (even private ones) to a class when redefining it. This schema change support is itself an optional HotSwap capabilitiy. Currently no JVM supports schema change, but the API does not restricts it.

HotSwap exists since Java 1.4, requires -Xdebug mode, and can only be called from a remote JVM (well almost, read below)

HotSwap is slightly modified in Java 1.5 and does not requires -Xdebug mode

So considering what HotSwap can do, and what bytecode instrumentation oriented AOP frameworks do to enable AOP in Java OOP, especially to enable dynamic AOP constructs of the first category I described previously, HotSwap seems very attractive, but is currently missing key things.

There might be some ways to address the problems thought...

The fact that the java level HotSwap API as described in Java 1.4 JPDA requires a remote JVM attached to the running JVM (or a JPDA LaunchingConnector) can be avoided by implementing a JNI layer on top of the C-level HotSwap API so that HotSwap is available at Java level, without coupling with the JPDA architecture: no need for a remote JVM, and no need for a LaunchingConnector (that would forbid further remote dedbugging ...).

I describe the advantage of this JNI glue in the work I submitted to the DAW AOSD 2004, and call it "in-process HotSwap". We have it in Java 1.5 so there is no problem on this side anymore.

The "in-process HotSwap" - Java level API - is standardized in Java 1.5 thru the JSR-163. Java 1.5 will not requires -Xdebug to enable the HotSwap API. As you might have understood, the HotSwap API is not that coupled to the JVM debugging capabilities, althought the API was part of the same block in Java 1.4.

The -Xdebug overhead issue raised by Bill is fixed (though I would like to do measurement of what -Xjavaagent in Java 1.5 involves).

The schema change support would be interesting for AOP join point redefinition, since as Bill explains, AOP frameworks that allow dynamic AOP (AspectWerkz, JBossAOP) require some structural changes to bring in enough indirection level between the join point (in the weaved classed) and the AOP constructs (handled by the AOP framework internals).

A simple bytecode decompilation of an AspectWerkz weaved class with execution pointcut will show you that AspectWerkz has added a wrapper method to handle the join point and renamed the original method, since it will be called by the framework as a result of the execution join point construct.

class Foo { public void doAction() { // the bytecode for your code } }

can appeared transformed after the weaving as (approximation)


class Foo {
 
  JoinPointManager __AW_joinPointManager =...
 
  public void doAction() {
    __AW_joinPointManager.proceed(this, "doAction()");
  }
 
  private void __AW_doAction() {
    // the bytecode for your code
  }
}

As a consequence, if we would like to be able to redefine (f.e. simply define) the "* Foo.doAction(..)" execution pointcut at runtime, we would need schema change support. The schema change support might not be needed though for caller side constructs.

The problem is thus more a HotSwap API problem, tight to current JVM limited capabilities.

I think that HotSwap - if largely adopted by JVM implementation - can be a good enabler for dynamic AOP as regards pointcut redefinition at runtime. I don't think that the debate is limited to a -Xdebug option, and Java 1.5 demonstrates it.

In the work I submitted to the DAW AOSD 2004, I describe how we are able in an AspectWerkz prototype to redefine pointcuts at runtime even without schema change support.

I agree that this solution (which I will publish later in my blog) is itself not optimal but I was able to demonstrates the feasability of such an on-demand AOP approach, instead of an invasive pointcut definition, that might itself lead to a globally large overhead. I was able to do the prototype with Java 1.4, and it will be even more easy to do with Java 1.5 and the standardized "in-process" Java level HotSwap.

I will debate if HotSwap is good for dynamic AOP at the DAW AOSD workshop, and you bet, some more JVM level AOP support might be crucial on this area.

So far so good, what do we wanted ?

Probably not a low level "addPointcut(<pointcut pattern>, Class klass)" API with a controlled and guaranteed minimal (or even zero) global overhead prior activation, but maybe a more usefull system wide control like "addAOPSystem(ClassLoader klassScope, SystemDefinition definition)" to be able to activate AOP constructs as a whole, at runtime, and leaving the internal recipe of it to AOP solutions and JVM implementations ...

Right ?

It's life

2004-02-17T13:02:00.000+01:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Most of my blog' fans (do I have blog fans ?) might think I am hacking AOP constructs and weaving architecture all the day.

Well I have to admit that this is not exactly true, thought almost true ;-).

These days I had to hack a small app that gather data from a WebLogic Server farm (more than 60 servers) by querying runtime JMX mbean so that we can then generate some graph to feed a NOC intranet.
Rather simple to play with JMX, have a fault tolerant client that does not crash when one instance is downed, and be able to handle at least 10 Go log daily.

For now the logs are parsed and consolidated as regular file. I would have liked to use a MySQL or HSQL database instead of parsing Go of files... but well, it is a customer driven story.

Here is a pretty cool graph that I got using JFreeChart that shows some http session data history grabbed from a subset of the farm.
Fun ?

AspectWerkz AOP - new pointcut expression implementation

2004-01-06T11:51:00.000+01:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Today I finished my implementation of the new AspectWerkz pointcut expression language.

AspectWerkz had already a good usability level as regards pointcut expression. Most of the needed syntax to bind pointcuts with advice or introduction had been implemented by Jonas, based on an algebra backed by Jexl.


/** @Aspect perJVM */
public class MyAspect extends Aspect {

  /** @Execution com.*.Foo.set(..) */
  Pointcut pc1;

  /** @Execution com.*.Foo.get(..) */
  Pointcut pc2;

  /** @Around pc1 OR pc2 */
  Object advice(Joinpoint jp) { .. }

}

The previous implementation supported the complete AND / OR / NOT algebra:


pc1 OR pc2
pc1 AND ( pc2 OR pc3 )
pc1 AND NOT pc2
...

And some synonyms were also provided, like &&, || and !, and other lowercased operators (or, and, not).

What I found bad with this Jexl based implementation was the following:

very hard to add new syntax operator (like IN for cflow pointcut composition)

added a programmed AST on top of the Jexl AST, but without providing a true Visitor pattern exposure

lost boolean expression optimization f.e. needed to evaluate all pointcut in an OR expression like pc1 OR pc2 even if pc1 matched

used only a tiny part of Jexl capabilities (Jexl is the evaluation engine of Velocity)

After some digging in Jexl extensibility I decided to start my own boolean expression implementation based on a JJTree grammar. In two days I had the complete operator set I needed, and a true Visitor pattern exposure.

In other two days I was able to refactor Jonas' work to plug my JJTree implementation in AspectWerkz. It allowed to solve a pending issue we had in the 0.9.RC1 release as regards cflow support for model 2 JSR-175 style Aspects.

Now we have many advantages at the hand:

no impact for the user

faster, especially when boolean optimizations are possible

cleaner, with Visitor pattern

more usable: with the new cflow composition thru the IN operator

From a user point of view this grammar add a new operator for cflow: the IN and NOT IN (or ! IN etc ...).
It allows to express in a natural way the pointcut expression with cflow.
pc1 OR pc2 IN cflow1
is then different from
( pc1 OR pc2 ) IN cflow1
[ the previous syntax was not natural : pc1 && cflow1 ]

In the next month, I should have time to add the needed glue to support several level of cflow composition - since this is already supported by our JJTree grammar :
pc1 IN ( cflow1 OR ! cflow2 )

This time, the trick will be more at the runtime during the joinpoint evaluation.

An awfull backend for AOP vs AspectWerkz (part 1)

2003-12-16T20:41:00.000+01:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Today while having my daily googling time on AOP, I found a 25 pages article about JMangler, a class load time weaving solution I had already bloged about.

The paper is here as a PDF.

I am rather surprised when I read that and mirror it with the fact that AOP would gain from massive adoption. I will defend my position in a very near future, but i could not avoid this flameware snip.

JMangler quotes itself as A Powerful Back-End for Aspect-Oriented Programming. And you might even buy a Prentice Hall book in 2004 to learn that.
I will save your time and money, by explaining you (again) why AOP does not gain from such a low concrete achievement strategy.

Just to let you think about by your own while I am writting my flameware:

Will you buy a book in 2004 to learn that JMangler is used in AspectWerkz ? This is false: JMangler had been in AspectWerkz but has been kicked out since july 2003 and we are happy.

Have your read that AspectWerkz hooking provides native support for BEA JRockit ?

Have you already worked with IBM JRE class loading scheme

Should a powerfull solution - that claim to be compliant with J2EE - fail under WebSphere environments ?
Should a powerfull solution be SUN JRE 1.4 centric ?

Should it hang when used in J2EE architectures ?

Should it run only in -Xdebug mode, with two JVM side by side ?

Is that a joke ? Abusing the AOP hype in the Academic sphere ?
Software is a moving thing, and AspectWerkz makes AOP move fast. Be ready.

AOP debugging makes its show

2003-11-21T07:37:00.000+01:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Samuel has done a viewlet where he demonstrates how to easily debug an application and its aspects with AspectWerkz from within Eclipse.

If you did not yet understood how standard and easy it was from my first annoucement, you have to watch it now !

All is explained step by step. This is a definitely a things to work with.

AspectWerkz announces support for BEA JRockit

2003-11-12T11:00:00.000+01:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

The latest snapshot of AspectWerkz is now supporting JRockit JVM for on the fly weaving.

By using JRockit JMAPI capabilities, AspectWerkz brings AOP one step ahead by officially supporting all major JVM (Sun, IBM, BEA). This allows to hook AspectWerkz in a seamless way to provide dynamic on the fly weaving of Aspects no matter your environment.

JRockit could not be supported before due to lack of HotSwap support and a limitation that forbids overriding of java.lang.ClassLoader with a -Xbootclasspath option. Users had to use offline mode until there.

The extension will be core part of coming 0.9 AspectWerkz release, so that all VM are supported with the smallest integration cost (Sun, IBM, JRockit).

Tests have been made with

JRockit 7SP4 (java 1.3.1)

JRockit 8.1SP1 (java 1.4.1_03)

The JMAPI JRockit API (management API) allows to add a classPreProcessor mechanism using a java snip or a command line option.

A single VM is thus running at full speed, both under java 1.3 and 1.4, without having to use bootclasspath override.

The classPreProcessor instance is responsible to modify bytecode of the class at load time. AspectWerkz was already providing this classPreProcessor based architecture since summer 2003 for Sun VM and Q3 2003 for IBM VM.

It is possible to hook or unhook AspectWerkz programmatically using JMAPI:


JVMFactory.getJVM().getClassLibrary().setClassPreProcessor(
   new JRockitPreProcessor());

To launch a JVM with AspectWerkz hooked in for class load time weaving this consists of a single option:


/jrockit/bin/java
-Xmanagement:class=org.codehaus.aspectwerkz.extension.jrockit.JRockitPreProcessor ....

If you are a JRockit fan, please help us and give a try to AspectWerkz online mode by your own. We need your feedback to make it suits your needs.

We will provide extensive documentation with the 0.9 release, but for the impatient :

grab a CVS snapshot

check bin/aspectwerk scripts to adapt them for JRockit (one line is commented out - notice that AspectWerkz needs to be in bootclasspath due to JMAPI)

have a look at org.codehaus.aspectwerkz.extension.jrockit.JRockitPreProcessor in src/extensions/

compile extensions.jar with ant/maven aspectwerkz:extensions:compile

give it a try

ThreadLocal and memory leaks

2003-11-03T12:14:00.000+01:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Last day I realized that using ThreadLocal can be very dangerous when it comes to long running applications and garbage collections.

When you read the documentation of the ThreadLocal, you read that the object put in a ThreadLocal context is associated to the thread, and will be garbaged once the thread is dead.

Each thread holds an implicit reference to its copy of a ThreadLocal as long as the thread is alive and the ThreadLocal object is accessible; after a thread goes away, all of its copies of ThreadLocal variables are subject to garbage collection (unless other references to these copies exist).

So if you use ThreadLocal to store some object instance there is a high risk to have the object stored in the thread local never garbaged when your app runs inside an app server like WebLogic Server, which manage a pool of working thread - even when the class that created this ThreadLocal instance is garbage collected.

To solve this issue, you will probably have to check for it first with a memory profiler tool like JProfiler, and then wrap the value put in the ThreadLocal in a WeakReference.

That did not appears to be obvious while reading the javadoc. Isn't it ?

AspectWerkz online mode and JSR 163

2003-10-20T11:58:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Since end of july the JSR 163 draft is in public review. I did not had time to read it before (shame on me !) and Jonas has send me the link yesterday.
(read the draft)

I am pretty happy with it. As you might know, we have struggled hard in AspectWerkz to provide a way to weave class at load time and keep compliant with complex class loader hierarchies as J2EE ones.
If JMangler was good for first AspectWerkz releases (prior to 0.8), having our own architecture was a major step forward (and read recent post on JMangler lists that state problems under J2EE environment where it is reported to freeze - as I reported it in Q2 2003).

On july 30 I complained about JVMPI API and was kidding at Sun, since they had forgetted some information I needed in the API.
(old article)

I forgot to subscribe to JSPA to gain access to JSR for early reviewing. I should have (shame on me!)
This JSR 163 sounds really good.
- First I will have the information I need in the class load hook event.
- Second, there seems to have support for in process hotswapping
- Last and not least, there seems to have no requirements for -Xdebug for the hotswapping part.

So adopt AspectWerkz online architecture today and you will have a preview of what this JSR 163 is going to provide in a simpler way for java 1.5. I have already a todo list for AspectWerkz online architecture when this java 1.5 will be ready for prime time.

And don't forget: subscribe to JSPA. I have sent the agreement to SUN this sunday !

An old crap we don't care about anymore

2003-10-01T10:42:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

With the 0.8 release and the new online mode architecture, a major limitation has been resolved. Now online mode (on the fly weaving) can be enabled with application servers (was released summer 2003).

Prior to 0.7.4 (including 0.7.4), it was not working correctly on some circumstances, due to a problem in the JMangler architecture which was used for online mode. You could use AspectWerkz offline mode as a fix to this.
Starting with 0.8, we have developped in AspectWerkz our own architecture, that fix the problem and brings much more functionnalities. AspectWerkz hook architecture is now used in CGLib as well since we made it available as a standalone module.

Some recent post on JMangler mailing lists reminded me this problem, and i want to give you some focus here about this topic, even if its an old story I don't care about anymore since the AspectWerkz way is now far from those limitations.

The problem was that JMangler is using a JPDA LaunchingConnector. This is a Sun provided JDI component that allow to launch a JVM in debug mode from a first JVM. The first JVM is responsible of hooking the low level layer in the second JVM when this one starts up. This interesting architecture provided by JMangler team has many problems.

The major one makes it bogus under heavy class loading scheme, where the application launched loads lots of class in a multi threaded way. This heavy class loading can appear when an application server starts. Lots of class are created and some execute thread queues are initialized, threads deploy applications, jdbc drivers are loaded ...

I did some test under a weblogic 7 startup sequence, and it just hangs. Hopefully AspectWerkz now works perfecly fine with the new 0.8 architecture.

If you are interested in reproducing the bug, which seems to be a JPDA bug that thus affects the whole JMangler 3.x architecture and thus AspectWerkz version prior to 0.8, you can use the following sample application (read the extended entry below).

Download the test application

Read more about AspectWerkz 0.8
Read more about the new on the fly weaving architecture

Description of the testing application:
A sample main class application starts T threads (configurable).
Each thread runs L loops (configurable).
At each loop the thread instantiates a new URLClassLoader with a search path on file system from which it loads a single class and instantiate a first instance. The class static init section (clinit) loads itself another class using another new URLClassLoader.

Each class loading is written on stdout: classname + hashcode. Hashcode allows to identify classes loaded by differents classloader which are not hierarchically linked.

This simulates a startup sequence where a big number of class is loaded by several threads ( 2 x T x L class). No class reference is kept so all are elligible to garbage collection.

Results
The app performs well with 3 threads and 2000 loops (thus 12 000 class loaded) (use a heap of 128m and a new of 32m, used -verbose:gc option). Turning on -Xdebug or -Runjdwp does not affect the correct behavior.

When started thru JMangler or used with a jpda LaunchingConnector - the component used in JMangler 3.x - it does not complete, and the launching VM garbage collection sometimes turns completely crazy making a lot of minor garbage collections at around 6000 class loaded, and the whole app always hangs.

Tests were run under win2000 SP3 and solaris, under different java 1.4 version, with the same results.

Explanation
I don't have explanations. AspectWerkz 0.8 uses a ListeningConnector coupled to an AttachingConnector instead, and it performs fine.
Now you can startup a complete application server with AspectWerkz hooked in in online mode with a minimal overhead.
And the 0.8 release comes with many more options on this important topic, that makes AspectWerkz on the fly weaving an easy and non intrusive operation.

Download the test application

Read more about AspectWerkz 0.8
Read more about the new on the fly weaving architecture

AspectWerkz has support for IBM

2003-09-03T16:06:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

I managed to bring IBM jdk support to AspectWerkz for on the fly weaving.
Read more on project blog

IBM bundles a specific JRE that I did not planned when hacking the JVM hook architecture that allow on the fly weaving, so some of you reported JVM crash. Now it is fixed (in CVS HEAD) and for other exotic VM, an explicit error message should be reported.

If you have issues with it, please provide feedback. The IBM jre (at least 1.3.x) seems to have many issues as regards jars packaging. f.e I had to remove index.lst from meta-inf/ in two third parties jars for it to work...

debug your AOP with AspectWerkz

2003-09-03T13:30:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Using the latest development of AspectWerkz, and thus the recently released 0.8.1, debugging step by step your aspects is made easy.

Indeed we now have support for standard IDE debugging, so that bytecode weaving process can be followed step by step as regular java.

Then when your weaved application is running, you can set breakpoints on your advices and debug the app as usual, as plain java, as you are probably used to do it.
Though the IDE does not have access to the source of the weaved classes, you can set breakpoints in the original method affected by the pointcuts where aspects are bounded.

Here are some screenshot with IntelliJ.

Here is a short how-to:

Create a new project in your IDE, with AspectWerkz source distribution and your own sources.

Configure all classpath as needed.

On java 1.3, copy piccolo-xy.jar in your JAVA_HOME/jre/lib/ext/ (and remove it after since Ant does not like to have it there...)

Run or debug the class org.codehaus.aspectwerkz.hook.impl.WeavingClassLoader as usual with your IDE.
Use this class as a java command replacement.

Do not forget to define the -Daspectwerkz.definition.file

For example I am running it with the following to run a single unit test.

JVM arg = -Daspectwerkz.definition.file=src\test\aspectwerkz-test.xml -Daspectwerkz.transform.verbose=yes -Daspectwerkz.transform.dump=test..*


Program parameter = test.DynamicDeploymentTest


Working directory = my AspectWerkz home directory

AspectWerkz - new on the fly weaving architecture released

2003-08-14T14:00:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

AspectWerkz 0.8 is released.

This version is a major change from the previous releases, since the whole on the fly weaving architecture has been reviewed.
There is now support for java 1.3, no debug mode required with certain options on java 1.4, and the most seamless way to plug the AspectWerkz AOP weaver in your environment thru a native module !


java -Xrunaspectwerkz [options] [classpath] my.Application

just hooks in the weaver for all the class loaded by the JVM (except bootstrap classloader).

Try it now

Read the official slides presenting the new architecture (pdf, ppt)

Read the documentation related to the new architecture

AspectWerkz online architecture - better than JVMPI - Sun needs to think about it for java 1.5

2003-07-30T12:05:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

I already talked about the coming AspectWerkz new architecture for online mode (on the fly weaving). The key advantages will be as I explained:
- java 1.3 support
- debug mode for JVM not required
- option for single jvm
You may think its useless since the online weaving is already working fine (almost) with JMangler 3.x, which came with the original terrific idea of hotswapping the java.lang.ClassLoader, but I can bet you will change your position once you will have played with it a little more, thru the official 0.8 release.

Today I gave a look at the JVMPI API. Its a C layer receiving events from the JVM. Two events could be usefull to handle for AOP implementations: JVMPI_EVENT_CLASS_LOAD (sent when a class is loaded in the VM) and
JVMPI_EVENT_CLASS_LOAD_HOOK (sent when the VM obtains a class file data, but before it constructs the in-memory representation for that class).

Coding a layer receiving JVMPI_EVENT_CLASS_LOAD_HOOK to transform the bytecode could be usefull, since it could be enabled with just a -XrunMyStuff option and a shared library. This is the approach many commercial or open source debugging and profiling tools have choosen.

But giving a deeper look implementing an AOP layer on it will fail shortly for the following reasons:
- the event does not give the caller classloader information
- it might be a hell to call the java implemented AOP layer (AspectWerkz off course) from the C/C++ MyStuff JVMPI addon. Unless we setup a kind of client / server mode with a protocol to transmit bytecode back and forth (see AdaptJ for this sort of things).
Could lead to the idea of a distributed weaving engine... but do you need it ? And how will we be able to handle the calller classloader distinction, for example to weave struts classes in a specifc war only ?

So I definetely think the JMangler idea was good, even if the implementation is not that much good. And thus I am sure the new AspectWerkz architecture which unleash the idea will be and is already killing...

JVMPI doc
adaptJ project
JMangler project

As a ending note, I am really surprised Sun implemented the JVMPI_EVENT_CLASS_LOAD_HOOK in JVMPI and not directly (and even both) in the java.lang.ClassLoader... since its almost the single issue an on the fly weaver has to solve... since we have to override a class of rt.jar for java 1.3 compatibility... since BEA has implemented it since more than 2 years (except for the system classloader, just for the custom classloader deploying application like ear, war, ejb-jar, rar, but this might change with WebLogic 9)... since several project tries to address this lack (JMangler, beSee, AspectWerkz, JBoss, AspectJ)

Sun engineers - wake up - put that stuff in java 1.5 - java AOP needs it !
Ready to make a lobby with me ?

beSee donated to AspectWerkz

2003-07-23T18:38:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur, read comments there)

Past 6 month I have founded and developped the beSee project. This project is now closed, but I am definitely happy since the core architecture is now part of AspectWerkz, founded by Jonas Boner, for which I am commiter.

AspectWerkz power will be unleashed in the next weeks, and let me explain why.

It all started with beSee-1-x. This project aimed to provide a runtime bytecode instrumentation solution for BEA WebLogic Server (v7), based on Javassist for bytecode instrumentation and BEA ClassPreProcessor feature, which allow to modify bytecode of deployed applications when they are loaded in the app server.
beSee-1-x allowed easy instrumentation, based on a property file, and provided JMX management feature for hot configuration. It was easy to instrument on the fly some classes and run a load test on my app.

But there was something missing: BEA ClassPreProcessor does not allow instrumentation of core server classes or jvm wide component, like a JDBC driver, so I was unable to instrument enough my CMPs to track down SQL query being executed.

I started beSee-2-x architecture. It provides a JVM wide PreProcessor hook, allowing to extend what BEA provided for deployed application to the whole JVM, and thus to any product running java (!).
I knew JMangler add this kind of feature, but after some testing it appeared to be heavy and not that much stable as regards app server integration (let me know and I will send you some test case for it, and read more, you'll learn why).

So the main features of beSee-2-x I implemented are:
- plugable JVM wide ClassPreProcessor mechanism for on the fly instrumentation
- bytecode kit independancy (BCEL and Javassist provided, any other supported)
- offline compiler (the AspectJ way)
- performance wise:
- beSee-2-x does not requires JVM to run in debug mode (JMangler does)
- beSee-2-x is java 1.3 compatible (JMangler is not)
- beSee-2-x auto adapts behavior for java 1.3 and 1.4 (process forking or hotswapping)
- beSee-2-x allow to disable dual process, thus a standard java ... command to launch the target application to instrument on the fly is enough - no need for a wrapper script and stream piping in background (required when hotswap or process forking is used).

beSee-2-x allowed me to unleash the power of beSee-1-x to the whole weblogic, not just the deployed app.
And crucial point, its open architecture was ready to unleash power and capabilities of next generation AOP framework - and AspectWerkz is the one !

So stay tune. All those features are right now in AspectWerkz CVS, and in some weeks, you won't feel the pain to plug AOP in your whole architecture, on the fly or offline, and with its full power - no debug mode required, java 1.3 supported, optional process forking (choose between tuning and easy set-up), not tight to any app server but certified app server ready.

BCEL patch submitted

2003-07-23T05:43:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur)

I have submitted a patch for the BCEL thread safe issue I described (read).

You'll find it on the BCEL dev mailing list archive. I am waiting for BCEL guy to put the patch in their CVS HEAD. For now you can whether incorporate the patch in the BCEL source and rebuild your own BCEL, or build a jar with only the two files I patched and put it in your classpath before the bcel.jar.

(read the post on the list archive)

(dowload the patch for BCEL 5.1)

BCEL internal

2003-07-19T12:01:00.000+02:00

(imported from http://blogs.codehaus.org/people/avasseur)

Using BCEL to do bytecode modification on a simple application I have, I encountered problems with thread safety. As soon as I got 2 threads instrumenting in parallel, the java Verifier begins to complains about generated method signatures.

I struggled half a day to find it out. Thread safety problem is not that easy to isolate. After some googling I found the following post on BCEL mailing lists

It appears that some of BCEL static class (org.apache.bcel.generic.Type and org.apache.bcel.classfile.Utiity at least), which should provides static method to compute human understandable signature from bytecode and vice-versa are miscoded.

If I understand that doing a thread safe application can be difficult, I am really thinking BCEL guy were lazy this time with a snip like

// sample purpose - pseudo code

package org.apache.bcel.classfile;

public abstract class Utility {

public static int consumed_chars;

public static String[] methodSignatureArgumentTypes(String) {
loop {
     // read and set consumed_chars field
     consumed_chars++;
}
}

public static String methodSignatureToString(String) {
loop {
    // read and set consumed_chars field
    consumed_chars++;
}
}

}

As we can see its obvious it is non thread safe because its bad designed: two static methods are using the same static field as a temporary storage to perform string operations.

I had to wrap the consumed_chars in a ThreadLocal and replace all calls. It works fine, but I cannot be sure I covered all the bad design BCEL put in.

Guy, ok to say BCEL is not designed for thread safety in the FAQ. But please try to avoid such ugly coding.

Another option for me could be to add some synchronized in my instrumentation application - but it would be a hell from a performance point of view.
Considering BCEL is used in app server modules (npulse, willy technology), it should be thread safe. In app server there is lots of classloading ocuring concurenlty (stub / skel generation, CMP impl at deployment time ...)