Trifork Blog

Spring Insight plugin for the Axon CQRS framework

November 13th, 2012 by
|

Introduction

In a previous blog post we introduced the Spring Insight module that’s part of SpringSource’s tc Server developer edition and gives you, well, insight into what’s taking up the time to process requests in your applications. Insight does this by instrumenting your application at load time using AspectJ: a single call to your application will result in a so-called trace consisting of various operations, and for each of these operations it’s measured how long it takes to execute. An operation can provide additional details: for example, when sending a query to a database using a JDBC Connection the actual SQL statement including prepared statement bindings will be stored in the operation.

Spring Insight is not a profiler: it doesn’t create operations for each and every little thing that happens, but only for ‘significant’ events, like a Spring-MVC controller invocation, the start of a transaction or communication with a database. These events are called ‘operations’. It does allow assigning operations to so-called endpoints. An endpoint could be a controller method, or a method on a class annotated with one of Spring’s stereotype annotations like @Service or @Repository for example. For these endpoints Insight can provide statistics across traces, so that you can measure the health of these endpoints during the lifespan of your application based on configurable thresholds.

Insight’s plugin model

Spring Insight consists of a core plus a set of plugins that are shipped with the tool. These plugins define the actual operations and endpoints that Insight knows about and exposes. One of the nice things about Spring Insight is that the plugins use a public API that’s not restricted to the built-in plugins: you can build your own plugins to teach the tool new tricks.

Although you could do this on a per-application basis to expose metrics relevant to your particular app, you wouldn’t usually write a dedicated plugin for that. Insight already exposes several application-level methods as operations if they’re part of your stereotype-annotated Spring beans, and you can use their set of annotations to expose additional application-specific operations and endpoints easily .

Plugins are much more useful for framework developers: their framework might contain several methods that would be interesting to expose as Insight operations or even endpoints to show users what the framework is doing and how long that takes. Earlier this year, that’s exactly what we did for the Axon CQRS framework that’s being developed within Trifork.

This blog briefly discusses the plugin’s implementation. All source code has been added to the Axon Framework itself and is available on Github.

Update:
After publishing this blog entry, VMware has contacted  us to host the source code for this plugin in their public GitHub repository, so that the plugin can be shipped out-of-the-box with the Spring Insight distribution. That means that the plugin sources are no longer found under the Axon repository. The link above has been updated to reflect this change.

The Axon Insight Plugin

Applications built with Axon create and handle commands and events. Commands are dispatched through a command bus to a single command handler, and the handling of a command typically leads to one or more events. These events are published to all interested event handlers for further consumption. There are some other relevant concepts like sagas, but for the purpose of explaining the plugin this is pretty much all you need to know about Axon’s model.

This means that Axon-related processing originates from commands. Both the dispatching and handling of a command are thus interesting operations to capture. To identify these sort of operations you have to write an AspectJ pointcut expression that matches the relevant method execution.

An Axon command handler method is typically identified by an annotation, but can also be the implementation of a CommandHandler interface. Moreover, the support for annotation-based handlers is actually provided by a framework implementation of this interface. This means we want to match executions of annotated methods and interface methods, but we should exclude the built-in CommandHandler implementation to avoid the creation of two operations for a single command handling event in the case of annotated methods.
Pointcuts allow you to express exactly this information in a concise manner. This is what the definition used by the plugin looks like:

public pointcut collectionPoint():
        execution(@org.axonframework.commandhandling.annotation.CommandHandler * *(*, ..)) ||
                (execution(* org.axonframework.commandhandling.CommandHandler.handle(*, org.axonframework.unitofwork.UnitOfWork))
                        && !within(org.axonframework.commandhandling.annotation.AnnotationCommandHandlerAdapter));

Based on that named pointcut you can then create an Operation instance. An operation contains information about the context it was captured in: this includes a type, label and source code location but can be extended with arbitrary additional information that’s relevant to your use case.

What’s interesting in the case of Axon is that the current version under development, version 2.0, has a different way of representing commands internally than the production 1.x versions. Prior to the 2.0 version a command was always a simple POJO, whereas starting with 2.0 the programming model still supports POJOs but internally a CommandMessage object is used. This CommandMessage can contain additional metadata next to the command object itself that it wraps as its payload. If your command handler accepts a CommandMessage rather than a POJO as its argument, it’s nice to let the plugin capture that metadata as part of the Operation that is created.

The plugin supports both Axon 1.x and 2.0, so it checks what version is being used and conditionally adds the extra info. Since most of the work of creating an Operation is boilerplate code (capturing the source location for a method invocation is always the same) it’s easiest to let your command creation aspect extend an Insight-provided base aspect. Here’s the full code (without imports)  for the aspect that captures command handler invocations and creates the corresponding operations:

public aspect CommandHandlerOperationCollectionAspect extends MethodOperationCollectionAspect {

    public pointcut collectionPoint():
            execution(@org.axonframework.commandhandling.annotation.CommandHandler * *(*, ..)) ||
                    (execution(* org.axonframework.commandhandling.CommandHandler.handle(*, org.axonframework.unitofwork.UnitOfWork))
                            && !within(org.axonframework.commandhandling.annotation.AnnotationCommandHandlerAdapter));

    @Override
    protected Operation createOperation(JoinPoint jp) {
        Operation operation = super.createOperation(jp).type(AxonOperationType.COMMAND_HANDLER);
        Object[] args = jp.getArgs();
        if (!AxonVersion.IS_AXON_1X) {
            if (Axon20OperationUtils.processCommandMessage(args, operation)) {
                // we're done here
                return operation;
            }
        }
        Object command = args[0];
        operation.put("commandType", command.getClass().getName());
        return operation;
    }
}

In this case it’s fairly straightforward to deal with the two Axon versions in a single class (the Axon20OperationUtils handles the metadata part if the command is in fact a CommandMessage instance): for some other cases, like the actual dispatching of a command by the CommandBus, the plugin has dedicated aspects per version with pointcuts that are specific to either Axon 1.x or 2.0.

The plugin defines operations for the following events:

  • Command dispatching by the command bus
  • Command handling by a command handler
  • Event publishing by the event bus
  • Event handling by an event handler
  • Event handling by a saga

The result of doing this can be seen in the following screenshot of a trace taken from the Axon Trader sample application:

Axon Trader Insight screenshot

Note the event sourcing in action: to handle the remove contact command, an aggregate is first loaded from a snapshot and subsequent events obtained from the event store. Had the aggregate been cached already, then obviously the trace wouldn’t have shown these JDBC SELECT statements.

Endpoint analyzers

In addition to providing these Axon-specific operations, the plugin also defines several endpoint analyzers. Endpoint analyzers can be used to define new endpoints that will show up in the Spring Insight UI so that their health can be monitored across requests handled by the application. Not every operation needs to be associated with an endpoint: endpoints are often application components that perform some type of handling of dispatched requests in your application, like Spring-MVC controllers. That means that it’s interesting to define endpoints for command handlers, and potentially for (saga or regular) event handlers as well. Currently the plugin supports all three, although feedback from real-world usage might lead to the conclusion that defining endpoints for event handlers is a bit too fine grained.

The endpoint analyzer implementation for command and event handlers is similar enough that most of the code has been extracted into a common baseclass. I won’t discuss the details here, but when you start with Insight plugin development you’ll see that the code is very similar to that of the built-in plugins for e.g. Spring-MVC Controller support.

By having something like a command handler endpoint, you can monitor how long it takes to handle certain types of commands during the lifecycle of your Axon application. Especially when event handling is done synchronously this will give you a good feel for what parts of your application might contain a bottleneck. By then drilling down into representative traces that have been captured you can see where the actual time is being spent.

Here’s a screenshot of what this looks like for the Axon Trader; note how HTTP and Axon endpoints are both shown:

Conclusion

As you can see, you can gain a lot of insight into the workings and performance of your Axon-based applications using the Insight plugin with relatively little effort. The current plugin is an initial proof of concept and can surely be expanded and improved: of course we’d love to hear your feedback for improvements and other suggestions!

7 Responses

  1. November 13, 2012 at 13:36 by William Louth

    >> it doesn’t create operations for each and every little thing that happens, but only for ‘significant’ events

    Please define “significant event” in the context of a trading workflow, online gaming world, a compiler such as scala,….hmmm

    The flaw in your logic and why SI has always being DOA is that you think you know what is significant whilst typing in an editor with little runtime or workload context to make even an “wild” guess never mind an informed guess. There is a reason why the JVM is adaptive in its compilation and optimization strategies and behaviors and that rule pretty much applies to performance measurement.

    Incidentally what does all those methods you deem “significant events” actually do…a bet they call millions of insignificant events…unless of course we are talking about the obvious problem with most web apps…meet by bottleneck…say hello database 😉

    • November 14, 2012 at 11:05 by Joris Kuipers

      Significant in the context of SI means an important operation for the average enterprise web app from a component-based view, something that fits within the mental model a developer has of the application and of which you might want to track the average execution duration over time. That’s not necessarily the actual operation that eats up your CPU time in case of a performance issue: if an operation does take more time than expected to execute, and it’s not something like a DB statement that is shown to be the culprit, you might use an actual profiler to drill down to find the root cause, sure.
      Suggesting that my logic is flawed by saying that I claim to know what is significant in the context of a profiler at development time is just a misinterpretation of my words, since I’m not talking about profiling.

      I’d prefer to reserve the comments section of this posting for comments that I consider to be on-topic, namely the development of Spring Insight plugins: please use other platforms to comment on your or other vendors’ tools.

  2. November 15, 2012 at 09:55 by William Louth

    Why would a “relevant” class based view be so different from a component based view when it comes to monitoring which is focused on workload (throughput, response time, capacity, consumption). Do you really want to have multiple tools measuring different things in different ways with different levels of accuracy and precision in production just so a developer can make some sense out of the flow irrespective of the fact this might have very little bearing on the success in detecting and resolving both software execution (inefficiencies) model and system execution (bottlenecks) model issues.

    Your logic is flawed because I actually attempted this…this is my domain of expertise and more qualified than anyone I know of to make such judgements…but maybe that is not so obvious to a person who has yet to travel on this path. In JXInsight 5.7 (before occur & adaptive/intelligent activity metering) we had 768 separate instrumentation extension libraries…covering all of the popular frameworks and OSS projects. But the problem was that even then a customer could very easily deploy 30 of so extensions and pollute their model both in terms of data collection and measurement overhead. So what was the customer to do but decide which of the extensions he would deploy and guess what…yes the generally guessed wrong when it came to an issue in production (though this might be biased sample). Even when they did have sufficient coverage at the component level it only confirmed what their users or external tx probes had observed…performance was bad…but it was not sufficient to help get to the root cause especially when we are not talking about poor performing SQL or RPC.

    Component views are good for (pre-)deployment purposes but when it comes to performance on the big scale its the code, consumption and cost view that makes the difference between “thats a nice pretty demo dashboard” to “we have accurately identified the underlying causes and factors”. They are only good for making people foolishly believe they have the proper insurance in place.

  3. December 12, 2012 at 20:08 by Lukasz

    Source code is no longer available.

  4. January 16, 2013 at 11:52 by Joris Kuipers

    The link to the sources has been updated: VMware has asked us if they could host the source as part of their GitHub repo, so they’re no longer found in the Axon repo.

  5. […] was motivated by our recent post on the Spring Insight plugin that we wrote for Axon Framework. In response to that post VMware has […]