Trifork Blog

Posts by Joris Kuipers

Twitter
« Older Entries

Integrating the AWS Parameter Store with Spring Cloud

July 20th, 2018 by
(https://blog.trifork.com/2018/07/20/integrating-the-aws-parameter-store-with-spring-cloud/)

I’ll tell you all my secrets (but I lie about my past)
— Tom Waits – Tango till they’re sore

tl;dr

We’ve integrated the AWS Parameter Store with Spring Cloud so that it can be used as a secure configuration backend for services deployed to EC2, including ECS. This code has recently been merged in Spring Cloud AWS and is available in its 2.0 release.

Introduction

At the moment I’m working on a project where we’re developing a microservices-based system based on Spring Cloud for the Dutch Lotteries. The services are deployed on Amazon Web Services using Amazon’s current Docker support (ECS).

When we started late last year, we decided to use Consul, both as a service registry and as a key-value store for configuration. Spring Cloud has excellent built-in integration with Consul, both for service discovery as well as for using it as a shared configuration backend.

However, we quickly found out that we needed an internal load balancer to allow ECS to perform health checks on the services, so we might as well use that for server-side load balancing. This eliminated the need for client-side routing and service discovery. Furthermore, we weren’t too happy with the options to easily restrict access to secrets stored as config in Consul and were looking for a configuration service provided by AWS (rather than e.g. Vault) so that we’d no longer need to operate our own Consul cluster or other middleware.

AWS Parameter Store

When we looked for alternative solutions we soon found the AWS Parameter Store: it’s an option provided by EC2 to store all sorts of configuration parameters, including secrets that are encrypted at rest. Using IAM roles you can restrict access to parameters, which can have nested paths that can be used to define ACL-like access constraints. It also integrates with ECS quite nicely, by allowing containers to retrieve credentials to access the store, and provides versioning of parameter values.

This screenshot provides an impression of the corresponding console:

However, when looking for integration with Spring Cloud I just found some open tickets, so I decided to try to develop some integration myself. This blog post describes the result of that effort.

Read the rest of this entry »

Tags: , ,
Posted in DevOps, Java, Software Development, Spring | 9 Comments »

How to manage Database Migrations with Flyway?

December 5th, 2017 by
(https://blog.trifork.com/2017/12/05/how-to-do-database-migration-with-flyway/)

Joris Kuipers, CTO at Trifork, presented a webinar on some usage patterns for Flyway. You can find the recording on our Trifork YouTube channel.

Tools like Flyway address a common concern for many people, which quickly leads to questions on how to pick a tool and then apply it in the best manner for one’s particular situation.

In this blog post, Joris has summarized the Q&A session – he provides the readers with his answers and ideas for managing database migrations.

1. How does Flyway compare to Liquibase?

When I was choosing a DB schema migration tool 4 or 5 years ago, I’ve looked at both Liquibase and Flyway. In general, I’d say that Flyway is a bit more light-weight: Liquibase addresses some requirements that Flyway explicitly chooses not to support. These include support to define migrations declaratively (e.g. in XML) and then generate the correct SQL DDL statements for your particular DBMS and the support to generate migration rollbacks (countering operations).
Read the rest of this entry »

Tags: , , , , ,
Posted in Software Development | No Comments »

How to send your Spring Batch Job log messages to a separate file

April 14th, 2017 by
(https://blog.trifork.com/2017/04/14/how-to-send-your-spring-batch-job-log-messages-to-a-separate-file/)

In one of my current projects we’re developing a web application which also has a couple of dozen batch jobs that perform all sort of tasks at particular times. These jobs produce quite a bit of logging output when they’re run, which is important to see what has happened during a job exactly. What we noticed however, is that the batch logging would make it hard to quickly spot the other logging performed by the application while also running a batch job. In addition to that, it wasn’t always clear in the context of what job a log statement was issued.
To address these issues I came up with a simple solution based on Logback Filters, which I’ll describe in this blog.

Logback Appenders

We’re using Logback as a logging framework. Logback defines the concept of appenders: appenders are responsible for handling the actual log messages emitted by the loggers in the application by writing them to the console, to a file, to a socket, etc.
Many applications define one or more appenders and them simply list them all as part of their root logger section in the logback.xml configuration file:

<configuration scan="true">

  <appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
    <destination>logstash-server</destination>
    <encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
  </appender>

  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>log/server.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
      <fileNamePattern>log/server.%d{yyyy-MM-dd}.log</fileNamePattern>
      <maxHistory>30</maxHistory>
    </rollingPolicy>
    <encoder>
      <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %mdc %-5level %logger{36} - %msg%n</pattern>
    </encoder>
  </appender>
  <root level="info">
    <appender-ref ref="LOGSTASH"/>
    <appender-ref ref="FILE"/>
  </root>

</configuration>

This setup will send all log messages to both of the configured appenders. Read the rest of this entry »

Tags: , , ,
Posted in DevOps, From The Trenches, Java | 1 Comment »

Using Spring Session for concurrent session control in a clustered environment

April 8th, 2016 by
(https://blog.trifork.com/2016/04/08/spring-session-concurrent-session-control/)

For a long time, Spring Security has provided support to limit the number of sessions a single user can have concurrently. This prevents users from being logged in from many different devices at the same time, for example to ensure that they won’t share their credentials to a paid site with their friends and family.

My former colleague Quinten Krijger has blogged about this feature before. Note the last paragraph, which explains how this support is limited to single-node applications.

Although running on a single node may suffice for many applications, there are plenty applications running in a clustered environment that should be able to benefit from concurrent session control as well. As hinted in the aforementioned blog, this requires both implementing a custom SessionRegistry as well as ensuring that expiring a session is propagated to all nodes in the cluster.

This is exactly what I’ve done recently using Spring Session, a framework that allows you to take control over managing sessions using a shared external registry like Redis. In this post I’d like to walk you through the code, which can be found here: https://github.com/jkuipers/spring-session-concurrent-session-control

UPDATE:

Based on the code I wrote for this blog I’ve opened a pull request for Spring Session. That request is scheduled for inclusion in Spring Session 1.3, but the code works just fine with the upcoming 1.2 release and removes the limitation of not providing an expiry notification after exceeding the maximum number of sessions.

Read the rest of this entry »

Tags:
Posted in Software Development, Spring | 5 Comments »

Spring-AMQP and payload validation: some notes from the trenches

February 29th, 2016 by
(https://blog.trifork.com/2016/02/29/spring-amqp-payload-validation/)

It’s been a while since I’ve written one of our from-the-trenches blogs: that’s mostly because I’ve been very busy in those trenches developing systems for our customers.

This week I completed a Spring Boot-based microservice which is responsible for interacting with some 3rd party SOAP service: its own clients communicate with it by sending request message over RabbitMQ, and the service then sends back a response to a response queue after handling the SOAP response message.

Of course I used Spring-AMQP to build this service. Spring-AMQP supports a nice annotation-based listener programming model, based on Spring’s generic Message support.
That allows writing listener methods like this:

@RabbitListener(queues = REQUEST_QUEUE) 
public DeclarationResponse submitDeclaration(DeclarationRequest request) { 
  // handle the request and return a response 
}

The request parameter here is the result of converting the incoming AMQP message using a Spring-AMQP MessageConverter, after which it is considered to be the payload of the message (even when headers are used in the conversion as well).

The request messages that the clients send have some required fields: without those fields, the service can’t make the SOAP calls. While reading the RabbitListener JavaDoc I noticed that Spring-AMQP allows you to apply validation to message payload parameters by annotating it. When using this, you also have to add the @Payload annotation (which is optional without validation if your method doesn’t have any other arguments), so the result looks like this:

@RabbitListener(queues = REQUEST_QUEUE)
public DeclarationResponse submitDeclaration(@Valid @Payload DeclarationRequest request) { … }

By the way, Spring’s own @Validated (even as a meta-annotation) and in fact every annotation whose name starts with “Valid” are supported for this purpose as well.

Now we can add some JSR-303 Bean Validation annotations to the fields in our DeclarationRequest, like @NotNull, to express our validation constraints.

Read the rest of this entry »

Tags: , , ,
Posted in From The Trenches, Software Development, Spring | 2 Comments »

Booting your Microservices Architecture with Spring and Netflix: the aftermath

November 26th, 2015 by
(https://blog.trifork.com/2015/11/26/booting-your-microservices-architecture-with-spring-and-netflix-the-aftermatch/)

On 25 November Trifork hosted a webinar in which I gave a short overview of Spring Cloud and its support for the Netflix OSS stack, focusing on Spring Cloud Config and the support for Netflix’s Eureka, Ribbon and Hystrix.

We’ve been investigating this stack over the last couple of months and are using parts of it in production already: we have found that a lot of the common concerns that you need to tackle as you’re moving into a distributed systems architecture are nicely covered and, in many cases, even abstracted by the Spring Cloud platform. In a typical Spring fashion, this allows you as an application developer to focus more on your business logic while letting the framework handle the concerns related to things like accessing shared configuration, working with service registries, handling failing downstream services, etc.

This blog provides you with background info to accompany the webinar, which has been recorded and can be found on our YouTube channel.
The code has been published on GitHub, as well as the accompanying config repository, in case you’d like to code along with the video.

Read the rest of this entry »

Posted in Software Development, Spring | Comments Off on Booting your Microservices Architecture with Spring and Netflix: the aftermath

Declarative multi-tenant security with Spring Security and Spring-MVC

September 5th, 2013 by
(https://blog.trifork.com/2013/09/05/multi-tenant-security/)

It’s been a while since our last ‘from the trenches’ entry, and as I’ve found I am better at authoring blogs than convincing colleagues to do the same I figured I’d write you another installment. This time I’d like to focus on an easy yet powerful approach that we used to secure a multi-tenant Spring-MVC application using Spring Security and its support for annotation-based declarative authorization.

Introduction

If you’re developing enterprise web applications, then you have certainly applied some form of security to your apps. In some cases it suffices to come up with a number of roles that you can assign to (groups of) users and to perform authorization based on that. However, in many cases that’s not enough and the concept of data access control comes into play: only users that are somehow related to the data they’re trying to work with should be allowed to access that data. This can be through direct ownership, access control lists, some temporary relation like a doctor-patient treatment relationship, etc. A common requirement in multi-tenant applications, where a single application instance is used by people from different organizations whose data should be strictly separated, is that data should only be accessible by people who work for the organization that that data belongs to.

In this blog we’ll show you one approach that we used to implement this with Spring Security.

Read the rest of this entry »

Tags: ,
Posted in From The Trenches, Software Development | 2 Comments »

Adding user info to log entries in a multi-user app using Mapped Diagnostic Context

June 6th, 2013 by
(https://blog.trifork.com/2013/06/06/adding-user-info-to-log-entries-in-a-multi-user-app-using-mapped-diagnostic-context/)

Have you ever been in the situation where you were looking at one of your production log files and had a hard time seeing what log entries belonged to what requests? In a multi-user web application many requests are handled in parallel, so without additional context it becomes almost impossible to see how a single user is interacting with your application.

Similarly, when someone reports a problem with the application you can’t really tell what to look for: how do you know that some error in the log was actually caused by the user reporting the issue?

The good news is that there’s an easy solution to this problem, that has been around for a long time already and is very easy to add to your existing applications: use a Mapped Diagnostic Context

MDC explained

By now, you’re either saying “duh, I’ve been doing that for ages” or “what the hell is an MDC?”. In my experience as a consultant, a surprising number of people fall in the latter category; if you do as well, then this new “from the trenches” blog is for you.

Read the rest of this entry »

Tags: , ,
Posted in From The Trenches, Software Development | Comments Off on Adding user info to log entries in a multi-user app using Mapped Diagnostic Context

From the trenches: Rendering Twitter Bootstrap form inputs for Spring-MVC with FreeMarker macros

May 27th, 2013 by
(https://blog.trifork.com/2013/05/27/bootstrap-spring-mvc-form-input-freemarker-macros/)

This is the first entry in a new series of what we hope will become a regular appearance on our Trifork blog: “from the trenches”, with blogs that explain in a no-nonsense just-the-tech fashion how we applied certain tools, technologies, libraries and frameworks in our own projects to solve real-world challenges that we faced while building solutions for our clients. We hope that by sharing the solutions we came up with, we can provide you with some examples and ideas that you can apply to your own projects — with the confidence of knowing that these techniques have proven themselves in real production systems.

We are also open to your suggestions on what technologies you might want us to cover. So don’t be shy and just drop us a note if you have any ideas.

In this first blog I’d like to cover how we used FreeMarker to build a small library of macros that make it very easy and effort-free to work with Twitter Bootstrap-based forms in a Spring-MVC application.

TL;DR: we’ve built a set of FreeMarker form macros for Spring-MVC apps with a Twitter Bootstrap-based frontend that allow one-liners in your form templates to render full Bootstrap-based HTML structures for various input types that include internationalized labels and rendering of binding and validation errors while allowing passing in custom attributes and nested contents. It’s awesome.

Read the rest of this entry »

Posted in From The Trenches, Frontend, Java, Software Development | Comments Off on From the trenches: Rendering Twitter Bootstrap form inputs for Spring-MVC with FreeMarker macros

Putting the pedal to the Mongo metal

March 1st, 2013 by
(https://blog.trifork.com/2013/03/01/putting-the-pedal-to-the-mongo-metal/)

At Trifork we’re always looking to get the best performance out of our systems. As a 10gen partner that means that we also try to squeeze the most out of our MongoDB deployments in terms of read and write throughput. Experience has shown that it matters greatly whether those deployments are performed on dedicated hardware or on virtual machines: especially having enough and fast RAM and disk IO can make all the difference.

If you’re interested in reading up on this topic, make sure to check out this article on SoftLayer’s blog. It shows how hosted MongoDB deployments can perform when given the appropriate hardware, and how having a hosting provider that understands the technology running on their infrastructure can help you to achieve the performance that your applications require.

If you’re in The Netherlands and want to learn more about MongoDB, whether you’re a newbee or an experienced user already, you should sign up for  the upcoming Mongo User Group meetup that’s taking place coming Tuesday as well: SoftLayer will be there to tell more about their hosted MongoDB offering, 10gen’s Alvin Richards will talk about what’s planned for the upcoming releases of MongoDB (spoiler: it includes search!) and we’ll have Open Space sessions where you can decide for yourself what topics you’d like to see covered. Hope to see you there!

Posted in MongoDB, NoSQL | 2 Comments »

« Older Entries