I’ll tell you all my secrets (but I lie about my past)
— Tom Waits – Tango till they’re sore

tl;dr

We’ve integrated the AWS Parameter Store with Spring Cloud so that it can be used as a secure configuration backend for services deployed to EC2, including ECS. This code has recently been merged in Spring Cloud AWS and is available in its 2.0 release.

Introduction

At the moment I’m working on a project where we’re developing a microservices-based system based on Spring Cloud for the Dutch Lotteries. The services are deployed on Amazon Web Services using Amazon’s current Docker support (ECS).

When we started late last year, we decided to use Consul, both as a service registry and as a key-value store for configuration. Spring Cloud has excellent built-in integration with Consul, both for service discovery as well as for using it as a shared configuration backend.

However, we quickly found out that we needed an internal load balancer to allow ECS to perform health checks on the services, so we might as well use that for server-side load balancing. This eliminated the need for client-side routing and service discovery. Furthermore, we weren’t too happy with the options to easily restrict access to secrets stored as config in Consul and were looking for a configuration service provided by AWS (rather than e.g. Vault) so that we’d no longer need to operate our own Consul cluster or other middleware.

AWS Parameter Store

When we looked for alternative solutions we soon found the AWS Parameter Store: it’s an option provided by EC2 to store all sorts of configuration parameters, including secrets that are encrypted at rest. Using IAM roles you can restrict access to parameters, which can have nested paths that can be used to define ACL-like access constraints. It also integrates with ECS quite nicely, by allowing containers to retrieve credentials to access the store, and provides versioning of parameter values.

This screenshot provides an impression of the corresponding console:

However, when looking for integration with Spring Cloud I just found some open tickets, so I decided to try to develop some integration myself. This blog post describes the result of that effort.

Spring Cloud Config support

If you’re not familiar with Spring Cloud’s configuration support, then what you need to know is that in essence it allows you to contact some central configuration backend at startup to obtain your service’s configuration: typically that configuration can be either shared or service-specific, and can also be dependent on what Spring profiles have been activated. The result will become available as one or more PropertySources within Spring’s Environment, just like when you’re using regular properties or YAML files (which you can still use as well, they simply have a lower precedence).

Spring Cloud provides its own Config Server for this as one option, which is typically backed by one or more Git repos. You can also use Consul, as I mentioned already.

This approach has several benefits over using local config files: you can have configuration that’s shared across all your services (still overridable on a per-service basis), you can change configuration centrally and have it applied to all services without redeploys, changes can even be pushed and trigger refresh-scoped beans to pick up their changed configuration without a full restart, etc.

Integrating with AWS’s Parameter Store

Using AWS’s API, you can programmatically query the Parameter Store.
Initially I was looking at an older version of the API and only found ways to retrieve the value of a specific key: that’s not very useful, as we want to discover all relevant key/value-pairs at startup and make them available to the application in the form of an EnumerablePropertySource.
However, in newer versions AWS added support for hierarchical property names with up to 15 path segments and methods to retrieve all properties under a given path: that makes it ideally suitable for Spring Cloud integration.

Let’s say that you’re using the Parameter Store not just as a Spring Cloud configuration backend, but for other purposes as well: then it makes sense to store your Spring configuration under a dedicated path, like /config. If you’re using a single EC2 setup to host multiple environment you could also choose for something like /config-test and /config-prod instead.

Then you want configuration that’s shared across all services under a dedicated path, perhaps /config/application (“application” is the default that’s used for Consul config support as well). Configuration for your service should be under /config/<your-service-name>.

When you have profiles enabled, they can have dedicated configuration as well that can override the standard service configuration: since path segments in a parameter name can only contain letters, digits, dots, dashes and underscores I’m using an underscore as separator, so if you’d have a profile called ‘aws’ enabled the configuration parameters under /config/my-service_aws/ would be picked up for a service called “my-service” and override any parameters with the same names present under /config/my-service/.

Integrating with the AWS Parameter Store from Java

AWS provides a Java SDK to interact with the many services it provides. The Parameter Store is part of a service called Systems Manager, for which there is a dedicated jar that you can use.

To retrieve these parameters, we use a getParametersByPath method which supports retrieving multiple parameters under a given prefix including decryption of encrypted parameters. The code for that can be found here.
It uses an AWSSimpleSystemsManagement client to connect to the Parameter Store. By default, this client uses a credentials chain to automatically obtain the necessary credentials to access the store. Read more about where it looks for those credentials here.

Initially I wrote an implementation just for the project I’m working on, but later I spent some time to turn that code into a proper pull request for Spring Cloud AWS. This code has been included in the 2.0 release of Spring Cloud AWS, so if you want to use this in your own project you can simply include a dependency on the appropriate starter.

When you do, the result of then checking the /actuator/env endpoint result of your service will look something like this:

Note that your application itself does not need to run on AWS to test this: as long as you’ve configured your AWS credentials the application will simply connect to AWS to retrieve the parameters even when running on your local machine! You can find a small example that you can run yourself as well on my Github: make sure to edit the application name in bootstrap.properties to match the name you’ve used in your parameter store path names!

Conclusion

Integrating the Parameter Store into Spring Cloud so that it can be used as a regular property source turned out to be fairly easy, but provides great benefits: the application itself remains unaware of where its configuration is coming from, secrets can be encrypted and both read and write access can be restrained, and configuration can easily be shared among services where desired.

If your application is running on AWS, make sure to check it out!

Tags: AWS, Cloud, EC2 Posted in: Custom Development, DevOps, Java, Spring