I’ll tell you all my secrets (but I lie about my past)
— Tom Waits – Tango till they’re sore
tl;dr
We’ve integrated the AWS Parameter Store with Spring Cloud so that it can be used as a secure configuration backend for services deployed to EC2, including ECS. This code has recently been merged in Spring Cloud AWS and is available in its 2.0 release.
Introduction
At the moment I’m working on a project where we’re developing a microservices-based system based on Spring Cloud for the Dutch Lotteries. The services are deployed on Amazon Web Services using Amazon’s current Docker support (ECS).
When we started late last year, we decided to use Consul, both as a service registry and as a key-value store for configuration. Spring Cloud has excellent built-in integration with Consul, both for service discovery as well as for using it as a shared configuration backend.
However, we quickly found out that we needed an internal load balancer to allow ECS to perform health checks on the services, so we might as well use that for server-side load balancing. This eliminated the need for client-side routing and service discovery. Furthermore, we weren’t too happy with the options to easily restrict access to secrets stored as config in Consul and were looking for a configuration service provided by AWS (rather than e.g. Vault) so that we’d no longer need to operate our own Consul cluster or other middleware.
AWS Parameter Store
When we looked for alternative solutions we soon found the AWS Parameter Store: it’s an option provided by EC2 to store all sorts of configuration parameters, including secrets that are encrypted at rest. Using IAM roles you can restrict access to parameters, which can have nested paths that can be used to define ACL-like access constraints. It also integrates with ECS quite nicely, by allowing containers to retrieve credentials to access the store, and provides versioning of parameter values.
This screenshot provides an impression of the corresponding console:
However, when looking for integration with Spring Cloud I just found some open tickets, so I decided to try to develop some integration myself. This blog post describes the result of that effort.
Spring Cloud Config support
If you’re not familiar with Spring Cloud’s configuration support, then what you need to know is that in essence it allows you to contact some central configuration backend at startup to obtain your service’s configuration: typically that configuration can be either shared or service-specific, and can also be dependent on what Spring profiles have been activated. The result will become available as one or more PropertySources within Spring’s Environment, just like when you’re using regular properties or YAML files (which you can still use as well, they simply have a lower precedence).
Spring Cloud provides its own Config Server for this as one option, which is typically backed by one or more Git repos. You can also use Consul, as I mentioned already.
This approach has several benefits over using local config files: you can have configuration that’s shared across all your services (still overridable on a per-service basis), you can change configuration centrally and have it applied to all services without redeploys, changes can even be pushed and trigger refresh-scoped beans to pick up their changed configuration without a full restart, etc.
Integrating with AWS’s Parameter Store
Using AWS’s API, you can programmatically query the Parameter Store.
Initially I was looking at an older version of the API and only found ways to retrieve the value of a specific key: that’s not very useful, as we want to discover all relevant key/value-pairs at startup and make them available to the application in the form of an EnumerablePropertySource.
However, in newer versions AWS added support for hierarchical property names with up to 15 path segments and methods to retrieve all properties under a given path: that makes it ideally suitable for Spring Cloud integration.
Let’s say that you’re using the Parameter Store not just as a Spring Cloud configuration backend, but for other purposes as well: then it makes sense to store your Spring configuration under a dedicated path, like /config. If you’re using a single EC2 setup to host multiple environment you could also choose for something like /config-test and /config-prod instead.
Then you want configuration that’s shared across all services under a dedicated path, perhaps /config/application (“application” is the default that’s used for Consul config support as well). Configuration for your service should be under /config/<your-service-name>.
When you have profiles enabled, they can have dedicated configuration as well that can override the standard service configuration: since path segments in a parameter name can only contain letters, digits, dots, dashes and underscores I’m using an underscore as separator, so if you’d have a profile called ‘aws’ enabled the configuration parameters under /config/my-service_aws/ would be picked up for a service called “my-service” and override any parameters with the same names present under /config/my-service/.
Integrating with the AWS Parameter Store from Java
AWS provides a Java SDK to interact with the many services it provides. The Parameter Store is part of a service called Systems Manager, for which there is a dedicated jar that you can use.
To retrieve these parameters, we use a getParametersByPath method which supports retrieving multiple parameters under a given prefix including decryption of encrypted parameters. The code for that can be found here.
It uses an AWSSimpleSystemsManagement client to connect to the Parameter Store. By default, this client uses a credentials chain to automatically obtain the necessary credentials to access the store. Read more about where it looks for those credentials here.
Initially I wrote an implementation just for the project I’m working on, but later I spent some time to turn that code into a proper pull request for Spring Cloud AWS. This code has been included in the 2.0 release of Spring Cloud AWS, so if you want to use this in your own project you can simply include a dependency on the appropriate starter.
When you do, the result of then checking the /actuator/env endpoint result of your service will look something like this:
Note that your application itself does not need to run on AWS to test this: as long as you’ve configured your AWS credentials the application will simply connect to AWS to retrieve the parameters even when running on your local machine! You can find a small example that you can run yourself as well on my Github: make sure to edit the application name in bootstrap.properties to match the name you’ve used in your parameter store path names!
Conclusion
Integrating the Parameter Store into Spring Cloud so that it can be used as a regular property source turned out to be fairly easy, but provides great benefits: the application itself remains unaware of where its configuration is coming from, secrets can be encrypted and both read and write access can be restrained, and configuration can easily be shared among services where desired.
If your application is running on AWS, make sure to check it out!
Thank you for your contribution in adding AWS Parameter Store support to spring cloud. I currently use spring cloud config server and I’m lost on where to add the configuration to use AWS parameter store. Is sprong cloud aws starter to be added on the spring cloud config server or on the client? What client configuration is required?
Any detailed instruction will help! Thanks in advance!
The idea is that you’re not using Spring Cloud Config Server at all in this case: it’s a replacement that uses the Parameter Store as the server config store. Just adding the dedicated starter provides you with the client. Check out the sample here for more details: https://github.com/jkuipers/param-store-config-demo
I was hoping I could use spring config server and AWS Parameter store together. I would use Parameter store only for secrets. Spring config would then send the config to the client which is transparent to the client.
There’s nothing that’s stopping you from using them both together: in fact, that was our original use case as well (Consul + Param Store).
Thank you for article. You wrote that with Config server it’s possible to pick up changed configuration without a full restart. Is it possible with AWS param store?
That’s currently not integrated. AWS does support notifications when config parameters change through CloudWatch, but you’d have to configure them and then implement some SNS listeners to act on those notifications by refreshing the context, or even build Spring Cloud Bus support for SNS: that didn’t seem like something this simple library could/should do.
With e.g. Consul it’s easier, as that supports HTTP long polling as an efficient way for every service to check for changes. The Parameter Store doesn’t have something like that, and continuously fetching all parameters to check for changes seemed too inefficient to me to build.
Hi,
Great work on the integration!
Was wondering how you managed to get the environment specific parameters to work? I assume the env gets passed from the IAM role maybe? I’ve got /config-int/application/my-param set up and the application running in the int environment doesn’t seem to pick it up (but does pick up non env specific ones i.e. /config/application/my-other-property)
Cheers
It’s simpler than that: in your bootstrap.properties (or .yml) you’d configure something like this:
aws.paramstore.prefix=/config-${ENVIRONMENT:}
Where ENVIRONMENT would be an environment variable or system property.
Ah i see, very simple!
Thanks!
Christian
[…] >>Integrating The AWS Parameter Store With Spring Cloud [blog.trifork.com] […]
Is there any way to override the Parameter Store properties from the command line? My attempts to override via –PROPERTY=VALUE are ignored for any property held in Parameter Store, but are set for any property not set in Parameter Store.