I’ll tell you all my secrets (but I lie about my past)
— Tom Waits – Tango till they’re sore
tl;dr
We’ve integrated the AWS Parameter Store with Spring Cloud so that it can be used as a secure configuration backend for services deployed to EC2, including ECS. This code has recently been merged in Spring Cloud AWS and is available in its 2.0 release.
Introduction
At the moment I’m working on a project where we’re developing a microservices-based system based on Spring Cloud for the Dutch Lotteries. The services are deployed on Amazon Web Services using Amazon’s current Docker support (ECS).
When we started late last year, we decided to use Consul, both as a service registry and as a key-value store for configuration. Spring Cloud has excellent built-in integration with Consul, both for service discovery as well as for using it as a shared configuration backend.
However, we quickly found out that we needed an internal load balancer to allow ECS to perform health checks on the services, so we might as well use that for server-side load balancing. This eliminated the need for client-side routing and service discovery. Furthermore, we weren’t too happy with the options to easily restrict access to secrets stored as config in Consul and were looking for a configuration service provided by AWS (rather than e.g. Vault) so that we’d no longer need to operate our own Consul cluster or other middleware.
AWS Parameter Store
When we looked for alternative solutions we soon found the AWS Parameter Store: it’s an option provided by EC2 to store all sorts of configuration parameters, including secrets that are encrypted at rest. Using IAM roles you can restrict access to parameters, which can have nested paths that can be used to define ACL-like access constraints. It also integrates with ECS quite nicely, by allowing containers to retrieve credentials to access the store, and provides versioning of parameter values.
This screenshot provides an impression of the corresponding console:
However, when looking for integration with Spring Cloud I just found some open tickets, so I decided to try to develop some integration myself. This blog post describes the result of that effort.
Spring Cloud Config support
If you’re not familiar with Spring Cloud’s configuration support, then what you need to know is that in essence it allows you to contact some central configuration backend at startup to obtain your service’s configuration: typically that configuration can be either shared or service-specific, and can also be dependent on what Spring profiles have been activated. The result will become available as one or more PropertySources within Spring’s Environment, just like when you’re using regular properties or YAML files (which you can still use as well, they simply have a lower precedence).
Spring Cloud provides its own Config Server for this as one option, which is typically backed by one or more Git repos. You can also use Consul, as I mentioned already.
This approach has several benefits over using local config files: you can have configuration that’s shared across all your services (still overridable on a per-service basis), you can change configuration centrally and have it applied to all services without redeploys, changes can even be pushed and trigger refresh-scoped beans to pick up their changed configuration without a full restart, etc.
Integrating with AWS’s Parameter Store
Using AWS’s API, you can programmatically query the Parameter Store.
Initially I was looking at an older version of the API and only found ways to retrieve the value of a specific key: that’s not very useful, as we want to discover all relevant key/value-pairs at startup and make them available to the application in the form of an EnumerablePropertySource.
However, in newer versions AWS added support for hierarchical property names with up to 15 path segments and methods to retrieve all properties under a given path: that makes it ideally suitable for Spring Cloud integration.
Let’s say that you’re using the Parameter Store not just as a Spring Cloud configuration backend, but for other purposes as well: then it makes sense to store your Spring configuration under a dedicated path, like /config. If you’re using a single EC2 setup to host multiple environment you could also choose for something like /config-test and /config-prod instead.
Then you want configuration that’s shared across all services under a dedicated path, perhaps /config/application (“application” is the default that’s used for Consul config support as well). Configuration for your service should be under /config/<your-service-name>.
When you have profiles enabled, they can have dedicated configuration as well that can override the standard service configuration: since path segments in a parameter name can only contain letters, digits, dots, dashes and underscores I’m using an underscore as separator, so if you’d have a profile called ‘aws’ enabled the configuration parameters under /config/my-service_aws/ would be picked up for a service called “my-service” and override any parameters with the same names present under /config/my-service/.
Integrating with the AWS Parameter Store from Java
AWS provides a Java SDK to interact with the many services it provides. The Parameter Store is part of a service called Systems Manager, for which there is a dedicated jar that you can use.
To retrieve these parameters, we use a getParametersByPath method which supports retrieving multiple parameters under a given prefix including decryption of encrypted parameters. The code for that can be found here.
It uses an AWSSimpleSystemsManagement client to connect to the Parameter Store. By default, this client uses a credentials chain to automatically obtain the necessary credentials to access the store. Read more about where it looks for those credentials here.
Initially I wrote an implementation just for the project I’m working on, but later I spent some time to turn that code into a proper pull request for Spring Cloud AWS. This code has been included in the 2.0 release of Spring Cloud AWS, so if you want to use this in your own project you can simply include a dependency on the appropriate starter.
When you do, the result of then checking the /actuator/env endpoint result of your service will look something like this:
Note that your application itself does not need to run on AWS to test this: as long as you’ve configured your AWS credentials the application will simply connect to AWS to retrieve the parameters even when running on your local machine! You can find a small example that you can run yourself as well on my Github: make sure to edit the application name in bootstrap.properties to match the name you’ve used in your parameter store path names!
Conclusion
Integrating the Parameter Store into Spring Cloud so that it can be used as a regular property source turned out to be fairly easy, but provides great benefits: the application itself remains unaware of where its configuration is coming from, secrets can be encrypted and both read and write access can be restrained, and configuration can easily be shared among services where desired.
If your application is running on AWS, make sure to check it out!
More information from Trifork:
Thank you for your contribution in adding AWS Parameter Store support to spring cloud. I currently use spring cloud config server and I’m lost on where to add the configuration to use AWS parameter store. Is sprong cloud aws starter to be added on the spring cloud config server or on the client? What client configuration is required?
Any detailed instruction will help! Thanks in advance!
The idea is that you’re not using Spring Cloud Config Server at all in this case: it’s a replacement that uses the Parameter Store as the server config store. Just adding the dedicated starter provides you with the client. Check out the sample here for more details: https://github.com/jkuipers/param-store-config-demo
I was hoping I could use spring config server and AWS Parameter store together. I would use Parameter store only for secrets. Spring config would then send the config to the client which is transparent to the client.
There’s nothing that’s stopping you from using them both together: in fact, that was our original use case as well (Consul + Param Store).
Would you happen to have an example how you configured Spring Config Server to use Parameter Store as an configuration “backend” and serve application specific configuration for application connected to the Config Server ?
I was hoping to use Parameter Store and Git together in Spring Config Server.
The idea is not that you’re using the Parameter Store as a backend for Spring Cloud Config Server: you use it instead of (or, if want to, next to) the Config Server. That means that your Spring Cloud services will connect directly to the Parameter Store at startup to retrieve their config, rather than to a Config Server. Spring Cloud does allow you to use multiple configuration backends, so you can combine the Config Server with the Parameter Store support, but they don’t integrate via each other in that case: they’re simply two different configuration stores that end up defining a list of PropertySources in the Environment.
Thank you for article. You wrote that with Config server it’s possible to pick up changed configuration without a full restart. Is it possible with AWS param store?
That’s currently not integrated. AWS does support notifications when config parameters change through CloudWatch, but you’d have to configure them and then implement some SNS listeners to act on those notifications by refreshing the context, or even build Spring Cloud Bus support for SNS: that didn’t seem like something this simple library could/should do.
With e.g. Consul it’s easier, as that supports HTTP long polling as an efficient way for every service to check for changes. The Parameter Store doesn’t have something like that, and continuously fetching all parameters to check for changes seemed too inefficient to me to build.
Hi,
Great work on the integration!
Was wondering how you managed to get the environment specific parameters to work? I assume the env gets passed from the IAM role maybe? I’ve got /config-int/application/my-param set up and the application running in the int environment doesn’t seem to pick it up (but does pick up non env specific ones i.e. /config/application/my-other-property)
Cheers
It’s simpler than that: in your bootstrap.properties (or .yml) you’d configure something like this:
aws.paramstore.prefix=/config-${ENVIRONMENT:}
Where ENVIRONMENT would be an environment variable or system property.
Ah i see, very simple!
Thanks!
Christian
[…] >>Integrating The AWS Parameter Store With Spring Cloud [blog.trifork.com] […]
Is there any way to override the Parameter Store properties from the command line? My attempts to override via –PROPERTY=VALUE are ignored for any property held in Parameter Store, but are set for any property not set in Parameter Store.
Yes, you can enable those overrides to work, but it requires some additional parameters in your remote configuration. This is documented here: https://cloud.spring.io/spring-cloud-static/spring-cloud.html#overriding-bootstrap-properties (not specific to the AWS support in any way).
The documentation doesn’t seem to be entirely correct: adding only the property spring.cloud.config.overrideSystemProperties=false to the Parameter Store was actually sufficient for me to allow command line args to override remote config.
Awesome, it worked, thanks. I tried initially with just spring.cloud.config.allowOverride=true, however as with you, spring.cloud.config.overrideSystemProperties=false was all I needed.
[…] a comprehensive article Integrating the AWS Parameter Store with Spring Cloud .Hence was trying to […]
works great with version 2.0.0.RELEASE, but not working any more with later versions?
org.springframework.boot
spring-boot-starter-parent
2.0.0.RELEASE
Versions of what: Spring Cloud AWS or Spring Boot? What exactly doesn’t work?
In your pom.xml, if I change the version in the parent section from 2.0.0.RELEASE to 2.1.0.RELEASE, it fails to work.
This works:
org.springframework.boot
spring-boot-starter-parent
2.0.0.RELEASE
This doesn’t:
org.springframework.boot
spring-boot-starter-parent
2.1.0.RELEASE
Hi Thanks man for this great work.
You speak of different profiles, I’ve been looking for a while how this works.
I know how to pass a param during local tests to make Spring Boot select the proper properties. But what I still haven’t figured out is how you do that in AWS. More precisely, how can I make sure that if my application is deployed in the TEST environment, it gets /TEST/servicename/* properties and when it is in ACCEPTANCE, it reads /ACCP/servicename/* from the configstore?
You have any link how to do that? Would be cool! Thanks.
In your bootstrap.properties (or .yml) you’d configure something like this:
aws.paramstore.prefix=/${ENVIRONMENT}
Where ENVIRONMENT would be an environment variable or system property that you’d pass to the process. We’re using Docker with ECS, so in our case this is part of the ECS Task Definition.
Nice article! I was able to implement this demo and it is working.
Quick question on aws paramter store. I feel that manually creating the parameters on AWS Console is not best practice because we need to maintain them in some repository for future edits moreover editing key/values on AWS console is not easy if we have numerous properties. Do you know the best approach to create properties in AWS Paramter Store. Do we have something like CloudFormation template where we can define key/values which can be automatically loaded into AWS Parameter Store (that way we can maintain CloudFormation template in git). Could you please let me know?
@Joris Kuipers
My AWS params are organized like below:
/config/application/s3password
/config/application/s3username
/config/application_prod/s3password
/config/application_prod/s3username
/config/demo-vgi_prod/cognitousername
When i run the the demo application () it is printing duplicate properties as shown below. Ideally i would expect that s3password & s3username should be overridden with application_prod. Please advice.
cognitousername = iamprodcognito
s3password = s3prodpassword
s3username = s3produser
s3password = s3pwd
s3username = s3user
It does override: the app simply prints out all property sources, but Spring will only resolve the highest priority value.
I was assuming i need to use values retrieved from property sources object. Are you saying that i need to create Spring @ConfigurationProperties class to read the properties? Any clue? Thanks!
It’s just another Spring PropertySource, so you can use the configured parameters any way you want: using @Value(“${param.name}”) String param, using Environment#getProperty(String name) or using an @ConfigurationProperties-annotated class.
Yes, you can do that. We do the same through Terraform, with most property values specified in Terraform as well and secrets coming from Gitlab CI variables.
I agree that that’s a best practice.