Secrets in Java services on Spring: where to get and how to update

The article will consist of two parts. In the first, I will talk about the features of working with secrets in Java on Spring Boot — where to get them and how to apply them to your service using the example of how we do it in Platform V Sessions Data (a distributed in-memory cache for client sessions that reduces the load on external services and the database). I will also talk about the standard options for updating secrets "on the fly" (without stopping, restarting services, or even removing the load from them) and what is wrong with them.

In the second part, I will detail how we update secrets "on the fly". These tips will help you improve your work with secrets, and thus make services more secure. After all, if secrets fall into the hands of attackers, they will be able to attack your service — disable it, steal confidential data, etc. And any successful attack is fraught with loss of money, nerves, time, and reputation for companies.

Why modern services need secrets

Secrets are configuration parameters that, on the one hand, are necessary for the operation of the service, and on the other hand, they are of particular interest to attackers. These can be, for example, SSL certificates (public certificate, its private key, root trusted certificate), credentials (username, password), etc.

Therefore, the architecture of secure work with secrets in services needs to be carefully thought out.

To make it clearer, let's look at the secrets of our service — Platform V Sessions Data (a distributed in-memory cache for client sessions).

Here is what the architecture looks like:

• in‑memory cache is located in the memory of the master nodes of the cluster;

• consumers receive data from the master through our client library, which caches them in the near cache;

• the master sends session lists to the servant application, which saves them in the database;

• the back of our admin panel — manager — requests this data from the servant from the database.

All three of our applications are written in Java and use Spring Boot.

Secrets are primarily needed here for secure integrations between applications, as well as for connecting to the database. In both cases, Mutual TLS (mTLS) is used, which requires SSL certificates. That is, in our case, SSL certificates are needed:

• in all interactions between applications (HTTPS with mTLS is used there);

• for servant's requests to the PostgreSQL database via JDBC.

In addition, we need credentials (username and password) to connect to the database.

Where to safely get secrets from

There are quite a few secret providers. Among them are HashiCorp Vault, Azure Key Vault, AWS Secret Manager, and many others. Their task is to minimize the risk of compromising secrets when storing and transmitting them to services.

One of the most popular is HashiCorp Vault. Its creators have put a lot of effort into the security of storage: secrets are stored in the Vault database, encrypted with a master key, which is encrypted with a root key, and it is also encrypted.

Vault also provides secrets to services securely:

• firstly, HTTPS with TLS is used for transport;

• secondly, Vault necessarily authenticates each service that requests it, for which there are many options for all occasions and for any cloud;

• thirdly, Vault authorizes services based on flexible policies, that is, it clearly knows who can be given which secrets.

Secrets from Vault get into applications deployed in containers/virtual machines in different ways:

1. Standard: use Vault Agent sidecar in Kubernetes. It fetches secrets from Vault via HTTPS and stores them as files in emptyDir (shared with the main container of the pod).

2. More elegant way: use External Secrets Operator. It also fetches secrets from Vault via HTTPS, but stores them as Kubernetes entities kind: Secret, providing secrets to the entire cluster. Services only need to mount the secret files into their container.

3. Another way: integrate directly with Vault via HTTPS. In this case, the service fetches secrets from Vault itself.

With any method of integrating with Vault, secrets for the service usually come to you as files. This creates a clear boundary between the layer that fetches secrets and the layer that consumes them. And if necessary, the provider can be easily changed.

To fetch secrets, we use not the vanilla HashiCorp Vault, but our own service, which is fully compatible with Vault via API.

Before diving into the Java code, where there will be examples from our Platform V Sessions Data service, I'll explain which secrets we use directly from Java.

When Istio saves the day

For HTTPS interactions in our Kubernetes clusters, we use Istio. It handles:

• mTLS termination for incoming traffic in Ingress;

• mTLS initiation for outgoing traffic in Egress.

Applications within the Kubernetes namespace do not need to use SSL certificates for HTTPS interactions themselves, as Istio handles everything for them. However, our master storage is deployed on virtual machines, so there is no Istio magic there — the master handles SSL certificates itself from Java code. This applies to both client and server certificates.

As for the database, as already mentioned, SSL certificates are also required for connecting to it, but this time for JDBC (Istio Egress does not save us here). In addition, credentials are also needed to connect to the database. These secrets have to be applied directly from the Java code of our servant application.

Thus, in our Platform V Sessions Data service applications, only the master storage and servant consume secrets directly in Java code, while our client jar and manager are lucky — they do not need secrets, Istio saves them.

How to apply secrets when starting Java services on Spring Boot

Now we have approached Java code and Spring Boot. Let me remind you once again that our secrets are files:

• server SSL certificates for HTTPS;

• client SSL certificates for HTTPS;

• client SSL certificates for JDBC;

• DB creds.

First of all, we need to be able to apply these secrets in the Java code of services when they start. It may seem simple: read the files and apply their secret content. But the devil, as usual, is in the details.

A) Applying server SSL certificates.

When starting our master repository, you need to apply server SSL certificates to the embedded Spring Boot Tomcat. We specify the paths to the certificate stores in application.yaml in the standard properties key-store and trust-store:

However, you cannot feed the passwords to these certificate stores to Spring Boot, as it needs strings, and we only have files with these strings inside.

In Platform V Sessions Data, we use a trick with the Tomcat customizer factory. When starting Spring Boot, we read three passwords from files ourselves and programmatically set them to Tomcat.

In such cases, we always remember best practice: in Java code, work with secret values not through regular strings, but through char[] and immediately zero out the array after use. This way, secrets will not be exposed in the JVM heap dump, and therefore will not fall into the hands of service administrators who are not supposed to know its secrets.

But in this particular case, the Spring Boot API can only accept passwords as regular strings. Therefore, it should be recognized: here best practice cannot be followed.

B) Applying client SSL certificates.

When starting the master repository, we also need to apply client SSL certificates for outgoing HTTPS interactions.

To send HTTP requests, we use the Jersey client. And to apply client SSL certificates to it, we ourselves assemble the SSLContext in Java code. To do this, we read the files with the certificate stores and their passwords:

This time we can follow best practice: read passwords in char[], zeroing out the array after use, and that's great.

The SSLContext built in this way is passed to the created instance of the Jersey client:

That's all. Next, our master service uses this built client instance for outgoing HTTPS requests (for example, to servant).

B) Using SSL certificates for database connections.

Our servant application needs to use SSL certificates to connect to the PostgreSQL database at startup. In this case, the paths to the certificate files are embedded in the JDBC URL. These are three files:

• certificate,

• its private key,

• the root certificate that must sign the server certificate Postgre.

Certificates are used in pem format (just base64 encoded), so no passwords for certificate stores are needed here.

When creating the HikariDataSource bean, we simply set it to such a JDBC URL, where the paths to the SSL certificates are embedded.

And that's all. Next, the Postgre JDBC driver, when connecting to the database, uses the SSL certificates specified in the URL. We don't even have to read the files ourselves.

C) Using database credentials

To connect to PostgreSQL, the servant must also use database credential files at startup. Unlike SSL certificates, we read the username and password from the secret files ourselves and set them when creating the HikariDataSource bean.

Here we again recall best practice: secrets should be read in char[] with zeroing out the array after use. Unfortunately, this will not work here either: as in the case of Spring, the HikariCP library requires credentials as plain strings.

Why update secrets

So, the secrets are applied, the service starts, works safely, using mTLS in all integrations. What else is needed? You need to be able to update secrets, because they can change. Here are some reasons:

• periodic rotation of SSL certificates;

• compromise of credentials requiring login/password replacement (God forbid), etc.

Therefore, it is not enough to obtain secrets once and for all, you also need to be able to update them.

With any method of integration with HashiCorp Vault, the service periodically receives updates of files with secrets from it. It remains only to respond in time to these file changes and apply new secrets. And there are also many nuances and complexities here.

Why not restart to update secrets

It would seem that you can simply restart the service to apply new secrets obtained from HashiCorp Vault. It sounds tempting: it is simple and reliable. However, this idea had to be abandoned, and here's why.

1) Let's start with Kubernetes. The first thing that comes to mind is to trigger a rolling update to sequentially restart all application pods to apply secret updates.

There are two options here.

1. During the rolling update, the number of service pods will temporarily decrease. We do not agree to this, as we do not want to increase the load on the remaining pods, so as not to provoke errors and not to degrade the quality of the service.

2. During the rolling update, the number of pods will temporarily increase. We are also not ready to do this with every secret update. We try to be as rational as possible in the use of hardware in clusters. There are secrets that are used by many services (for example, a trust store for mTLS). If, when such secrets are changed, dozens and hundreds of services request additional pods, then at the moment the consumption of hardware in the cluster will sharply increase. For such a task as applying new secrets, this is too expensive.

2) Let's continue with our master repository on virtual machines.

There is no rolling update out of the box. Moreover, the master is a stateful application that stores data in RAM. Therefore, it is necessary to avoid restarting its nodes as much as possible, especially for such a purpose as simply applying new secrets.

We, having performed such an analysis, decided to apply secret updates "on the fly", without stopping, restarting services, or even removing the load from them. We call this hot reload. And it became a real challenge.

Why not @RefreshScope for updating secrets

Of course, first we looked for standard tools that could perform hot reload of secrets. The first thing we tried was using the @RefreshScope annotation from Spring Cloud. It looks very convenient - you hang this annotation on a bean that needs to be recreated when the Spring configuration changes:

And after changing the files with secrets (they are part of the Spring configuration):

• either call the RefreshEndpoint actuator via HTTP: http://:/actuator/refresh -request POST

• or directly in the application code publish RefreshEvent: applicationContext.publishEvent(new RefreshEvent(this, , ))

And that's it! The first next call to the bean will lead to its recreation according to the new configuration (that is, in our case, according to the new secrets).

However, in practice, this method was only suitable for recreating the HTTP client bean with new SSL certificates.

And to update Tomcat server certificates via @RefreshScope, there was simply no suitable bean in Spring Boot. The most obvious solution is to hang the annotation on the Tomcat factory bean (TomcatServletWebServerFactory). But then Spring Boot stops starting due to an error:

Unable to start ServletWebServerApplicationContext due to multiple ServletWebServerFactory beans: scopedTarget.tomcatServletWebServerFactory,tomcatServletWebServerFactory

For some reason, Spring Boot perceives the Tomcat factory in RefreshScope as an additional, second bean of this type, although the factory should be singleton.

But even if we assume that we managed to hang the annotation on the Tomcat factory, nothing would come of it, because the factory bean is no longer used after the application starts, which means that the Tomcat server recreated by the new factory would still not be created.

As for the bean HikariDataSource, the use of @RefreshScope leads to the termination of all current connections to the database: threads waiting for a response from the database receive a PSQLException. This happens because HikariDataSource implements the AutoClosable interface, and RefreshScope, when recreating the bean according to the new Spring configuration, destroys the previous instance using the close() method for AuthCloseable beans.

Unfortunately, we had to admit that we cannot use @RefreshScope for hot reload of secrets.

Why not SSL bundles for updating secrets

We started looking for other solutions for hot reload and tried SSL bundles, which appeared in Spring Boot 3.2.

One of their main features is the hot reload of SSL certificates when they are changed. You just need to specify a named bundle with SSL certificates in application.yaml and set the reloadOnUpdate flag for it.

This enables file tracking of certificates and hot reload of the bundle, which is indeed very convenient.

However, in practice, reload SSL bundles did not suit us for several reasons.

1. Does not work in k8s when secrets are mounted from kind: Secret. External Secrets Operator can change kind: Secret on the fly. At the same time, the mounted files with secrets are symlinks that remain unchanged, so there is no reaction to the change of secrets.

2. Does not track changes to keyStore and trustStore passwords. In application.yaml, passwords for bundles are set as strings, not file references, so it is impossible to update passwords.

3. Reload triggers on any file change events. In practice, it was necessary to track specific events (for example, only deletions).

In general, the option with SSL bundles also turned out to be not for us because they are still raw for use in production. I hope that in the next version of Spring Boot the above shortcomings will be fixed.

As a result, we had to implement our own universal tool for hot reload of any secrets in Java services on Spring Boot, which can be deployed both in Kubernetes and on virtual machines. I will tell you more about this in the next article.

Results and conclusions

1. Use an external secrets provider for your services. This will allow you to centrally, flexibly, and securely manage secrets, regardless of what your services are written in and how they are deployed. At the same time, ensure a clear boundary between the layer that brings secrets and the layer that consumes them. It is convenient to work with secrets in the form of files — this will allow you to easily change the secrets provider without changing the code of your service.

2. Do not use rolling update for secret rotation, especially if you have a high-load service. Even a temporary decrease or increase in the number of pods is too expensive for updating secrets.