Kubernetes Secrets Management with Sealed Secrets and Helm: GitOps way

Let's consider an application that is deployed in a Kubernetes cluster using Helm Chart and GitOps. According to the principles of GitOps, all data necessary for deploying the application should be stored in a git repository. Artifacts such as docker images, Helm charts, etc., can be stored in separate registries or repositories but must be uniquely identified, for example, using versioning. Thus, the git repository is the single source of truth for deploying the application. However, storing secrets in git in plain text, or as suggested by standard Kubernetes and Helm tools, simply in base64, is completely unsafe.

Of course, you can use special tools like HashiCorp Vault to store secrets when justified by the scale of the project. In this article, I want to focus on a simple solution that requires almost no additional external dependencies and minimal effort in operation. It is quite applicable for small systems and simple security policies.

To solve the problem, we will use the following tools:

• Universal Helm Chart from Nixys

• Flux CD as a GitOps tool

• Sealed Secrets for encrypting secrets

A similar construction can be implemented for any Helm Chart and other GitOps systems, such as ArgoCD.

Sealed Secrets is a solution from Bitnami, specifically designed for organizing the storage of secrets in a git repository and working in conjunction with GitOps systems. The secret is pre-encrypted and can be stored in git as an object of type SealedSecret. The Sealed Secrets controller decrypts the secrets and provides them to applications in the usual way. It is quite lightweight, requires no configuration, and practically does not consume cluster resources. Secrets are encrypted using the kubeseal console command. In the standard usage method, it creates a ready-made manifest for the SealedSecret object.

But there is one inconvenience. If you place the secret as a separate manifest, it becomes unavailable for management from the application's Helm Chart. For example, it is difficult to track its changes for restarting the application, as well as to ensure the presence of the secret before the pod starts. One solution might be to include the encrypted secret directly in the application's Helm Chart.

Implementation

For the convenience of using our solution, we will add a template and a helper for Sealed Secrets to the universal Helm Chart.

{{- range $sName, $val := .Values.sealedSecrets -}}
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: {{ include "helpers.app.fullname" (dict "name" $sName "context" $) }}
  namespace: {{ $.Release.Namespace | quote }}
  labels:
    {{- include "helpers.app.labels" $ | nindent 4 }}
    {{- with $val.labels }}{{- include "helpers.tplvalues.render" (dict "value" . "context" $) | nindent 4 }}{{ end }}
  annotations:
    {{- include "helpers.app.hooksAnnotations" $ | nindent 4 }}
    {{- with $val.annotations }}{{- include "helpers.tplvalues.render" (dict "value" . "context" $) | nindent 4 }}{{ end }}
spec:
  encryptedData:
    {{- include "helpers.sealedSecrets.render" (dict "value" $val.encryptedData) | indent 4 }}
  template:
    metadata:
      name: {{ include "helpers.app.fullname" (dict "name" $sName "context" $) }}
      namespace: {{ $.Release.Namespace | quote }}
      labels:
        {{- include "helpers.app.labels" $ | nindent 8 }}
        {{- with $val.labels }}{{- include "helpers.tplvalues.render" (dict "value" . "context" $) | nindent 8 }}{{ end }}
      annotations:
        {{- with $val.annotations }}{{- include "helpers.tplvalues.render" (dict "value" . "context" $) | nindent 8 }}{{ end }}
{{- end }}

{{- define "helpers.sealedSecrets.render" -}}
{{- $v := dict -}}
{{- if kindIs "string" .value -}}
{{- $v = fromYaml .value }}
{{- else -}}
{{- $v = .value }}
{{- end -}}
{{- range $key, $value := $v }}
{{ printf "%s: %s" $key $value }}
{{- end -}}
{{- end -}}

Our template will create secrets from the .Values.sealedSecrets section, adding labels and annotations defined for both the application and the resource itself. Encrypted data is placed in encryptedData as a standard dictionary.

It is worth noting that hooks are used here, with which Helm creates the SealedSecret object before creating and launching the application pods. This approach is used in the Helm Chart from Nixys for objects like ConfigMap and Secret. It ensures that the application receives the correct version of the configuration at startup, but the resource will not be automatically deleted when it is no longer in use. Similarly, you can define a template without hooks if such behavior is inconvenient.

If you need the application to restart automatically when the secret changes, you can add an annotation with the checksum of all secrets to its pods.

checksum/secrets: '{{ include "helpers.workload.checksum" (printf "%s" $.Values.sealedScrets) }}'

Now we can encrypt the secret, for example like this:

kubeseal --raw --scope=namespace-wide --namespace=yournamespace --from-file=yoursecret.txt

Thus, we get a string containing the content of the yoursecret.txt file in encrypted form. We specified the namespace-wide scope here so as not to be tied to the resource name that Helm can generate when rendering the chart.

We will add the resulting string to Values as follows:

sealedSecrets:
  yoursecretname:
    annotations:
      sealedsecrets.bitnami.com/namespace-wide: "true"
    encryptedData:
      FOO: "encrypted-secret-string"

It is worth noting that here we additionally add the annotation sealedsecrets.bitnami.com/namespace-wide: "true" so that the resource scope matches our encrypted data.

Verification

Let's describe our application through the values of the universal chart. For example, let's take the test microservice podinfo, which does not require any configuration, but will allow us to test the correct transfer of the secret.

First, let's encrypt our secret. For convenience, we will pass it directly from the command line:

echo -n 'very-secret-string' | kubeseal --raw --scope=namespace-wide --namespace=podinfo --from-file=/dev/stdin

To deploy the application via Flux CD, we will create a description of the HelmRelease object, containing the minimally necessary values parameters for deploying podinfo using the universal chart. We will define deployment, service, ingress, and SealedSecret. We will insert the previously encrypted string into sealedSecrets.app-secret.encryptedData.

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: podinfo
  namespace: podinfo
spec:
  interval: 10m
  chart:
    spec:
      chart: universal-chart
      version: '>=2.8.0'
      sourceRef:
        kind: HelmRepository
        name: your-helm-repository
        namespace: your-repository-namespace
      interval: 10m
  values:
    deployments:
      app:
        containers:
        - name: podinfo
          image: stefanprodan/podinfo
          imagePullPolicy: IfNotPresent
          ports:
          - name: http
            containerPort: 9898
          envSecrets:
          - app-secret
    services:
      app:
        type: ClusterIP
        ports:
        - name: http
          protocol: TCP
          port: 9898
    ingresses:
      app:
        hosts:
        - hostname: podinfo.example.com
          paths:
          - serviceName: app
            servicePort: 9898
            path: "/"
    sealedSecrets:
      app-secret:
        annotations:
          sealedsecrets.bitnami.com/namespace-wide: "true"
        encryptedData:
          SECRET_VARIABLE: 

After deployment, let's check if the secret was correctly transferred to the application. For podinfo, it is enough to execute the command:

curl -X 'GET' 'https://podinfo.example.com/env'

In response, we should receive an array of variables containing our secret:

[
  ...
  "SECRET_VARIABLE=very-secret-string",
  ...
]

Similarly, you can add a template for SealedSecret in any other "library" Chart, for example, generated by helm create. The only differences will be the helpers used inside and the structure of the values file.