Kubernetes – Add vault secrets into environment variables

We already have a full kubernetes and Jenkins-X stack for our application’s CI/CD. For a fully setup application, I’m already able to tweak the secrets settings and how the secrets are populated from vault to environment variable. However, for an app that has no secrets setup, it took me days to figure it out.

Initial Setup

The following are already being setup and copied from a perfectly working application which as newly created by me and apparently, my Jenkins-X version already sets up the secrets properly for me.

Vault

Vault is already setup as part of the Jenkins-X and kubernetes infrastructure. I already populated the secrets in vault using Vault UI.

charts/app-name/templates/secrets.yaml

apiVersion: v1
kind: Secret
metadata:
  name: app-name-secrets
data:
  JWT_SECRET: {{ .Values.secret.jwtSecret | b64enc }}

charts/app-name/values.yaml

I also set the values.yaml properly and is copied as it appears from a working application.

secret:
  jwtSecret: "vault:app-cluster1/app-name-secrets:jwtSecret"
env:
  FOO: bar
envFrom:
  - secretRef:
      name: app-name-secrets

What’s missing?

With this configuration, the application is still not able to react the JWT_SECRET from the environment variables. There must be something missing. I did some digging on the pod details using the kubectl CLI tool.

# List all namespaces
kubectl get ns
# List pods in a namespace, like a preview namespace
kubectl get pods -n some-namespace
# Get pod definition in yaml format
kubectl get pods pod-name -n namespace -o yaml

I compared the pod definition of an app with working secrets vs this app with broken secrets. I noticed that envFrom is missing in the definition. This is where I thought of looking deeper into the deployment.yaml since this is the place where magic happens.

I found out that envFrom is not defined in my old app with broken secrets. I just copied the line from the working app and then secrets are now working finally.

charts/app-name/templates/deployment.yaml

      containers:
      - name: {{ .Chart.Name }}
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        imagePullPolicy: {{ .Values.image.pullPolicy }}
        env:
{{- range $pkey, $pval := .Values.env }}
        - name: {{ $pkey }}
          value: {{ quote $pval }}
{{- end }}
        envFrom:
{{ toYaml .Values.envFrom | indent 10 }}

See that envFrom injection? Apparently, my Jenkins-X version already populated that line in deployment.yaml. The older app does not have it configured.

That’s it!