Kubernetes ConfigMaps and Secrets: how settings and passwords get into your pods

With Docker Compose your settings and passwords usually sit in a .env file. In Kubernetes there are two objects for that. A ConfigMap is for normal settings, and a Secret is for passwords and keys. Your containers get both either as environment variables or as files. When you change a setting, an environment variable only gets the new value when the pod starts again. A file gets it while the app runs, after a minute or so. And a Secret isn't encrypted by itself, its values are only base64.

I tried both ways with the settings and passwords of a ticket shop. I made the shop up, it's called Seatlark and it sells concert tickets online. Then I turned on encryption for Secrets in etcd, that's the database where Kubernetes stores everything. And I moved the shop's payment key into HashiCorp Vault, which is a separate store for passwords.

With the environment variable, the running app never saw the change. The pods had the old value until I restarted them. With the file, the change came without a restart. It took about 45 seconds on the first pod and 70 seconds on the second, and in between the two pods showed different messages. So a file changes on its own, but not on all pods at the same time.

One command turns the base64 of a Secret back into the password. etcd stored the Secrets in plain text until I turned on encryption at rest. That protects them in a copy of the etcd data, as long as the copy doesn't have the key. Inside the cluster, the permissions in Kubernetes protect a Secret. With Vault, the payment key came from Vault and not from a Kubernetes Secret. Vault adds a helper container to the pod, and it wrote the key into a file before the app started.

ConfigMaps as environment variables

The first way puts each setting into an environment variable of the container, and your app reads it like any other environment variable. The shop's web app can show a message from the organiser at the top of the page. That's a setting, so it goes into a ConfigMap. A ConfigMap is a list of names and values:

apiVersion: v1
kind: ConfigMap
metadata:
  name: storefront-config
data:
  SALE_MESSAGE: Doors open at 7 pm. Bags get checked at the entrance.

The Deployment describes the pods of the web app. In it, envFrom turns each value of the ConfigMap into an environment variable:

      containers:
        - name: storefront
          image: registry:5000/seatlark/storefront:1.7.0
          envFrom:
            - configMapRef:
                name: storefront-config

If you only need some of the values, or want them under other names, env with valueFrom.configMapKeyRef picks single keys.

After kubectl apply, the message was on the page. Then I changed it to 6:30 pm and applied the ConfigMap again. The page kept showing 7 pm, and so did the environment of the running container:

$ kubectl exec deploy/storefront -- printenv SALE_MESSAGE
Doors open at 7 pm. Bags get checked at the entrance.

That's not a bug. The kubelet is the part of Kubernetes that runs on each server and starts the containers. It sets the environment when it starts a container, and a running process never sees a change to it. The new value only came with new containers:

kubectl rollout restart deployment/storefront

A rollout restart replaces the pods with the Deployment's update strategy, by default a few at a time. With readiness checks and enough pods kept ready, the shop stays up while that happens. You can automate this. Helm charts often put a hash of the ConfigMap into an annotation of the pod template. When helm upgrade applies a changed ConfigMap, the hash also changes, and that starts a rollout. Editing the ConfigMap directly doesn't change the hash. Tools like Stakater Reloader watch ConfigMaps and restart the Deployments that use them.

ConfigMaps as files

If you want a change to arrive without a restart, the other way is to mount the ConfigMap as a folder. Each value becomes a file named after its key:

      containers:
        - name: storefront
          image: registry:5000/seatlark/storefront:1.7.0
          volumeMounts:
            - name: config
              mountPath: /etc/seatlark
      volumes:
        - name: config
          configMap:
            name: storefront-config

The app reads /etc/seatlark/SALE_MESSAGE on each request. When I changed the ConfigMap to 6 pm, I didn't restart anything. The kubelet on each server updated the file by itself, about 45 seconds later on one pod and 70 seconds later on the other. In between, requests to the shop got a different message depending on which pod answered:

$ for i in 1 2 3 4 5 6; do curl -s worker-1:30080/ | grep -o 'Doors open at [^.]*'; done
Doors open at 6 pm
Doors open at 6:30 pm
Doors open at 6:30 pm
Doors open at 6 pm
Doors open at 6:30 pm
Doors open at 6 pm

The delay comes from how the kubelet works. It checks the mounted ConfigMaps of its pods on a regular sync, by default every minute. It also caches what it reads from the API. A minute or two is normal on a healthy cluster with the default settings, but it isn't a deadline. And different pods change at different times.

Inside the pod the folder looks like this:

$ kubectl exec deploy/storefront -- ls -la /etc/seatlark
lrwxrwxrwx    1 root     root    32 Nov  7 15:37 ..data -> ..2026_11_07_15_37_59.4081817578
lrwxrwxrwx    1 root     root    19 Nov  7 15:36 SALE_MESSAGE -> ..data/SALE_MESSAGE

The kubelet writes the new files into a new folder and then switches the ..data link, so the app never reads a half-written file. Three things decide whether this works for you:

An app that reads its settings once at start needs a restart either way. If you want the opposite, settings that can't change by accident, you mark the ConfigMap with immutable: true. Then a change means a new ConfigMap, and you point the Deployment at it. That also starts a rollout.

Kubernetes Secrets

Passwords get their own object, the Secret. It looks a lot like a ConfigMap. The shop's passwords came from the .env file of Docker Compose:

POSTGRES_USER=seatlark
POSTGRES_PASSWORD=demo-only-not-a-secret
POSTGRES_DB=seatlark_orders
PAYMENT_KEY=demo-only-not-a-secret

kubectl turns a file like that into a Secret:

kubectl create secret generic seatlark-env --from-env-file=.env

A pod uses a Secret the same way as a ConfigMap. Either envFrom with a secretRef turns the values into environment variables, or you mount it as files. This is what Kubernetes stored:

$ kubectl get secret seatlark-env -o yaml
apiVersion: v1
data:
  PAYMENT_KEY: ZGVtby1vbmx5LW5vdC1hLXNlY3JldA==
  POSTGRES_DB: c2VhdGxhcmtfb3JkZXJz
  POSTGRES_PASSWORD: ZGVtby1vbmx5LW5vdC1hLXNlY3JldA==
  POSTGRES_USER: c2VhdGxhcms=
kind: Secret
...

The values look encrypted, but they're base64, a way to write bytes as text. One command turns them back:

$ kubectl get secret seatlark-env -o jsonpath='{.data.POSTGRES_PASSWORD}' | base64 -d
demo-only-not-a-secret

So the difference between a Secret and a ConfigMap isn't encryption. Kubernetes does treat Secrets a bit more carefully than ConfigMaps:

• They can be encrypted in etcd.
• The kubelet keeps mounted Secrets in memory instead of on disk.
• Permissions usually allow reading ConfigMaps more widely than Secrets.

What protects a Secret are those permissions. Each pod talks to the Kubernetes API as a service account, and the shop's pods use the default one of their namespace. That account can't fetch Secrets from the API:

$ kubectl auth can-i get secrets --as system:serviceaccount:seatlark:default
no

The containers still get the values, because the kubelet passes them in. As the cluster admin, I can read every Secret. Two things about this are easy to get wrong.

Anyone who can create pods in a namespace can read its Secrets. They start a pod that mounts the Secret and print the file. So permission to create pods counts as permission to read Secrets.

And kubectl apply keeps a copy. I also created the same Secret with kubectl apply and stringData. Then the annotation kubectl.kubernetes.io/last-applied-configuration held all the values in plain text, right next to the base64 ones. kubectl create doesn't add that annotation.

Secrets don't belong in Git either, not even as base64. If you want everything in Git, tools like Sealed Secrets or SOPS store an encrypted version there. Or you keep the passwords in an outside store like Vault and bring them in from there.

Where Kubernetes stores Secrets: etcd

Permissions only count when someone goes through the Kubernetes API. The values also sit on a disk, so I looked there. Kubernetes keeps all its objects in etcd, a database that runs on the control plane. The control plane is the server that runs Kubernetes itself. With kubeadm, the tool I set the cluster up with, etcd runs as a pod there:

$ kubectl get pods -n kube-system -l component=etcd -o wide
NAME             READY   STATUS    RESTARTS   AGE   IP             NODE
etcd-control-1   1/1     Running   0          37m   172.31.1.184   control-1

etcd has its own command line tool, etcdctl, and the etcd image includes it. This small script runs it with the certificates of the control plane:

#!/bin/bash
kubectl -n kube-system exec etcd-control-1 -- etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key \
  get "$1"

The stored Secret is binary, and strings shows the readable parts of it:

$ ./etcd-get.sh /registry/secrets/seatlark/seatlark-env | strings
/registry/secrets/seatlark/seatlark-env
...
kubectl-create
Update
...
PAYMENT_KEY
demo-only-not-a-secret
POSTGRES_DB
seatlark_orders
POSTGRES_PASSWORD
demo-only-not-a-secret
POSTGRES_USER
seatlark
Opaque

The password is right there. Anyone with a copy of the etcd data, or of an etcd backup, has all the passwords of the cluster.

Kubernetes encryption at rest for Secrets

That's what encryption at rest is for. The API server, the part of Kubernetes that takes your kubectl commands, encrypts Secrets before it writes them to etcd. You tell it how in a file:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - secretbox:
          keys:
            - name: key1
              secret: <32 random bytes, base64>
      - identity: {}

The first provider in the list encrypts everything new. The ones after it are only used to read what's already there, and identity means plain text. I made the key with head -c 32 /dev/urandom | base64. These are the providers Kubernetes offers:

The API server reads the file when it starts. On a kubeadm cluster it runs as a static pod. That's a pod the kubelet starts from a file on the server's disk, so a change to that file changes the pod. My script did two things. It copied the file to /etc/kubernetes/enc/ on the control plane. And it added the flag --encryption-provider-config and that folder to /etc/kubernetes/manifests/kube-apiserver.yaml. The kubelet saw the changed file and started the API server again. It was back after about 45 seconds.

Secrets that already exist stay in plain text until they get written again. So I read all of them and wrote them back:

kubectl get secrets -A -o json | kubectl replace -f -

After that, the current versions of the Secrets were encrypted in etcd. The prefix says with which provider and key:

$ ./etcd-get.sh /registry/secrets/seatlark/seatlark-env | strings | head -3
/registry/secrets/seatlark/seatlark-env
k8s:enc:secretbox:v1:key1:
lIe*

kubectl get secret still showed the values, because the API server decrypts them. It helps to know what this protects and what it doesn't:

• It protects the Secrets in a copy of the etcd data, as long as the copy doesn't include the key.
• It doesn't protect against someone on the control plane, because the key is in a file there. With kms the main key stays outside, but a running API server can still decrypt Secrets.
• Older versions of a Secret stay in etcd until it compacts them, so a backup made right after the rewrite can still have the plain text. Compaction isn't a secure wipe of the disk either, and backups made before stay as they were.

On EKS and other managed clusters, the provider runs etcd for you, and EKS encrypts Secrets with AWS KMS by default.

One thing happened after the API server restart in both of my runs. A Service I created a minute or two later didn't resolve in DNS. CoreDNS, the DNS server inside the cluster, answered "no such host" for about 100 seconds. Restarting CoreDNS fixed it at once. So after you restart the API server, check that the rest of the cluster has caught up.

Kubernetes and Vault

The key for encryption at rest still sits on the control plane. A lot of companies keep their passwords outside Kubernetes altogether, in a secret store like HashiCorp Vault. The app gets them while it runs. Vault can log each access if you turn on auditing, and its access tokens expire. The payment key I stored here doesn't expire by itself.

I installed Vault in dev mode with its Helm chart. A chart is a package of Kubernetes objects that Helm installs at once:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault --version 0.34.1 \
  -n vault --create-namespace --set server.dev.enabled=true --wait

Dev mode keeps everything in memory. It starts unsealed, so it's unlocked from the start, and it uses a fixed root token. It's only for tests. The chart also installs the Vault Agent Injector. That's the part that later puts the key into the pod.

Then Vault needed the key, and a way to know which pod may read it:

vault kv put secret/payment key=pk_demo_4f9c2b7e81a3
vault auth enable kubernetes
vault write auth/kubernetes/config kubernetes_host=https://kubernetes.default.svc
vault policy write payment - <<P
path "secret/data/payment" {
  capabilities = ["read"]
}
P
vault write auth/kubernetes/role/payment-service \
  bound_service_account_names=payment-service \
  bound_service_account_namespaces=seatlark \
  policies=payment ttl=1h

I ran these inside the Vault pod with kubectl -n vault exec -i vault-0 -- sh < vault-setup.sh. The Kubernetes auth method lets a pod log in to Vault with the token of its service account. Vault asks the Kubernetes API whether the token is valid, and the role decides what that service account may read. Here only the service account payment-service in the namespace seatlark may read the payment key.

payment-service got its own service account, kubectl create serviceaccount payment-service, because a pod can only use one that exists. Then the Deployment got that service account and a few annotations:

  template:
    metadata:
      labels:
        app: payment-service
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: payment-service
        vault.hashicorp.com/agent-inject-secret-payment-key: secret/data/payment
        vault.hashicorp.com/agent-inject-template-payment-key: |
          {{- with secret "secret/data/payment" -}}{{ .Data.data.key }}{{- end -}}
    spec:
      serviceAccountName: payment-service
      containers:
        - name: payment-service
          image: registry:5000/seatlark/payment-service:1.5.0
          env:
            - name: PAYMENT_KEY_FILE
              value: /vault/secrets/payment-key

The injector is a webhook that changes pods as they get created. For these annotations it added two containers to the pod:

$ kubectl get pods -l app=payment-service
NAME                              READY   STATUS    RESTARTS   AGE
payment-service-5b4856cb4-b7g76   2/2     Running   0          88s
$ kubectl get pod -l app=payment-service -o jsonpath='{.items[0].spec.initContainers[*].name} | {.items[0].spec.containers[*].name}'
vault-agent-init | payment-service vault-agent

vault-agent-init is an init container, so it runs to the end before the other containers start. It logs in to Vault and fetches the key. Then it writes the key into /vault/secrets/payment-key, a volume that all containers of the pod share. After that the app starts and reads the file:

$ kubectl logs deploy/payment-service -c payment-service
payment-service listening on 9000, payment key from /vault/secrets/payment-key

vault-agent is the second container. It runs next to the app the whole time and keeps the file up to date, for example when the key changes in Vault. That kind of helper is called a sidecar. My app reads the key once at start, so it would only pick up a new key after a restart.

A pod with more than one container is normal in Kubernetes. The containers share the pod's network and can share volumes, so a helper next to the app can do a job the app doesn't know about. Logging agents and proxies work the same way.

A test purchase went through with the key from Vault. Then the payment key could come out of the Kubernetes Secret:

kubectl patch secret seatlark-env --type json -p '[{"op":"remove","path":"/data/PAYMENT_KEY"}]'

Vault still ran inside this cluster, and the key still exists there, in Vault's storage and in the file inside the pod. What changed is that the key came from Vault, and the Kubernetes Secret no longer had it. Older copies, like earlier versions in etcd, don't go away with that patch.

Vault isn't the only way. The External Secrets Operator copies passwords from Vault or AWS Secrets Manager into normal Kubernetes Secrets, so your apps don't have to change. The Secrets Store CSI Driver mounts them as files without a Secret in between. Vault itself has changed its license to the Business Source License. OpenBao is a fork under an open source license that works the same way.

Questions you might have

ConfigMap or Secret? Passwords go into a Secret, and so do keys and tokens. Anything that would hurt if it leaked belongs there. Everything else goes into a ConfigMap. Both can hold at most 1 MiB.

Environment variables or files for passwords? Both are common. Files have two advantages. They can change while the app runs. And they don't show up in places that list the environment, like a crash report or a child process.

Can I see which pods use a Secret? Not directly. You have to look through the pod specs, for example with kubectl get pods -o json and a search for the name.

What about Docker Compose's .env file? Kubernetes doesn't read it. kubectl create secret generic --from-env-file and kubectl create configmap --from-env-file turn simple KEY=value lines into a Secret or a ConfigMap once. They keep quotation marks and don't fill in references like ${PASSWORD}, so check a file that was written for Compose before you import it.

How I ran this

The cluster had three servers on AWS with Ubuntu 24.04. A t3.medium ran the control plane, and two t3.large servers were the workers. The shop's web app, the storefront, ran in two copies, one on each worker. I tested this on November 7, 2026.

These are the versions:

• Kubernetes 1.37.2, set up with kubeadm
• etcd 3.7.0
• Cilium 1.20.3, next to kube-proxy
• Helm 4.3.1
• Vault 2.0.4, from the Helm chart 0.34.1

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com