Kubernetes persistent storage for databases: local volumes and Ceph

When a container is gone, its own files are also gone. For a web app that's OK, but a database needs its data somewhere that stays when the pod goes away. In Kubernetes that's a PersistentVolume. With a volume on the disk of one server, the data is still there when the pod gets replaced. But then the database can only run on that one server. With a network disk that other servers can reach, the database can also start on another server when its server dies.

I tried three setups with the orders database of a ticket shop on a small cluster. I made the shop up, it's called Seatlark and it sells concert tickets online. First the database had no volume at all. Then it got a folder on the disk of one server, and after that a disk from Ceph. Ceph stores copies of the data on several servers. With the folder and with Ceph, I also switched off the server that the database ran on.

Without a volume, I only had to delete the pod and the orders were gone. With the folder, the orders were still there after a new pod. But while that server was off, the database couldn't run anywhere else, because Kubernetes could only use that folder on that server. With Ceph, the database came back on another server with all its orders. That took 55 seconds after I told Kubernetes that the old server was off.

Both times I had to tell Kubernetes myself that the server was off, before it replaced the database pod. Kubernetes can't tell a dead server from one that's only cut off from the network.

A database without a volume

In Kubernetes a container runs inside a pod, and a Deployment is the normal way to run an app. It makes a new pod when the old one goes away. At the start, the database was a plain Deployment with no volume, so Postgres wrote its files into the container:

      containers:
        - name: orders-db
          image: postgres:17-alpine
          envFrom:
            - secretRef:
                name: seatlark-env
          env:
            - name: PGDATA
              value: /pgdata

I bought a ticket, and it became order 1042. Then I deleted the pod. The Deployment made a new pod right away, from the same image. And the image has no orders in it. The seat was free again, and the next ticket got the number 1042 again. So the order was gone.

The same happens whenever a pod gets replaced, for example with a new version or when a server gets drained and all its pods have to leave. A crash also loses them. Kubernetes usually restarts a crashed container inside the same pod, but the new container starts with fresh files. Only the files in a volume survive that.

PersistentVolume, PersistentVolumeClaim and StorageClass

For data that has to stay around, the pod needs storage that lives longer than the pod. Kubernetes splits that into three objects:

The claim gets bound to a volume that fits, and the pod mounts the claim. The pod only names the claim, and the claim says what it needs, like the size and the class. Usually Kubernetes picks a volume that fits, but a claim can also name a particular one.

I started with the simplest kind, a local volume. That's a folder on the disk of one server, here worker-1.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-disk
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: orders-data-worker-1
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-disk
  local:
    path: /mnt/orders-data
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - worker-1

kubernetes.io/no-provisioner means this class doesn't create volumes, so I made the volume by hand. WaitForFirstConsumer waits with the binding until a pod needs the claim. Then the scheduler, which picks a server for each pod, can take the volume's server into account. The nodeAffinity says that this volume only exists on worker-1. Retain keeps the data when the claim gets deleted.

A StatefulSet with a volume claim template

The database needs a pod that comes back with the same data, and for that Kubernetes has the StatefulSet. Its pods get fixed names, here orders-db-0. The volumeClaimTemplates part gives each of those pods its own claim:

  volumeClaimTemplates:
    - metadata:
        name: orders-data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: local-disk
        resources:
          requests:
            storage: 5Gi

The claim is called orders-data-orders-db-0. When orders-db-0 gets replaced, the new pod gets the same claim, and with it the same data. A Deployment can also use a claim. But all its pods would share that one claim, and that doesn't work for a database with more than one copy.

ReadWriteOnce means that only one server can mount the volume at a time. Most disks work that way. ReadWriteMany lets many servers mount it at once. For a shared folder that needs a shared file system like NFS or CephFS.

After kubectl apply, the claim was bound:

$ kubectl get pvc
NAME                      STATUS   VOLUME                 CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
orders-data-orders-db-0   Bound    orders-data-worker-1   5Gi        RWO            local-disk     <unset>                 5s

I bought a ticket and deleted the pod. The new orders-db-0 started on worker-1 with the same folder, and the order was still there.

When the server with a local volume dies

Now everything depended on worker-1, so I switched it off with sudo poweroff -f. That's about the same as pulling the plug. Kubernetes marked the server as not ready 53 seconds later. The database pod still showed as running. That status comes from the kubelet, the part of Kubernetes that runs on each server. And on worker-1 nothing answered anymore:

$ kubectl get pod orders-db-0 -o wide
NAME          READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
orders-db-0   1/1     Running   0          75s   10.0.3.148   worker-1   <none>           <none>

Kubernetes doesn't move a StatefulSet pod off a server it can't reach. The server could be off, or it could be cut off from the network with the database still running. If Kubernetes started a second orders-db-0 somewhere else, two databases could write to the same data. With the default settings, Kubernetes tries to delete the pod about five minutes after the server stops answering. But the pod then stays in Terminating, because nothing on worker-1 confirms it. It stays there until the server comes back, or until someone deletes the node or forces the delete.

When you know that the server is off, you tell Kubernetes with a taint. A taint is a mark on a server, and this one says that the server is out of service:

kubectl taint node worker-1 node.kubernetes.io/out-of-service=nodeshutdown:NoExecute

Kubernetes calls this non-graceful node shutdown. With the taint, Kubernetes deletes the pods on that server and releases their volumes, so the pods can start somewhere else. Only set it when you're sure that the server is off.

Kubernetes deleted the old pod and made a new one, but the new one stayed Pending:

$ kubectl get pod orders-db-0
NAME          READY   STATUS    RESTARTS   AGE
orders-db-0   0/1     Pending   0          9s
$ kubectl describe pod orders-db-0 | tail -1
  Warning  FailedScheduling  15s (x2 over 15s)  default-scheduler  0/4 nodes are available: 2 node(s) didn't match PersistentVolume's node affinity, 2 node(s) had untolerated taint(s). preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.

The volume only exists on worker-1, so no other server could take the pod. A local volume survives a new pod. But while its server is down, the database can't run anywhere else. And if that server's disk is gone, the data is gone with it. On AWS the server's disk is an EBS volume, and you could attach it to another server by hand. A local volume doesn't do that for you. When worker-1 was back, I removed the taint with the same command and a minus at the end. The database started on worker-1 again, with its orders.

A network disk from Ceph with Rook

For a database that has to keep running when a server dies, the disk has to be reachable from other servers. On AWS that's an EBS volume with the EBS CSI driver, the piece that connects Kubernetes to EBS. Kubernetes can then attach the volume to another server in the same availability zone. On servers you run yourself, a common choice is Ceph, and Rook runs Ceph inside Kubernetes.

Ceph spreads data over the disks of several servers and keeps copies of each piece. It has a few parts:

Each worker got an empty 20 GB disk for Ceph. I installed Rook with the manifests from its quick start. Those are the YAML files that describe what to create:

git clone --depth 1 --branch v1.21.0 https://github.com/rook/rook.git
cd rook/deploy/examples
kubectl create -f crds.yaml -f common.yaml -f csi-operator.yaml -f operator.yaml
kubectl create -f cluster.yaml

cluster.yaml asks for three monitors and uses all empty disks on all servers. A few minutes later Ceph was ready, with one OSD per worker:

$ kubectl -n rook-ceph get pods -l app=rook-ceph-osd -o wide
NAME                               READY   STATUS    RESTARTS   AGE   IP           NODE
rook-ceph-osd-0-7659786b76-2wrvd   2/2     Running   0          31m   10.0.2.141   worker-3
rook-ceph-osd-1-64c4488d94-ccf8q   2/2     Running   0          27m   10.0.3.192   worker-1
rook-ceph-osd-2-668764845f-6t7ps   2/2     Running   0          31m   10.0.0.112   worker-2

Then I applied the example file for the StorageClass. It also has a CephBlockPool, and Rook makes a pool from it. A pool is a group of data in Ceph with its own rules, like how many copies to keep:

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 3
    requireSafeReplicaSize: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  clusterID: rook-ceph
  pool: replicapool
  imageFormat: "2"
  imageFeatures: layering
  ...
allowVolumeExpansion: true
reclaimPolicy: Delete

size: 3 with failureDomain: host keeps three copies of each piece of data, each on a different server. The provisioner is Ceph's CSI driver, so this class makes volumes by itself. A claim for rook-ceph-block gets a new Ceph disk, called an RBD image, and nobody has to make a PersistentVolume by hand.

I ran into three problems on the way.

Ceph 20 reported HEALTH_WARN right away. Its monitors still allowed insecure key types, and the CSI driver's keys used one. That's about how Ceph's clients log in, and the data was fine. I muted the three warnings with ceph health mute.

The Helm chart didn't start the CSI driver. I first tried Rook's Helm chart instead of the manifests. The chart installed the CSI operator, but the service accounts its drivers expected were missing, so no claim got a disk. The quick start manifests worked.

Rook wouldn't use disks it had used before. When I installed Ceph a second time on the same disks, Rook started no OSDs. It logged that each disk belonged to a different Ceph cluster. Deleting the old cluster hadn't cleared the disks, and neither had wipefs or blkdiscard on EBS. I wrote zeros over the start of each disk, and over the extra label copies that Ceph keeps at 1 GiB and 10 GiB. After that they counted as empty again.

Moving the database to Ceph

Now the database had to move onto a Ceph disk. You can't change the claim template of a StatefulSet in place, and a claim that's bound can't switch to another class. So that took a few steps. First I saved the orders with a dump:

kubectl exec orders-db-0 -- pg_dump --clean --if-exists -U seatlark seatlark_orders > orders.sql

Then I deleted the StatefulSet and created it again, with a new claim name and the Ceph class:

  volumeClaimTemplates:
    - metadata:
        name: orders-ceph
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: rook-ceph-block

By default, deleting a StatefulSet doesn't delete its claims. So the old claim stayed on worker-1, and Kubernetes made a new one in Ceph:

$ kubectl get pvc
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      VOLUMEATTRIBUTESCLASS   AGE
orders-ceph-orders-db-0   Bound    pvc-514c0b42-1fee-4f81-b8cd-01a9dfe79ecc   5Gi        RWO            rook-ceph-block   <unset>                 13s
orders-data-orders-db-0   Bound    orders-data-worker-1                       5Gi        RWO            local-disk        <unset>                 47m

The dump went back in with psql, and the orders were there. In a shop with customers, orders could come in between the dump and the restore. So you'd stop the writes before the dump and keep them stopped until the new database runs. For less downtime, Postgres replication can copy the later changes, and then the writes only stop for the switch.

When the server with a Ceph disk dies

Then I did the same test as with the local volume. I switched off the server the database ran on, worker-1. Kubernetes marked it as not ready, and I set the same taint. This time Kubernetes started orders-db-0 on worker-3 and attached the Ceph disk there. The database ran again 55 seconds after the taint, with all the orders:

$ kubectl get pod orders-db-0 -o wide | cut -c1-90
NAME          READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE
orders-db-0   1/1     Running   0          55s   10.0.2.244   worker-3   <none>

Postgres started like after a power cut. Its log said database system was not properly shut down; automatic recovery in progress. Then it replayed its write-ahead log, the record of recent changes it keeps for cases like this. A moment later it accepted connections. Ceph kept serving the data with one of three servers missing:

$ kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status | head -12
  cluster:
    id:     28f3dcad-3ed1-45e8-a74f-8cb6dbbf39a2
    health: HEALTH_WARN
            1/3 mons down, quorum a,b
            1 osds down
            1 host (1 osds) down
            Degraded data redundancy: 37/111 objects degraded (33.333%), 33 pgs degraded, 33 pgs undersized
            (muted: AUTH_INSECURE_CLIENT_KEY_TYPE AUTH_INSECURE_KEYS_ALLOWED AUTH_INSECURE_KEYS_CREATABLE)

  services:
    mon: 3 daemons, quorum a,b (age 111s) [leader: a], out of quorum: c
    mgr: a(active, since 13m), standbys: b

One monitor out of three was down, and a third of the copies were missing. But every piece of data still had two copies, so nothing was lost. When the server came back, Ceph made the third copies again by itself.

So the database survived a dead server. It still wasn't automatic, because I had to confirm that the server was off. Some tools do that step for you. They make sure the server is off, for example through the AWS API, and then they set the taint. The other way is a database that keeps its own copy on another server. Database operators do that.

Database operators and CloudNativePG

A storage layer like Ceph keeps the files safe. A database operator keeps the database running. An operator is a program inside the cluster that runs one kind of software for you. CloudNativePG, for example, runs a Postgres cluster with one primary and standby copies in other pods. Each of them has its own volume, so a failover doesn't need the dead server's disk. When the primary fails, CloudNativePG promotes a standby by itself. The app reconnects through a Service that then points to the new primary.

By default the standbys get the changes a moment later than the primary. So a standby can miss the last few writes when the primary dies. With synchronous replication, each commit waits until a standby has it. You can also set up backups to object storage like S3, and restore them into a new cluster.

With an operator like that, local disks are a common choice again, because the database itself keeps the copies. They only help with a dead server when the primary and the standbys run on different servers.

Database inside or outside the cluster

You can also keep the database out of Kubernetes. A lot of companies run their production database outside, for example on Amazon RDS. Then AWS does the backups and the upgrades, and with a Multi-AZ setup also the failover. Kubernetes only runs the apps. Running the database inside Kubernetes gives you the same tools for everything, and it can cost less. But then these are your job:

• Storage that survives a dead server.
• Backups that you have tested.
• A failover that works.

Questions you might have

What's the difference between a volume and a PersistentVolume? A volume is anything a pod mounts. Some volumes, like emptyDir, live and die with the pod. A PersistentVolume is storage that exists on its own, and a pod gets it through a claim.

What does the reclaim policy do? It decides what happens to a volume when its claim gets deleted. Delete deletes the disk, and that's the default for classes that make volumes. Retain keeps it, and you clean up by hand.

Why not use hostPath instead? A hostPath volume mounts a folder of whatever server the pod lands on. Without node affinity, a new pod on another server uses that server's folder, which can be empty or missing. A local PersistentVolume ties the data and the pod to the same server.

Is a StatefulSet required for a database? No, but it's the usual choice, because each pod keeps its name and its own claim. Operators like CloudNativePG manage their pods directly instead.

How I ran this

The cluster had four servers on AWS with Ubuntu 24.04. A t3.medium ran the control plane, and three t3.large servers were the workers, each with an extra empty 20 GB disk. I switched servers off with sudo poweroff -f and on again through the AWS API. I tested this on November 7, 2026.

These are the versions:

• Kubernetes 1.37.2, set up with kubeadm
• Cilium 1.20.3, next to kube-proxy
• Postgres 17, the image postgres:17-alpine
• Rook 1.21.0, from its example manifests
• Ceph 20.2.4

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com