What is Kubernetes? How it runs your app on several servers

Kubernetes runs your containers on a bunch of servers. You tell it what should run, and it tries all the time to make that happen. When a container crashes, Kubernetes starts it again. When a whole server dies, it starts new copies on the other servers. But your database doesn't move along by itself, because its data is still on the dead server.

I tried this on a small cluster with three servers on AWS. On it ran a ticket shop. I made it up, it's called Seatlark and it sells concert tickets online. While a load test acted like 120 people using the shop, I shut down one of the servers. Then I deleted the database container, and later I also shut down the server that the database ran on.

The web app had copies on two servers. When one of them went down, more than half of the requests failed for a bit less than a minute, because Kubernetes first has to notice that the server is gone. After that it only sent traffic to the copies on the other server, and I didn't have to do anything for that. About five minutes later it started new copies to replace the missing ones. It waits that long in case the server comes back, and you can change that.

The database is different. When I deleted its container, Kubernetes started a new one with the same disk, and the order was still there. But that disk was a folder on one server. So when I shut down that server, the database was down, and nobody could buy a ticket until the server was back.

So Kubernetes replaces the parts of your app that can run anywhere, as long as another server has room for them. Where your data lives and what happens to it when a server dies, you still have to figure out yourself.

Kubernetes architecture: control plane and workers

A cluster is a group of servers that run Kubernetes together. Mine had three:

$ kubectl get nodes
NAME        STATUS   ROLES           AGE   VERSION
control-1   Ready    control-plane   51m   v1.37.2
worker-1    Ready    <none>          51m   v1.37.2
worker-2    Ready    <none>          51m   v1.37.2

These are plain Linux servers on AWS, but nothing in this article depends on AWS. The control plane decides what runs where, and your containers run on the workers.

You talk to the cluster with a command line tool called kubectl, and it only talks to the API server. The scheduler and the controller manager don't call each other either. They watch the API server for changes and write back what they decide. The kubelet watches it for pods that belong on its server, and then asks containerd to run them. You'll see this pattern a few times below. Something changes in the API server, and another part notices and acts on it.

With kubeadm the control plane parts run as pods themselves, and that's why control-1 also has a kubelet and containerd.

Pods on different servers also have to reach each other. A network plugin does that, and here I used Cilium.

What is a Kubernetes pod?

A pod is the smallest thing you can run in Kubernetes, and usually it's one container. This is a pod for storefront, the web app of the shop:

apiVersion: v1
kind: Pod
metadata:
  name: storefront
spec:
  containers:
    - name: storefront
      image: seatlark/storefront:1.4.0

kubectl apply -f pod.yaml sends this file to the API server. The scheduler picks a worker, and the kubelet on that worker starts the container. kubectl logs storefront shows its output, like docker logs does.

Kubernetes pod vs container

It's called a pod and not a container because a pod can hold more than one container, and those containers share a network. I added a small helper to the pod:

    - name: helper
      image: busybox:1.37
      command: ["sleep", "infinity"]

You can't add a container to a pod that's already running. Kubernetes answers with spec.containers: Forbidden: pod updates may not add or remove containers. So I deleted the pod and created it again. Then I called the web app from inside the helper:

kubectl exec storefront -c helper -- wget -qO- localhost:3000/healthz

That printed ok, so localhost in the helper reached the web app in the other container. With Docker Compose, localhost normally only reaches the same container.

Containers in a pod also always run on the same server. When the pod is replaced they all get replaced, but one container can crash and restart while the other keeps running. So you put two containers in one pod only when they belong together. A common case is Nginx in front of PHP-FPM. Another is a sidecar container, which is a small helper next to your app that sends its logs somewhere or fetches secrets for it. Two parts that could also run on their own, like a web app and its database, don't belong in one pod.

A pod that you start on its own doesn't come back when you delete it:

$ kubectl delete pod storefront --now
pod "storefront" deleted from default namespace
$ kubectl get pod storefront
Error from server (NotFound): pods "storefront" not found

The same thing happens when its server dies. So you almost never write pods by hand.

Kubernetes Deployment: how many copies you want

Instead you write a Deployment. It says what a pod should look like and how many copies you want:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: storefront
spec:
  replicas: 3
  selector:
    matchLabels:
      app: storefront
  template:
    metadata:
      labels:
        app: storefront
    spec:
      containers:
        - name: storefront
          image: seatlark/storefront:1.4.0

The part under template is the same pod as before. replicas says how many you want, and the labels are how the Deployment finds its pods. You'll see pod names like storefront-5b4f79f6c9-phm9f. The Deployment creates a ReplicaSet to manage its copies, and the first random part of the name comes from that ReplicaSet.

When I deleted one of the three pods, a new one started with a new name. This is what happened inside the cluster:

1. The controller manager saw two pods where there should be three, and created a new pod through the API server.
2. The scheduler saw a pod without a server and picked a worker for it.
3. The kubelet on that worker saw that it had a new pod to run, and started it.

A crash is different. I crashed the app inside a pod, and the pod kept its name while RESTARTS went up to one:

$ kubectl get pods -l app=storefront
NAME                          READY   STATUS    RESTARTS     AGE
storefront-5b4f79f6c9-jtv82   1/1     Running   0            12s
storefront-5b4f79f6c9-kdx6v   1/1     Running   0            8s
storefront-5b4f79f6c9-ndd44   1/1     Running   1 (3s ago)   12s

In all three cases you start with a new container, so anything the app wrote into the container's own filesystem is gone. A mounted volume has its own lifetime. An emptyDir volume for example survives a crash but not a new pod. So data you need after that has to go on storage that stays around, and we get to that below.

To scale, I changed replicas: 3 to replicas: 50 and applied the file again. Each worker got 25 pods, and it took the small servers about half a minute to start them all. No new servers appear when you do this, because Kubernetes doesn't create servers by itself. These pods also all use the same database, so more copies don't help if the database is the slow part.

There's also a kubectl scale command. I changed the file because the file goes into Git, and the next kubectl apply would otherwise put the old number back.

What happens when a Kubernetes server dies

On one server, a dead server takes the whole app down. With Kubernetes the copies on the other servers keep going. For this test I ran six copies of storefront, three on each worker. The shop's front door and checkout also ran on both workers. The database and a few other parts ran only on worker-1, and I set that up on purpose. So losing worker-2 only costs pods that can run anywhere.

For the traffic I used k6. It's a load test tool, and it acted like 120 people using the shop. Each request got two seconds before it counted as failed.

Requests reach the pods through a Service. It gives all the copies one address, and it only sends new connections to copies that Kubernetes has marked as ready. I reached the shop through a NodePort, which is a port that's open on each worker. Here that was port 30080 on worker-1.

Then I shut down worker-2:

ssh worker-2 sudo poweroff

Why about a minute? The kubelet on each worker reports to the API server regularly. By default Kubernetes marks a server as NotReady when it hasn't heard from it for 50 seconds. The wait is there so a server with a short problem doesn't get treated as dead. You can change this with --node-monitor-grace-period, and a shorter time means more false alarms.

Why five more minutes? Each pod gets a default rule that lets it stay on an unreachable server for 300 seconds. That's in case the server comes back, because restarting everything elsewhere has a cost. You can set this per pod with tolerationSeconds. While the wait runs, your app only has the copies on the servers that are left, so there has to be enough room and CPU on them.

Why doesn't anything move back? The scheduler only places new pods, and it doesn't move running ones. After this test all six copies were on worker-1, and worker-2 was empty. kubectl rollout restart deployment storefront gives the scheduler new copies to place. That can spread them over both workers again, but it doesn't guarantee an even split. If that matters to you, set spreading rules and check the result.

I ran this test both ways. With a new connection for each request, small spikes of failures went on for about two more minutes after Kubernetes stopped sending new traffic to the dead copies. With k6 reusing its connections, which browsers also do, those spikes were gone, and that's the run in the table. I didn't find out the cause, so I can't say if this came from my test or from how the cluster handled those connections. I also didn't look further into the short dip at six minutes.

There's also a limit in my setup. All traffic came in through worker-1, so if that server had died, the address would have been gone with it. A cluster in production has a load balancer in front that can send traffic to any worker.

StatefulSet vs Deployment: what happens to the data

The database is harder, because its data has to be on a disk that the new pod can reach. I bought a ticket and looked it up in Postgres, which also runs in a pod here:

$ kubectl exec orders-db-0 -- psql -U seatlark seatlark_orders -c "select id, seat from orders"
  id  | seat
------+------
 1042 | D7
(1 row)

Then I deleted that pod with kubectl delete pod orders-db-0. It came back with the same name, and order 1042 was still there.

Two things make this work. orders-db is a StatefulSet, and a StatefulSet gives each pod a name that stays the same. And I gave it a disk that stays around when the pod gets deleted. This is the part of the file that does that:

kind: StatefulSet
metadata:
  name: orders-db
spec:
  serviceName: orders-db
  replicas: 1
  template:
    spec:
      containers:
        - name: orders-db
          image: postgres:17-alpine
          env:
            - { name: PGDATA, value: /pgdata }
          volumeMounts:
            - { name: orders-data, mountPath: /pgdata }
  volumeClaimTemplates:
    - metadata:
        name: orders-data
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: local-disk
        resources:
          requests:
            storage: 5Gi

PGDATA tells Postgres to write its database into the mounted folder. The claim at the bottom asks for a disk, and Kubernetes keeps that claim when the pod goes away. The new pod with the same name gets the same claim.

Here that disk is the folder /mnt/orders-data on worker-1, so this pod always has to run on that server. When worker-1 dies, the database is down. I placed an order, shut down worker-1 and tried the shop through worker-2:

• The pages of the shop still loaded from worker-2, because there were copies of the front door and the web app on it.
• Asking for the seats or holding a seat gave a 502, so nobody could buy a ticket.
• After about six minutes orders-db-0 showed as Terminating, and it stayed like that. The StatefulSet won't create another orders-db-0 while the old pod still exists, and the old pod can't finish terminating while its server doesn't answer. Its disk was also on worker-1, so a new pod couldn't have run on worker-2 anyway.
• The other parts that I had tied to worker-1, like Redis, got new pods that stayed in Pending, because they had nowhere to go.
• Once worker-1 was Ready again, the database came back about a minute later. The order was still there, and the shop could show the seats again.

Kubernetes doesn't move your data for you. To survive that, you can use a disk that another server can reach, like a disk on the network. Another way is a second copy of the database on another server, and tools like CloudNativePG run those copies for Postgres inside the cluster. Or you keep the database outside of the cluster, with a service like Amazon RDS. All of these need a safe way to take over after a crash, and you still need backups.

Not all data needs that. In this shop seat-holds is Redis with saving switched off, so the holds in it are gone when its pod is replaced. People then pick their seat again, and Postgres still decides who owns a seat. I wrote more about these choices in Docker Compose vs Kubernetes.

Settings and passwords come back on their own. They live in the cluster as a Secret, so a new pod gets them again.

Who runs Kubernetes

I set up this cluster myself with kubeadm, so the control plane is my job. You can also rent it, and then the provider runs the control plane for you. The workers are usually still servers you pay for.

OpenShift is Red Hat's platform on top of Kubernetes. kubectl and these YAML files work on all of them, apart from a few details. The parts that differ are mostly load balancers and disks, because those come from the provider. Permissions also differ. OpenShift is also stricter by default, for example about running containers as root.

Questions you might have

Do I still need Docker? You still build images, and Docker is a normal way to do that. The servers in the cluster only need a container runtime like containerd, and they run the same images.

What if the control plane dies? The apps that are running keep running, and the kubelets still restart crashed containers. But nothing new gets scheduled, and no pods get replaced. My cluster has one control plane server, so that's a weak spot. Clusters in production usually have three.

How did the images get onto the servers? For this test I loaded them onto the workers by hand. Normally the kubelet pulls them from a registry.

How do I try this without AWS? Any Linux servers work with kubeadm. On your laptop, kind runs a whole cluster in containers.

Is Kubernetes worth it for a small app? Often not. It's a lot to learn and to run, and one server gets you further than you'd think. I load tested the same shop on one server in Docker Compose vs Kubernetes: when is one server enough?

How I ran this

I used three servers on AWS with Ubuntu 24.04 for the cluster. The control plane ran on a t3.medium, and the two workers were t3.large servers. A fourth t3.medium ran k6. I tested this on November 6, 2026.

These are the versions:

• Kubernetes 1.37.2, set up with kubeadm
• containerd 2.4.2
• Cilium 1.20.3
• Postgres 17

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com