Kubernetes rolling updates without failed requests

When you release a new version on Kubernetes, the Deployment replaces your pods step by step. That's called a rolling update, and it's supposed to give you a release without downtime. But with the default settings, some requests still fail. Kubernetes sends traffic to new pods that aren't ready yet, and also to old pods that are already shutting down.

Two settings fix this, a readiness probe and a short wait before a pod stops. I tried it with a ticket shop that I made up, it's called Seatlark and it sells concert tickets online. While a load test acted like 120 people using the shop, I released new versions of it. I started with a plain Deployment and added one setting at a time, until a release went through without a single failed request. At the end I released a broken version and rolled it back.

Each of the 120 people in the load test loaded the shop's page and its seat data, again and again. That came to about 120 requests per second, a bit less while requests failed. The page comes from the web app I released, and the seat data from a separate app that I didn't touch. The table counts both kinds of requests together:

The counts other than zero are estimates. The load test printed the share of failed requests for each second, rounded to a whole percent, and I multiplied that by the number of requests in that second.

With the readiness probe, Kubernetes asks the app in each new pod if it's ready, and it only sends requests there when the answer is yes. The short wait is a preStop hook. With it, an old pod still answers for a few seconds while Kubernetes takes it out of the Service. The Service is the part of Kubernetes that spreads the requests over the pods.

The broken release didn't take the shop down. Its new pods crashed, so the rolling update stopped right there, and five of the six old pods kept running. Rolling it back is one command, but Kubernetes doesn't run it for you.

A liveness probe is for an app that still runs but doesn't answer anymore. When I froze one pod, some requests failed for about six seconds. Then the readiness probe failed, and Kubernetes took the pod out of the Service. A bit later the liveness probe gave up, and Kubernetes restarted the container.

What a rolling update does

A Deployment doesn't change the pods that are running. When you change the pod template, for example the image, it makes a new ReplicaSet for the new version. The pod template is the part of the Deployment that says how the pods should look, and a ReplicaSet is the group of pods for one version. Then the Deployment makes the new group bigger step by step, and the old one smaller. After the first release, the old one was down to zero:

$ kubectl get replicasets -l app=storefront
NAME                    DESIRED   CURRENT   READY   AGE
storefront-56b48d64db   0         0         0       29s
storefront-7994546d5c   6         6         6       27s

The old ReplicaSet stays around with zero pods, and a rollback uses it later. Two settings decide how fast the switch goes:

$ kubectl get deployment storefront -o jsonpath='{.spec.strategy}'; echo
{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"}

So with six pods, Kubernetes starts two new pods and stops one old one right away. Once that old pod is on its way out, there's room for a third new pod. After that it only stops more old pods when new ones are ready. Without a readiness probe, ready only means that the container runs, even if the app in it can't answer yet.

Why requests fail during a rolling update

The first release failed requests because the pods had no readiness probe. This is the Deployment I started with:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: storefront
spec:
  replicas: 6
  selector:
    matchLabels:
      app: storefront
  template:
    metadata:
      labels:
        app: storefront
    spec:
      containers:
        - name: storefront
          image: registry:5000/seatlark/storefront:1.4.0

Version 1.5.0 of the web app loads the prices of all events when it starts. In my test app that's simply an eight-second wait before it opens its port. Without a readiness probe, a pod counts as ready as soon as its container runs. So Kubernetes sent requests to the new pods before the app in them listened on its port. And it stopped the old pods, because the new ones looked ready. The whole rollout took four seconds. For the next ten seconds or so, about half of the requests failed. The seat data still came from the other app, so that was nearly every request for the page itself.

Nothing was wrong with the new version. It only needed a few seconds before it could answer.

Kubernetes readiness probe

So Kubernetes needs a way to ask the app if it's ready, and that's what a readiness probe is for. It's a check that the kubelet runs against each pod. The kubelet is the part of Kubernetes that runs on each server and starts the pods there. As long as the check fails, the pod isn't ready, and the Service doesn't send it new requests. The Deployment also counts only ready pods when it decides whether to stop more old ones.

          readinessProbe:
            httpGet:
              path: /healthz
              port: 3000
            periodSeconds: 2

Here the kubelet calls /healthz on port 3000 every two seconds. The app only opens the port once it's done loading, so the check fails until then. Instead of an HTTP request, a probe can also open a TCP connection with tcpSocket, or run a command in the container with exec. For gRPC services there's grpc.

With the probe, the new pods stayed not ready for about eight seconds, and the old pods kept the traffic until then. The rollout took about 24 seconds instead of four. A few requests still failed, and those came from the old pods that got stopped.

Graceful shutdown: preStop hook and terminationGracePeriodSeconds

The failures that were left came from old pods that Kubernetes stopped. When Kubernetes stops a pod, two things start at about the same time. The pod gets removed from the Service, and the container gets its stop signal. The removal has to reach the network rules on every server, and that takes a moment. In that moment, new requests can still arrive at a pod whose app has already stopped.

My app exits right away when it gets the signal. That cuts off requests it's still working on, and the requests that arrive before the routing has caught up fail. For the second part, the usual fix is a short wait before the signal, and in Kubernetes that's a preStop hook:

          lifecycle:
            preStop:
              sleep:
                seconds: 5

During those five seconds the pod is already on its way out of the Service, and the app keeps answering what still arrives. After that the container gets the stop signal. The sleep action is built into Kubernetes, so the image doesn't need its own sleep program. In older clusters people wrote exec: command: ["sleep", "5"] instead.

The first release with the hook still had about 17 failed requests, because the pods that got stopped in that release were the old ones, and those didn't have the hook yet. A hook only helps a pod that has it when it gets stopped. So I ran kubectl rollout restart deployment/storefront, which replaces all pods once more with the same settings. This time every pod that got stopped had the wait, and not a single request failed.

So when you add a shutdown setting, the release that adds it doesn't show its effect yet. Only the next one does. Five seconds were enough here, and your setup may need a longer wait.

Two more things matter for a clean shutdown:

• terminationGracePeriodSeconds is how long Kubernetes waits in total before it kills the container. The default is 30 seconds, and the preStop wait is part of those 30.
• The app itself should stop taking new connections when it gets the signal, and it should finish the requests it already has. The fixed wait helps with the routing, but it doesn't help a request that needs longer than the time that's left.

A bad release: CrashLoopBackOff and kubectl rollout undo

A broken version doesn't take the shop down, because the rolling update stops when the new pods don't get ready. Version 1.6.0 of the web app needs a new setting, PRICE_API_URL, and I didn't put it in the Deployment. So the app stopped right after it started:

$ kubectl get pods -l app=storefront
NAME                          READY   STATUS             RESTARTS      AGE
...
storefront-5985cfd76b-fbvdj   1/1     Running            0             87s
storefront-5985cfd76b-fh724   1/1     Running            0             87s
storefront-5985cfd76b-k2b4s   1/1     Running            0             75s
storefront-5985cfd76b-krwlc   1/1     Running            0             75s
storefront-ff744dcb7-blwxf    0/1     CrashLoopBackOff   2 (7s ago)    22s
storefront-ff744dcb7-x7dzv    0/1     CrashLoopBackOff   2 (5s ago)    22s
storefront-ff744dcb7-zwkw9    0/1     Error              2 (19s ago)   22s

CrashLoopBackOff means the container keeps crashing, and Kubernetes waits longer before each new start, up to five minutes. Error is the same pod in the moment right after a crash, before the next wait starts. kubectl logs with --previous shows what the last crashed container printed, and the reason is usually in there.

The shop kept working, because of the same two limits as before. Kubernetes started three new pods and stopped one old one. The new pods never got ready, so it didn't stop any more. Five old pods carried the load. In the recorded run, about two requests failed in the second after the apply, and none after that. In two earlier runs of the same step, none failed at all.

Nothing rolls this back by itself. After ten minutes without progress, the Deployment reports ProgressDeadlineExceeded, and that's all it does. You roll back with:

kubectl rollout undo deployment/storefront

That makes the previous ReplicaSet the current one again. kubectl rollout history deployment/storefront lists the revisions, and --to-revision picks an older one.

rollout undo only changes the Deployment in the cluster. Your file stays as it is, and kubectl even warns about that:

Warning: resource deployments/storefront was previously managed with 'kubectl apply'. Rolling back will not update the kubectl.kubernetes.io/last-applied-configuration annotation, which may cause unexpected behavior on future 'kubectl apply' operations. Consider using 'kubectl apply' with your previous configuration file instead.

My file still said 1.6.0, so the next kubectl apply would have brought the broken version back. If your files are in Git, fix it there.

Readiness vs liveness probe

A readiness probe only keeps requests away from a pod. If the app in it hangs for good, the pod stays not ready, and nothing restarts it. A liveness probe is a check that restarts the container when it fails. It's for an app that still runs but doesn't answer anymore, for example because it's stuck in a loop. I added one next to the readiness probe:

          livenessProbe:
            httpGet:
              path: /healthz
              port: 3000
            periodSeconds: 5

Then I made one pod freeze, so it didn't answer anything anymore. In the recorded run, requests to it failed for six seconds, at most 14% in one second. The readiness probe checks every two seconds, and by default it needs three failed checks in a row. So the pod counted as not ready after about six seconds, and then the change still had to reach the network rules. In the rehearsal I saw the pod out of the Service after about eight seconds.

The liveness probe checks every five seconds, so it gave up after about 15. Then the restart took longer than I expected. A frozen app doesn't even react to the stop signal. So Kubernetes ran the five-second preStop wait, and then it waited for the rest of the 30 seconds before it killed the container. In the rehearsal the container restarted about 40 seconds after the freeze, and the pod was ready again after about 50:

$ kubectl get pod storefront-c96965574-gb5qd
NAME                         READY   STATUS    RESTARTS     AGE
storefront-c96965574-gb5qd   0/1     Running   1 (2s ago)   74s
$ kubectl get pod storefront-c96965574-gb5qd
NAME                         READY   STATUS    RESTARTS      AGE
storefront-c96965574-gb5qd   1/1     Running   1 (24s ago)   96s

If that's too slow for you, a liveness or startup probe can have its own terminationGracePeriodSeconds. Kubernetes uses that shorter time when this probe restarts the container, and the preStop wait also counts toward it. Normal rollouts still use the pod's 30 seconds.

If you add a startup probe, the other two don't run until it has passed once. That helps with apps that sometimes need a minute to start, because the liveness probe can't kill them while they're still starting.

Be careful with liveness probes. If the check is too strict, Kubernetes restarts pods that are only busy, and under load that takes away capacity when you need it most. And don't let a liveness probe check the database or other services. If the database is down, restarting all your web pods doesn't help.

Blue green and canary deployments in Kubernetes

A rolling update mixes the old and the new version for a short time. Two other ways to release are common.

Blue green. The new version runs next to the old one with full capacity. Then you point the Service at the new pods, for example by changing its selector. That change takes a moment to reach every server, and connections that are already open can stay with the old pods. So keep the old pods running for a while. As long as they run, switching back is quick, but you need double the pods for that time.

Canary. A small share of the traffic goes to the new version first, and you watch your error rates before you move more traffic over. With two Deployments behind one Service, the share of new connections roughly follows the number of ready pods. Connections that stay open can make the split uneven. Tools like Argo Rollouts or a Gateway with traffic weights do it more precisely.

Both still need readiness probes and a clean shutdown, so they build on the same settings.

Questions you might have

What does kubectl rollout restart do? It starts a rolling update with the same version, by changing an annotation in the pod template. It's useful when the pods should start fresh, for example after a Secret changed that they only read when they start. I used it to show that the preStop wait works once all pods have it.

What does kubectl rollout status do? It waits until the rollout is done, and it returns an error if that doesn't happen within --timeout. In a deploy script, that's how you notice a stuck release.

Do I need a /healthz endpoint? It helps. For readiness it should answer only when the app can serve requests. For liveness it should only check that the process itself still works.

Does this work the same on EKS and other managed clusters? Yes. Rolling updates and probes are part of Kubernetes itself, and so is the preStop hook. If a cloud load balancer sends traffic straight to the pods, it has its own way of adding and removing them, and that needs its own care. On EKS, the AWS Load Balancer Controller can use readiness gates, so old pods keep running until their replacements are healthy in the load balancer. The five-second wait from this test might not be enough there.

How I ran this

The cluster had three servers on AWS with Ubuntu 24.04. A t3.medium ran the control plane, and two t3.large servers were the workers. The web app ran six copies behind an Nginx front door. A fourth t3.medium built the images and ran the registry. The load test also ran on that server. It was k6, a tool that acts like a lot of people using your site. A request counted as failed when it got an error, or no answer within two seconds. I tested this on November 7, 2026.

These are the versions:

• Kubernetes 1.37.2, set up with kubeadm
• containerd 2.4.2
• Cilium 1.20.3, next to kube-proxy
• k6 1.3.0

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com