Kubernetes Jobs, CronJobs and queue workers

A worker that takes jobs from a queue runs all the time, so in Kubernetes it's a normal Deployment. Work that runs once and then ends, like a database migration, is a Job. And work on a schedule, like a report every night, is a CronJob. With a queue, a job can also run twice. If a worker stops before it confirms a job, the queue gives the job to another worker, and that one does the work again. So your background work has to be safe to repeat.

I tried this with a ticket shop on a small cluster. I made the shop up, it's called Seatlark and it sells concert tickets online. Its worker takes jobs from a RabbitMQ queue and makes the PDF tickets. The next version of the worker needed a database migration, and I ran that as a Job. The new worker also got an init container, that's a container that runs before the app, and it waited for the migration. The nightly sales report became a CronJob. Then I deleted the worker pod in the middle of a job.

The migration Job was done after 3 seconds, and the new worker waited in its init container until the table was there. When I deleted the worker pod while it made a PDF, RabbitMQ gave the job to the new pod. So one order got two ticket numbers for one seat. Then a second migration made the order number unique, and the same test gave one ticket per order. The new worker found the ticket that was already there and went on with that one.

This isn't a bug in RabbitMQ. The first pod had already saved a ticket number when RabbitMQ gave the job to the new pod. The nightly report also has to be safe to repeat, because Kubernetes can miss a scheduled run, and in rare cases it starts one twice.

Kubernetes workload types

This is what each kind is for, and where it shows up in the shop:

A worker that takes jobs from a queue

When someone buys a ticket, checkout-api saves the order and puts a message into the RabbitMQ queue tickets.requested. ticket-worker takes the messages one at a time. For each one it makes the PDF and marks the order as ready. Then it confirms the message, which is called an acknowledgement or ack. Until the ack arrives, RabbitMQ keeps the message. If the worker's connection closes before the ack, RabbitMQ puts the message back into the queue for another worker.

That's what keeps a job from getting lost when a worker dies. It's also what makes a job run twice, because the first worker may have done part of the work before it died.

RabbitMQ can keep queues and messages through a restart, which it calls durable queues and persistent messages. But that only works when RabbitMQ still has its data files after the restart. In this shop, RabbitMQ runs without a volume, so its files are inside the container. A new pod starts empty, and so does a restarted container. To keep the messages, RabbitMQ needs a volume that survives the pod.

Database migrations as a Kubernetes Job

The next version of the worker gives each ticket a number in a new table. The table has to exist before the new version starts, so a Job creates it. The SQL comes from a ConfigMap. That's the Kubernetes object for settings, and a pod can mount it as files:

apiVersion: v1
kind: ConfigMap
metadata:
  name: db-migrations
data:
  001-tickets.sql: |
    CREATE TABLE IF NOT EXISTS tickets (
      id SERIAL PRIMARY KEY,
      order_id INT NOT NULL,
      created_at TIMESTAMPTZ NOT NULL DEFAULT now()
    );

The Job runs psql, the Postgres command line, once for each file:

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate-1-5-0
spec:
  backoffLimit: 3
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: postgres:17-alpine
          command: ["sh", "-ec", "for f in /migrations/*.sql; do psql -v ON_ERROR_STOP=1 -f $f; done"]
          env:
            - name: PGHOST
              value: orders-db
            ...
          volumeMounts:
            - name: migrations
              mountPath: /migrations
      volumes:
        - name: migrations
          configMap:
            name: db-migrations

A Job is complete when its pod exits without an error. If the pod fails, the Job starts a new one, up to backoffLimit times. After that the Job counts as failed. Three things matter here.

The exit code has to be right. ON_ERROR_STOP makes psql stop at the first error, and sh -e makes the loop stop at the first psql that fails. Without that, a failing file followed by a working one would look like a success.

The migration can run more than once. A retry runs the same SQL again, so it has to be safe to repeat. CREATE TABLE IF NOT EXISTS is. The second run printed a notice that the table already existed and went on.

And applying a finished Job again doesn't run it again. The Job needs a new name, or you delete the old one first. Here each release gets its own name, with the version in it.

This is what the Job looked like after it ran. 1/1 means one pod finished, and one was all it needed. It took 3 seconds:

$ kubectl get jobs
NAME               STATUS     COMPLETIONS   DURATION   AGE
db-migrate-1-5-0   Complete   1/1           3s         4m54s

Outside a test like this one, the migration Job often comes from the deployment tool. With Helm, the package manager for Kubernetes, the Job can be a hook that runs before an upgrade. Helm updates the other objects only after that hook, so the SQL and the database password have to be there already. With Argo CD, which deploys what's in Git, you put the Job in an earlier sync wave than the app.

Kubernetes init containers

The new worker must not start before the table exists. An init container takes care of that. Init containers run before the app's containers. If there are several, they run one after the other, and each has to finish without an error before the next one starts:

    spec:
      initContainers:
        - name: wait-for-tickets-table
          image: postgres:17-alpine
          command: ["sh", "-c", "until psql -h orders-db -c 'SELECT 1 FROM tickets LIMIT 1' >/dev/null 2>&1; do echo waiting for the table tickets; sleep 2; done"]
          env:
            ...
      containers:
        - name: ticket-worker
          image: registry:5000/seatlark/ticket-worker:1.5.2

I applied the new worker before the Job, and its pod waited:

$ kubectl get pods -l app=ticket-worker
NAME                             READY   STATUS     RESTARTS   AGE
ticket-worker-5cf979bf87-gzqj5   1/1     Running    0          47s
ticket-worker-84578d8764-sx7fk   0/1     Init:0/1   0          10s

Init:0/1 means the pod has one init container, and it hasn't finished yet. The old worker kept running in the meantime, because a rolling update only removes the old pod when the new one is ready. After the Job had created the table, the init container finished, and the new worker started.

You could also run the migration itself in an init container. Then every new pod tries it, and with several copies of the app, several pods try it at the same time. A Job keeps the migration out of the app's start. But even a Job can run it more than once, so the SQL still has to be safe to repeat.

When a worker dies in the middle of a job

Pods get deleted all the time in Kubernetes, for example during a deployment or when a server gets drained for maintenance. When that happens to a worker in the middle of a job, the job runs again on another worker. The new worker first saves a ticket number in the database and then makes the PDF. So that I could stop it in the middle, I made it wait five seconds between the two steps. I bought a ticket and deleted the worker pod during those five seconds. The worker stopped right away without an ack, so RabbitMQ gave the message to the new pod. The new pod saved a second ticket number for the same order:

$ kubectl exec orders-db-0 -- psql -U seatlark seatlark_orders -c "SELECT order_id, count(*) AS tickets FROM tickets GROUP BY order_id"
 order_id | tickets
----------+---------
     1049 |       2
(1 row)

The test left two ticket numbers for one order. If a real ticket system sent out both, there would be two valid barcodes for one seat at the door. The same happens with anything a job does before its ack. An email goes out twice, or a card gets charged twice.

When Kubernetes deletes a pod, the container first gets a stop signal. My worker stops right away on that signal, because I wrote it that way. A worker that finishes its current job first, within the time Kubernetes gives it in terminationGracePeriodSeconds, gets fewer duplicates. It still gets some, because a crash or a dead server doesn't give it that time.

Making the work safe to repeat

The usual fix is a key for each piece of work, so a second run can see that the work was already done. Here the key is the order number. A second migration removes the extra tickets and lets the database refuse a second ticket for the same order:

DELETE FROM tickets a USING tickets b WHERE a.order_id = b.order_id AND a.id > b.id;
CREATE UNIQUE INDEX IF NOT EXISTS tickets_one_per_order ON tickets (order_id);

The new worker version tries to insert the ticket. If the order already has one, it carries on with that one:

row = conn.execute(
    "INSERT INTO tickets (order_id) VALUES (%s) ON CONFLICT (order_id) DO NOTHING RETURNING id", (order,)
).fetchone()
if row is None:
    row = conn.execute("SELECT id FROM tickets WHERE order_id = %s", (order,)).fetchone()

The check happens in the database. A worker that first asks whether there is a ticket and then inserts one has a gap between the two steps. Another worker could insert a ticket in that gap. The unique index has no such gap.

In my test no ticket came in while the migration ran. In a shop with traffic, a new duplicate could come in between the DELETE and the index, and then creating the index fails. So for a migration like this you'd stop the workers first, and start the new version after it.

I ran the same test with this version. This is the log of the new pod:

$ kubectl logs deploy/ticket-worker
ticket-worker 1.6.2 waiting for orders
order 1050: ticket 3 already exists, carrying on with it
order 1050: ticket 3 for seat G3, making the PDF
order 1050: ticket ready, 1473 bytes

The ticket from the first pod was already there, so the new pod made the PDF for that one. The order ended up with one ticket. The worker still makes the PDF again on a retry. Here that's fine, because it only makes it. A step that sends the PDF to the customer would need the same kind of check.

For work outside your database, you need the same thing from the other side. Payment providers and many APIs take a key with each request, called an idempotency key. When you retry a request with the same key, they don't run it twice. They only keep the keys for a while, Stripe for example for at least 24 hours. A retry after that can run again.

Jobs that keep failing

A job that fails every time, for example because of a broken order, would go back into the queue forever. RabbitMQ's quorum queues have a limit for how often a message gets delivered again, and by default it's 20. Above the limit RabbitMQ drops the message. It only keeps it when a dead letter exchange sends it on to a separate queue for failed jobs. Here that would be tickets.failed, and someone would look at those by hand. I didn't set this up or test it.

Kubernetes CronJob

The nightly report also has an end, and it has to run again each day. For that there's the CronJob. It starts a Job on a schedule, and this one makes the sales report every night at three:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: payout-report
spec:
  schedule: "0 3 * * *"
  timeZone: Europe/Berlin
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: report
              image: postgres:17-alpine
              command: ["psql", "-h", "orders-db", "-c", "SELECT count(*) AS tickets_sold, max(created_at) AS last_sale FROM orders WHERE paid"]
              ...

The schedule has the usual cron format. timeZone makes three o'clock mean three o'clock in Berlin, instead of in the time zone of the controller that starts the Jobs. concurrencyPolicy: Forbid means a scheduled run doesn't start while the last one is still running. It can still start once the last one is done, and startingDeadlineSeconds says how late that may be. Replace would stop the old one and start the new one. Allow is the default, and it lets them overlap.

To test it without waiting until three, I created a Job from the CronJob's template:

$ kubectl create job --from=cronjob/payout-report payout-now
$ kubectl logs job/payout-now
 tickets_sold |           last_sale
--------------+-------------------------------
            9 | 2026-11-07 19:56:55.970943+00
(1 row)

The report says 9 tickets were sold and when the last one was. A Job made like this doesn't count as a scheduled run, so Forbid doesn't apply to it. And for now the report only goes to the pod's log. In the shop it would have to go to the organisers.

Kubernetes can also miss a scheduled run, for example while the CronJob controller is down, and in rare cases it starts one twice. So the report should be safe to run twice, like the jobs from the queue.

Kubernetes DaemonSet

The last kind is for programs that each server needs, like networking. A DaemonSet keeps one pod on each server it's meant for, and the cluster already had a few:

$ kubectl get daemonsets -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,PODS:.status.currentNumberScheduled
NAMESPACE     NAME                                       PODS
kube-system   cilium                                     4
kube-system   cilium-envoy                               4
kube-system   kube-proxy                                 4
rook-ceph     rook-ceph.cephfs.csi.ceph.com-nodeplugin   3
rook-ceph     rook-ceph.rbd.csi.ceph.com-nodeplugin      3

The networking DaemonSets run on all four servers, the control plane included, because every server needs networking. The drivers for Ceph, the cluster's storage, run only on the three workers, where pods with Ceph disks can run. When a new server joins the cluster, every DaemonSet that's meant for it starts a pod there by itself. Log collectors and monitoring agents usually run the same way.

Questions you might have

Why not run the worker as a Job? A Job is for work that ends. A worker that waits for messages never ends, so it's a Deployment. Kubernetes can also run a fixed batch of work items as one Job, with completions and parallelism.

How do I scale workers with the queue? The usual tool is KEDA. It watches the length of a queue and changes the number of worker pods.

What happens to finished Jobs? They stay until you delete them. ttlSecondsAfterFinished deletes them after a time you set, and a CronJob keeps only its last few runs. Their pods and logs can go away earlier, so a report that matters has to be saved somewhere else.

Should the app run its own migrations at start? It can work when the migration tool keeps track of what already ran and takes a lock in the database, so only one copy changes the tables. Without that, several copies try at once. And a failed migration then crashes the app in a loop. A Job before the release keeps the migration apart from the app.

How I ran this

The cluster had four servers on AWS with Ubuntu 24.04: a t3.medium for the control plane and three t3.large workers. I tested this on November 7, 2026.

These are the versions:

• Kubernetes 1.37.2, set up with kubeadm
• RabbitMQ 4.4.0
• PostgreSQL 17.11
• ticket-worker in Python 3.13 with pika 1.3.2

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com