Kubernetes Services: how the parts of an app find each other

In Kubernetes your app runs in pods, and pods get replaced all the time. Each new pod also gets a new IP address, so the parts of your app can't talk to each other through these addresses. That's what a Service is for. It gives a group of pods one name and one address that doesn't change, and the DNS inside the cluster turns the name into that address.

I tried this with a ticket shop on a small cluster. I made the shop up, it's called Seatlark and it sells concert tickets online. I wrote the files for its two main parts by hand. One of them needed its image from a registry, and the pods waited until I pushed the image there. At the end I changed one letter in the label that a Service looks for. That's the most common mistake with Services, and the shop showed an error page.

When I deleted a pod, the new pod had a different IP address. The shop's code doesn't care about that. It connects to names like orders-db, and each of these names belongs to a Service. The Service has the same address the whole time, while the pods behind it come and go. Kubernetes has a list of the pods behind it and updates it by itself.

The two pods of checkout-api first showed ImagePullBackOff, because their image wasn't in the registry yet. That means Kubernetes waits a bit before it tries the download again. After I pushed the image, each pod started at its next try.

A short name like checkout-api only works inside the same namespace. A namespace is basically a group of names in the cluster. From a pod in another namespace, checkout-api wasn't found, but checkout-api.seatlark worked.

With the wrong letter, the Service was still there and still had its address. But no pod matched anymore, so the shop showed a 502. Kubernetes didn't warn about it at all. So when a Service doesn't answer, I think the first thing to check is the list of pods behind it.

Why pods need a Service

Each pod gets its own IP address, and that address only lasts as long as the pod does. These are the parts of the shop that already ran, with their addresses:

$ kubectl get pods -n seatlark -o custom-columns=NAME:.metadata.name,IP:.status.podIP,SERVER:.spec.nodeName
NAME                               IP           SERVER
orders-db-0                        10.0.1.136   worker-1
payment-service-76fdd8d646-b2vh4   10.0.2.79    worker-2
seat-holds-8689b49fc7-7j7vb        10.0.2.183   worker-2
ticket-queue-7b7f7bb58d-4bsxh      10.0.1.27    worker-1
ticket-worker-5cf979bf87-zkm2r     10.0.2.31    worker-2

Then I deleted the pod of seat-holds. Its Deployment started a new one, seat-holds-8689b49fc7-rq9v8, and that one got a new address, 10.0.2.21. A Deployment keeps as many pods running as you asked for, so it replaced the deleted one. The same thing happens with each deploy and each time a server dies. A crash that replaces the pod does the same. So the parts of an app can't use pod addresses to reach each other.

The code of the shop doesn't. checkout-api connects to its database by name:

DB = dict(host="orders-db", user=os.environ["POSTGRES_USER"], ...)

That name belongs to a Service. Services and DNS come further down, after the images.

Where Kubernetes gets the image: registries and ImagePullBackOff

Before a container can start, its image has to be on that server. The kubelet, the part of Kubernetes that runs on each server and starts the pods there, has the container runtime download it when it's missing. It comes from a registry, which is a server that stores images. Docker Hub is the one most people know. Cloud providers and GitLab have their own. Some companies run one themselves, for example with Harbor.

The image name says where to download it from:

registry:5000/seatlark/checkout-api:1.4.1

This is the Deployment I wrote for checkout-api. The image name is in there, and so is the number of copies:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      containers:
        - name: checkout-api
          image: registry:5000/seatlark/checkout-api:1.4.1
          envFrom:
            - secretRef:
                name: seatlark-env

The image wasn't in the registry yet when I applied it, so both pods failed:

$ kubectl get pods -l app=checkout-api
NAME                            READY   STATUS             RESTARTS   AGE
checkout-api-6f9746455d-6kz4h   0/1     ImagePullBackOff   0          17s
checkout-api-6f9746455d-wcjjj   0/1     ImagePullBackOff   0          17s

kubectl describe pod shows the reason at the bottom: failed to resolve reference "registry:5000/seatlark/checkout-api:1.4.1": ... not found.

ErrImagePull means a download failed. ImagePullBackOff means Kubernetes is waiting before it tries again. The wait gets longer with each try, up to five minutes, and each pod has its own wait. So after I pushed the image, the pods didn't start at once. Each one started when its next try came. If you don't want to wait, delete the pods. The Deployment makes new ones, and those try at once.

These are the usual reasons for ImagePullBackOff:

I saw the last one while I set this up. My registry is a small one on a helper server, and it has no TLS. containerd, which runs the containers on each server, has to be told about that, and I did that with the file /etc/containerd/certs.d/registry:5000/hosts.toml. On my Ubuntu servers containerd still tried HTTPS. I worked around it with use_local_image_pull = true in containerd's config, which makes Kubernetes downloads go another way through containerd. My servers had two registry config folders listed together, and as far as I can tell from the source, containerd 2.2.1 handles that list differently on the two ways. If you keep the registry for longer than a test, give it a certificate, and then that problem doesn't come up.

A private registry with a login

At a company, the registry normally needs a login. In Kubernetes you put that login into a Secret, which is how Kubernetes stores passwords. The Secret has to be in the same namespace as the pods, here seatlark, and you name it under imagePullSecrets. In a Deployment that goes under spec.template.spec:

kubectl -n seatlark create secret docker-registry registry-login \
  --docker-server=registry.example.com \
  --docker-username=... --docker-password=...

    spec:
      imagePullSecrets:
        - name: registry-login
      containers:
        - name: checkout-api
          image: registry.example.com/seatlark/checkout-api:1.4.1

My registry had no login, so I didn't test this part.

Tags and pull policy

The tag also decides when Kubernetes checks the registry again. If you leave out imagePullPolicy, a new Deployment with a tag like 1.4.1 gets IfNotPresent, which uses the copy on the server if there is one. With latest, or no tag, it gets Always, which checks the registry each time a container starts and downloads what's missing. Changing the tag later doesn't change that policy. That's one reason to tag images with a version number. The other is that latest doesn't tell you which version runs.

What a Kubernetes Service is

A Service gives a group of pods one name and one address. The name orders-db in the code of checkout-api is such a name, and it belongs to the Service of the database. This is the Service I wrote for checkout-api itself, in the same file as its Deployment:

---
apiVersion: v1
kind: Service
metadata:
  name: checkout-api
spec:
  selector:
    app: checkout-api
  ports:
    - port: 8000

The selector says which pods belong to the Service. Here that's all pods in the same namespace with the label app: checkout-api, and the pods get that label from the template of the Deployment. port is the port of the Service. Without a targetPort, the Service sends traffic to the same port on the pods, and here that's 8000, where the app listens.

After kubectl apply, the Service got its own address:

$ kubectl get service checkout-api
NAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
checkout-api   ClusterIP   10.102.238.6   <none>        8000/TCP   44s

That address is a ClusterIP. It only works inside the cluster, and it stays the same for as long as the Service exists. Behind it, Kubernetes keeps a list of the pods that belong to the Service, in objects called EndpointSlices. When I scaled the Deployment from two to three copies, the third address showed up in that list by itself:

$ kubectl get endpointslices -l kubernetes.io/service-name=checkout-api
NAME                 ADDRESSTYPE   PORTS   ENDPOINTS                         AGE
checkout-api-9ksjn   IPv4          8000    10.0.2.168,10.0.1.34,10.0.2.243   44s

When a pod goes away, it leaves the list. You don't have to update anything for that.

Only pods that are ready get new connections, and connections that are already open can keep going. Without a readiness check, a pod counts as ready as soon as its container runs, even if the app inside is still starting. If you give the pod a readiness check, it only gets traffic once that check passes.

Who sends the traffic

The Service's address is a virtual one. On each server, the networking sends connections for that address to one of the pods in the list. In a lot of clusters, a part called kube-proxy writes those rules. My cluster also has kube-proxy. But Cilium, the network plugin of my cluster, takes the traffic from pods to Service addresses before it reaches those rules. You can see Cilium's view of the Service:

$ kubectl -n kube-system exec ds/cilium -- cilium-dbg service list | grep -A2 10.102.238.6
22   10.102.238.6:8000/TCP     ClusterIP      1 => 10.0.1.34:8000/TCP (active)
                                              2 => 10.0.2.168:8000/TCP (active)
                                              3 => 10.0.2.243:8000/TCP (active)

The address on the left is the Service, and the addresses on the right are the pods behind it. With kube-proxy or with Cilium, the pod gets picked once per connection. If a client keeps one connection open, all its requests go to the same pod.

Kubernetes service types

checkout-api and storefront use the default type, ClusterIP, and that only works inside the cluster. The users of the shop are outside, so the front door uses a NodePort instead. These are the types:

This is how the front door looked from the cluster:

$ kubectl get service front-door
NAME         TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
front-door   NodePort   10.96.57.208   <none>        80:30080/TCP   111s

The 80:30080 means port 80 on the Service, and port 30080 on each server. You wouldn't normally let users in through a port on each server. On a cloud you'd use a LoadBalancer Service, or a Gateway with a load balancer in front of it, and that can also handle HTTPS.

Kubernetes DNS: how a name becomes an address

The code connects to orders-db, and something has to turn that name into the address of the Service. For that, each cluster runs a DNS server. If you set the cluster up with kubeadm, like I did, that's CoreDNS, and it runs as two pods in the namespace kube-system. For each Service it answers to a name like this:

checkout-api.seatlark.svc.cluster.local

The first part is the name of the Service, and the second is its namespace. Then comes svc, and at the end the domain of the cluster. Nobody types all of that, because each pod gets a list of domains to try. To test this, I started a small pod in a namespace of its own, called dns-test. This is the file from that pod:

$ kubectl exec -n dns-test dns-test -- cat /etc/resolv.conf
search dns-test.svc.cluster.local svc.cluster.local cluster.local eu-north-1.compute.internal
nameserver 10.96.0.10
options ndots:5

The first line is that list. The last entry in it comes from AWS, because the server's own search domains get added to the list. A short name like checkout-api gets dns-test.svc.cluster.local added first, because the test pod ran in the namespace dns-test. That gave checkout-api.dns-test.svc.cluster.local, which doesn't exist. The other entries didn't find anything either, so the short name failed:

$ kubectl exec -n dns-test dns-test -- wget -qO- checkout-api:8000/api/seats
wget: bad address 'checkout-api:8000'

With the namespace in the name, checkout-api.seatlark got svc.cluster.local added from the second entry. That name exists, and checkout-api answered:

$ kubectl exec -n dns-test dns-test -- wget -qO- checkout-api.seatlark:8000/api/seats
{"sold":[],"held":[],"opens_in":0}

The nameserver line is the address of the Service in front of CoreDNS. ndots:5 means that a name with fewer than five dots goes through the list before it's tried as it is. checkout-api.seatlark worked because of that rule. The same rule sends a lookup of an outside name like api.stripe.com through the list first, so it makes a few useless queries inside the cluster.

Kubernetes namespaces

The short name only worked from inside the shop's own namespace. A namespace is a group of names inside one cluster. All of the shop runs in seatlark. Two Services can have the same name as long as they're in different namespaces. While I recorded this, an older copy of the shop still had a Service called checkout-api in the namespace default, and the two didn't get in each other's way.

kubectl works in one namespace at a time. You pick it per command with -n, or once, like I did:

kubectl config set-context --current --namespace seatlark

Two things are worth knowing about namespaces:

• Some objects don't belong to a namespace. A server, which Kubernetes calls a node, is the same for the whole cluster. The same goes for PersistentVolumes and StorageClasses.
• A namespace keeps names apart. It doesn't block traffic. A pod in dns-test reached checkout-api in seatlark without any problem. If you want to stop that, you need NetworkPolicies.

The most common Service mistake

A Service finds its pods through the selector, so one wrong letter there is enough to break it. This is what the Service for storefront looked like:

apiVersion: v1
kind: Service
metadata:
  name: storefront
spec:
  selector:
    app: storefront
  ports:
    - port: 3000

I changed storefront in the selector to storefrnt and applied it. kubectl said service/storefront configured, so nothing looked wrong. The Service kept its address. But no pod has the label app: storefrnt, so the list behind it was empty:

$ kubectl get endpointslices -l kubernetes.io/service-name=storefront
NAME               ADDRESSTYPE   PORTS     ENDPOINTS   AGE
storefront-d94hv   IPv4          <unset>   <unset>     41s

The front door of the shop is Nginx, and it couldn't connect to anything behind that address. So the shop showed 502 Bad Gateway. Nothing in Kubernetes warns you about a selector that matches no pods. When I changed the letter back, the list filled up again, and the shop was back.

When a Service doesn't answer, I'd check these in order:

1. Is the list behind it empty? kubectl get endpointslices -l kubernetes.io/service-name=storefront
2. If it is, do the labels match? kubectl get pods --show-labels, and compare with the selector.
3. Are the pods ready? A pod that isn't ready can still be in the list, so add -o yaml to the command in step 1 and look at conditions.ready for each address.
4. Is it the right port? If the app listens on another port than the Service's port, set targetPort.
5. Is it the right namespace? From another namespace, use storefront.seatlark. The short name could reach a different Service with the same name in the caller's namespace.

Questions you might have

How is this different from Docker Compose? In Compose each service gets a name on the network by itself. In Kubernetes a Deployment doesn't, so you write a Service for each part that others need to reach. The Service then spreads the connections over all copies of that part, and those copies can run on different servers.

Does every Deployment need a Service? Only if something connects to it. In this shop ticket-worker has none, because it only takes jobs from the queue and nothing calls it.

Can I use pod names instead? The name of a Deployment's pod normally isn't a DNS name other pods can use, and its replacement gets a different name anyway. A StatefulSet with a headless Service gives each pod a DNS name that stays the same, made of the pod name and the Service name. Databases with several copies use that.

Is a Service a load balancer? A normal ClusterIP Service spreads connections over the pods, and that's all. It doesn't retry a failed request, and it doesn't look at the HTTP request itself. A headless Service leaves the choice of pod to the client, and ExternalName only gives another DNS name.

How I ran this

I used a cluster I had set up with kubeadm. It has three servers on AWS with Ubuntu 24.04. The control plane is a t3.medium, and the two workers are t3.large. A fourth t3.medium built the images and ran the registry. I tested this on November 7, 2026.

These are the versions:

• Kubernetes 1.37.2, set up with kubeadm
• containerd 2.2.1, from Ubuntu's packages
• Cilium 1.20.3, next to kube-proxy
• CoreDNS 1.14.6
• The registry: distribution 3.1.2, the image registry:3

What I'm building

Delegate tasks. Get software.

Give Vroni a GitHub issue, bug report, spec, or rough idea. It reads the repo, plans the change, writes code, runs checks, and works toward a review-ready pull request.

Take a look at vroni.com