# How a request finds a pod

Alfonso Campodonico · 2026-11-12 · https://alfonsocampodonico.com/posts/how-a-request-finds-a-pod

> Kubernetes, one `curl http://web`, followed through DNS, iptables and a hop to another server.

## Why follow one request

This post follows one HTTP request from one pod to another, across two servers, and names every piece that touches it.

It's part of a series. The opener, [What happens when a server dies](/posts/what-happens-when-a-server-dies), had a step called "traffic moves first": a node goes quiet, and within about a minute requests stop going to its pods. I skipped over how. It's most of what I debug, because "the service is down" almost always ends in one of the places below.

The short version: a Service is a name and a virtual IP, and nothing sits behind either. Every node keeps its own packet rules, which turn the virtual IP into a real pod's address before the packet leaves.

## How it works, in one minute

1. **A name.** The app asks the cluster's phone book for "web".
2. **A stand-in.** The answer is an address no machine owns.
3. **A list.** Kubernetes tracks which copies of web pass their check-ups.
4. **A swap.** Every server rewrites that address to one healthy copy, picked by a row of coin flips.
5. **One long call.** The rest of the conversation goes to the same copy.

It works like a switchboard, except every phone in the building has its own copy.

## The deep dive

The example throughout: a pod called `client` in the `default` namespace, on node-1, runs `curl http://web`, and the request ends on a pod on node-2. Defaults are upstream Kubernetes.

## The setup: a Service and three pods

The app is the opener's Deployment: `web`, three pods of `nginx:1.27` on port 80, one per node. They're web-7f9c-k2xpl on node-1 (10.244.1.5), web-7f9c-m8vqd on node-2 (10.244.2.8) and web-7f9c-r4tzn on node-3 (10.244.3.4). In front of them sits a Service:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: default
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 80
```

With no `type`, it's a ClusterIP Service. The API server gives it an address from the service range, 10.96.0.42 here. Nothing starts: no process, no container.

## The name: DNS and the search list

The kubelet wrote the pod's `/etc/resolv.conf` when it started:

```kubectl
$ kubectl exec client -- cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5
```

The nameserver is CoreDNS, behind a Service still called `kube-dns` for compatibility. `ndots:5` means a name with fewer than five dots gets the search domains first.

`web` has no dots, so the resolver asks for `web.default.svc.cluster.local`, and CoreDNS answers 10.96.0.42. `api.example.com` has two, so it gets NXDOMAIN once per search domain before the name as typed goes upstream. Each name gets an A and an AAAA query, and each query a reply: 8 queries and 16 packets for one lookup[^resolvers]. On a busy cluster, these misses are often most of CoreDNS's load[^ndots].

*Interactive diagram on the page: Looking up a name, one guess at a time.*

## The address: a ClusterIP nothing listens on

curl opens a TCP connection to 10.96.0.42:80. In the default mode, no interface has that address[^ipvs] and no process binds to it. It exists only as a match in packet rules, built from a list.

## The list: EndpointSlices and readiness

The EndpointSlice controller watches each Service and the pods its selector matches. It writes EndpointSlice objects labelled `kubernetes.io/service-name: web`, with up to 100 endpoints each:

```yaml
endpoints:
  - addresses: ["10.244.2.8"]
    conditions: { ready: true, serving: true, terminating: false }
    nodeName: node-2
    targetRef: { kind: Pod, name: web-7f9c-m8vqd }
```

`ready` means the pod's containers have started and, if it has one, its readiness probe passes. The opener's Deployment has none. Adding `readinessProbe: { httpGet: { path: /, port: 80 } }` changes the pod template, so it rolls out as a new ReplicaSet, web-4h6c. The kubelet then probes every 10 seconds. Three failures in a row (`failureThreshold: 3`) mark it not Ready, and one success marks it Ready again. A pod that starts failing keeps getting traffic for up to 30 seconds, plus the time the change takes to reach every node.

This was the opener's "traffic moves first" step. When node-2 lost power, no probe ran: the node lifecycle controller marked its pods not Ready, and the same chain followed.

*Interactive diagram on the page: When a copy stops answering its check-ups.*

## The rules: kube-proxy writes the coin flips

kube-proxy runs on every node and turns Services and EndpointSlices into kernel rules. In the default iptables mode, three kinds of chain in the `nat` table do the work:

- **KUBE-SERVICES.** One rule per Service port: TCP to 10.96.0.42:80 jumps to `KUBE-SVC-…`.
- **KUBE-SVC-.** One rule per ready endpoint, tried in order.
- **KUBE-SEP-.** One per endpoint, ending in `DNAT --to-destination 10.244.2.8:80`.

iptables can't pick from a list, but it can match at random with `-m statistic --mode random --probability`. With three endpoints, the first rule is a coin flip that matches 1 in 3, the second 1 in 2 of what's left, and the last takes everyone left. Each pod gets exactly a third. With no ready endpoints, a REJECT rule refuses the connection, and curl reports `Failed to connect to web port 80 after 0 ms: Couldn't connect to server`[^curl].

*Interactive diagram on the page: Picking a copy at random, fairly.*

KUBE-SERVICES is walked rule by rule, which shows with tens of thousands of Services. IPVS mode uses hash tables, and was deprecated in 1.35. nftables mode uses maps, has been stable since 1.33, and is now the recommended one[^modes].

## The hop: DNAT, then the pod network

On node-1, the SYN hit KUBE-SERVICES, flipped its coins, and landed in web-7f9c-m8vqd's KUBE-SEP chain. DNAT rewrote the destination to 10.244.2.8:80. The source is still the client pod, 10.244.1.7.

From here it's an ordinary packet between pods, left to the CNI plugin, with one rule: every pod can reach every other pod's IP without NAT. Plugins keep that promise with VXLAN tunnels, plain routes or eBPF[^cni].

nginx replies to 10.244.1.7. Back on node-1, the kernel rewrites the reply's source to 10.96.0.42:80, so curl sees the address it called.

## The connection: conntrack keeps it on one pod

That rewrite is conntrack's job. The nat table only sees a connection's first packet; the kernel records the decision and rewrites every later packet, both ways, to match:

```console
$ conntrack -L -d 10.96.0.42
tcp 6 86397 ESTABLISHED src=10.244.1.7 dst=10.96.0.42 sport=51234 dport=80 src=10.244.2.8 dst=10.244.1.7 sport=80 dport=51234 [ASSURED]
```

So the coins are flipped once per connection, not per request. A keep-alive connection or a gRPC channel is one long call: every request goes to the pod it first landed on, and new pods wait until clients reconnect. That's the most common "the new pods get no traffic" ticket I see.

## From outside: NodePort and LoadBalancer

The other Service types are layers on top:

- **NodePort.** Opens a port from 30000–32767 on every node, feeding the same lottery.
- **LoadBalancer.** Adds a NodePort, and asks the cloud provider's controller for a load balancer in front of it.

`externalTrafficPolicy` decides what a node does next. With `Cluster`, the default, any node picks any pod, and swaps the source for its own IP so the reply comes back through it. With `Local`, a node only picks its own pods and keeps the source. A node with none drops the packet, and its `healthCheckNodePort` answers 503 so the load balancer skips it.

*Interactive diagram on the page: Coming in from outside: any server, or only its own?.*

| | `Cluster` | `Local` |
| --- | --- | --- |
| Pods a node picks | Any ready pod | Only its own |
| Client IP in logs | Lost: node's IP | Kept |
| Extra hop | Often | Never |
| Node with no pod | Forwards it | Drops it |

Each side has a difference that bites. With `Cluster`, rate limits and allowlists keyed on client IP all see a handful of node IPs. With `Local`, the load balancer splits per node, so a node with one pod gets as much traffic as a node with five.

That's the trip: a name that's mostly search domains, an address that's only a rule, and a coin flip on node-1 that the rest of the connection never repeats.

[^resolvers]: This is glibc's behaviour. musl, the C library in Alpine images, has its own resolver with its own rules, so the same lookup can behave differently there.
[^ndots]: Ways to cut the extra lookups: end external names with a dot (`api.example.com.`), lower `ndots` for a pod with `dnsConfig.options`, or run NodeLocal DNSCache so the misses are answered on the node.
[^ipvs]: In IPVS mode, kube-proxy adds every ClusterIP to a dummy interface called `kube-ipvs0`, so the kernel accepts packets for it. Still, no process listens.
[^modes]: IPVS was deprecated in Kubernetes 1.35 ([KEP-5495](https://kubernetes.dev/resources/keps/5495)). nftables went alpha in 1.29, beta in 1.31 and stable in 1.33 ([KEP-3866](https://kubernetes.dev/resources/keps/3866)). iptables is still the default on Linux.
[^curl]: Older curl versions end the same error with "Connection refused".
[^cni]: For example, [Flannel](https://github.com/flannel-io/flannel) uses VXLAN by default, [Calico](https://www.tigera.io/project-calico/) can route pod IPs with BGP, and [Cilium](https://cilium.io) can replace kube-proxy with eBPF programs.
