# From a bound pod to a running process

Alfonso Campodonico · 2026-11-11 · https://alfonsocampodonico.com/posts/from-a-bound-pod-to-a-running-process

> Kubernetes, what a node's agent does with a pod once the scheduler hands it over.

## Why follow one pod

The scheduler writes a node's name into a pod, and that's all it does. This post follows what happens next on that node, until the pod serves traffic, and then what happens when it misbehaves.

This is part of a series. In the opener, [What happens when a server dies](/posts/what-happens-when-a-server-dies), this stage got one paragraph. That paragraph hides most of the incidents I get paged for: slow starts, restart loops, `OOMKilled`, evicted pods. They all happen on one node, with no controller involved.

The short version: the kubelet turns a pod into a few Linux processes in a few cgroups, and nearly everything that goes wrong later was decided by the numbers you wrote in `resources` and in the probes.

## How it works, in one minute

Each server runs an agent called the kubelet.

1. **It notices.** It sees new work assigned to its server.
2. **It builds a home.** It sets up a box with its own network address, and fetches the app.
3. **It fences the app in.** Each app gets a budget of processor time and memory.
4. **It asks "are you OK?" on a timer.** The app gets traffic only once it answers.
5. **It cleans up.** It restarts crashed apps, slower each time, and removes the least important work when memory runs out.

It works like a stage manager. The director picks the stage. The stage manager builds the set, checks the actors are ready, and gets someone off when they faint.

## The deep dive

The example is the opener's `web` Deployment, with the parts the kubelet cares about filled in: its own app image instead of nginx, an init container, resources and probes. That's a fuller spec than the opener's, so it's a new ReplicaSet, `web-8b7j`, and the pod bound to node-3 is `web-8b7j-x7k2q`. Here's `web.yaml`:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels: { app: web }
  template:
    metadata:
      labels: { app: web }
    spec:
      initContainers:
        - name: migrate
          image: ghcr.io/example/web:1.4
          command: ["./migrate"]
      containers:
        - name: web
          image: ghcr.io/example/web:1.4
          ports:
            - containerPort: 8080
          resources:
            requests: { cpu: 250m, memory: 256Mi }
            limits: { cpu: 500m, memory: 512Mi }
          readinessProbe:
            httpGet: { path: /ready, port: 8080 }
          livenessProbe:
            httpGet: { path: /healthz, port: 8080 }
```

Defaults are upstream's, and some depend on the version[^versions].

## The kubelet notices: one worker per pod

The kubelet watches the API server for pods whose `spec.nodeName` is its own node. Each one gets a pod worker, a goroutine that runs `syncPod` whenever something about that pod changes. Changes on the node itself come from the pod lifecycle event generator (PLEG), which lists what's running every second and compares.

## Admission: the node gets a veto

The kubelet checks the pod again against its own view: allocatable CPU and memory for the requests, host ports, node affinity, taints. If a check fails, the pod goes straight to `Failed` with a reason like `OutOfcpu`, and its ReplicaSet makes a new one.

## The sandbox: a pause container and an IP

Everything from here goes through the Container Runtime Interface (CRI), a gRPC API the runtime serves on a local socket[^cri]. The first call is `RunPodSandbox`. The runtime starts a tiny pause container whose only job is to hold the pod's namespaces open. Every container in the pod joins its network namespace, so they share one IP.

Then the runtime, not the kubelet, calls the CNI plugin with `ADD`. The plugin creates a veth pair, puts one end in the namespace, and assigns an address. The kubelet writes it into `status.podIP`.

## The image: pulled by layer

Next is `PullImage`, and `imagePullPolicy` decides whether it happens:

- **IfNotPresent.** Pull only if the image is missing. The default unless the tag is `latest` or left out.
- **Always.** Ask the registry every time. Layers already on the node aren't downloaded again.
- **Never.** Use what's there, or fail with `ErrImageNeverPull`.

An image is a manifest and a stack of content-addressed layers, and only the missing layers are downloaded. A failed pull shows as `ErrImagePull`, then `ImagePullBackOff`.

## Init containers, one at a time

`migrate` runs first: `CreateContainer`, `StartContainer`, then wait for exit code 0. Only then does the next container start[^sidecars].

## Create, start, and the cgroups

Then the same two calls for `web`. `CreateContainer` carries the resource numbers, and the runtime writes them as cgroup v2 files[^cgroups]:

- **CPU request → `cpu.weight`.** It only matters when the CPU is contended. Then the weights split it.
- **CPU limit → `cpu.max`.** 500m becomes `50000 100000`: 50 ms of CPU time per 100 ms period. Use it up early, and the process waits. That's throttling.
- **Memory limit → `memory.max`.** Go over it and the kernel's OOM killer kills the process.
- **Memory request → nothing, by default.** It matters to the scheduler, to eviction and to the OOM score.

## Ready is a probe, not a process

The readiness probe runs when the container starts, then every `periodSeconds`, 10 by default. Ready turns true on the first success, not when the app starts listening. That's the last line of the opener's recovery: Ready flips, the EndpointSlice controller adds the pod's IP, and traffic arrives.

*Interactive diagram on the page: Getting an app ready to open its doors.*

## Requests, limits and the QoS class

From its requests and limits, every pod gets a QoS class. Think of airline seats: the request is the seat you booked, and the limit is the most you're allowed. Guaranteed is a confirmed seat, Burstable a standard ticket, BestEffort standby:

| | Guaranteed | Burstable | BestEffort |
| --- | --- | --- | --- |
| When | CPU and memory limits on every container, and requests equal them | Anything set, but not Guaranteed | Nothing set |
| `oom_score_adj` | −997 | 3 to 999, lower with a bigger memory request | 1000 |
| Under memory pressure | Never over its request, so it goes after every pod that is | Evicted first if over its request | Always over its request of zero |
| The difference that bites | Its CPU limit still throttles it, even on an idle node | A limit far above the request promises memory the node may not have | New ones are refused while the node is under memory pressure |

CPU and memory fail differently at the limit. Too much CPU makes the app slow, but it lives. Too much memory kills it:

```kubectl
$ kubectl describe pod web-8b7j-x7k2q
...
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
```

137 is 128 plus 9, the number of `SIGKILL`. The process gets no chance to clean up.

*Interactive diagram on the page: What happens when an app wants more than its maximum.*

## When it crashes: probes and the backoff

Three probes, three jobs:

- **Startup.** Holds off the other two until it passes. If it fails `failureThreshold` times, the container is killed.
- **Liveness.** After three failures in a row, the default, the kubelet kills the container and restarts it.
- **Readiness.** On failure, the pod leaves the EndpointSlices. Nothing restarts.

A crash, an `OOMKilled` or a failed liveness probe all end in a restart, because a Deployment's pods have `restartPolicy: Always`. The first restart is immediate. After that the kubelet waits: 10 seconds, doubling on each crash, capped at 5 minutes, like redialling a number that doesn't answer. While it waits, the status is `CrashLoopBackOff`. Ten minutes of running resets the delay.

The backoff protects the node and whatever the app depends on. The cost is recovery time: if the database comes back during a five-minute wait, the pod finds out when the wait ends.

*Interactive diagram on the page: Why a crashing app waits longer each time.*

## When the node runs short: eviction

Every 10 seconds or so, the kubelet computes `memory.available`: capacity minus the working set. The default hard threshold is `memory.available<100Mi`[^thresholds].

Below it, the kubelet sets the node's `MemoryPressure` condition, which becomes a `node.kubernetes.io/memory-pressure` taint. Then it evicts, one pod at a time, with no grace period. It ranks pods by whether they're over their memory request, then by priority, then by how far over they are. An evicted pod ends `Failed` with reason `Evicted`, and its controller replaces it elsewhere.

If memory goes faster than the kubelet checks, the kernel's OOM killer gets there first, and picks by `oom_score_adj`.

*Interactive diagram on the page: Who gets asked to leave when a server runs out of memory.*

Set requests to what the app really uses, and the kubelet will keep it running through most of what the node throws at it.

[^versions]: The restart delay is 10 s to 300 s by default. With `KubeletCrashLoopBackOffMax`, beta and on by default since 1.35, `crashLoopBackOff.maxContainerRestartPeriod` in the kubelet config lowers the cap per node. The alpha gate `ReduceDefaultCrashLoopBackOffDecay` (1.33) changes the defaults to 1 s and 60 s.
[^cri]: The common runtimes are [containerd](https://containerd.io) and [CRI-O](https://cri-o.io). Both create containers through an OCI runtime such as runc or crun.
[^sidecars]: An init container with `restartPolicy: Always` is a sidecar: it starts in order like the others, but keeps running next to the app. Native sidecars are stable since 1.33.
[^cgroups]: On cgroup v1 the same numbers go to `cpu.shares`, `cpu.cfs_quota_us` and `memory.limit_in_bytes`. The shares-to-weight conversion depends on the OCI runtime: runc 1.3.2 and crun 1.23 and later give one CPU a weight of 100, the cgroup v2 default; older versions give it 39.
[^thresholds]: The other default hard thresholds on Linux are `nodefs.available<10%`, `nodefs.inodesFree<5%`, `imagefs.available<15%` and `imagefs.inodesFree<5%`. Soft thresholds, with a grace period, are off unless you set them.
