# What happens when a server dies

Alfonso Campodonico · 2026-10-20 · https://alfonsocampodonico.com/posts/what-happens-when-a-server-dies

> Kubernetes, explained through one broken server and the small loops that bring the app back.

## Why follow one failure

Kubernetes makes one promise: you describe what should be running, and it keeps it running. This post follows that promise through one server's bad day, from the moment it goes quiet to the moment its work runs somewhere else.

I run Kubernetes for a living. Most explanations of it start with a list of nouns: pods, nodes, services, deployments. The nouns make more sense once you've watched them do something, and a dying server is the best demo there is, because Kubernetes fixes it on its own.

This is the first post in a series. Later ones take one stop from this trip and go deeper.

The short version: Kubernetes is a set of small loops. Each one compares what you asked for with what's true, and fixes the difference. A dead server is just a big difference.

## How it works, in one minute

1. **You write down a wish.** "Keep three copies of my app running."
2. **Servers check in.** Each one says "I'm alive" every few seconds.
3. **Small programs keep comparing.** Each watches one thing and closes any gap between the wish and reality.
4. **A server goes quiet.** After about a minute, traffic stops going to it.
5. **The gap gets closed.** A few minutes later, a new copy starts on a healthy server.

It works like a thermostat. You set the temperature once, and it keeps correcting the room.

That's the whole story, in about two minutes:

<div class="mt-8 aspect-video overflow-hidden rounded-2xl bg-black shadow-xl">
<iframe src="https://www.youtube-nocookie.com/embed/Lk0LduIVOHY" title="What happens when a server dies" class="size-full" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
</div>

## The deep dive

The rest of this post takes the same failure one step at a time, with the real names, objects and timeouts. The example: a web app with three copies on three servers, and the second server loses power. Timings are the defaults, and most are flags you can change[^versions].

## The pieces: a control plane and some nodes

A cluster has two halves:

- **The control plane.** The API server is the front desk: every read and write goes through it. Behind it, etcd stores every object, and it's the only source of truth[^ha]. Next to it run the scheduler and the controller manager, one process that holds dozens of small loops called controllers.
- **The nodes.** These are the servers that run your app. Each runs a kubelet, an agent that starts containers through a container runtime[^cri], and a network proxy that routes traffic for services.

Nothing talks to etcd except the API server, and the components rarely call each other. Each one watches the API server for the objects it cares about, and writes back what it did.

*Interactive diagram on the page: One wish, four helpers.*

## The wish: three copies, written down

The app is a Deployment with three replicas:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels: { app: web }
  template:
    metadata:
      labels: { app: web }
    spec:
      containers:
        - name: web
          image: nginx:1.27
```

That's a wish, not a command. Applying it stores an object, and then four loops each do one small thing. The Deployment controller makes a ReplicaSet, the ReplicaSet controller makes three Pod objects, the scheduler gives each pod a node, and each node's kubelet starts its container.

## The loop: compare and fix, forever

Every controller has the same shape. Here's a sketch of the ReplicaSet controller:

```go
for {
    want := rs.Spec.Replicas
    have := countActive(podsOwnedBy(rs))
    if have < want {
        createPods(want - have)
    } else if have > want {
        deletePods(have - want)
    }
    waitForAChange()
}
```

It doesn't react to an event like "a pod died". It looks at the current state and compares. That's called level-triggered, and it's why Kubernetes recovers from things nobody planned for. A missed event, a restarted controller or a stale cache all end the same way: the next pass sees the gap and closes it.

*Interactive diagram on the page: A thermostat for copies.*

## The heartbeat stops

Every 10 seconds, each kubelet renews a small Lease object in the `kube-node-lease` namespace: "node-2 is alive".

When node-2 loses power, the renewals stop. The node lifecycle controller checks every 5 seconds. Once no renewal has arrived for the grace period, 50 seconds (40 before 1.32), it sets the node's Ready condition to Unknown:

```console
$ kubectl get nodes
NAME     STATUS     ROLES    AGE   VERSION
node-1   Ready      <none>   12d   v1.34.1
node-2   NotReady   <none>   12d   v1.34.1
node-3   Ready      <none>   12d   v1.34.1
```

It also taints the node with `node.kubernetes.io/unreachable`, so the scheduler places nothing new there.

*Interactive diagram on the page: What happens when a server goes quiet.*

## Traffic moves first

The pods on node-2 might still be running. Kubernetes can't tell: all it knows is that the node stopped talking. So it does the safe thing first, and marks those pods not Ready.

The EndpointSlice controller removes them from the service's endpoints, the proxy on every node rewrites its rules, and requests stop going to node-2 within seconds. Nothing has been deleted yet. If node-2 comes back now, its pods turn Ready again and nothing else happens.

## Eviction, after five minutes

Every pod gets a default toleration for the unreachable taint: "I can live with this for 300 seconds". Most outages are short, and a node that blipped for a minute shouldn't have its whole workload thrown away.

After 300 seconds, the taint-eviction controller deletes the pods on node-2. Deleting is a request, not a fact. The pods get a deletion timestamp, but the kubelet that should confirm they stopped is the dead one, so they sit in Terminating.

## The ReplicaSet notices

The ReplicaSet controller doesn't count pods that are being deleted. It wants three and counts two, so on its next pass it creates a third Pod object. That pod has no node yet. It's a record that says "this should exist".

## The scheduler picks a node

The scheduler watches for pods with no node. For each one it:

1. **Filters** out nodes that can't take it: node-2 for its taint, and any node without enough free CPU and memory for the pod's requests.
2. **Scores** the rest: spread across nodes and zones, room left over, images already pulled.
3. **Binds** the pod to the winner, node-3, by writing the choice back through the API server.

If no node passes the filters, the pod stays Pending, and its events list the reason for each node.

*Interactive diagram on the page: Finding a seat for the new copy.*

## The kubelet starts it

node-3's kubelet sees a pod bound to it. It pulls the image if it isn't there yet, and creates the pod's sandbox. A CNI plugin gives the pod an IP address, and the container starts.

The kubelet marks the pod Ready once the container is running, or once its readiness probe passes if it has one. The EndpointSlice controller adds it to the service, and the proxies send it traffic. Three copies again.

## The whole recovery

*Interactive diagram on the page: How long recovery takes, and why it waits.*

With the defaults:

| Time | What happens | Who does it |
| --- | --- | --- |
| 0 s | node-2 loses power, and Lease renewals stop | nobody: that's the failure |
| 50 s | Node marked Unknown and tainted; its pods marked not Ready | node lifecycle controller |
| seconds later | Traffic stops going to node-2 | EndpointSlice controller, every node's proxy |
| 5 min 50 s | Pods on node-2 deleted, and stuck in Terminating | taint-eviction controller |
| right after | A replacement Pod object created | ReplicaSet controller |
| a moment later | Pod bound to node-3 | scheduler |
| seconds to minutes | Image pulled, container started, pod Ready | node-3's kubelet |

Users notice for under a minute, while the dead copy still gets traffic. Full capacity is back after about six minutes.

Not every workload recovers like this:

| | Deployment | StatefulSet | DaemonSet |
| --- | --- | --- | --- |
| Replaced after eviction | Yes, right away | Not until the old pod is truly gone | Not evicted: it belongs on that node |
| Why | Copies are interchangeable | Each pod has an identity, and two copies of one identity is worse than none | One copy per node, by design |
| The difference that bites | A volume still attached to the dead node can hold up the new pod | The pod stays stuck until someone deletes the node or marks it out of service | Nothing to recover until the node returns |

That middle column is where on-call engineers earn their pay. Kubernetes won't guess that a silent node is really dead when guessing wrong could mean two databases writing to the same disk.

That's the trip: a minute to notice, five minutes of patience, then a few seconds of loops doing exactly what they always do.

[^versions]: The grace period is `--node-monitor-grace-period` on the controller manager, and the 300 seconds come from `--default-unreachable-toleration-seconds` on the API server, or from a pod's own tolerations.
[^ha]: In production, the control plane runs several copies of everything, and etcd runs as a cluster of three or five members that agree on every write.
[^cri]: Container runtimes speak the Container Runtime Interface (CRI). The common ones are [containerd](https://containerd.io) and [CRI-O](https://cri-o.io).
