# Why your database pod is stuck

Alfonso Campodonico · 2026-11-13 · https://alfonsocampodonico.com/posts/why-your-database-pod-is-stuck

> Kubernetes, one database pod whose server dies, and why Kubernetes won't bring it back on its own.

## Why follow one disk

This post follows one database disk from the moment it's requested to the moment its server dies, and explains why Kubernetes then leaves the database down.

In the series opener, [What happens when a server dies](/posts/what-happens-when-a-server-dies), a web app recovered on its own. A StatefulSet didn't: its pod sat in Terminating until someone stepped in. I get paged for that row more than any other.

The short version: a disk can only be plugged into one machine at a time, and Kubernetes won't move it until it's sure the old machine has let go, because two databases writing to one disk is worse than no database.

## How it works, in one minute

1. **You ask for storage.** Kubernetes writes the request down as a claim.
2. **A disk gets made.** A storage plugin creates it, ideally once it knows where the app will run.
3. **The disk follows the app.** It's plugged into the app's server and mounted inside the app.
4. **The app keeps its name.** Copy 0 is always copy 0, with the same disk and the same address.
5. **A server dies.** Nobody can confirm copy 0 stopped, so Kubernetes waits until a human says so.

It works like a library book. If the borrower disappears, the library won't lend it again until someone confirms it's back on the shelf.

## The deep dive

The rest of this post traces a MySQL StatefulSet called `db`, with three replicas, from its first claim to the loss of `db-0`'s node.

## The claim and the class

A PersistentVolumeClaim (PVC) asks for a size, an access mode and a class. A StorageClass says how to make the disk:

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: disk.csi.example.com
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
```

The `provisioner` names a CSI driver, the plugin that talks to the real storage system. Kubernetes runs it next to a few standard helpers[^sidecars]. The external-provisioner sees an unbound claim, calls the driver's `CreateVolume`, and creates a PersistentVolume (PV) object for the new disk.

## When the disk gets made

`volumeBindingMode` decides when `CreateVolume` runs. It matters because most block disks live in one zone.

With `Immediate`, the default, the disk is made as soon as the claim exists, in a zone the provisioner picks with no pod to go by. The PV gets node affinity for that zone. If the pod can't run there, it waits forever[^reasons]:

```kubectl
$ kubectl describe pod db-0
...
Events:
  Warning  FailedScheduling  default-scheduler  0/3 nodes are available: 1 node(s) didn't match PersistentVolume's node affinity, 2 node(s) didn't match Pod's node affinity/selector.
```

With `WaitForFirstConsumer`, the scheduler picks a node first, writes it on the claim as `volume.kubernetes.io/selected-node`, and the disk is made in that node's zone. For anything zonal, it's the right choice.

*Interactive diagram on the page: Where should the disk be made?.*

## Binding, attaching, mounting

Three loops then move the disk into the container:

1. **Binding.** The PV controller binds the PV to the claim, and the claim turns Bound.
2. **Attaching.** The attach-detach controller creates a VolumeAttachment, "this disk, on this node". The external-attacher calls `ControllerPublishVolume` to plug it in.
3. **Mounting.** The kubelet calls `NodeStageVolume`, which formats the disk once and mounts it for the node, then `NodePublishVolume`, which bind-mounts it into the pod.

Attach is decided centrally. Mount happens on the node. When the node dies, only the central half is left to ask.

## Access modes: one node, one pod, or many

- **ReadWriteOnce (RWO).** One node. Several pods on that node can share it.
- **ReadWriteOncePod (RWOP).** One pod, cluster-wide, GA since 1.29. The scheduler refuses a second.
- **ReadWriteMany (RWX).** Many nodes. Only storage built for it, like a file share.

The scheduler doesn't enforce RWO. A second pod on another node gets scheduled, and the attach-detach controller refuses the second attach. A file share usually never attaches, so it never gets that check.

*Interactive diagram on the page: Can two apps share one disk?.*

## Identity: db-0 is always db-0

```yaml
apiVersion: v1
kind: Service
metadata:
  name: db
spec:
  clusterIP: None
  selector: { app: db }
  ports:
    - port: 3306
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db
  replicas: 3
  selector:
    matchLabels: { app: db }
  template:
    metadata:
      labels: { app: db }
    spec:
      containers:
        - name: mysql
          image: mysql:8.4
          env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef: { name: db-root, key: password }
          volumeMounts:
            - name: data
              mountPath: /var/lib/mysql
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: fast-ssd
        resources:
          requests:
            storage: 20Gi
```

The pods are `db-0`, `db-1` and `db-2`. The headless Service (`clusterIP: None`) gives each Ready pod its own DNS record, like `db-0.db.default.svc.cluster.local`.

`volumeClaimTemplates` makes one claim per pod: `data-db-0`, `data-db-1`, `data-db-2`. The claims outlive the pods, and a recreated `db-1` gets `data-db-1` back. `persistentVolumeClaimRetentionPolicy`, stable since 1.32, can delete them on scale-down, but the default is `Retain`.

With the default `podManagementPolicy: OrderedReady`, pods are created from 0 upwards, each after the one before is Ready, and removed from the top down. `Parallel` drops the waiting for scaling. Rolling updates still go one pod at a time.

*Interactive diagram on the page: Copies that keep their name and their disk.*

## The node dies

Now node-2, `db-0`'s node, loses power. At 50 seconds it's marked NotReady and tainted unreachable (40 s before 1.32). At 5:50, the taint-eviction controller deletes `db-0`.

Deleting is a request, and the kubelet that should confirm it is dead. So `db-0` sits in Terminating. A Deployment would create a replacement with a new name. A StatefulSet can't: the name `db-0` is taken until the old pod object is gone. It waits, forever if it has to.

The disk is stuck too. The attach-detach controller still sees a pod on node-2 using `data-db-0`, so the VolumeAttachment stays.

## Multi-Attach and the six-minute wait

Suppose the old pod object does go away. The new `db-0` lands on node-3, in the same zone, and its attach fails with what everyone still calls the Multi-Attach error[^multiattach]:

```kubectl
$ kubectl describe pod db-0
...
Events:
  Warning  FailedAttachVolume  attachdetach-controller  Waiting for detach for volume "pvc-4f1c2a9e-7d3b-4b8e-9a51-0c6f2d8e1b77" Volume is already exclusively attached to one node, waiting on detach before it can be attached to another node
```

As far as the controller knows, the disk is still mounted on node-2, and detaching a mounted disk can corrupt it. So it waits for an unmount, up to `maxWaitForUnmountDuration`: six minutes. Then, if the node is still unhealthy, it force-detaches[^forcedetach]. Newer versions let you turn that off with `disable-force-detach-on-timeout`.

## Four ways out

Once node-2 is NotReady:

- **Wait.** If node-2 returns, its kubelet finishes the deletion and lets go of the disk.
- **Force-delete the pod.** `kubectl delete pod db-0 --grace-period=0 --force`. kubectl warns: "Immediate deletion does not wait for confirmation that the running resource has been terminated." The new pod then waits out the six minutes.
- **Delete the Node.** `kubectl delete node node-2`. PodGC deletes pods on a missing node after a 40-second quarantine. The deleted Node's last status still lists the disk as in use, so the detach still waits out the six minutes[^nodedelete].
- **Taint it out of service.** `kubectl taint nodes node-2 node.kubernetes.io/out-of-service=nodeshutdown:NoExecute`. Non-graceful node shutdown, GA since 1.28. PodGC force-deletes the terminating pods, and the volumes detach without the wait. It's the only fix that skips it.

None of the three active fixes checks that the node is really off. If node-2 only lost its network, the old `db-0` is still writing. Pull the disk and you risk torn writes; let anything reach it and you have two primaries.

*Interactive diagram on the page: When the database’s server dies.*

With the defaults, each fix applied at 6:40:

| Fix | Old pod gone | Disk detached | db-0 back |
| --- | --- | --- | --- |
| Wait | If node-2 returns | Cleanly | If node-2 returns |
| Force-delete | At once | Forced, 6 min | ~6 min later |
| Delete Node | PodGC, 40 s | Forced, 6 min later | ~7 min later |
| Taint | PodGC, next 20 s pass | At once | Under a minute |

Each has its catch. Waiting might never end. Force-deleting still waits six minutes for the disk, and its warning is easy to ignore. Deleting the Node throws away its labels and taints, and if the machine returns it registers as a new node. The taint is built for this, but it's only safe if the node really is powered off.

My order: confirm the machine is off, out of band, then taint it out of service.

That's the opener's StatefulSet row in full: Kubernetes keeps one identity, one disk and one writer, and it will wait forever rather than guess, so the guessing is left to you.

[^sidecars]: The usual CSI sidecars are the external-provisioner, external-attacher, external-resizer and external-snapshotter, alongside the driver's controller, plus the node-driver-registrar next to the driver on every node. Drivers that don't need an attach step, like many file share drivers, set `attachRequired: false` in their CSIDriver object and skip VolumeAttachment altogether.
[^forcedetach]: The Kubernetes docs warn that a force detach while a workload is still using the volume violates the CSI spec, which says `ControllerUnpublishVolume` must only be called after `NodeUnstageVolume` and `NodeUnpublishVolume` have succeeded, and may corrupt data.
[^reasons]: Before 1.33, the reason read "node(s) had volume node affinity conflict".
[^multiattach]: Up to 1.34, the message read `Multi-Attach error for volume "pvc-…" Volume is already exclusively attached to one node and can't be attached to another`. 1.35 changed it to the wording above.
[^nodedelete]: Kubernetes's main branch now clears that in-use status when a Node is deleted, so a future release will detach at once. No release does yet.
