# Why is my pod Pending?

Alfonso Campodonico · 2026-11-10 · https://alfonsocampodonico.com/posts/why-is-my-pod-pending

> Kubernetes, one pod through the scheduler, and how to read the "0/5 nodes are available" it leaves behind.

## Why follow one pod

A pod is Pending until the scheduler finds it a node, and this post follows one pod through that decision, first when it fails and then when it works.

Pending is the status I get asked about most. The scheduler has usually already said exactly why, in one dense line that most people scroll past.

This is part of a series. The opener, [What happens when a server dies](/posts/what-happens-when-a-server-dies), gave the scheduler three steps: filter, score, bind. This post opens them up.

The short version: the scheduler only counts what pods ask for, rules out every node that can't take the pod, scores the rest, and if nothing is left it tells you why, node by node, in the FailedScheduling event.

## How it works, in one minute

1. **It joins a queue.** A new pod waits in line, and more important pods go first.
2. **Its order is read.** The pod says how much processor and memory it needs, and that number is trusted.
3. **Servers get crossed off.** Too full, reserved or in the wrong place: each check crosses some off.
4. **The rest get marked.** Every server left gets marks out of 100, and the best total wins.
5. **Or it waits.** If nothing's left, the pod stays Pending with a note saying why.

It's like a host seating a party: they count the seats you asked for, skip the reserved tables, and pick the best of the rest. If nothing fits, you wait at the bar.

## The deep dive

This follows the opener's `web` Deployment on a cluster of five nodes, with requests added. A changed pod template means a new ReplicaSet, so its 500m pods are `web-5c8d-…`. Then someone raises the request to 3 CPU, more than any node it's allowed on has free. That pod fails first, and then we fix it. The defaults are kube-scheduler's, and some differ between versions[^versions].

## The queue: active, backoff, unschedulable

The scheduler watches for pods with an empty `spec.nodeName` and puts them in a queue that is really three:

- **activeQ.** Pods ready for an attempt, ordered by priority, then age. The scheduler pops one at a time.
- **backoffQ.** Pods that failed recently, waiting out a backoff: 1 second after the first failure, doubling up to 10.
- **unschedulablePods.** Pods waiting for the cluster to change. A new node, a deleted pod or a removed taint moves them back. Otherwise they're retried after five minutes.

## Requests are all it counts

Our pod, `web-6d4b-q8zxm`, asks for this:

```yaml
resources:
  requests:
    cpu: "3"
    memory: 512Mi
  limits:
    memory: 512Mi
```

The scheduler adds up the requests of every pod on a node and compares the total with the node's allocatable: its capacity, minus what's reserved for the system. It never looks at real usage, so a node at 10% CPU can still be full on paper.

Limits are for the kubelet. The scheduler ignores them, except that a container with only a limit gets a request equal to it. A container with no request counts as zero for fitting. NodeResourcesFit scores it as 100m of CPU and 200 MiB, but BalancedAllocation uses the real requests and skips a pod that has none.

## Filters: who can take it

Filter plugins look at one node each and say yes or no. A node is out at the first no. The main ones, in order[^plugins]:

- **TaintToleration.** A NoSchedule or NoExecute taint the pod doesn't tolerate. That's what keeps node-2, tainted `node.kubernetes.io/unreachable`, empty. node-5, a GPU node, is tainted `dedicated=gpu:NoSchedule` for GPU work only.
- **NodeAffinity.** The `nodeSelector` and required node affinity must match the node's labels.
- **NodeResourcesFit.** The requests must fit in what's left of allocatable.
- **VolumeBinding.** A claim bound to a volume in one zone needs a node in that zone.
- **PodTopologySpread.** With `DoNotSchedule`, the matching pods in the node's zone, plus this one, minus the emptiest zone, must not exceed `maxSkew`.

*Interactive diagram on the page: Which tables can seat this party?.*

For our 3-CPU pod, node-2 and node-5 fall to their taints, and the other three have 1, 2.5 and 0.5 CPU free. Nothing's left.

## Scores: who should take it

When more than one node survives, each score plugin gives each node 0 to 100. The scheduler multiplies by the plugin's weight and adds up:

- **NodeResourcesFit, weight 1.** LeastAllocated by default: the share of CPU and memory left free once the pod is placed. MostAllocated flips it, to pack nodes tight.
- **NodeResourcesBalancedAllocation, weight 1.** Prefers nodes where CPU and memory would be used in the same proportion.
- **ImageLocality, weight 1.** Prefers nodes that already have the image, scaled by its size and how many nodes have it.
- **PodTopologySpread, weight 2.** Prefers the node and zone with the fewest matching pods. A Deployment's pod gets a soft default even if it sets none.

Spreading's raw score is a count, so it normalises: the best node gets 100. Ties are broken at random.

*Interactive diagram on the page: Picking the best table.*

## Reserve, Permit, Bind

The scheduler first reserves the winner in its own cache, so the next cycle sees the space as taken. Permit plugins can then approve, deny or hold the pod; the default set holds nothing. That ends the scheduling cycle.

Binding runs on its own. Volumes get bound, then the default binder posts a Binding object, which sets `spec.nodeName`. For a 500m pod, the event reads `Successfully assigned default/web-5c8d-k2x4p to node-3`. If binding fails, the reservation is undone and the pod goes back to the queue.

## Pending, in the scheduler's own words

Our pod got no node, and `kubectl describe pod` says why:

```kubectl
$ kubectl describe pod web-6d4b-q8zxm
...
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  12s   default-scheduler  0/5 nodes are available: 2 node(s) had untolerated taint(s), 3 Insufficient cpu. preemption: 0/5 nodes are available: 2 Preemption is not helpful for scheduling, 3 No preemption victims found for incoming pod.
```

It's a tally. Each rejection reason is counted, the "count reason" strings are sorted as text and joined. A node short of both CPU and memory counts twice, and the two taints fold into one line because the reason no longer names them. The `preemption:` part says whether evicting pods could help: never for a taint, and not here, because nothing has a lower priority.

*Interactive diagram on the page: The party waiting at the bar.*

A cluster change moves the pod out of unschedulablePods. A change to the Deployment replaces the pod with a new one.

## Preemption: making room

A PriorityClass gives pods a number:

```yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: critical
value: 100000
description: "Customer-facing. May evict batch work."
```

Pods that name it in `priorityClassName` get that priority; pods without one get 0, unless a class is `globalDefault`. Higher priority goes earlier in activeQ, and can preempt.

When filtering fails, DefaultPreemption runs. On each node where eviction could help, it removes every lower-priority pod, checks the new pod fits, then puts back as many as it can, most important first. The rest are victims. It picks the node that breaks the fewest PodDisruptionBudgets, then the one whose most important victim is least important, then the lowest total, then the fewest victims.

The pod gets `status.nominatedNodeName`, the victims get their graceful shutdown, 30 seconds by default, and the pod binds on a later attempt. With `preemptionPolicy: Never`, it still jumps the queue but evicts nobody.

*Interactive diagram on the page: Making room for a VIP.*

Most placement rules come in a hard form, a filter, and a soft one, a score:

| | Hard: a filter | Soft: a score |
| --- | --- | --- |
| Taints | `NoSchedule` | `PreferNoSchedule` |
| Node labels | `nodeSelector`, required affinity | preferred affinity |
| Spreading | `DoNotSchedule` | `ScheduleAnyway` |

A hard rule that no node passes leaves the pod Pending, with a FailedScheduling event, and one stale label can keep it there forever. A soft rule never leaves a pod Pending: it always schedules, so it can quietly break the rule you thought you had.

In the opener, the scheduler got one line, skip node-2 and pick node-3, and behind it is a queue, five filters, four scores and a tally that `kubectl describe pod` will always show you.

[^versions]: The backoffs and the five-minute retry are set in the scheduler's configuration, which checks for pods to retry every 30 seconds. Plugin names and weights are those of recent versions. Older versions worded the taint reason differently: 1.33 and earlier named the taint, as in "untolerated taint {dedicated: gpu}", so each taint got its own line. In clusters of more than 100 nodes, the scheduler stops once it has found enough feasible nodes, a share that is by default 50%, shrinking as the cluster grows, to no less than 5%.
[^plugins]: The full default set also includes NodeUnschedulable, NodeName, NodePorts, VolumeRestrictions, NodeVolumeLimits, VolumeZone, InterPodAffinity and DynamicResources (on by default since Dynamic Resource Allocation went GA in 1.34) as filters, and TaintToleration (weight 3), NodeAffinity (2) and InterPodAffinity (2) as scores.
