# What happens when you run kubectl scale

Alfonso Campodonico · 2026-10-24 · https://alfonsocampodonico.com/posts/kubernetes-api-server-and-etcd

> Kubernetes, one kubectl scale, followed through the API server and etcd to everything watching.

## Why follow one write

This post follows one command, `kubectl scale deployment web --replicas=5`, from the terminal into etcd and back out to every component that cares.

It's part of a series. [What happens when a server dies](/posts/what-happens-when-a-server-dies) followed a dead node through the cluster. I wrote that each component "watches the API server for the objects it cares about, and writes back what it did", and left it there. That phrase hides the most important part of Kubernetes, and the part I debug most often.

The short version: nothing in Kubernetes calls anything else. Every component writes to the API server and watches it, only the API server touches etcd, and etcd agrees on every write by majority vote.

## How it works, in one minute

1. **You ask.** Your terminal sends "five copies, not three" to one central service.
2. **It checks you.** It works out who you are and whether you're allowed.
3. **It checks the change.** Plugins can adjust or refuse it, and it must be well formed.
4. **It writes it down.** A database stores it with a number that only goes up.
5. **Everyone listening hears.** Each program that cares gets the new version and does its small job.

It works like a clerk at a locked door: check the badge, read the note, write it in the book with the next page number, and shout the number to everyone waiting outside.

That's the whole story, in under two minutes:

<div class="mt-8 aspect-video overflow-hidden rounded-2xl bg-black shadow-xl">
<iframe src="https://www.youtube-nocookie.com/embed/rM_iFl2k-Ok" title="What happens when you run kubectl scale" class="size-full" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
</div>

*Interactive diagram on the page: One kubectl scale, end to end.*

## The deep dive

The rest of this post takes that command step by step, with the real names, status codes and defaults. The example is the opener's Deployment, `web`, in the `default` namespace.

## The request: a patch to a subresource

`kubectl scale` doesn't send the whole Deployment. It sends a JSON merge patch to its `scale` subresource, a small object that only holds the replica count:

```http
PATCH /apis/apps/v1/namespaces/default/deployments/web/scale
Content-Type: application/merge-patch+json

{"spec":{"replicas":5}}
```

Every request, from you, a controller or a kubelet, goes through the same pipeline: authentication, authorisation, mutating admission, schema validation, validating admission, and only then storage. Each stage says no with its own status code[^more].

## Who are you: authentication

The API server tries each configured authenticator in turn: client certificates, bearer tokens, OIDC, service account tokens. The first that recognises the credentials gives the request a user name and groups.

Credentials that are there but wrong, like an expired token, get `401 Unauthorized`:

```kubectl
error: You must be logged in to the server (Unauthorized)
```

No credentials at all usually isn't a 401. Anonymous requests are on by default, so the request carries on as `system:anonymous` and is stopped at the next stage.

## May you: authorisation

On almost every cluster, the authoriser is RBAC, and its question is precise: may this user `patch` the resource `deployments/scale` in the `apps` group, in `default`? Permission to edit Deployments doesn't cover the subresource unless a Role names it. Without one, the answer is `403 Forbidden`:

```kubectl
Error from server (Forbidden): deployments.apps "web" is forbidden: User "alice" cannot patch resource "deployments/scale" in API group "apps" in the namespace "default"
```

That message names the verb, resource and namespace your missing rule needs.

## Admission, in two halves

Admission is where the cluster's own policy runs, in two halves with schema validation between them:

- **Mutating admission.** Plugins and webhooks can change the object: add defaults, inject a sidecar. A webhook that can't be reached fails the request with `500` by default, because its `failurePolicy` is `Fail`.
- **Schema validation.** A negative count fails here with `422 Unprocessable Entity`. `kubectl scale` never sends one, because kubectl refuses it first, but a raw patch can.
- **Validating admission.** Webhooks and policies can refuse but not change. A webhook picks its own status code, and gets `400 Bad Request` if it sets none.

Validation runs after mutation on purpose, so a mutating webhook can't sneak an invalid object past it.

*Interactive diagram on the page: Six checks before anything changes.*

## Into etcd

Only now does anything touch storage. The API server applies the patch and writes the whole Deployment, as protobuf, under one key:

```etcd
/registry/deployments/default/web
```

etcd keeps one counter for the whole cluster, the revision, and every write moves it up by one. The new revision, the clerk's next page number, becomes the object's `resourceVersion`. Clients are told to treat it as opaque, but underneath it's etcd's revision.

kubectl prints `deployment.apps/web scaled`. Nothing has started yet. All that's happened is a write.

## Two writers at once

Kubernetes doesn't lock objects. Every update says which `resourceVersion` it was based on, and the API server only stores it if that's still current. A stale one gets `409 Conflict`:

```kubectl
Operation cannot be fulfilled on deployments.apps "web": the object has been modified; please apply your changes to the latest version and try again
```

The fix is in the message: read again, reapply your change, retry. Controllers do this constantly, and client-go has `RetryOnConflict` for it. A 409 isn't a fault. It's the system working.

`kubectl scale` never sees one, because its patch carries no `resourceVersion` and the API server applies it to whatever is current. `kubectl replace` sends the whole object with its version, so it can.

*Interactive diagram on the page: Two people, one page.*

## Back out: everyone is watching

Nobody asks the API server what changed. Every component lists the objects it cares about, notes the list's `resourceVersion`, and holds a watch open from there: "send me every change after N". client-go wraps this in an informer, which keeps a local cache and calls the component's code on each change.

Our write sets off a chain:

1. **The Deployment controller** sees `web` change, and sets the ReplicaSet `web-7d4b9c8f6d` to five.
2. **The ReplicaSet controller** counts three pods, and creates two with no node.
3. **The scheduler** binds each new pod to a node.
4. **Each kubelet** sees a pod bound to it, and starts it.

Each step is a write, followed by a watch event. Polling thousands of objects would hammer the API server and still be late. A watch costs almost nothing while nothing changes, and delivers each change in order within milliseconds.

## Compaction, and 410 Gone

etcd keeps old revisions so a watcher can ask for "everything after 41207", but not forever. Every 5 minutes, the API server compacts etcd, throwing away revisions older than the ones it saw on its previous run[^compaction].

A watcher that comes back asking for history that's gone gets `410 Gone`, as an `ERROR` event on the stream:

```log
too old resource version: 41207 (41209)
```

The recovery is the same as at startup: list everything, then watch from the list's version. Informers do it on their own, and ask for bookmark events, empty events that keep an idle watcher's version fresh, to make it rare.

It's the opener's level-triggered design again. A controller that missed events doesn't replay them. It reads the current state and fixes the difference.

*Interactive diagram on the page: Everyone listens for new pages.*

## etcd: majority or nothing

A production etcd runs three or five members, using the Raft consensus algorithm:

- **One leader.** All writes go through it.
- **Heartbeats.** The leader sends one every 100 ms. A follower that hears nothing for 1,000 ms, the default election timeout, starts an election for a new term.
- **Majority.** A write commits once ⌊n/2⌋ + 1 members have it on disk. Electing a leader needs the same majority.

Three members survive losing one. Five survive two. Four still survive only one, which is why etcd clusters have odd sizes.

Without a majority, the leader steps down and no new one can be elected. Writes stop. Running pods keep running, because kubelets don't need etcd to keep a container alive, but nothing new happens: no scaling, no scheduling, no recovery from the next dead node.

*Interactive diagram on the page: Saving a page needs more than half.*

| | 3 members | 5 members |
| --- | --- | --- |
| Majority needed | 2 | 3 |
| Failures tolerated | 1 | 2 |
| Each write waits for | 1 follower | 2 followers |
| Good for | most clusters, one per zone across three zones | surviving two members down at once |
| The difference that bites | one member in maintenance, and the next failure stops all writes | slower writes, and more members to keep healthy |

That's the trip the opener took without saying so: the node lifecycle controller's taint, the EndpointSlice update, the eviction, the new pod, the binding and the kubelet's Ready were all writes through this one door, each heard by whoever was watching.

[^more]: I've left out a few stops. Audit logging records the request, and API Priority and Fairness can answer `429 Too Many Requests` when the server is overloaded.
[^compaction]: The interval is `--etcd-compaction-interval`. The API server also serves most watches from an in-memory watch cache, which holds a limited window of recent events, so a watcher can get 410 Gone from the cache before etcd has compacted anything.
