# Training at frontier scale

Lecture 17 · Training models · Distributed Systems I
Paper: Llama 3, DeepSeek-V3, DiLoCo
The page also has slides and an interactive simulator: https://alfonsocampodonico.com/classes/distributed-systems/entrenamiento-a-escala-de-frontera?lang=en
Code: https://github.com/AlfonsoCampodonico/ta050-sistemas-distribuidos/tree/main/repos/entrenamiento-distribuido (`gh repo clone AlfonsoCampodonico/ta050-sistemas-distribuidos`)

> 4D parallelism, experts, failures as the normal state and DiLoCo: training at the scale of Llama 3 and DeepSeek-V3.

Lecture 16 was Megatron-LM: data, tensors and pipeline over thousands of GPUs. This lecture is about how training is done today, with two cases published in a lot of detail —Meta's Llama 3 and DeepSeek-V3— and what is known about training across more than one datacenter. The same three questions still guide us: what gets split, what has to be communicated and over which link, and what happens when something fails. What's new is that the third one stops being an edge case.

## 1. From Megatron to today

The ideas from Lecture 16 are still the foundation; what changes is the scale and what that scale forces you to do.

|  | Megatron-LM (2021) | Today |
| --- | --- | --- |
| GPUs | 3,072 A100 | 16,384 H100 (Llama 3) |
| Parallelism | data, tensors, pipeline | + context, + experts |
| Precision | 16 bits | FP8 in the large multiplications (DeepSeek-V3) |
| Failures | — | one unexpected interruption every three hours (Llama 3) |
| Where | one cluster | several datacenters (Gemini) |

Every row is a systems row, even precision: it's how many bytes move for each number.

## 2. The scale of a frontier training run

Llama 3 405B was trained on up to **16,384 H100s at once**, in a 24,000-GPU cluster, with 3.8·10²⁵ floating-point operations in total. Each GPU sustained between 380 and 430 TFLOPs in BF16, between 38% and 43% of its peak, and each training step processes 16 million tokens (§3.3, table 4).

The back-of-the-envelope math with those numbers:

```latex
\frac{3.8 \cdot 10^{25}\ \text{FLOPs}}{16{,}384 \times 400 \cdot 10^{12}\ \text{FLOPs/s}} \approx 5.8 \cdot 10^{6}\ \text{s} \approx 67\ \text{days}
```

Not every run used every GPU all the time, but that's the order of magnitude: more than nine weeks of continuous compute. With an unexpected failure every three hours, 67 days means more than 500 of them, and each one stalls the whole cluster.

The 60% of capacity that goes unused is lost to communication, pipeline bubbles and memory movement. Each technique in this lecture wins back a piece of it: at 50% MFU instead of 40%, the same training run would take about 54 days instead of 67.

## 3. 4D parallelism

Llama 3 combines four kinds of parallelism and orders them from most to least communication: **\[TP, CP, PP, DP\]** (§3.3.2). Each GPU has one coordinate per dimension and the first one varies fastest, so it lands on neighboring GPUs.

| Dimension | What it splits | When it communicates | Where it lives |
| --- | --- | --- | --- |
| Tensors (TP) | each matrix | 4 all-reduces per layer and per microbatch | inside the server, NVLink |
| Context (CP) | the sequence | all-gather of K and V at every layer | inside the pod |
| Pipeline (PP) | the layers | activations to the neighboring stage, per microbatch | inside the pod |
| Data (DP, with FSDP) | the batch | weights and gradients, once per step | between pods |

**The network.** The 405B cluster uses RoCE at 400 Gb/s per GPU. 192 racks of 16 GPUs make up a pod of 3,072 GPUs with full bandwidth; eight pods make up the cluster, and above the pod there's 1:7 oversubscription (§3.3.1). That's why data goes on the outside: it syncs once per step and tolerates the slowest tier. Since collectives generate a few very large flows, the library opens 16 flows per GPU pair and E-ECMP spreads them over several paths; in the core they use deep-buffer switches instead of congestion control.

**The configurations** (table 4). The product TP·CP·PP·DP is the number of GPUs, and there are always 16 million tokens per step:

| GPUs | TP | CP | PP | DP | Sequence | TFLOPs/GPU |
| --- | --- | --- | --- | --- | --- | --- |
| 8,192 | 8 | 1 | 16 | 64 | 8,192 | 430 |
| 16,384 | 8 | 1 | 16 | 128 | 8,192 | 400 |
| 16,384 | 8 | 16 | 16 | 8 | 131,072 | 380 |

With sequences 16 times longer, 16 times fewer sequences fit in a step: data drops from 128 to 8 and context goes up from 1 to 16.

**FSDP.** Today's data parallelism is ZeRO implemented in PyTorch: replicas don't store the full state. Llama 3 shards the optimizer state and the gradients, and the weights gathered for the forward pass aren't resharded until the backward: more memory, one fewer all-gather. TorchTitan reimplemented FSDP by sharding each parameter separately on top of distributed tensors, with about 7% less memory per GPU (ICLR 2025).

**Context.** At 128K tokens, the activations of one sequence don't fit on a GPU. The sequence is split into 2·CP chunks and each GPU gets one from the beginning and its mirror from the end, because with the causal mask the chunks at the end have more attention. Instead of passing the keys and values around a ring, Llama 3 does an all-gather before each attention: with grouped-query attention, K and V are small, and attention costs on the order of S² while the all-gather costs S.

**Pipeline.** It's the interleaved schedule from Lecture 16, with bubble (PP−1)/(V·M), plus three tweaks: a tunable number of consecutive microbatches per stage (the batch no longer has to be a multiple of the number of stages), one fewer layer in the first and last stages to make up for the embedding and the output, and asynchronous messages with memory freed as soon as it's no longer needed. With that they trained with 8K tokens without activation checkpointing.

In the repository, `go run ./cmd/demo 4d` works out where each group lands on this network; with the order reversed (`-orden dp,pp,cp,tp`), a tensor group ends up spread across pods.

## 4. Experts: DeepSeek-V3

A dense model uses all of its parameters for every token. A *mixture of experts* (MoE) replaces the feed-forward layer with many experts and a router that picks a few per token. DeepSeek-V3 has **671B parameters in total and 37B active per token**: 256 routed experts plus 1 shared one per layer, and each token goes to 8 routed ones. You pay the memory of a big model and the compute of a small one.

**Expert parallelism (EP).** The experts are spread across GPUs: EP64 over 8 nodes. Each MoE layer does two all-to-alls: one to send each token to the GPUs holding its experts (*dispatch*) and another to bring the results back (*combine*). Unlike the data all-reduce, this traffic depends on which experts the router picked, and it goes over the network at every layer.

**Node-limited routing.** Inside the node there's NVLink at about 160 GB/s; between nodes, InfiniBand at about 50 GB/s, more than three times less. DeepSeek limits each token to experts on **at most 4 nodes**: it first picks the nodes with the best score and then the experts inside them. The token crosses InfiniBand once per destination node and is fanned out over NVLink inside it, so the network cost stays bounded.

The demo `go run ./cmd/demo moe` shows this with 256 experts on 8 nodes: with free routing a token touches 5.29 nodes on average (4.62 remote); with the limit, 3.95 (3.46 remote).

**Balancing without an auxiliary loss.** If the router sends everything to a few experts, those GPUs become the bottleneck. Instead of an auxiliary loss that competes with the model's loss, DeepSeek-V3 adds a bias to each expert that only affects the selection, and raises or lowers it depending on whether the expert was under- or overloaded in the previous step.

**DualPipe.** With EP, the ratio between compute and communication is around 1 to 1: if you wait for the all-to-alls, half the time is lost. DualPipe splits each microbatch into attention, dispatch, MLP and combine, and overlaps the compute of one microbatch with the communication of another, in forward and backward at the same time. It also feeds the pipeline from both ends: the bubble drops to (PP/2 − 1)·(F&B + B − 3W), versus (PP − 1)·(F + B) for 1F1B, at the cost of keeping two copies of the parameters. Communication uses only 20 of the H800's 132 SMs.

**FP8.** They train the multiplications in FP8 with block-wise scales: 1×128 for activations and 128×128 for weights, so one extreme value doesn't ruin the precision of the whole tensor. They accumulate in FP32 every 128 elements. The relative error in the loss stays below 0.25% compared to BF16.

**The cost.** 14.8 trillion tokens on 2,048 H800s, 3.7 days per trillion tokens, **2.788 million GPU hours** (2.664 million for pretraining, plus context extension and post-training), about 5.576 million dollars at 2 dollars an hour. The paper makes clear that this only covers the official run: it doesn't include research, ablations or failed runs. The lesson for this course isn't the number but where it came from: designing the model (MoE, node-limited routing) and the system (DualPipe, FP8) together, around the network they had.

## 5. Failures are the normal state

With thousands of GPUs, something breaking isn't an exception. In 54 days of Llama 3 405B pretraining there were **466 interruptions**: 47 planned and **419 unexpected**. About 78% of the unexpected ones were hardware, and 58.7% came from GPUs (the GPU itself, its HBM3 or SRAM memory, its system processor, silent data corruption). Even so, they kept more than 90% effective training time, with automated diagnosis and only three significant manual interventions (§3.3.4).

**The math.** If failures are independent, the cluster's mean time between failures is a single GPU's divided by the number of GPUs:

> Cluster MTBF = MTBF of one GPU / N

With the Llama 3 data (419 failures in 54 days over 16,384 GPUs) you get an MTBF of about 50,677 hours per GPU, almost six years. With 16K GPUs that's one failure every 3 hours; with 131K, one every 23 minutes.

**How often to save.** Each checkpoint costs C seconds; each failure loses whatever was done since the last checkpoint, plus the restart. Saving often spends time on checkpoints; saving rarely loses a lot of work per failure. The interval that minimizes the loss is Young and Daly's:

> τ\* = √(2·C·M)

where M is the cluster's MTBF. With C = 30 s and M = 3 h, τ\* ≈ 14 minutes.

**Asynchronous checkpoints.** If the checkpoint only copies the state to host memory and the write to storage carries on in the background, C drops a lot and the optimal interval gets shorter. TorchTitan reports between 5 and 15 times less overhead with asynchronous checkpoints.

The demo `go run ./cmd/demo fallas` (and `-gpus 131072` for the second row) simulates this with a 30 s synchronous checkpoint, a 3 s asynchronous one and a 3-minute restart, each at its own optimal interval:

| GPUs | Interval (sync / async) | Synchronous | Asynchronous |
| --- | --- | --- | --- |
| 16,384 | 14 min / 4 min | 91.4% | 96.1% |
| 131,072 | 5 min / 2 min | 71.4% | 82.9% |

At 131K GPUs, even with asynchronous checkpoints you lose almost a fifth of the cluster. That's why the problem at the next scale isn't just bandwidth: it's that a synchronous job, where every step waits for every GPU, stops entirely each time one fails. The next section goes after exactly that.

**For infrastructure.** This is the same as in Kubernetes: you don't stop nodes from failing, you design so that failing is cheap. Detect fast (GPU health checks, NCCL not responding), pull the node, restart from the last checkpoint with spare nodes already warm. Llama 3 also mentions that the cluster's power draw swings by tens of megawatts when all the GPUs wait at the same time, which is a problem for the power grid.

## 6. Syncing less: DiLoCo

Everything so far assumes a very fast network and a synchronous job: at every step, all the data replicas do an all-reduce and wait for the slowest one. That forces the cluster to be in a single building and means one failure stalls everyone.

**DiLoCo** (Douillard et al., 2023) changes the rule. Each replica takes H steps on its own with its own optimizer (AdamW) and only then do they sync. What gets averaged isn't gradients but each replica's change since the shared weights, the *pseudo-gradient*, and it's applied with an outer optimizer with Nesterov momentum. With H = 500 and 8 replicas they match synchronous training while communicating **500 times less**. With H = 1, SGD as the inner optimizer and SGD with rate 1 as the outer one, it's plain data parallelism. The demo uses exactly that: SGD inside and a plain average outside, without momentum.

The demo `go run ./cmd/demo diloco` reproduces it at small scale, with 8 replicas and 400 steps:

| Sync every | Rounds | Floats sent | Loss |
| --- | --- | --- | --- |
| 1 step (plain data) | 400 | 25,600 | 1.00e-04 |
| 10 steps | 40 | 2,560 | 1.00e-04 |
| 50 steps | 8 | 512 | 1.00e-04 |
| 200 steps | 2 | 128 | 1.00e-04 |

This is the best possible case: the model is convex and the replicas see similar data. With LLMs, the paper's result is that it works, but not for free: you have to tune H and the outer optimizer.

**Decoupled DiLoCo** (Google DeepMind, April 2026) takes the idea across several datacenters. Each group of replicas syncs on its own and asynchronously, so a failure at one site doesn't stall the others. They report:

- to sustain 95% utilization across 8 datacenters, the bandwidth needed drops from 198 Gbps to 3.3 Gbps (0.84 Gbps when communicating in int4);
- with simulated failures over 1.2 million chips, 88% goodput versus 58% for elastic data parallelism (27% without elasticity);
- comparable quality: on a 5B dense model trained on 1T tokens, the text evaluation average is 69.8 versus 70.1 for data parallelism without failures, and 69.8 versus 69.7 with simulated failures (table 2).

**Why it matters.** If you don't need to sync every step, the cluster no longer has to be in one place, and power, which today is the real limit on building a one-gigawatt datacenter, can be sourced across several sites. It's the same shift we saw in this course with systems that move from strong to eventual consistency: you accept that the replicas diverge for a while in exchange for communicating less and tolerating failures.

## 7. What about the labs?

What we know in detail comes from the labs that published their systems. For the rest there are hints, and they have to be presented as hints.

| Lab | What it published | What is known |
| --- | --- | --- |
| Meta | the Llama 3 paper | 16K H100s, 4D parallelism, RoCE, failures |
| DeepSeek | the V3 report | MoE, FP8, DualPipe, 2,048 H800s, cost |
| Google | Gemini, DiLoCo | TPUs, several datacenters, syncing less |
| OpenAI | almost nothing | GPT-4.5 on several clusters |
| Anthropic | compute deals | Trainium2 (Project Rainier) and Google TPUs |

- **Google.** The Gemini technical report (2023) says Gemini Ultra was trained on TPUs across several datacenters, with synchronous data parallelism between superpods. The DiLoCo line, up to Decoupled DiLoCo in 2026, is their published work on syncing less.
- **OpenAI.** Publishes very little about its training infrastructure; it said GPT-4.5 was trained across several clusters.
- **Anthropic.** Hasn't published how it trains either. What's public are its compute deals: AWS's Project Rainier, with close to half a million Trainium2 chips, and a deal with Google for up to a million TPUs, announced in October 2025.

**Is it the same everywhere?** Not being published doesn't mean they do something different. The physical constraints are the same: a hierarchical network where inside is fast and outside is slow, a hardware failure rate that grows with scale, and power. It's reasonable to expect similar solutions (splitting along several dimensions ordered by the network, hiding communication, saving state often, syncing less between sites), but we don't know.

**Question for the class.** What changes if the hardware isn't an NVIDIA GPU but a TPU or a Trainium? The links and their speeds change (TPUs, for example, are connected in a torus inside the pod), and with that, how worthwhile each dimension is. The three questions from Lecture 16 stay the same.

## 8. Demo, questions and reading

**The repository.** `entrenamiento-distribuido` adds four demos to the ones from Lecture 16 (allreduce, train and pipeline):

| Command | What it shows |
| --- | --- |
| `go run ./cmd/demo 4d` | where each dimension lands on the Llama 3 network |
| `go run ./cmd/demo moe` | how many nodes a token touches, with and without the limit |
| `go run ./cmd/demo fallas` | failures, checkpoint interval and effective training time |
| `go run ./cmd/demo diloco` | syncing every H steps |

`go test ./...` checks what the lecture claims: the Llama 3 order keeps tensors inside the server, limiting nodes cuts traffic, the Young and Daly interval beats the others, and syncing every 10 steps converges just the same. The *Training at scale* simulator lets you play with failures and 4D in the browser.

New exercises in the README: hot spares and in-memory checkpoints for the failure simulator, and expert balancing with the DeepSeek-V3 bias for the MoE one.

**Review questions.**

1. Why is the order \[TP, CP, PP, DP\] and not the other way around? *Tensors communicates at every layer and microbatch, on the critical path: it goes over NVLink. Data syncs once per step and tolerates the oversubscribed tier between pods.*
2. With 128K sequences, which dimension grows and which one shrinks? *Context goes up from 1 to 16 and data drops from 128 to 8: 16 times fewer sequences fit in the step's 16M tokens, with the same GPUs.*
3. Why is communication all-to-all with experts? *Each token goes to experts on different GPUs, so each pair of GPUs exchanges different data. Limiting to 4 nodes cuts remote sends from 4.6 to 3.5 per token, a 25% drop.*
4. With 65,536 GPUs and 30 s checkpoints, how often should you save? *M ≈ 50,700 h / 65,536 ≈ 0.77 h; τ\* = √(2 · 30 s · 0.77 h) ≈ 7 minutes.*
5. Why does DiLoCo handle failures better? *A failure only stops its own group, which rejoins when it comes back; with synchronous data parallelism it stops everyone.*

**Reading.**

- Grattafiori et al., [The Llama 3 Herd of Models](https://arxiv.org/abs/2407.21783), 2024, §3.3.
- DeepSeek-AI, [DeepSeek-V3 Technical Report](https://arxiv.org/abs/2412.19437), 2024, §3.
- Douillard et al., [DiLoCo: Distributed Low-Communication Training of Language Models](https://arxiv.org/abs/2311.08105), 2023.
- Liang et al., [TorchTitan](https://arxiv.org/abs/2410.06511), ICLR 2025.
- Google DeepMind, Decoupled DiLoCo, April 2026.
- Gemini Team, [Gemini: A Family of Highly Capable Multimodal Models](https://arxiv.org/abs/2312.11805), 2023.

**Next lecture (18).** We move from training to serving: how a model is served, with the vLLM paper.

## Run the examples

With Go installed and no dependencies, from the root of the repository:

```sh
cd repos/entrenamiento-distribuido
go run ./cmd/demo 4d
go run ./cmd/demo moe
go run ./cmd/demo fallas
go run ./cmd/demo diloco
```
