# Distributed Systems I

TA050 · DISTRO · FIUBA · https://alfonsocampodonico.com/classes/distributed-systems?lang=en

> Lectures 16 to 22: how a language model is trained across thousands of GPUs, how it is served, how a cluster is shared among many models and jobs, and how the two halves meet in post-training.

Training and serving a large language model is, before it is a machine learning problem, a distributed systems problem. These seven lectures close the course: two on training (data, tensor and pipeline parallelism, and the scale of Llama 3 and DeepSeek-V3), three on serving (vLLM from the inside, what happens outside each replica with Mooncake, and how hundreds of models and thousands of fine-tunes fit on a few GPUs), one on how a GPU cluster is shared among many jobs, and a last one on reinforcement-learning post-training, where training and serving happen at the same time.

Each lecture has an anchor paper, its slides, its notes, and dependency-free Go code to run in class. From lecture 17 on, each also has a simulator in the browser.

The lectures were given in Spanish; this English version is a translation.

## Code

- [ta050-sistemas-distribuidos](https://github.com/AlfonsoCampodonico/ta050-sistemas-distribuidos) · [zip](https://github.com/AlfonsoCampodonico/ta050-sistemas-distribuidos/archive/refs/heads/main.zip) · `gh repo clone AlfonsoCampodonico/ta050-sistemas-distribuidos`

## Lessons

### Training models

- [Lecture 16: Distributed model training](https://alfonsocampodonico.com/classes/distributed-systems/entrenamiento-distribuido.md?lang=en): Data, tensors and pipeline: how model training gets split up, and what each split costs.
- [Lecture 17: Training at frontier scale](https://alfonsocampodonico.com/classes/distributed-systems/entrenamiento-a-escala-de-frontera.md?lang=en): 4D parallelism, experts, failures as the normal state and DiLoCo: training at the scale of Llama 3 and DeepSeek-V3.

### Serving models

- [Lecture 18: How a language model is served](https://alfonsocampodonico.com/classes/distributed-systems/como-se-sirve-un-modelo.md?lang=en): Continuous batching and PagedAttention: what happens inside an inference engine like vLLM.
- [Lecture 19: From the engine to the cluster](https://alfonsocampodonico.com/classes/distributed-systems/del-motor-al-cluster.md?lang=en): Storing, finding, splitting and scaling the KV cache: from one replica to a service, with Mooncake.
- [Lecture 20: Many models, few GPUs](https://alfonsocampodonico.com/classes/distributed-systems/muchos-modelos-pocas-gpus.md?lang=en): Hundreds of models and thousands of fine-tunes on a handful of GPUs: cold starts in seconds with ServerlessLLM and many LoRAs at once with S-LoRA.

### Sharing the cluster

- [Lecture 21: Sharing a GPU cluster](https://alfonsocampodonico.com/classes/distributed-systems/repartir-un-cluster-de-gpus.md?lang=en): Who runs, when and on which GPUs: all or nothing, locality, fragmentation and scheduling without knowing durations, with the Philly trace and Tiresias.
- [Lecture 22: Training with what you serve](https://alfonsocampodonico.com/classes/distributed-systems/entrenar-con-lo-que-se-sirve.md?lang=en): Reinforcement-learning post-training brings the two halves together: rollouts that are served, gradients that are trained, and where to put each one, with HybridFlow.
