# Training with what you serve

Lecture 22 · Sharing the cluster · Distributed Systems I
Paper: HybridFlow (EuroSys 2025)
The page also has slides and an interactive simulator: https://alfonsocampodonico.com/classes/distributed-systems/entrenar-con-lo-que-se-sirve?lang=en
Code: https://github.com/AlfonsoCampodonico/ta050-sistemas-distribuidos/tree/main/repos/post-entrenamiento (`gh repo clone AlfonsoCampodonico/ta050-sistemas-distribuidos`)

> Reinforcement-learning post-training brings the two halves together: rollouts that are served, gradients that are trained, and where to put each one, with HybridFlow.

## 1. Where the two halves meet

Lectures 16 and 17 were about training: splitting a model across many GPUs and surviving failures. Lectures 18 and 19 were about serving: KV cache, continuous batching, router, prefill and decode. Lectures 20 and 21 were about sharing GPUs between models and between jobs. This lecture closes the series with the case where everything shows up at once: **reinforcement-learning post-training** (RL), which turned language models into reasoning models.

Today's paper is **HybridFlow** (Sheng et al., EuroSys 2025), whose open implementation is verl. The companion readings are DeepSeek-R1 (why RL, and GRPO), Kimi k1.5 (partial rollouts and hybrid deployment) and AReaL (asynchronous RL with bounded staleness).

| Lectures | What we do | The state that matters |
| --- | --- | --- |
| 16 and 17 | train: split the model along dimensions | weights, gradients, optimizer state |
| 18 and 19 | serve: continuous batches, router, prefill and decode | the KV cache |
| 20 and 21 | share GPUs between models and between jobs | what occupies each GPU |
| Today | generate, score and update, over and over | the weights, in two layouts, and their **version** |

## 2. The loop

An RL step has three parts:

1. **Generate.** The current model answers each problem in a batch G times. That's decode, the serving workload from Lecture 18.
2. **Score.** A verifier decides which answers are right: the final answer matches, or the code passes the tests.
3. **Update.** A gradient step with those answers: the training workload from Lecture 16.

Then generate again with the new weights. Unlike pre-training, the data is generated by the model being trained: generating the next batch depends on training with the current one.

**Why since 2025.** DeepSeek-R1-Zero was trained with RL alone, using rule-based rewards: accuracy (correct answer, or tests that pass) and format (the reasoning between `<think>` and `</think>`). They don't use a neural reward model because it "may suffer from reward hacking" at scale. On AIME 2024, R1-Zero goes from **15.6% to 71.0%** accuracy with a single answer and reaches **86.7%** by voting across several (January 2025 version; the revised one reports 77.9%). DeepSeek-R1 reaches 79.8%; Kimi k1.5, from the same month, 77.5.

**GRPO.** The policy gradient needs each answer's advantage: whether it was better or worse than expected. PPO estimates it with a critic, a model "typically the same size as the policy". GRPO drops it: it generates G answers to the same problem and uses the group as the baseline,

> Aᵢ = (rᵢ − mean(r₁…r_G)) / std(r₁…r_G)  (with std 0 the advantage is taken as 0)

If the whole group got it right or the whole group got it wrong, the advantage is zero and that problem teaches nothing in that step. The rest is like PPO (ratio between new and old policy, clipped) plus a KL term against the reference policy. The revised version of R1 gives the details: 16 answers per problem, a cap of 32,768 tokens that rises to 65,536 from step 8,200, and 512 H800s for about 198 hours for R1-Zero.

**The models.** HybridFlow describes classic RLHF (PPO) with up to four models and three stages:

| Model | What it does | Stages |
| --- | --- | --- |
| Actor | the policy: generates and learns | generation, training |
| Critic | estimates value (PPO; GRPO doesn't use it) | preparation, training |
| Reference | the original policy, for the KL | preparation |
| Reward | scores; or a rule-based verifier | preparation |

The actor is in two stages with opposite workloads: generating is decode, bound by reading the weights, and wants little tensor parallelism and many replicas; training is bound by compute and optimizer memory, and wants to split the model. **Same model, two layouts.**

**Answers get longer.** In R1-Zero the average answer length grows throughout RL (figure 3): the model learns on its own to think more. Kimi k1.5 does RL with a 128k-token window. The loop's serving workload isn't fixed: it grows, and its tail gets longer.

## 3. The rollout dominates

**The long tail.** Answer lengths have a long tail. A synchronous step can't train until the last answer in the batch finishes. In the demo (`go run ./cmd/sim cola`, 256 answers on 8 GPUs, up to 64 per GPU):

| Policy | Mean | p50 | p99 | Max | 1st GPU finishes | Last | GPU idle |
| --- | --- | --- | --- | --- | --- | --- | --- |
| At the start | 353 | 305 | 1,165 | 1,388 | 16 s | 30 s | 86% |
| Already trained | 3,822 | 3,478 | 8,192 | 8,192 | 178 s | 192 s | 75% |

"Idle" counts the empty slots in the batch: from Lecture 18, a decode iteration costs almost the same with one sequence as with 64. A GPU with one long answer and 63 empty slots is busy and produces 2% of what it could. AReaL puts it this way: generation has to wait for the longest output in the batch before updating, and that underutilizes the GPUs.

**It's the barrier from Lecture 16.** In synchronous data parallelism, a straggler slows down everyone's all-reduce; here, one long answer slows down everyone's step. The fixes look alike: don't wait for the slowest, bound how far ahead the fast ones can get, or split up the slow one's work.

**How much each part weighs** (HybridFlow §2.3 and §8.2):

- Actor training and generation are **58.9%** of RLHF time with HybridFlow.
- In NeMo-Aligner, which generates with the training engine, generation reaches **81.2%** of the step.
- With generation and training on different GPUs, sending the 140 GB of weights of a 70B model can take up to **36.4%** of an iteration (citing OpenRLHF).

## 4. Where each job goes

**Three placements** (HybridFlow §8.3):

| Placement | What it is | Who |
| --- | --- | --- |
| Colocated | all models on the same GPUs, taking turns | DeepSpeed-Chat; the 3D-HybridEngine case |
| Separate | each model on its own group of GPUs | OpenRLHF (Ray, vLLM, DeepSpeed) |
| Split | actor and reference in one group, critic and reward in another | NeMo-Aligner |

For PPO there are 15 possible placements. From 16 to 64 GPUs, putting everything together wins; with 96 to 128 GPUs and 34B models (or 96 GPUs with 13B), splitting wins, and with 13B on 128 GPUs, separating does. Colocated doesn't leave GPUs waiting on others but doesn't overlap stages; separate lets each model use its own parallelism, but each group waits on the other.

**Colocated: the layout transition.** If the actor generates and trains on the same GPUs, you have to change how it's split between one stage and the next. In the paper's notation, training is p-t-d and generation is pg-tg-dg-d, with N = p·t·d = pg·tg·dg·d. In the figure 7 example, 4 GPUs train with t = 2, d = 2 and generate with tg = 1: each GPU ends up with the whole model. It's worth it: with 16 GPUs and training at t = 8, generating with tg = 2 (7B) or tg = 4 (13B) cuts generation latency by 60.3% and 36.4%.

The transition is an all-gather (the ZeRO collective, Lecture 16), and table 2 compares three ways to do it:

| Transition | Volume | Duplicated memory | Example t = 8, d = 2 → tg = 2 |
| --- | --- | --- | --- |
| DS-Chat | (tpd − 1)/(tpd) · M | M/(tpd) | 0.94 M |
| HybridFlow-V | (tp − 1)/(tp) · M | M/(tp) | 0.88 M |
| HybridFlow | (tp − tg·pg)/(tg·pg·tp) · M | 0 | 0.38 M |

The **3D-HybridEngine** picks the generation groups so that each GPU already holds part of what it needs, gathers only what's missing and leaves no copies behind. Measured: transition time drops **55.2%** on average (11.7 s) and up to **89.1%** (78.2 s) with 70B models. Overall, HybridFlow reports between **1.53 and 20.57** times more throughput than DeepSpeed-Chat, OpenRLHF and NeMo-Aligner, on 128 A100s (16 machines of 8, 600 GB/s NVLink, 200 Gb/s between machines).

**Separate: one broadcast per step.** With separate pools, the trainer sends the weights to the rollout GPUs after every step. With 140 GB at 25 GB/s (`go run ./cmd/sim ubicacion`):

| Rollout GPUs | Naive (one copy to each) | Ring / pipeline |
| --- | --- | --- |
| 6 | 34 s | 5.9 s |
| 48 | 269 s | 8.2 s |
| 384 | 2,150 s | 27 s |

It's the ring all-reduce argument from Lecture 16: split into chunks and forwarding while receiving, the time barely depends on n.

**Kimi k1.5, a middle case.** Megatron (training) and vLLM (inference) run in separate containers in the same Kubernetes pod, sharing the GPUs. A process they call the checkpoint engine passes the weights from Megatron to vLLM, and across nodes they're moved by **Mooncake** over RDMA, the system from Lecture 19. vLLM starts with dummy weights and swaps them out. They report under a minute from training to generation and about ten seconds the other way.

**The demo, all synchronous** (`go run ./cmd/sim ubicacion`, 8 GPUs):

| Placement | s per step | GPU idle | Where the time goes |
| --- | --- | --- | --- |
| Colocated | 217 | 69% | generate 89%, train 10%, transition 1% |
| Separate, 6 + 2 | 279 | 76% | rollout idle 77%, trainer 70% |
| Separate, 4 + 4 | 237 | 71% | rollout idle 60%, trainer 82% |

When everything is synchronous, separating only adds waiting: while some generate, the others watch.

## 5. Asynchronous

**The idea.** Let the rollout generate the next batch while the trainer trains on the current one. No pool waits, but the batch that produces v2 was generated by v0: the data has **staleness**. That's off-policy learning: the policy gradient assumes the samples come from the policy being updated, and if they come from an old one it's biased. It's what happened with an asynchronous parameter server, where a worker computed its gradient with weights that had already changed.

**Stale Synchronous Parallel** (Ho et al., NIPS 2013). Each worker has a clock; the fastest and the slowest can't be more than s clocks apart; a worker at clock c is guaranteed to see all updates up to c − s − 1. With s = 0 it's Bulk Synchronous Parallel.

| | Parameter server with SSP | Asynchronous RL (AReaL) |
| --- | --- | --- |
| The fast ones | workers computing gradients | rollout GPUs generating |
| The shared state | the parameters on the server | the trainer's weights |
| The bound | at most s clocks ahead of the slowest | at most η versions ahead |
| With bound 0 | BSP | synchronous RL |

AReaL has a hyperparameter η, the maximum staleness. The controller doesn't let an answer start if ⌊(N_r − 1)/B⌋ > i + η, with N_r the answers requested, B the answers in a batch and i the trainer's current version (equation 3). With η = 0, the paper says, it degenerates into synchronous RL.

**The honest part:** the bound is applied when an answer starts. A long answer that started early finishes late, and gets trained on with more staleness than η. In the demo, with η = 4 the average is 3.6 and the max 10.

**AReaL** (Fu et al., 2025), three pieces:

- **Interruptible rollouts.** When new weights arrive, generation is interrupted, the KV cache computed with the old ones is discarded, recomputed with the new ones, and generation continues. One answer can have pieces from several versions. That alone raises throughput 12% and 17% with 1.5B and 7B models.
- **Decoupled PPO.** It separates the policy that generated the data (behavior) from the one used as the anchor for clipping (proximal, a recent one). If the anchor were the old one, it would pull the new policy towards older, worse versions.
- **No barrier.** Three quarters of the GPUs generate and one quarter trains; η = 8 for math and 4 for code.

Result: up to **2.77×** less training time than its synchronous version on the same GPUs (1.5B: 41.0 h versus 14.8 h), with equal or better final results, on a cluster of 512 H800s and models from 1.5B to 32B.

**What staleness does to learning** (AReaL, table 2, 1.5B, AIME 2024):

| η | Plain PPO | Decoupled |
| --- | --- | --- |
| 0 (synchronous) | 42.0 | 42.0 |
| 1 | 41.8 | 42.1 |
| 2 | 40.0 | 41.8 |
| 4 | 23.3 | 42.2 |
| 8 | 35.7 | 41.0 |
| 16 | 35.8 | 38.7 |
| unbounded | 34.0 | 36.9 |

With the decoupled objective, "moderate staleness (e.g., η ≤ 8) has minimal impact on final performance". The middle column isn't monotonic: these are single runs, and an RL run that destabilizes doesn't always recover the same way. What's robust is the gap between the columns.

**The demo, asynchronous** (`go run ./cmd/sim asinc`, 8 GPUs, 40 steps; 6 generate and 2 train, interruptible and with correction):

| | Time | Ans./s | GPU idle | Mean staleness | Max | vs colocated |
| --- | --- | --- | --- | --- | --- | --- |
| Colocated, synchronous | 8,692 s | 1.18 | 69% | 0 | 0 | 1.00× |
| Separate, synchronous | 11,141 s | 0.92 | 76% | 0 | 0 | 0.78× |
| η = 1 | 5,785 s | 1.77 | 53% | 0.97 | 2 | 1.50× |
| η = 2 | 3,867 s | 2.65 | 33% | 1.92 | 5 | 2.25× |
| η = 4 | 2,901 s | 3.53 | 12% | 3.60 | 10 | 3.00× |
| η = 8 | 2,874 s | 3.56 | 8% | 3.95 | 10 | 3.02× |

**The demo, learning** (`go run ./cmd/sim aprender`). The task: the model decides how much to think; thinking more gets more right, but going past 8,192 tokens scores zero. The optimum is in the middle (median of about 3,300 tokens, reward 0.799).

| η | Staleness | Without correction | With correction |
| --- | --- | --- | --- |
| 0 | 0 | 0.795 | 0.795 |
| 1 | 0.97 | 0.791 | 0.796 |
| 2 | 1.92 | 0.695 | 0.794 |
| 4 | 3.30 | 0.663 | 0.794 |
| 8 | 4.41 | 0.205 | 0.792 |
| unbounded | 7.00 | 0.000 | 0.792 |

Why? A gradient computed with samples from an old policy is, on average, the old policy's gradient. When the old one thought too little, that gradient says "think more" even if the current one is already at the optimum: it overshoots the cap, the answers get cut off and score zero, and if they all score zero GRPO has no signal and doesn't recover. It's the classic instability of delayed gradients. With **importance correction** (each sample is weighted by π_current/π_old, truncated at 2; the same idea as the decoupled objective) every η reaches the same reward, at the cost of a few more steps.

**Partial rollouts (Kimi k1.5).** Kimi is colocated. So that one long answer doesn't hold everything up, it sets a token budget per round: if an answer goes over it, "the unfinished portion is saved to the replay buffer and continued in the next iteration". Only the current piece is computed with the current policy; the earlier ones are reused. That way "no single lengthy trajectory monopolizes the system's resources". It's asynchrony in another form, with the staleness concentrated in the long answers. The paper doesn't publish an ablation with numbers. In the demo (`go run ./cmd/sim parciales`):

| | Time | GPU idle | Staleness | Reward |
| --- | --- | --- | --- | --- |
| Colocated, synchronous | 8,692 s | 69% | 0 | 0.795 |
| Colocated, partial rollouts | 5,121 s | 47% | 0.97 | 0.787 |
| … and with correction | 5,108 s | 48% | 0.97 | 0.795 |
| Separate, η = 4, finishes what it has | 4,251 s | 41% | 2.30 | 0.791 |
| Separate, η = 4, interruptible | 2,901 s | 12% | 3.60 | 0.794 |

## 6. When something fails

| Piece | If it goes down | What you do |
| --- | --- | --- |
| A rollout GPU | half-finished answers are lost | regenerate them: they have no side effects, like a map in MapReduce |
| The verifier | a sandbox that hangs or is slow | timeout and retry; it's a separate service (Kimi describes a code-execution service as part of the reward model) |
| The trainer | the optimizer state is lost | checkpoints, as in Lecture 17 |
| The weights in transit | a rollout is left with an old version | the version travels with each answer: it's detected and bounded |

Synchronous, a rollout GPU that goes down stalls the step like an infinite straggler. Asynchronous, the controller has one worker fewer: less throughput, not a broken barrier.

## 7. Closing the series

| | Training (16–17) | Serving (18–19) | Sharing (20–21) | The RL loop (22) |
| --- | --- | --- | --- | --- |
| What gets split | the model, in 4 dimensions | the KV cache, in blocks and tiers | the GPUs, between models and between jobs | the cluster, between generating and training |
| What gets communicated | gradients and activations | KV cache between prefill and decode | weights, from SSD or network to the GPU | weights, after every step |
| Who you wait for | the straggler, in every all-reduce | the long prefill, in every iteration | the cold start; gathering the gang's GPUs | the longest answer, in every step |
| If something fails | go back to the checkpoint | recompute the request | reload the checkpoint; evict and requeue | regenerate; checkpoint the trainer |

**Takeaways.**

- **Don't wait for the tail.** The longest answer decides the synchronous step; partial or asynchronous rollouts remove that wait.
- **Move the weights well.** Colocated: change layout by gathering only what's missing. Separate: ring broadcast, not one by one.
- **Bound the staleness.** Bounded staleness, like SSP, and a correction so that old data doesn't push too hard.

## 8. Demo, questions and readings

**The repository.** `post-entrenamiento` simulates the loop in virtual time: rollout GPUs with continuous batching (an iteration costs 20 + 0.2·n ms), a trainer, the weights in transit, and a toy problem that actually learns.

| Package | What it has |
| --- | --- |
| `tarea` | the policy that chooses how much to think, the verifier reward and GRPO |
| `pesos` | naive and ring broadcast; the volumes from HybridFlow's table 2 |
| `bucle` | colocated or separate, synchronous or with η, partial and interruptible, with or without correction |

| Command | What it shows |
| --- | --- |
| `go run ./cmd/sim cola` | a step waits for the longest answer |
| `go run ./cmd/sim ubicacion` | colocated vs separate, and moving the weights |
| `go run ./cmd/sim asinc` | bounded staleness: throughput and waiting |
| `go run ./cmd/sim parciales` | partial and interruptible rollouts |
| `go run ./cmd/sim aprender` | what stale data does to learning |
| `go test ./...` | the tests for what the lecture claims |

The tests check that a synchronous step waits for the longest answer and leaves most of the GPU·time idle; that with η = 0 there's no staleness; that more η gives more throughput and more staleness; that asynchronous finishes in less than half the time with the same reward; that without correction η = 8 ruins learning and with correction it doesn't; that partial rollouts and interruptions speed things up; and that answers get longer as the model learns. The *RL loop* simulator shows each GPU's batches, the trainer, the weights in transit and a live η slider.

Exercises in the README: discard stale data instead of using it, PPO with clipping, an adaptive GPU ratio, rollout GPU failures, and dynamic sampling of groups that carry signal.

**Review questions.**

1. Why does an RL step have a serving workload and a training workload? What depends on what? *Generating is decode with the current model; updating is a training step. Training needs the answers, and the next generation needs the new weights.*
2. What does GRPO remove from PPO, and why does it matter for the system? *The critic, typically the size of the policy: the advantage is computed against the group. One less model to train, store and move; in exchange, the trainer needs the complete group.*
3. Why does a synchronous step leave the GPU idle most of the time even though they're all "working"? *The step waits for the longest answer, and decode costs almost the same with one sequence as with 64: GPUs with few answers are busy but produce little.*
4. Colocated or separate: what do you pay to get the weights to where generation happens? *Colocated, the layout transition, an all-gather that can be designed to move little. Separate, a broadcast per step, which has to be ring or pipelined to scale.*
5. What is η in AReaL and what is s in SSP? Why can the measured maximum exceed η? *How many versions generation can get ahead, and how many clocks the fastest worker can. With 0, both are synchronous. The bound is applied when an answer starts, and a long one finishes late.*
6. Without correction, why does staleness make the demo's model overshoot the cap? *Because the gradient from old samples is that of the old policy, which still asked to think more.*
7. What do Kimi's partial rollouts and AReaL's interruptible ones have in common? *An answer ends up made of pieces generated by different versions, and no long answer holds up the rest.*

**Readings.**

- Sheng et al., [HybridFlow: A Flexible and Efficient RLHF Framework](https://arxiv.org/abs/2409.19256), EuroSys 2025, and [verl](https://github.com/volcengine/verl). This is the lecture's paper.
- DeepSeek-AI, [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://arxiv.org/abs/2501.12948), 2025.
- Kimi Team, [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://arxiv.org/abs/2501.12599), 2025.
- Fu et al., [AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning](https://arxiv.org/abs/2505.24298), 2025.
- Hu et al., [OpenRLHF](https://arxiv.org/abs/2405.11143).
- Wu et al., [LlamaRL](https://arxiv.org/abs/2505.24034), 2025.
- Ho et al., More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server, NIPS 2013.

**Closing the series.** In Lecture 16 we split a model to train it; in 18 we served it; in 20 and 21 we shared the GPUs between models and between jobs; in 22 we saw that, for reasoning models, training is serving thousands of times over: the same model, in two layouts, going back and forth, with a version stuck to every piece of data. The questions are still the same: what gets split, what gets communicated, what happens when it fails.

## Run the examples

With Go installed and no dependencies, from the root of the repository:

```sh
cd repos/post-entrenamiento
go test ./...
go run ./cmd/sim cola
go run ./cmd/sim ubicacion
go run ./cmd/sim asinc
go run ./cmd/sim parciales
go run ./cmd/sim aprender
```
