# Many models, few GPUs

Lecture 20 · Serving models · Distributed Systems I
Paper: ServerlessLLM (OSDI 2024), S-LoRA (MLSys 2024)
The page also has slides and an interactive simulator: https://alfonsocampodonico.com/classes/distributed-systems/muchos-modelos-pocas-gpus?lang=en
Code: https://github.com/AlfonsoCampodonico/ta050-sistemas-distribuidos/tree/main/repos/servir-tokens (`gh repo clone AlfonsoCampodonico/ta050-sistemas-distribuidos`)

> Hundreds of models and thousands of fine-tunes on a handful of GPUs: cold starts in seconds with ServerlessLLM and many LoRAs at once with S-LoRA.

## 1. From one popular model to many unpopular ones

In Lectures 18 and 19 we served one model: first a single replica from the inside, then many replicas behind a router. A real platform doesn't serve one model: it serves hundreds of open models and thousands of versions fine-tuned by its customers, and almost all of them get a request only every now and then. Giving each one its own GPU is throwing money away; sharing the GPUs means loading the model when a request arrives, and a model is tens of gigabytes.

Today's paper is **ServerlessLLM** (Fu et al., University of Edinburgh, OSDI 2024): how to make that cold start take seconds instead of minutes. Its companion is **S-LoRA** (Sheng et al., Berkeley, MLSys 2024): how to serve thousands of fine-tunes of the same base model on one GPU.

| | Lecture 19: one model | Today: a fleet |
| --- | --- | --- |
| How many models | one, with many replicas | hundreds, almost all idle |
| What moves | the KV cache | the weights: tens of GB |
| The scheduler decides | where the prefix is | where the checkpoint is |
| Fine-tunes | — | thousands of adapters on one base model |

The structure is the one from Lecture 19 —a large piece of state and a scheduler that knows where it is—, but now the state is the weights.

## 2. The problem

**The workload.** There are no public serverless traces for language models, so ServerlessLLM uses the Azure Functions one (Shahrad et al., *Serverless in the Wild*, ATC 2020). With a 5-minute keep-alive, **more than 40%** of functions cold-start on more than a quarter of their requests, and **about 25%** on more than 60%. For a function, a cold start means bringing up a container; for a language model it means fetching the checkpoint, and providers report tens of seconds.

**Serverless for models** (§2.2, figure 1). The customer uploads their checkpoint and pays per request. A **request router** sends each request to a GPU that already has the model; if there isn't one, it asks the **loading scheduler** to start one, and that server downloads the checkpoint, passes it through SSD and DRAM and moves it up to the GPU. SageMaker, Azure, Google, Hugging Face, Together, Replicate, Fireworks and others offer this; KServe is the open version on Kubernetes.

**Statistical multiplexing.** With one GPU per model, each GPU is sized for its model's peak and sits idle the rest of the time. With a shared fleet, the peak of the sum is enough: bursts from different models don't line up. **AlpaServe** (Li et al., OSDI 2023) takes the idea further —splitting a model across several GPUs even when it fits on one, so a burst can use many at once— and reports up to **10×** more requests per second or **6×** burstier traffic with more than 99% of requests within the SLO.

## 3. Anatomy of a cold start

**The checkpoint is huge** (§2.3, in FP16): LLaMA-2-70B 130 GB, DBRX 250 GB, Mixtral-8x22B about 280 GB, Grok-1 over 600 GB. Downloading 130 GB from S3 over a 5 GB/s network takes **at least 26 s**. Generating a token, by contrast, takes under 100 ms.

**The server already has room for it** (§3.1). An 8-GPU server can have up to 4 TB of DRAM, 64 TB of NVMe and 192 TB of SATA. With PCIe 5.0 each GPU has its own link to DRAM: 512 GB/s combined; from NVMe in RAID 0 to DRAM, about 60 GB/s. KServe and Ray Serve used a fraction of the memory and almost none of the SSDs. Storing checkpoints there is cheap (the hardware is already there), scales on its own and is going to get better (a Grace Hopper has 1 TB of CPU memory with 900 GB/s to the HBM).

The alternatives don't cut it (§2.4): keeping GPUs warm is expensive with large models; caching only in DRAM isn't enough for hundreds of GB; and separate storage servers cost money: even at 100 Gb/s the download takes over 20 s, and an AWS cache server with 210 GB and 200 Gb/s costs $16.3 an hour, the same as an 8-GPU server.

| Tier | Capacity in an 8-GPU server | Bandwidth |
| --- | --- | --- |
| GPU (HBM) | whatever the model uses | 512 GB/s from DRAM, across all 8 |
| DRAM | up to 4 TB | ≈ 60 GB/s from NVMe in RAID 0 |
| SSD | 64 TB NVMe, 192 TB SATA | — |
| Remote | no practical limit | ≥ 26 s for 130 GB at 5 GB/s |

**Even from the local SSD, loading takes a while** (§7.2, figure 6a, NVMe in RAID 0 at 12 GB/s, system caches flushed):

| Model | PyTorch | Safetensors | ServerlessLLM |
| --- | --- | --- | --- |
| OPT-2.7B | 3.0 s | 1.8 s | 0.5 s |
| OPT-30B | 34 s | 18.5 s | 4.5 s |
| Falcon-40B | 50 s | 25 s | 6.2 s |
| LLaMA-2-70B | 84 s | 48 s | 10.3 s |

Between 3.6× and 8.2× faster. PyTorch reads tensor by tensor and copies twice (host, then GPU); Safetensors uses mmap, and when cold every page is a page fault (112,000 for LLaMA-2-7B). ServerlessLLM's time depends only on size. With a 1 GB LoRA adapter for LLaMA-70B: 83.5 ms versus 370 ms for Safetensors.

**A format designed for reading** (§4.1). In training, checkpoints are written a lot and read rarely; in serverless they are uploaded once and loaded thousands of times. ServerlessLLM converts them on upload:

- **Per-GPU partitions:** one file per GPU with only the bytes of its tensors, back to back. It is read sequentially, in large blocks.
- **Tensor index:** name → (GPU, offset, size), aligned. Restoring a tensor is base + offset.
- **Loading separately:** a *model manager* allocates the memory and loads the bytes while the inference process builds the model; at the end it hands over the base addresses (CUDA IPC) and each tensor is pointed at its place.

**The multi-tier loader** (§4.2, figure 7). Starting from reading tensor by tensor, each optimization multiplies the throughput: reading in 16 MB chunks, **1.2×** (a third of the tensors are smaller than 1 MB); O_DIRECT, bypassing the kernel page cache, **2.1×**; multiple threads, because an SSD has many channels, **2.3×**; pinned memory, so the copy to the GPU is DMA, **1.4×**; and pipelining across tiers, **1.5×**. Multiplied together, about 12 times. Four CPU cores are enough. DRAM is a pool of fixed-size chunks and each GPU uses its PCIe link in parallel.

The key thing about the pipeline is that it makes loading **predictable**: while one chunk goes up to the GPU, the next one is already in DRAM, so the time is size over the bandwidth of the slowest tier, not the sum.

**Connection to the course.** It's the same storage hierarchy as Mooncake's KV cache (Lecture 19), with different contents. And the format is a layout decision driven by access pattern, like a log or a columnar format.

## 4. Where to start each model

**The scheduler estimates** (§6). To load a model on a server:

> startup time = q + n / b

where q is the server's loading queue (loads go one at a time on purpose, so they don't fight over the SSD and the estimate stays valid), n is the size and b is the bandwidth of the tier where the checkpoint sits on that server, corrected with what it measures as it goes. To move an inference in progress:

> resume time = α · (t_in + t_out) + β

with α and β fitted from measurements

linear in the input and generated tokens; t_out is estimated as duration over time per token, without asking the server. The measured error is bounded at 5 ms on GPU and 40 ms from SSD; cleaning up GPU state after a migration makes it worse (26 ms on average; once, 623 ms across 119 migrations). It's Mooncake's prefix-aware router with a different formula: there, queue plus prefill of what's missing; here, queue plus loading what isn't there.

**The good server is busy** (§5.1, figure 3). S1 has a free GPU, A in DRAM and B on SSD; S2 is running an inference of A and has B in DRAM. A request for B arrives:

| Policy | Model B (arriving) | Model A (running) |
| --- | --- | --- |
| Availability | loads from S1's SSD: slow | unaffected |
| Locality | waits for A to finish on S2 | unaffected; S1 sits idle |
| Preemption (Shepherd) | starts right away on S2, from DRAM | gets cut off: reloads and recomputes on S1 |
| Live migration | starts on S2 after A is moved | continues on S1 with a minimal pause |

Waiting doesn't work because you don't know how long an LLM inference will last: it depends on how many tokens it generates.

**Live migration** (§5.2). Snapshotting the process takes tens of seconds or minutes, and dirty-page migration isn't supported with GPUs. The observation: the state of an inference is its **tokens** (tens to hundreds of KB); the KV cache (1 to tens of GB) is derived and can be recomputed, and recomputing is a prefill: prefilling 1,000 tokens takes about as long as generating 100. So it's done in rounds: the source sends the tokens and keeps generating; the destination recomputes; the next round has far fewer tokens. When the gap is small, the source stops and sends the last bit.

**Connection to Lecture 19.** Mooncake fetches the KV cache instead of recomputing it, because it has 800 Gb/s RDMA and prompts thousands of tokens long. ServerlessLLM has 10 Gb/s Ethernet and the destination is idle. Same question —fetch or recompute—, different answer: it depends on bandwidth versus compute. And it's the pre-copy approach from virtual machine live migration, with a twist: you copy the state that generates everything (the tokens), not the derived state.

**The protocol** (§5.3, figure 4):

1. Load A on the destination, if there's no idle instance.
2. The scheduler asks the source to migrate.
3. The source sends the tokens to the destination and keeps generating.
4. The destination recomputes the KV cache; repeat with whatever is new.
5. The source stops and hands everything back to the **router** marked "migrated".
6. A is unloaded from the source and B is loaded.
7. The router updates the destination in its table and continues the stream. The client never notices.

Failures (§5.4): the source keeps generating throughout, so if the destination fails, the source just carries on. If the source fails before migrating, the migration is aborted and the destination is unloaded; during the migration, the destination discards the KV cache. If the inference finishes on its own midway, the destination is notified and stops recomputing. Nothing is irreversible until step 5.

**The scheduler's state** (§6, §6.3). Which GPUs are busy, which checkpoints are in each server's DRAM and SSD, and their loading queues live in a reliable key-value store (etcd, ZooKeeper). Order matters: when it assigns a load it marks the GPU; when the server confirms, it updates the storage info and only then notifies the router. If the scheduler goes down, the new one reads the state and syncs it with the servers. It handles thousands of loads per second on one server; distributing it is left as future work. The map is a cache of the real state and can be stale (a checkpoint already evicted): the estimate is wrong but the load is still correct.

**Results** (§7.3 and §7.4, 4 servers with 4 A40s, 512 GB of DRAM, a 2 TB NVMe and 10 Gb/s Ethernet; bursty Azure trace):

| | ServerlessLLM | Ray Serve | Ray Serve with SSD cache |
| --- | --- | --- | --- |
| OPT-6.7B, GSM8K, mean startup | 0.8 s | 12.1 s | 8.2 s |
| OPT-30B, GSM8K, mean startup | 7.5 s | 213 s | 199.2 s |
| OPT-30B, requests under 300 s | 89% | | 26% |

With a network 10 times faster, Ray Serve would drop to 3.8 s: still 4.7 times slower. KServe takes 128 s to first token, 114 of them downloading the checkpoint over 1 Gb/s (28 s with the same improvements as Ray Serve). Under high load, the scheduler's p99 is 1.27 times better than Shepherd* (which preempts) and 1.95 times better than random serverless. With one GPU per server ServerlessLLM gets 4 s; Ray Serve with a cache needs four to reach 12. Overall, **10 to 200 times** lower latency.

## 5. Keep-alive

After the last request, how long does the model stay on the GPU? It's a **TTL**: longer means fewer cold starts and more GPUs busy doing nothing, which in the cloud is money. *Serverless in the Wild* proposes an adaptive per-application keep-alive instead of a fixed one; in the ServerlessLLM evaluation, each system's keep-alive equals its loading time.

The right value for a TTL depends on how much a miss costs. With a checkpoint in DRAM and a pipelined loader, a startup costs half a second: you can free the GPU right away and leave the model in DRAM, which is much cheaper. In the repository demo, the fleet also evicts an idle model if its GPU is needed, like an entry in an LRU cache: TTL and LRU together, as in a web cache.

## 6. Many fine-tunes: S-LoRA

**LoRA** (Hu et al., 2021). A fine-tune replaces each matrix W with

> W′ = W + A·B, with A of h × r, B of r × d and r much smaller (8, 16, 64)

and only A and B are trained, typically on the query, key, value and output projections. For a 4096 × 4096 matrix and rank 8 that's 65,000 parameters instead of 16 million. It cuts trainable parameters by orders of magnitude (10,000×, for example) with comparable quality, and there are hundreds of thousands of published adapters. An adapter for a 7B model weighs tens of MB; the model, 13 GB.

**Merge or compute separately** (§4.1).

- **Merge:** add A·B to W. It costs nothing per token, but a batch can only use one adapter, and switching means subtracting one and adding another with the GPU stalled. The naive version, one merged copy per adapter (vLLM-packed), fits fewer than 5 copies on an 80 GB A100.
- **Compute separately:** x·W + x·A·B. The expensive part, x·W, in a single batch with all requests; x·A·B, which is small, separately for each request with its adapter. What you gain from batching is much more than what x·A·B costs. Adapters have different ranks and requests different lengths, so instead of a padded batched GEMM they use custom kernels that gather weights from non-contiguous memory: MBGMM (prefill) and MBGMV (decode), the latter based on the **Punica** kernel (Chen et al., 2023), which with the same idea reports 12× more throughput while adding about 2 ms per token.

It's batching again (Lecture 18): decode is bound by reading the weights, so the more requests share a read of W, the better.

**Unified Paging** (§5.1). All adapters live in host memory; the ones for the current batch are brought to the GPU. Swapping adapters of different sizes in and out fragments memory, like the KV cache before PagedAttention. The observation: a KV cache of S tokens in one layer is S × H; a LoRA weight of rank R is R × H. They share H. So a page is a vector of H, and KV cache and adapters share a single paged pool that uses all the memory not taken by the base weights. Also (§5.2), while an iteration runs they look at the waiting queue and prefetch the adapters for the next batch.

**S-LoRA also does tensor parallelism** (§6): it splits A and B aligned with the base model's Megatron partitioning, and the extra communication is 5(N−1)Br/N versus 2(N−1)Bh/N for the base model, negligible because r ≪ h.

**Results** (table 3, Llama-7B on an 80 GB A100, rank-8 adapters, requests per second):

| Adapters | S-LoRA | vLLM-packed | PEFT |
| --- | --- | --- | --- |
| 5 | 8.05 | 2.04 | 0.88 |
| 100 | 7.99 | out of memory | 0.25 |
| 1,000 | 7.64 | out of memory | — |
| 2,000 | 7.61 | out of memory | — |

Up to **4×** more than vLLM-packed and **30×** more than PEFT. Past about 100 adapters (figure 5) throughput stops dropping: the extra cost depends on the adapters in a batch, bounded by the batch size, not on how many exist. And the honest ablation (figure 9): with a single adapter, merging is better; with more than two, computing separately wins.

S-LoRA also has admission control (§4.3): its *early abort* drops requests that won't meet the SLO anymore (first token within 6 s), like Mooncake's early rejection.

## 7. The course lens

| Today | We already saw it as |
| --- | --- |
| Checkpoints in DRAM, SSD and remote | cache hierarchy; LRU per tier |
| Keep-alive | TTL; the cost of a miss decides the TTL |
| Starting where the checkpoint is | moving compute to the data, like MapReduce |
| The map of what is where | state in etcd or ZooKeeper: consensus |
| Migration in rounds | pre-copy live migration of VMs |
| Rejecting if it won't meet the SLO | admission control (S-LoRA, Mooncake) |
| Shared fleet | statistical multiplexing |

What consistency does the scheduler's map need? If it thinks a checkpoint is in DRAM and it isn't, the load takes longer but is correct: for efficiency, eventual information is enough. What can't happen is the router sending requests to a GPU that doesn't have the model yet, and that's why that notification only goes out after the server confirms.

## 8. Demo, questions and readings

**The repository.** `servir-tokens` adds the `flota` package: a serverless platform in virtual time, event-driven (bursty arrivals, per-tier loads, instances serving up to 8 requests with keep-alive, migration and preemption), plus a separate server with many LoRA adapters. The bandwidths (10 Gb/s network, 6 GB/s SSD, 25 GB/s PCIe) are in the same ballpark as the paper's testbed, not measured.

| Command | What it shows |
| --- | --- |
| `go run ./cmd/sim arranque` | per-tier loads, placement, keep-alive and one GPU per model |
| `go run ./cmd/sim arranque -gpus 2` | with few GPUs: migrate or preempt |
| `go run ./cmd/sim lora` | merged versus mixed batch |
| `go test ./flota` | the tests for what the lecture claims |

**Loading from each tier** (seconds to the GPU):

| | DRAM | SSD | Remote |
| --- | --- | --- | --- |
| 13 GB, naive loader | 1.7 | 9.0 | 19.4 |
| 13 GB, pipelined loader | 0.5 | 2.2 | 10.4 |
| 130 GB, naive loader | 17.3 | 89.6 | 193.6 |
| 130 GB, pipelined loader | 5.2 | 21.7 | 104.0 |

**Where to start** (64 models, 3,996 bursty requests in one hour, 4 servers with 4 GPUs):

| Policy | Cold | Loads from DRAM | From remote | Mean delay | p99 |
| --- | --- | --- | --- | --- | --- |
| Random, naive loader | 62% | 33% | 34% | 24.5 s | 188 s |
| Random | 53% | 31% | 35% | 7.5 s | 80 s |
| Locality-aware | 47% | 70% | 7% | 5.5 s | 83 s |
| Locality + preemption (49) | 50% | 68% | 7% | 6.3 s | 102 s |
| Locality + migration (6) | 48% | 69% | 7% | 5.5 s | 83 s |

The loader matters most; next, starting where the checkpoint is. Preempting makes both the mean and the tail worse. With GPUs to spare, migration barely kicks in; with 2 GPUs per server it migrates 27 times and preempts 114, and the mean delay comes out at 9.4 s versus 14.6 s. In the paper migration matters more because inferences are longer and the GPUs are full. The tail is dominated by bursts, under any policy.

**Keep-alive:**

| Keep-alive | Classic: cold | Delay | With local cache: cold | Delay |
| --- | --- | --- | --- | --- |
| 0 s | 86% | 60 s | 68% | 5.6 s |
| 10 s | 79% | 56 s | 60% | 5.5 s |
| 60 s | 61% | 47 s | 42% | 5.4 s |
| 600 s | 58% | 47 s | 35% | 5.4 s |

Classic: download every time, naive loader, random placement. With a local cache, cold starts drop just the same but the delay barely changes: a miss is cheap, so a long keep-alive no longer pays for itself (at 600 s the GPU sits idle 82% of the time to save 0.2 s). And versus one GPU per model: 64 GPUs busy 4% of the time, versus 16 busy 16% of the time.

**LoRA** (capacity in requests per second, with load to spare):

| Adapters | Merged | Mixed batch | Adapters per batch |
| --- | --- | --- | --- |
| 1 | 21.0 | 18.9 | 1 |
| 2 | 15.7 | 18.9 | 2 |
| 100 | 10.4 | 18.0 | 47 |
| 2,000 | 2.2 | 16.9 | 85 |

With 200 adapters at 10 requests per second, the merged approach handles 3 and the queue grows without bound; the mixed batch handles all 10 with a p99 TTFT of 44 ms. Keeping adapters in the pool brings each one in once: 200 loads versus 1,545, although since fetching one costs under a millisecond, latency barely changes.

The *GPU fleet* simulator shows the 16 GPUs live, with each model's color and which tier each load came from; you can change the policy, the keep-alive and the loader.

Exercises in the README: initial placement by popularity, adaptive keep-alive and prefetching adapters from disk.

**Review questions.**

1. Why, with a pipelined loader, is the load time that of the slowest tier and not the sum? *The tiers work at the same time on different chunks: the flow is limited by the narrowest pipe.*
2. What does ServerlessLLM's scheduler estimate to pick a server? *q + n/b: the server's loading queue, the checkpoint size and the bandwidth of the tier where it sits on that server. For migration, α·(t_in + t_out) + β.*
3. Why does ServerlessLLM send the tokens and Mooncake the KV cache? *It depends on network versus compute. Mooncake has 800 Gb/s RDMA and long prompts; ServerlessLLM has 10 Gb/s Ethernet, KV caches of several gigabytes and an idle destination to recompute on.*
4. If a startup from DRAM costs half a second, what happens to the optimal keep-alive? *It goes down: a miss is cheap, so it pays to free the GPU and leave the model in DRAM. The optimal TTL grows with the cost of a miss.*
5. Why does S-LoRA's throughput barely drop from 100 to 2,000 adapters? *The extra cost depends on the distinct adapters in a batch, bounded by the batch size.*
6. Why is preempting an inference worse than migrating it? *The preempted one loses what it had generated: it reloads the model and recomputes everything. The migrated one keeps generating while the destination catches up, and only stops for the last round.*
7. In what order does the scheduler update its store and notify the router, and why? *It marks the GPU when assigning, updates the storage info when the server confirms, and only then notifies the router: that way it never routes to a model that isn't there yet.*

**Readings.**

- Fu et al., [ServerlessLLM: Low-Latency Serverless Inference for Large Language Models](https://arxiv.org/abs/2401.14351), OSDI 2024. This is the lecture's paper.
- Sheng et al., [S-LoRA: Serving Thousands of Concurrent LoRA Adapters](https://arxiv.org/abs/2311.03285), MLSys 2024.
- Chen et al., [Punica: Multi-Tenant LoRA Serving](https://arxiv.org/abs/2310.18547), 2023.
- Li et al., [AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving](https://arxiv.org/abs/2302.11665), OSDI 2023.
- Shahrad et al., Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider, USENIX ATC 2020.
- Hu et al., [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685), 2021.

**Wrap-up.** Serving many models on few GPUs comes down to three decisions: **bring** the weights closer (the server's DRAM and SSD, with a loader that saturates every tier), **place** each model where its checkpoint already is (and move, rather than cut off, whatever is in the way), and **share** what is common (one base model for thousands of adapters). Lecture 21 continues with how to split a GPU cluster among different jobs.

## Run the examples

With Go installed and no dependencies, from the repository root:

```sh
cd repos/servir-tokens
go run ./cmd/sim arranque
go run ./cmd/sim lora
```
