vLLM and PagedAttention: serving LLMs to many users efficiently
TL;DR
vLLM serves LLMs efficiently thanks to PagedAttention (paging the KV cache into blocks) and continuous batching, with 2–4× higher throughput per the original paper. It's the default for self-hosting for many users, but count the full operating cost.
Running an open model for one user is easy. Serving hundreds of concurrent users on a few expensive GPUs is the real problem: GPUs are fast but usually bottlenecked on memory, not compute. vLLM, born at UC Berkeley, is today's most popular LLM serving engine thanks to an idea borrowed from operating systems: PagedAttention.
The problem: the KV cache wastes memory
When generating text, the model stores the Keys/Values of every previous token (the KV cache) to avoid recomputing them. This cache is large and its length varies per request. Systems before vLLM allocated one contiguous block for the maximum possible length, causing fragmentation and waste, so each GPU served few concurrent requests.
PagedAttention: paging like virtual memory
- The KV cache is split into fixed-size blocks that don't need to be contiguous, like memory pages in an operating system.
- Blocks are allocated as tokens are generated, so waste is limited to roughly the last block.
- Requests sharing a prompt prefix (the same system prompt) can share blocks.
The PagedAttention paper (Kwon et al., SOSP 2023) reported 2–4× higher throughput than the leading systems of the time at the same latency.
Key features
| Feature | Benefit |
|---|---|
| Continuous batching | New requests join the batch as soon as there's room, without waiting for the whole batch |
| Prefix caching | Reuses the KV cache of shared prompt prefixes |
| Quantisation (FP8, INT8, INT4, GPTQ, AWQ…) | Run bigger models on fewer GPUs |
| Speculative decoding | Lower token generation latency |
| Tensor, pipeline and expert parallelism | Split large models across GPUs, MoE support |
| OpenAI-compatible API (and Anthropic Messages) | Existing apps only need a base URL change |
Starting a server
uv pip install vllm
vllm serve <hugging-face-model>
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "<model>", "messages": [{"role":"user","content":"Hello"}]}'vLLM or Ollama?
| Ollama | vLLM | |
|---|---|---|
| Target | One user, a laptop | Many users, GPU servers |
| Priority | Easy install and use | Throughput, GPU efficiency |
| Operations | Almost none | Requires understanding GPUs, memory and tuning |
Risks and caveats
- Self-hosting isn't free: GPUs, operations, monitoring, security updates. Compare total cost with an API before deciding.
- Memory tuning: maximum context length and the share of GPU memory reserved for the cache directly affect concurrency; measure with real load.
- Securing the endpoint: the server has no strong authentication by default; put it behind a gateway with auth.
- A fast-moving project: flags and configuration change between versions; pin versions in production.
How to apply it
- Dev: Use the OpenAI-compatible API so your code isn't tied to the engine; benchmark with a load tool using real prompts.
- Tech Lead: Enable prefix caching when many requests share a long system prompt; monitor p95 latency and throughput.
- PM: Consider self-hosting only for data requirements, large steady traffic, or when an open model is good enough for the task.
Sources & references
- vllm — vLLM project (GitHub)
- Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al. (arXiv)