vLLM internals and production serving

Serving is where tokens become latency and money. PagedAttention and vLLM are the names to know. The idea: the KV cache is a paged block table, like virtual memory, so you can pack many sequences without giant contiguous reservations. Continuous batching adds new requests as others finish. Tensor parallel splits weights across GPUs when one GPU cannot hold the model.

Throughput and latency fight. A huge batch raises tokens per second and first-token wait. Prefix sharing and cache reuse help when many prompts overlap. You will measure both, not pick a slogan.

Coding: serve a replica

python -m vllm.entrypoints.openai.api_server \
  --model ./checkpoints/tiny-lm \
  --max-model-len 2048 \
  --tensor-parallel-size 1

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model="tiny-lm",
    messages=[{"role": "user", "content": "Say one honest sentence about KV cache."}],
    max_tokens=64,
    temperature=0.2,
)
print(resp.choices[0].message.content)
print(resp.usage)

If vLLM will not load a 10M toy because of a converter gap, serve with a Hugging Face generate loop and still write the same measurements: time to first token, inter-token latency, and tokens per second at batch 1 and batch 8. The replica is the log, not the logo.

What you must be able to explain

• KV cache: at decode, Q is one token, K and V grow. Memory is layers * heads * seq * head_dim * bytes * 2.
• Tensor parallel: split linear maps on the hidden dim, all-reduce as needed. Helps large weights. Hurts if the GPU count is vanity.
• Batching: prefill is compute-heavy, decode is memory-heavy. Mixing them without a scheduler wastes the GPU.

Assignment

• Serve the small checkpoint. Hit it with 1 and 8 concurrent prompts.
• Log TTFT and inter-token latency. No invented p99 from a vendor slide.
• Compute a KV-cache byte estimate for seq 2048 and your layer count.

Opinion checkpoint

Core project 7 starts: an OpenAI-compatible vLLM server and a latency log.

Next: Triton and hand-rolled kernels.