LLM Serving

vLLM Internals: PagedAttention, Continuous Batching, and Where the 2-4x Comes From

• Lakshmanan LN
• Llm serving, Machine learning, Llms

A decode step moves every weight in the model from HBM into the SMs to produce exactly one token per sequence. At batch size 1 that is a …