Speculative Decoding: How Guessing Tokens You Then Throw Away Speeds Up LLM Inference
Here’s a claim that sounds like it can’t possibly be true: you can use a small, weak model to guess several of a large …
Read More
vLLM Internals: PagedAttention, Continuous Batching, and Where the 2-4x Comes From
A decode step moves every weight in the model from HBM into the SMs to produce exactly one token per sequence. At batch size 1 that is a …
Read More