Draft-Model

Speculative Decoding: How Guessing Tokens You Then Throw Away Speeds Up LLM Inference

• Lakshmanan LN
• Llm serving, Machine learning, Llms

Here’s a claim that sounds like it can’t possibly be true: you can use a small, weak model to guess several of a large …