Blog Posts

Home /
Blog Posts

Speculative Decoding: How Guessing Tokens You Then Throw Away Speeds Up LLM Inference

Speculative Decoding: How Guessing Tokens You Then Throw Away Speeds Up LLM Inference

Here’s a claim that sounds like it can’t possibly be true: you can use a small, weak model to guess several of a large …

Agent Harnesses: Why the Same Model Scores 34 Points Apart on the Same Benchmark

Agent Harnesses: Why the Same Model Scores 34 Points Apart on the Same Benchmark

Take Claude Sonnet 4.5 — same weights, same API, same day — and run it on SWE-bench Verified under two different scaffolds. Under a …

Context Engineering: Why Contexts Fail, and the Four Moves That Fix Them

Context Engineering: Why Contexts Fail, and the Four Moves That Fix Them

In June 2025, Andrej Karpathy posted a one-line reframe that stuck: “+1 for ‘context engineering’ over ‘prompt …

vLLM Internals: PagedAttention, Continuous Batching, and Where the 2-4x Comes From

vLLM Internals: PagedAttention, Continuous Batching, and Where the 2-4x Comes From

A decode step moves every weight in the model from HBM into the SMs to produce exactly one token per sequence. At batch size 1 that is a …

Scaling Laws: Kaplan, Chinchilla, and Why Nobody Trains Chinchilla-Optimal

Scaling Laws: Kaplan, Chinchilla, and Why Nobody Trains Chinchilla-Optimal

Say you’ve got loss numbers from a 40M-parameter run and a 400M-parameter run, same data, same architecture family, same optimizer. …

Attention From Scratch: How Transformers Replaced a Bottleneck with a Lookup

Attention From Scratch: How Transformers Replaced a Bottleneck with a Lookup

Take a vanilla encoder-decoder RNN doing machine translation. It reads a source sentence token by token, updating one hidden-state vector as …