Tag: inference
All the articles with the tag "inference".
-
The three regimes of agentic KV eviction
On real Claude Code traces under memory pressure, LRU does 60 percent more recompute than a strong offline oracle. Three 2026 papers propose policies to close that gap. Measured against the oracle on shared traces, most do not beat LRU; the one signal that reliably helps, lifecycle retirement, does so only in a middle band of pressure and only by a variable amount. Here is the map, with the error bars.
-
Static batching is slow even when your server is idle
At 0.5 requests per second, a nearly idle server, p95 time-to-first-token was 2 seconds under static batching and 45 milliseconds under continuous batching. Same engine, same traffic, identical throughput. The pathology is not a load phenomenon.
-
HuggingFace's greedy decoding isn't always greedy
My inference engine diverged from HF generate() at token 3. Three independent forward passes outvoted the wrapper. The bug was a repetition penalty shipped in the checkpoint's generation_config, applied even under do_sample=False.