Tag: batching

All the articles with the tag "batching".

• Static batching is slow even when your server is idle

  20 Jul, 2026

  At 0.5 requests per second, a nearly idle server, p95 time-to-first-token was 2 seconds under static batching and 45 milliseconds under continuous batching. Same engine, same traffic, identical throughput. The pathology is not a load phenomenon.