TokTier makes prompt caching pay off sooner

Prompt caching does not remove tokenization. In TokTier's coding-agent traces, the fleet prompt-cache hit rate was 94.1%, yet tokenization reached 10–64% of time-to-first-token as the hit rate approached 0.99. The system keeps token IDs and byte spans per session, retokenizes only a bounded append window, and falls back to the reference tokenizer when it cannot prove a stable boundary. 1

Start in shadow mode: measure continuation and full-rebuild traffic, persist session state, and compare every repaired result with the reference path. The paper reports 16–34% lower median TTFT with vLLM, but the result is workload-specific; two of 17 tokenizer families used the full path, WordPiece stayed on CPU, and no public code release is stated. 1