routing

Cloudflare published an excellent post this week: Cutting our inference bill by 68% by being obsessive about optimization. Their AI code reviewer went from $2.43 to $0.77 per review, and the median review time fell from 4m 09s to 1m 32s. It is a masterclass in squeezing the token economics of a single provider: shared prompt-cache prefixes, fewer LLM calls per session, and relentless model/effort A/B tests.

It is also a study in what optimisation inside the call cannot reach. Every one of their levers operates after the request has already been committed to a provider, a model, and an effort level. Their architecture answered the question of which lane each request deserves by experiment rather than by a decision mechanism. That layer is what terraphim-llm-proxy exists for, and the interesting part is that its answer costs zero tokens.