Three things happen to every LLM feature that stays in production long enough. The provider has a bad hour. The pricing or the rate limits change. And the model behind a name you have been calling for six months behaves slightly differently one morning. None of the three involves you, and all three land on your on-call rotation.
The reaction I want to argue against is the heavy abstraction layer — the interface so generic that you cannot use any provider's actually useful features, written by a team that will support exactly one provider forever. What you want instead is thin and specific: a routing decision per task, pinned versions, an eval suite that gates changes, and a failure policy that degrades rather than amplifies.
Route by task, not by a global default
The common shape is one model constant imported everywhere, which means classification, extraction, summarisation and multi-step reasoning all run on whatever the team picked for the hardest of those jobs. That is the single largest source of avoidable spend in most LLM features I have looked at, and it is also a latency problem: a routing call that gates a user interaction does not need a frontier model to decide between six intents.
Declare the route per task in configuration rather than scattering model names through the code. On one product, moving classification and first-pass summarisation to a small model cut inference spend by about sixty percent with no measurable change in quality. The important half of that sentence is 'no measurable' — it was only knowable because an eval set existed for both tasks. Without one, that change is a guess you make on a Friday and discover was wrong through support tickets.
Fallback that does not make things worse
The naive version is a try/except that retries on a second provider. It fails in four ways, and each one shows up precisely when you are already having a bad day: you double the user's latency because the fallback only starts after a sixty second timeout has expired; every client retries simultaneously and the retry traffic becomes its own outage; the alternate model returns something your downstream schema validation rejects, so the fallback produces an error with extra steps; and the cost of the incident is invisible until the invoice.
- Hedge rather than only failing over. For latency-sensitive calls, start the secondary at the primary's p95 rather than at its timeout, take the first valid response, and cancel the loser. Failing over after a sixty second timeout means the user has already waited sixty seconds before the retry even begins.
- Circuit-break per provider, not per request. Once the error rate for a provider crosses a threshold, route away from it for a cooldown so that requests stop paying failure latency one at a time before doing the useful thing.
- Only fall back within a validated tier. The alternate has to pass the same eval suite and emit the same structured schema. A fallback you have never evaluated is not resilience — it is an untested branch that executes only during incidents.
- Give the fallback a budget. Cap hedges and retries at a small fraction of total requests, say ten percent, so that a provider outage degrades the feature instead of multiplying your bill by the number of providers you configured.
- Make the non-determinism observable. Tag every response with model, version, route reason, whether it hedged, latency, tokens and cost. Without that, 'the outputs were different last Tuesday' is an unanswerable question.
ROUTES = {
'classify': ['small-fast', 'small-alt'],
'extract': ['small-fast', 'frontier'],
'synthesise': ['frontier', 'frontier-alt'],
}
async def complete(task: str, prompt: str, *, hedge_after: float, deadline: float):
candidates = [m for m in ROUTES[task] if breakers[m].allows()]
if not candidates:
raise NoHealthyModel(task)
attempts = [asyncio.create_task(call(candidates[0], prompt, timeout=deadline))]
# Hedge at the primary's p95, not at its timeout. Waiting for the timeout
# before starting the alternate is how a slow provider becomes an outage.
done, pending = await asyncio.wait(attempts, timeout=hedge_after)
if not done and len(candidates) > 1 and budget.allow_hedge():
attempts.append(asyncio.create_task(call(candidates[1], prompt, timeout=deadline)))
done, pending = await asyncio.wait(attempts, return_when=asyncio.FIRST_COMPLETED)
for task_ in pending:
task_.cancel()
result = first_schema_valid(done) # an unparseable win is not a win
breakers[result.model].record(ok=True)
emit('llm.call', route=task, model=result.model, version=result.version,
hedged=len(attempts) > 1, latency_ms=result.latency_ms,
cost_usd=result.cost_usd)
return resultPin the version and treat a model change as a dependency upgrade
Where a provider offers dated or pinned identifiers, use them. An alias that silently moves is an unannounced deploy into the middle of your product, made by somebody who has never seen your prompts. You would not let a library upgrade itself in production on the vendor's schedule, and a model is a far more behaviourally sensitive dependency than most libraries.
Then make the upgrade a real process, because it is one. Run the eval suite against the candidate. Shadow it on a sample of live traffic and diff the outputs on the metrics that matter for the task — schema validity, groundedness, refusal rate, latency, cost — not on vibes from ten examples. Roll it forward behind a flag with the old version one config change away. The whole loop takes an afternoon once it exists, and it is the difference between adopting a better model deliberately and finding out about it from a customer.
A fallback path you have never evaluated is not resilience. It is an untested branch that only ever executes during an incident.
None of this is large. A route table in config, pinned versions, an eval suite in CI, a circuit breaker, a budgeted hedge, and structured telemetry on every call — a few hundred lines in total, and most of it is the boring reliability engineering you would apply to any third-party dependency in the request path. Skip it and your product's availability is defined, exactly and entirely, by somebody else's status page.