Overview
By the time we had six internal products calling LLMs, every team had reimplemented retries, streaming parsers, and provider fallbacks — differently and mostly wrong. A provider incident meant six emergency deploys. The gateway consolidates this: one internal API, SSE streaming normalized to a common event shape, per-model fallback chains, and centralized usage accounting per team.
The design center is streaming-aware failover. Non-streaming fallback is trivial; the hard case is a stream that dies 40 tokens in. The gateway buffers the head of each stream briefly and, if a provider fails before first token (the common failure mode), silently retries the next provider in the chain. Mid-stream failures surface an explicit error event with a retry hint rather than pretending the truncated answer was complete — silent truncation was the one failure mode product teams said they could never tolerate.
Architecture
- Hono-based Node service exposing one /v1/chat endpoint; provider adapters translate a canonical request into Anthropic, OpenAI, or Bedrock wire formats.
- Each logical model maps to an ordered fallback chain (e.g. claude-sonnet -> bedrock-claude -> gpt-4.1) defined in config, not code.
- Health scores per provider live in Redis: rolling error rate and p95 first-token latency; a provider breaching thresholds is skipped for a cooldown window (circuit breaker).
- Pre-first-token failures fail over transparently; mid-stream failures emit a typed error event — the gateway never silently truncates.
- Every request logs tokens in/out, cost, latency, and fallback depth to a usage table keyed by team API key.
- Prompts never persist in the gateway — it logs metadata only, which kept the security review to one meeting.
Code Snippets
async function* streamWithFallback(
req: ChatRequest,
chain: ProviderTarget[]
): AsyncGenerator<GatewayEvent> {
let lastError: Error | undefined;
for (const target of chain) {
if (await circuitBreaker.isOpen(target.id)) continue;
try {
const stream = adapters[target.provider].stream(req, target.model);
let firstToken = false;
for await (const event of stream) {
firstToken = true;
yield { ...event, provider: target.id };
}
return; // completed cleanly
} catch (err) {
lastError = err as Error;
await circuitBreaker.recordFailure(target.id);
if (hasYieldedTokens(err)) {
// Mid-stream death: never silently retruncate onto another provider.
yield { type: "error", recoverable: true, message: "stream interrupted" };
return;
}
// Pre-first-token failure: fall through to next target.
}
}
yield { type: "error", recoverable: false, message: String(lastError) };
}const WINDOW_SEC = 60;
const ERROR_THRESHOLD = 0.25;
const COOLDOWN_SEC = 30;
async function recordFailure(providerId: string): Promise<void> {
const key = "cb:" + providerId;
await redis
.multi()
.hincrby(key, "failures", 1)
.hincrby(key, "total", 1)
.expire(key, WINDOW_SEC)
.exec();
const stats = await redis.hgetall(key);
const rate = Number(stats.failures) / Math.max(Number(stats.total), 1);
if (rate > ERROR_THRESHOLD && Number(stats.total) >= 10) {
await redis.set("cb:open:" + providerId, "1", "EX", COOLDOWN_SEC);
}
}Lessons Learned
- Almost all provider failures happen before the first token. Optimizing for pre-stream failover covered ~95% of incidents; mid-stream recovery wasn't worth its complexity.
- Never silently truncate. Product teams unanimously preferred an explicit error event over a plausible-looking half answer — trust is the actual product of a gateway.
- Provider 'compatible' APIs aren't. Tool-call deltas, stop reasons, and token accounting differ subtly across Anthropic, OpenAI, and Bedrock; the adapter layer is where the real work lives.
- Fallback chains in config, not code. Incident response became 'edit a YAML file and watch recovery' instead of an emergency deploy.
- Centralized token accounting changed team behavior more than any optimization — the first per-team cost dashboard triggered three prompt-efficiency refactors in a week.