The service was async from top to bottom, ran on Kubernetes, autoscaled beautifully, and fell over at a throughput the boring synchronous version had handled without complaint. Nothing in the code was obviously wrong. The three causes were the ones that always seem to be the causes: something was blocking the event loop, the connection pool was sized per process instead of per fleet, and there was no backpressure anywhere in the request path.
Async Python is genuinely excellent for the workload most cloud services actually have — waiting on other people's networks. It just fails differently from threaded code, and the failure is global rather than local. Here is what I check first, in order.
The event loop is a shared resource
One blocking call does not slow one request. It stalls every coroutine on that worker, which is why the symptom is never 'this endpoint is slow' — it is 'everything is slow at once, and the CPU graph looks fine'. The usual culprits are a synchronous database or HTTP client that someone imported out of habit, a CPU-bound transform like parsing or scoring a large payload, and an SDK that is quietly synchronous underneath an async-looking wrapper.
- Turn on asyncio debug mode in staging; slow-callback warnings point straight at the offender.
- Export an event-loop lag metric and alert on it. It is the single best early indicator of a blocked loop.
- Send CPU-bound work to a bounded executor — never inline, never unbounded.
- Audit your dependencies for sync-under-async wrappers; one requests call in a hot path is enough.
Connection pools multiply by replica count
Pool settings are written per process and paid per fleet. A pool of twenty looks conservative until the horizontal autoscaler runs twenty replicas and the database sees four hundred connections, which is both over its limit and well past the point where more connections make anything faster. The autoscaler then reacts to the resulting latency by adding replicas. That feedback loop is a genuinely expensive way to take down a database.
from concurrent.futures import ProcessPoolExecutor
import asyncio
# The database sees replicas * (pool_size + max_overflow), not pool_size.
# 12 * (8 + 2) = 120 of a 200 connection limit, leaving headroom for
# migrations, admin sessions and a rolling deploy running double replicas.
MAX_REPLICAS = 12
engine = create_async_engine(
DSN,
pool_size=8,
max_overflow=2,
pool_timeout=5, # fail fast instead of queueing forever
pool_recycle=300, # stay under the proxy idle timeout
)
CPU = ProcessPoolExecutor(max_workers=2)
GATE = asyncio.Semaphore(32) # bound in-flight work: backpressure, made explicit
async def handle(request):
async with GATE:
rows = await fetch(request)
loop = asyncio.get_running_loop()
return await loop.run_in_executor(CPU, score, rows)Size the pool from the ceiling downwards: take the database connection limit, subtract headroom for admin work and for a rolling deploy briefly running double replicas, then divide by maximum replica count. If that leaves each process with an uncomfortably small pool, you need a connection pooler in front of the database rather than a bigger number in the config file.
Concurrency without backpressure is just a longer queue
Async makes it trivially cheap to accept work, which is exactly the trap. Ten thousand awaiting coroutines are ten thousand requests whose clients have almost certainly given up. A semaphore around the expensive section, a bounded queue for background work, and a fast 503 when the gate is full will hold p99 far better than unlimited acceptance. The client's timeout is your real SLA, and work that outlives it is waste you are paying to produce.
Cold starts live in the import graph
On serverless and during rolling deploys, startup cost is user-visible latency, and in Python it is dominated by imports. A module that pulls in a data-science stack at the top level costs seconds before a single request is served. Import heavy libraries lazily inside the function that needs them, construct clients and model handles once in a warm singleton rather than per request, and keep the image slim. Measuring time-to-first-successful-request, rather than time-to-container-running, is what makes this visible.
Shutdown is part of the runtime
Every deploy is a shutdown, so if shutdown is unhandled, every deploy is a burst of 502s that you have trained yourself to ignore. Handle SIGTERM, stop accepting new work, let in-flight requests finish inside the grace period, cancel background tasks explicitly and await their cleanup, then close pools. Pair it with a preStop hook so the load balancer stops routing before the process starts draining — without that gap, Kubernetes will keep sending traffic to a pod that is already saying goodbye.
Async does not remove the queue. It moves it somewhere you forgot to put a limit on.
Every one of these failures has the same root: async makes accepting work nearly free, while doing the work costs exactly what it always did. Put an explicit bound on every place work can accumulate — the loop, the pool, the gate, the queue, the grace period — and async Python will hold up under load that a threaded service could not touch.