Event-Driven Autoscaling: Scaling Python Workers on Queue Depth, Not CPU

Our queue workers sat at twelve percent CPU while a backlog of two hundred thousand jobs built up behind them. The horizontal pod autoscaler, watching CPU, was completely satisfied. It was right about the metric and wrong about the system, because the workers were waiting on an external API, and waiting does not consume CPU.

This is the default state of most Python worker deployments. The work is I/O-bound, the scaling signal is CPU, and the two are unrelated. Event-driven autoscaling fixes the mismatch by scaling on the thing you actually care about: how much work is waiting and how long it has been waiting.

Scale on the backlog

KEDA is the standard way to do this on Kubernetes. It polls a source — queue length, Kafka consumer lag, a Postgres query, a Prometheus expression — and drives the underlying HPA from that value. The mental model is simple: pick a target backlog per replica, and the controller adds replicas until the backlog divided by replicas is under the target.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: enrichment-worker
spec:
  scaleTargetRef:
    name: enrichment-worker
  minReplicaCount: 0          # scale to zero between bursts
  maxReplicaCount: 40
  pollingInterval: 15
  cooldownPeriod: 300         # wait before scaling to zero; cold starts are not free
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 300   # scale up fast, down slowly
  triggers:
    - type: redis
      metadata:
        address: redis:6379
        listName: queue:enrichment
        listLength: "50"     # target backlog per replica
    - type: prometheus       # a second trigger: age of the oldest job
      metadata:
        serverAddress: http://prometheus:9090
        query: max(queue_oldest_job_seconds{queue="enrichment"})
        threshold: "120"

The second trigger is the one people forget. Backlog length alone is blind to a small queue that is not moving — five jobs stuck for an hour is an incident that no length-based rule will catch. Scaling on the age of the oldest item catches starvation, poison messages, and a dependency that has gone slow rather than down.

Scale-to-zero is a trade, not a free win

Dropping to zero replicas between bursts is genuinely valuable for spiky workloads — nightly batches, webhook floods, report generation. The cost is that the first job after idle pays the full cold start: image pull, interpreter boot, imports, connection pool warm-up. If that is eight seconds and your SLA is five, scale-to-zero is not available to you until you have fixed the start-up path.

• Scale up aggressively, scale down with a long stabilisation window. Flapping costs more than a few idle pods.
• Cap maxReplicaCount against your downstream limits — the database connection ceiling and the third-party rate limit, not just the node pool.
• Add the oldest-item-age trigger. Length-based scaling cannot see a stalled queue.
• Handle SIGTERM properly so scale-down does not abandon in-flight jobs mid-work.
• Keep the image small and imports lazy; on a scale-from-zero path, start-up time is user-visible latency.

Autoscaling on CPU for an I/O-bound worker measures how hard the process is failing to wait efficiently.

The failure worth naming explicitly is the amplification loop: scaling up thirty workers against a dependency that is already struggling, which makes it slower, which grows the backlog, which triggers more replicas. Every scaler needs a ceiling derived from what the downstream can actually absorb. Autoscaling multiplies whatever your workers do — including the harm.