The uncomfortable truth about most Kubernetes bills is that you're paying for fear. Fear-driven resource requests, fear of spot interruptions, fear of scaling down. When we audited a production cluster running a Laravel-and-Node SaaS workload, average CPU utilization was 11%. We were paying for nine idle cores out of every ten. Over a quarter, we cut the compute bill by 55% with no reliability regression — and every step was boring.
Requests are a bet; most teams over-bet
Requests drive scheduling and therefore cost: the scheduler reserves what you request, not what you use. Every team sets requests once, during an incident or a guess, and never revisits them. Start by measuring actual P95 usage per workload over two weeks (kube-prometheus makes this a one-query job), then set CPU requests near P95 usage and let limits breathe. For CPU, I set requests honestly and either omit limits or set them high — CPU is compressible, and CPU limits mostly buy you throttling artifacts. Memory is not compressible: set memory requests equal to limits and size them from observed peaks plus headroom, because the OOM killer does not negotiate.
resources:
requests:
cpu: 250m # P95 observed: 210m. Was 2000m "just in case".
memory: 768Mi # peak observed: 610Mi + headroom
limits:
memory: 768Mi # memory: requests == limits, no surprises
# no cpu limit: compressible, throttling hurts more than sharingThat single change — right-sizing requests across roughly forty Deployments — recovered more capacity than any other step. Nodes went from 11% to around 45% average utilization, and the cluster autoscaler quietly drained a third of the nodes.
Spot nodes: cheap compute for honest workloads
Spot/preemptible nodes are 60-90% cheaper, with the caveat that they vanish on two minutes' notice. The mistake is treating that as a reason to avoid them; the correct response is to sort workloads by interruption tolerance. Queue workers, cron jobs, CI runners, and stateless replicas behind a load balancer are all fine candidates — interruption just means a job retries or a pod reschedules. We ran two node pools: a small on-demand pool for stateful and latency-critical pods, and a spot pool for everything else, steered by taints and node affinity.
- Taint spot nodes and make workloads opt in explicitly; accidental placement is how spot gets a bad reputation.
- Set PodDisruptionBudgets so voluntary and spot evictions can't take out all replicas of a service at once.
- Handle SIGTERM properly — finish the in-flight job, stop pulling new work. Two minutes is plenty if you use it.
- Diversify spot instance types; a single instance-type pool can be reclaimed en masse in one capacity event.
Autoscale on the signal users feel
CPU-based HPA is fine for web tiers, but for queue workers it's actively wrong — a worker waiting on I/O shows low CPU while the backlog grows. Scale workers on queue depth per replica via KEDA or external metrics, scale web tiers on requests-per-second or CPU, and let the cluster autoscaler (or Karpenter) translate pod demand into nodes. The other half nobody does: scale down. Non-production environments running nights and weekends are pure waste; a scheduled scale-to-zero on preview and staging environments was worth 12% of the total bill by itself.
Utilization is a proxy for honesty. A cluster at 10% utilization isn't reliable — it's unexamined.
Make cost a metric, not an audit
The reason clusters drift back to waste is that cost is invisible at decision time. Put per-namespace cost (OpenCost or your cloud's allocation tooling) on the same dashboards as latency and error rate, and review it in the same weekly ritual. Optimization you do once is a project; optimization you can see is a habit. The 55% didn't come from one clever trick — it came from making waste show up on a graph someone actually looks at.