Self-hosting an open model is usually justified with one of three reasons: cost at volume, data residency, or latency control. All three can be valid. The mistake is treating it as an infrastructure project when it is primarily an economics one, because a GPU node bills continuously and your traffic does not arrive continuously.
The number that decides it is utilisation. A GPU instance running at eight percent costs the same as one at eighty, and at eight percent an API provider is almost always cheaper. Before any of the engineering below matters, you need an honest projection of sustained token throughput, not peak.
Continuous batching is the whole game
Naive serving processes one request at a time and leaves most of the GPU idle, because generation is memory-bandwidth bound rather than compute bound. Continuous batching — as implemented by vLLM and similar servers — interleaves requests at the token level, adding new ones into the running batch as others finish, instead of waiting for a whole batch to complete.
The throughput difference is not incremental; it is commonly an order of magnitude. Paired with paged attention, which stops the KV cache from requiring one contiguous worst-case allocation per sequence, it is the difference between a GPU serving a handful of concurrent users and serving dozens.
containers:
- name: vllm
image: vllm/vllm-openai:latest
args:
- --model=/models/instruct-8b
- --max-model-len=8192 # the KV cache scales with this; be honest
- --gpu-memory-utilization=0.90 # leave headroom or you OOM under burst
- --max-num-seqs=64 # concurrency ceiling per replica
- --enable-prefix-caching # shared system prompts stop being re-computed
resources:
limits: { nvidia.com/gpu: 1 }
readinessProbe:
httpGet: { path: /health, port: 8000 }
initialDelaySeconds: 180 # weight loading is minutes, not seconds
---
# Scale on queue depth and time-to-first-token, never on GPU utilisation:
# a busy GPU is the goal, not the alarm.
triggers:
- type: prometheus
metadata:
query: vllm:num_requests_waiting
threshold: "8"Two parameters carry most of the memory risk. Maximum context length determines the KV cache footprint per sequence, and setting it to the model's theoretical maximum when your prompts are two thousand tokens wastes a large fraction of the card. Prefix caching is nearly free money if your requests share a system prompt, which in an enterprise deployment they almost always do.
Autoscaling GPUs is not autoscaling pods
- Cold start is minutes: node provisioning, image pull of many gigabytes, then weight loading. Pre-pull images and consider keeping one warm replica permanently.
- Scale on queue depth and time-to-first-token. GPU utilisation is a terrible scaling signal because high utilisation is the desired state.
- Scale-to-zero suits batch and internal tools. For interactive traffic, a warm floor of one replica is usually cheaper than the user-visible cost of a cold start.
- Separate prefill-heavy and decode-heavy traffic if you can; long documents and chat have different resource profiles and interfere with each other.
- Quantisation is the cheapest capacity increase available — measure quality on your own evals rather than trusting a general benchmark.
An idle GPU is the most expensive thing in your cluster, and the second most expensive is a GPU serving one request at a time.
The pattern that has held up best for me is hybrid: self-host the high-volume, well-defined, latency-sensitive workload where utilisation is genuinely high, and route the long tail — the occasional hard request, the overflow during a spike — to a hosted API. You get the unit economics of owning the common case without paying for capacity to sit idle waiting for the rare one.