A realtime feature is the easiest thing in the world to demo and one of the more awkward things to operate. You open a socket, push a JSON message, a number on the dashboard ticks up, and everybody is pleased. Then it meets a few thousand concurrent users on unreliable networks, and you find that every assumption the rest of your stack is built on — stateless workers, rolling deploys, autoscaling on CPU — quietly stopped holding.
I have shipped realtime into enterprise products twice: a live operations dashboard for field teams, and an in-app notification and collaboration layer. Both times the hard parts were the same four, and not one of them was the WebSocket protocol.
A connection is state, and state changes how you deploy
An HTTP request lives for eighty milliseconds. A socket lives for hours. That single difference moves your capacity ceiling from CPU to memory and file descriptors: budget somewhere between ten and forty kilobytes per connection once you count send buffers and per-connection application state, and fifty thousand connections is suddenly a real number on a real machine. Autoscaling on CPU is actively wrong here — a node can be at eight percent CPU and completely full. Scale on connection count, and set the pod limit from measured memory per connection rather than hope.
Scale-in is the sharper edge. Removing a pod disconnects everyone on it, so an aggressive scale-in policy manufactures the exact reconnect storm you are trying to avoid. The same applies to deploys: every deploy of the connection layer is a mass reconnect event, and if all clients come back in the same second you will DDoS your own authentication endpoint. Two things fix it. Drain slowly — close connections in batches over several minutes with a close code the client understands as 'come back later'. And mandate jittered exponential backoff on the client, because a reconnect loop with no jitter is a synchronised herd by construction.
The structural fix is to deploy the connection layer separately from the application. Business logic ships several times a day; the gateway that holds sockets should ship when the gateway changes. Coupling them means every routine deploy churns every connection in the product.
Fan-out is where the cost actually lives
Publishing one message to a channel with N members is N deliveries, and that multiplication is the thing that breaks. The naive architecture has every gateway node subscribed to every channel, filtering locally — which means every node pays the full firehose regardless of who it is actually serving. It works beautifully at ten nodes and one thousand channels, and falls over at the point where the product becomes successful.
Two changes carried us a long way. First, a gateway node subscribes upstream only to the channels for which it currently holds a local connection, and unsubscribes when the last one goes. Second, coalesce: buffer events per connection and flush once per tick rather than writing a frame per event. A fifty millisecond window is invisible to users and collapsed our busiest dashboard from roughly forty frames a second per client to twenty, with a matching drop in syscalls and egress. The tick loop is also the natural place to handle slow consumers, which otherwise accumulate unbounded buffers until the node dies.
// A gateway node subscribes upstream only to channels it actually holds,
// and coalesces bursts into one frame per tick.
const CHANNELS = new Map<string, Set<Connection>>();
const TICK_MS = 50;
async function subscribe(conn: Connection, channel: string) {
let members = CHANNELS.get(channel);
if (!members) {
members = new Set();
CHANNELS.set(channel, members);
await broker.subscribe(channel); // first local member: subscribe upstream
}
members.add(conn);
}
broker.on('message', (channel: string, event: Event) => {
for (const conn of CHANNELS.get(channel) ?? []) {
conn.pending.push(event); // buffer, do not write yet
}
});
setInterval(() => {
for (const conn of connections) {
if (conn.pending.length === 0) continue;
if (conn.socket.bufferedAmount > BACKPRESSURE_LIMIT) {
// Slow consumer: drop the buffer, make it resync from its cursor.
conn.pending.length = 0;
conn.send({ type: 'resync', from: conn.lastSeq });
continue;
}
conn.send({ type: 'batch', events: conn.pending.splice(0) });
}
}, TICK_MS);Presence is an approximation, so bound it explicitly
- Presence is eventually consistent by nature. A phone that walks into a lift is indistinguishable from a client that is simply quiet, so heartbeat and expire: refresh a key every fifteen seconds, expire it at forty-five, and tell product that 'online' means 'seen in the last forty-five seconds'. Agreeing that sentence early prevents a lot of bug reports that are not bugs.
- Keep presence in an expiring in-memory store, never in a database row you write on every heartbeat. Ten thousand users at four heartbeats a minute is a sustained write load whose only purpose is a green dot.
- Reference-count per user, not per connection. The same person has a laptop, a phone and three tabs; presence goes away when the last connection does, and a naive implementation flickers every time a tab closes.
- Debounce presence fan-out separately and more aggressively than message fan-out. Nobody needs sub-second accuracy on a roster, and joins and leaves in a busy channel generate far more traffic than the actual content.
- Make reconnect cheap and idempotent. A client returning should resubscribe and ask for everything after a cursor, not trigger a full state dump for every channel it was in.
Missed messages are the requirement nobody writes down
The socket is a delivery optimisation, not a source of truth. Networks drop, nodes restart, mobile clients suspend. If the only path by which a change reaches the screen is an open socket, then every blip is silent data loss, and users respond exactly as you would expect: they stop trusting the screen and start refreshing it, which is the feature failing in the most expensive way possible.
The fix is boring and mandatory. Give every channel a monotonic sequence number, have the client persist the last sequence it processed, and expose an HTTP endpoint that returns everything after a cursor. On reconnect the client sends its cursor; if the gap is small the server replays it, and if the gap is beyond the retention window the server says so and the client refetches the underlying resource over REST. That protocol is perhaps two hundred lines, and it turns the socket back into what it should be — a latency improvement over polling, with the correctness guarantee living underneath it.
The socket is an optimisation. If the feature is only correct while the socket is open, it is not a feature, it is a demo.
One last piece of pragmatism: use a managed service or an off-the-shelf server until you have a specific reason not to. Nothing interesting about your product lives in the WebSocket frame parser. The engineering that actually differentiates is the sequencing protocol, the presence semantics you agreed with product, and the shape of your fan-out — and all three are yours to design regardless of who terminates the connection.