Notifications look like a feature and behave like a subsystem. The first version is a mail call inside a controller. The tenth version has preferences, channels, batching, quiet hours, deduplication, retries, and an unsubscribe flow that legal reviewed — and by then it is tangled through the whole application because nobody drew a boundary at the start.
Separate the event from the notification
The single most important structural decision: application code emits domain events, and a notification service decides who hears about them and how. The code that approves an inspection should not know that three people want an email and one wants a daily digest. It should say 'inspection approved' and stop.
# Domain code emits a fact. It knows nothing about channels or preferences.
events.publish(InspectionApproved(inspection_id=i.id, tenant_id=i.tenant_id,
approver_id=user.id, at=now()))
# The notification service owns the whole decision tree.
async def on_inspection_approved(event):
for recipient in subscribers_for(event): # fan-out
prefs = preferences.get(recipient.id, event.type)
if not prefs.enabled:
continue
# Deduplicate: the same fact reaching the same person twice is the
# complaint you will get most often.
key = f'{event.type}:{event.inspection_id}:{recipient.id}'
if not dedupe.add(key, ttl=3600):
continue
if prefs.cadence == 'digest':
digest.append(recipient.id, event) # batched later
elif quiet_hours(recipient, now()) and event.priority < Priority.URGENT:
schedule.at(next_waking_hour(recipient), recipient.id, event)
else:
await deliver(recipient, event, prefs.channels) # own retry policyFan-out is where the scaling decision lives
Fan-out on write — computing recipients and writing a row per person when the event occurs — makes reads trivial and works well when the audience is small and bounded. Fan-out on read, where the feed is assembled when someone opens it, avoids writing millions of rows for a broadcast but makes every read more expensive. Most enterprise systems have small audiences per event and should simply fan out on write; the hybrid only becomes necessary when one event can reach tens of thousands of people.
Whichever you choose, the fan-out itself must be asynchronous and chunked. A synchronous loop over four thousand recipients inside a request handler is a timeout with a notification feature attached.
The parts users actually judge you on
- Deduplication with an idempotency key per recipient per event. Duplicate notifications erode trust faster than late ones.
- Digest batching for high-volume event types, with a real summary rather than twenty concatenated messages.
- Preferences at the right granularity — per event type and channel, not one global switch that people use to turn everything off.
- Per-channel retry and failure handling: a bounced email, a revoked push token and a rate-limited SMS gateway are three different problems.
- An in-app inbox as the durable record, with email and push as delivery hints. Channels fail; the record should not.
The engineering problem is delivery. The product problem is restraint, and the second one is what determines whether people turn notifications off.
One metric is worth instrumenting from the first version: the rate at which users disable each notification type. It tells you precisely which messages are unwanted, and it is the only feedback signal that reliably distinguishes a notification people value from one they tolerate until they find the settings page.