Webhooks look like the easy half of an integration: make an HTTP request when something happens. The difficulty is that the receiver is someone else's system, with their uptime, their timeouts, their deploys and their occasional decision to return 200 for a request they did not actually process.
Retries, and the storm you can cause
Deliveries fail routinely, so retry with exponential backoff and jitter over a long window — hours, not minutes — because a customer's endpoint being down for a deploy is normal. After the window expires, stop, mark the endpoint unhealthy, and notify the customer rather than retrying indefinitely.
Circuit-break per endpoint. When a customer's receiver has failed fifty times in a row, continuing to send is both futile and rude — you are hammering a system that is already in trouble, and filling your own queue with work that will not succeed.
def sign(payload: bytes, secret: str, timestamp: int) -> str:
# Sign the timestamp WITH the body. Signing the body alone lets an
# attacker who captures one delivery replay it indefinitely.
signed = f'{timestamp}.'.encode() + payload
return hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest()
async def deliver(endpoint, event, attempt=0):
ts = int(time.time())
body = json.dumps(event.payload).encode()
headers = {
'X-Signature': f'v1={sign(body, endpoint.secret, ts)}',
'X-Timestamp': str(ts),
'X-Event-Id': event.id, # stable across retries: the receiver
'X-Delivery-Id': uuid4().hex, # deduplicates on THIS id
}
try:
r = await http.post(endpoint.url, content=body, headers=headers, timeout=10)
if r.status_code < 300:
return record_success(endpoint, event)
except httpx.RequestError:
pass
if attempt >= 12: # ~24h of backoff, then give up loudly
return mark_endpoint_failing(endpoint, event)
delay = min(3600, 2 ** attempt) * random.uniform(0.8, 1.2)
await schedule(deliver, endpoint, event, attempt + 1, delay=delay)The two identifiers matter. The event ID is stable across retries so the receiver can deduplicate — you will deliver twice eventually, and saying so in the documentation is better than pretending otherwise. The delivery ID changes per attempt so both sides can discuss one specific delivery in a support thread.
Ordering, and why you should not promise it
Retries reorder events by construction: event A fails and is retried while event B succeeds immediately, so B arrives first. You can preserve per-resource ordering by serialising deliveries for that resource and blocking on failure, but that means one stuck event halts everything behind it for that customer.
The better contract is to include a sequence number and a timestamp, state plainly that delivery is unordered and at-least-once, and let receivers ignore anything older than what they have already applied. Almost every mature webhook API takes this position, and it is the honest one.
- Give customers a delivery log they can see: payload, response code, attempts, and a manual replay button. It removes most of your support load.
- Keep payloads small and include an ID the receiver can fetch. Large payloads are expensive to retry and awkward to version.
- Document the signature verification with working code in three languages. Every hour spent here saves ten in support.
- Version the payload and never silently change a field's meaning — receivers are even more fragile than API clients.
- Send from a stable, published set of addresses, and support a customer-supplied secret they can rotate without your involvement.
You will deliver the same event twice. The only choice is whether the receiver was told to expect it.
One practice repays itself faster than any other: a test endpoint customers can point at during their build, which shows them exactly what you send and whether their signature check would have passed. Most integration failures are not reliability problems — they are a developer on the other side, at eleven at night, unable to tell whether the problem is your payload or their verification.