Idempotency Keys: Faking Exactly-Once in Payment and Webhook Systems

A client sends a charge request. The charge succeeds, but the response is lost to a network blip. The client, doing exactly what a well-behaved client should, retries. Without idempotency, you've just billed a customer twice — and no amount of 'be careful' fixes it, because the retry is correct behavior. Timeout ambiguity is fundamental: the caller cannot distinguish 'failed' from 'succeeded, response lost'.

Having built payment flows on Stripe and consumed more webhook firehoses than I care to count, my position is simple: any endpoint with side effects that can be retried must be idempotent, and the mechanism has to live server-side. Here's the design that has held up.

Exactly-once delivery is impossible; exactly-once effect is engineering

Distributed systems give you at-most-once or at-least-once delivery — pick one. Every serious system picks at-least-once and pushes deduplication to the receiver. The contract: the client generates a unique idempotency key per logical operation (not per HTTP attempt), sends it on every retry of that operation, and the server guarantees the side effect happens once and every retry gets the same response.

The server-side state machine

The naive implementation — check if key exists, then insert — has a race: two concurrent retries both pass the check. The claim must be atomic. A unique constraint plus insert-first is the simplest correct version:

public function charge(Request $request)
{
    $key = $request->header('Idempotency-Key');
    abort_if(!$key, 400, 'Idempotency-Key required');

    try {
        $record = IdempotencyKey::create([
            'key' => $key,
            'request_hash' => hash('sha256', $request->getContent()),
            'status' => 'in_progress',
        ]);
    } catch (UniqueConstraintViolationException) {
        $record = IdempotencyKey::where('key', $key)->firstOrFail();
        if ($record->request_hash !== hash('sha256', $request->getContent())) {
            abort(422, 'Key reused with different payload');
        }
        if ($record->status === 'in_progress') {
            abort(409, 'Original request still processing');
        }
        return response()->json($record->response_body, $record->response_code);
    }

    $response = $this->processCharge($request);
    $record->update(['status' => 'completed', ...$response->toStorable()]);
    return $response;
}

• Store the response, not just a 'seen' flag — retries must return the original result, including the original error.
• Hash the payload and reject key reuse with a different body; otherwise a client bug silently returns the wrong customer's charge.
• Return 409 while the first attempt is in flight rather than blocking; let the client retry with backoff.
• Expire keys on a policy (Stripe uses 24 hours) so the table doesn't grow forever — and document that window.

Webhooks: the same problem, inverted

When you consume webhooks, you're the server in this story. Stripe, Microsoft Graph, SendGrid — all of them deliver at-least-once, and all of them will re-deliver during their retry storms precisely when your system is already degraded. Treat the event ID as the idempotency key: atomically record it before processing, skip if already recorded, and keep the handler itself safe to re-run because your dedupe window won't be infinite.

Retries are not an edge case. They are the normal operation of a distributed system observed over enough time.

Where the effect isn't yours

Idempotency gets harder when the side effect lives in someone else's system. If your handler calls Stripe, pass your own derived idempotency key downstream — Stripe's API accepts one for exactly this reason. Chain the keys: inbound event ID becomes the outbound idempotency key. Now a redelivered webhook that crashes halfway through can be safely reprocessed end to end.

What this buys you

Once effects are idempotent, aggressive retry policies stop being scary, at-least-once queues stop being a correctness risk, and incident recovery becomes 'replay the last hour of events' instead of a forensic reconciliation project. That last one is the real payoff. The teams that recover fastest from outages aren't the ones that prevent every duplicate — they're the ones whose systems don't care.