Here's a bug I've seen in three different codebases, written by three good teams: save the order to the database, then publish an OrderCreated event to the broker. Two writes, two systems, no transaction spanning them. The service crashes between the two, or the broker times out, and now the order exists but downstream — billing, inventory, notifications — never hears about it. The inverse ordering is worse: publish first, and a rolled-back transaction means consumers act on an order that doesn't exist.
This is the dual-write problem, and it isn't rare. At a few thousand events an hour, a 0.01% failure window still corrupts state daily. Retries don't fix it; they just change which half is missing.
One transaction, one source of truth
The transactional outbox pattern resolves it by refusing to do two writes. You write the business row and the event into the same database, in the same local transaction. An outbox table holds the pending events; a separate relay process reads them and publishes to the broker. If the transaction commits, the event is durably queued. If it rolls back, the event never existed. Atomicity comes from the one place you already have it — your database.
func (s *OrderService) CreateOrder(ctx context.Context, o Order) error {
return s.db.WithTx(ctx, func(tx *sql.Tx) error {
if err := insertOrder(tx, o); err != nil {
return err
}
evt := OutboxEvent{
ID: uuid.New(),
AggregateType: "order",
AggregateID: o.ID,
EventType: "order.created",
Payload: mustJSON(o),
CreatedAt: time.Now().UTC(),
}
// Same transaction: commit means both exist, rollback means neither.
return insertOutbox(tx, evt)
})
}The relay: polling vs. CDC
Something has to move events from the table to the broker. Two viable designs. A polling relay selects unpublished rows (with FOR UPDATE SKIP LOCKED so multiple relay instances don't fight), publishes, and marks them sent — simple, debuggable, adds 100ms-1s of latency. Change data capture with Debezium tails the database WAL and streams outbox inserts to Kafka with no polling and near-zero lag — more moving parts, but the right call at serious volume.
My honest advice: start with polling. I've run a polling relay at hundreds of events per second on unremarkable Postgres hardware. You can graduate to CDC when latency or load demands it, and the outbox table schema doesn't change — that's the beauty of the pattern as an interface.
Consumers still need to earn their keep
The outbox gives you at-least-once publication — the relay can crash after publishing but before marking the row, and the event goes out again. That's the correct trade. It means consumers must deduplicate on the event ID, which you conveniently generated inside the producing transaction. Ordering also needs care: partition the broker topic by aggregate ID so all events for order 123 land in one partition, in commit order.
- Carry the event ID end to end; it's the idempotency key for every consumer.
- Partition by aggregate ID to preserve per-entity ordering without global ordering costs.
- Version event payloads from day one — v2 of order.created will arrive sooner than you think.
- Monitor outbox lag (oldest unpublished row age); it's the single best health signal for the pipeline.
- Prune published rows on a schedule, or the outbox becomes your largest and least useful table.
You don't need a distributed transaction. You need one local transaction and the honesty to handle at-least-once everywhere else.
Where I'd reach for it
Any time a state change in one service must reliably trigger work in another and losing that trigger costs real money or trust: order lifecycles, billing events, provisioning workflows, audit trails, notifications. The pattern's overhead is one table, one relay, and a dedup check per consumer. Against a 2 a.m. reconciliation script that guesses which orders never got invoiced, it's the cheapest insurance in distributed systems.