Change Data Capture: Streaming the Database Without Dual Writes

You need the search index, the analytics warehouse and the notification service to know when an order changes. The obvious implementation — write to the database, then publish an event — is a dual write, and dual writes fail at exactly the wrong moment: the transaction commits, the process dies, the event never publishes, and three downstream systems are quietly wrong until someone notices next quarter.

Change data capture removes the second write entirely. The database already maintains a durable, ordered log of every committed change for its own replication; CDC reads that log and turns it into a stream. If the transaction committed, the event exists. There is no window in which they can disagree.

The log is the source of truth

Reading the write-ahead log, rather than polling a table with an updated_at column, is what makes this reliable. Polling misses changes that happen between polls with the same timestamp, cannot see deletes, and puts load on the primary. Log-based capture sees every committed change exactly once, in commit order, including deletes, with almost no impact on the database's own work.

The connector tracks its position in the log, so a crash resumes from the last acknowledged offset. The two operational concerns that follow from that are worth internalising early: if the connector is down long enough, the database retains log segments it would otherwise recycle and disk fills — this is the classic CDC outage — and a replication slot left behind by a deleted pipeline will do the same thing silently for weeks.

{
  "name": "orders-cdc",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "plugin.name": "pgoutput",
    "slot.name": "orders_cdc",
    "publication.autocreate.mode": "filtered",
    "table.include.list": "public.orders,public.order_lines",
    "snapshot.mode": "initial",
    "topic.prefix": "prod.sales",
    "heartbeat.interval.ms": "10000",
    "key.converter.schemas.enable": "false",
    "transforms": "unwrap",
    "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
    "transforms.unwrap.delete.handling.mode": "rewrite"
  }
}

The heartbeat setting is not cosmetic. On a low-traffic table the connector otherwise has nothing to acknowledge, its recorded position stops advancing, and the database keeps every log segment since. A heartbeat keeps the offset moving even when your table is quiet, which prevents a class of 3 a.m. disk alert that is genuinely unpleasant to diagnose.

A row change is not a domain event

This is the design mistake that turns a good pipeline into a distributed coupling problem. A CDC message says 'column status changed from 2 to 3 on row 9912'. A domain event says 'order shipped'. If consumers subscribe to raw row changes, every one of them now depends on your table structure, and a routine column rename becomes a breaking change for four teams who do not report to you.

• Treat CDC topics as internal. Put a translation stage between raw changes and the events other teams consume.
• Register schemas and enforce compatibility, so a producer-side migration cannot silently break consumers.
• Plan the backfill: an initial snapshot of a large table is a serious operation, and incremental snapshotting is usually the gentler path.
• Key topics by primary key so compaction and ordering per entity behave as you expect.
• Monitor replication slot lag as a first-class alert. It is the metric that predicts the outage.

CDC gives you every change the database made. Turning that into something another team can depend on is a separate, deliberate piece of design.

Used with that boundary in place, CDC is one of the highest-leverage patterns available for integration work: search indexes that are never stale, warehouses that are minutes rather than a day behind, and cache invalidation driven by facts instead of guesses — all without asking the application team to remember to publish anything.