Kafka Exactly-Once in Practice: Partitions, Transactions, and Rebalances

Kafka's exactly-once semantics are frequently described as marketing and frequently deployed as a checkbox. Both are wrong. The guarantee is real, it is precisely scoped, and almost every disappointment I have seen came from expecting it to cover a boundary it explicitly does not.

What it covers: consume from a topic, process, produce to a topic, and commit the consumer offset — all as one atomic transaction. Either the output records and the offset commit both land, or neither does. What it does not cover: the HTTP call you made to a payment provider in the middle, or the row you wrote to Postgres. Those are outside the transaction and always were.

Partition keys are your consistency model

Ordering in Kafka exists within a partition, and nowhere else. The partition key is therefore not a load-balancing detail — it is the declaration of which events must be processed in order relative to each other. Key by the entity whose sequence matters: account ID, tenant ID, aggregate ID. Key by something random and you have chosen, explicitly, that no ordering exists.

The trap is skew. Key by tenant when one tenant is forty percent of your volume and one partition is permanently hot while the others idle; you cannot fix that by adding consumers, because a partition is consumed by exactly one member of the group. The usual resolution is a composite key — tenant plus a sub-entity — which shards the hot tenant while keeping the ordering that actually matters.

producer = Producer({
    'enable.idempotence': True,        # no duplicates from producer retries
    'transactional.id': 'enricher-1',  # stable per instance, survives restarts
    'acks': 'all',
    'max.in.flight.requests.per.connection': 5,
})
producer.init_transactions()

for batch in consumer.consume_batches():
    producer.begin_transaction()
    try:
        for msg in batch:
            producer.produce('enriched', key=msg.key(), value=enrich(msg))
        # The offset commit rides inside the transaction, not after it.
        producer.send_offsets_to_transaction(
            consumer.position(consumer.assignment()), consumer.consumer_group_metadata())
        producer.commit_transaction()
    except Exception:
        producer.abort_transaction()
        raise

# Downstream consumers MUST read with isolation.level=read_committed,
# or they will happily consume records from transactions that later aborted.

Rebalances are the failure mode nobody rehearses

A consumer group rebalance reassigns partitions, and with the default eager protocol it stops the entire group to do it. Deploys trigger it, scaling triggers it, and a consumer that blocks too long between polls triggers it — which is how a slow downstream dependency turns into a rebalance storm where the group spends more time reassigning than consuming.

• Use cooperative rebalancing so a deploy does not stop every consumer in the group.
• Keep max.poll.interval.ms above your realistic worst-case processing time, and do slow work asynchronously rather than inside the poll loop.
• Make processing idempotent anyway. Rebalance during a transaction means the next owner reprocesses from the last committed offset.
• Alert on consumer lag per partition, not in aggregate — one stuck partition is invisible in a total.
• Size partition count for peak consumer parallelism up front; increasing it later rehashes keys and breaks ordering guarantees you were relying on.

Exactly-once is a property of a closed loop. The moment your processing touches a system outside Kafka, you are back to at-least-once with idempotency keys.

That last point is the practical summary. If your consumer's job is to transform events into other events, use transactions and enjoy a genuinely strong guarantee. If it writes to a database or calls an external API — which is most consumers — design for at-least-once, make the write idempotent, and treat exactly-once as what it is: an optimisation for one specific topology, not a property of the system as a whole.