Kafka transactions and "exactly once": output topic and offsets committed together Advanced

The problem

A processor reads OrderPaid at offset 7 and writes InvoiceCreated to another topic. It publishes, then dies before committing the input offset: the new consumer rereads offset 7 and publishes the invoice a second time. Reverse the order and a crash in between drops the invoice (offset commit article).

When both the input offset and the output live in Kafka, a transaction combines them into one commit boundary. The scope of “exactly once” is: read Kafka, process, write Kafka (plus the input offset). Writing PostgreSQL, calling a payment API or sending email is outside it.

Mechanism

One Kafka transaction covers the output and the input progress

Enlarge diagram

Input → Processoroffset 7

◀ Prev▶ PlayNext ▶↺ RestartStep 1/4

The processor reads the record at offset 7; the group's committed position is still 7.

• A stable transactional.id lets Kafka identify the session and fence (block) the old producer when a new instance replaces it.
• Within the transaction: produce the output, send the input offset (the next read position, i.e. 8 after record 7) into the transaction, then commit.
• Commit: output and offset are confirmed together. Abort: the output is treated as aborted, the offset does not advance, and the input is read again.
• The idempotent producer is the foundation; transactions add atomicity across a group of writes and the offset.
• The consumer reading the input must set enable.auto.commit=false; after an abort, the application must reset its read position (an abort does not move the in-memory cursor back to the old offset).

Situation	read_committed sees the output?	Input offset advances?
Abort	No	No
Commit	Yes	Yes, in the same transaction
PostgreSQL written, then Kafka aborts	No	The DB is already written: systems diverge

LSO: an open transaction blocks the whole partition

A read_committed consumer only reads up to the LSO (last stable offset: the first offset of a still-open transaction, or the end of the log if none). Every record after the LSO is held back, including regular committed records from other producers.

An open transaction makes a read_committed consumer wait

◀ Prev▶ PlayNext ▶↺ RestartStep 1/5

Producer A opens a transaction and writes a record at offset 10. The transaction has not ended.

Operational consequence: a long-open transaction increases read latency for the whole partition. The broker aborts transactions exceeding transaction.timeout.ms on its own, so keep it short enough.

Config cheat sheet

Config	Default	Notes
transactional.id (producer)	null	Without it, transactions cannot be used
transaction.timeout.ms (producer)	60000	Exceeding the broker’s transaction.max.timeout.ms is rejected
transaction.max.timeout.ms (broker)	900000 (15 minutes)	Upper bound for the timeout
isolation.level (consumer)	read_uncommitted	Downstream consumers must set read_committed
enable.auto.commit (input consumer)	true	Set false in a transactional processor
transaction.state.log.replication.factor (broker)	3	A single-broker dev cluster must lower it to 1
transaction.state.log.min.isr (broker)	2	Same as above; do not copy dev config to production

Since Kafka 4.0, the stronger transaction protocol (KIP-890) is enabled when the upgrade is finalized: the producer epoch increases with each transaction. The upgrade docs advise applications to treat both TimeoutException and TransactionAbortableException as reasons to abort the transaction.

Practical recommendations

• Use Kafka Streams with processing.guarantee=exactly_once_v2 instead of hand-writing the consume-transform-produce loop: the library handles fencing, rebalances and position resets.
• Hand-write it with the Java consumer/producer only if you accept managing group metadata, epochs and commit/abort errors.
• When the processor also writes PostgreSQL: use idempotent writes keyed on eventId, or an outbox and inbox; a Kafka transaction cannot lock the DB.

When to use / not use

Use	Do not use
Kafka-to-Kafka pipelines (enrich, join, aggregate)	The final destination is a DB or an external API
Need atomic output across multiple partitions/topics	Only need to prevent retry duplicates (the idempotent producer is enough)
Using Kafka Streams	The team cannot yet operate at-least-once plus an idempotent consumer

Real-world example

Two levels of guarantee in one pipeline

A processor turns OrderPaid into InvoiceRequested, publishes with a transactional producer and commits the input offset in the same transaction; output consumers use read_committed. A restart mid-batch means downstream never sees aborted output.

Another service writes invoices to PostgreSQL with a unique eventId and commits its offset after the DB transaction. The team documents it explicitly: exactly-once on the Kafka-to-Kafka flow, at-least-once plus idempotency on Kafka-to-PostgreSQL.

An in-memory state model (an array and a number; it creates no real transaction markers or fencing):

npx --prefix examples ts-node examples/src/kafka-transactions/transaction-model.ts

Output: on the aborted attempt, read_uncommitted sees the event, read_committed does not, and the input offset stays at 7; after the successful attempt, read_committed sees one invoice-created and the input offset is 8. The repo only has a single-node Kafka 3.6 broker, so real transactions have not been run.

Trade-offs

The Kafka-to-Kafka boundary

• Output and input offset commit or abort together.
• read_committed hides the output of aborted transactions.
• Fencing stops the old producer from writing after a new instance replaces it.

Operational limits

• More complex than at-least-once: aborts, position resets, rebalances, transaction duration.
• Long-open transactions increase read latency via the LSO.
• PostgreSQL, email and HTTP are outside the transaction.

Production checklist

1. Use only for Kafka-to-Kafka flows; use idempotency everywhere else.
2. A stable transactional.id, unique per instance/partition; enable.auto.commit=false for the input.
3. Downstream consumers set isolation.level=read_committed.
4. Keep transaction.timeout.ms short enough; monitor LSO-induced latency.
5. Production: transaction.state.log RF=3, min ISR=2.
6. Prefer Kafka Streams exactly_once_v2.
7. Test a crash between produce and commit, restarts and rebalances.

Common mistakes

Common pitfall

Enabling transactions but still auto-committing the input offset separately. The offset can run ahead of the output. Put the offset into the transaction.

Common pitfall

Downstream reads with read_uncommitted. It sees records from aborted transactions too. Set read_committed.

Common pitfall

Calling a pipeline with a DB exactly-once end to end. Kafka does not commit the DB for you. Define the boundary and add idempotency or an outbox.

Summary

Key takeaways

• A transaction combines Kafka output and the input offset into one commit.
• read_committed only reads up to the LSO; a long-open transaction slows the whole partition.
• Exactly-once does not extend to PostgreSQL or external APIs.
• The most practical path: Kafka Streams exactly_once_v2.

Self-check: (1) The output was produced but the transaction aborted: what does read_committed see? (2) Why must auto-commit of the input offset be disabled? (3) How does an open (uncommitted) transaction affect read_committed consumers of that partition?

References

• Message Delivery Semantics and Using Transactions — Apache Kafka 4.2
• Producer Configs — Apache Kafka 4.2
• Upgrade notes — Apache Kafka 4.2