Kafka idempotent producer: retrying the same batch does not duplicate, a new send still can Intermediate

The problem

The order service sends OrderPaid. The broker has appended the record but the response is lost because the connection dropped; the producer only sees a timeout and does not know whether the first request succeeded.

Without a retry the event may be lost; with a blind retry it may be written twice, and billing books the entry twice.

The idempotent producer solves exactly this uncertainty at the producer-to-Kafka boundary: the broker recognizes a retry of the same batch and does not append it a second time. A new send call for the same event is not protected.

Mechanism

The broker identifies a producer by its producer ID (PID) and a per-partition sequence number. A retry keeps the PID and sequence of the original batch.

The broker appended, the ack was lost, the producer retries with the same sequence

Enlarge diagram

App → Producerevent-42

◀ Prev▶ PlayNext ▶↺ RestartStep 1/4

The app sends an event; the producer attaches a producer ID and sequence number to the batch for the partition.

Send	PID / sequence	Broker has seen it?	Result
First batch	7 / 0	No	Append
Retry of the same batch after a lost ack	7 / 0	Yes	No second append
App issues a new send with the same eventId	7 / 1	No	Appends a second record

The broker does not understand OrderPaid or orderId; it only compares PID and sequence. Two independent send calls with the same payload are still two valid records.

Config cheat sheet (Java producer 4.2)

Config	Default	Notes
enable.idempotence	true (default since 3.0)	If set explicitly with a conflicting config, a ConfigException is thrown; if relying on the default with a conflict, idempotence is silently disabled
acks	all	Idempotence requires all
retries	2147483647	The practical limit is delivery.timeout.ms
delivery.timeout.ms	120000	Total time for one send, including retries
max.in.flight.requests.per.connection	5	Must be <= 5 with idempotence on
linger.ms	5 (changed from 0 in 4.0)	Wait time to fill a batch
transactional.id	null	Only needed when using transactions

Check the effective config after merging defaults, framework settings and environment variables: an acks=1 from the environment can silently disable idempotence.

Scope and limits

• Ordering: with idempotence and max.in.flight<=5, retries do not reorder batches within a partition. The producer config docs only state the <= 5 requirement; the reason lies in the broker implementation (it keeps sequence state for only a few recent batches per producer/partition pair), not a documented contract.
• PID lifecycle: a restarted producer gets a new PID, so a send after the restart is not treated as a retry of the earlier batch.
• Scope: one producer session and one partition. It does not span multiple partitions, multiple sessions, or the DB.
• Does not replace acks=all: idempotence prevents duplicates on retry; durability is the job of acks, ISR and min.insync.replicas (ISR article).

A scenario right at the boundary: the DB records the order as paid but does not store that the event was published. After a restart, the application calls send again. This is a new batch, and the topic has two OrderPaid.

The fix: a transactional outbox on the publishing side and an idempotent consumer on the receiving side.

When to use / not use

• Always on (already the default in 4.x); there is no reason to turn it off for a regular producer.
• Not enough when you need atomic writes across partitions or tied to input offsets: use Kafka transactions.
• Does not replace a business event ID and consumer-side dedupe.

Real-world example

Three layers, three failure windows

The order service stores OrderPaid (eventId=event-42) in the outbox within the same transaction that changes the order status. A relay publishes with an idempotent producer.

1. The Kafka ack is lost: the producer retries the same batch, no duplicate.
2. The relay crashes after Kafka received it but before marking the outbox row: the new relay publishes it again as a new send.
3. Billing writes the ledger with a unique eventId, so the side effect applies only once.

Each layer handles its own window; none can replace the other two.

A model of PID/sequence state using a Map (not a real producer or broker; not for verifying config). The repo has a KafkaJS 2.2 demo running against a Kafka 3.6 broker; this article describes config per 4.2.

npx --prefix examples ts-node examples/src/kafka-idempotent-producer/retry-model.ts

Output: first send: appended, same batch after lost ack: duplicate retry, new logical send: appended; the simulated log ends with two order-42-paid.

Trade-offs

Producer retry protection

• Retrying the same batch after a lost ack does not append a copy.
• Keeps batch order within a partition when configured correctly.
• The application does not have to guess write state from a timeout.

Limits

• Conflicting config can disable idempotence or make the producer fail.
• Does not dedupe a new send with the same payload or event ID.
• Does not make a DB-to-Kafka pipeline exactly-once.

Production checklist

1. Keep enable.idempotence=true, acks=all, max.in.flight<=5; log the effective config at startup.
2. Important topics use RF=3, min.insync.replicas=2.
3. Create the eventId once at the business moment, store it durably, and reuse it for every publish.
4. Use an outbox when the DB and Kafka must stay consistent.
5. Consumers dedupe by eventId.
6. Handle the final send error by error type; do not wrap it in while(true) send().

Common mistakes

Common pitfall

Seeing enable.idempotence=true and dropping the event ID. The broker does not know two sends are the same business action. Keep a stable event ID and dedupe in the consumer.

Common pitfall

Checking config in only one file. acks=1 or max.in.flight>5 from the environment causes a conflict. Check the effective config.

Common pitfall

Treating a producer timeout as “Kafka did not write”. The broker may have appended it. Do not publish a new event with a new ID before resolving the unknown outcome.

Summary

Key takeaways

• The idempotent producer dedupes retries of the same batch using the PID and a per-partition sequence.
• Defaults in Kafka 4.x: enable.idempotence=true, acks=all, max.in.flight=5.
• A new send or a producer restart can still create business duplicates.
• An outbox and an idempotent consumer are still needed for windows outside producer retries.

Self-check: (1) The broker appended, the ack was lost, and the producer retries the same batch: how many records result? (2) What if the app restarts and sends the same eventId again? (3) Why do acks=all and idempotence solve two different problems?

References

• Message Delivery Semantics — Apache Kafka 4.2
• Producer Configs — Apache Kafka 4.2