Retry topics and DLQ in Kafka: handling failing messages without blocking the partition Intermediate

The problem

Topic orders has 12 partitions. An event in partition 3 has a broken payload, and the handler throws every time it reads it. The consumer retries forever, and partition 3’s lag climbs from 0 to tens of thousands.

The other 11 partitions run normally, so the total throughput dashboard still looks fine.

This is a poison message. It blocks the whole partition because the committed offset is contiguous progress: committing offset 5 means “everything before 5 is done”.

So until offset 4 is processed, you cannot commit past it without dropping it (offset commit). Every order after it in that partition waits.

Unlike RabbitMQ, Kafka’s consumer API has no built-in DLQ: no nack, no dead-letter exchange. Retry and DLQ are patterns you build yourself with regular topics. Built-in DLQs exist only in higher layers: Kafka Connect (sink connectors) and Kafka Streams (the default exception handler).

Three options

	In-place retry (blocking)	Delayed retry topic	Immediate DLQ
How	Retry inside the consumer, with backoff between attempts	Publish to orders.retry-5s, orders.retry-1m… then keep committing	Publish to orders.dlq, keep committing
Same-key ordering	Kept	Lost (the failing message is skipped past)	Lost (the failing message is left behind)
Main partition	Stalls during retries	Keeps running	Keeps running
Delay of the failing message	Short (seconds)	Per delay tier (seconds to minutes)	Until someone handles it
Complexity	Low	High: extra topics, consumers, headers, alerts	Medium
Fits	Very short errors (timeout, deadlock)	Longer transient errors (downstream 503)	Permanent errors (schema, validation)

The sensible default is a combination: a few in-place retries with short backoff, then retry topics for transient errors, and finally the DLQ. Permanent errors go straight to the DLQ, skipping retries.

The main → retry-5s → retry-1m → DLQ flow

order-44 fails with 503 three times: attempt increases through each topic, then it lands in the DLQ

Enlarge diagram

orders → Main consumeroffset 4

◀ Prev▶ PlayNext ▶↺ RestartStep 1/6

The main consumer receives order-44 at offset 4 (attempt 1). The payment service returns 503: a transient error.

Each delay tier is its own topic. Because every message in orders.retry-5s has the same delay, the message at the head of a partition is always the one due soonest. The consumer only needs to look at the head message.

How the delay is implemented

Kafka has no delayed delivery: the broker hands over records as soon as the consumer fetches. The retry topic consumer waits on its own:

1. Read the record and compare the due-at header with the current time.
2. Not due yet: pause that partition, seek back to this record, resume when it is due.
3. Due: process it. If it fails again, publish to the next tier with attempt + 1.

Spring Kafka does exactly this: the retry topic consumer “checks the timestamp and if it’s not due it pauses the consumption for that topic’s partition”.

Their docs add that the delay is a lower bound: never processed early, but possibly later when the consumer holds many partitions or earlier records process slowly.

With the Java client, a paused partition still needs regular poll() calls (poll returns nothing for that partition), so max.poll.interval.ms is not violated. Conversely, a 1-minute sleep in the handler is a reliable way to get kicked out of the group once total time exceeds max.poll.interval.ms (default 300000 ms) (rebalance).

KafkaJS has no built-in retry topic or DLQ in its docs, but it has per-partition pause(). A sketch of the retry consumer:

await consumer.run({
  eachMessage: async ({ topic, partition, message, pause }) => {
    const dueAt = Number(message.headers?.['x-due-at']?.toString() ?? 0);
    const wait = dueAt - Date.now();
    if (wait > 0) {
      const resume = pause();              // pauses only the current topic-partition
      setTimeout(resume, wait);
      throw new Error('not due yet');      // offset not resolved, the record is read again after resume
    }
    await handleWithRetryRouting(topic, partition, message);
  },
});

The “pause, schedule resume, then throw” pattern comes from an example in the KafkaJS docs. This sketch has not been run against a real broker; retest the crash window with your exact client version.

Headers to carry

A message in a retry topic or DLQ must describe itself: where it came from, how many attempts, what error.

Header (example name)	Value	Used for
x-attempt	1, 2, 3…	Deciding the next tier or the DLQ
x-due-at	epoch ms	Lets the retry consumer know when to process it
x-original-topic	orders	Replaying to the right topic
x-original-partition	3	Tracing, matching against logs
x-original-offset	48121	Tracing, dedupe during investigation
x-error-class, x-error-message	PaymentUnavailable, 503	Quick triage when browsing the DLQ
x-first-failed-at	epoch ms	Measuring error age, alerting

Keep the original key and value unchanged; do not wrap the payload in a new envelope. Keeping the key means the message still follows its key’s partition, and replaying to the main topic does not change partitions.

References from two existing implementations:

• Spring Kafka DeadLetterPublishingRecoverer adds headers such as DLT_ORIGINAL_TOPIC, DLT_ORIGINAL_PARTITION, DLT_ORIGINAL_OFFSET, DLT_EXCEPTION_FQCN, DLT_EXCEPTION_MESSAGE, DLT_EXCEPTION_STACKTRACE, DLT_ORIGINAL_CONSUMER_GROUP. By default it sends to topic <original topic>-dlt, in the same partition as the original record, so the DLT needs at least as many partitions as the original topic.
• Kafka Connect adds headers prefixed __connect.errors. when errors.deadletterqueue.context.headers.enable is on.

The ordering trap: retries reorder same-key events

Retry topics solve head-of-line blocking by skipping past the failing message. The price: later events of the same key are processed first. Spring Kafka says it plainly: “By using this strategy you lose Kafka’s ordering guarantees for that topic.”

order-42: the failing created goes around the retry loop, paid runs first, then created overwrites it

◀ Prev▶ PlayNext ▶↺ RestartStep 1/4

order-42 created (offset 1) fails with 503. The main consumer pushes it to retry-5s and moves on.

Ordering matters when the consumer applies per-entity state (message key): order state machines, balances, projections. It does not matter when each event is independent: sending email, generating thumbnails, writing logs.

When reordering is unacceptable, pick one of these:

Approach	How it works	Cost
In-place retry, no retry topic	Blocks the partition until it succeeds or goes to the DLQ	Whole-partition lag while waiting
Park by key	If a key has a message in retry, later messages of that key are also pushed to a waiting area (DB table or retry topic) and run only after the earlier one finishes	Must durably store the set of “failing keys”; complex
State machine rejects invalid transitions	paid for an order not yet created is not applied but sent to retry	Needs a version or status on the entity
Versioned update	Write only when the event’s version is greater than the stored version	Events must carry an increasing version

Default for stateful events: state machine plus versioned update. It tolerates both retries and redeliveries, not just retry topics.

Classify errors before retrying

Error type	Examples	Action
Transient, short	Network timeout, PostgreSQL deadlock 40P01, serialization failure 40001	Retry in place 2-3 times, short backoff
Transient, long	Downstream 503/429, open circuit breaker	Tiered retry topics
Permanent: data	Deserialization error, wrong schema, validation 400/422	DLQ immediately
Permanent: code/config	Bug, 401/403 from wrong credentials	DLQ immediately; fix, then replay
Ambiguous	“Entity not found” (maybe the event arrived early)	A few retry topic tiers, then DLQ

Spring Kafka’s DefaultErrorHandler treats some exceptions as fatal by default and does not retry them, e.g. DeserializationException, MessageConversionException, ClassCastException. Declare your own business list of non-retryable errors; do not retry every exception.

Backoff between in-place retries should have jitter so many consumers do not hammer the downstream at once (retry, backoff, jitter).

Runnable simulation

An in-memory simulation with a virtual clock, no Kafka: topics are FIFO arrays, headers are objects. Scenario: order-42 created fails with 503 twice then succeeds, order-43 has a bad schema, order-44 fails with 503 forever.

npx --prefix examples ts-node examples/src/kafka-retry-dlq-sim/retry-dlq-sim.ts

Real output of the naive mode:

=== mode: naive ===
t=  0s  orders          order-41:created attempt=1 OK
t=  0s  orders          order-42:created attempt=1 FAIL payment-svc 503 -> orders.retry-5s (due t=  5s)
t=  0s  orders          order-42:paid attempt=1 OK
t=  0s  orders          order-43:created:bad-schema attempt=1 FAIL schema invalid -> DLQ
t=  0s  orders          order-44:created attempt=1 FAIL payment-svc 503 -> orders.retry-5s (due t=  5s)
t=  0s  orders.retry-5s pause partition, wait until t=  5s
t=  5s  orders.retry-5s order-42:created attempt=2 FAIL payment-svc 503 -> orders.retry-1m (due t= 65s)
t=  5s  orders.retry-5s order-44:created attempt=2 FAIL payment-svc 503 -> orders.retry-1m (due t= 65s)
t=  5s  orders.retry-1m pause partition, wait until t= 65s
t= 65s  orders.retry-1m order-42:created attempt=3 OK
t= 65s  orders.retry-1m order-44:created attempt=3 FAIL payment-svc 503 -> DLQ
applied order : order-41:created, order-42:paid, order-42:created
DLQ           : order-43:created:bad-schema [attempt=1, orig=orders-0@3, error=schema invalid]; order-44:created [attempt=3, orig=orders-0@4, error=payment-svc 503]
order-42 order: REORDERED (paid before created)

The park-by-key mode runs the same scenario but holds order-42:paid until created finishes (excerpt of the lines that differ from naive):

t=  0s  orders          order-42:paid PARK (key in retry)
t= 65s  orders.retry-1m order-42:created attempt=3 OK
t= 65s  unpark order-42:paid
t= 65s  parked          order-42:paid attempt=1 OK
applied order : order-41:created, order-42:created, order-42:paid
order-42 order: preserved (created before paid)

Note: order-43 lands in the DLQ with attempt=1 (permanent errors are not retried), order-44 with attempt=3. The main partition processes all 5 records at t=0s despite 3 failing records.

Replaying from the DLQ

A DLQ is not a trash bin. Every message in it is a business action not yet completed.

1. Triage: group by x-error-class and x-original-topic; find the common cause.
2. Fix the cause: deploy a fix, correct the configuration, or wait for the downstream to recover.
3. Check idempotency: the handler must tolerate reprocessing, since the message may have been partially applied (idempotent consumer).
4. Bounded replay: a tool reads the DLQ and republishes to x-original-topic with the same key, value and event ID, resetting x-attempt. Rate-limit it; try a few messages first.
5. Track progress: the replay tool is its own consumer group on the DLQ; commit the offset after a successful publish so a tool crash does not replay twice.
6. Unrecoverable messages: record the reason and who decided to drop them, then mark them done. Never delete silently.

Ordering note: replaying order-42 created after paid has run is still a reorder. The state machine must handle it.

Alerting and monitoring

Signal	Meaning	Suggested threshold
New messages entering the DLQ (rate)	There are unhandled errors	Any message for an important topic
Unprocessed messages in the DLQ (lag of the replay/audit group)	Error backlog	Growing continuously
Age of the oldest DLQ message	Forgotten errors	Approaching the DLQ retention
Lag of each retry topic	Retry consumer slow or dead	Several times that tier’s delay
Share of messages entering retry vs. total	Downstream is down	Sudden spike

Set DLQ retention long enough to investigate (e.g. a few weeks); a short retention turns the DLQ into delayed data loss.

Existing options

Tool	What it offers	Limits
Kafka Connect (sink)	errors.tolerance=all plus errors.deadletterqueue.topic.name	Sink connectors only; the default empty topic name means the DLQ is off
Spring Kafka @RetryableTopic	Auto-creates retry topics per delay (-retry-1000, -retry-2000…) and -dlt, pauses partitions until due	No support for batch listeners and container transactions
Spring Kafka DefaultErrorHandler	In-place (blocking) retry via seek; default FixedBackOff(0L, 9), i.e. 10 attempts, no delay, then log	Blocks the partition during retries
Kafka Streams	errors.dead.letter.queue.topic.name (4.2 docs): the default handler sends failing records to this topic	Default null; ignored if you use a custom exception handler
KafkaJS	Per-partition pause/resume	No retry topic or DLQ; build it yourself
Share groups (Kafka 4.2+)	Counts deliveries per record; group.share.delivery.count.limit default 5 (2 to 10)	No DLQ; ordering is lost (share groups)

Kafka Connect DLQ config (Kafka 4.2):

Config	Default	Notes
errors.tolerance	none	none: an error fails the task; all: skip failing records
errors.deadletterqueue.topic.name	empty	Empty means no DLQ
errors.deadletterqueue.topic.replication.factor	3	When Connect creates the DLQ topic itself
errors.deadletterqueue.context.headers.enable	false	Adds __connect.errors.* headers
errors.retry.timeout	0	0 no retry, -1 retry forever
errors.retry.delay.max.ms	60000	Maximum delay between two retries

Share groups are an alternative for independent jobs: many workers on one partition, RELEASE to redeliver, REJECT to stop, a configurable maximum delivery count. The Kafka 4.2 docs describe no DLQ for share groups; records rejected or out of delivery attempts still need a store you design.

When to use / not use

• Use retry topics: the downstream is down for a few minutes, events are independent or already have a state machine/versioning.
• Use in-place retry: short errors (deadlock, timeout); or per-partition ordering is an invariant and lag is acceptable.
• Use an immediate DLQ: data errors and code errors, where retrying does not change the outcome.
• Do not use retry topics for stateful events without a mechanism to prevent reordering.
• Consider RabbitMQ when the workload is a job queue needing built-in TTL, DLX and counted retries (RabbitMQ dead-letter).

Real-world example

Syncing orders to the warehouse system

A consumer reads orders (24 partitions, about 300 events/second) and calls the warehouse API. The warehouse API is under maintenance for 10 minutes every night and returns 503.

• Before: infinite in-place retry with a 1-second backoff. Every night lag reaches about 180,000 events (300 × 600 seconds), the whole topic stalls for 10 minutes, and some orders are processed late the next morning.
• After: 3 in-place retries (200 ms, with jitter), then orders.retry-1m, orders.retry-10m, and after 4 rounds orders.dlq. Other customers’ orders are not blocked.
• Ordering: the warehouse orders table has a version column; updates apply only when the version is newer, so a paid arriving before created is held for retry instead of being written incorrectly.
• Operations: alert when the DLQ gets new messages; a replay runbook by x-original-topic, keeping the event ID so the warehouse dedupes.

The numbers above are an illustrative scenario, not measurements.

Trade-offs

Retry topics + DLQ

• A failing message does not block other messages in the partition.
• Long backoff (minutes) without violating max.poll.interval.ms.
• Permanent errors have a place to live, with headers for investigation and replay.

The cost

• Same-key ordering is lost; needs a state machine or park-by-key.
• More topics, consumer groups, alerts and runbooks to operate.
• The delay is only a lower bound; not for precise scheduling.

🎮 Try it

• Kafka consumer lag keeps growing: a poison message blocks the partition, and retry topics.
• Scale Lab: notifications: retry with backoff + DLQ when the provider goes down.

Production checklist

1. Classify errors clearly: permanent errors go to the DLQ immediately; only transient errors take the retry loop.
2. One topic per delay tier; retry topics and the DLQ have at least as many partitions as the original topic if you keep the original partition.
3. Required headers: attempt, due-at, original topic/partition/offset, error; keep the key and value unchanged.
4. The retry consumer uses pause/resume, never a long sleep in the handler.
5. Publish to retry/DLQ successfully first, then commit the original message’s offset.
6. Stateful events: a state machine or versioned update to tolerate reordering; idempotent handlers.
7. Alert on new DLQ messages and on the oldest message’s age; the DLQ has an owner and long retention.
8. A rate-limited replay tool that keeps event IDs, tested on staging.

Common mistakes

Common pitfall

Committing the offset before publishing to the retry topic. A crash in between loses the message entirely: it is neither in the main topic nor in retry. Publish, wait for the producer ack, then commit.

Common pitfall

One retry topic for several delay levels. A 10-minute message at the head blocks a 5-second message behind it, because the consumer pauses based on the head message. One topic per delay.

Common pitfall

Retrying every exception. A wrong-schema payload goes through every retry tier, wasting minutes and resources, and still ends up in the DLQ. Permanent errors go to the DLQ immediately.

Common pitfall

A DLQ nobody reads. No alerts, default retention, and a few weeks later the messages are deleted. It is still data loss, just slower.

Summary

Key takeaways

• The committed offset is contiguous progress, so one poison message blocks the whole partition unless you move it elsewhere (retry topic, DLQ).
• Kafka’s consumer API has no built-in DLQ; Kafka Connect sinks have errors.deadletterqueue.*, Kafka Streams has errors.dead.letter.queue.topic.name.
• Retry topics: one topic per delay tier, the consumer pauses until due; the delay is a lower bound.
• Retry topics reorder same-key events; stateful events need a state machine, versioning or park-by-key.
• A DLQ needs complete headers, alerts, an owner, long retention and an idempotent replay process.

Self-check: (1) Why does committing offset 5 while offset 4 still fails amount to skipping offset 4? (2) Why is mixing 5-second and 10-minute delays in one retry topic wrong? (3) Replaying created after paid has been applied: what mechanism does the consumer need?

References

• Kafka Connect Configs — Apache Kafka 4.2
• KIP-298: Error Handling in Connect
• Non-Blocking Retries: How the Pattern Works — Spring for Apache Kafka
• Back Off Delay Precision — Spring for Apache Kafka
• Handling Exceptions — Spring for Apache Kafka
• Consuming Messages — KafkaJS
• Kafka Streams Configs — Apache Kafka 4.2
• Broker Configs — Apache Kafka 4.2