Consumer lag in Kafka: measuring, alerting and fixing Intermediate

The problem

The dashboard shows lag = 40 for group billing. The number is small and nobody notices. In reality, the oldest record has waited 90 seconds because the database behind it is timing out, and customers see late invoices.

Another day the dashboard shows lag = 50,000 at 9 a.m., and on-call gets paged. Ten seconds later lag is back to 0: the producer just wrote a burst batch.

Both are the same mistake: measuring lag in offsets and alerting on a fixed threshold. This article shows how to measure correctly, alert correctly and diagnose the cause.

🎮 Try it

• Kafka consumer lag keeps growing: hot partition, rebalance storm, poison message.

Quick summary

Question	Short answer
What is lag?	Per partition: log-end offset minus the group’s committed offset
Unit?	An offset distance, not exactly a record count (compaction, transaction markers)
Where to measure?	Externally, from committed offsets (CLI, Admin API, exporter); client metrics are supplementary
SLI to use	Time lag: age of the oldest uncommitted record
Alert when	Time lag exceeds the SLO for a sustained period, or lag grows continuously; on the worst partition
Adding consumers helps when	Some partitions have no consumer and the bottleneck is consumer CPU

How lag is computed

Each partition has three positions worth knowing:

Position	Meaning
High watermark (HW)	The offset after the last fully replicated record; read_uncommitted consumers can read up to here
Last stable offset (LSO)	For read_committed: stops before the first open transaction (LSO)
Committed offset	The offset the group reads next, committed by the consumer (offset commit)

lag = end offset − committed offset, computed per partition. Group lag is the sum, but the sum hides the worst partition.

Log-end offset and committed offset over time (illustrative numbers)

◀ Prev▶ PlayNext ▶↺ RestartStep 1/6

A partition's lag = log-end offset minus committed offset. At first the consumer keeps up: 0 offsets, 0 seconds.

Two shapes to remember from the diagram:

• Large offset count, small age: a burst being drained. Usually harmless.
• Small offset count, large age: a consumer stuck on a low-traffic partition. This is an incident.

Reading lag with kafka-consumer-groups.sh

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group my-group

Per the ConsumerGroupCommand source (4.1 branch), the printed header is:

GROUP  TOPIC  PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID  HOST  CLIENT-ID

The example in the Kafka 4.2 docs omits the GROUP column; real output has it. Adding --verbose adds a LEADER-EPOCH column.

Column	How to read it
CURRENT-OFFSET	The group’s committed offset for that partition
LOG-END-OFFSET	The tool asks for the latest offset via the Admin API with the default read_uncommitted isolation, i.e. the high watermark
LAG	The difference of the two columns above
CONSUMER-ID = -	No member holds the partition: the group is empty or rebalancing
CURRENT-OFFSET = -	The group has never committed for this partition

Other useful options (per the 4.2 docs):

• --members --verbose: which member holds which partition; spots members with no work.
• --state: the group STATE (Stable, rebalancing…) and #MEMBERS.
• --reset-offsets: only run when consumers are stopped; by default it only prints, --execute is needed to apply.

The CLI suits manual investigation. Dashboards and alerts need an exporter that reads the same data (committed offset + end offset) via the Admin API on a schedule.

Consumer client metrics

MBean kafka.consumer:type=consumer-fetch-manager-metrics (Java client, per the 4.2 Monitoring docs):

Metric	Scope	Meaning
records-lag-max	client	The largest lag (in records) of any partition within the window
records-lag, records-lag-avg, records-lag-max	per partition (tags topic, partition)	Latest / average / max lag of the partition
records-lead-min, records-lead	client / partition	Distance from the read position to the start of the log; near 0 means about to be deleted by retention
records-consumed-rate	client	Records/second received by the consumer

The MBeans consumer-coordinator-metrics and consumer-metrics cover the “why”:

Metric	Used for
rebalance-rate-per-hour, rebalance-total	Detecting rebalance storms
time-between-poll-max, last-poll-seconds-ago	Slow handlers about to exceed max.poll.interval.ms
poll-idle-ratio-avg	Near 0: the consumer spends almost all its time in processing code, not waiting for data
commit-latency-avg, commit-latency-max	Slow commits

Three differences from CLI lag:

1. The docs state that records-lag-max is computed from the current position, not the committed offset. The position is how far the client has fetched, which can be ahead of what has finished processing.
2. With isolation.level=read_committed, the client computes lag up to the LSO; with read_uncommitted, up to the high watermark (per SubscriptionState.partitionLag).
3. The metric only updates when the client fetches. If the consumer dies or is stuck in a handler, the metric freezes or disappears, exactly when you need it most.

Conclusion: alert on lag computed from committed offsets outside the consumer; use client metrics for diagnosis.

Why record-count lag misleads

Situation	Offset lag says	Reality
Producer writes a burst	Very large	Drained within seconds
Low-traffic partition, stuck consumer	Very small	Records have waited minutes
Compacted topic	Larger than the real record count	Offsets have gaps after compaction (compaction)
Topic with transactions	Never exactly 0	Control markers take offsets; an open transaction holds back the LSO
read_committed consumer, long-open transaction	CLI (measuring to HW) sees lag rising	The consumer is waiting on the LSO, not slow
Partitions with different throughput	A shared threshold is meaningless	1,000 offsets is 1 second on one partition, 1 hour on another

An in-memory simulation (no Kafka): each second the producer writes, then the consumer processes at a fixed capacity:

npx --prefix examples ts-node examples/src/kafka-lag-sim/lag-sim.ts

Excerpt of real output:

t(s) | partition              | log-end | committed | lag(offset) | lag(s)
  10 | P1 burst               |    6500 |      1100 |        5400 |      0
  15 | P1 burst               |    6750 |      4100 |        2650 |      5
  20 | P1 burst               |    7000 |      7000 |           0 |      0
  60 | P2 low traffic, stuck  |       7 |         1 |           6 |     50

P1 receives a 6,000-record burst at second 10 and drains it before second 20. P2 has a handler stuck since second 5: lag is only 6 offsets, but the oldest record has waited 50 seconds. A “lag > 1,000” threshold fires falsely on P1 and stays silent on P2.

Time lag: the SLI to use

A partition’s time lag = now − timestamp of the record at the committed offset (the oldest unprocessed record). It is 0 when offset lag is 0.

It answers the question users actually care about: “how late are events?”. It is comparable between fast and slow partitions, and maps directly to an SLO (SLI/SLO).

Getting the timestamp:

• The exporter reads the committed offset, then fetches that exact record to get its timestamp. Accurate, but costs one read per partition per cycle.
• Or interpolate: store (log-end offset, time) pairs periodically, then estimate which time the committed offset corresponds to. Cheaper, less accurate when traffic is irregular.

Notes on timestamps:

• message.timestamp.type defaults to CreateTime: the producer sets the timestamp, so it depends on the producer clock and may not increase monotonically within a partition.
• The broker rejects records whose timestamp is ahead of the broker clock by more than message.timestamp.after.max.ms (default 1 hour); the past direction (before.max.ms) is unlimited by default.
• If you need a more accurate clock, use LogAppendTime on the topic, at the cost of losing the original event creation time.

Also measure end-to-end latency in the consumer (now − record.timestamp() when processing finishes) as a histogram. It complements time lag but does not replace it: a stuck consumer processes no records, so the histogram goes silent.

Alerting: three rules

Rule	Condition	Catches
Time lag above SLO, sustained	The worst partition’s time lag stays above the threshold for N minutes	Slow or stuck consumers, including low-traffic partitions
Lag growing continuously	The lag derivative (offsets/second) is positive over a long window, e.g. 10–15 minutes	Consumer capacity below the write rate, before the SLO is breached
No progress	The committed offset does not change while the log-end offset grows	Dead consumer, empty group, poison message blocking the partition

To reduce noise:

• Take the max per partition, not the sum or average. Group by group, topic; keep the partition label only when the partition count is small (cardinality).
• Always add a sustained condition (for: a few minutes). Short bursts resolve themselves.
• Set the time lag threshold from the business SLO, not from a number that “looks big”.
• Add a separate alert for small lead: time lag approaching retention.ms means unread data is about to be lost.

Sketch (metric names depend on your exporter):

# Pseudo-PromQL: replace metric names to match your exporter
- alert: KafkaConsumerTimeLagHigh
  expr: max by (group, topic) (consumer_group_time_lag_seconds) > 60
  for: 5m
- alert: KafkaConsumerNoProgress
  expr: |
    max by (group, topic, partition) (delta(consumer_group_committed_offset[10m])) == 0
    and max by (group, topic, partition) (delta(topic_partition_end_offset[10m])) > 0
  for: 5m

Diagnosis: what to look at when lag rises

Loading diagram…

Common causes and fixes

Cause	Signal	Fix
Slow handler	Low poll-idle-ratio-avg, all partitions evenly slow	Profile the handler; batch DB writes; drop work that need not be synchronous
Rebalance storm	High rebalance-rate-per-hour, CONSUMER-ID often empty	Cooperative / group.protocol=consumer, static membership (rebalance)
Exceeding max.poll.interval.ms (default 5 minutes)	time-between-poll-max hits the limit, members removed repeatedly	Lower max.poll.records (default 500), raise the interval to the worst batch
Hot partition / key skew	One partition lags, others at 0	Change the key or split hot keys (message key)
More consumers than partitions	--members --verbose shows members with no partitions	Add partitions (mind the key mapping) or remove consumers
Saturated downstream DB/API	DB latency rises at the same time, timeout errors	Limit concurrency, fix queries/indexes; do not scale consumers
Poison message	The committed offset is frozen on one partition	Cap retries, send to a dead-letter topic, log the skipped offset
Long-open transaction (read_committed)	CLI lag rises, consumer not busy	Find the producer holding the transaction, check transaction.timeout.ms

When adding consumers does not help

Adding consumers only raises throughput when both conditions hold: some partitions have no dedicated consumer, and the bottleneck is the consumer’s own CPU/IO. Otherwise:

• Consumers already equal partitions: the new consumer gets no partition.
• Hot partition: one partition has only one consumer in the group; its lag does not change.
• Saturated downstream: more consumers mean more connections and queries against an overloaded DB, and lag grows faster.
• Rebalance storm: every new member is one more rebalance.
• Poison message or open transaction: the partition is blocked by a record or the LSO, not a lack of workers.

For more workers than partitions on independent jobs, see share groups.

Real-world example

A total-lag alert fires falsely and misses the real incident

Topic invoice-events with 12 partitions, group billing with 12 consumers. Old alert: total lag above 10,000 for 1 minute.

• Every morning a reconciliation job writes about 80,000 events in a few seconds; the alert fires and clears itself within a minute.
• One day partition 7 gets an event with a broken payload, and the handler retries forever. That partition receives only a few dozen events per minute, so total lag stays under 10,000. Invoices for customers on partition 7 are three hours late.

The team switches to two alerts: worst-partition time lag above 2 minutes sustained for 5 minutes, and committed offset not advancing for 10 minutes. The handler caps retries at 5, then pushes to a dead-letter topic. (Numbers are an illustrative scenario.)

Trade-offs

Alerting on time lag

• Maps directly to the business SLO (“how late are events”).
• Comparable between fast partitions and low-traffic ones.
• No false alarms when the producer writes a burst.

The cost

• Needs an exporter that reads timestamps or interpolates; not available in client metrics.
• Depends on producer timestamps when using CreateTime.
• Per-partition labels increase the number of series to store.

Production checklist

1. Measure lag outside the consumer from committed offsets (an exporter via the Admin API), not only from the client’s records-lag-max.
2. Alert on the worst partition’s time lag, with a sustained condition; take the threshold from the SLO.
3. Add a no progress alert: the committed offset is frozen while the log-end offset grows.
4. The dashboard shows: per-partition lag, rebalance-rate-per-hour, time-between-poll-max, poll-idle-ratio-avg, downstream latency.
5. Track time lag against retention.ms so unread data is not lost.
6. Cap retries per record and have a dead-letter topic, so one bad record does not block the whole partition.
7. Before scaling consumers: check the partition count, key skew and downstream load.

Common mistakes

Common pitfall

Alerting on the group’s total lag. The sum hides stuck partitions and exaggerates bursts. Use the per-partition max, measured in time.

Common pitfall

Relying only on consumer client metrics. When the consumer dies, the metric disappears; the dashboard looks “clean” right during the incident. Always have a lag source from broker-side committed offsets.

Common pitfall

Autoscaling consumers on lag. If lag rises because of a slow downstream or a hot partition, scaling is useless or harmful, and each scale triggers a rebalance. Cap replicas at the partition count.

Summary

Key takeaways

• Lag = end offset − committed offset, per partition; the unit is offsets, not exactly records.
• kafka-consumer-groups.sh --describe measures to the high watermark; the client’s records-lag measures from the position, to the LSO under read_committed.
• Record-count lag fires falsely on bursts and stays silent when a low-traffic partition is stuck.
• The SLI to use is the worst partition’s time lag, alerted with a sustained condition plus a “no progress” alert.
• Adding consumers only helps when partitions are free and the bottleneck is the consumer itself.

Self-check: (1) What does a partition with 5 offsets of lag but a 20-minute-old oldest record tell you? (2) Why can records-lag-max be 0 while the CLI still shows lag? (3) The downstream DB is at 100% CPU; will adding consumers reduce lag?

References

• Basic Kafka Operations — Apache Kafka 4.2
• Monitoring — Apache Kafka 4.2
• Consumer Configs — Apache Kafka 4.2
• Topic Configs — Apache Kafka 4.2
• ConsumerGroupCommand.java — Apache Kafka (GitHub)