Kafka consumer groups and rebalance: eager, cooperative and KIP-848 Intermediate

The problem

You add instances to the service reading topic orders to speed it up, but lag spikes during the rebalance before it drops. If an autoscaler keeps adding and removing consumers, the group is almost always rebalancing and throughput ends up lower than before scaling.

Rebalance is normal: the group redistributes partitions when members join, leave, crash, or the partition count changes. The operational problem is its frequency and duration.

Protocol summary

	Classic eager	Classic cooperative	KIP-848 (consumer)
Enabled by	Default in many clients (including KafkaJS)	CooperativeStickyAssignor assignor (since Kafka 2.4)	group.protocol=consumer (opt-in, GA since 4.0)
On a change	Every consumer releases all partitions	Only partitions that must move are revoked, in two rounds	Incremental via heartbeats, no group-wide barrier
Who computes the assignment	Group leader (client)	Group leader (client)	Broker (server-side assignor)
Disruption	Whole group	Only the partitions changing owner	Only the partitions changing owner

The Java consumer defaults partition.assignment.strategy to RangeAssignor, CooperativeStickyAssignor, and group.protocol defaults to classic (per the 4.2 consumer configs docs).

Consumer B joins: three ways to handle it

◀ Prev▶ PlayNext ▶↺ RestartStep 1/4

Consumer B joins a group that is running steadily. The three rows compare three ways of handling it.

Consumer group rules

• Each partition in a group has one consumer at a time; the maximum number of useful consumers equals the partition count.
• Extra consumers remain members but get no work; adding consumers cannot split a hot partition.
• Another group reading the same topic has its own assignment and offsets.
• Keys, offsets and commits: see message key and offset commit. After a rebalance, the new consumer resumes from the group’s committed offset; work processed but not committed before the handover is delivered again, so side effects must be idempotent.

Observing an eager rebalance with KafkaJS

The demo (examples/src/kafka-rebalance/) uses KafkaJS (eager) against the repo compose’s Kafka 3.6 broker, topic orders with 3 partitions, group billing-service.

Rebalance when a new consumer joins the group

Enlarge diagram

Partition 0 · Partition 1 · Partition 2 · Consumer A

◀ Prev▶ PlayNext ▶↺ RestartStep 1/6

Consumer A starts alone in the "billing-service" group → it is assigned all 3 partitions.

Only Consumer A:

[consumer-A] GROUP_JOIN — memberId=kafkajs-1a6a...4910, assigned partitions: [0, 1, 2]

Consumer B joins the same group:

[consumer-A] GROUP_JOIN — memberId=kafkajs-1a6a...4910, assigned partitions: [2]
[consumer-B] GROUP_JOIN — memberId=kafkajs-0a70...975c, assigned partitions: [0, 1]

State	P0	P1	P2
Only A	A	A	A
During eager rebalance	revoked	revoked	revoked
After B joins	B	B	A

During the rebalance, messages stay in Kafka; lag rises because producers keep writing. The GROUP_JOIN log does not show how long the disruption lasted; measure lag and the gap between two processing events.

KIP-848: the new protocol (Kafka 4.0+)

Per the Kafka 4.2 docs, this protocol is GA since 4.0 and enabled by default on the server, but the client must opt in:

• Enable it with group.protocol=consumer on the Java consumer; it is not the default.
• The broker computes the assignment (group.consumer.assignors, default uniform and range; the client picks one with group.remote.assignor).
• Heartbeat and session are server-controlled: group.consumer.heartbeat.interval.ms=5000, group.consumer.session.timeout.ms=45000.
• The configs session.timeout.ms, heartbeat.interval.ms, partition.assignment.strategy cannot be used once consumer is chosen.
• Limitations listed in the docs: no client-side assignors yet, rack awareness is incomplete.
• Online migration: the group switches from classic to consumer when the first consumer using the new protocol joins (only if the classic assignor does not embed custom metadata).

Recommendation: Java services on Kafka 4.x should try group.protocol=consumer in staging. For non-Java clients (KafkaJS, for example), check that the library supports it before planning.

Config cheat sheet (classic protocol, Java client)

Config	Default	Meaning and recommendation
session.timeout.ms	45000	A member without heartbeats for too long is removed from the group
heartbeat.interval.ms	3000	Usually no more than 1/3 of the session timeout
max.poll.interval.ms	300000	Exceed it between two poll() calls and the member is removed; slow handlers cause rebalance storms
max.poll.records	500	Lower it so each poll loop finishes within max.poll.interval.ms
group.instance.id	null	Set it for static membership: a fast restart within the session timeout does not trigger a rebalance
partition.assignment.strategy	Range, CooperativeSticky	Use cooperative to avoid stopping the whole group

KafkaJS has no max.poll.interval.ms; long handlers must keep heartbeating within the session timeout.

Rebalance storm: a slow handler gets kicked out of the group

1. A consumer fetches 500 records, and each takes longer to process than expected.
2. The gap between two poll() calls exceeds max.poll.interval.ms; the consumer is considered failed and removed.
3. Its partitions move to another consumer, which is just as slow and is removed too.
4. The group loops in rebalance, lag grows, and uncommitted records are processed again and again.

Fixes: lower max.poll.records, set max.poll.interval.ms to the worst-case processing time, move heavy work to a separate thread while continuing to poll(), and measure per-batch processing time.

When to use / not use

• Cooperative or KIP-848: the default for large groups or autoscaling.
• Static membership: workloads on Kubernetes rolling updates, where pods restart quickly.
• Eager: only when the client supports nothing else; accept the group-wide pause.
• Share group (separate article): when you need more workers than partitions for independent jobs.

Real-world example

Adding consumers without reducing lag

A payment flow keyed by merchantId, a topic with 3 partitions (illustrative assumption), 2 consumers. The team scales to 4 consumers: the fourth gets no partition, and a large merchant still lands on one partition, so that partition’s lag does not change.

Before adding instances, measure lag per partition, then decide whether to add partitions or change the key.

Trade-offs

Pros

• Spread load by adding consumers while partitions remain.
• Preserves record order within a partition.
• When a consumer dies, the group moves its partitions to the remaining consumers.

Cons

• Eager rebalance stops the whole group; measure it and include it in the latency budget.
• The number of useful consumers is capped by the partition count.
• Slow handlers exceeding max.poll.interval.ms cause rebalance storms.

Production checklist

1. Use CooperativeStickyAssignor (or KIP-848 on 4.x); avoid eager.
2. Set a stable group.instance.id for workloads that restart often (Kubernetes).
3. Keep max.poll.interval.ms above the worst-case batch processing time; lower max.poll.records if needed.
4. Commit offsets in onPartitionsRevoked (offset commit).
5. Alert on rebalances per hour and per-partition lag.
6. Do not autoscale consumers beyond the partition count.

Common mistakes

Common pitfall

Adding partitions without accounting for key movement. The hash(key) mod N mapping changes, so old and new records of the same key land in two partitions. See message key.

Common pitfall

Adding consumers beyond the partition count. Extra consumers sit idle and do not raise throughput.

Common pitfall

Ignoring max.poll.interval.ms. Slow handlers are removed from the group, causing a continuous rebalance loop.

Summary

Key takeaways

• Each partition in a group has only one consumer; the maximum number of useful consumers equals the partition count.
• Eager stops the whole group; cooperative and KIP-848 move only what must change.
• KIP-848 is GA since Kafka 4.0, opt-in via group.protocol=consumer; classic is still the default.
• Rebalance storms usually come from slow handlers exceeding max.poll.interval.ms.

Self-check: (1) With three partitions and four consumers in one group, how many consumers have work? (2) How many rounds does a cooperative rebalance need, and why? (3) Once group.protocol=consumer is set, does the client’s session.timeout.ms still have any effect?

References

• KIP-848 consumer rebalance protocol — Apache Kafka 4.2
• Consumer Configs — Apache Kafka 4.2
• Consuming messages — KafkaJS
• Confluent — Kafka Rebalance Protocol