Scenario: Kafka consumer lag climbs mid-campaign Intermediate

Four incidents hit one consumer group back to back on campaign night. Each step has a fix “everyone has tried once” and a fix that is actually right.

• Read the CLI output and metrics carefully before choosing: the answer is usually in individual partitions, not in the total.
• If you pick wrong, read the explanation and Choose again. Only your first choice at each step counts toward the score.

Scenario

11.11 campaign: order-events lag keeps growing

You are on call for the fulfillment service, which reads the order-events topic (12 partitions, 12 pods) and calls the warehouse API. Every choice immediately shows its consequences on the metrics below.

Time lag of worst partition17 min

Processing rate (producer writes 2,400/s)1,800 events/s

Rebalances per hour0

Events silently skipped0

Step 1/4 · Lag is growing, but where?

20:10, the campaign started 30 minutes ago. Alert: orders reach the warehouse 17 minutes late, against a 1-minute SLO. The producer writes 2,400 events/s; each pod handles about 400 events/s, 4,800/s for the whole group, plenty of headroom. You run kafka-consumer-groups.sh --describe.

message key

merchantId

merchant M-001 (flagship)

38% of campaign traffic

partition 7 receives

~1,000 events/s, consumer handles 400/s

PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG      CONSUMER-ID
6          1204511         1205380         869      fulfillment-6-…
7          3880120         4291402         411282   fulfillment-7-…
8          1198004         1198760         756      fulfillment-8-…
... remaining 9 partitions: LAG < 1,000

What do you do?

AScale the deployment from 12 to 24 podskubectl scale deploy/fulfillment --replicas=24BIncrease order-events partitions from 12 to 48 right nowkafka-topics.sh --alter --topic order-events --partitions 48CChange the key from merchantId to orderId, since the business only needs ordering within each orderDTotal group lag is about 420,000 and the group handles 4,800/s, so wait 2–3 minutes and it drains

When you finish, read Consumer lag in Kafka for the full diagnostic tree and how to alert on time lag.