Kafka is a distributed log: topics, partitions and offsets Basic

The problem

The order service publishes three events for one order: created, paid, shipped. Billing reads them to charge, analytics reads them for reports. If you treat Kafka as a traditional queue (a consumer takes a message and it disappears), you cannot explain:

• Why both services read all three events.
• Why analytics can read again from the beginning.
• Why a slow consumer can lose data without anyone deleting it manually.

The right model is a distributed log: each partition is an append-only sequence of records, and each record has an offset. Reading does not delete records. The broker cleans up data per retention/compaction, regardless of whether consumers have read it.

Quick summary

Concept	In one sentence
Topic	The name of a record stream, for example orders
Partition	One ordered log; an offset only means something within a (topic, partition) pair
Offset	A record’s position in the partition, increasing, not a business counter
Consumer group	A set of consumers dividing partitions; each group keeps its own offsets
Retention	How long/how much data is kept, independent of consumers
Replication	Each partition has a leader and followers on different brokers (ISR article)

Anatomy of a topic

Loading diagram…

• Data is split into partitions to scale horizontally; each partition is replicated to other brokers.
• Ordering is only guaranteed within one partition; there is no topic-wide order. How a record’s partition is chosen: see message key.
• Within a group, each partition belongs to only one consumer at a time (consumer groups and rebalance).
• Other groups read independently, so billing and analytics do not affect each other.

The path of one record

The producer appends; each consumer reads and commits its own position

Enlarge diagram

Producer → Partition logorder-created

◀ Prev▶ PlayNext ▶↺ RestartStep 1/4

The producer sends a record with a key. The partitioner picks a partition; the demo has only one.

After three appends to a new partition, the log looks like this:

Offset	Value
0	created
1	paid
2	shipped

Once billing finishes offset 2 it commits 3 (the next offset to read). A new analytics group can still read from offset 0, as long as retention has not removed the data. Commit details are in the offset commit article.

Retention: Kafka does not wait for consumers

With cleanup.policy=delete (the default), Kafka deletes whole old segments per retention.ms (default 7 days) or retention.bytes (default unlimited). A consumer lagging beyond this window cannot reread the deleted part. Details and compaction: retention and compaction.

Design rule: the retention window must exceed the worst outage you can accept before consumers catch up.

Kafka 4.x, KRaft and a note on the demo

Since Kafka 4.0, ZooKeeper has been removed: Kafka runs only in KRaft mode (cluster metadata is managed by Kafka’s own controllers using Raft, with no separate ZooKeeper ensemble). New clusters should be built on KRaft; old ZooKeeper clusters must migrate before upgrading to 4.0.

The repo demo runs on the Confluent Platform 7.6 image (Kafka 3.6) with ZooKeeper, via examples/docker-compose.yml. Offsets, partitions and consumer groups behave the same across versions; only cluster setup differs.

The other articles in this series describe configuration for Kafka 4.x and say explicitly when a demo uses an older version.

When to use / not use

Use the Kafka log when	Consider another tool when
Many services need to read the same events independently	You only need to distribute jobs to workers, no replay (see RabbitMQ queue types or share groups)
You need replay to rebuild projections or investigate bugs	You need complex pattern-based routing, priority queues
High throughput, sequential writes	Very small volume and no need for retention

Real-world example

Billing stops for an hour

Topic orders has 6 partitions, retention 7 days (default). Billing and analytics are two groups. Billing stops for an hour for a deploy: events stay in the log, and billing resumes from its committed offset.

If billing writes to the DB and then crashes before committing the offset, it will read that event again, so write the ledger with a unique eventId (idempotent consumer). Kafka only manages the log and offsets; it does not know whether the side effect succeeded.

Run the single-broker demo (one partition, replication factor 1, only for looking at offsets; it does not prove multi-broker fault tolerance):

docker compose -f examples/docker-compose.yml up -d zookeeper kafka
npx --prefix examples ts-node examples/src/kafka-log/log-demo.ts

The script creates a new topic, writes created, paid, shipped, then a new group reads from the beginning and prints partition, offset, key, value (partition 0, offsets 0 to 2). KafkaJS JSON logs and TimeoutNegativeWarning mixed into the output are client noise.

Trade-offs

Benefits of a log

• Many consumer groups read the same data at their own pace.
• Replay within the retention window to rebuild projections or investigate bugs.
• Partitions raise throughput and keep per-key ordering.

Responsibility shifted to the application

• Ordering is only within a partition; key skew creates hot partitions.
• Retention does not wait for slow consumers.
• Offset commits and side effects outside Kafka are separate; idempotency or an outbox is needed.

Production checklist

1. Set replication.factor=3 and min.insync.replicas=2 for important topics (broker defaults are RF=1, min ISR=1).
2. Choose the partition count from throughput and maximum parallel consumers, because partitions cannot be reduced.
3. Set retention from the recovery SLA; do not keep the 7-day default without thinking.
4. Choose the key by the entity that needs ordering.
5. Monitor consumer lag per partition and over time.
6. Use KRaft for new clusters; do not plan around ZooKeeper.

Common mistakes

Common pitfall

“Once a consumer reads a message, Kafka deletes it.” Kafka keeps data per retention/compaction. Other groups can still read the same record while it is in the log.

Common pitfall

“Offset 10 is the 10th record of the whole topic.” Offsets belong to a partition. When debugging, always log topic, partition and offset together.

Common pitfall

“Committing the offset confirms the payment is done.” Kafka only knows the read position. Side effects need their own proof of success and idempotency.

Summary

Key takeaways

• A topic is split into partitions; each partition is a log with its own offsets and ordering.
• Reading does not delete; each consumer group has an independent read position.
• Retention/compaction decides what data is left to replay, independent of consumers.
• Kafka 4.0 dropped ZooKeeper; only KRaft remains.

Self-check: (1) Billing committed offset 3; can a new analytics group read offset 0, and what must be true?

(2) Why does using the same key help ordering but not guarantee the order of all events in the topic? (3) The DB was written but the offset was not committed, then the consumer crashed: what happens on restart?

References

• Design — Apache Kafka 4.2
• Topic Configurations — Apache Kafka 4.2
• Upgrade notes — Apache Kafka 4.2