Kafka retention and log compaction: keep history or keep the latest state? Intermediate

The problem

Service A rebuilds a table of current emails from topic user-email: it only needs the latest value per user. The audit service needs every email change. Both heard “Kafka stores a log, so you can replay”, but the cleanup policy decides what data actually remains.

Need	Policy
History over a time window (event log)	delete (default)
Latest state per key (snapshot, CDC state)	compact
Snapshot with a time cap	delete,compact
Full, permanent audit	Do not use compact; store it elsewhere

Segments: the unit of cleanup

A partition’s log is split into segment files. Kafka deletes or compacts per segment:

• The segment being written (active) is never compacted (docs: “the active segment will not be compacted”).
• With delete, retention also removes whole segments (“Retention and cleaning is always done a file at a time”).
• segment.ms exists to force segments closed “to ensure that retention can delete or compact old data”.

So a segment that is continuously receiving writes is usually not deleted yet; how an expired active segment is handled when retention runs is an implementation detail that varies by version.

Loading diagram…

A segment closes when it reaches segment.bytes (default 1 GiB) or segment.ms (default 7 days). A small, rarely written topic may not have closed any segment yet: a common reason behind “why hasn’t retention/compaction run?”.

Config cheat sheet (defaults per the 4.2 docs)

Config	Default	Meaning
cleanup.policy	delete	delete, compact or both
retention.ms	604800000 (7 days)	Maximum time to keep a segment
retention.bytes	-1 (unlimited)	Size cap per partition
segment.bytes	1073741824 (1 GiB)	Segment size
segment.ms	604800000 (7 days)	Force-close segments by time
min.cleanable.dirty.ratio	0.5	Minimum “dirty” ratio before the cleaner compacts
min.compaction.lag.ms	0	A record must be at least this old to be compacted
max.compaction.lag.ms	very large (effectively unlimited)	Upper bound on waiting for compaction
delete.retention.ms	86400000 (1 day)	How long tombstones are kept
remote.storage.enable	false	Enables tiered storage (see end of article)

Two properties to remember: the actual deletion time does not match each record’s expiry to the millisecond, and retention does not wait for slow consumers.

delete retention

A closed segment is deleted when retention.ms expires or the topic exceeds retention.bytes. A consumer lagging beyond this window cannot read the old data, even though it never committed past it.

Rule: the retention window must be longer than the worst outage plus catch-up time. Monitor lag in time (age of the oldest unprocessed record), not only in record count.

compaction

With compact, Kafka keeps at least the latest version of each key in the partition. The cleaner runs asynchronously and only compacts when the dirty (not yet compacted) portion exceeds min.cleanable.dirty.ratio.

Loading diagram…

• Offsets are not renumbered: offset 2 stays 2; the readable sequence has gaps.
• A caught-up consumer may see both the old and new versions; the guarantee is that reading from the start rebuilds the final state per key.
• A tombstone is a record with a key and a null value (not the string 'null'). The cleaner removes it after delete.retention.ms (1 day): a consumer bootstrapping slower than this window can miss the tombstone.
• The key must be stable (userId, not eventId); otherwise every event is its own key and compaction produces no snapshot.

When to use / not use

Use compact	Do not use compact
Per-key state: CDC snapshots, configuration, cache rebuilds	Audit trails, event sourcing that needs every change
__consumer_offsets (Kafka itself uses compact)	Events without a stable key
New consumers that need to bootstrap quickly from the final state	You need to know when A changed to B

Tiered storage (remote.storage.enable, default false) moves old segments to external storage to keep long retention at low broker cost; it is enabled per topic, and you should check support in the version you use.

Real-world example

A user profile projection

Topic user-profile-state is compacted, keyed by userId, and used to rebuild a cache after deploys; older changes may be cleaned up, but the projection only needs the latest version. A separate audit flow keeps retention/archival per legal requirements.

When a user is deleted, the producer publishes a tombstone. The team makes sure bootstrapping consumers read past tombstones within the default 1 day of delete.retention.ms, and tests recovery on anonymized data. They do not use the compacted topic as an audit log.

An in-memory model of the final state (it does not simulate the real cleaner):

npx --prefix examples ts-node examples/src/kafka-compaction/compaction-model.ts

Output: four records at offsets 0–3; the latest per key is 2:user-1=new@example.test and 3:user-2=null; applying the tombstone leaves user-1=new@example.test. On a real broker, compaction does not run immediately and tombstones have their own retention.

Trade-offs

Choosing the policy by need

• Delete: predictable size, fits a windowed event log.
• Compact: rebuild the latest state without keeping every change.
• Offsets of remaining records stay stable.

Limits

• Delete can remove data before a slow consumer reaches it.
• Compact keeps no history and does not run instantly.
• Tombstones and delete,compact require accounting for the bootstrap window.

Production checklist

1. Choose the policy from the need: history (delete) or snapshot (compact).
2. Set retention.ms from the recovery SLA; do not keep the 7-day default without thinking.
3. Compact: stable keys, backward-compatible value schema.
4. Compact: bootstrapping consumers must finish reading within delete.retention.ms.
5. Alert on the age of the oldest unprocessed record relative to retention.
6. Audit data that must be kept long goes to a separate store or tiered storage.

Common mistakes

Common pitfall

Using a compacted topic as an audit log. Older versions of the same key will be cleaned up.

Common pitfall

Assuming offsets are renumbered after compaction. Offsets do not change; the readable sequence has gaps.

Common pitfall

Using the string 'null' as a tombstone. A delete marker must be a null value at the record level.

Summary

Key takeaways

• Delete removes segments per retention.ms (7 days) or retention.bytes, not per consumer.
• Compact keeps the latest state per key; the active segment is never compacted.
• Tombstones live for delete.retention.ms (1 day) and are then removed.
• Choose the policy from the need: replaying history or restoring a snapshot.

Self-check: (1) Why do offsets have gaps after compaction? (2) You need to audit every email change; is a compacted topic enough? (3) If the producer uses eventId as the key, is per-user compaction still effective?

References

• Log Compaction — Apache Kafka 4.2
• Topic Configurations — Apache Kafka 4.2
• Broker Configurations — Apache Kafka 4.2