Foundations
Distributed log: topic, partition, offset.
Message key and ordering: how far ordering is guaranteed.
Messaging Basic
Kafka is a log with read positions: reading does not delete records, many groups read independently, and data can be replayed. RabbitMQ is a router plus queues: the exchange decides which queues receive a message, and once acked the message leaves the queue.
The two models answer three questions differently: does a message still exist after it is read, who decides where a message goes, and what “sent successfully” means.
Kafka or RabbitMQ?
Section titled “Kafka or RabbitMQ?”| Need | Choose | Why |
|---|---|---|
| Replay, many independent consumers reading the same data | Kafka | The log is kept per retention, each group has its own offset |
| Per-key ordering at high throughput | Kafka | Ordering within a partition, the key picks the partition |
| Event streaming, CDC, stream processing | Kafka | Connect, Streams, compaction ecosystem |
| Work queue, one worker per job, per-job ack | RabbitMQ (quorum queue) | Per-message ack, DLX, prefetch |
| Flexible pattern-based routing (topic, headers) | RabbitMQ | Exchanges and bindings |
| Retry with backoff, per-message dead-lettering | RabbitMQ | DLX, TTL, x-delivery-limit |
| Need both queue-like and log semantics in one system | Consider Kafka Share Groups (4.2) or RabbitMQ streams | See Share Groups and queue types |
Rule of thumb: need replay or streaming, use Kafka; need work distribution and routing, use RabbitMQ. Both are at-least-once at the application level, so an idempotent consumer is always needed.
Kafka: a log with read positions
Section titled “Kafka: a log with read positions”Durability and progress
Replication, ISR, acks: when a write is acknowledged.
Offset commit: the gap between side effects and offsets.
Operations
Share Groups: many workers on one partition.
Consumer lag: measure, alert and diagnose.
Retry topics and DLQ: failing messages do not block the partition.
Preventing duplicate writes
Idempotent producer: dedupe retries of the same batch.
Transactions and exactly-once: the boundary stops at Kafka.
RabbitMQ: routing and acks
Section titled “RabbitMQ: routing and acks”| Article | Answers the question |
|---|---|
| Exchanges and bindings | Which queue does a message reach? Does an error-free publish mean it reached a queue? |
| Ack, prefetch, publisher confirms | How do the three acknowledgment boundaries differ? What is consumer_timeout? |
| Dead-lettering, TTL, counted retries | Where do failed jobs go, how many retries, where does a poison message stop? |
| Classic, quorum, stream | Which queue type for each flow? |
The boundary both share
Section titled “The boundary both share”The broker manages messages and delivery state; it does not know whether the service’s side effect succeeded. Write to the database, then crash before acknowledging, and the message is processed again; acknowledge before writing, and the work is lost.
Handling duplicates
Idempotent consumer: inbox table, unique key, one transaction. Works for both Kafka and RabbitMQ.
Dual write
Transactional outbox: write state and the intent to publish an event in one DB transaction. Prevents lost events; duplicates are still possible.
Broader context: the eight fallacies of distributed computing.
Read by problem
Section titled “Read by problem”| Situation | Read in this order |
|---|---|
| New to Kafka | Distributed log → Message key → Consumer group |
| New to RabbitMQ | Exchange → Ack and prefetch → DLX and TTL |
| An event is processed twice | Idempotent consumer → Offset commit |
| The producer reported success but data was lost | Replication, ISR, acks |
| Adding consumers does not raise throughput | Share Groups |
| Need to replay but the data is gone | Retention and compaction |
| Lag is rising for unknown reasons | Consumer lag → Consumer group |
| One failing message blocks a whole Kafka partition | Retry topics and DLQ → Idempotent consumer |
| RabbitMQ queue is full, jobs loop forever | Dead-lettering and counted retries |
| DB committed but the event never reached the broker | Transactional outbox |
| A producer changed the event shape and an older consumer broke | Event schema evolution |