Transactional outbox: the DB committed but the event never reached the broker Intermediate

The problem

The order service updates orders.status = 'paid' and then publishes OrderPaid to Kafka. The database commits, and the process dies before calling the broker: the order is paid but Shipping never receives the event.

Reversing the order also fails: publish first, then the DB rolls back, and Shipping processes an order that was never paid.

This is a dual write: one request writes to two systems with independent transactions. try/catch and in-process retries cannot close the gap, because the process itself can die.

Transactional outbox: in one DB transaction, change the order status and insert a row into an outbox table. A separate relay reads pending rows and publishes them.

Both rows commit or roll back together, so there is no longer a state of “paid, but nobody remembers to publish the event”. The outbox prevents lost events, not duplicate ones.

One DB transaction writes the order and the outbox; the relay publishes the event later

Enlarge diagram

App → DB + OutboxCOMMIT

◀ Prev▶ PlayNext ▶↺ RestartStep 1/4

The app writes the order state and the outbox event in the same DB transaction.

Schema

CREATE TABLE outbox (
  id           bigserial   PRIMARY KEY,
  event_id     uuid        NOT NULL UNIQUE,   -- stable across every retry
  aggregate_id text        NOT NULL,          -- used as the message key
  event_type   text        NOT NULL,
  payload      jsonb       NOT NULL,
  created_at   timestamptz NOT NULL DEFAULT now(),
  published_at timestamptz                    -- NULL = pending
);

-- partial index: only pending rows are indexed, so it stays small
CREATE INDEX outbox_pending_idx ON outbox (id) WHERE published_at IS NULL;

Written in the same transaction as the business change:

BEGIN;
UPDATE orders SET status = 'paid' WHERE id = 42;
INSERT INTO outbox (event_id, aggregate_id, event_type, payload)
VALUES (gen_random_uuid(), 'order-42', 'OrderPaid', '{"orderId":42}');
COMMIT;

The payload is a public contract across the broker: do not serialize the raw ORM entity, and do not leak secrets.

Polling relay with SKIP LOCKED

Several relay instances can run together if each claims a different batch of rows. FOR UPDATE SKIP LOCKED skips rows locked by another relay (see NOWAIT and SKIP LOCKED).

BEGIN;
SELECT id, event_id, aggregate_id, event_type, payload
  FROM outbox
 WHERE published_at IS NULL
 ORDER BY id
 LIMIT 100
   FOR UPDATE SKIP LOCKED;
-- publish this batch to the broker, wait for ack/confirm
UPDATE outbox SET published_at = now() WHERE id = ANY($1);
COMMIT;

Run on PostgreSQL 17 (scratch database) with 5 rows: SELECT ... LIMIT 3 FOR UPDATE SKIP LOCKED returns the first three rows by id, and after the UPDATE 2 rows remain pending.

A note on the transaction: keeping it open while publishing means holding row locks and a connection. Keep batches small and put a timeout on the publish. If the relay dies, the transaction rolls back, the locks are released, and another relay picks up the pending rows.

Ordering. SKIP LOCKED lets relay B take later rows while relay A holds earlier ones, so two events of the same aggregate can be published out of order.

Two options: run a single relay per shard (e.g. hash(aggregate_id) mod N), or have the consumer check a per-aggregate version. created_at does not reflect commit order, so use id or a version.

Polling or CDC (Debezium)

	Polling publisher	CDC (Debezium Outbox Event Router)
How it reads	Queries the outbox table	Reads the WAL, turns outbox rows into events
Latency	The poll interval plus publish time	Lower, near real time
DB load	Repeated queries (light with a partial index)	A replication slot; a stuck slot bloats the WAL
Extra infrastructure	Only relay code	Kafka Connect, Debezium, replication permissions
Needs a published_at column	Yes	No; the Debezium docs say the SMT filters out DELETE operations on the outbox table, so rows can be deleted right after insert (check against your connector configuration)
Ordering	You must design it (table above)	Follows the connector’s log read order; aggregateid as the message key keeps order within a Kafka partition
When it fits	Moderate scale, fewer moving parts	High throughput, low latency needed, Kafka Connect already in place

This article has no latency or throughput measurements because they depend on workload and configuration; measure with your own system. Both approaches are at-least-once.

Stop points and recovery

Where the process stops	DB order	Outbox	Broker	Recovery
Before the DB commit	Not paid	None	None	The request retries per business rules
After commit, before the relay	Paid	Pending	Not yet	The relay reads pending rows and publishes
After the broker received it, before mark sent	Paid	Pending	Possibly already there	The relay publishes again; the consumer dedupes
After mark sent	Paid	Sent	Received	Monitor the consumer separately

Only mark sent after a broker signal: the Kafka send ack per acks, or a RabbitMQ publisher confirm.

Even so, the gap between publishing and dying before the mark always remains, so consumers must be idempotent by event_id. Kafka’s idempotent producer cannot save this case because the new relay is a new producer.

Cleaning up the outbox

The outbox table only grows. Pending rows are never deleted for being “old”.

Strategy	How	When
Batch deletes	DELETE ... WHERE published_at < now() - interval '7 days' in small batches	Low to moderate volume
Daily partitions	DROP PARTITION for old sent rows	High volume: no dead tuples, no heavy vacuum
CDC	Rows can be deleted right after insert (Debezium filters out DELETE); the log still has the insert record	When using Debezium

Keep sent rows long enough for investigation and replay (e.g. a few days; the exact number is your choice). Monitor: age of the oldest pending row, retry counts, and latency from commit to broker.

In-memory model demo

npx --prefix examples ts-node examples/src/transactional-outbox/outbox-model.ts

The script prints three states. After “DB commit”: the order is paid, the event pending. After a relay crash: the simulated broker has received one event but the row is still pending. After a retry: the published array contains event-42 twice, and the outbox row is marked sent.

It shows that the outbox prevents losing the publish request but not duplicates. The Map in the demo is not a real transaction; the SQL above is the real protection.

Three tests worth writing: roll back right after the outbox insert (both order and event disappear); block the broker after commit and then reopen it (the relay must publish); crash after the ack, before mark sent (the consumer still applies it once).

Real-world example

Checkout publishes OrderPaid

Checkout writes the order and an OrderPaid outbox row in one PostgreSQL transaction. Two relays run SKIP LOCKED in batches of 100, with orderId as the Kafka key.

• Kafka is down for 20 minutes: payment requests still complete, the outbox builds a backlog, and the relays drain it when Kafka returns.
• Alert when the oldest pending row is older than 60 seconds; the dashboard shows the latest few failing rows with PII masked.
• Shipping creates jobs with a unique eventId, so a duplicate publish from the relay does not create two jobs.
• If product requires Shipping to start synchronously right away, an asynchronous outbox does not meet that need.

When to use / not use

• Use: you need to update the DB and publish an event, and losing the event is unacceptable.
• Not needed: losing events is acceptable (logs, metrics), or the event is written straight to Kafka by one system within a Kafka transaction.
• Not enough: you need a global order, or a synchronous response from the consumer.

🎮 Try it

• Asynchronous checkout: outbox, broker, inbox and the dual write and 2PC traps.
• Scale Lab: notifications: queue, worker, retries when the provider goes down.

Production checklist

1. Insert into the outbox in the same transaction as the business change.
2. event_id generated once, unique, stable across retries.
3. A partial index on pending rows.
4. Claim with FOR UPDATE SKIP LOCKED in small batches; publish with a timeout.
5. Mark sent only after the broker ack/confirm.
6. Per-aggregate ordering: shard the relays or use a version in the consumer.
7. Idempotent consumers.
8. Cleanup: batch deletes or dropped partitions; monitor pending age, retries, latency.

Common mistakes

Common pitfall

Writing the outbox after committing the order. Die between the two steps and the dual write is still there. The outbox insert must be inside the transaction that changes the state.

Common pitfall

Marking sent before the broker ack. If the relay dies after the mark but before the broker receives it, the event is lost. Wait for the ack, then mark.

Common pitfall

Assuming the outbox creates exactly-once. A crash after publish, before the mark, still causes duplicates. A stable event_id and an idempotent consumer are needed.

Summary

Key takeaways

• The outbox writes state and the intent to publish an event in one DB transaction.
• A polling relay uses FOR UPDATE SKIP LOCKED; CDC with Debezium gives lower latency but adds infrastructure.
• Duplicates still happen, and order across aggregates can flip; consumers must dedupe.
• Clean up with batch deletes or dropped partitions, and monitor pending age.

Self-check: (1) Why does writing the outbox in a separate transaction still lose events? (2) What results when the relay dies after the broker ack, before mark sent? (3) Why can SKIP LOCKED reorder events?

References

• Transactional Outbox pattern — Microservices.io
• The Locking Clause — PostgreSQL 17
• Outbox Event Router — Debezium