Webhooks: verifying signatures, handling retries and preventing duplicate processing Intermediate

The problem

A sales system receives an order.paid webhook. The handler updates the order and returns 200. One day the same event arrives twice: the first attempt was written to the database but the 200 response got lost on the way back, so the provider, not knowing the outcome, retried.

If the handler adds revenue on every POST it receives, one payment creates two business effects. Conversely, anyone who knows the URL can send fake JSON to mark an order as paid.

The two failures need two different mechanisms, plus a convention for ACK:

Mechanism	Answers	Does not answer
Signature verification	Was the payload signed with the sender’s secret, and is it fresh?	Has this event been processed yet
Idempotency (dedupe)	Has this valid event already produced its effect?	Whether the source is trustworthy
ACK (2xx)	How far has the handler gotten?	Whether downstream work is complete

This article uses a generic model: HTTP POST, HMAC, an event ID that stays stable across retries. The demo in the repo is not compatible with Stripe signatures; for real Stripe, use the SDK and the exact Stripe-Signature format.

Handler: verify first, then dedupe

The required order: raw body → signature → timestamp → parse → insert into inbox → ACK.

// express.raw keeps the Buffer intact; do NOT use express.json() for this route
app.post('/webhooks/pay', express.raw({ type: 'application/json' }), async (req, res) => {
  const raw: Buffer = req.body;
  const { t, v1 } = parseSignatureHeader(req.header('x-signature'));  // "t=...,v1=..."

  const expected = createHmac('sha256', SECRET).update(`${t}.`).update(raw).digest();
  const given = Buffer.from(v1 ?? '', 'hex');
  const ok = given.length === expected.length && timingSafeEqual(given, expected);
  const fresh = Math.abs(Date.now() / 1000 - Number(t)) <= 300;        // 5 minutes
  if (!ok || !fresh) return res.status(400).end();                     // nothing written yet

  const event = JSON.parse(raw.toString('utf8'));
  const r = await db.query(
    `INSERT INTO webhook_inbox (provider, event_id, payload, status)
     VALUES ('pay', $1, $2, 'pending') ON CONFLICT (provider, event_id) DO NOTHING`,
    [event.id, raw.toString('utf8')]);
  // r.rowCount === 0 means a retry of an event already received; still ACK
  res.status(200).end();                                               // a worker processes it later
});

Notes on the parts that commonly break:

• Raw body: parsing and re-serializing can change whitespace and key order and break the signature. This is the number one bug on Express, Nest and Next.
• timingSafeEqual requires both buffers to have the same length, so check the length first.
• Timestamp tolerance: the Stripe library defaults to 5 minutes; don’t set it to 0, because that disables the check. Server clocks need NTP sync.
• ON CONFLICT DO NOTHING is what arbitrates the race between two concurrent requests; if (!exists) insert does not.
• The inbox row is already a durable record, so ACKing after the insert is safe. This pattern is the idempotent consumer; effects that go out to other systems use the transactional outbox.

A valid webhook can be delivered more than once

Enlarge diagram

Provider → Webhook handlerPOST + signature

◀ Prev▶ PlayNext ▶↺ RestartStep 1/4

The provider sends the raw body, an event ID, and a signature. The handler does not trust this data yet.

Failure matrix: where it crashes and what happens

Crash point	Provider sees	Outcome	Prevention
Before the inbox insert	No 2xx, retries	Processed normally on the next attempt	Nothing extra needed
After the insert, before 2xx	No 2xx, retries	The retry is detected as a duplicate and still ACKed; the worker processes from the inbox	The inbox is the source of truth
Worker dies before the effect	Got 2xx, will not resend	The row stays pending forever unless something picks it up	Worker scans pending, lease/timeout
Effect done, status not updated	Got 2xx	The worker runs the effect a second time	Effect + status change in the same transaction, or an idempotent effect

The last row is why you cannot rely on dedupe at the entrance alone: you need idempotency at every boundary that has an effect.

Retry, replay and retention

For Stripe (checked against the docs):

Topic	Value
Live mode retries	Up to 3 days, exponential backoff
Sandbox retries	3 times over a few hours
Replay tolerance (library)	5 minutes
Manual resend	Dashboard 15 days, CLI 30 days
Ordering	Not guaranteed; use the event ID, not created
Secret rotation	Keeps the old secret for up to 24 hours, Stripe signs one signature per secret
Response	Return 2xx quickly, do complex processing asynchronously

Design consequences:

• Keep seen event IDs longer than the retry window (at least 3 days plus a margin, longer if you resend manually often).
• Since ordering is not guaranteed, use a state machine (pending to paid only when the current state allows it) instead of trusting “the later event is the newest”.

Other providers have different numbers; read their docs.

When to use it

Use inbox + dedupe	Not needed
Effects that must not repeat (money, goods, email)	Effects that are naturally idempotent (overwriting state with the same value)
The provider retries at-least-once	The internal source already has exactly-once at another layer
Slow processing that needs a fast ACK	Very fast processing in the same transaction

Real-world example

A payment event arrives twice

The inbox stores provider, event_id, status, received_at with a unique (provider, event_id). The worker moves the order from pending to paid in one transaction, only when the current state allows it.

When the event arrives again, the handler ACKs because the row already exists. Customer notifications go through the outbox so they are not lost if the process dies right after the commit.

No step relies on “the webhook is surely sent only once”.

Run the demo (no server or third-party account needed):

npx --prefix examples ts-node examples/src/webhook-delivery/webhook-demo.ts

The demo signs the raw body with HMAC-SHA256, calls the handler twice with the same event, then modifies one byte of the body while keeping the old signature. Expected output: ack: evt-42, ack: duplicate event, reject: invalid signature, and effects equal to 1.

The demo signs only the body, without a timestamp, uses a hard-coded secret and an in-RAM Set; do not copy it to production. Try crashing between processed.add and effects += 1 to see the last row of the failure matrix.

Production checklist

1. The webhook route uses the raw body; test with payloads containing unusual whitespace.
2. The signature includes a timestamp; compare with timingSafeEqual after checking the length; tolerance around 5 minutes.
3. The inbox has a unique constraint on (provider, event_id); dedupe with ON CONFLICT.
4. Return 2xx after the durable write; heavy processing runs in a worker.
5. The worker has a lease to pick up pending rows again; the effect shares a transaction with the status change.
6. Keep event IDs longer than the retry window; handle secret rotation by accepting two secrets during the transition.
7. Limit body size; keep sensitive payloads out of logs; have a manual replay procedure.

Trade-offs

What this mechanism gives you

• A signature over the raw body rejects requests that were modified or sent without the secret.
• A unique event ID keeps retries from doubling effects.
• A durable inbox lets you ACK early and decouple slow processing.

Costs and boundaries

• Secret rotation, tolerance and raw-body middleware need correct configuration per provider.
• The dedupe table takes storage; you must choose a long enough retention.
• Side effects outside the database still need an outbox and idempotency at the destination.

Common mistakes

Common pitfall

Parsing JSON before verifying the signature. The parser changes whitespace or field order when re-serializing. Keep the raw bytes and verify before trusting the content.

Common pitfall

Treating the signature as proof of “exactly once”. HMAC only proves integrity and knowledge of the secret; it does not limit how many times the provider resends a valid payload. You need an event ID and a unique constraint.

Common pitfall

ACKing before there is a durable record. If the process dies after 200 but before writing to the inbox, the provider considers it done and will not resend. Test by injecting failures at each crash point.

Summary

Key takeaways

• Verify on the raw body, check the timestamp, and only then parse and dedupe.
• Signature, idempotency and ACK answer three different questions.
• INSERT ... ON CONFLICT DO NOTHING on (provider, event_id) is the arbiter; keep IDs longer than the retry window.
• Every boundary with a side effect needs its own idempotency (outbox, effect in the same transaction).

Self-check: (1) Why does JSON with “the same content” fail signature verification after re-serialization? (2) If the worker dies midway after the ACK, which row saves the event? (3) Why must event ID retention exceed the retry window?

References

• Receive Stripe events — Stripe Docs
• Resolve signature verification errors — Stripe Docs
• Crypto API — Node.js