Retry, backoff and jitter: retrying without making the failure worse Intermediate

The problem

An overloaded dependency returns 503 to 1,000 requests. If every caller retries immediately, it receives another 1,000 requests at its weakest moment. If they all wait a fixed 1 second, the second wave still arrives all at once.

Retries get past short failures, but they can also turn a small incident into a self-amplifying loop.

A retry policy answers five questions: which errors to retry, the maximum number of attempts, how far apart, within which deadline, and whether the side effect is safe to repeat.

Terms to tell apart

Term	Meaning
Attempt	One real call. maxAttempts = 3 means 1 initial attempt plus 2 retries
Exponential backoff	Wait min(cap, base × 2^n) after the n-th failure
Jitter	Add randomness to the wait so clients do not wake up at the same time
Retry budget	Limits retries to a proportion of original requests

An attempt fails before deciding whether to retry

Enlarge diagram

Gateway → Service503

◀ Prev▶ PlayNext ▶↺ RestartStep 1/4

Many clients hit a transient error at the same time while the service is overloaded.

Three jitter variants (AWS Architecture Blog)

With base, cap, and attempt starting at 0:

Variant	Sleep formula	Characteristics
Full jitter	random(0, min(cap, base × 2^attempt))	Spreads most evenly; a wait can be close to 0
Equal jitter	t = min(cap, base × 2^attempt); t/2 + random(0, t/2)	Always keeps at least half of the backoff
Decorrelated jitter	min(cap, random(base, prevSleep × 3))	Depends on the previous wait, not on the attempt number

According to AWS:

• No jitter is the worst choice for both call count and completion time.
• Full and Equal produce roughly the same number of calls, but Equal takes much longer to complete.
• Decorrelated makes more calls than Full, while Full does less work but is slightly slower.

The blog post says choosing between Full and Decorrelated is “less clear” and does not name a default; this article picks Full jitter as the default because its formula is the simplest (no need to remember the previous wait).

function backoffMs(attempt: number, baseMs = 200, capMs = 800): number {
  const ceiling = Math.min(capMs, baseMs * 2 ** attempt); // attempt 0,1,2 -> 200, 400, 800
  return Math.random() * ceiling;                         // full jitter
}

Without jitter vs with jitter, four clients failing together

Actual output of examples/src/retry-jitter/jitter-demo.ts:

without jitter: [ 800, 800, 800, 800 ]
full jitter: [ 296, 592, 88, 384 ]
retry amplification for 4 clients × 3 attempts: 12 calls

The demo uses a fixed sequence to illustrate the spread, not a production random generator. Redrawn on a time axis (ms):

no jitter:      0    200   400   600   800
                                         ████  4 clients at once at 800 ms

full jitter:    0    200   400   600   800
                 █(88)  █(296) █(384)  █(592)   each client at its own time

A peak of 4 requests at one point becomes 4 separate points. Jitter reduces synchronization; it does not add capacity.

Retry budgets: the numbers

Source	Rule
Google SRE book	At most 3 attempts per request; and a client retries only while its retry/request ratio is below 10%. The worst-case load increase drops to about 1.1x instead of nearly 3x
Envoy	retry_budget: budget_percent defaults to 20%: concurrent retries are capped at 20% of active plus pending requests; min_retry_concurrency defaults to 3
gRPC	Retry throttling with a token bucket: maxTokens, tokenRatio; each failure subtracts 1 token, each success adds tokenRatio; below half of maxTokens, retries stop

The SRE book also states a rule: only the layer directly above the failing dependency retries; layers further up return the error immediately. Add an “overloaded, do not retry” signal to inform the layers above.

Amplification by number of layers

Each layer retries up to 3 attempts; the worst-case number of calls reaching the bottom layer for one top-level request:

Layers that retry	Calls reaching the bottom layer
1	3
2	9
3	27
4	81

An SDK retrying implicitly, an app wrapper retrying, a gateway retrying: the total attempts differ wildly from the number in any single config file. Pick one layer responsible for retries (the layer closest to the dependency, per the SRE book) and turn the others off.

When to retry and when not to

Retry	Do not retry
Network errors before a response, connection reset	400, 401, 403, 404, 422
503, 502, 504	Business or validation errors
429 with Retry-After (wait as the header says)	Timeout of an operation with side effects but no idempotency
408 per the dependency’s contract	When the overall deadline is about to run out

A status code alone is not enough: an endpoint can return 500 after it has committed. Write operations need an idempotency key that stays the same across every attempt (see eight fallacies and idempotent consumer). Every attempt, including sleep time, must fit within the original deadline (timeout budget).

Real-world example

An ordering service calls a stock-check provider

The provider occasionally returns 503 for a few hundred ms while switching instances.

• 3 attempts, retrying only 503 and network resets before a response.
• Full jitter with base = 200 ms, cap = 800 ms.
• Stop if the remaining deadline cannot fit the wait plus one more call.
• The stock-reservation command uses the same reservation key for every attempt.
• Check whether the provider’s SDK already retries, to avoid multiplying calls.

Testing: use fake timers; the first call returns 503, the next 200; assert the number of attempts, that delays fall within range, and the same idempotency key across attempts.

Also test that 400 is not retried, the deadline expiring mid-backoff, 429 with Retry-After, and the case where the server committed but the response was lost. Inject the random generator instead of comparing against a random value.

🎮 Try it

• Scale Lab: notifications: retrying when the provider goes down, and draining the backlog.
• Retry storm: retries at every layer multiply load 27 times.

Production checklist

1. State maxAttempts explicitly; do not confuse it with maxRetries.
2. An explicit list of retryable errors (network, 502/503/504, 429); do not retry other 4xx.
3. Full jitter with a cap; respect Retry-After.
4. A retry budget: a retry/request ratio (for example 10% per the SRE book) or a token bucket.
5. Only one retrying layer per call path; compute the worst case if that is not possible.
6. All attempts fit within one overall deadline.
7. Write operations have an idempotency key that stays the same across attempts.
8. Metrics: retry_attempts, retry ratio, reason and delay of each attempt; logs contain no sensitive payloads.

Trade-offs

When it helps

• Gets past brief network errors or overload without the user having to act again
• Backoff and jitter reduce synchronized retries after a shared incident
• Limits on attempts, budget and deadline keep latency and load under control

Trade-offs

• Adds traffic exactly when downstream is weak; a bad policy amplifies the incident
• Lengthens response time and makes failures harder to trace without per-attempt logs
• Side effects need real idempotency, not just added delay

Common mistakes

Common pitfall

Retrying every error. A 400 caused by bad input and a 401 caused by missing permissions do not go away after a few hundred ms.

Common pitfall

Fixed backoff for a thousand clients. They all wake up and hit again together. Add jitter and monitor the retry rate.

Common pitfall

Granting a new deadline to each attempt. The request runs far past the client’s timeout while still holding resources. All attempts fit within the original deadline.

Common pitfall

Retrying a POST after a timeout with a different key. A lost response does not mean the operation did not commit. Keep the same idempotency key.

Summary

Key takeaways

• A retry is a policy: which errors, how many attempts, delay, deadline, and side-effect safety
• Default to Full jitter: random(0, min(cap, base × 2^attempt)); Equal and Decorrelated are AWS’s other two options
• Retry budgets: SRE book 3 attempts and 10%; Envoy defaults to 20%; gRPC uses a token bucket
• Retries at several layers multiply: 3 layers × 3 attempts = 27 calls
• A timeout on an earlier attempt does not prove the side effect failed

Self-check: (1) How many retries is maxAttempts = 3? (2) Why does a fixed backoff still cause load peaks? (3) What is the maximum number of calls created by 3 retrying layers with 3 attempts each?

References

• Exponential Backoff and Jitter — AWS Architecture Blog
• Retry with backoff pattern — AWS Prescriptive Guidance
• Handling Overload — Google SRE Book
• RetryBudget — Envoy
• Retry — gRPC