Circuit breaker and bulkhead: a slow provider must not take down the whole API Intermediate

The problem

A sales API has two endpoints: POST /checkout calls a payment provider, GET /health calls nothing. Each request borrows one of 16 workers (like Tomcat’s thread pool). Normally the provider answers within 200 ms at p99.

One afternoon, the provider’s p99 rises to 10 s. Each checkout holds a worker while it waits; at 20 req/s, all 16 workers are taken in under a second.

/health queues behind them, the probe fails, and the pod is considered broken. One slow dependency takes down the whole API: cascading failure.

The question: how do you make provider failures affect only checkout, and make checkout fail fast instead of waiting?

What to choose (summary first)

Situation	Choose	Condition
Every outbound call	Timeout	Always; based on measured p99/p99.9 and the remaining deadline
Transient errors lasting a few hundred ms	Retry with jitter	The error is retryable and the operation idempotent (retry, jitter)
One dependency can take every worker or a shared connection pool	Bulkhead	A separate concurrency limit per dependency
A dependency is slow or failing for a long time	Circuit breaker	A fallback exists; enough traffic to compute an error rate

The four mechanisms complement each other; none replaces another.

Core concepts

• Bulkhead: limits concurrent calls to one dependency and rejects when it is full.
• Circuit breaker: measures the error rate and, above a threshold, fails fast without sending the request.
• Fallback: an alternative response when a call is rejected.

Nygard popularized both patterns in Release It!; the bulkhead takes its name from the partitions in a ship’s hull.

How it works

A breaker is a state machine wrapped around calls to one dependency. The figure below follows the payment breaker in the demo.

Payment breaker: opens when errors cross the threshold, retries in a limited way, closes when the provider recovers

Enlarge diagram

Checkout API → Breaker paymentpayBreaker payment → Payment provider200 ms

◀ Prev▶ PlayNext ▶↺ RestartStep 1/6

CLOSED: every call goes through; the breaker records results in a window of the last 10 calls.

State	Calls to the provider	Leaves the state when
CLOSED	Pass through, results go into the sliding window	minimumNumberOfCalls is reached and the failure rate or slow-call rate ≥ threshold → OPEN
OPEN	Rejected immediately (CallNotPermittedException)	waitDurationInOpenState elapses → HALF_OPEN
HALF_OPEN	Only a few trial calls	Trial calls below threshold → CLOSED, otherwise → OPEN

Slow calls are counted too: the provider still returns 200, just after 10 s.

Parameters and defaults

Resilience4j 2.x parameter	Default	Demo
slidingWindowType / slidingWindowSize	COUNT_BASED / 100	count, 10 calls
minimumNumberOfCalls	100	10
failureRateThreshold	50%	50%
slowCallRateThreshold / slowCallDurationThreshold	100% / 60 000 ms	50% / 400 ms
waitDurationInOpenState	60 000 ms	1 000 ms
permittedNumberOfCallsInHalfOpenState	10	3

• minimumNumberOfCalls stops the breaker from opening on the first few errors: with a value of 10, nine failed calls out of nine still do not open it.
• The default slow threshold of 60 s is almost never reached; set it yourself based on the provider’s SLO.
• Polly v8: FailureRatio 0.1, MinimumThroughput 100, BreakDuration 5 s.

Bulkhead: a semaphore and a bounded queue

A bulkhead counts in-flight calls to one dependency. If there is room, the call goes; if not, it waits in a bounded queue, and is rejected when the wait times out.

Loading diagram…

Payment holds at most 4 workers; the remaining 12 are kept for /health. Resilience4j has SemaphoreBulkhead (maxConcurrentCalls default 25, maxWaitDuration 0) and FixedThreadPoolBulkhead (queueCapacity 100).

Size the bulkhead with Little’s law: concurrent calls ≈ rate × latency. 20 req/s × 0.2 s = 4 slots. At 40 req/s, 4 slots fall short even when the provider is healthy: in the demo, 4/18 checkouts got 503 in the steady phase.

Ordering with retry and timeout

Source	Order (outer → inner)
Resilience4j, Spring Boot aspect	Retry(CircuitBreaker(RateLimiter(TimeLimiter(Bulkhead(fn)))))
.NET standard resilience handler (Polly v8)	Rate limiter → total timeout → retry → circuit breaker → attempt timeout

Both put retry outside, the breaker in the middle, the per-attempt timeout innermost. The breaker sees each attempt, so the error rate is accurate; when it opens, attempts fail within a few ms.

Do not retry immediately on a “breaker open” error: Polly suggests putting BreakDuration into the context so that the retry waits exactly that long.

Operational details

The fallback is the application’s job. Fowler lists three options:

• Fail fast: 503 with Retry-After (remaining seconds of OPEN), with a Problem Details body.
• Stale data from a cache, with its timestamp clearly stated.
• Process later: store the order as pending_payment, and a worker charges when the provider recovers, using an idempotency key.

Scope: one breaker and one bulkhead per dependency (or per endpoint of it).

Which errors count: Resilience4j counts every exception as a failure by default; use ignoreExceptions for 4xx caused by bad input. The demo ignores bulkhead-full errors because those are the caller’s own limit.

Observability: Fowler requires logging every state change. You need a state gauge per breaker name, the number of rejected calls and bulkhead slots in use; alert when a breaker opens.

Real-world example

Checkout while the payment provider is 10 s slow

Order 101 arrives while the breaker is OPEN: the API returns 503 with Retry-After: 1 after 3 ms, and the frontend shows “payments temporarily unavailable”.

Order 102 arrives after the provider recovers: the 3 trial calls in HALF_OPEN all finish in about 200 ms and the breaker closes. /health answers within 2 ms throughout the incident, and the pod is not restarted.

The demo examples/src/circuit-breaker-bulkhead/ hand-writes a breaker and a semaphore, the provider on port 7010, the API (16 workers, three modes) on port 7011, with load generated by parallel fetch.

npx --prefix examples ts-node examples/src/circuit-breaker-bulkhead/load.ts
CHECKOUT_RPS=40 npx --prefix examples ts-node examples/src/circuit-breaker-bulkhead/load.ts timeout

A 6 s scenario: checkout at 20 req/s (the client gives up after 3 s), /health every 250 ms (gives up after 2 s); the provider is 10 s slow from second 0.5 to 3.5. Real measurements on Node 24.17, slow-provider phase:

Mode	/health	/checkout	Provider in-flight max
none (no timeout)	8/12 timeout, p50 2 001 ms	58/58 timeout	16
timeout 500 ms	p50 2 ms	58/58 504 after ~505 ms	63
timeout 500 ms, 40 req/s	p50 259 ms, p95 503 ms	96/115 504	112
protected	p50 2 ms, p95 5 ms	p50 3 ms; 43 503, 15 504	19

t=1.52s CLOSED -> OPEN (failure rate 50% over 10 calls)
t=2.53s OPEN -> HALF_OPEN (waited 1000 ms)
t=3.13s HALF_OPEN -> OPEN (half-open: failure rate 100% over 3 calls)
t=4.18s OPEN -> HALF_OPEN (waited 1000 ms)
t=4.49s HALF_OPEN -> CLOSED (half-open: 3/3 trial calls passed)

1. No timeout: the provider recovers at second 3.5, but /health still times out 10/10 because workers are stuck on old calls.
2. Timeout alone is enough when req/s × timeout < pool (20 × 0.5 = 10 < 16); at 40 req/s it is 20 > 16 and /health gets 100 times slower.
3. A timeout cancels only the caller’s side; the provider keeps running abandoned requests (in-flight 63, 112). Only the breaker actually reduces load (19).

Numbers vary per run; load.ts discards results if the event loop stalls for more than 500 ms. The demo lacks time-based windows, metrics, retries and a fallback queue.

Trade-offs

Benefits

• A failure in one dependency does not spread to other endpoints
• Failing fast frees workers
• A failing dependency is not hit with even more load

Costs and limits

• Modal behavior is hard to test; Brooker (AWS) notes a breaker can prolong recovery
• Wrong parameters: it opens because of low traffic, or never opens
• A bulkhead that is too small rejects calls even when the dependency is healthy

Production checklist

1. A timeout on every outbound call based on p99/p99.9, within the request’s deadline.
2. A separate breaker and bulkhead per dependency, named after the dependency.
3. Bulkhead size = req/s × p99 latency plus headroom; verify with a load test.
4. Slow-call threshold based on the dependency’s SLO, not the 60 s default.
5. minimumNumberOfCalls matched to real traffic so a few isolated errors do not open the breaker.
6. A fallback per endpoint: 503 + Retry-After, stale data, or a queue with an idempotency key.
7. Order retry → breaker → per-attempt timeout; do not retry when the breaker rejects.
8. Metrics for breaker state, rejected calls and bulkhead slots; alert when a breaker opens.

Common mistakes

The most common mistake:

One breaker for every dependency

✕ Common

const breaker = new CircuitBreaker('outbound', cfg, log);
await breaker.exec(() => payment.charge(order));
await breaker.exec(() => email.send(order)); // email fails -> checkout gets blocked

✓ Prefer

await paymentBreaker.exec(() => paymentBulkhead.run(() => payment.charge(order)));
await emailBreaker.exec(() => email.send(order));

Why

A breaker measures the health of one dependency. Lump them together and a failure on one side blocks the other.

The second mistake:

Dropping the timeout because there is a breaker

A breaker opens only once enough calls have been recorded. Without a timeout, hanging calls never finish, so they never enter the window, and the workers are stuck before the breaker gets a chance to open.

Misconception	Consequence	What to do
A timeout is enough to isolate failures	At 40 req/s, /health p50 is 259 ms	Add a bulkhead when req/s × timeout approaches the pool size
Counting only exceptions	The provider returns 200 after 10 s and the breaker never opens	Enable the slow-call threshold

Self-check

1. Why does /health still time out in none mode after the provider has recovered?
2. 20 req/s, a 500 ms timeout, a 16-worker pool: do you need a bulkhead? What about 40 req/s?
3. minimumNumberOfCalls = 100 while the dependency receives only 30 calls/minute: what does the breaker do in the first minute of an incident?
4. Why does Resilience4j put Retry outside CircuitBreaker?

Summary

Key takeaways

• Timeouts are always needed; bulkheads isolate resources; a breaker stops calling when errors or slow calls cross a threshold
• CLOSED counts, OPEN fails fast, HALF_OPEN probes in a limited way
• Bulkhead size ≈ req/s × latency
• Retry outside, breaker in the middle, timeout innermost; one breaker per dependency
• Fallback: 503 + Retry-After, stale data, or process later

References

• Release It! — Nygard: the origin of both patterns.
• CircuitBreaker — Fowler: half-open, monitoring, fallback.
• Resilience4j CircuitBreaker, Bulkhead, aspect order: parameters, defaults.
• Microsoft Learn — HTTP resilience, Polly: ordering, Polly v8 defaults.
• AWS — Timeouts, retries, backoff: choosing timeouts, breaker drawbacks.