Timeout budget: how much time does a request have to finish? Intermediate

The problem

A create-order request passes through the gateway, the order service, inventory and payment. Each layer sets its own 2-second timeout. Call by call it looks safe; across the chain, the client gave up after 1.5 seconds while the order service is still calling payment.

The client’s retry creates more work, and the server holds connections, workers and transactions for a job nobody is waiting for.

The problem is not the “2 seconds” figure but the boundary of responsibility for time: each layer needs to know how long the request has left and whether to start the next step at all.

Core concepts

Concept	Meaning	Example
Timeout	Bounds one operation	Call inventory for at most 300 ms
Deadline	The latest point at which the whole request still matters	15:00:01.500
Remaining budget	Deadline minus now; only decreases	330 ms left
Cancellation	A “stop if you can” signal	AbortSignal

The effective timeout of each call is min(own timeout, remaining - margin). When the budget is gone, don’t start new calls.

The time budget shrinks across the steps of a request

Enlarge diagram

Client → API500 ms

◀ Prev▶ PlayNext ▶↺ RestartStep 1/4

The client starts the request with a 500 ms budget. The API records the deadline before doing any work.

Over the network: gRPC does not send an absolute timestamp; it converts the deadline into a timeout minus the time already elapsed, avoiding clock skew; Java and Go propagate it by default, C++ needs it enabled (gRPC deadlines).

If you build your own HTTP header, define the unit, a maximum cap, how to handle forged headers and how to subtract elapsed time. Within a process, use a monotonic clock (performance.now()), not the wall clock.

Waterfall: how to split 800 ms

Target 800 ms. The gateway takes 100 ms. The order service reserves the last 80 ms for writing the response and cleanup. Per-step caps: inventory 300 ms, payment 400 ms (the sum of caps exceeding the budget is intentional).

Step	Starts at	Own cap	Remaining before the step (minus 80 ms reserve)	Applied timeout	Actually used
Gateway	0	-	-	-	100 ms
Inventory	100 ms	300	620	min(300, 620) = 300	290 ms
Payment	390 ms	400	330	min(400, 330) = 330	times out at 330 ms
Reserve	720 ms	-	-	-	80 ms, finishes at exactly 800 ms

If payment kept its 400 ms cap, the request would overrun the deadline by 70 ms.

When payment times out, the API stores the idempotency key, returns an error per the contract (problem+json) and lets the client look up the result; it must not create a new payment on its own.

Timeout hierarchy: outer larger than inner

Timeouts must shrink as you go inward: client larger than gateway, gateway larger than service, service larger than dependency. If reversed, the outer layer gives up before the inner layer can return a proper error.

Layer	Timeout type	Verified default
nginx proxy_connect_timeout	Establishing a connection to the upstream	60 s (in practice cannot exceed 75 s)
nginx proxy_read_timeout	Between two successive reads, not the whole response	60 s
AWS ALB connection idle timeout	Connection with no data	60 s (range 1-4000 s)
Application	Server keep-alive/idle	Set it larger than the LB idle timeout, otherwise the LB may send into a just-closed connection and return 502 (AWS recommendation)

The 60 s defaults are usually far larger than an 800 ms API budget, so they are only a last safety net; the application deadline determines the real behavior. Distinguish three kinds: connect (should be short), read/response (per the budget), idle (for connection pooling).

Node: AbortSignal.timeout and AbortSignal.any

AbortSignal.timeout() exists since Node 17.3/16.14; AbortSignal.any() since Node 20.3 / 18.17.

const remaining = deadlineMs - performance.now();
if (remaining <= 50) throw new Error('deadline exceeded');

const signal = AbortSignal.any([
  callerSignal,                                          // client disconnected
  AbortSignal.timeout(Math.min(400, remaining - 50)),    // dependency cap and budget
]);
const res = await fetch(paymentUrl, { signal });

abort() only cancels locally; the downstream may already have committed. On the server side, also stop work when the client has gone away (gRPC cancels automatically with CANCELLED; for HTTP, listen for the request’s close).

deadlineMs must come from the same monotonic clock within the process; pass it to other processes as capped remaining ms, never as performance.now(). The 50 ms margin is only illustrative, measure your own.

When to use it

Use deadline propagation	Per-call timeouts are enough
Chains of two or more hops	One call, no fan-out
Retries within the same request	Background jobs nobody waits on
Synchronous requests with a latency SLO	Long batches with their own resume mechanism

Real-world example

Creating an order within 800 ms

The waterfall table above is a realistic scenario for a create-order API: after inventory spends 290 ms, payment only has min(400, 330) = 330 ms.

When payment runs out of time, the API stores the idempotency key and returns a “processing” state for the client to look up instead of reporting a definite failure. A timeout does not prove the downstream has not committed; see retry, backoff and jitter and the eight fallacies.

Run the pure TypeScript simulation:

npx --prefix examples ts-node examples/src/timeout-budget/budget-demo.ts

The demo uses fixed timestamps: a 500 ms request, 150 ms auth, downstream capped at 350 ms; a subsequent retry has only 50 ms left, and after the deadline it must not start. Change the auth time to 480 ms to see the remaining budget nearly exhausted.

🎮 Try it

• Retry storm: retries at every layer multiply load 27 times; the deadline is propagated down.

Production checklist

1. Set the deadline at the system edge, from measured p95/p99, not from gut feeling.
2. Every outbound call has a timeout of min(dependency cap, remaining - margin).
3. Retries stay within the original deadline; check the budget after backoff.
4. Outer timeouts are larger than inner ones; check the LB and proxies too.
5. Propagate the remaining timeout across services, and cap the value received from clients.
6. The server stops work when the client cancels; don’t hold a DB transaction while waiting on a remote call.
7. Log remaining_ms before/after calls and the error type (caller cancel, local timeout, downstream).

Trade-offs

Benefits

• Bounds resources held by requests that are already past due; latency becomes predictable.
• Every service knows how much time is left.
• Retries have a total bound instead of many independent timeouts.

Costs and limits

• Deadlines that are too tight increase spurious errors; you need to measure and tune.
• Propagating deadlines across multiple protocols needs a contract.
• Cancellation does not undo side effects that have already committed.

Common mistakes

Common pitfall

Resetting the timeout in every service or on every retry. The request lives far longer than the client is willing to wait. Compute from the original deadline and take the min with the call’s own timeout.

Common pitfall

Passing monotonic timestamps between machines. performance.now() only means something within the local process. Send a capped remaining timeout.

Common pitfall

Outer timeout smaller than inner timeout. Gateway 5 s, service 30 s: the gateway gives up while the service keeps running. Make timeouts shrink from the outside in.

Summary

Key takeaways

• A timeout bounds one step, a deadline bounds the whole request, and the remaining budget only decreases.
• Call timeout = min(own cap, remaining - margin).
• AbortSignal.any([caller, AbortSignal.timeout(ms)]) chains caller cancellation with a timeout (Node 20.3+).
• Timeouts shrink from the outside in; the 60 s defaults of nginx/ALB are only the last net.

Self-check: (1) Inventory used 290 ms; how much can payment get at most? (2) Why must the app idle timeout be larger than the ALB’s? (3) Does abort() undo side effects downstream?

References

• Deadlines — gRPC
• AbortSignal — Node.js
• ngx_http_proxy_module — nginx
• Connection idle timeout — AWS ALB