Eight fallacies of distributed computing: seen through one request timeout Basic

The problem

A client sends POST /payments. The service has written the result to the database, but the response arrives after the client’s deadline. The client sees a timeout and retries. If every request creates a new payment, the user is charged twice.

From the client’s logs there is no way to tell where the request failed. This is partial failure: one side does not know the other’s state for sure, even though both are still alive.

Origin of the list

The list started in 1994 with seven items recorded by L. Peter Deutsch (Sun Microsystems). According to Wikipedia, four of them came from an earlier list, “The Fallacies of Networked Computing”, by Bill Joy and a colleague at Sun.

Around 1997, James Gosling added the eighth item, “the network is homogeneous”. This article uses the eight-item list from Oracle’s article.

The eight fallacies in one table

Fallacy	Backend counterexample	Design question
The network is reliable	An established TCP connection still drops between request and response	After a timeout, what does the client know? Is a retry safe?
Latency is zero	Every RPC adds waiting and tail latency	How many synchronous hops does one request go through?
Bandwidth is infinite	Large payloads and fan-out multiply total bytes	Are you sending extra data or fetching multiple times?
The network is secure	Internal networks still have compromised machines and leaked credentials	Where are authentication, authorization, TLS and secrets?
Topology doesn’t change	Pod restarts, leader failover, IP changes	Does the client have discovery and reconnect?
There is one administrator	Many teams own the gateway, services, DB, DNS	Who handles incidents; how do logs/traces connect the parties?
Transport cost is zero	Serialization, TLS, egress and hops all cost money	Batch requests, cache, or move the service boundary?
The network is homogeneous	Mobile, WAN, VPN, different clusters	Do timeouts and protocols fit every path?

Four points where an RPC can fail

A local function charge() returns a result or throws immediately. An RPC charge() has four failure points, and the client sees the same TimeoutError for all four.

Loading diagram…

Point	What the server did	Retrying the same intent
1. Request lost	Knows nothing	Needed, safe
2. Received, not done	Changed nothing	Needed, safe
3. Committed	The side effect happened	Must return the old result, not redo it
4. Response lost	The side effect happened	Same as point 3

Because points 1-2 cannot be told apart from 3-4, operations with side effects need an idempotency key: when the server receives the same key, it returns the old result instead of redoing the work. Key design, durable storage and handling two concurrent requests are covered in idempotent consumer.

How it works

The client times out but the service has already committed

Enlarge diagram

Client → ServicePOST /payments

◀ Prev▶ PlayNext ▶↺ RestartStep 1/4

The client sends a create-payment request. A connection that works at send time does not guarantee the response arrives on time.

A client timeout does not cancel work running on the server. Retrying with a new key creates a second payment; retrying with the same key makes the server look up the old result.

A local demo (no Docker needed) starts an HTTP server on 127.0.0.1; the client has a 20 ms deadline, the server waits 80 ms and then commits. Actual output:

client timeout: TimeoutError
server committed: payment-1
retry result: payment-1 committed count: 1

The demo uses an in-memory Map, so it only illustrates a retry after the first request finishes. Production needs durable storage and a unique key.

Latency and availability multiply with the number of hops

The availability of a synchronous call chain is the product of each part’s availability:

Synchronous dependencies, each 99.9%	Chain availability	Downtime per month (30 days)
1	99.9%	about 43 minutes
10	99.0%	about 7.2 hours
20	98.0%	about 14.3 hours

(0.999^10 ≈ 0.990, 0.999^20 ≈ 0.980.) Even when each hop’s p95 is small, 20 sequential RPCs add up latency and multiply risk.

The fix is not necessarily more services:

• combine APIs per use case;
• run independent parts in parallel;
• keep data within the same boundary;
• or return results asynchronously.

Allocate the time budget from the outside in (timeout budget).

Real-world example

Create-order API: write the order, call payment, publish an event

The client times out while the service is waiting on the payment provider. Answering “just retry” is dangerous. You need:

• A stable request ID (idempotency key) passed all the way down to the provider.
• An order status that can be queried (pending, paid, failed).
• Events published through a transactional outbox after the decision is durably stored, not published from memory in the hope that the network holds up.

🎮 Try it

• Retry storm: a slow network turns into cascading failure.

Checklist when adding a synchronous RPC

1. Operations with side effects accept an idempotency key; retries reuse the same key and never generate a new one.
2. The client has a deadline and a way to query the status (pending, paid, failed) when the outcome is unclear.
3. Retry only appropriate errors, with limits and backoff (retry, backoff, jitter).
4. Count synchronous hops on the critical path and multiply availability; consider combining APIs, running in parallel or returning results asynchronously.
5. Internal calls still have authentication, authorization and TLS.
6. The client has discovery and reconnect for topology changes (Pod restart, failover, IP change).
7. Events go through a transactional outbox, not published from memory.

Trade-offs

Designing with realistic assumptions

• Timeouts, idempotency and status queries give the client a way to handle unclear outcomes
• Limiting hops and payload size makes latency and cost predictable
• Ownership, logs and traces across services shorten the time to find partial failures

Costs to accept

• Idempotency needs durable storage and an expiration policy
• Retry and backoff increase load without a budget (retry)
• Security, discovery and observability add operational work; splitting services is worth it only when the benefit exceeds the cost

Common mistakes

Common pitfall

“A timeout means the server did nothing.” A timeout only says the client did not receive a response in time; the server may have committed (points 3 and 4).

Common pitfall

“Retries make things more reliable.” Blind retries create duplicate side effects and multiply load. Retry only appropriate errors, with limits, backoff and idempotency.

Common pitfall

“The internal network is secure and free.” Serialization, TLS, egress and hops have costs; authentication is still needed. Treat every RPC as a boundary for failure, security and ownership.

Summary

Key takeaways

• The list was recorded by Deutsch in 1994 (seven items); Gosling added the eighth around 1997
• An RPC has four failure points; a timeout cannot tell them apart
• Operations with side effects need an idempotency key; details in idempotent consumer
• Synchronous chain availability is a product: 10 dependencies at 99.9% leave about 99.0%
• Measure end to end before splitting off more services

Self-check: (1) The client times out after POST /payments: what can you conclude about the server’s DB? (2) How does retrying with a new key differ from retrying with the same key? (3) What availability do 15 sequential RPCs at 99.9% each give, roughly?

References

• Fallacies of distributed systems — Oracle
• Fallacies of distributed computing — Wikipedia
• Making retries safe with idempotent APIs — Amazon Builders Library
• Cloud Design Patterns — Microsoft Learn