Distributed locks with Redis: what if the lock expires while the owner is still running? Advanced

The problem

An inventory sync job with the supplier runs on 3 pods, reads the supplier’s figures, and writes qty into the inventory table. Only one pod may run at a time; if two pods write, an old figure can overwrite a newer one.

The usual approach is a Redis lock key with a TTL. But the job sometimes hits a GC pause or stalls longer than the TTL.

What does a Redis lock guarantee when the lock owner is stalled past its expiry, and what else is needed to keep the data correct?

What to choose? (summary first)

Situation	Choose	Condition
Avoid duplicate work, rare duplicates are acceptable	SET key token NX PX on one Redis, release via Lua	The job is idempotent
Must be strictly correct	A fencing token checked at the target resource, a consensus system (etcd, ZooKeeper), or a database constraint	The target rejects old tokens
The target is PostgreSQL	Advisory lock or a row lock	Combined with a conditional write

Kleppmann calls the first two needs locks for efficiency and for correctness; confusing the two is the main mistake.

Core concepts

A Redis lock is really a lease: it expires by itself when the TTL hits 0, even if the owner is still running. An advisory lock is the opposite: no expiry, lost when the connection drops.

	Redis SET NX PX	PostgreSQL advisory lock
Lock lost when	TTL expires, failover	Connection drops
Does the owner know it lost the lock?	No	When the next query reports a connection error
Fencing token	No	No

How it works

Acquiring the lock: why not SETNX then EXPIRE

The value is a fresh random token for each acquisition. SETNX then EXPIRE is two commands: dying in between leaves a key with no expiry (TTL returns -1), and the lock is stuck forever.

SETNX is deprecated since 2.6.12; use a single SET key token NX PX 30000 (OK, or nil if the key already exists).

Releasing the lock: compare the token in Lua

A plain DEL removes any lock at the key, including one another pod just acquired after yours expired. The script runs atomically, so there is no gap between GET and DEL (see Lua scripts):

if redis.call("get",KEYS[1]) == ARGV[1] then
    return redis.call("del",KEYS[1])
else
    return 0
end

Redis 8.4 adds DELEX key IFEQ value, which does exactly this in one command; Redis 7.x still uses the script.

The lock expires while the owner is still running

Redis does not need to fail: Pod A only has to stall longer than the TTL. The diagram follows the demo run:

Pod A pauses past the TTL, Pod B acquires the lock with token 34; PostgreSQL rejects the late write carrying token 33

◀ Prev▶ PlayNext ▶↺ RestartStep 1/6

Pod A takes the lock with a 300 ms TTL and INCRs to get fencing token 33.

Step	Redis holds the lock for	Pod(s) believing they own it	inventory
1–3	A, then nobody	A	qty = 100, fence_token = 0
4	B	A and B	unchanged
5	B	A and B	qty = 95, fence_token = 34
6	B	A and B	A is rejected

Drop the fence_token < $token condition and A writes qty = 120 over B’s newer figure. The lock does not prevent this; only the target resource can.

A renewal watchdog does not survive a GC pause

A watchdog (present in some libraries such as Redisson) is a timer that periodically extends the TTL, running in the same process as the job: a GC pause freezes the timer too. In the demo, the watchdog returns 0 when A wakes up, too late.

Checking the lock right before writing is not enough either: the pause can happen right after the check.

Failover: the replica never received the lock key

Redis replication is asynchronous; both Sentinel and Cluster fail over to a replica:

Loading diagram…

Redlock: N independent masters and a quorum

Redlock (redis.io) uses N masters without replication, for example N = 5 on 5 machines:

1. Record the start time, send SET key token NX PX ttl to all 5 nodes in parallel, with a per-node timeout small relative to the TTL.
2. Success means acquiring a majority (≥ 3) while the elapsed time is less than the TTL.
3. Validity time = TTL − elapsed time − clock drift allowance.
4. On failure, release on every node and retry after a random delay.

A node that restarts and loses its keys (see RDB and AOF) can let two clients both reach a majority; the docs recommend delaying restarts beyond the maximum TTL.

Operational details

The 2016 debate around Redlock; redis.io links to both sides:

Point	Martin Kleppmann	antirez (author of Redis)
Fencing	Redlock cannot produce monotonically increasing numbers	If the target can check a token, it can already provide exclusion; check-and-set with a random token works
Time	Relies on assumptions about network delay, pauses, clocks	Only needs to count elapsed time with bounded error
Pause after acquiring the lock	Causes two owners	Affects every self-expiring lock, not just Redlock
Recommendation	Efficiency: one Redis. Correctness: consensus system + fencing	Redlock is usable when exclusion is needed

redis.io now recommends implementing fencing tokens and states that the TTL does not use a monotonic clock: adjusting the system time can grant the lock to multiple processes.

A fencing token needs a durable increasing source. INCR on a replicated Redis can repeat tokens after failover. Safer sources: the target database (version, sequence) or a consensus system (etcd revision, ZooKeeper zxid).

Real-world example

Syncing inventory for SKU-001 across 3 pods

Each pod runs SET ... NX PX 30000; on nil it skips its turn. On OK it INCRs to get a fencing token, reads the supplier, then runs UPDATE inventory ... WHERE fence_token < $token. 0 rows: a newer run has already written, so the job stops.

Demo examples/src/redis-distributed-lock/demo.ts (Redis 7.4.9, PostgreSQL 17): Pod A runs in a worker thread; the GC pause is simulated with Atomics.wait for 900 ms, blocking the thread’s whole event loop. TTL 300 ms, watchdog every 100 ms.

Real output, trimmed (ms timestamps vary between runs)

2. A releases the lock with a plain DEL after the lock has expired
   A: SET NX PX 300 -> OK
   B: SET NX PX 5000 -> OK
   A: DEL -> 1 (deletes B's lock)
   C: SET NX PX 5000 -> OK => B and C both believe they hold the lock
3. Same as step 2, releasing via Lua
   A: EVAL release(tokenA) -> 0 (token mismatch, nothing deleted)
4. Watchdog + GC pause: the lock expires while A is still running, no fencing
   [  47 ms] A: SET NX PX 300 -> OK, fencing token = 33
   [ 249 ms] A: watchdog PEXPIRE 300 -> 1 (still renewable)
   [ 298 ms] A: GC pause of 900 ms begins
   [ 555 ms] B: SET NX PX 5000 -> OK, fencing token = 34
   [ 560 ms] B: write qty = 95 with token 34 -> UPDATE 1 row
   [1198 ms] A: wakes up, writes qty = 120 with token 33
   [1200 ms] A: watchdog PEXPIRE 300 -> 0
   [1200 ms] A: UPDATE -> 1 row
5. Same as step 4, with fencing: UPDATE ... WHERE fence_token < $token
   [1187 ms] A: UPDATE -> 0 rows (rejected: stale token)
   Result: { qty: 95, fence_token: '34', updated_by: 'pod-b' }

Step 4: the old figure 120 overwrites the newer 95. Step 5: PostgreSQL keeps 95. The counter is preset to 32 so the tokens match the diagram.

Trade-offs

Pros

• One command, low latency.

• The TTL frees the lock automatically when the owner dies.

• Good enough for efficiency locks.

< slot="cons">

• Expires while the owner is still running; failover can lose the lock.

• No fencing token; the target resource must check on its own.

• Redlock costs 5 nodes and still relies on timing assumptions.

Cons

Production checklist

1. State explicitly whether the lock is for efficiency or correctness; the latter requires fencing.
2. Release with a token-comparing script (Redis 8.4+: DELEX ... IFEQ); plain DEL is forbidden.
3. TTL larger than the job’s p99 duration; the job has its own deadline and stops when it passes.
4. Every write carries a fencing token from a durable source; the target checks fence_token < $token.
5. The job stays idempotent: fencing does not block external side effects such as calling an API twice.
6. Alert when the release script returns 0: the lock expired midway.

Common mistakes

The most common mistake is in the release step inside finally:

Releasing the lock: a plain DEL also deletes another pod's lock

✕ Common

await redis.set(key, '1', 'PX', 30000, 'NX');
try { await syncInventory(); } finally { await redis.del(key); }

✓ Prefer

const token = randomBytes(16).toString('hex');
if ((await redis.set(key, token, 'PX', 30000, 'NX')) !== 'OK') return;
try { await syncInventory(); } finally { await redis.eval(RELEASE, 1, key, token); }

Why

If the job outlives the TTL, DEL deletes another pod’s lock (demo step 2).

The second mistake is believing renewal is enough:

Watchdogs and Redlock do not replace fencing

The watchdog freezes along with the process during a GC pause; Redlock only reduces the risk of losing the lock because one node died. Neither stops the old owner from waking up and writing; only a target resource that checks the token can block it.

Misconception	Consequence	What to do
Checking the lock before writing is enough	A pause right after the check	Fencing at the target resource
INCR in Redis is a durable token	Failover repeats tokens	Get the token from the database or a consensus system

Self-check

1. In demo step 2, why does A’s DEL return 1 even though A’s lock had expired?
2. The watchdog renews every 100 ms with a 300 ms TTL. Why can B still acquire the lock while A pauses for 900 ms?
3. Describe a sequence of events in which two clients both hold a 3/5 majority of Redlock nodes.

Summary

Key takeaways

• A Redis lock is a lease: SET key token NX PX ttl, released via a token-comparing Lua script, never a plain DEL.
• If the old owner pauses past the TTL, there are two owners at once; a watchdog cannot save it.
• Failover can lose the lock; Redlock reduces the risk but still relies on timing assumptions.
• For correctness: a fencing token checked at the target resource.

References

• Redis Docs — Distributed Locks: single-instance recipe, Redlock, failover, fencing, clocks.
• Redis Docs — SET and SETNX: NX, PX, deprecation.
• Redis Docs — Replication: asynchronous replication.
• Martin Kleppmann — How to do distributed locking: efficiency vs. correctness, fencing tokens.
• antirez — Is Redlock safe?: rebuttal on fencing, clocks, pauses.