Pipelining, MULTI/EXEC and Lua: when is a sequence of Redis commands truly atomic? Intermediate

The problem

A flash sale API keeps SKU 42’s stock in Redis: stock:{sku42} (units left) and buyers:{sku42} (the set of buyers). 10 units go on sale and 50 requests arrive at once.

The first version of the code: GET the stock, and if there is some left, SET it minus 1, then SADD.

The demo below lets 50 people buy 10 units. Separately, the job that loads 10,000 keys before the sale opens takes about a second, because it sends one command at a time and waits for each reply.

Both problems share one question: what does batching Redis commands buy you: speed, running back to back without interleaving, or conditional atomicity?

What to choose? (summary first)

Situation	Choose	Condition
Many independent commands (loading keys)	Pipeline	Other clients’ commands can interleave
Several commands must run back to back, no need to read intermediate results	MULTI/EXEC	Runtime errors do not roll back
Check then write, where the written value depends on the read value	Lua (EVALSHA or FUNCTION)	Short script; keys declared via KEYS
Same as above, logic must live in the app, low contention	WATCH + retry	Cap retries; switch to Lua under high contention
A single command is enough (INCR, SET NX)	One command	Each command is already atomic

Core concepts

	No interleaving by other commands	Condition on a read value	Rollback
Pipeline	No	No	No
MULTI/EXEC	Yes	No	No
WATCH + MULTI/EXEC	Yes, aborted if the key changed	In the app, with retries	Aborted before running
Lua / Function	Yes	On the server	No

All four cut round trips (RTT). Atomic here means no other client can slip in midway, not “undo on error”.

How it works

Why does GET then SET oversell even though each Redis command is atomic? Scaled down to 1 unit and 2 clients:

GET then SET from the app sells 1 unit to 2 people; Lua reads and decrements in one go, so it never oversells

◀ Prev▶ PlayNext ▶↺ RestartStep 1/7

One unit left. A calls GET stock:{sku42}, receives 1, and starts processing in the app.

Step	stock	buyers
A and B GET	1	empty
A SET 0, SADD	0	user-a
B SET 0, SADD	0	user-a, user-b (oversold)
Script A (B waits)	0	user-a
Script B: reads 0, writes nothing	0	user-a

The gap lies between the two commands, on the app side: a lost update.

Pipeline: fewer RTTs, same gap

Loading diagram…

The server still runs each command as a standalone command, so other clients’ commands can interleave. A side benefit per the Redis docs: fewer read()/write() calls.

MULTI/EXEC: queue, then run back to back

1. MULTI: every subsequent command returns QUEUED and does not run yet.
2. EXEC: runs the whole queue in one go, with no other client interleaving.
3. A syntax error while queueing: EXEC returns EXECABORT, and no command runs.
4. A runtime error (wrong type): only that command fails; the other commands still run.

Commands inside MULTI have no results yet, so you cannot write “if stock > 0 then decrement”. The condition has to live in WATCH or Lua.

WATCH: optimistic locking

If a WATCHed key changes before EXEC, EXEC returns nil and no command runs; the app rereads and retries. This is Redis’s form of optimistic locking.

Lua: logic running on the server

A script receives keys via KEYS and arguments via ARGV, and runs atomically: every other client waits until the script finishes (see Redis event loop).

-- KEYS[1] = stock:{sku42}, KEYS[2] = buyers:{sku42}, ARGV[1] = user id
local stock = tonumber(redis.call('GET', KEYS[1]) or '0')
if stock <= 0 then return 0 end
if redis.call('SADD', KEYS[2], ARGV[1]) == 0 then return -1 end -- duplicate purchase
redis.call('DECR', KEYS[1])
return 1

Scripts do not roll back either: if an error occurs midway, earlier writes remain (tested: SET then a failing INCR, the key still holds x). Check every condition before the first write.

Operational details

Per redis.io, verified on Redis 7.4.9:

• SCRIPT LOAD / EVALSHA: call a script by its SHA1. The script cache is lost on restart, failover, or SCRIPT FLUSH; EVALSHA then returns NOSCRIPT No matching script. The client must fall back to EVAL (ioredis defineCommand does this automatically).
• FUNCTION LOAD (Redis 7.0+): libraries are stored with the data (RDB, AOF) and replicated; call with FCALL.
• busy-reply-threshold (the Redis 7.0+ name for lua-time-limit; CONFIG GET on either returns 5000): past 5 seconds, other clients get BUSY. The script keeps running.
• Declaring keys: every key a script uses must go through KEYS; Cluster relies on this for routing.
• Cluster: every key in a MULTI or a script must be in the same hash slot. The hash tag {sku42} makes Redis hash only the part inside the braces.
• Large pipelines: the server buffers replies in memory; split into batches (the docs use 10,000 commands as an example).

Tested on a separate cluster node (Redis 7.4.9):

CLUSTER KEYSLOT stock:sku42 → 7674    CLUSTER KEYSLOT buyers:sku42 → 8214
CLUSTER KEYSLOT stock:{sku42} → 6177  CLUSTER KEYSLOT buyers:{sku42} → 6177
EVAL ... 2 stock:sku42 buyers:sku42 u1 → CROSSSLOT Keys in request don't hash to the same slot
MULTI / DECR stock:sku42 / SADD buyers:sku42 u1 / EXEC → CROSSSLOT

Real-world example

A flash sale of 10 units, 50 buyers at once

50 buyers, one connection each, stock = 10. GET/SET oversells by 40 units; Lua and WATCH both stop at exactly 10, but WATCH retries about 250 times.

Demo examples/src/redis-pipelining-lua/demo.ts (ioredis, prefix demo:redis-pipelining:, cleans up when done):

docker compose -f examples/docker-compose.yml up -d redis
npx --prefix examples ts-node examples/src/redis-pipelining-lua/demo.ts

Real output (Redis 7.4.9 in Docker Desktop, Node on the same macOS arm64 machine, via 127.0.0.1:6380):

GET/SET from app: stock=9, buyers=50, oversold=40
Lua EVALSHA: stock=0, buyers=10, oversold=0, returned 1=10, returned 0=40
WATCH + MULTI/EXEC: stock=0, buyers=10, oversold=0, times EXEC returned null (retries)=254
Loading 10000 SETs: sequential=992 ms, pipeline=20 ms, x48.9 faster
pipeline: 20000 INCR then GET in the same batch → GET=8236 (expected 20000)
multi: 20000 INCR then GET in the same batch → GET=20000 (expected 20000)
MULTI per-command results:
  [0] OK
  [1] error: ERR value is not an integer or out of range
  [2] OK
  after EXEC: a=1, b=2
FCALL buy user-1: 1st call=1, 2nd call (duplicate purchase)=-1, stock=9

Reading the output:

• GET/SET: all 50 clients read 10 and all write 9. Stock is left at 9, with 50 people who “bought”.
• Timing: three runs gave 992/1059/1064 ms sequential and 16–20 ms pipelined. This is localhost through Docker; with real network RTT, the gap is even larger.
• A pipeline is not atomic: another client runs SET counter 0 midway, and the GET at the end of the pipeline returns 8236 (other runs: 9905, 9132). Inside MULTI it is always 20000.
• EXECABORT (tested with redis-cli): an INCRBY missing an argument inside MULTI makes EXEC return EXECABORT Transaction discarded because of previous errors., and the preceding SET does not run.

What the demo leaves out for production: timeouts, an idempotency key for repeated requests, and durably recording the order outside Redis.

Trade-offs

Gains

• A pipeline cuts RTT dozens of times over for independent commands
• Lua/Function: check-then-write in one round trip, no retries
• WATCH keeps the logic in the app and does not block the server

Costs

• No mechanism rolls back a runtime error midway
• Slow Lua blocks every client; logic is split between app and server
• WATCH retries explode when many clients contend for one key

Production checklist

1. Name keys with hash tags (stock:{sku42}) from the start, even before using Cluster.
2. Pass every key through KEYS; never build key names inside Lua.
3. Use FUNCTION LOAD, or a client that automatically falls back from NOSCRIPT to EVAL.
4. Keep scripts short and watch SLOWLOG; busy-reply-threshold is only a safety net.
5. Scripts check every condition before the first write.
6. WATCH has a retry cap and backoff with jitter.
7. Split large pipelines into batches and check each reply for errors.

Common mistakes

The most common one: treating MULTI like SQL’s BEGIN.

Reading stock, then decrementing inside MULTI

✕ Common

const stock = Number(await redis.get(STOCK)); // outside the transaction
if (stock > 0) await redis.multi().decr(STOCK).sadd(BUYERS, user).exec();

✓ Prefer

await redis.evalsha(buySha, 2, STOCK, BUYERS, user); // read, check, write on the server

Why

MULTI only keeps DECR and SADD running back to back. The gap between GET and MULTI is still there.

The second trap is in error handling.

Believing that a failed EXEC means nothing was written

EXEC only discards everything on a queueing error (EXECABORT) or when a WATCHed key changed (nil). Runtime errors live inside the result array, and the other commands have already run. You must walk every reply.

Misconception	Consequence	What to do
A pipeline is a transaction	Other clients’ commands interleave	MULTI or Lua
Redis is single-threaded, so GET-SET is safe	Overselling	Combine read and write in Lua
EVALSHA always works	NOSCRIPT after restart/failover	Fall back to EVAL or use Functions

Self-check

1. Why can the GET at the end of a pipeline of 20,000 INCRs return less than 20,000, but not inside MULTI?
2. A MULTI contains SET a, an INCR on "abc", SET b: after EXEC, do a and b exist? What if it is an INCRBY missing an argument?
3. Why does WATCH need retries while a Lua script does not?
4. Why can’t stock:sku42 and buyers:sku42 be used in one script on Cluster?

Summary

Key takeaways

• A pipeline cuts round trips (demo: about 1 second down to 20 ms for 10,000 commands) and is not atomic.
• MULTI/EXEC runs back to back without interleaving; a syntax error discards the transaction, a runtime error does not roll back.
• Check-then-write needs Lua/Function or WATCH + retry; GET then SET from the app oversells.
• Lua is atomic but blocks the whole server; declare keys via KEYS and use hash tags for Cluster.

References

• Pipelining: RTT, syscalls, batching.
• Transactions: EXECABORT, no rollback, WATCH.
• Scripting with Lua: atomicity, KEYS, NOSCRIPT.
• Functions: FUNCTION LOAD, persistence and replication.
• Cluster specification: hash slots, hash tags.