Scale Lab: build a system, run traffic, watch it fall over Basic
Estimated reading time: 4 min read
Your shop grows round by round. Drag components onto the board, wire the boxes together, then press ▶ Run.
- 8 rounds, from your first app to a flash sale and a Redis restart at peak hour.
- Pass a round by answering ≥ 99% of requests with average response time within the limit and cost within budget. The cheapest design earns 3 stars.
- What if it breaks? The failed box is circled in red, and the feedback says why. Click a feedback item to jump to that box.
- Wiring: drag from one box’s dot to another box, or click a box, then 🔗 Connect to another box and tap a highlighted box. Removing a wire: click the wire, then ✕, or use the Remove button in the box’s panel.
- On phones: the flow runs top to bottom; tap a component to add it, tap a box to configure it.
Finished all 8 rounds? Try the multi-round interview problems (your design carries over between rounds):
| Problem | Difficulty | Main pressure |
|---|---|---|
| URL shortener | Medium | Read-heavy, viral links, spam bots |
| Notifications | Medium | Slow and failing provider, retries, duplicate sends |
| Concert ticketing | Hard | One hot inventory row, row locks, scalper bots |
| Realtime chat | Hard | Polling or WebSocket, Pub/Sub between gateways, reconnect storm |
Or set your own traffic and incidents in the Sandbox. The numbers in the game are round assumptions meant to show trends; see the model assumptions.
Model assumptions
Section titled “Model assumptions”| Component | Capacity per instance (S / M / L) | Price/month |
|---|---|---|
| Server | 500 / 1,000 / 2,000 req/s | $15 / $30 / $60 |
| Database, Read Replica | 500 / 1,000 / 2,000 query/s | $15 / $30 / $60 |
| Cache | 10k / 25k / 50k ops/s, 90% hit rate when warm | $20 / $40 / $80 |
| CDN | absorbs 95% of static file requests | $20 / $40 / $80 |
| Load Balancer, Rate Limiter | LB 10k / 25k / 50k; Rate Limiter 20k / 50k / 100k req/s | $10 / $20 / $40 |
| Queue / Worker | Worker 100 / 200 / 400 job/s | $10 / $20 / $40 |
| WebSocket Gateway | 10k / 20k / 40k connections | $15 / $30 / $60 |
| Pub/Sub | 50k / 100k / 200k msg/s | $20 / $40 / $80 |
- Every dynamic request goes through a server and produces one query: reads go to the cache (if any) then the database/replica, writes go to the primary.
- Service time grows with utilization ρ = load / capacity, queueing-style:
base / (1 − ρ). Above 100%, the excess is dropped. - Adding server replicas only helps when a Load Balancer sits in front; the Load Balancer takes dead replicas out of rotation.
- With request coalescing enabled on the cache, only ~5% of the extra misses reach the database when the cache is empty; the rest wait for the result of the in-flight query.
- The Rate Limiter returns 429 to bots at the edge; real users’ requests pass through normally.
- Hot row (ticketing problem): a row lets only one transaction hold its lock, so it can be decremented at most
1000 / lock hold time (ms)times per second, multiplied by the number of rows if inventory is split. - Lock hold time (ticketing problem): calling payment inside the transaction ~500 ms,
FOR UPDATE~20 ms, a singleUPDATEstatement ~2 ms. - Chat: without a gateway, clients poll the server every 2 seconds (messages arrive 1 second late on average). With 2 or more gateway instances but no Pub/Sub, messages to recipients on another gateway are lost.
- Chat reconnects: when a gateway dies, its clients reconnect, and every reconnect is an authentication request. Without jitter they all land in the same second; with jitter they spread over ~20 seconds.
- Provider outage (notifications problem): every send fails. Without retries the message is dropped; with retries it goes back to the queue, and ~5% of the “failed” sends actually arrived, so they are sent twice unless the worker deduplicates.
This model is deliberately simple to show the right trends (bottlenecks, the cost of each choice), not to predict real numbers. Real systems need measurement: see SLI, SLO and error budget.