Scenario: a 5-million-follower account posts and every feed falls 40 minutes behind Advanced

Fan-out on write is fast to read and cheap on average. This scenario is about the tail: one account with 5 million followers, and a feed that pushes every post into every follower’s timeline. All capacity numbers are illustrative and consistent with each other.

• Four steps: diagnose, mitigate, redesign, then handle the edge cases.
• Compute writes per post before choosing; most wrong answers ignore that number.

Scenario

A 5-million-follower account posts a thread, and everyone's feed falls 40 minutes behind

You are the on-call backend dev for the feed. Timelines are Redis sorted sets filled by fan-out on write: every post is pushed into each follower's tl:{user}. Every choice immediately shows its consequences on the metrics.

Fan-out lag (ordinary posts)40 min

Redis RSS per node58 GB

Timeline writes per @star post5,000,000

Feed read p9945 ms

Step 1/4 · What is backing up?

20:00, the evening peak. At 19:55 the account @star (5 million followers) posted a 3-part thread within 5 minutes. Since then, support tickets say friends' posts show up in the feed half an hour late.

The fan-out lag alert reads 40 minutes. Redis nodes are close to their 64 GB limit.

fan-out lag

40 min and rising

fan-out workers

8,000 timeline writes/s, all busy

normal evening load

~2,000 timeline writes/s

Redis RSS

58 GB of 64 GB

What do you do?

AScale fan-out workers from 20 to 80 right awayBBreak the backlog down by producer (author) and compute writes per postpending timeline writes grouped by author_id; writes per post = followers fanned out toCLook at total queue depth and consumer lag in messagesDRestart the fan-out workers, they look stuck

When you finish, read Designing a news feed for the load estimate, the data model and the hybrid threshold.