Scenario: a 3 a.m. incident, checkout p99 up to 4 seconds Intermediate
Estimated reading time: 1 min read
The pager goes off at 3 a.m.: checkout is 20 times slower, errors are climbing, and there was a deploy 40 minutes ago. You go through four decisions: the first action, an update for stakeholders, finding the cause and writing the postmortem.
- Read the evidence carefully before clicking: CPU, event loop delay and
pg_stat_activityalready point most of the way. - If you pick wrong, read the explanation and Try again; scoring counts only your first choice at each step.
3 a.m. incident: checkout p99 goes from 200 ms to 4 seconds
You are the backend on-call for the checkout team. Tonight it is just you and a secondary who is being paged awake. All timestamps below are UTC (03:07 Vietnam time = 20:07 UTC).
Step 1/4 · The pager fires at 03:07
Checkout SLO burn-rate alert. p99 has gone from 200 ms to over 4 seconds, and 5xx errors are rising. You have acked, opened the incident doc and declared SEV2, acting as IC yourself.
Most recent change: api v3.14 deployed at 19:27 UTC, 40 minutes ago, adding loyalty points on payment. This release has no migration.
- p99 starts climbing
- 19:58 UTC
- api CPU
- 15%
- event loop delay p99
- 11 ms
- burn rate
- ~250x
-- pg_stat_activity at 20:08 UTC state | wait_event_type | wait_event | count ---------------------+-----------------+---------------+------ idle in transaction | Client | ClientRead | 41 active | Lock | transactionid | 118 idle | Client | ClientRead | 9 -- api log Error: timeout exceeded when trying to connect (pool 30/30 per pod, waiting 300+)
What do you do?
After playing, read Incident response for the “symptom → what to look at → what to mitigate” table and a full on-call checklist.