Scenario: checkout p99 hits 1.2 s while CPU sits at 40% of the limit Intermediate

A CPU limit is a quota per 100 ms, shared by every thread in the container. A Node service can burn it in a few milliseconds of parallel hashing, then wait, while the 1-minute average still reads “40%”.

• Four steps: find the signal, resist the quick fixes, fix the fleet, add guardrails.
• Hint: compare what each metric averages over with how long a CFS period lasts.
• Only your first choice at each step counts toward the score. The numbers are illustrative.

Scenario

Checkout p99 hits 1.2 s while CPU sits at 40% of the limit

You are the backend on-call for checkout-api (NestJS, Node 24, 6 replicas, each cpu: 1 and memory: 768Mi as request and limit). Every choice moves the metrics below right away.

p99 of POST /checkout1,200 ms

Throttled CFS periods38 %

504 at the gateway (1 s timeout)0.8 %

CPU requested by checkout-api (cost)6 cores

Step 1/4 · Slow checkout, quiet dashboards

A promotion started at 20:00. p99 of POST /checkout went from 180 ms to 1.2 s, and the gateway returns 504 for requests over 1 s. The database p99 is a flat 8 ms.

The standard dashboard shows each Pod at 40% of its CPU limit, no restarts, no OOMKilled. The HPA (target 70% CPU) has not moved. Where do you look first?

Pod CPU, 1-min average

~400m of 1000m

database p99

8 ms, unchanged

traffic

+60%, half of it guest checkout

HPA

40% / target 70%, 6 replicas

$ kubectl top pod -l app=checkout-api
NAME                          CPU(cores)   MEMORY(bytes)
checkout-api-7c9d-2kq8n       412m         431Mi
checkout-api-7c9d-5wz7r       398m         428Mi
checkout-api-7c9d-9hd4m       405m         440Mi
...

What do you do?

ACPU is at 40%, so this is not a CPU problem. Look for a slow downstream call insteadBGraph the throttling ratio per Podrate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m])CCheck node CPU: if nodes are not saturated, CPU is fineDOpen traces of slow checkouts

The mechanism, with a measured Docker demo, is in Kubernetes requests and limits.