Ordering: a Pod gets replaced during a rolling deploy Intermediate
Estimated reading time: 1 min read
On every rolling deploy, the few 502s and reprocessed messages usually come from one place: the shutdown steps run in the wrong order, or the app assumes traffic has stopped when it has not.
Order the steps below, from the moment the Pod is deleted to the moment the container stops. Press Check to see which steps are out of place and what breaks when they are.
A Pod replaced during a rolling deploy: what happens, and in what order
The orders-api Pod both serves HTTP and consumes the payments topic from Kafka. Spec: terminationGracePeriodSeconds: 30, preStop: sleep 5, and an exec-form Dockerfile so Node is PID 1.
A rolling deploy deletes this Pod. Order the steps from Pod deletion to container stop. Two things that run in parallel have been merged into one step.
Use ▲▼ to move each step, or tap a step’s ⠿ and then tap its new position. On a computer you can drag and drop.
- The API server marks the Pod as being deleted (Terminating); the 30 s grace period clock starts
- Close the DB pool, flush logs and telemetry
- The app stops accepting new work: readiness returns 503,
server.close(), the consumer stops polling new messages preStopfinishes, the kubelet sendsSIGTERMto the container's PID 1- Wait for in-flight work: running HTTP requests send their responses, messages being processed commit to the DB, then commit offsets and leave the consumer group
- The process exits on its own with exit code 0, before the 30 s run out
Too late: the kubelet sends SIGKILL - In parallel: the kubelet runs
preStop(sleep 5), while the EndpointSlice marks the endpointready: falseand kube-proxy/ingress gradually stop sending new traffic
Time budget, exec-form Dockerfile and drain code for Node: SIGTERM during deploys.