Deployment strategies: pick rolling, blue-green or canary by what you can undo Intermediate

The problem

A NestJS checkout service runs as 10 Pods on Kubernetes. Version 2 changes how promotions are calculated. Each successful checkout writes an order row, publishes an OrderPlaced event to Kafka and sends a confirmation email.

A plain kubectl apply gives a rolling update: a pricing bug in v2 reaches every customer within a minute. Undoing the Deployment brings v1 back, but the wrong prices are already in the database, in the events and in customers’ inboxes.

So the question is not “which strategy is best”. It is: how many users may see a bad version, for how long, and what does going back cost?

What to choose? (summary first)

Situation	Choose	Condition
Internal tool, a short outage is acceptable, or two versions must never run together	Recreate	Downtime equals stop time + start time
Most stateless services, low-risk changes	Rolling update (Kubernetes default)	Correct readiness probe; old and new versions compatible with the same schema and messages
Need an instant switch and an instant way back, can pay for 2× capacity	Blue-green	Database and events are compatible with both colours
Risky change on a high-traffic path (checkout, payment)	Canary, ideally with automated analysis	Metrics split by version; enough traffic for the canary to be statistically meaningful
Change is mostly in behaviour, not binary	Feature flag on top of any strategy	The old code path stays until the flag is removed

Loading diagram…

Core concepts

	Recreate	Rolling update	Blue-green	Canary	Progressive delivery
Downtime	Yes	No, if readiness is correct	No	No	No
Extra capacity	None	maxSurge Pods	100% (two full stacks)	A few Pods, or none	Same as canary
Two versions serve at once	No	Yes, for minutes	Briefly (in-flight requests)	Yes, by design	Yes, by design
Blast radius of a bug	Everyone	Grows with each batch	Everyone after the switch	Canary share only	Canary share, stopped automatically
Rollback	Redeploy	New rollout back to old ReplicaSet	Flip the selector back	Scale canary to 0	Controller aborts by itself

Deploy puts code on machines; release lets users reach it. A feature flag separates the two: v2 ships with the new pricing off, then the flag widens exposure without touching Pods.

How it works

Rolling update: readiness decides the pace

The Deployment controller creates a new ReplicaSet and shifts Pods between the two. maxSurge and maxUnavailable both default to 25%. Surge rounds up, unavailable rounds down, so with 10 replicas: at most 13 Pods, at least 8 available.

Rolling update of 10 checkout Pods: readiness and minReadySeconds gate every batch

Enlarge diagram

◀ Prev▶ PlayNext ▶↺ RestartStep 1/6

Before the deploy: 10 v1 Pods are ready; the Service sends all checkout traffic to v1.

1. The controller only scales v1 down by what keeps 8 Pods available.
2. A new Pod receives traffic once its readiness probe passes and every condition in readinessGates is True (for example a load balancer controller confirming registration).
3. It counts as available only after staying ready for minReadySeconds (default 0). Only then does the controller remove more v1 Pods.
4. A v2 that never turns ready stalls the rollout with 8 v1 Pods serving: capacity drops 20% until someone acts.

Kubernetes does not roll back on its own. After progressDeadlineSeconds (default 600), it sets Progressing=False with reason ProgressDeadlineExceeded; a pipeline or person must run kubectl rollout undo.

Blue-green: one switch, one shared database

Blue (v1) and green (v2) both run at full size. Green is tested through a preview Service, then the production selector flips:

# Example only: production Service
apiVersion: v1
kind: Service
metadata:
  name: checkout
spec:
  selector:
    app: checkout
    slot: green   # was: blue
  ports:
    - port: 80
      targetPort: 3000

Rollback is the same flip back, while blue still runs. The catch: both colours share one database and one Kafka topic. Green’s schema changes must be readable by blue, so a destructive migration needs expand–contract whatever the strategy.

Canary: replicas or percentages

Plain Kubernetes splits traffic by Pod count: a track: canary Deployment and a track: stable Deployment behind one Service. One canary Pod next to 9 stable ones gets roughly 10%, and the smallest step is one Pod.

Exact percentages need something that routes by weight: a service mesh, an ingress controller, or a Gateway API HTTPRoute with backendRefs[].weight (weights are proportional, not percentages). Then 1% is possible with 2 canary Pods.

Progressive delivery: the controller watches the metrics

Argo Rollouts and Flagger automate the canary steps and judge each step with queries. In Argo Rollouts, a failed AnalysisRun aborts the rollout and sets the canary weight back to zero:

# Example only (Argo Rollouts v1.10). No trafficRouting:
# setWeight is approximated by replicas (10% of 10 = 1 Pod).
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout
spec:
  replicas: 10
  selector:
    matchLabels: { app: checkout }
  template: { ... }            # same as the Deployment
  strategy:
    canary:
      analysis:                # background analysis during the steps
        templates:
          - templateName: checkout-error-rate
        startingStep: 1
      steps:
        - setWeight: 10
        - pause: { duration: 10m }
        - setWeight: 30
        - pause: { duration: 10m }
        - setWeight: 50
        - pause: {}            # wait for a human to promote
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: checkout-error-rate
spec:
  metrics:
    - name: error-rate
      interval: 1m
      failureLimit: 2
      successCondition: result[0] < 0.005
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{service="checkout",track="canary",code=~"5.."}[5m]))
            / sum(rate(http_requests_total{service="checkout",track="canary"}[5m]))

Flagger uses stepWeight, maxWeight and threshold (failed checks before rollback), with built-in request-success-rate and request-duration metrics.

Operational details

What a rollback cannot undo. Every strategy rolls back Pods; none rolls back the world outside them:

Side effect of v2	After rollback	What to do before releasing
Destructive migration (drop, rename)	v1 fails on the new schema	Expand–contract; contract only after the rollback window
OrderPlaced events in a new format	Already in Kafka; v1 consumers may choke	Additive, versioned event schema
Confirmation emails with wrong prices	Already sent	Canary small; compensating email; flag guarding the new calculation
Rows written by v2	Still there, possibly wrong	Reconciliation query keyed by version

What to watch during a rollout

• Split every SLI by version: add a version or track label to request metrics, or the canary’s 5% error rate hides as 0.5% overall.
• Judge by burn rate, not raw error count. For a 99.9% SLO, the SRE Workbook starting point pages at burn rate 14.4 over 1 hour (confirmed by 5 minutes), which spends 2% of a 30-day budget.
• A canary step lasts minutes, shorter than that window: abort criteria compare canary with stable per step; burn-rate alerts are the safety net.
• Watch business signals: conversion and average discount per order catch a pricing bug that returns HTTP 200.

Real-world example

Checkout v2 on a test cluster: a broken rollout stalls, it does not roll back

The demo applies a 10-replica checkout Deployment (a few lines of Node standing in for the NestJS app) to a single-node k3s v1.33.5 cluster in Docker. The strategy field is omitted; Kubernetes fills in maxSurge: 25%, maxUnavailable: 25%.

Setting VERSION=v2, the ReplicaSets went 10/0 → 8/5 → 3/10 → 0/10 in about 13 seconds. A v3 whose readiness endpoint returns 503 stalled at 8 old + 5 new. After 60 seconds (progressDeadlineSeconds lowered for the demo), it reported ProgressDeadlineExceeded and stayed there.

kubectl rollout undo restored ten v2 Pods in 6 seconds, fast only because 8 v2 Pods were still running. Rolling back a finished rollout recreates every Pod.

Run the demo (needs Docker and kubectl)

docker run -d --name k3s --privileged -p 16443:6443 \
  -e K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml -e K3S_KUBECONFIG_MODE=666 \
  -v "$PWD/k3s:/output" rancher/k3s:v1.33.5-k3s1 server --disable=traefik
sed -i.bak 's#127.0.0.1:6443#127.0.0.1:16443#' k3s/kubeconfig.yaml
export KUBECONFIG=$PWD/k3s/kubeconfig.yaml

kubectl apply -f examples/src/deployment-strategies/checkout-deployment.yaml
kubectl get deploy checkout -o jsonpath='{.spec.strategy}'
# {"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"}

examples/src/deployment-strategies/watch-rs.sh 45 &
kubectl set env deploy/checkout VERSION=v2

Output, abridged (timings vary per run):

t=0s  total/updated/available=10/10/10 | v1=10
t=1s  total/updated/available=13/5/8   | v1=8  v2=5 (not ready)
t=6s  total/updated/available=13/10/8  | v1=3  v2=10 (5 available)
t=13s total/updated/available=10/10/10 | v1=0  v2=10

Broken version, then undo:

kubectl set env deploy/checkout VERSION=v3 READY_FAIL=1
kubectl rollout status deploy/checkout
# error: deployment "checkout" exceeded its progress deadline
kubectl rollout undo deploy/checkout   # back to v2, 6 s in this run

Trade-offs

Canary / progressive delivery gives

• A bug reaches only the canary share, and analysis can stop it without a human.
• Real production traffic tests the change, not a staging guess.
• Rollback is scaling the canary to zero, usually seconds.

And costs

• Needs metrics split by version and enough traffic per step.
• Two versions coexist longer, so schema and events must stay compatible longer.
• Another controller and CRDs to operate; exact weights need a mesh or Gateway.

🎮 Try it

• Canary & feature flags: order the steps to release a risky checkout change.
• Rename a column: put the expand–contract steps in an order that survives a rolling deploy.

Production checklist

1. The readiness probe reflects “can serve checkout” (health probes), and shutdown drains in-flight requests (graceful shutdown).
2. Set minReadySeconds so a Pod that crashes soon after turning ready stops the rollout.
3. Lower progressDeadlineSeconds to a realistic value and make the pipeline run kubectl rollout undo on ProgressDeadlineExceeded.
4. Every migration in the release is backward compatible; destructive steps wait for a later release.
5. Request metrics carry a version/track label; dashboards and alerts compare canary with stable.
6. Abort criteria (error rate, p99, a business metric) are written down before the rollout starts.
7. Rehearse rollback on staging and time it, including the rollback of the feature flag.

Common mistakes

The most common mistake is expecting the platform to undo a bad release:

Waiting for Kubernetes to roll back

✕ Common

kubectl apply -f checkout.yaml   # pipeline reports success and exits

✓ Prefer

kubectl apply -f checkout.yaml
kubectl rollout status deploy/checkout --timeout=5m \
  || kubectl rollout undo deploy/checkout

Why

apply returns once the object is stored, not when Pods are ready. A stalled rollout only sets a condition; something must read it and act.

The second mistake hides inside blue-green, because the switch itself always looks clean:

Blue-green with a migration only green understands

Green’s migration renames a column, then the switch fails its checks. Flipping back to blue now breaks blue as well, because it queries a column that no longer exists. The instant rollback only exists when the schema is compatible with both colours.

Misconception	Consequence	What to do
“The canary looks fine on the overall dashboard”	5% errors on the canary show as about 0.5% overall	Split SLIs by version
setWeight: 5 with 10 replicas means 5%	Without a traffic router, 1 Pod = 10%	Use a mesh or Gateway, or more replicas
A feature flag replaces a safe deploy	A crash on startup ignores the flag	Canary the binary, flag the behaviour

Self-check

1. With 20 replicas and default maxSurge/maxUnavailable, what are the maximum Pod count and the minimum available count during a rollout?
2. A v2 Pod passes readiness, then crashes after 3 seconds. Which setting stops the controller from continuing?
3. The canary emitted 4,000 events in a new format before the abort. What does rolling back the Pods not fix?

Summary

Key takeaways

• Rolling update is the default: 25% surge (rounded up) and 25% unavailable (rounded down), paced by readiness and minReadySeconds.
• Kubernetes reports ProgressDeadlineExceeded but never rolls back by itself.
• Blue-green buys an instant switch with double capacity; the shared database must suit both colours.
• Native canary splits by Pod count; exact percentages need a mesh, ingress or Gateway API weights.
• Rollback restores Pods, not migrations, events or emails; compare canary with stable by burn rate.

References

• Deployments — Kubernetes: strategy defaults and rounding, minReadySeconds, progressDeadlineSeconds.
• Canary deployments — Kubernetes: splitting by replica ratio.
• Pod readiness — Kubernetes: readinessGates.
• HTTP traffic splitting — Gateway API: weighted backendRefs.
• Canary, Analysis and BlueGreen — Argo Rollouts: steps, replica approximation, abort.
• Deployment strategies — Flagger: strategies and analysis fields.
• Alerting on SLOs — Google SRE Workbook: burn-rate alert parameters.
• Canarying Releases — Google SRE Workbook: canary size, duration, metrics.