Kubernetes probes: startup, readiness and liveness handle three different states Basic

The problem

An API needs 40 seconds to load data on startup. Kubernetes calls /health after a few seconds and sees an error.

1. If /health is also the liveness probe, the container is restarted before it can initialize, and this repeats forever.
2. “Fix” it by making /health return 200 immediately, and the Pod receives traffic before it has opened its database connection, returning 500 to users.
3. Finally, a database hiccup of a few seconds drags every Pod into a restart loop, and the database that just recovered faces the startup load of the whole fleet at once.

These three situations are three different questions, which is why there are three probes:

Probe	Question	On failure
Startup	Has initialization finished?	Kill the container, apply the restart policy
Readiness	Can it receive traffic?	Remove the Pod from the EndpointSlice, no restart
Liveness	Is the process stuck beyond recovery?	Restart the container after the threshold

The rule: readiness or liveness?

Ask one question: would restarting this container fix the problem?

• Yes (deadlock, stuck event loop, internally broken process) → liveness.
• No, but the Pod cannot serve yet/right now (loading, lost connection) → readiness.
• No, and it does not affect core functionality → no probe at all.

Incident	Failing probe	Why
Handler deadlock, stuck event loop	Liveness	Only a restart gets out of it
Loading an index/cache at startup	Startup (then readiness)	Wait, don’t restart
This Pod’s connection pool is broken, other Pods are fine	Readiness	Pull just this Pod; it recovers on its own
A shared database is down	Careful, see below	Restarting is useless; failing readiness on the whole fleet = no endpoints left
A secondary dependency (email, product recommendations) fails	No probe	It only degrades secondary features

Shared dependencies: beware of pulling the whole fleet

Kubernetes suggests readiness may check required backends.

But if the shared database is down and every Pod’s readiness calls it, every Pod goes not ready together, the Service runs out of endpoints and users get errors from the load balancer. That is usually worse than an API that returns controlled errors or serves cached data.

Principle: readiness checks what this Pod itself needs and what may differ from other Pods (initialized, pool still connected, internal queue not full). For shared dependencies, prefer degraded mode, a circuit breaker or sensible request-level timeouts, and alert on SLOs instead of letting a probe decide.

Liveness never calls external dependencies. The Kubernetes docs warn that misconfigured liveness causes cascading failures: restarts under high load, failed client requests, and load piling onto the remaining Pods.

How it works

The kubelet checks the Pod; the Service only sends traffic once readiness passes

Enlarge diagram

kubelet → Podstartup

◀ Prev▶ PlayNext ▶↺ RestartStep 1/5

The container starts. The startup probe has not succeeded yet, so liveness/readiness checks have not started.

At each stage, each probe has its own state:

The state of each probe across the Pod lifecycle

◀ Prev▶ PlayNext ▶↺ RestartStep 1/5

Starting up: only the startup probe runs. Liveness and readiness wait until startup succeeds.

What to remember from the timeline:

• With a startup probe, liveness and readiness do not run until it passes. After that, it does not run again.
• Readiness runs throughout the Pod’s lifetime, so a Pod can leave and rejoin the endpoints many times without restarting.
• Liveness does not wait for readiness; the two probes are independent.

Startup probe or initialDelaySeconds?

Both delay liveness.

• initialDelaySeconds is a constant: set it to 60 seconds and an app that starts in 5 seconds still waits 60 seconds.
• A startup probe tries every periodSeconds and hands over to liveness as soon as the app is ready; the maximum window is failureThreshold × periodSeconds (for example 30 × 10 = 300 seconds in the Kubernetes docs).

For apps that start slowly or with variable startup time, use a startup probe.

Commonly tuned parameters

Field	Meaning	Default
periodSeconds	How often the probe is called	10
timeoutSeconds	Wait time per attempt	1
failureThreshold	Consecutive failures to count as failed	3
successThreshold	Consecutive successes to count as passing again	1

A successThreshold greater than 1 is only allowed for readiness (liveness and startup must be 1). Use it so the Pod doesn’t keep flapping in and out of endpoints when a dependency is flaky.

Minimal manifest

startupProbe:
  httpGet: { path: /health/startup, port: 3000 }
  periodSeconds: 5
  failureThreshold: 24          # window of about 120 seconds
readinessProbe:
  httpGet: { path: /health/ready, port: 3000 }
  periodSeconds: 5
  timeoutSeconds: 2
  successThreshold: 2           # must pass twice in a row before taking traffic again
livenessProbe:
  httpGet: { path: /health/live, port: 3000 }
  periodSeconds: 10
  failureThreshold: 3

/health/live only answers “is the process still handling requests”, without calling the database. For gRPC services, the kubelet calls the gRPC health check directly (stable since Kubernetes v1.27):

livenessProbe:
  grpc:
    port: 9090      # the server must implement the gRPC Health Checking Protocol
  periodSeconds: 10

gRPC probes run against the Pod IP, do not use TLS by default, and any error counts as a failure.

When a Pod is deleted

Readiness is not a draining mechanism. When a Pod is deleted, its endpoint is updated with ready = false regardless of probes, so you don’t need a readiness probe just for draining.

But this update does not propagate instantly, so you still need preStop, SIGTERM handling and a grace period; see SIGTERM during deploys.

When to use / not to use

Probe	Use when	Can skip when
Startup	Slow or unstable startup (loading indexes, warming caches, light migrations)	The app is ready within a few seconds
Readiness	The Pod receives traffic through a Service	The workload receives no traffic (a pure queue-pulling worker)
Liveness	There are failure modes where the process gets stuck without exiting	The process crashes on error: the restart policy already handles it

Real-world example

A product API that loads its index for 30–60 seconds

The startup probe has a 120-second window, enough for a legitimately slow load yet short enough to catch a hung loader. Readiness passes once the index is loaded and the Pod’s connection pool is usable. Liveness only checks that internal handlers still respond, without calling the datastore.

When the datastore loses connectivity for a few seconds, each Pod pulls itself out of the endpoints and comes back; no Pod restarts and the index cache stays intact.

When the loader hangs for more than 120 seconds, the container restarts and an alert fires. (The thresholds are examples; choose yours from real startup and latency measurements.)

The demo simulates 5 states and prints three booleans for each (matching the timeline above):

npx --prefix examples ts-node examples/src/k8s-probes/probe-demo.ts

For dependency outage expect readiness: false, liveness: true; for process stuck, liveness: false. The demo only simulates the endpoints’ decisions; there is no kubelet, thresholds or EndpointSlice.

To verify for real, try it on a test cluster:

• Break a dependency: kubectl get endpointslice changes but the restart count does not increase.
• Make the process hang: kubectl describe pod records a liveness failure event followed by a restart.

Trade-offs

Separating three signals

• Startup allows a slow start while keeping liveness strict afterwards.
• Readiness pulls a Pod out of traffic without restarting it.
• Liveness recovers a stuck process that won’t exit on its own.

Costs and risks

• Wrong health endpoints cause premature traffic or restart storms.
• Probes that check dependencies too deeply/expensively become a source of load and failure spread.
• No probe replaces graceful shutdown, timeouts, retries or business error metrics.

Common mistakes

Common pitfall

One /health that pings the database for both readiness and liveness. When the database fails, every container restarts, loses its cache, and adds load to the dependency that just recovered. Split the two endpoints, and keep liveness away from external dependencies.

Common pitfall

Readiness calling a shared database. A slow database makes the whole fleet go not ready together, and the Service runs out of endpoints. Only check things specific to the Pod; handle shared dependency failures with degraded mode.

Common pitfall

Returning 200 as soon as the process listens. An open port does not mean initialization data or the pool are ready. Readiness must reflect the conditions real requests require.

Common pitfall

timeoutSeconds too low under load. Probe timeouts caused by GC or high CPU can trigger a restart storm exactly when the system is already weak. Keep liveness cheap, separate it from heavy processing paths, and keep failureThreshold wide enough.

🎮 Try it

• Probe restart cascade: a liveness probe that calls the database restarts the whole fleet.

Production checklist

1. Three separate endpoints: /startup, /ready, /live; each is cheap and time-bounded.
2. Liveness does not call the database, cache or other services.
3. Readiness only checks things specific to the Pod; there is a degraded-mode plan for shared dependencies.
4. The startup probe window is derived from real startup time (p99), not copied from a sample manifest.
5. Readiness successThreshold is enough to avoid flapping; liveness failureThreshold is enough to ride out GC pauses.
6. terminationGracePeriodSeconds, preStop and SIGTERM handling are configured per the graceful shutdown article.
7. Alerts exist for a rising restart count and for ready endpoints dropping to 0.
8. Dependency failures and stuck processes have been tested on a test cluster, confirming the right probe fails.

Summary

Key takeaways

• Startup: has initialization finished. Readiness: can it take traffic. Liveness: does it need a restart.
• The rule: would a restart fix it? If yes, liveness; if no, readiness or no probe.
• Readiness only checks things specific to the Pod; shared dependencies are handled with degraded mode.
• A deleted Pod leaves the endpoints on its own but not instantly: you still need graceful shutdown.

Self-check: (1) Why can a liveness probe that pings the database make an outage worse? (2) How is a startup probe better than initialDelaySeconds? (3) When a shared database dies, what should readiness do? (4) Does a failing readiness probe guarantee that in-flight requests complete?

References

• Liveness, Readiness, and Startup Probes — Kubernetes
• Configure probes — Kubernetes