Kubernetes Liveness Is Not Readiness: Why a Health Check Can Amplify an Outage
KUBERNETES FAILURE RECOVERY
A failed health check is an observation. Restarting a container and withholding traffic are different responses, and choosing the wrong one can amplify an outage.
By Hossam
The important question is what failure should trigger
A web application loses access to its shared database. Its processes are still running, but requests that require database access fail. A health endpoint reports the dependency problem. Should Kubernetes restart every application container, or stop directing those requests to it? The endpoint alone cannot choose the right recovery action.
Kubernetes separates startup, liveness and readiness probes because they answer different operational questions. Startup protects initialization, liveness can trigger container termination and restart handling, and readiness controls eligibility for traffic from matching Services. A green check is meaningful only within the check and response that were designed.
This is a documentation-based design guide, not a report of a cluster experiment. No production outage, benchmark or deployment test was performed for this article. The examples are explanatory scenarios. Check documentation for your cluster version before implementing a configuration; this article does not rely on newer alpha probe features.
Sources: Liveness, Readiness, and Startup Probes, Configure Liveness, Readiness and Startup Probes.
Three probes, three failure boundaries
A startup probe asks whether the application has completed the startup condition that the endpoint represents. When configured, it prevents liveness and readiness probes from executing until it succeeds. Repeated startup failures reaching the configured threshold cause the kubelet to kill the container, with subsequent handling governed by its restart policy.
A liveness probe asks whether the application has entered a state for which terminating the container is an appropriate remedy. Kubernetes gives a deadlock as an example. Once the failure threshold is reached, the action targets that container. This is not a promise that the entire Pod will be recreated on another node, or that restarting will repair its dependencies.
A readiness probe asks whether the container is ready to accept traffic. On failure, the container continues running while the kubelet keeps checking. The Pod becomes not ready. For normal selector-based Service routing, that removes it from the set of ready backends. It does not itself restart the container.
Sources: Liveness, Readiness, and Startup Probes, Pod Lifecycle.
Do not turn a shared dependency outage into a restart wave
Kubernetes documents a useful separation for strict backend dependencies: liveness can report that the application itself is healthy while readiness additionally checks the required backend. If the database is unavailable, failing readiness can prevent directing ordinary Service traffic to an application that cannot satisfy it. Failing liveness solely for the same external outage may restart healthy local processes without restoring the database.
That distinction is conditional, not a rule that every dependency belongs in readiness. If an optional recommendation service is down but the application can still complete checkout, withdrawing the whole application may discard useful capacity. Decide which operations must work for the Pod to be a suitable backend, and which failures should instead produce a degraded response.
Nor does separating endpoints guarantee safety. If every replica depends on the same unavailable system, every replica can become unready together. Readiness makes that inability visible to routing; it does not create a healthy alternative. Recovery still needs dependency repair, appropriate client behavior and an explicit decision about degraded service.
Sources: Liveness, Readiness, and Startup Probes, Addressing Cascading Failures.
An open port does not prove that a useful request works
A TCP probe succeeds when a connection can be established. The Kubernetes documentation notes that a remote peer closing the connection immediately after it opens still counts as healthy. That can be a suitable narrow check, but it is not evidence that authentication, database access or a representative application operation works.
An HTTP probe judges the response status, with codes from 200 through 399 counted as successful. It does not interpret an application-specific JSON field that says a dependency is down. A handler returning HTTP 200 with an error message in its body therefore does not communicate probe failure through that message.
Keep the health endpoint small and give it an intentional contract. A local process-progress check, a required-dependency readiness check and an end-to-end user transaction can all be useful, but they establish different facts. Name them accordingly rather than treating any responding URL as proof that the service works.
Sources: Liveness, Readiness, and Startup Probes, Configure Liveness, Readiness and Startup Probes.
Startup protection is not an end-to-end availability deadline
A slow initialization sequence can look like a deadlock to an aggressively configured liveness probe. A startup probe provides a separate allowance so that steady-state liveness detection does not need to tolerate the full initialization duration every time.
The Kubernetes configuration guide illustrates failureThreshold 30 and periodSeconds 10 as a five-minute startup allowance. That product is useful when choosing the startup budget, but it is not a complete service-restoration guarantee. Image pulling, scheduling, shutdown grace, restart handling and the application becoming useful can sit outside the simplified probe calculation.
Choose the startup condition carefully. A process that has merely opened its health port may still need to load data before accepting traffic. Once startup succeeds, readiness must continue to express whether that later work permits serving requests. Conversely, a startup endpoint that unnecessarily waits on an unavailable external dependency can keep triggering startup termination rather than letting a healthy local process wait for recovery.
Sources: Liveness, Readiness, and Startup Probes, Configure Liveness, Readiness and Startup Probes, Pod Lifecycle.
Probe timing is a policy, not a magic constant
periodSeconds controls the interval between checks, timeoutSeconds limits how long a check can take, and failureThreshold controls consecutive failures before the failure response. successThreshold controls the consecutive successes needed after failure, but must be one for liveness and startup probes. The fields are not interchangeable.
Readiness checks can run at times other than the configured period while the container is not ready, to make it ready faster. Liveness does not wait for readiness to succeed. A failed readiness check therefore cannot be assumed to pause an independently configured liveness check.
Use measured initialization times and realistic overload tests to choose settings. A timeout that is harmless on an idle development machine may classify a temporarily starved server as dead under load. Raising every threshold is not a universal fix either: it can leave a genuinely stuck container serving, or eligible to serve, for longer. Record the failure mode each setting is intended to handle.
Sources: Liveness, Readiness, and Startup Probes, Addressing Cascading Failures.
Readiness is routing information, not a firewall
The simplified phrase that an unready Pod receives no traffic needs a boundary. EndpointSlices retain backend addresses and conditions, and consumers use that information for routing. The endpoint does not have to disappear from the API object for ordinary Service traffic eligibility to change.
Kubernetes documents an important exception: for a Service with spec.publishNotReadyAddresses set to true, the EndpointSlice ready condition is always true. Do not assume that adding a readiness probe overrides a Service explicitly configured to publish not-ready addresses.
Readiness should not be used as a security barrier. A direct client of a Pod address, or a separate proxy with its own behavior, is outside a blanket claim about normal Service selection. Inspect the actual traffic path and its handling of endpoint conditions. Changing readiness is also not, by itself, a complete strategy for already established connections.
Sources: EndpointSlices, Explore Termination Behavior for Pods And Their Endpoints.
Overload can make the recovery response become the problem
Kubernetes explicitly warns that incorrect liveness probes can cause cascading failures: high load leads to restarts, lost capacity increases pressure on remaining Pods, and more requests fail. Restarting can remove the local work and warm state that might otherwise have recovered once pressure eased.
Readiness is not immune to the same capacity feedback. Google's SRE discussion explains that servers failing health checks without crashing can reduce available load-balancing capacity much like crashed servers. Moving all traffic away from a temporarily slow replica can overload the ones that remain.
The response needs more than probe tuning. Consider cheap rejection of excess work, bounded queues, a degraded response where the product permits it, and retry behavior that does not multiply the incident. These are design options to test, not features automatically supplied by a readiness endpoint. Keeping liveness cheap does not establish that all application requests will remain healthy.
Sources: Liveness, Readiness, and Startup Probes, Addressing Cascading Failures.
A fictional failure review with three different answers
Imagine Cedar runs four identical application replicas. The shared database becomes unavailable. A readiness endpoint reports that required database work cannot complete, while the local liveness endpoint still succeeds. The processes stay running and ordinary Service routing stops treating them as ready. That behavior is consistent with the chosen contracts, but users still need an appropriate unavailable response or a separate degraded path.
In a second scenario, one replica deadlocks while the database and other replicas remain healthy. A liveness endpoint that actually detects the loss of local progress can trigger termination and restart handling for that container. A handler that always returns success from an unrelated thread might miss the deadlock. The endpoint must observe the state that motivates the remedy.
In a third scenario, all replicas are overloaded. A very short liveness timeout begins restarting them, reducing usable capacity further. Replacing it with an equally aggressive readiness cutoff may still shift too much traffic onto the remaining backends. The diagnosis is not that probes are useless; the configured reaction and the capacity model have to be tested together. These scenarios are invented illustrations, not measured results.
Sources: Liveness, Readiness, and Startup Probes, Addressing Cascading Failures.
Termination and connection draining are a separate review
A Pod being deleted is different from a container temporarily failing readiness. Kubernetes updates terminating endpoint conditions so ordinary load balancers can stop selecting that endpoint for regular traffic, while the application may still be processing work during its grace period.
The termination tutorial shows an endpoint with ready false, serving true and terminating true. Those fields let clients that understand them make draining decisions. EndpointSlice documentation also notes that Service proxies may route to serving, terminating endpoints if all available endpoints are terminating. The exact traffic-policy behavior matters; a single slogan about removing the Pod is too broad.
Review application shutdown, grace periods and the actual proxy or load balancer together. A readiness endpoint alone does not prove that existing work finishes cleanly or that clients will never encounter a request during a transition. Test the path the application uses rather than inferring connection behavior only from a Pod status display.
Sources: EndpointSlices, Explore Termination Behavior for Pods And Their Endpoints.
What to verify before calling the design healthy
Write down what each endpoint measures, its expected success and failure responses, and the Kubernetes action it can trigger. Distinguish a local failure that restart can repair from a dependency failure that restart cannot repair. If a process already exits reliably when it becomes irrecoverable, Kubernetes notes that a separate liveness probe may not be necessary.
Then test slow startup, a single local deadlock, a required backend outage, an optional backend outage, overload and graceful termination in a suitable non-production environment. Observe container restart counts, Pod conditions, EndpointSlice conditions, application errors and user-visible outcomes together. Passing one health request is not acceptance evidence for every case.
Keep the manifest, cluster version, endpoint implementation and test conditions with the result. The useful conclusion is specific: this check detected this failure and the resulting action improved this outcome under these conditions. That is stronger engineering evidence than a dashboard full of green checks with undefined meaning.
Sources: Liveness, Readiness, and Startup Probes, Pod Lifecycle, EndpointSlices, Addressing Cascading Failures.
Related reading
Health contracts and artifact identity solve different deployment risks. For the latter, see container image digests, signatures and provenance.
Sources and scope
This guide uses the official Kubernetes documentation and Google's SRE chapter listed below. The Kubernetes pages are maintained documentation, not a claim about features supported by every installed cluster. No workload was deployed or benchmarked for this article.
Comments
Post a Comment