Add a health check that detects useful failure states

A health endpoint returning success because the process can answer HTTP is useful evidence, but it does not prove that normal requests can succeed. At the other extreme, a liveness check that fails whenever the database is briefly unavailable can restart every application instance during a shared outage. The useful design distinguishes process viability, traffic readiness, and startup progress.

Consider an illustrative API that needs initialization and database access before handling normal requests. It also sends optional notifications through a provider. The database is critical for core traffic, while notification failure can be represented as a pending state. The probes should reflect these differences instead of treating every dependency as equally important.

Define what each check means

Liveness asks whether the process should continue running or be restarted. Readiness asks whether the instance should receive normal traffic. Startup checks allow initialization to finish before ordinary liveness behavior begins where the platform supports that distinction. Write these meanings in the application's operational documentation.

Do not use one broad check for all three simply because it is easy to implement. A live process waiting for a database may be temporarily unready without needing a restart. An instance performing a long initialization may need a startup allowance. A background worker has another readiness question: can it safely claim and process work? The correct check depends on the action the platform takes when it fails, not merely on the status code returned.

Identify critical dependencies and degraded modes

List dependencies by the user operations they support. Database access may be required for every authenticated request. Email may be optional for browsing but consequential for recovery flows. An unavailable analytics collector should usually not remove the application from service. Define which failures produce degraded behavior and which make the instance unsuitable for traffic.

Use this list to keep checks small. A readiness endpoint may need a cheap bounded validation of a critical dependency. It should not run a complete report or exercise every provider. Expensive probes can consume scarce resources during the outage they are detecting. Avoid writes unless the health design explicitly requires a controlled synthetic operation; routine health checks should not create customer-visible records.

Keep liveness independent where possible

A liveness failure should indicate a state that restarting can plausibly improve. If the process is unable to serve its own basic handler because it is irrecoverably stuck, restart may help. If a shared provider is down, restarting application replicas usually does not repair that provider and may make recovery harder through repeated initialization.

Implement a cheap process-level endpoint or signal appropriate to the runtime. Be cautious about asserting that one successful endpoint proves the whole event loop or worker system is healthy. It is one observation. Document what the liveness check detects and what it misses. Use external monitoring for user journeys instead of trying to make a simple restart signal prove application correctness.

Bound readiness work

Set explicit timeouts for dependency checks and avoid unbounded connection creation. Reuse supported connection behavior where appropriate, and ensure concurrent probes do not multiply work. Return a stable status with a safe diagnostic category. Do not expose credentials, detailed topology, or private errors through a public endpoint.

Consider briefly cached readiness state when checking a dependency is costly, while defining the acceptable delay in detecting failure and recovery. Thresholds and probe frequency should match expected startup and transient behavior. An overly aggressive configuration can flap traffic routing; an overly loose one can send users to broken instances for too long. Select values from observed behavior and operational requirements, not from a copied example presented as universally correct.

Verify platform behavior, not only the endpoint

In an isolated deployment, delay initialization and observe whether startup handling prevents premature restart. Make the critical dependency unavailable and confirm readiness changes as intended. Verify that traffic is removed from the instance and returns after recovery. Check that liveness does not trigger a needless restart for the same temporary outage.

Inspect platform events and application logs together. A healthy endpoint with a misconfigured path or port is useless to the platform. A probe can fail because its timeout is too short, because the process listens on a different interface, or because the platform cannot reach it. Test the actual configuration. Keep the result separate from the application's response when called manually from inside the container.

Add a synthetic journey separately

A readiness probe may confirm connectivity while an authorization query or a route-specific serializer is broken. Add a controlled synthetic check for a critical journey when its value justifies the cost. Use a dedicated test identity and safe data, and ensure it cannot modify real customer state.

For example, an authenticated read of a synthetic account can test routing, identity handling, and data access together. Keep it separate from the frequent readiness endpoint so a route bug does not create a restart storm. Record which journey it covers and its limitations. Connect these results to deployment evidence. Health signals should support decisions, not create an illusion that every feature is continuously verified.

Handle draining and worker shutdown

Readiness matters during deliberate shutdown as well as failure. Stop accepting new traffic or jobs before terminating work, using the platform's supported lifecycle behavior. Give in-flight work a bounded completion period and define what happens afterward. A worker must not leave a permanently claimed job if it exits during shutdown.

Do not return ready while the instance is already draining. Conversely, do not assume that readiness removal cancels existing requests. Long-running work needs its own recovery design. Document how application shutdown, worker leases, and load-balancer routing interact. A well-designed probe is one part of operational lifecycle management; it cannot replace durable job state or a graceful termination policy.

Ask AI to review consequences

Give an assistant the probe endpoints, platform configuration, initialization steps, and dependency list. Ask what action follows each failure and whether that action can improve the condition. This exposes the common mistake of checking external dependencies in liveness without considering restart behavior.

Have it propose isolated verification cases and identify expensive or sensitive diagnostic work. Check suggested platform fields against current official documentation and your deployed version. Do not let it claim availability from a passing endpoint test. The useful output is a small, explicit health contract with known limits, suitable thresholds, and evidence that the actual platform reacts as intended.

Sources

Kubernetes liveness, readiness, and startup probe documentation is the primary reference for those probe semantics.

Kubernetes Deployment documentation provides rollout context. The API dependency classification and verification sequence are illustrative.