Diagnose a production incident with structured logs and request IDs

A user reports that saving a change failed, while server logs contain thousands of disconnected messages. Searching for error finds provider timeouts, invalid requests, and unrelated background jobs. The missing ingredient is correlation: which events belong to the user's operation, what durable state changed, and where the journey stopped.

Structured logs and request identifiers help answer those questions when their fields and boundaries are designed deliberately. This article uses an illustrative document-save operation that writes a record and schedules indexing. It does not describe an actual incident. The aim is to make diagnosis possible without logging full document contents, credentials, or every internal object.

Define a small event contract

Use consistent fields for timestamp, severity, event name, request identifier, route or operation, outcome, duration, and a safe error category. Add deployment identity where it helps separate releases. Choose stable event names such as document.save.accepted or indexing.schedule.failed rather than free-form prose that changes with every refactor.

Keep fields relevant to diagnosis. A record identifier may be sensitive in some contexts, so decide how it is represented and who can read it. Avoid using arbitrary user input as field names or event names. A practical artifact is a logging schema with allowed fields, redaction rules, and examples containing synthetic values. This makes review easier and gives assistants a predictable structure to analyze without asking for broad data dumps.

Establish correlation at entry

Create or validate a request identifier at the trusted boundary. If an incoming identifier is accepted, bound its format and length; it is untrusted text. Propagate the resulting identifier through the operation's relevant logs and downstream calls where supported. Return a safe reference the user can provide to support when useful.

Do not confuse request identity with business operation identity. A retry creates another transport attempt but may belong to the same save operation. A background job starts later and should carry a link to the initiating operation while keeping its own attempt identity. This distinction helps explain duplicate requests and delayed work. Without it, a log search may show one request as successful while hiding the later indexing failure the user actually experiences.

Log meaningful state boundaries

Record durable transitions and consequential outcomes, not every line of execution. For document save, distinguish validation rejected, database committed, indexing accepted, and indexing failed. If the response says saved while indexing remains pending, the events should make that partial state visible.

Include durations around expensive boundaries where they support diagnosis. Do not log complete request or response bodies just to obtain convenient context. Store a safe count, size, category, or controlled identifier instead. Sensitive content can appear in exception messages, so sanitize provider and database diagnostics too. A concise state trail is usually more useful than a large volume of detailed payloads that create privacy risk and obscure the causal sequence.

Build a focused incident query

Start with the reported time window, request reference, route, and deployment. Gather the matching events, then expand to connected operation and job identifiers. Compare the successful and failed paths for the same kind of request. Identify the last confirmed durable transition and the first missing or failed transition.

Write findings as facts and hypotheses. The database commit event is evidence of the application's reported commit outcome; a missing log may reflect instrumentation gaps rather than absence of work. A provider timeout may leave delivery uncertain. Record those limits. A useful incident artifact contains the user symptom, timeline, evidence links, affected scope, and unanswered questions. It should not turn a plausible cause into certainty before the relevant state is checked.

Check durable state safely

Use bounded read-only queries to confirm the document's state and indexing job status. Scope the query to the affected operation and authorized tenant. Return only necessary fields, avoiding document content. Check whether the user saw a network failure after the database commit, because retry behavior depends on that distinction.

If indexing failed after saving, the correct recovery may be to retry indexing rather than repeat the save. Identify the business invariant before proposing a repair. Use the existing operational authorization process for writes. Correlated logs help choose the smallest repair, but they do not authorize broad replay. See safe AI incident triage for separating diagnosis from consequential actions.

Verify the instrumentation in isolation

Exercise a successful save, an invalid request, a database failure, and an indexing failure with synthetic documents. Inspect whether every event carries the intended correlation fields and whether the durable state matches the logged outcome. Confirm that asynchronous work can be traced back to its initiating operation.

Search the emitted logs for sensitive payloads and credentials. Check log volume and performance impact under a bounded representative workload. A logger that blocks requests or serializes huge objects can create another failure mode. Verify behavior when the logging destination is unavailable according to the application's requirements. Logging should support the application rather than accidentally becoming a mandatory external dependency for every user operation.

Define access and retention

Limit access to logs according to the data they contain. Redaction does not eliminate the need for permissions and retention. Define how long operational correlation remains useful and whether particular identifiers need shorter handling. Keep audit requirements distinct from diagnostic convenience; an audit record may need stronger integrity and completeness than ordinary application logs provide.

Avoid treating logs as the only durable business record. They can be sampled, delayed, or lost. Store essential operation state in the application where recovery depends on it. Keep the event schema versioned or compatible enough that queries survive routine refactors. Review new fields and logging statements during code review, especially when AI-generated debugging code adds broad object serialization.

Use AI on a bounded evidence bundle

Give an assistant redacted events for one incident window, the event schema, and the relevant state transitions. Ask it to construct a timeline, mark missing evidence, and propose read-only checks that distinguish hypotheses. Require it to cite event identifiers or timestamps for claims rather than presenting an unsupported narrative.

Do not ask the model to infer private content from identifiers or to propose immediate broad repairs. Review its suggested queries for tenant scope and limits. AI can summarize a large but bounded trail and notice inconsistent outcomes, yet it should explicitly preserve uncertainty around timeouts and absent logs. The final diagnosis should connect observed failure to a reproducible or verified cause before changes are made.

Sources

OpenTelemetry logs documentation provides the primary reference for log correlation concepts.

OWASP secrets management guidance supports careful handling of sensitive diagnostics. The document-save event contract and investigation steps are illustrative.