Debug a production-only bug with a small evidence bundle

“It only happens in production” is a description of missing evidence, not a diagnosis. The deployed system differs from a local environment in runtime, configuration, data shape, traffic, network boundaries, and build output. A useful investigation makes those differences explicit without copying production secrets or private records into a debugging prompt.

Imagine an illustrative file-upload feature that works locally but returns an error after deployment. The cause could be authentication, request limits, runtime capabilities, provider configuration, or a file shape the local test never exercised. Starting with a library replacement changes the system before establishing which boundary failed. A small evidence bundle narrows that search while remaining safe to share internally.

Name one failed operation and its impact

Describe the action, expected result, actual result, and who is affected. Include whether the failure occurs for all users, one role, one file type, or one region. A statement such as “authenticated avatar uploads fail before a provider URL is returned” is more actionable than “uploads are broken.” It identifies the journey and the point where evidence becomes incomplete.

Record a time window and a request identifier if available. These let maintainers correlate logs without searching every event in the system. Do not include a user's full session token or raw request body merely to make the report look complete. The best evidence bundle contains the minimum necessary information for the investigation.

Check the response status and user-facing behavior separately. A server may reject a request correctly while the interface reports a confusing generic message. Alternatively, a success response may hide a failed side effect. Both are worth understanding, but they should not be mixed into one presumed root cause.

Capture the deployment identity

Record the deployed commit or release identifier, runtime version, relevant dependency versions, hosting environment, and region when appropriate. Compare them with the environment where the operation succeeds. A local checkout ahead of production is not a reliable baseline for deciding whether the deployed code behaves incorrectly.

Use configuration presence and purpose rather than values in the bundle. “The media provider secret is configured” can be useful; the secret itself is not. Where a value's structure matters, use a safe description such as whether an origin is local or public, or whether a database connection uses the expected deployment environment.

Keep build-time and runtime configuration distinct. A public variable embedded in client code may not change when a server secret changes. A process may also need a restart to observe new settings. Trace how the application reads the relevant setting rather than assuming every environment variable is evaluated on every request.

Follow the failing request across boundaries

Map the path from UI event to frontend request, API handler, validation, authorization, storage or provider call, and response serialization. At each boundary, ask what was observed and what is still inferred. If the provider was never called, provider-side troubleshooting is probably premature. If it returned success, investigate how that result was interpreted afterward.

A concise bundle can use a structure like this:

Journey: signed-in user uploads a small PNG avatar
Expected: image URL returned and profile can save it
Observed: API returns 500 before provider success is logged
Release: recorded commit ID
Runtime: recorded production version
Reproduction: synthetic file, test account, exact steps
Evidence: request ID and redacted error category
Unknown: whether multipart parsing reached validation

The entries are illustrative. Fill them with observed information, and leave unknowns visible. A fabricated “confirmed cause” makes the next engineer repeat the investigation while trying to reconcile contradictory evidence. Honest gaps are useful because they identify the next measurement.

Compare a small set of concrete differences

For an upload issue, compare supported runtime APIs, multipart field names, MIME and size validation, authentication cookies, proxy limits, and provider call behavior. Do not compare every dependency or setting at once. Choose differences that could explain the observed boundary and test the most discriminating one first.

Inspect the production build rather than relying solely on development mode. Server and browser code can run in different environments, and development tools may hide a missing assumption. A constructor available in one Node version, for example, may not exist in another supported runtime. Verify the actual request object and documented runtime contract.

Use synthetic inputs that preserve the relevant shape. If a particular record contains a missing nested field, construct a test record with that omission. You usually do not need the original customer's name, email, or document content to reproduce a null-handling failure. This keeps the debugging artifact reusable and reduces the risk of unnecessary disclosure.

Ask AI for competing hypotheses

An assistant can help compare code paths and suggest discriminating experiments. Give it the redacted bundle and ask for several explanations consistent with the evidence, along with an observation that would reject each one. This is more useful than asking for the “most likely” cause and immediately implementing its answer.

Restrict any connected tooling to the task's required scope. A read-only log query may be appropriate; changing production configuration or running migrations is a different action. Make consequential proposals concrete and reviewable. An AI tool's ability to execute a command does not establish that the command is a sensible diagnostic step.

Require references for runtime or library claims that could vary by version. If the assistant proposes a method absent from installed typings or package code, stop and verify the API. The evidence bundle should shrink uncertainty, not provide a setting where plausible inventions become deployment instructions.

Reproduce before changing the architecture

Create the smallest local or isolated staging case that exhibits the relevant failure. Keep the inputs, runtime, and boundary as close to the observed condition as necessary. If reproduction fails, record the mismatch instead of claiming the suspected cause is disproven. A missing production network condition may explain the difference.

When a cause is confirmed, implement a narrow fix and a regression test that fails under the old behavior. Test successful recovery and the adjacent error paths. For file uploads, that includes invalid fields, unauthorized users, oversized inputs, and provider failure where relevant. Do not turn one runtime compatibility fix into a new media architecture without evidence.

Verify the deployed result and preserve the lesson

After rollout, repeat the original bounded journey with an appropriate test account and synthetic input. Correlate the new request identifier with the intended release. Confirm both server outcome and user-visible behavior. A passing local test supports the code change but does not prove the production path has recovered.

Keep a short resolution note with the cause, relevant difference, fix, regression check and operational limits. Link it to a checklist for recurring incidents when the same assumption may appear elsewhere. The reusable outcome is an evidence trail that makes the next production-only bug easier to identify safely.