Design a background job for work that should not block a request

A request that generates a report, converts a file, and emails the result can exceed its time budget even when every dependency is functioning. Moving the work into an untracked asynchronous callback may make the response quick, but a process restart can lose the task. The real requirement is not simply to avoid waiting. It is to define when work is durably accepted, how it progresses, and what happens when execution is repeated or fails.

Use an illustrative report export: an authenticated user requests a report, the application stores the job, a worker produces an artifact, and the user downloads it later. The report may take longer than an interactive request. The following design is a reviewable starting point, not a promise that a particular queue or worker guarantees completion.

Diagnose what belongs outside the request

Measure or estimate the work budget separately from the request budget. Identify CPU work, database reads, provider latency, and large file operations. A slow database query may need fixing before it is placed in a worker. Moving an inefficient operation does not reduce its resource demand; it changes who waits and how capacity is controlled.

Determine which result the user needs immediately. Validation, permission checks, and acceptance can remain synchronous. Artifact generation can be deferred if the UI supports a pending state. If the user must receive the complete result in the same response, a background job changes the product contract and needs deliberate agreement. Name the expected completion window and failure behavior instead of hiding an indefinite wait behind a spinner.

Persist acceptance before responding

Create a durable job record or publish through a mechanism that makes acceptance reliable. The response should contain a job identifier and its current state. A server returning accepted before the job exists durably creates a misleading promise. If job creation fails, return an actionable failure instead of pretending work is underway.

Bind the job to the requesting account and organization. Store the minimal inputs needed to execute it, with an explicit policy for whether the report reflects request-time data or execution-time data. Avoid persisting credentials or large private payloads unnecessarily. When the database update and queue publication are separate writes, consider an outbox. A background worker cannot recover work that never reached durable storage.

Define a small state machine

Use states with clear transitions: queued, running, succeeded, failed, and cancelled may be enough. Record creation time, attempt count, last relevant error category, and artifact reference. If workers lease jobs, record lease ownership and expiry. Explain how expired running work becomes eligible for recovery after a worker disappears.

A practical artifact is a transition table naming who can move each state and under what condition. The worker may claim queued work, but a user may request cancellation only for their own job. Success requires an accessible completed artifact, not merely a function returning without error. Keep raw provider errors in controlled diagnostics and expose a stable user-facing explanation. This table helps prevent contradictory behavior when retries and cancellation arrive together.

Design for repeated execution

Assume a worker can crash after producing the artifact but before recording success. A retry may then execute the same job again. Use the job identifier in output paths or a conditional result write so repeated execution cannot overwrite another job's artifact. Decide whether replacing the same job's incomplete artifact is safe.

External effects require special care. An email notification can sometimes tolerate duplication; charging a user usually cannot. Use provider idempotency support where available and verify its contract rather than assuming the queue prevents duplicates. Keep business operation identity separate from worker attempt identity. Each attempt should be observable while still belonging to one requested report. Duplicate protection belongs around consequential effects, not merely around the initial HTTP request.

Bound retries and concurrency

Retry transient failures with a finite budget and backoff suited to the dependency. Invalid report inputs should fail permanently, while a temporary provider outage may justify another attempt. Classify errors carefully: retrying a programming defect indefinitely wastes capacity and makes diagnosis harder. Record the next eligible attempt time where the scheduler needs it.

Set a concurrency limit based on the expensive resource. Ten workers producing memory-heavy reports can overload a database or exhaust memory even if the queue is healthy. Include per-tenant limits if one organization can occupy all capacity. Define what operators do with exhausted jobs: inspect, correct, replay with authorization, or cancel. A dead-letter destination is useful only when someone owns review and recovery of its contents.

Make progress and cancellation honest

Show the last known durable state and an appropriate refresh mechanism. Avoid fabricated percentages when the work has no measurable stages. Queued and generating are often more honest than a progress bar that stalls at ninety percent. Tell the user when a report is ready and when they can safely retry.

Cancellation is a request, not necessarily immediate termination. A worker may need to check cancellation between bounded stages. Decide whether already completed artifacts are retained, removed, or made unavailable. If a provider call cannot be interrupted, cancellation must not claim that it never occurred. Document the race between completion and cancellation. The UI and worker should use the same transition rules so a report does not appear cancelled while remaining downloadable through an unprotected link.

Verify recovery rather than only success

Use synthetic data and a controlled worker environment. Stop execution after claiming a job, after writing an artifact, and before recording completion. Restart the worker and inspect the final state and artifact ownership. Inject a transient dependency failure and a permanent input failure to verify that retries differ appropriately.

Test access to job status and downloads from another account. Confirm that expired artifacts produce a clear result and that storage cleanup does not leave success pointing to a missing file. Record which recovery cases were exercised; do not describe a queue's advertised behavior as your application's tested behavior. For release readiness, link these cases to the deployment evidence checklist.

Use AI to review the state transitions

Give an assistant the transition table, worker claim logic, and effect boundaries. Ask for concrete interruption points and the resulting durable state. This is more useful than asking for a generic background job framework. Require the assistant to distinguish queue guarantees from application guarantees and to cite installed library documentation for proposed APIs.

Have it suggest bounded failure-injection cases and inspect whether logs expose secrets or private report inputs. Review those suggestions before running them. Keep the implementation small until actual scheduling and capacity requirements justify more machinery. AI can help enumerate races and recovery states, but only the chosen storage and provider contracts, verified in your environment, can establish what happens after a real interruption.

Sources

AWS's transactional outbox guidance explains a durable approach to coordinating database changes and outgoing messages.

OWASP authorization guidance supports checking access to jobs and artifacts. The job state machine above is illustrative and must be adapted to the application's requirements.