Reduce AI feature costs before adding another cache

Your AI support feature costs more each month, so the first proposal is to cache responses. Yet most questions contain different account context, the prompt repeats long instructions, and failed requests are retried three times. A cache could add freshness and privacy risks while leaving the largest waste untouched.

Cost optimization starts with a request-level model of usage. This article uses an illustrative support-summary service. It discusses quantities and tradeoffs without assuming current provider prices or claiming a measured saving. Exact prices should be checked against the provider contract used by the product.

Build a usage ledger

Record request count, input usage, output usage, selected model, retries, and outcome. Use provider-reported usage where available and label estimates separately. A request identifier should connect application attempts to provider calls.

Separate user operations from provider attempts. One clicked Summarize action may create several calls because of retries, validation repair, or multi-stage processing. Counting only user actions hides the amplification. Include background refreshes and abandoned operations if they still incur provider usage.

Avoid storing full sensitive prompts just to measure cost. Aggregate token counts, template versions, and safe request metadata can reveal many problems. When deeper inspection is necessary, use approved synthetic or sanitized examples and the application's retention rules.

Find the largest drivers before optimizing

Group usage by feature, model, prompt version, and input size. Determine whether volume, context, output length, or repeated attempts dominates. A small number of oversized documents may matter more than thousands of short questions.

Inspect duplicated context. A request may include the same policy text in instructions, retrieved chunks, and conversation history. Removing unnecessary duplication can reduce input without changing the task, but verify that required evidence remains available.

Check repeated automatic requests. A frontend effect can call the model on every render or field edit. Fixing an accidental request loop may be more valuable than selecting a cheaper model. Usage analysis should include application behavior, not only prompt wording.

Reduce context with a quality check

Include the information needed for the task and omit unrelated material. For a support summary, current issue details may matter while years of unrelated account history do not. Retrieval filters and explicit field selection can make this boundary understandable.

Do not remove context solely to reach an arbitrary token target. A missing policy exception can degrade correctness. Compare candidate context strategies on a representative evaluation set, including cases where essential evidence appears near the end of a long document.

Set supported input limits and explain them in the product. Truncating silently can create a fluent summary of only part of the source. If the feature supports partial processing, label what was included and preserve a way to handle the remaining material.

Control output and orchestration

Ask for the output the product actually needs. A short structured summary may not require an extensive explanation. Validate length and format according to the contract, while allowing enough detail to preserve important facts.

Review multi-call workflows. A draft, critique, rewrite, and validation sequence may improve quality, but each stage needs evidence of value. Compare a simpler call with deterministic validation before assuming an agent loop is necessary.

Bound retries and repair attempts. Track whether retries recover failures or merely repeat unsupported requests. Separate transient provider errors from semantic validation failures. A fixed attempt limit protects the budget and makes worst-case behavior easier to understand.

Evaluate model routing deliberately

A smaller or cheaper model may handle bounded extraction or classification adequately. Test it against the same contract and difficult examples used for the current model. Do not choose based on one attractive response.

Routing adds complexity: deciding which request uses which model, maintaining evaluations, and handling fallback. A fallback to a larger model can erase savings if it occurs frequently. Measure route proportions and combined cost per completed user operation.

Keep consequential policy decisions outside the model where possible. A deterministic eligibility rule costs no generation tokens and is easier to verify. The guide on functions versus LLMs explains how to identify those boundaries.

Add caching only with a freshness contract

Decide what makes two requests equivalent. The key may need prompt version, model configuration, document version, user or tenant scope, and relevant input. Omitting permission scope can leak one account's answer to another.

Define how long a response may remain valid and what invalidates it. A summary of an immutable document has different freshness requirements from an answer about a changing account balance. Cache design begins with staleness tolerance, not an impressive hit-rate target.

Distinguish provider prompt caching from application response caching. They have different semantics and availability. Verify the provider's current behavior rather than assuming repeated prompt text automatically creates a discount. Application caches remain responsible for authorization, invalidation, and user-visible correctness.

Use AI to inspect waste and verify outcomes

A coding assistant can map request paths, identify repeated context assembly, and suggest synthetic evaluation cases for a reduced prompt. Ask it to cite call sites and avoid making savings claims without usage evidence.

An illustrative request is: “Trace all provider calls created by one summary operation. Identify retries, repair calls, and repeated context. Propose one bounded reduction and the evaluation cases needed to detect lost information.” This produces an inspectable optimization candidate.

Compare cost per completed operation alongside quality, latency, and failure outcomes. A lower token count is not a win if users retry more often or receive unsupported answers. Record the measured period and configuration; do not extrapolate a tiny synthetic run into production savings. The goal is a simpler, less wasteful feature whose usefulness remains demonstrated.

Include evaluation expenditure in the optimization budget. Testing a cheaper model or a smaller context strategy consumes resources, but it can prevent a misleading production change. Keep experiments bounded, reuse representative cases, and record why each candidate was accepted or rejected. Once a reduction is verified, monitor whether user behavior changes: more manual edits or repeated submissions may reveal quality loss that token metrics alone miss. Cost control should remain connected to completed useful work rather than minimizing provider usage in isolation.

Review cache eviction and failure behavior before deployment. A cache outage should not automatically multiply provider calls without a bounded fallback, especially during peak traffic. A cold cache after release can also create a burst that the steady-state hit rate hides. Test these transitions with synthetic operations and observe concurrency limits. The optimization should remain understandable when the cache is empty or unavailable, since those states are normal operational conditions rather than exceptional proof that the design cannot fail.

Sources

Anthropic: Building effective agents discusses cost and latency when adding workflow complexity.

Anthropic: Demystifying evals for AI agents provides guidance for comparing behavior across changes.