The cost of an assistant does not come only from the tokens of the last call. A RAG generates embeddings, searches, reranks, sometimes calls multiple models, and uses tools. An agent can repeat, wait, and request human validation. Infrastructure, logs, and support complete the cost.
Optimizing requires measuring a complete task and its success. A cheap request that produces a poor response and manual follow-up is a false economy.
Build an economic unit
Choose a unit related to the job: qualified file, processed document, useful response, prepared order, or resolved incident. For each unit, count:
- calls;
- tokens ;
- embeddings ;
- reranking ;
- tools;
- storage;
- retries;
- validation;
- errors;
- infrastructure;
- support.
This unit allows for comparing an architecture, a model, and the current process.
Calculate the cost per successful task
A simplified formula is:
(coûts variables + coûts fixes affectés + validation + reprises) / tâches réussies
The success rate must comply with a definition. A displayed response is not a successful task if the user has to redo everything.
The scenarios incorporate average volume, peak, growth, and seasonality. The rates are dated and verified, as they change.
Breakdown of the cost of an AI task between context, research, model, tools, retries, validation, and deployment.
Avoid calling the model
The most effective lever is not to call an LLM when:
- one rule is enough;
- a response exists in cache;
- the request is invalid;
- the data is missing;
- the user does not have the right;
- a classical research responds;
- an operation can be grouped.
The initial classification can be deterministic or use a small model. Frequent paths are prioritized for optimization.
Router according to the difficulty
Not all tasks require the most capable model. A router can choose:
- rule;
- lightweight model;
- specialized model;
- more powerful general model;
- humane treatment.
Routing uses intention, length, risk, language, and trust. It is evaluated: a routing error can cost more than the savings.
For sensitive tasks, policy and data may require an environment regardless of cost.
Reduce the context
Prompts sometimes accumulate history, documents, and instructions. A longer context increases cost and latency, and may reduce accuracy.
The levers are:
- more precise recovery;
- deduplication;
- filters;
- compression with provenance;
- structured memory;
- summary of ancient exchanges;
- different context depending on the stage;
- removal of unnecessary examples.
The gain must be validated by the evaluations.
Optimize the RAG
A poor search leads to sending too many passages. Improving metadata, hybrid, and reranking can reduce the final context.
Embeddings are calculated only for the changed content. Batches, dimensions, and models are chosen according to the corpus. A change in embedding requires a comparison before full reindexing.
Reranking is reserved for queries where it provides a benefit. A small number of candidates reduces the cost.
Cache
Several levels exist:
- exact answer;
- search result;
- query embeddings;
- prompt prefix;
- tool output;
- agentic sub-task.
The key must include version, rights, language, and freshness. A cache shared between users must not expose private data. Invalidation follows the sources and policies.
Suppliers can offer context caching mechanisms; their conditions and savings are checked.
Batch and asynchronous
Non-interactive tasks can be grouped and processed with less expensive capacity or at a suitable schedule. Any batch APIs have specific delays and rates.
The architecture separates real-time needs from background processing. A user receives a status and a notification rather than an unnecessarily maintained connection.
Limit retries and loops
A timeout does not always mean failure. Idempotence and verification prevent repeating an operation. Retries use a maximum number, a backoff, and a distinction between temporary and permanent errors.
The agents have a budget of steps, duration, and cost. They stop when no new evidence appears.
Structured outings
A clear format reduces correction calls. The schema is validated and deterministic errors are sent back to the model with a limited number of attempts.
For extraction, a specialized model or a post-processing rule may be more cost-effective than a general conversation.
Open models and private capacity
A private infrastructure can reduce the variable cost at a stable volume, but adds investment and operation. It is necessary to calculate the utilization rate, peak capacity, energy, support, and renewal.
A hybrid architecture routes predictable volume to reserved capacity and peaks or complex tasks to an API. Quality and sovereignty remain constraints.
Negotiate and book
At high volume, suppliers can offer suitable capacities, commitments, or rates. A reservation is only profitable if usage is stable and dependence is accepted.
The contract must address versions, availability, region, reversibility, and price evolution. Negotiation does not replace technical optimization.
Reduce validation costs
The interface shows sources, differences, and uncertain areas to speed up the review. Deterministic controls filter errors before the human. Sampling is adapted to risk and demonstrated quality.
Human time is tracked in the cost per task.
FinOps for AI
A FinOps practice combines visibility, ownership, and optimization. Each team knows:
- usage;
- budget;
- unit cost;
- model;
- quality;
- anomalies;
- forecast.
Tags and identifiers assign calls to the use case. Alerts detect spikes, loops, context changes, and success rate drift.
Budgets and safeguards
Limits exist per user, team, task, and period. When a budget is approaching, the system can:
- use a lighter model;
- reduce the steps;
- switch to asynchronous;
- request a validation;
- refuse a non-priority task.
Risky decisions should not be quietly downgraded to save money.
Measure quality and cost together
A table compares the versions on:
- success;
- critical errors;
- latency;
- cost;
- validation;
- satisfaction.
The Pareto frontier shows the options that improve one dimension without excessively degrading the others. The choice is made by use case.
A continuous approach
Start by instrumenting, identifying the three major positions, testing a lever, and checking the quality. Sustainable savings often come from the process: better routing, less context, fewer reworks, and fewer unnecessary calls.
Partitech can instrument costs, build the router, optimize RAG and agents, and compare hosting scenarios. The goal is a controlled cost per task with proven quality and security.
Let's talk about your project
Audit and optimize the cost of your AI platform with Partitech. Contact Partitech.