In 2026, model announcements are happening rapidly. OpenAI introduced GPT-5.6 in July as a model intended notably for professional and long-term agentic workflows. Mistral presented Medium 3.5 in May in public preview, with open weights, long context, and capabilities combining instruction, reasoning, vision, and code.
These examples illustrate the broadening of choice: frontier APIs, open weights, compact models, code models, OCR, voice or reasoning. The reflex of looking for "the best model" becomes less relevant. The company must look for the best system for a given task, risk, and cost.
Models, versions, prices, licenses, and availability can change within a few weeks. Any comparison must record the exact identifier and the date.
Why overall rankings are not enough
A public benchmark measures a set of tasks under defined conditions. It can be useful for pre-selection, but it does not answer all questions:
- quality in French profession;
- respect of an internal pattern;
- use of your tools;
- quotations;
- latency in your region;
- cost with your contexts;
- adapted refusals;
- behavior on your mistakes;
- security;
- version stability.
The results can also be sensitive to the prompt, to the temperature, to the reasoning budget, to the tools, and to the evaluation mode.
A gap of a few points on a leaderboard does not justify a migration.
Define success before choosing the candidates
Write a task sheet:
- entrance;
- exit;
- users;
- frequency;
- impact;
- difficult cases;
- acceptable errors;
- forbidden errors;
- deadline;
- cost;
- data;
- human control.
For an invoice extraction, success is measured by field and by document, not by overall appreciation. For an agent, it includes completion, tools, incorrect actions, and recovery. For a document assistant, it includes retrieval, fidelity, and citations.
Conceptual chart showing a Pareto frontier between quality, cost, and latency for several candidate models.
Build a representative test set
The dataset contains:
- frequent cases;
- high-value case;
- long cases;
- noisy inputs;
- ambiguities;
- languages;
- refusal;
- attacks;
- edge cases;
- historical incidents.
Separate development and final testing. Teams can optimize on the former without seeing the latter.
Each case has criteria. A response can be rated automatically for a JSON and by an expert for a summary. Model judges can help, but they are calibrated against humans and do not judge critical cases alone.
Compare specific versions
Write in each result:
- supplier;
- model;
- version or snapshot;
- date ;
- endpoint or runtime;
- settings;
- prompt ;
- tools;
- region;
- context;
- repetition.
A trade name can point to an updated version. For a reproducible decision, use a snapshot when one exists or maintain a change detection campaign.
During a new version, rerun a subset of non-regression tests before canary.
Evaluate GPT-5.6 without assuming it is suitable everywhere
GPT-5.6 is presented by OpenAI as a frontier model capable of ambitious professional and agentic workflows, with risk-adjusted control layers. It can be a candidate for complex tasks, research, tools, and long horizons.
But a simple classification task does not need the most capable model. Measure:
- real gain compared to a smaller model;
- reasoning budget;
- latency;
- cost;
- stability of the tools;
- quality of structured outputs;
- review rate.
The frontier model can become a fallback for difficult cases rather than a single route.
Evaluate Mistral Medium 3.5 with its status and license
Mistral describes Medium 3.5 as a dense model with 128 billion parameters, 256k context, open weights under a modified MIT license, and possible self-hosting on a multi-GPU setup. It notably powers agentic functions presented in 2026.
The benchmark must check:
- quality in French and in the field;
- tools;
- vision;
- formats;
- long context;
- flow;
- self-hosted capacity;
- support;
- preview status;
- license obligations.
Self-hosting offers control but adds capacity, high availability, security, and updates.
Include the specialized models
An OCR model, code, embeddings, reranking, transcription, or classification can outperform a generalist on its task and cost less. The architecture selects step by step:
- OCR for the document;
- embedding for the index;
- reranker for search;
- compact model for router;
- capable model for synthesizing;
- deterministic validator to control.
Compare a specialized pipeline to a single multimodal call. Simplicity can win, but it must be measured.
Measure quality by category
An overall average masks significant failures. Present:
- score by type;
- distribution;
- worst decile;
- blocking rate;
- critical cases;
- trust;
- expert comments.
A model may be excellent at writing and insufficient at calculation. Routing tasks allows exploiting its strengths.
Add robustness and security
Tester:
- prompt injection ;
- contradictory instructions;
- sensitive data;
- unauthorized tools;
- false sources;
- malformed outputs;
- toxic content;
- requests outside the scope;
- exfiltration;
- overconfidence.
Set blocking criteria. A cheaper model does not make up for an incorrect critical action.
Security also depends on the harness, the sandbox, and the permissions.
Measure latency correctly
Follow:
- time at the first token;
- total time;
- p50 ;
- p95 ;
- p99 ;
- flow;
- error rate;
- throttling
- performance in competition.
Test from the production region with the real context sizes. An average in the lab does not reflect the peaks.
For an interactive experience, the first response can matter more than the total time. For a night batch, the throughput dominates.
Calculate the cost per accepted task
The price per token is just one component. Add:
- context;
- tools;
- retries;
- research;
- stash;
- human review;
- infrastructure;
- observability;
- errors;
- support;
- migration.
Useful formula:
coût par tâche acceptée = coût total du scénario / nombre de résultats acceptés
A more expensive model can be economical if it significantly reduces review. A cheap model can remain optimal for standard cases.
Integrate data, contract and sovereignty
The scorecard covers:
- use of data;
- retention;
- region;
- subcontractors;
- rights;
- license;
- audit;
- availability;
- support;
- reversibility;
- end of service.
A technically superior model can be excluded if the framework does not allow the relevant data.
Build a Pareto frontier
There is not always a winner. Several models can be non-dominated:
- high quality, high cost;
- low latency, sufficient quality;
- strong control, heavier exploitation;
- Excellent specialization, narrow scope.
The Pareto frontier shows the trade-offs. The weights change according to the task. A sensitivity analysis verifies that the decision does not depend on an arbitrary weight.
Router instead of imposing a single model
A multi-model architecture can:
- detect the task;
- apply the data policy;
- choose a fast model;
- to escalate complex cases;
- use a specialized model;
- fallback in case of unavailability;
- compare in shadow mode.
The router must be simple, observable, and evaluated. A wrong rule can cost more than the gain.
Avoid multiplying suppliers without operational capacity. Two or three controlled profiles are often enough.
Deploy through canary and shadow traffic
Before migration:
- offline replay on authorized data;
- shadow traffic without user impact;
- canary on a fraction;
- comparison;
- alerts;
- rollback.
Sensitive content is protected and retention policies are followed. The new model does not automatically receive all production data.
Set an expiration date for the decision
A model selection is valid for one version and one period. Review triggers:
- new model;
- price change;
- drift;
- incident;
- new data;
- volume change;
- end of preview;
- contractual evolution;
- regulatory requirement.
Keeping the automated benchmark allows reevaluation without starting from scratch.
Document the decision
The report includes:
- objective;
- candidates;
- versions ;
- dataset ;
- metrics;
- results;
- case of failure;
- costs;
- constraints;
- recommendation;
- fallback ;
- limits;
- review date.
It indicates what has been measured and what remains a hypothesis.
The right model is an architectural choice
In 2026, the diversity of models allows technology to be better adapted to needs. It also requires discipline: test games, versions, costs, policies, routing, and operations.
Partitech can build the benchmark, integrate GPT, Mistral or open models, set up the gateway, routing, canary, and continuous evaluations. The selection then becomes a reproducible and reversible decision, not a bet on the latest published ranking.
Let's talk about your project
Organize a benchmark of reproducible models and a multi-model architecture with Partitech. Contact Partitech.