Two models are arriving in Copilot, but their availability does not answer the essential question: which one genuinely helps your team deliver a correct change? Here is how to prepare a reproducible comparison on your own tasks, without manufacturing a benchmark.
What the announcements change — and what they do not prove
Two dates, access to establish
GitHub announced Gemini 3.8 Flash in Copilot on 3 September 2026, then the general availability of GPT-6 Astra on 4 September 2026. In both cases, GitHub describes a phased rollout, eligible plans and service-specific administration settings. These are announcement facts, reviewed on 9 September 2026; they do not establish that a model is already enabled for every account.
| Model | GitHub announcement | Access in your environment |
|---|---|---|
| Gemini 3.8 Flash | 3 September 2026 | Not verified |
| GPT-6 Astra | 4 September 2026 | Not verified |
The first action is therefore straightforward: record the plan, applicable policies, client in use and the model that can actually be selected. A dated record avoids conflating an announcement, an administrative activation and an observable development experience.
Availability is not a benchmark win
GitHub presents its own assessments in these announcements. They show how the publisher positions the service; they do not replace an independent comparison on a given repository, review rules and continuous-integration pipeline. Partitech's analysis is therefore this: the choice should concern a change accepted by the team, not the smoothness of a conversation or an isolated demonstration.
Building a task set that resembles real work
Short changes with acceptance criteria
Prepare tasks small enough to review, yet realistic enough to expose project constraints. For example: fixing PHP validation that accepts an invalid value; adding a documented field to an API response without breaking existing clients; repairing a JavaScript test that has become unstable. These examples are fictional: they describe a protocol, not an observed result.
Every task must start from a known commit, provide the strictly necessary context, identify targeted tests and define the expected result. “The test passes,” “no out-of-scope file is modified” and “the response remains compatible” are verifiable criteria. A vague request such as “improve this module” does not produce a usable unit of comparison.
Making conditions comparable
Freeze the commit, dependencies, tool permissions, supplied context and generation budget. Separate tasks already known from those reserved for evaluation, so the team does not inadvertently adapt the protocol to a previous response. Plan several runs where relevant, without turning that intention into a result before trials have actually been run.
Measuring the accepted result, not only the generated response
An empty grid is more honest than a table filled with decorative numbers. It records what was observed after execution, using the conventions adopted by the team.
| Task | Targeted tests | Review defects | Rework | Human time | Observed cost | Result |
|---|---|---|---|---|---|---|
| To be completed | Not measured | Not measured | Not measured | Not measured | Not measured | Not measured |
Cost per validated change can be tracked as follows: (generation + review + rework) / accepted changes. Define what “review” and “rework” cover: engineering time, billed executions, CI waiting time, or only active time. If no change is accepted, unit cost remains uncomputable: keep expenditure and failures separate. The formula helps compare documented trials; it predicts no return.
Interpreting differences without overreaching
Classify failures before attributing them: functional error, regression, out-of-scope change, incomplete context or unavailable tooling. An output rejected because a test was not run does not mean the same as a fix that breaks a business rule. If trials are published, state the set size, exclusions and limits of representativeness.
The decision can then be conditional. Retain a model for a task family if changes are regularly accepted and review remains sustainable. Test further if results depend heavily on context or vary from one run to another. Limit its use to a precise scope if rework or irrelevant modifications become too frequent. This matrix is an editorial recommendation, not a ranking of GPT-6 Astra or Gemini 3.8 Flash.
Moving from a trial to version tracking
Retain version identifiers, prompts, outputs, test commands and review decisions, while excluding secrets and customer data. Re-run a representative subset after a substantial change to the service, a model, your dependencies or your policies. This record makes a decision revisable instead of turning it into a lasting preference.
The protocol will not answer every question: on its own, it does not measure long-term architectural quality or the business value of a change. It does, however, make it possible to decide what to authorize, observe and suspend on the basis of local evidence. To place this discipline in the broader context of agents, also see our analysis of critical cyber agents.
Choosing a model in Copilot thus means buying local evidence, not a general promise. Verify access, freeze conditions, measure accepted changes and retain the material that will make it possible to revisit the decision.