Discuss your project
Architecture IA

How can you tell whether a coding agent has improved?

A coding agent fixes a test on its first attempt and edits the wrong file on its second. Use the Copilot Evals preview to build a repeatable comparison of two configurations on your repository.

Comment vérifier qu’un agent de code s’améliore vraiment ?

Your agent fixes a date test. On the next run, it disables the assertion instead of correcting the calculation. A successful demo gives you one observation. A repeatable evaluation shows which behaviors recur.

A preview for testing your own setup

GitHub opened Copilot Evals in public preview on November 17. GitHub lists Individual, Business, and Enterprise plans. It uses Vally, an extensible local evaluation engine. It helps build tests for skills, custom agents, and MCP tools, which connect the agent to other services. GitHub lists a VS Code extension, a CLI plugin, and a Copilot app plugin. The engine runs locally. Configured models and tools can still exchange data elsewhere.

Compare two versions of a skill

  1. Set up a disposable repository with a failing date test. Record the expected result, editable files, and allowed commands. Add a check that fails if the test is deleted.
  2. Pin the repository revision, model, and permissions. Run five trials per skill version, starting from a clean copy each time. Record the test result, modified paths, duration, and any manual work.
  3. Add an ambiguous specification, where asking for clarification may be correct. Inspect failures individually. With five runs, the success rate is a pilot signal.

Promote the update when it improves the intended tasks while meeting every requirement you marked as blocking. Keep the failures as fixtures for the next version.

Share this article