How to evaluate coding agents on your codebase

September 24, 2026 - 4 minute read

A coding agent evaluation should end with changes your team can verify. A completed session, a convincing explanation, or a large diff can coexist with incorrect behavior. Define the acceptance evidence before asking an agent to work.

Factory's Legacy-Bench research gives a concrete reason to do this. Its tasks combine instructions, reproducible environments, and verification tests that the agent cannot see. The published results describe agents producing plausible output and declaring success despite failing the independent checks. Those findings belong to that benchmark and its tested configurations, not to every repository or model.

For a team choosing a coding agent, the next step is a smaller evaluation grounded in its own work. Use real task shapes, explicit access boundaries, and reviewers who can judge the resulting changes.

Choose tasks that represent the work

Start with the decisions the pilot needs to inform. A team evaluating review assistance needs a different task set from a team delegating maintenance changes. Choose work from the backlog that reflects the language, dependency structure, and review requirements the agent would encounter after adoption.

Include routine work and at least one task where a plausible answer can be wrong. A parser may accept ordinary input but mishandle a boundary case. A migration may compile while changing serialization behavior. A bug fix may remove a symptom without preserving an authorization check.

Write down why each task belongs in the evaluation. Do not select only tasks that already produced an impressive demo. Keep setup failures and difficult examples in the record, because excluding them can hide costs the team will face later.

Describe the desired outcome without leaking the solution. Give the agent the context a real engineer would receive, including reproduction steps and repository instructions. Keep reviewer-only checks separate when they are intended to test whether the solution generalizes beyond the obvious example.

Factory's Agent Readiness documentation can help identify repository conditions that affect execution. Record those conditions before the pilot. Otherwise, a missing test service or broken bootstrap script can be mistaken for a difference in coding ability.

Make the environment and permissions reproducible

Use the same starting commit, dependency versions, task description, and acceptance criteria for each candidate. Give each run an independent worktree or disposable environment. One agent should not receive another agent's patch, generated artifacts, or task-specific notes.

Record the product version, model selection, reasoning settings where exposed, and execution environment. Also record which commands, network destinations, and external tools are allowed. A run with production credentials and unrestricted network access is not equivalent to a run with read-only repository access.

When comparing products, preserve their documented supported workflow and disclose any differences in setup. When comparing models inside one product, keep the surrounding instructions and tools stable where possible. If the product selects models automatically, record that mode rather than attributing its outcome to a model you did not observe.

For scripted Factory runs, Droid Exec provides a non-interactive entry point with output formats and autonomy controls. Choose the minimum permissions required by the task. Do not disable safeguards merely to make an unattended benchmark complete.

Define what happens when a run stalls. Set the stopping rule before execution, and record retries, permission failures, and manual unblocking consistently. Quietly rescuing one candidate while leaving another untouched makes the result difficult to interpret.

Write the acceptance checks before running the task

Separate the agent's working tests from the evidence that determines acceptance. The agent should use tests to develop its solution, while the reviewer retains an independent way to confirm the intended behavior.

For a bug fix, preserve the failing reproduction and require it to pass after the change. Check nearby behavior that the patch could damage. For a feature, cover the requested flow and relevant failure paths. For a migration, test the contracts that consumers depend on, such as output formats, error behavior, and compatibility.

Make the checks sensitive to the actual failure. A test that passes before the fix cannot establish that the fix solved the reported bug. Where the result is user-visible, include a browser, CLI, or API check instead of relying solely on compilation.

Factory's Missions planning documentation treats validation and runnable development environments as part of planning. Apply that discipline to the pilot even when the work is small enough for a normal Droid session.

Reviewers should inspect scope as well as correctness. A patch that passes acceptance tests by removing a security check or changing an unrelated interface should fail review. The accepted unit is the complete change under its constraints, not a green command in isolation.

Record human work and unsuccessful attempts

Keep a per-run record of the starting revision, prompt, environment, relevant settings, changed files, checks, and reviewer decision. Record every intervention that materially helped the agent, including supplying missing context, fixing setup, rewriting code, and explaining failed tests.

Measure elapsed time and human time separately. An agent may run while an engineer does other work, but review and correction still consume attention. Include failed attempts and abandoned patches in the cost record rather than reporting only successful sessions.

For code-review evaluations, Factory's published review benchmark illustrates the use of known findings, repeated runs, and separate precision and recall measures. Its results are tied to its model versions and selected pull requests. Reuse the methodological lesson without presenting the old ranking as the current result for your codebase.

Report accepted changes alongside the failures and remaining uncertainty. A small pilot can expose a recurring workflow problem, but it cannot establish a universal product ranking. If repeated attempts vary sharply, show that variation rather than selecting the best run for the summary.

Use the result to choose the next bounded rollout

Start with the task class where the evidence supports adoption. If maintenance changes pass reliably but larger features need substantial intervention, expand maintenance first. If environment failures dominate, improve setup before repeating the comparison.

Connect the pilot to operating measures after rollout. Factory's Agent Effectiveness announcement describes connecting sessions to cycle time, work intent, and shipped artifacts. Those signals help track what happens after adoption. They do not replace independent review of correctness or prove that the agent alone caused a change in delivery speed.

Keep the task descriptions, acceptance checks, and environment setup so the evaluation can be rerun after a meaningful product or model change. Update the task set when the team's work changes, while retaining enough of the old set to explain differences over time.

The useful outcome is a decision with a boundary: which work to delegate, under which controls, and what a reviewer still needs to verify.