Evaluating self-improving software after release

September 24, 2026 - 5 minute read

Evaluating self-improving software requires a record of what changed and a stable definition of success. A workflow can close more issues because it received easier work. It can finish sooner because checks were skipped. It can appear less expensive because the dashboard excludes the engineer who repaired its output.

Treat each proposed improvement as a change to a running engineering process. Record the previous behavior, the candidate version, the work eligible for comparison, and the evidence needed before the change becomes normal practice. This is a release decision about one workflow over time, rather than a ranking of coding agents on unrelated demonstrations.

Define success when evaluating self-improving software

Choose the behavior that motivated the change before inspecting the new results. For a maintenance workflow, success might mean a compatible dependency update accepted with all required tests intact. For a review workflow, it might mean identifying a confirmed defect without sending reviewers a growing queue of incorrect findings.

Keep the output check separate from the agent's explanation. Anthropic's guide to agent evaluations distinguishes the transcript of a run from its outcome and discusses choosing graders for the part being assessed. A convincing explanation is useful diagnostic material, but it cannot establish that a patch works in the intended environment.

Write down what would make the candidate unacceptable. A reduction in completion time should not compensate for a regression in a critical flow unless the organization has explicitly decided that trade-off. Preserve the original required checks so the candidate cannot improve its recorded result by changing the test it is supposed to satisfy.

Factory's Automated QA documentation describes evaluating applications through web, CLI, and API surfaces and producing evidence. Use those checks alongside repository tests and review findings. If an environment prevents a required check from running, keep the outcome blocked until the missing evidence is available. Excluding it from the report would make the candidate appear more reliable than the evaluation supports.

Compare the same work under known conditions

Record the workflow revision, model selection, tools, instructions, repository revision, and environment used for each run. If several of those change together, the resulting comparison measures the combined intervention. It cannot isolate the effect of a rewritten skill or a different tool response on its own.

Build the comparison set from the work the workflow is expected to handle. Retain ordinary tasks as well as the incidents that motivated the change. Reserve cases that were not used to develop the candidate. Anthropic's tool-engineering account describes using held-out tests to check whether an improvement generalized beyond its development examples.

For an illustrative maintenance pilot, compare dependency updates with similar compatibility requirements, test coverage, and repository conditions. A patch-only update and a major API migration should not silently become interchangeable successes. Preserve the reason for excluding a task, and keep failures caused by missing dependencies separate from failures in the proposed code.

With Factory Automations, review the configured run target, working directory, model, and instructions when recording the pilot setup. The documentation also describes run history for inspecting results. Store the evaluation's task definitions and expected outcomes in the team's versioned evaluation materials. Do not assume an automation's execution history is itself a controlled experiment or a complete regression suite.

Count the work that remains with engineers

Track human effort at the points where the workflow needs intervention. Preparation, clarification, review, repair, and release follow-up can all consume engineering time even when the agent completes its own run quickly. A change that moves work from implementation into review may still be useful, but the report should describe that trade-off.

Use accepted outcomes as the denominator for the cost question. Retain the cost of failed attempts and repeated runs that preceded an accepted patch. Report waiting time separately where possible, because an environment queue and an agent's incorrect implementation call for different remedies. Keep source records that let a reviewer reconstruct the total rather than relying on a single average.

Factory's Agent Effectiveness overview documents Throughput, Output, and Attribution views connecting agent activity to delivery work. The product is in Private Preview. Its setup requirements include organization-level integrations and the Advanced Analytics enterprise control, with Owner or Manager access needed for setup.

These views can help locate the work associated with a session and inspect its delivery context. Supplement them with whatever reviewer-effort records your pilot requires. Do not claim a dashboard measures every minute of correction or proves the causal effect of an instruction change. State which work is represented, which systems are connected, and what still requires manual review.

Evaluate self-improving software after deployment

A candidate that passes offline checks can still produce a different result on later work. Watch for recurrence of the original failure, changes in the mix of incoming tasks, and regressions in behavior the candidate was expected to preserve. Keep the prior version available while the team gathers that evidence.

Factory's January 2026 Signals research describes an internal system that correlates session friction with releases and turns recurring patterns into proposed fixes. Its human approval step and dated deployment observations make the scope important. The report is evidence of Factory's internal research process, not a guarantee that every customer automatically receives identical session analysis.

Arize's discussion of persistent agent improvements treats evaluation and promotion as decisions between a candidate change and the version future runs inherit. Apply that discipline to repository guidance too. A new instruction can pass its motivating example yet become harmful when the next release changes the underlying tool or directory structure.

Factory's Effectiveness Dashboard documentation cautions that cycle-time trends need interpretation in the context of release cadence, holidays, and headcount changes. It also limits attribution coverage to connected systems. Use those qualifications in an improvement report. A change in the observed trend is a reason to investigate the associated work, not permission to attribute every movement to the newest automation.

Make the next release decision explicit

Keep a release record that connects the candidate to its evidence and owner. It should identify what changed, which tasks were evaluated, which checks were blocked, what human correction remained, and which later signals could justify reverting it. The record should be useful to someone who did not participate in the original investigation.

When evidence is mixed, preserve the uncertainty. A candidate may succeed on the original incident while failing a neighboring task. It may reduce manual correction but increase operating cost. It may look promising on too little work to justify a broader rollout. Each result can support a different decision without being flattened into an unsupported claim that the software improved.

Decide whether to retain the current version, revise the candidate, or accept a bounded rollout under existing approval rules. An automated reviewer can contribute findings, but the release owner still needs the evidence required by the repository and the organization. Changes to required tests, access controls, or the evaluation itself deserve their own review.

The resulting record gives the next improvement attempt a trustworthy starting point. It preserves successful outcomes, failed hypotheses, and the conditions under which each conclusion held. For a Factory workflow, that means connecting its versioned instructions and execution evidence to reviewed changes and observed delivery work, while keeping the limits of the measurement visible.