What it Takes for Coding Agents to Complete Large Software Tasks

Models have become very good at problems with compact, stable criteria for success. Much of the past year's progress in mathematics and constrained optimization falls into this category - machine-checked proofs of long-open Erdős problems, gold-medal performance at the IMO, new bounds on decades-old combinatorial problems. The search space can be enormous, but the result can ultimately be judged as a whole.

Large software tasks are different. A software specification can semantically cover the desired outcome without specifying what must be run, inspected, and compared before the work can be called complete. Requirements state what must be true. By themselves, they do not measure whether the work achieves it.

Without that measurement, an agent constructs one piecemeal as it works. It decomposes the task, validates each piece in the context that produced it, and eventually decides that it is finished. Every local judgment may be reasonable while parts of the whole remain unmeasured.

Humans currently close the loop by supervising the agent: holding the whole outcome and steering the agent back to it.

We wanted to know whether the model could close the loop on its own.

gdal, from scratch

To test this, we compared single-agent and multi-role runs across 24 selected ProgramBench tasks and three models. Take gdal. We asked Droid to rebuild it from scratch in two ways. In both runs, Droid could execute the reference program without limit, but could not access its source, tests, or the internet.

As a single agent, Droid implemented, checked its own work, and decided for itself when it was done. It wrote 17,000 lines of C++ and reproduced 36 percent of the program's behavior. The code was solid and the common paths worked, but most of the program was still missing. It did not run out of time or budget. It stopped because, by its own assessment, it was done.

We then arranged Droid into a system consisting of separate roles. Before any implementation, one role built an executable standard of completion - its own account of what the reimplementation must do and what evidence would prove it - and the implementation was then held to that standard. This run grew to 115,000 lines and reached 90 percent behavioral parity.

What the system run produced is its own program: it resembles the original in neither shape nor size - a fraction of GDAL's codebase, organized its own way.

This was not an outlier case. Across the 24 tasks, the same system took its 7-Zip recreation from 54 percent parity to 95 percent and its DuckDB recreation from 34 to 80 percent. Several recreations reached the upper 90s.

The underlying model did not change. But when held to its own standard of completion, it reproduced far more of each program's behavior.

Why the same agent stops early

Coding agents usually validate their own work as they go. They implement a piece, write or run a few checks, inspect the output, and decide whether to continue. For a small change, this works well: the task, implementation, and evidence fit in one view.

Large tasks have to be decomposed into features, subsystems, and successive rounds of work. As the agent reaches each piece, it also decides what evidence would count and whether that evidence is sufficient. These checks inherit the scope of the work that produced them. They can establish everything the agent thought to build, but exclude features, interactions, or constraints it never represented.

An agent can therefore make steady, locally correct progress and stop with much of the outcome absent. The problem is not necessarily that it could not implement the rest. It never established a complete account of what remained.

Define validation before the work

Establishing a whole outcome requires more than a list of requirements. The system needs an inventory of what must be established, procedures for establishing each part, and current evidence that those procedures pass against the artifact being shipped.

That standard should be derived from the requirements and relevant sources of truth before implementation narrows the task into individual work items. It need not remain frozen. Checks can be added, replaced, or refined as the system learns. But the standard of completion must not quietly collapse around whatever has already been built.

Why this is rarely done by humans

Separating requirements from evidence is not new. Safety-critical projects use requirements traceability and independent verification and validation. Standards bodies publish conformance suites that many implementations must pass. Product teams write acceptance tests.

What is unusual is deriving and maintaining a comprehensive standard for each project. A conformance suite can spread its cost across many implementations; a product team bears that cost again for each application, rewrite, or migration. Most teams therefore validate incrementally and rely on review, product feedback, and the continuity of the people involved to preserve the whole.

Agents change both sides of this tradeoff. They can produce work faster than humans can inspect it, making informal supervision the bottleneck. But the same capacity can be applied to the standard itself: inventorying the outcome, constructing checks, and rerunning them as the artifact changes.

ProgramBench represents a demanding limit case: the reference program is available, but the model must discover both the behavior space and how to measure it.

ProgramBench

ProgramBench is a cleanroom software-engineering benchmark. Each task provides a reference program, fixtures, and partial documentation. The reference is a black-box oracle: it may be run, but never read, decompiled, or traced.

The goal is to reproduce the observable behavior of the reference program from scratch. Each task is graded and scored against a hidden suite of behavioral checks.

Completeness of a black-box implementation is difficult to measure. Any single behavior is trivially verifiable by running the reference against the candidate, but the whole is not. The shipped documentation covers a slice of the interface; the rest of the program's behavior has to be discovered.

That leaves three questions:

1. Can a frontier model construct its own measure of completeness for a large, unknown program?
2. Can that measure remain useful through a long implementation?
3. Does answering to it produce a better artifact?

Task selection

We selected 24 of the benchmark's hardest tasks, based on the current top leaderboard score.

The set was hand-picked, weighted toward low best-public scores. Some hard-end candidates (php-src, pueue, ditaa, quickjs, chroma, miller) were dropped during screening for persistent safety blocks, single-agent saturation, or a score gated by one undocumented environment variable.

Experimental design

For each selected task and model panel, we ran one campaign in each of two conditions. The system condition added an independent measure of completion without replacing the implementer's ordinary development loop.

Both conditions began from the same task scaffold and fixtures. Within each non-substituted model panel, the single agent and all three system roles used the same model at the same reasoning level. Both conditions could execute the reference program without limit, but neither could read, decompile, or trace it, inspect the benchmark tests, or access the internet. Six disclosed Fable cells used Opus after Fable was safety-blocked.

Once launched, each campaign ran without human intervention. The campaigns were not compute-matched; each continued until the single agent or system orchestrator decided to ship.

Each cell represents one campaign, not an average across repeated runs. After a campaign ended, its final candidate was graded once using the official pb-1.2.0 metric.

How the system closed the loop

The instrument

Full behavioral parity is not directly measurable. A program can accept an effectively unbounded set of inputs, flags, file formats, combinations, and error conditions. Any practical validation strategy has to sample that space.

In our system, that sample is an instrument. Before implementation begins, a validator surveys the reference program and maps where its behavior lives. It then builds a weighted body of cases and the comparison rules needed to judge the candidate's output. We asked for the instrument only in outline. Everything inside it - which behaviors matter, how they are weighted, what counts as evidence - the validator decides.

Two cases from the instrument the validator built for gdal:

D('hillshade.combined', ['raster', 'hillshade', '--variant', 'combined', 'dem.tif', 'out.tif']),
D('contour.levels', ['raster', 'contour', '--levels', '120,150,180', 'dem.tif', 'out.geojson']),

Each case describes the invocation; the grading policy says how to judge the result:

Primary comparator: for every case, the runner executes oracle and candidate sequentially in the same absolute sandbox path (wiped between runs), same env, same fixture bytes and mtimes, and byte-compares four channels:

1. exit code
2. stdout bytes
3. stderr bytes
4. full work-tree delta: the set of files created/modified/deleted by the run, and the exact bytes of each (includes sidecars: .aux.xml, .ovr, .msk, shapefile companions, tile trees).

Byte identity is the default on every channel. A relaxation exists only as a named normalizer attached to specific cases, each carrying a license below.

The validator wrote hundreds of cases like these, and licensed exactly two relaxations across all of them - a masked heap address in debug traces, a date embedded in a file header - each with recorded evidence that the reference cannot produce stable bytes there.

The outer loop

The system consists of three roles: orchestrator, implementer, and validator. The orchestrator delegates to both the implementer, which builds the candidate program, and the validator, which measures it against its instrument.

The validator builds the instrument first. Once implementation starts, the loop goes:

1. The orchestrator chooses what should be measured.
2. The validator tests the current candidate and interprets the failures.
3. The orchestrator decides which findings are real and what work should happen next.
4. The implementer investigates the reference and advances the candidate.

When the instrument stops revealing meaningful differences between successive candidates, the orchestrator can ask the validator to expand a weak area or begin targeted differential testing against the reference.

The wall

In ProgramBench, the standard required an additional boundary because its cases were only a sample of a much larger behavior space.

The boundary holds in both directions. The validator can expand the instrument as it learns more about the reference, but cannot weaken or revise it to accommodate what the candidate happens to contain. And the implementer never authors it, runs it, or sees its cases or raw output: once a sparse sample becomes visible, it becomes the target, and passing it establishes those cases, not the space they were meant to represent.

The validator runs the instrument against the current candidate and groups failures by root cause. The orchestrator reviews those findings, rejects noise or invalid measurements, and turns the remaining problems into a directive at the level of missing features, subsystems, or behavior. The implementer receives the directive, investigates the reference independently, and decides how to change the candidate.

Here is part of a directive from the gdal run:

## 1. Stub frontier (dominant mass, ~60 verbs)

Ordered by measured weight (approximate case mass in brackets). Probe each against the oracle first, as always.

- vgrid: all 11 grid methods [14]
- mdim group on multidim VRT + classic-input pins [12]
- sozip: create / list / validate / optimize [11]
- DEM suite [~30]: contour [7], hillshade incl. variants [6], aspect [6], slope [4], viewshed [4], roughness, tpi, tri [1-3 each]
- rasterize [7], pixel-info [7], calc [7], mosaic [6] + stack, tile [5], footprint [5]

The directive tells the implementer where the candidate is weak without giving away the sample.

Results

We ran the experiment with three frontier models - Fable 5, Kimi K3, and GPT 5.6 Sol.

Every score here comes from the benchmark's hidden suite - a sample of the same behavior space that no role ever saw. The gains transferred from the instrument the system built to an independent measure.

The system runs were far longer and more expensive; for gdal, 14 times the credits and 13 times the wall time. But budget was not what separated the conditions. Every single-agent campaign ended because the agent decided to end it. Additional compute does not help an agent that will not spend it. What our approach changed was the judgment of completion; the compute followed from that judgment.

Every task, every model

Method notes

Conclusion

The single agent didn't lack skill. It lacked a standard of completion. An independent standard, authored by the same model, drove the implementation much closer to behavioral parity with the reference.

ProgramBench gave that standard a particular shape: an inventory and weighted sample recovered from a black-box reference.

Other tasks draw their standard from different sources. A product task may draw on user-approved flows and designs; a migration, on the system being replaced.

What generalizes to real software work is the need for an external, executable standard of completion - one derived from the outcome, before implementation narrows attention, and kept current until the work meets it.

We are building this structure into the next generation of Missions.

To get on the waitlist, contact Factory or contact your Factory account team.