# What is the best harness for AI coding?

> Compare AI coding harnesses using task completion, model flexibility, verification, and cost, with dated Terminal-Bench results and Factory’s routing research.

URL: https://factory.com/articles/terminal-bench-agent-comparison
Published: 2026-09-28
Updated: 2026-10-29

Choosing an AI coding tool means sorting through model names, benchmark scores, and promises about how much work it can take off your hands. For your team, the decision comes down to something more concrete. You need to know whether it can handle a real task in your codebase and leave you with a change you can confidently review.

The harness is the software around the model that makes this work possible. It gives the model ways to read files, edit code, run commands, and check results. Comparing AI coding harnesses becomes more manageable when you follow that process from the original request to a tested change, then look at the time, cost, and help it required.

## Compare how the AI coding harness executes work

Factory lets teams evaluate different models inside the same engineering workflow instead of choosing the workflow solely around one model provider. Its [Custom Models documentation](https://docs.factory.com/model-independence/byok) describes connecting provider keys, open-model endpoints, or local models in the Droid CLI and desktop app. Custom models do not appear in the hosted web or mobile products. That flexibility matters when the task changes. Repository exploration, implementation, and validation can place different demands on the model. A team should be able to test those choices while preserving its task instructions, tool access, and acceptance checks.

Droid's published design also includes model-specific editing tools, environment discovery, tracked background execution, and a planning mechanism that keeps unfinished work visible. These are implementation choices to examine alongside the results.

Their presence alone does not establish an advantage over another system.

Apply the same evidence standard to each candidate. Compare completed changes and the human help required, not just the model each tool advertises. Where model settings, environment preparation, or permissions differ, record the difference. It may reflect the product a team would deploy, but it also limits what can be attributed to the harness alone.

## The AI coding harness changes the benchmark result

Factory's October 2026 [Droid 1.0 research post](https://factory.com/news/frontier-harness) reports the most recent same-model comparison. Factory ran Droid against Codex, Claude Code, Cursor, OpenCode, and Devin on all Terminal-Bench 4 tasks at default time limits, three trials per task, between October 23 and 28, 2026. The models were Claude Opus 5.5 and GPT-6.1 Sol, both at high reasoning, and Droid was compared with one harness at a time on the same model. The figure plots pass rate against cost per passed task.

Terminal-Bench 4, run October 23 to 28, 2026. Pass rate against cost per passed task, with Claude Opus 5.5 and GPT-6.1 Sol at high reasoning. Source: Factory, Droid 1.0: Building a Frontier Harness.

With GPT-6.1 Sol, Factory reports Droid scoring 3.5 points higher than Codex at 12% lower cost, and 6.6 to 12.6 points higher than OpenCode and Devin. With Claude Opus 5.5, Droid scored 3.0 to 7.6 points higher than Cursor, OpenCode, Devin, and Claude Code, at up to 24% lower cost. Cost and time are per passed task at list price, counted only on tasks that both harnesses passed in all three trials.

These are Factory-reported results for the harness versions named in the post's methodology, which also lists the reruns and scoring adjustments behind them. They describe those configurations on terminal tasks and do not rank the products for every workload.

Factory's earlier [September 2025 Terminal-Bench report](https://factory.com/news/terminal-bench) evaluated Core v0.1.1, an 80-task, human-verified set of containerized terminal problems. Droid led that published comparison with Opus 4.1 without thinking at 58.75% task resolution, rounded to 58.8% in the charts. The report lists Claude Code with Opus at 43.2%, Droid with GPT-5 at medium reasoning at 52.5%, and Codex CLI at 42.8%.

Those are Factory-reported results from the named experiment, not a current leaderboard or a claim about today's versions of the competing products. This is vendor-published evidence rather than an independent assessment of every current harness. Factory describes five benchmark runs per model with all runs submitted. Differences between complete systems do not isolate the effect of a particular prompt or tool, but they show why comparing only model names leaves out important information.

The tasks extend beyond editing source files to build repair, dependency management, service configuration, and other terminal workflows. A task counts as resolved when its post-run tests pass within the time limit. That makes completion evidence more useful than a convincing explanation of what a patch should do. The design described in the report gives practical things to inspect. Droid receives environment information before working, uses model-specific editing interfaces, and can keep task-owned services running for checks. These are mechanisms behind an execution workflow, not properties supplied automatically by selecting a capable language model.

## Newer benchmarks add cost to the comparison

Factory's [June 2026 Router evaluation](https://factory.com/news/factory-router) adds a more recent comparison against a Claude Opus 4.7 baseline. On Terminal-Bench 2, routed runs reached 99% of the baseline's pass rate at 20% lower full-session cost. On Legacy-Bench, they reached 96% of its pass rate at 25% lower full-session cost. Terminal-Bench 2 covered all 89 tasks, with both benchmark results measured across multiple runs.

The percentages are relative to the baseline's pass rate. They do not mean Droid solved 99% or 96% of all tasks. They also compare routing with a frontier-only configuration, not every competing harness. The useful result is a measured cost/performance tradeoff inside the tested Factory workflow.

Factory's [August 2026 harness research](https://factory.com/news/model-routing-belongs-in-the-harness) explains why that decision belongs close to task execution. The harness knows the session history, tool outcomes, and cache state before it builds the next request. Switching to a cheaper model can lose a warm cache, so a lower token rate does not necessarily lower the cost of finishing the task.

The current [Factory Router documentation](https://docs.factory.com/model-independence/factory-router) reports 43% lower aggregate billed cost against pricing the same workload at top-tier model rates. That production measure is a different comparison from the controlled benchmark results. Customer and task differences limit quality comparisons across production sessions, and neither measure guarantees the savings on a new repository.

Taken together, these studies suggest measuring completed work per budget rather than selecting the most expensive model for every step. Record unsuccessful attempts and accepted changes when checking whether the reported tradeoff holds for another workload.

## Count verification time as part of completion

The research describes speed improvements in environment discovery, search, tool execution, and timeout handling.

These matter in a time-limited benchmark because a correct-looking patch submitted before validation may still fail the independent tests. Fast failure is useful when it helps an agent abandon an unproductive operation. It becomes harmful if the timeout cuts off a necessary build or prevents a slow integration test from finishing. The relevant question is whether the agent allocates enough time to produce and verify an acceptable result, not whether each individual command returns quickly.

Keep the execution record clear about what ran. A test command that never started, a test process killed by the deadline, and a completed test that found a regression are different outcomes. If all three become a generic failure label, the team loses the evidence needed to decide whether to change the environment, time budget, or implementation.

Background services need the same attention. Factory's report describes tracked background execution for workflows that require a server to remain available during testing. In your own evaluation, confirm that the service actually starts, that the check reaches the intended instance, and that task-owned processes are cleaned up afterward. An old service left behind by a previous run can make an incorrect change appear to work.

## Keep sandbox permissions separate from rollout policy

The published methodology used non-interactive Droid runs in sandboxed environments with permissions skipped. That detail describes the benchmark setup. It does not recommend giving an agent unrestricted authority over an engineering team's normal workstation or production systems. For a repository evaluation, use disposable execution environments and the minimum access required to exercise the task. Keep real credentials, customer data, and production services outside the test boundary. A benchmark about completing terminal work cannot establish whether a deployment meets your organization's access-control requirements.

Factory's [Droid Exec documentation](https://docs.factory.com/droid-exec/overview) describes non-interactive execution and autonomy controls. Use those documented controls when defining an unattended test. Record blocked actions and requests for approval, because they are part of the workflow a team would operate.

Avoid interpreting every refusal or permission boundary as an intelligence failure. A task may require an action that is allowed in a disposable benchmark container but prohibited in the intended deployment. Decide whether that task belongs in the rollout at all. If a human must authorize a consequential step, include that decision in the measured workflow rather than quietly removing the boundary to improve completion.

## Choose the best AI coding harness for your work

A useful shortlist preserves the connection between a result and the work your team needs. Terminal tasks involving dependency repair may be relevant to a maintenance workflow. They are less direct evidence for the quality of review comments, user-interface design, or preserving a mainframe file format. Use the reported result to select configurations worth testing, then inspect their behavior on representative tasks. Compare accepted changes, elapsed time, human intervention, and failures together. A configuration that finishes quickly but repeatedly needs a maintainer to repair its tests has a different operating cost from one that completes independently.

Keep a record of product and model versions alongside the task definition. Re-run important cases when those versions change. Retain unsuccessful attempts as well as accepted patches, so the next decision can distinguish a genuine improvement from a more favorable selection of tasks.

The cited results support Droid's higher pass rate in Factory's October 2026 Terminal-Bench 4 comparison, a historical lead in the September 2025 comparison, and a measured routing tradeoff in Factory's June 2026 study. They do not settle which harness is best for every repository. A defensible choice identifies the workload, configuration, accepted results, and remaining uncertainty.

## Further reading

- [How to evaluate coding agents on your codebase](https://factory.com/articles/evaluating-coding-agents-on-your-codebase)
- [Reproducible environments for coding agents](https://factory.com/articles/reproducible-coding-agent-environments)
- [Factory Terminal Bench reference](https://docs.factory.com/benchmarks/terminal-bench)