Why model routing must be in the harness

Inference spend is rising as agents are deployed more widely, take on more complex tasks, and run longer autonomously. Enterprises are starting to put model routing into production to keep that spend in check. Among models capable of serious agent work, prices span more than fiftyfold, and a single task can take dozens or hundreds of model calls as the agent reads files, runs tools, and revises its approach. In conversations with leaders at systemically important banks, cybersecurity companies, and major technology firms, the same architecture question keeps coming up. A gateway receives model requests from applications and forwards them to model providers. Every model call already passes through one, so why put model routing in the harness?

At the gateway, an agent's model call looks like any other completion request: a prompt goes out, tokens come back, and the connection closes. Inside the harness, that call is one step in a computation that can run for hours.

Model routing belongs in the harness because the harness can price model choices against session cache and history, assign models to work it creates before a gateway request exists, and connect those choices to task outcomes.

Harness routing reduces cost and latency without lowering quality

We built Factory Router to make these model choices in production, and it has routed customer work for over two months. Across production sessions, routing cut aggregate cost 58% against using the frontier model for every call. The median routed session saved 76%, and more than nine in ten saved at least half.

Routed sessions matched frontier-pinned sessions across eight production measures, including task completion, command and test failures, repeated edits, delivered artifacts, and whether users accepted the result or started over.

Median turn latency also fell from 81 seconds in frontier-pinned sessions to 49 seconds with routing as efficient models respond faster on the turns they handle.

The harness chooses while constructing the next request

Model selection could happen at any of the three layers a request crosses on its way to an answer, but each layer would make the choice with different information.

Model serving sits deepest: the provider's infrastructure that holds model weights, executes requests, and streams tokens back. It sees the request, available capacity, and endpoint state, then routes across deployments to balance load and avoid unhealthy hardware.

The gateway sits in front of the providers and sees every request, the organization's allowed models and usage policies, and provider health. It enforces the allowed set, applies policy, selects providers, and handles failover. This central position makes gateway model routing attractive: one service can choose models for every application behind it without changing each application. It is also easier to deploy and maintain than routing embedded in an agent. For request-sized work such as single-shot Q&A, the request itself can contain enough information to make the choice.

The harness enables the agent loop that turns a task into model calls. It assembles prompts from the session and runs the tools the model calls for. When the model makes a defined subagent call, the harness creates and runs the worker session. It also creates review jobs and checks finished work. Its record includes session history and cache state, the current job and tool results, and the tests, artifacts, and outcomes that show how the work is going.

Model selection is one sub-step in constructing the next request. Suppose the next turn will edit a file. For one model, the harness supplies a find-and-replace editor and instructions for using it; for another, it supplies a diff-based editor and different instructions. Model families are not always wire-compatible: switching families can also discard encrypted reasoning content that the next call would otherwise reuse.

The harness must choose the model before it assembles the request's system instructions, tool definitions, reasoning settings, and context. By the time the gateway receives the request, the editing interface and instructions already reflect that choice. If the gateway substitutes another model, the harness must rebuild the request around that model's interface. The gateway can enforce the allowed model set, choose a provider, and handle failover, but the final model choice belongs where the request is built. The response returns to the harness, which executes the requested edit and uses the result to construct the next request.

The right model changes as the work develops

Frontier models are highly effective at planning and hard diagnosis but unnecessarily costly for straightforward work a lighter model can handle. Another common pattern is that efficient models are well suited to searching, running targeted checks, and assembling context for a frontier model to synthesize when the work requires stronger reasoning.

For example, a task to fix a flaky-test moves through both patterns. An efficient model suits its opening turns: at a fraction of the frontier price and with faster responses, it reads the CI log, finds the retry wrapper around the failing call, reproduces the failure, and maps the relevant corner of the repository.

Then a run under load shows two test suites colliding on a shared fixture. The failure is a race. The easy explanations are gone, and the diagnosis calls for stronger reasoning than the exploration did. For those turns, a frontier model is worth several times the efficient model's price. Once it identifies the race and writes a plan of mechanical edits, the remaining work suits the efficient model again. But deciding whether a switch pays takes more than comparing two price lists.

Only the harness can judge when a model switch is worth the cache miss

Every model call repeats most of the session. Each turn sends the new tool results and text along with the accumulated transcript, so the model processes nearly the whole session again. Providers cache that processed prefix for a particular model and typically charge a tenth of the fresh-input rate when it is reused. Move the same transcript to another model, and the accumulated history must be processed again at fresh rates, five to ten times the cached-input rate.

Whether that cost is justified depends on what has happened in the task. Repeated failed fixes can make it worth paying fresh rates to hand a stronger model twenty turns of gathered context. Steady progress usually cannot. A gateway can see the transcript and tool results, but only the harness owns the task state used to interpret them: whether the work has stalled, what the latest result changed, and how much work remains. The cheapest next call often comes from the model that is already warm, even when another model lists a lower price. Staying put is itself a routing decision.

The cost of switching changes during the session. Caches expire when a session idles between calls. When the harness compacts a long history into a summary, the next call carries a shorter prefix, but that prefix must still be processed at fresh-input rates until its own cache warms. The same switch can be cheap at turn five and expensive at turn ninety.

Our data shows how those stakes grow:

A gateway that switches models without preserving the session's warm prefix is cache-blind and can make the same work cost significantly more. In our data, the modeled fully uncached cost exceeds an all-frontier single-model baseline by turns 6–20, then reaches 2.12 times that baseline at turns 61–150 and 2.37 times at turns 151–200. Actual cache-aware routing remains between 0.19 and 0.28 times the same baseline. The comparison holds each turn's observed model, input, output, and task trajectory constant and excludes cache-write premiums. The cost of cache blindness grows with the transcript: by turns 61–150, the median call carries 7.6 times the input of a call in the first five turns, while warm-prefix reuse limits cost to 4.4 times the first-five-turn level. In production, inference spend is concentrated in the longest-running sessions.

Three completed sessions show the range of decisions cache-aware routing produces:

Each savings figure compares the completed session against the same session priced as if every call had used the frontier model. All three finished their tasks. The catalog loader never needed more than the efficient model. The artifact viewers used the frontier model for part of the work and the efficient model for the rest. The router kept the Prisma migration on the frontier model for all 67 turns because the work never offered a safe place to step down. Zero savings was the right answer for that session.

The harness fits the job and the model to each other

Every job the harness creates is a joint design choice: what work to assign and which model will do it. Model capabilities affect how the harness divides the task and specifies each assignment. The resulting scope, context, and completion conditions determine which model can perform the job efficiently.

Partway through an ordinary session, a model may propose a subagent to explore a directory or attempt a fix in isolation. The harness writes the worker's assignment as a short specification and selects its model. The worker starts without the parent's warm cache but receives the specification instead of the full session history. The parent can remain on its current model and preserve its warm prefix. A strong parent can use efficient workers for focused exploration or implementation, then synthesize their results.

In a Factory Mission, a planner session turns the user's instruction into an ordered plan and writes each worker's purpose, prerequisites, and completion conditions. The harness starts each worker with a model selected from that job description. The median mission spans 423 turns over ten sessions and about twelve hours. Routing saves 37.8% of the complete mission cost against pricing every call at the frontier model's rate.

The harness also defines review as a separate job. At a milestone, it writes one review job for a body of finished work. An efficient model can handle the implementation while a stronger model reviews the result. A useful validator should fail where the model it reviews does not, so the reviewer depends on the implementer. For that reason, our default validator comes from a different model family than our default implementer.

Task outcomes improve later routing

A successful model response establishes that tokens came back. Whether the task succeeded shows up elsewhere: in command failures, repeated edits, test results, delivered artifacts, and the user's decision to accept the result or start over. The harness sees all of it because it runs the tools and checks the work.

Some of those signals act immediately. Repeated failures on the same fix can change the next model choice before the session ends. Other signals arrive only when the task is complete. By then, a long session has tied many model choices to one outcome, making it hard to credit any single choice. Harness-created work tightens that link. A worker passes or fails against its completion conditions, and a review grades one specific piece of the work, so each result attaches to fewer model choices.

Each routing record captures the session state at a choice, the model selected, and the result that followed. We use that history to refine the policy Factory Router runs today. The same data can train future routing policies.

Production work varies by customer and task, which makes quality comparisons between sessions imperfect. On our published benchmarks, we tested routing against a frontier-only baseline on the same tasks. Routed runs reach 99% of the frontier model's pass rate on Terminal-Bench 2 and 96% on Legacy-Bench, at about 20% lower cost per successful run. Both benchmarks are more challenging than typical production workflows.

For agents, the next model choice begins before a gateway request exists. The harness knows how much history the current model has cached, what new work the agent is about to create, and whether earlier work passed its tests. Factory Router uses that context to choose models as the work develops. In production, those choices have reduced cost and turn latency while holding frontier quality. Because the harness sees task outcomes as well as requests, completed work can improve later routing choices. Model routing belongs in the harness.