Software Factory
Enterprise AI
Self-improving software needs a failure record
September 24, 2026 - 2 minute read
A task can finish successfully after the user corrects the same mistake several times. Completion counts hide that cost. Self-improving software needs a record of the friction, a way to propose a correction, and evidence that the next run behaves better.
Factory’s Signals research, published in January 2026, describes this inside Factory’s own development process. Signals analyzes session patterns, identifies recurring problems, and creates work for Droid. This is a published account of an internal system, rather than a customer-facing feature that automatically retrains models.
Give self-improving software evidence to act on
The Signals pipeline abstracts sessions into categorized observations. It looks for patterns such as repeated rephrasing, rejected tool calls, and recovery from errors. Factory’s research describes surfacing abstracted patterns rather than raw conversations to human analysts.
The paper describes recurring friction crossing a threshold, an issue being filed, and Droid implementing and reviewing a proposed fix. Its human-approval step matters as much as the automated detection. It states:
“A human still approves the PR before merge.”
That distinction keeps the mechanism concrete. The system can prepare its own improvement while people retain the decision to accept the change. An organization evaluating the approach should inspect the evidence attached to the proposal, the tests that exercise the failure, and who can authorize deployment.
The same report connects releases to subsequent friction. For a change addressing ambiguous requests, Factory reported a 30% drop in repeated-rephrasing friction within 48 hours. That is a dated observation from Factory’s internal system, not a promised improvement for every workflow.
Test whether self-improving software actually improved
Begin with a recurring failure that engineers can reproduce. Preserve the relevant error, affected version, and acceptance test. A vague instruction to “learn from this” leaves the next run without a checkable objective.
Factory’s Automations provide documented triggers and instructions for repeated engineering work. Its automated QA provides a separate way to exercise the resulting application. Connect those mechanisms to your existing review and release gates instead of granting broader permissions because a workflow runs repeatedly.
Measure the same behavior after the change. Fewer retries are useful only if the task still meets its requirements. Check for displaced failures, increased operating cost, and regressions in neighboring behavior before expanding the automation.
Factory’s software-factory framing extends the feedback loop across engineering signals, implementation, validation, shipping, and monitoring. Factory is the product used to operate that delivery model. Engineers still define what counts as an improvement and which changes may ship.
Discuss a recurring failure with Factory, the evidence available today, and a bounded workflow for addressing it.
