Build a self-improving software workflow

September 24, 2026 - 5 minute read

A self-improving software workflow needs a specific recurring problem and a way to show that the next run handles it better. Begin with work that already has an owner, a reproducible failure, and an established review path. Let the automation prepare evidence and a proposed correction before deciding whether it should receive more authority.

In an illustrative repository, an automated maintenance task repeatedly edits generated files instead of changing the generator input. The task must change the source, regenerate the output, and pass the repository's consistency check without a reviewer having to redirect it. This scenario is a worked example, not a reported Factory customer result.

Scope one self-improving software workflow

Write down where the workflow starts and what it is allowed to produce. For the example, use a selected queue of maintenance issues as the input. The output should be a proposed patch with the reproduction, changed files, and validation results attached. Keep deployment, changes to repository permissions, and edits to required CI checks outside its initial authority.

Factory's custom Automations support recurring work through schedules, Slack messages, GitHub events, and webhook triggers. The documentation exposes the trigger, instructions, run target, and visibility as separate configuration choices. Webhook triggers are currently in Private Preview. A scheduled pilot avoids depending on that access when the team only needs to inspect a bounded queue.

Choose a repository and a class of maintenance task narrow enough for a reviewer to recognize a correct result. Confirm the working directory, required tools, and identity under which the automation will run. Use existing access controls rather than granting production credentials merely because the workflow is recurring.

Record the stop conditions in the task instructions. Missing generator dependencies should produce a blocked report. An ambiguous ownership boundary should produce a question. An unrelated failing test should be reported with its evidence. None of those conditions should quietly authorize a broader patch or a weaker definition of success.

Capture evidence before proposing a correction

Preserve the failing revision, the relevant command, its result, and the source file that should have been edited. Retain the original maintenance request so the reviewer can distinguish a misunderstanding from a repository defect. Remove sensitive values from reusable fixtures and link to restricted evidence through the organization's approved access path.

Factory's Triage documentation describes classifying incoming work, reducing duplicates, and routing it to an owner or workflow. Triage is in Private Preview. If the team does not have access, keep the pilot on a manually reviewed issue queue rather than implying that automated intake is a prerequisite for the procedure.

Before starting another implementation, check whether an existing issue or pull request already addresses the same failure. Grouping repeated reports under one investigation helps the reviewer see recurrence without creating parallel patches for the same underlying problem. Leave uncertain matches for a person instead of closing a report because its wording looks similar.

For the generated-file example, the useful distinction is whether the generator command was absent, incorrect, or ignored. An absent command calls for better instructions. An incorrect command may require a repository fix. An ignored command needs evidence about what context the run actually received. Saving that distinction prevents the improvement step from becoming a reflexive instruction rewrite after every failed task.

Make self-improving software retain a useful change

Ask Droid to propose the smallest correction supported by the evidence. If the generator command is stable across the repository, add it to the existing project instructions. If the sequence applies only to a release or maintenance procedure, keep it with that workflow. The location determines which future sessions should inherit the lesson.

Factory's AGENTS.md guidance covers durable project instructions, exact commands, and scope boundaries. Its skills documentation covers reusable procedures loaded when relevant. These are concrete files a team can review and version. Neither needs to become a transcript archive.

In the example, the instruction should name the generator input, the regeneration command, and the consistency check. It should also say what to do when the command is unavailable. A reminder to avoid mistakes adds little operational value. A command that has been verified in the intended environment gives the next run a checkable action.

Arize's discussion of self-improving agents separates changes that persist into future runs from recovery within a single run. Apply that distinction when reviewing the patch. Rerunning a failed task may resolve the current issue. The workflow has a reusable improvement only when an accepted change affects later work and evidence supports keeping it.

Validate the patch and the reusable lesson separately

First reproduce the original failure against the old revision. Then run the proposed correction in a clean environment. Keep the task request and acceptance condition stable enough that the reviewer can attribute the changed outcome to the patch rather than to extra hints supplied during the second run.

Next try a neighboring case. A different generated file in the same package can reveal whether the new instruction describes the process or merely names the first answer. A task in a package without generated output can reveal an instruction that has become too broad. These are proposed checks for this example, not a claim that Factory generates every evaluation automatically.

Factory's Automated QA documents project-specific checks and evidence across application surfaces. For a web change, include the affected user flow. For the generator example, retain the command output and resulting diff. Its default failure-learning mode suggests updates in the report for manual review, which is a useful starting point while the team learns what should persist.

Add automated code review if it fits the repository's review process. The documented workflow skips draft pull requests, so do not mistake an absent review on a draft for a successful review. Keep required checks and human approval in the existing source-control process, and promote the pull request for review only when its evidence is ready.

Review later runs before expanding the scope

After an authorized reviewer accepts the correction, observe subsequent eligible tasks. Record whether they edited the correct source, whether regeneration succeeded, and whether a person still had to redirect the work. Keep blocked tasks and failed attempts visible. Counting only accepted pull requests would hide much of the work the pilot is meant to improve.

Anthropic's tool-improvement work describes evaluating changes on held-out cases rather than relying entirely on the examples used during development. For this workflow, reserve maintenance tasks that were not included in the correction request. Keep their expected outcomes out of the reusable instructions.

Check for new costs as well as fewer repetitions of the original error. A stricter generator procedure may add unnecessary work to unrelated changes. An instruction may become stale after the repository reorganizes its build system. Remove or revise the guidance when its scope no longer matches the project, and keep a straightforward path to restoring the previous version.

Expand only after the team can explain the observed result. Add another task class, repository, or trigger deliberately, rather than changing all of them at once. A pilot that still depends on repeated human redirection has produced useful diagnostic evidence, but it has not yet earned broader release authority. Preserve that distinction in the report presented to the next reviewer.