AI Coding Agents
Testing
Finding the cause of flaky tests with coding agents
September 24, 2026 - 2 minute read
Flaky test triage starts with a failure that cannot be trusted to reproduce on demand. A test may depend on timing, shared state, execution order, network behavior, or an environment difference. Re-running the job until it turns green restores the pipeline, but it removes the best evidence about the fault.
The pytest guide to flaky tests recommends treating flakiness as a distinct engineering problem. It identifies uncontrolled system state, overly strict assertions, and thread safety as common causes. Coding agents can help because they can collect repeated observations, compare runs, and keep the investigation tied to a reviewable change.
Preserve the first failing run
Start with the original job, test name, seed, runner image, dependency lockfile, and relevant logs. Keep timestamps and failure output together. If the suite records screenshots, traces, or temporary service logs, preserve those artifacts before another run replaces them.
Give the coding agent a bounded task. Ask it to reproduce the named failure, identify the smallest causal difference between passing and failing runs, add a regression test where possible, and avoid broad production changes. Factory's remote delegation guidance recommends including the desired outcome, reproduction steps, acceptance criteria, verification commands, and repository links. That structure prevents a flaky-test investigation from becoming an open-ended cleanup.
Use controlled variation during flaky test triage
Repeated runs are useful when each run tests a hypothesis. Change one dimension at a time, such as test order, worker count, timezone, locale, database state, or network latency. Record the command and result for every attempt. A batch of identical reruns can estimate frequency, but it rarely explains the cause.
The agent should inspect fixtures and cleanup paths before changing timeouts. It should look for mutable global state, non-unique test data, fixed ports, wall-clock assumptions, unordered collections, and assertions against eventually consistent systems. A longer sleep may lower the observed failure rate while leaving the race intact.
Run the narrow test first, then its containing package, then the affected CI job. When browser behavior matters, Factory's Automated QA can drive the affected flow and capture screenshots or other evidence. That user-level check complements unit and integration tests rather than replacing them.
Require a causal fix and reviewable evidence
A sound fix explains why the failure was intermittent. Examples include waiting on an observable condition instead of a fixed delay, isolating test records, freezing time, removing order dependence, or making cleanup deterministic. Quarantining a test can protect the main pipeline temporarily, but the quarantine needs an owner and an exit condition.
The pull request should show the original symptom, the identified cause, the change, and the exact commands used to verify it. Compare repeated runs before and after the fix when that evidence is practical. Keep unrelated formatting and refactors out of the diff so a reviewer can judge the causal link.
Scheduled recurrence checks can catch a residual failure that a short pull request run misses. Factory Custom Automations can run a focused prompt on a schedule or from a GitHub event, using a stable service-account identity and run target. Configure the automation to report failures and preserve evidence, not to retry until green.
