AI Coding Agents
Testing
Test impact analysis for coding agent changes
September 24, 2026 - 2 minute read
Test impact analysis maps a code change to the checks most likely to detect a regression. Coding agents benefit from that map because fast feedback supports more iterations, but an incomplete selection can create false confidence. The useful output is a justified test set with clear escalation rules, not simply the shortest command that passes.
A leaf formatter and a shared authorization module can have similarly sized diffs but require different checks. The formatter may need its unit tests and consumer package. Authorization needs coverage of direct consumers and allowed and denied paths, even if only one condition changed.
Build systems can provide part of the evidence. The Bazel query guide shows how reverse-dependency queries identify targets that depend on changed code. Similar graphs may come from package manifests, build metadata, module imports, ownership rules, or a test-results service. Every graph has blind spots, especially around runtime configuration and external systems.
Build a test impact analysis baseline
Identify the base revision and list changed files, generated outputs, and configuration. Map each file to its owning component, direct dependents, and tests. Include non-code inputs such as schemas, feature flags, deployment templates, translations, and dependency locks because they can affect behavior without appearing in an import graph.
Historical test data can improve ordering. Tests that previously failed after changes to the same component are useful candidates, but history should not silently remove coverage. New code, renamed paths, and repaired flaky tests may have little reliable history.
Apply Factory’s remote delegation guidance by naming the minimum local checks and the changes that require escalation. For a shared authorization edit, require consumer and integration tests even when the affected-target script selects only unit tests. State which full suite CI will still run and require the agent to report any gap between that policy and its selected checks.
Combine dependency data with risk
Dependency reachability answers which targets could be affected according to the graph. Risk analysis covers effects the graph does not represent. Authentication, serialization, concurrency, public APIs, database schemas, build tooling, and shared test fixtures deserve broader validation even when the code diff is small.
Ask the agent to explain selected and skipped test groups against that policy. An unclear mapping should expand the test set rather than shrink it.
Keep deterministic selection logic in a script or CI configuration when possible. The agent can inspect the change and propose additional checks, while the repository records the baseline policy. This makes the decision reproducible for reviewers and later runs.
Verify beyond the selected tests
Run focused tests first for rapid diagnosis, then the required broader suites. Compare the selected set with CI rather than disabling the existing gate. Track cases where an unselected test catches a regression. Use those misses to correct dependency metadata or selection policy.
User-facing changes need behavioral evidence. Factory’s Automated QA can exercise web, mobile, API, and CLI flows and return structured pass, fail, or blocked results. That validation covers behavior that a static dependency graph may not see.
The pull request should record the base revision, changed components, selection method, tests run, results, and deferred CI checks. If generated files, dynamic loading, or an unknown build edge prevents a defensible impact map, withhold the narrow-coverage claim and run broader checks. Record any unresolved gap before approval.
