AI Coding Agents
Testing
Test impact analysis for coding agent changes
September 24, 2026 - 2 minute read
Test impact analysis maps a code change to the checks most likely to detect a regression. Coding agents benefit from that map because fast feedback supports more iterations, but an incomplete selection can create false confidence. The useful output is a justified test set with clear escalation rules, not simply the shortest command that passes.
Build systems can provide part of the evidence. The Bazel query guide shows how reverse-dependency queries identify targets that depend on changed code. Similar graphs may come from package manifests, build metadata, module imports, ownership rules, or a test-results service. Every graph has blind spots, especially around runtime configuration and external systems.
Build a test impact analysis baseline
Identify the base revision and list changed files, generated outputs, and configuration. Map each file to its owning component, direct dependents, and tests. Include non-code inputs such as schemas, feature flags, deployment templates, translations, and dependency locks because they can affect behavior without appearing in an import graph.
Historical test data can improve ordering. Tests that previously failed after changes to the same component are useful candidates, but history should not silently remove coverage. New code, renamed paths, and repaired flaky tests may have little reliable history.
Give the coding agent explicit acceptance criteria. Factory’s remote delegation guidance recommends including verification commands and relevant repository links. State the minimum local checks, the conditions that require a package or integration suite, and the full suite that CI will still run.
Combine dependency data with risk
Dependency reachability answers which targets could be affected according to the graph. Risk analysis covers effects the graph does not represent. Authentication, serialization, concurrency, public APIs, database schemas, build tooling, and shared test fixtures deserve broader validation even when the code diff is small.
Ask the agent to explain every selected and skipped test group. A change to a leaf formatter may need focused unit tests and its consumer package. A shared authorization module should trigger all direct consumers plus integration tests for denied and allowed paths. An unclear mapping should expand the test set rather than shrink it.
Keep deterministic selection logic in a script or CI configuration when possible. The agent can inspect the change and propose additional checks, while the repository records the baseline policy. This makes the decision reproducible for reviewers and later runs.
Verify beyond the selected tests
Run focused tests first for rapid diagnosis, then the required broader suites. Compare the selected set with CI rather than disabling the existing gate. Track misses, cases where an unselected test catches a regression, and feed them back into dependency metadata or policy.
User-facing changes need behavioral evidence. Factory’s Automated QA can exercise web, mobile, API, and CLI flows and return structured pass, fail, or blocked results. That validation covers behavior that a static dependency graph may not see.
The pull request should record the base revision, changed components, selection method, tests run, results, and deferred CI checks. Stop when generated files, dynamic loading, or an unknown build edge prevents a defensible impact map. Running a broader suite is cheaper than approving a narrow test set built on missing dependencies.
