AI Coding Agents
Reliability
Backup recovery tests for agent-generated services
September 29, 2026 - 2 minute read
Backup recovery testing proves that stored data can be restored into a usable service. A coding agent may change a schema, encryption setting, storage path, or deployment dependency without touching the backup job. The backup can continue reporting success while the restore procedure becomes incomplete.
NIST SP 800-34 Revision 1 describes contingency planning as preparation for restoring systems after an interruption. For an application team, the strongest evidence is a repeatable restore into an isolated environment followed by integrity and behavior checks.
Define the recovery target before testing
List the data stores, object storage, configuration, keys, and external dependencies required for a usable recovery. Separate durable business data from caches and derived indexes that can be rebuilt. Document the order in which components must return.
Record the recovery point objective and recovery time objective approved for the service. The test should measure against those targets without turning one successful run into a guarantee. Data volume, region availability, and dependency failures can change the result.
Use a backup created by the same mechanism as production. A hand-built fixture can test migrations, but it does not prove that the operational backup format is restorable.
Exercise a complete backup recovery
Restore into an isolated account, project, or namespace with no route to production. Start from empty infrastructure so hidden dependencies cannot leak in from an existing environment. Verify checksums or native integrity reports before bringing up the application.
Run schema migrations in the documented order. Test a backup from the oldest version the recovery policy still supports, not only one created by the current commit. If encryption keys are required, use isolated test keys and confirm the process fails clearly when a key is missing or wrong.
After startup, query representative records, relationships, permissions, and file attachments. Rebuild derived indexes and compare counts or sampled identifiers to the restored source. Then run one read and one reversible write through the application interface.
Give a coding agent bounded recovery work
Provide the approved runbook, infrastructure boundary, synthetic data set, and recovery objectives. Require the agent to stop before any production endpoint or credential is used. Destructive commands should target an environment created for the test and named explicitly.
Factory’s Missions planning supports explicit requirements and acceptance criteria for long-running work. A recovery mission can treat provisioning, restore, integrity checks, and cleanup as reviewable steps while keeping production actions outside the task.
Keep cleanup separate from proof. Preserve sanitized logs long enough for review, then remove the isolated recovery environment through the approved workflow.
Review backup recovery evidence
Record the backup timestamp, restore start and finish, component versions, integrity results, and failed checks. Do not record secrets or customer data. Note every manual intervention because an undocumented step is part of the recovery time.
Repeat the exercise after changes to schemas, encryption, retention, or storage infrastructure. A passing backup job measures creation. A recovery test measures whether the organization can use what it created.
