How to test log redaction in agent-written code

To test log redaction, plant canary values in fixtures, run both success and failure paths, and scan the final emitted logs, traces, and artifacts for those values in exact and encoded form, while asserting that safe diagnostic fields remain. Coding agent changes to logging should carry those tests and a security review before merge.

Log redaction fails when a change removes one obvious token but leaves the same value in an error object, URL, trace attribute, or serialized request body. Coding agents can trace those paths and implement a bounded fix. Reviewers still need proof that sensitive data is gone and operational context remains.

The OWASP Logging Cheat Sheet identifies data that should usually be removed, masked, hashed, or encrypted before it reaches logs. Treat that list as a starting point, then apply the organization's own data classification and retention rules.

Build a log redaction inventory

Start with the reported exposure and follow the value from input to every logging sink. Include application logs, exception handlers, audit events, traces, metrics labels, browser telemetry, and CI artifacts. Structured logging can duplicate a value across a message and attached fields.

Give the agent representative test values rather than real credentials or personal data. Name the allowed output for each field. An account identifier might be safe as a stable opaque token, while an authorization header should be removed completely. Do not let the task invent policy from field names alone.

Writing the policy down as a small table gives the agent and the reviewer the same target. The rows below are an example. The real values come from your data classification and the security or privacy owner.

Factory's Droid Exec supports a constrained repository task and structured artifacts. Use it against sanitized fixtures, limit tool access, and keep production log stores outside the execution boundary.

Include third-party SDKs in the inventory. An HTTP client or error reporter may capture headers and payload fragments without an explicit application log statement.

Test redaction at the final sink

Unit-test the redaction function, then capture the final emitted record. A formatter, exporter, or error serializer may add fields after the first filter runs. Exercise success, validation failure, downstream timeout, and unexpected exception paths because sensitive values often appear only in failure logs.

OpenTelemetry's guidance for handling sensitive data recommends controls at instrumentation and collector layers. Test both when the system uses a collector. A collector processor can provide a backstop, but source-level minimization reduces the number of places that ever receive the value.

Use canary strings in fixtures and scan captured logs, traces, and test artifacts for exact and encoded forms. Also assert that the event name, safe identifiers, error class, and correlation fields remain. A redaction change that removes all diagnostic value creates pressure to bypass it during the next incident.

Make log redaction review repeatable

The pull request should identify the sensitive field, every sink checked, the chosen transformation, and tests for direct and nested values. Include a retention or deletion follow-up when earlier records may already contain the data. Code changes cannot remove historical copies by themselves.

Separate application diagnostics from immutable audit events. Apply the approved policy to each, and involve the security or privacy owner when requirements conflict. The coding agent should implement and verify the decision rather than decide which regulated data the organization may retain.

How Factory checks logging changes

Factory is a platform for running a software factory, where agents called Droids take work from a signal to a reviewed pull request. For logging changes, Factory adds a security reviewer that looks for data exposure, and keeps the agent's own access narrow.

Security review on the pull request. Factory's security review traces changed data flows across validation, database, network, and filesystem boundaries and checks them against STRIDE, which includes information disclosure, and the OWASP Top 10. It reports validated findings inline with severity and a suggested fix.

Repository-specific rules. A security-review-guidelines skill can list the fields your organization treats as sensitive and the sinks that must never receive them. The reviewer then flags a new log statement, error serializer, or trace attribute that carries one of those fields.

Narrow access for the agent. Run the redaction task against sanitized fixtures. The sandbox in Agent Safety & Controls can deny reads of sensitive paths and limit network access to approved domains, which keeps production log stores outside the execution boundary. Teams that cannot send source code to a hosted service can run Factory in their own environment with Factory Private.

To run a security review on your own code, use these commands in the Droid CLI or desktop app.

Frequently asked questions

What is log redaction?

Log redaction removes, masks, hashes, or encrypts sensitive values before they reach logs, traces, metrics, and error reports. The OWASP Logging Cheat Sheet lists data that usually needs this treatment.

Where do sensitive values leak in logs?

Values often appear in error objects, URLs, trace attributes, serialized request bodies, and third-party SDKs such as HTTP clients and error reporters. Failure paths leak more often than the success path.

How do you test that redaction works?

Use canary strings in fixtures, capture the final emitted record after every formatter and exporter, and scan logs, traces, and test artifacts for the canary in exact and encoded forms. Also assert that safe diagnostic fields remain.

Does fixing the code remove sensitive data already logged?

No. Earlier records may still contain the value. Plan a retention or deletion follow-up alongside the code change.

How does Factory help review logging changes?

Factory security review checks pull requests for data exposure using STRIDE and OWASP methodology and reports validated findings inline. A security-review-guidelines skill can list the fields your organization treats as sensitive.

Further reading

• Telemetry schema checks
• AI agent telemetry inside private networks
• Secret scanning
• Private AI security remediation
• OWASP Logging Cheat Sheet