AI Coding Agents
Observability
Coding agent observability that reviewers can verify
September 24, 2026 - 2 minute read
Coding agent observability work often looks harmless because instrumentation sits beside application logic. A changed attribute, missing trace context, or unbounded label can still break dashboards, increase storage costs, or hide the failure a team needs to diagnose. Reviewers need evidence that a telemetry change preserves semantics and remains useful under real traffic.
Suppose a unit test confirms that a span includes an error category, but the deployed collector drops that attribute. The dashboard query can no longer distinguish the failure operators need to investigate. The change needs validation through the collector and query, beyond the instrumentation test.
OpenTelemetry defines signals such as traces, metrics, and logs, with profiles as an emerging signal. Instrumentation should begin with an operational question and select the signal that answers it. Adding all available context to every event creates noise and can expose sensitive data.
Scope coding agent observability work
Start with the behavior that operators need to see. Name the service, code path, failure mode, existing dashboard or alert, and expected signal. Include the project’s naming rules, semantic conventions, sampling policy, and fields that must never enter telemetry.
Trace the signal from the instrumentation call through context propagation and collector processing to the stored result. Then run the dashboard or alert query against that result. Each stage can change what operators can see.
Apply Factory’s remote delegation guidance by stating the operational question, such as distinguishing dependency timeouts from application validation failures. Supply a representative synthetic request for each case and the query that should separate them. Require the agent to verify that the permitted error category survives export without adding request bodies or customer identifiers.
Control names, cardinality, and sensitive data
Stable names let dashboards and alerts survive refactors. Reuse established semantic conventions and repository helpers before creating a new metric or attribute. Check that units and aggregation types match existing instruments. A counter should represent an event that accumulates, while a histogram should use boundaries suited to the measured distribution.
Review every attribute for cardinality. User IDs, request IDs, raw URLs, exception messages, and unbounded customer values can create a new time series for each observation. Keep those values out of metric labels. Place high-cardinality correlation data in traces or structured logs only when policy permits it.
Telemetry can contain source fragments, prompts, headers, database values, and credentials. Require tests or processor rules that redact sensitive fields. Factory’s Droid Shield scans added lines for potential credentials during Droid-driven commits and pushes, but it does not establish a telemetry privacy policy. Review data collection and export boundaries directly.
Verify behavior across the signal path
Run the changed service with a controlled request that covers success and failure. Capture the emitted signal, then verify the exported form or a local collector query. Confirm parent and child trace relationships, metric units, log severity, error status, and resource attributes. Repeat enough requests to catch duplicate registration or per-request instrument creation.
The pull request should show the operational question, signal definition, before-and-after query, sample output with sensitive values removed, and focused test results. Include expected dashboard or alert changes when names or labels move.
Stop when the destination, retention policy, or allowed data fields are unknown. Instrumentation without a defined consumer and privacy boundary creates operational debt rather than observability.
