AI Coding Agents
Observability
Coding agent observability that reviewers can verify
September 24, 2026 - 2 minute read
Coding agent observability work often looks harmless because instrumentation sits beside application logic. A changed attribute, missing trace context, or unbounded label can still break dashboards, increase storage costs, or hide the failure a team needs to diagnose. Reviewers need evidence that a telemetry change preserves semantics and remains useful under real traffic.
OpenTelemetry defines signals such as traces, metrics, and logs, with profiles as an emerging signal. Instrumentation should begin with an operational question and select the signal that answers it. Adding all available context to every event creates noise and can expose sensitive data.
Scope coding agent observability work
Start with the behavior that operators need to see. Name the service, code path, failure mode, existing dashboard or alert, and expected signal. Include the project’s naming rules, semantic conventions, sampling policy, and fields that must never enter telemetry.
Ask the coding agent to trace the complete signal path. That path may include the instrumentation call, context propagation, SDK configuration, collector processing, export, storage, and the query used by a dashboard or alert. A unit test around one span proves little when a collector later drops its attributes.
Factory’s remote delegation guidance recommends specifying the outcome, reproduction details, acceptance criteria, and verification commands. For observability changes, attach a representative request or failure and define the query that should expose it.
Control names, cardinality, and sensitive data
Stable names let dashboards and alerts survive refactors. Reuse established semantic conventions and repository helpers before creating a new metric or attribute. Check that units and aggregation types match existing instruments. A counter should represent an event that accumulates, while a histogram should use boundaries suited to the measured distribution.
Review every attribute for cardinality. User IDs, request IDs, raw URLs, exception messages, and unbounded customer values can create a new time series for each observation. Keep those values out of metric labels. Place high-cardinality correlation data in traces or structured logs only when policy permits it.
Telemetry can contain source fragments, prompts, headers, database values, and credentials. Require tests or processor rules that redact sensitive fields. Factory’s Droid Shield scans added lines for potential credentials during Droid-driven commits and pushes, but it does not establish a telemetry privacy policy. Review data collection and export boundaries directly.
Verify behavior across the signal path
Run the changed service with a controlled request that covers success and failure. Capture the emitted signal, then verify the exported form or a local collector query. Confirm parent and child trace relationships, metric units, log severity, error status, and resource attributes. Repeat enough requests to catch duplicate registration or per-request instrument creation.
The pull request should show the operational question, signal definition, before-and-after query, sample output with sensitive values removed, and focused test results. Include expected dashboard or alert changes when names or labels move.
Stop when the destination, retention policy, or allowed data fields are unknown. Instrumentation without a defined consumer and privacy boundary creates operational debt rather than observability.
