AI Coding Agents
Observability
Telemetry schema checks for coding agent changes
September 28, 2026 - 2 minute read
Telemetry schema checks verify the names, types, and allowed values emitted by changed code. Coding agent changes can rename a span attribute, attach an unbounded identifier to a metric, or place sensitive content in a log while the application remains functionally correct. Downstream alerts and dashboards may fail only after deployment.
OpenTelemetry publishes semantic conventions for common operations and attributes across traces, metrics, logs, profiles, and resources. Those conventions provide a stable starting point. Each organization still needs a local contract for domain attributes, cardinality, privacy, and versioning.
Define telemetry schema checks as code
Keep an allowlist or typed schema near the instrumentation library. Record each signal name, attribute name, value type, unit, and whether the value may contain customer data. Mark attributes used for aggregation so a reviewer can spot a new high-cardinality value.
Capture telemetry from a deterministic test request and validate it before export. The test should fail on an unknown attribute, a changed type, a missing required field, or a forbidden value pattern. Normalize timestamps and generated identifiers before snapshot comparison so ordinary variation does not hide meaningful changes.
Test errors as well as success. Exception paths often include raw messages, request parameters, or stack details that the normal path never emits.
Version the local schema with the application. A pull request should make a telemetry contract change visible beside the code that produces it, rather than relying on a dashboard failure to reveal the difference.
Protect cardinality and sensitive values
Metrics need bounded dimensions. Route templates, result categories, and service names can be reasonable attributes. Raw URLs, user IDs, trace IDs, and error messages can create one series per request. Assert an allowed vocabulary where the domain is finite, and reject values that match request-specific patterns.
OpenTelemetry’s sensitive data guidance recommends controls at instrumentation, collection, and backend layers. Test the earliest practical control. Collector redaction can provide defense in depth, but it should not excuse emitting secrets from application code.
Use synthetic values that resemble sensitive formats without copying real credentials or customer records.
Give a coding agent explicit telemetry checks
Name the changed operation, expected signals, schema file, and focused test command. Require the agent to compare emitted data before and after the change. A new attribute needs a stated consumer and cardinality bound.
Factory’s Automated Code Review supports custom repository guidance. Teams can require a review finding when instrumentation changes without schema updates, fixture coverage, or privacy analysis. Review should cover generated instrumentation and collector configuration when both change in one pull request.
Review telemetry schema evidence
A useful CI report lists added, removed, and changed fields with sample redacted values. It should also state which workload produced each signal. Passing counts alone do not show whether a dashboard dependency changed.
Coordinate intentional breaking changes with alert, dashboard, and retention owners. Keep compatibility aliases only for a defined migration period. Once consumers move, remove the old field and its tests so the schema remains understandable.
