How to catch telemetry schema breaks in CI

To check a telemetry schema, keep a typed allowlist of signal and attribute names next to the instrumentation, capture the telemetry a deterministic test request emits, and fail CI on unknown attributes, changed types, missing required fields, unbounded values, or sensitive patterns. Build on OpenTelemetry semantic conventions and version the local schema with the code.

Telemetry schema checks verify the names, types, and allowed values emitted by changed code. Coding agent changes can rename a span attribute, attach an unbounded identifier to a metric, or place sensitive content in a log while the application remains functionally correct. Downstream alerts and dashboards may fail only after deployment.

OpenTelemetry publishes semantic conventions for common operations and attributes across traces, metrics, logs, profiles, and resources. Those conventions provide a stable starting point. Each organization still needs a local contract for domain attributes, cardinality, privacy, and versioning.

Define telemetry schema checks as code

Keep an allowlist or typed schema near the instrumentation library. Record each signal name, attribute name, value type, unit, and whether the value may contain customer data. Mark attributes used for aggregation so a reviewer can spot a new high-cardinality value.

Capture telemetry from a deterministic test request and validate it before export. The test should fail on an unknown attribute, a changed type, a missing required field, or a forbidden value pattern. Normalize timestamps and generated identifiers before snapshot comparison so ordinary variation does not hide meaningful changes.

Most telemetry breaks fall into a few categories, and each has a direct check.

Test errors as well as success. Exception paths often include raw messages, request parameters, or stack details that the normal path never emits.

Version the local schema with the application. A pull request should make a telemetry contract change visible beside the code that produces it, rather than relying on a dashboard failure to reveal the difference.

Protect cardinality and sensitive values

Metrics need bounded dimensions. Route templates, result categories, and service names can be reasonable attributes. Raw URLs, user IDs, trace IDs, and error messages can create one series per request. Assert an allowed vocabulary where the domain is finite, and reject values that match request-specific patterns.

OpenTelemetry’s sensitive data guidance recommends controls at instrumentation, collection, and backend layers. Test the earliest practical control. Collector redaction can provide defense in depth, but it should not excuse emitting secrets from application code.

Use synthetic values that resemble sensitive formats without copying real credentials or customer records.

Give a coding agent explicit telemetry checks

Name the changed operation, expected signals, schema file, and focused test command. Require the agent to compare emitted data before and after the change. A new attribute needs a stated consumer and cardinality bound.

Review telemetry schema evidence

A useful CI report lists added, removed, and changed fields with sample redacted values. It should also state which workload produced each signal. Passing counts alone do not show whether a dashboard dependency changed.

Coordinate intentional breaking changes with alert, dashboard, and retention owners. Keep compatibility aliases only for a defined migration period. Once consumers move, remove the old field and its tests so the schema remains understandable.

How Factory flags instrumentation changes

Factory is a platform for running a software factory, where agents called Droids take work from a signal to a reviewed pull request. For telemetry, Factory's job is to run the schema check where the Droid works and to make instrumentation changes visible in review.

Review with telemetry rules. Factory's automated code review picks up a review-guidelines skill from .factory/skills/review-guidelines/SKILL.md and injects it into every review run. A team can require a finding when instrumentation changes without a schema update, fixture coverage, or privacy analysis.

Check after every edit. A PostToolUse hook runs right after a tool completes, and its documented uses include running validation after an edit. A hook that runs the schema test when instrumentation files change gives the Droid the failure while it still has context.

Keep the command in one place. AGENTS.md holds the project's run and verify commands, so the schema file and focused test command are the first things the Droid reads.

Frequently asked questions

What is a telemetry schema check?

A telemetry schema check validates the names, types, units, and allowed values of the spans, metrics, and logs that code emits, and fails CI when they drift from a versioned local schema.

Why do metrics need bounded dimensions?

Each unique combination of attribute values creates a new series. Route templates and result categories stay bounded, while raw URLs, user IDs, and error messages can create one series per request.

Where should sensitive telemetry be removed?

OpenTelemetry recommends controls at instrumentation, collection, and backend layers. Test the earliest practical control, and treat collector redaction as defense in depth.

How does Factory catch telemetry schema changes?

Factory automated code review applies a review-guidelines skill to every pull request, which can require a schema update with any instrumentation change. A PostToolUse hook can run the schema test after each edit.

Further reading

• Log redaction
• Coding agent observability
• AI agent telemetry inside private networks
• API rate limit tests
• OpenTelemetry semantic conventions