AI agent telemetry inside private networks

AI agent telemetry should answer a specific operational question. A team investigating slow validation may need execution timing and CI results. It may have no reason to retain the source code or complete conversation behind the task.

Metrics summarize activity and duration. Traces connect operations within a request or turn. Message content contains the actual prompts, responses, and tool inputs or results, and can be attached to spans only when the relevant export is enabled. Decide which of those answers the operational question before collecting more detail.

AI agent telemetry separates metrics from content

Factory's telemetry documentation describes metrics-only customer export by default. Optional message-content spans go to a customer-configured collector. The hosted Analytics API is a separate source.

Its privacy reference shows where each data type can go.

Raw content export is off by default and has no automatic redaction. If enabled, it may include prompts, file contents, command arguments, and tool results. Approve the collector for those data classes before collecting them.

Set AI agent telemetry policy centrally

Organization-managed settings can pin the customer collector and content policy. Project, folder, and user copies of the telemetry block are ignored. Verify the effective configuration on the actual execution account.

Keep collector credentials out of broadly readable settings. Factory supports environment-variable references so the actual credential can arrive through the managed environment.

Test a missing credential and an unreachable collector. A dashboard receiving data from some machines may conceal another runner that never exported successfully.

Review downstream destinations too. A customer-operated collector can forward information to another service. Its storage, access, and retention policy belong in the same boundary review.

Choose identity detail deliberately

Aggregate granularity removes specified user and change identifiers and prevents message-content export. Some session, model, and workload dimensions remain. Review those fields before calling the dataset anonymous.

The export format affects trace availability. Factory documents no spans for the legacy format in aggregate mode, while the GenAI format retains trace structure with content attributes removed.

Inspect a sample from a non-sensitive task. Confirm that it matches the approved schema and use that sample as a check when changing collector transformations.

In an airgap, only the customer collector exists. Without a usable endpoint, the pipeline does not run. Plan an internal way to detect missing evidence rather than assuming a hosted fallback.

Connect monitoring to accepted work

Tool counts establish activity, not whether a change was correct. Use CI results, source-control history, and reviewer approval to establish the delivered outcome.

The customer OTEL metric set does not include token and cost data. Obtain inference cost from an appropriate provider, gateway, or infrastructure record. Label allocation assumptions when reporting estimates.

Droid exports metrics with delta temporality: each export describes activity since the previous collection interval. For example, two intervals containing three and five tool calls represent eight calls in total. Sum the interval values over the reporting window rather than treating the latest value as a lifetime total.

For the slow-validation investigation, join tool timing with the CI run and repository revision. That can distinguish a long-running test from repeated failed attempts without retaining the code or conversation. Keep the measurement definition stable when comparing configurations.

The result should be enough evidence to operate and review the workflow, with no unnecessary expansion of the sensitive data retained.

Further reading

• AI code data residency
• SOC 2 for AI coding tools
• Telemetry data reference