Introducing Agent Effectiveness

Agent Effectiveness is now generally available in Factory Analytics for Enterprise organizations. It connects Factory sessions to cycle time, work intent, and shipped artifacts, so engineering leaders can see where delivery is moving faster, what Droids are working on, and how Factory usage maps to shipped work.

The gap between usage and impact

Engineering output increases in scale and velocity with Droid. Proving how much faster, where the change is concentrated, and what caused it is a different problem.

Engineering leaders are spending more on AI, but most still measure the return with a mix of usage dashboards, delivery metrics, and surveys. Those signals leave a basic question unanswered: what is our AI spend producing?

Agent Effectiveness is a key part of Factory's answer. It connects Droid usage to delivery outcomes, giving leaders an operational view of AI ROI: how much faster teams ship, where gains are concentrated, and whether added capacity is reaching priority work.

Three questions engineering leaders need answered

Agent Effectiveness reads from the project management, issue-tracking, and source control tools you already use, then links Factory sessions to the work those systems track.

It turns that data into answers to three questions.

Are we shipping faster?

The Throughput view shows how project, issue, and pull request cycle times change alongside Factory usage, and where those changes are concentrated. You can compare usage across projects, pods, and users against your targets for the quarter.

Are Droids working on the right priorities?

The Output view maps spend to intent. Session intents classify work as feature engineering, maintenance, bug fixing, or exploration. You can compare the resulting split of engineer time and Factory Standard Credits against your plan. If the mix drifts, you can correct it during the quarter instead of explaining it afterward.

What was the output?

The Attribution view shows output by project, issue, session, and user. Using Local Signals (described below), you can trace a session back to the specific issues, projects, and artifacts a user worked on. Organization-level spend is no longer an abstract number. It is tied to concrete work.

How Local Signals works

Local Signals is the classification layer behind Agent Effectiveness. When an organization turns on Advanced Analytics, Factory runs a small set of rubrics over windows of each session. Factory will refine, add, and retire rubrics over time.

• Intent and completion records what the session set out to do (feature work, bug fixing, maintenance, or exploration), whether it got there, and which pull requests, tickets, and documents it worked on. It powers the Output and Attribution views. Organization admins see it in Advanced Analytics.
• Friction records moments that made a session worse: errors that blocked progress, the agent going in circles, the user repeating themselves or undoing the agent's work, or visible frustration. The normal write-test-fix loop and one-off retries don't count. Each moment gets a severity and a cause, such as model provider errors, permission gates, tool bugs, billing limits, the user's environment, the agent's own behavior, or third-party outages. Friction exists to find where the product, the model, or the environment gets in the way. Customers see it in Software Factory as counts and severities per workstream run.
• Expertise estimates how familiar the user was with the task in that window, and how many planning and execution decisions the user made versus the agent. It is adapted from Anthropic's research on agentic coding and expertise, and it deliberately leaves out occupation and dollar-value estimates. Expertise is not shown in any customer UI or API.

How to enable it

Agent Effectiveness relies on connecting Factory to the tools where your work lives and enabling the Advanced Analytics enterprise control. Setup is an admin task and takes a few minutes.

1. Connect your organization integrations

Configure your organization-level integrations for the tools you use: Jira and Linear for issue tracking and project management, GitHub and GitLab for source control.

These integrations let Factory map sessions to issues, projects, and pull requests. Attribution and the Output view only cover the systems you connect, so connect every tool your teams track work in.

2. Turn on Advanced Analytics

Turn on the Advanced Analytics enterprise control to make the Throughput, Output, and Attribution views available to organization admins. Advanced Analytics enables Local Signals, which classifies sessions and links them to related work and artifacts. It also enables analysis of eligible session history that was already synced to Factory.

Once this is set, the dashboards populate on their own. There is no per-repository configuration. Full setup instructions are in the Agent Effectiveness documentation.

Backfilling history

Backfill rebuilds effectiveness data from session history that was already synced to Factory, back to the start of the account. It only runs after an admin turns on Advanced Analytics, and only for organizations that:

• Are on an Enterprise plan
• Allow Factory-Managed Inference
• Use a supported inference region
• Have session sync on
• Don't use customer-managed encryption keys

Backfilled sessions go through the same Local Signals rubrics as new sessions.

Privacy and controls

Agent Effectiveness is opt-in. It is off by default and available on Enterprise plans only. Organization admins can turn it on. For organizations that haven't enabled it, Local Signals doesn't run.

For Factory-managed Local Signals, new classification requests require cloud sync to be on at the organization level. Classifier selection is checked against your organization's effective model policy, and it may use a different model from your chat session.

Custom (BYOK) classifiers use a separate inference route and are checked against the applicable custom-model and endpoint permissions.

Under our enterprise agreements, customer data isn't used to train models.

From adoption metrics to operating decisions

Measuring adoption is easy. Measuring impact is what changes decisions.

Agent Effectiveness gives engineering leaders the evidence to direct resources toward priority work, see where Droids improve delivery, and correct course while there is still time to act. Autonomous engineering should be measured by outcomes, not logins, session counts, or survey estimates.

Agent Effectiveness is now available for Enterprise organizations. To get started, contact your Factory account team.