AI Coding Agents
Enterprise AI
Context compaction must preserve the next step
September 24, 2026 - 2 minute read
Context compaction reduces a long session to information an agent can carry forward. The engineering test is whether the agent can continue correctly. A short summary that loses the modified file, the failed approach, or the user’s constraint can create more work than it removes.
Factory’s December 2025 compression study evaluated hundreds of compression points over 36,611 messages from opted-in software engineering sessions. It tested recall, artifact tracking, continuation, and decisions using probe questions and a GPT-5.2 judge.
The study’s design makes the intended measurement clear: retained information useful for continuing work. It is vendor-run research about the approaches tested at that time, not a current ranking of every coding product.
Context compaction needs more than a ratio
The published overall scores were 3.70 for Factory, 3.44 for Anthropic, and 3.35 for OpenAI on a five-point scale. Factory’s approach used a persistent structured summary and merged newly truncated material into it. The evaluation associated that approach with stronger retention of technical details.
The same results expose an important limit. Artifact tracking was the weakest dimension for all three methods. Factory scored 2.45, Anthropic 2.33, and OpenAI 2.19. A session that produces a fluent summary can still lose track of which files changed.
Treat the working tree, patch, and test output as independent evidence. Before resuming a consequential edit, reconcile the summary with the repository. Avoid asking the agent to reconstruct exact file state entirely from memory.
Compression ratio belongs beside continuation quality in an evaluation. Reducing the next request does not establish lower total cost if the agent must reread files, repeat failed attempts, or ask the engineer to explain an earlier decision again.
Context compaction should survive real working sessions
Chainguard’s case study provides a complementary customer account. One Droid session lasted two weeks, spanned six repositories, and built 80 packages. Josh Wolf described treating Droid like a colleague that remembers the ongoing discussion. The story shows why continuity matters in sustained work, but it does not measure token savings or isolate compaction as the sole cause.
Use a representative task that naturally crosses a context boundary. Before compaction, record the task constraint, files modified, validation status, rejected approach, and next action. Afterward, ask the agent to continue without restating those facts. Check the resulting work against the record.
Include a correction in the test. For example, a reviewer may reject an earlier approach after the agent has already summarized it. Verify that the new instruction replaces the stale decision instead of living beside it ambiguously.
Factory’s telemetry reference documents usage measurements that can support the cost side of an evaluation. Pair those measurements with successful completion, repeated reads, and human repair. Token counts without outcomes cannot show whether the saved context was useful.
A good compaction result leaves the agent able to take the next correct step with less reconstruction. That is a more demanding target than generating the smallest possible summary.
