Coding agents fix more code when tests stay frozen
ExecCritic decouples test generation from code repair with a fail-closed verification lock, raising agent bug resolution on SWE-bench Verified to 72.6%.
When an autonomous coding agent attempts to fix a repository bug, standard product architectures grant it full autonomy: inspect the issue, edit files, write tests, and submit when local commands succeed. This workflow conceals a dangerous structural failure. When a single model trajectory writes both the source patch and the validating test, the test inherits the exact same blind spots as the code. A flawed repair passes a flawed test, giving engineering teams false confidence until an edge case triggers an outage in production.
A research team from Microsoft Research and the University of Wisconsin–Madison has introduced a framework called ExecCritic 1, demonstrating that coding agents fix complex repository bugs more reliably when test creation and source repair are assigned to separate models with partitioned permissions. By isolating test construction in a specialized agent, qualifying the test in a fail-closed sandbox, and freezing the test before code revision begins, the system achieves a 72.6% resolved rate on SWE-bench Verified. For product managers building developer tools and automated maintenance agents, this architecture establishes a practical blueprint for replacing self-grading agent loops with verifiable engineering gates.
What changed: Decoupling test creation from code repair
ExecCritic restructures the repository repair loop into two distinct phases: Learn to Test and Test to Improve 1. The system divides the workload across specialized roles and enforces asymmetric write permissions across the codebase.
The execution protocol proceeds in strict sequence:
- Independent test generation: The Test agent inspects the natural-language issue description and the base repository. It produces a structured test bundle containing a repository-native test patch, the exact command to execute it, and a JSON behavior contract specifying target assertions.
- Fail-closed qualification: An execution harness runs the test bundle against a clean checkout of the buggy repository. To qualify, the test must execute cleanly and exit with a verified failure on the unpatched codebase. This failure confirms that the test actively detects the defect instead of passing vacuously. When the Test agent exhausts its five generation attempts, the harness marks a qualification failure, bypasses iterative revision, and submits the baseline fix directly to conserve compute.
- Frozen verification lock: Once a test qualifies, the harness freezes all test files in the workspace. The Repair agent receives the issue, the repository, and the failure trace from the frozen test.
- Source-only revision: The Repair agent holds write permissions exclusively for production source files. Filesystem locks enforce strict read-only protection across all test files, guaranteeing that assertions remain immutable throughout repair. When a candidate patch passes the frozen test, the controller terminates the loop and submits the pull request. If the test fails, stdout and stderr return to the Repair agent for up to five bounded revision cycles.
The underlying infrastructure is open-sourced under Microsoft Research's Orchard agent framework 2.
The evidence: When test feedback helps and when it hurts
The central empirical discovery of ExecCritic is that adding execution feedback to a coding agent is double-edged: feedback improves code only when tests accurately capture the intended behavior 1.
When the authors evaluated an off-the-shelf Qwen-3.5-35B-A3B Repair agent on SWE-bench Verified, the baseline model resolved 61.2% of tasks in a standalone single-turn pass. Providing the agent with tests generated by an untrained Qwen Test agent caused task resolution to fall to 57.3%—a 3.9 percentage point drop. The flawed tests encoded incorrect requirements, actively misleading the repair agent into breaking valid code. When the authors replaced the Qwen tests with tests generated by GPT-5.6-sol, resolution rose to 65.3% (+4.1 points). When supplied with privileged human Oracle tests, resolution reached 69.4% (+8.2 points). An unreliable test behaves as an active distractor.
To eliminate this bottleneck using open-weight infrastructure, the researchers developed a role-specific post-training curriculum using supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO):
- Test agent training: SFT on 5,000 reasoning trajectories raised the Qwen Test agent's Base-to-Gold accuracy (failing on buggy code while passing on reference fixes) from 22.2% to 39.6%. On-policy reinforcement learning rewarding balanced accuracy across candidate patches further boosted validity to 62.2%, matching specialized models like Codex-5.3 (61.0%).
- Repair agent training: Multi-turn RL training with a direct-solve bonus increased initial Round-0 repair resolution from 61.2% to 68.3% (+7.1 points).
- Composed performance: Composing the trained Test agent with the trained Repair agent achieved 72.6% resolution on SWE-bench Verified. This represents an 11.4 percentage point gain over the original baseline and a 4.3 percentage point improvement over the trained model's own initial fixes. Across the trajectories that entered iterative revision, the agent required an average of only 13 additional turns to converge.
As practitioner tracking on X noted 3, these gains reflect an expanded system workflow with dedicated test generation and revision passes. The authors document three critical boundaries:
- Single-behavior versus multi-case scope: On SWE-bench Pro, where issues require satisfying an average of 14.43 test cases compared to 3.03 on SWE-bench Verified, generated test bundles raised GPT-5.6 resolution by only 0.7 percentage points (61.6% to 62.3%), even though Oracle tests unlocked an 11.8 percentage point gain (to 73.4%). A single test bundle effectively validates focused bug repairs, whereas complex multi-feature tasks require generating comprehensive test matrices.
- Language dependencies: While the Python-trained Test agent transferred well to Rust (66.7% Base-to-Gold success) and C++ (58.3%), performance dropped on C (26.1%) and collapsed on Java (2.4%), where complex enterprise test harnesses resisted zero-shot generation.
- Harness fidelity: The qualification gate verifies execution status and failure on base checkouts, yet semantic alignment with human intent still depends on the Test agent's reasoning capabilities.
Production reality check: Beyond green test suites
The gap between passing benchmark assertions and deploying durable production code is a recognized challenge among AI engineering teams. As Runway co-founder Cristóbal Valenzuela observed 4, benchmark success often conceals destructive side effects and unhandled edge cases in real execution environments.
Enterprise CI/CD implementations mirror this tension. In an engineering analysis of deployment-stage evaluation gates 5, Harness highlighted that standard unit passes often mask severe functional regressions. In production pilots, agents achieved perfect relevance scores of 1.0 while failing task completion (scoring 0.3) due to hallucinated parameters, requiring automated pipeline gates that block deployment whenever task completion falls below 70%.
Similarly, architectural analyses by developer channel Boundary 6 demonstrate that self-testing models frequently craft vacuous assertions that confirm runtime execution while skipping boundary contracts. Their findings emphasize that independent verifiers and immutable test boundaries are essential for sustainable agent-authored codebases. While this explainer provides general systems context rather than an evaluation of ExecCritic, its architectural conclusions directly support separating test authors from code authors.
How to implement now: The 4-week PM pilot
For product teams deploying autonomous coding assistants, internal PR bots, or automated issue resolvers, ExecCritic provides an actionable pattern: replace monolithic agent prompts with a two-role pipeline governed by strict workspace write boundaries.
The deployment unit
Structure the production repair loop around six coordinated modules:
| Component | Responsibility | Permissions & Constraints | Acceptance Gate |
|---|---|---|---|
| Context Assembler | Extracts issue text, call sites, and recent commits into an isolated container | Read-only access to repository | Workspace mounts cleanly |
| Test Agent | Generates standalone test file, execution command, and assertion contract | Write access restricted to test directory | Produces valid syntax and explicit assertions |
| Qualification Harness | Executes generated test against unpatched base checkout | Runs inside ephemeral container with 90s timeout | Test exits with clean non-zero error code (Base Failure) |
| Repair Agent | Analyzes error trace and produces candidate source fix | Write access restricted to production source files; test files locked | Modifies valid source files; zero edits to test suite |
| Iteration Controller | Runs candidate patch against frozen test; collects stderr | Maximum 5 revision rounds of 40 turns | Candidate patch passes frozen test suite |
| Verification Gate | Executes full regression suite before opening human PR | Full container access | Clean run across entire existing test suite |
4-week pilot rollout plan
- Week 1: Baseline audit & repository selection. Identify one internal microservice or library with high test coverage, deterministic test runtimes under 60 seconds, and active issue backlog. Benchmark your current agent workflow on 30 historical bug tickets. Record baseline resolution rate, developer review time, and the frequency of regressions discovered during PR review.
- Week 2: Sandbox isolation & permission gates. Build the ephemeral execution harness using Docker or lightweight microVMs. Implement strict Linux filesystem write permissions: lock test directories as read-only during the repair phase. Build the fail-closed qualification gate requiring clean failure on base checkouts.
- Week 3: Shadow evaluation. Run incoming bug tickets in parallel through your existing coding pipeline and the new two-role frozen-test harness. Store candidate patches in evaluation databases before any branch creation. Have senior engineers conduct blind reviews assessing patch correctness, test validity, and presence of unhandled edge cases.
- Week 4: Production pull request rollout. Connect the frozen-test harness to live repository webhooks. Configure the pipeline to open pull requests with attached telemetry: the qualified test diff, the source patch, the iteration count, and execution logs. Require manual developer approval before merging.
Pilot measurement scorecard
Track four core metrics to govern rollout decisions:
| Metric | Target Threshold | Minimum Viable Gate | Rollback Trigger |
|---|---|---|---|
| Verified Task Resolution | ≥ 65% of scoped bug tickets | ≥ 55% | < 45% (below single-agent baseline) |
| Base-to-Gold Test Validity | ≥ 60% of generated tests | ≥ 50% | < 35% (tests fail to isolate bugs) |
| Developer PR Acceptance | ≥ 70% accepted on first review | ≥ 50% | < 40% (high developer rework) |
| Compute Overhead | ≤ 20 additional turns per ticket | ≤ 30 turns | > 40 turns with flat resolution |
Urgency and action window
- Immediate priority (current quarter): Teams deploying automated PR-generation bots, bug-triage auto-fixers, or enterprise coding agents that modify shared production codebases. The risk of unverified self-generated tests shipping broken logic to production makes permission separation an urgent architectural safeguard.
- Monitor and defer: Teams building inline code autocomplete, conversational doc search, or read-only developer explainers. In these contexts, execution sandboxes are absent and immediate single-token latency outweighs multi-round verification benefits.
References
- 1
- 2
- 3
- 4
- 5
- 6
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.
Related content
More from this channel
- Frontier models are stalling on legacy prompts: why agent teams are moving to enforced execution
- Multimodal prompt injection triggers unsafe agent actions ten times more often than completed attacks show
- Agent payment protocols verify the transaction while leaving the purchasing decision unconstrained
- The same model weights score 26% on one serving route and 62% on another
- A model upgrade can break agent memory before the API does
- The next computer-use agent will choose between the screen and the shell
- GPT-6 Astra puts computer use on the product roadmap. The hard part is still the commit.
- Terminal-Universe Turns Frozen Agent Traces into Reusable Coding Environments