The Statistical Reality of Coding Agents
For technical decision-makers, leaderboard order can look more conclusive than the underlying comparisons allow. The ExpertCrowd-SWE study, an author-reported coding-agent benchmark submitted on November 9, 2026, provides a nuanced look at seven frontier coding agents. Rather than a simple ranking, the study identifies four statistical tiers, highlighting where evidence separates models and where it remains inconclusive.
Four tiers, not seven distinct places
The paper evaluated seven models on 100 expert-authored software-engineering tasks drawn from 87 open-source projects and 11 languages. Each model attempted each task five times. The paper reports 3,495 scored runs. Gemini 3.8 Flash had no valid run on one assigned task, so its score covers 99 tasks. The reported averages are: Opus 5.5 (67.3%), GPT-6 Astra (63.7%), Fable 5.1 (46.5%), Gemini 3.8 Flash (40.0% across 99 tasks), Qwen 3.8 Max (33.5%), DeepSeek V4.1 Flash (32.5%), and Grok 4.6 (32.2%).
These averages form a four-tier pattern. The bottom three models—Qwen, DeepSeek, and Grok—are not statistically separated from one another in pairwise comparisons. Similarly, the top two models, Opus and Astra, are not statistically separated. While Opus leads Astra by 3.6 percentage points, the paired 95% confidence interval ranges from −1.4 to +9.7, with a p-value of 0.20. Because this interval crosses zero, the study does not establish that Opus outperforms Astra on this specific benchmark.
It is critical to distinguish between the two types of uncertainty reported. The intervals printed beside each model’s score (±4.7 for Opus and ±5.5 for Astra) measure task-sampling uncertainty, indicating how the aggregate might vary across a different set of tasks. In contrast, the paired interval measures the difference between two models on the exact same tasks. Keeping these estimates separate avoids treating score order as a settled result.
What the test actually measures
ExpertCrowd-SWE evaluates whether an autonomous agent can implement a short ticket in a sandboxed, offline repository. Tasks involve multi-file changes, and each includes an expert-authored rubric awarding partial credit, with specific blockers and gates for properties a maintainer would require. The authors explicitly state that this benchmark is not a ranking of coding products or overall model quality.
The evaluation controls for task, prompt, container, resource limits, step budget, and grader. However, the common HUD harness exposes each provider’s native tool interface. Consequently, the benchmark compares model-and-interface configurations rather than model weights isolated from repository interaction. The authors identify a single shared-tool comparison as future work.
Furthermore, the task set is intentionally bounded. The paper reports 76 held-out tasks and releases 24; while the public dataset subset is useful for inspection, it does not represent the full 100-task evaluation. The study measures one specific capability: autonomous work on repository changes judged by task-specific criteria. It does not establish performance in production environments.
Why aggregate scores are only a starting point
The inconclusive average comparison between the top two models does not mean they are interchangeable. Their task scores have a moderate correlation of r=0.42; they differ by at least 40 points on 14 tasks, and both earn full credit together on only four. For builders, this implies that choosing between models requires testing representative work rather than relying on average scores.
Effort metrics also provide a distinct perspective. The paper notes that step count is a poor predictor of success. GPT-6 Astra’s median was 33 steps per run, compared with 160 for Opus 5.5 and 166 for Gemini 3.8 Flash. These figures do not suggest that fewer steps cause better results, nor do they equate step count with reasoning quality. Instead, they demonstrate that evaluation plans should track outcome and effort separately, rather than treating longer trajectories as evidence of superior work.
Finally, grading rules influence the results: 7.8% of runs were capped after a blocker failed, and 2.3% were zeroed after a gate failed. For builders, these rules are part of the benchmark’s design. The useful conclusion is bounded: ExpertCrowd-SWE distinguishes four groups on its authored tasks, but leaves the top pair unresolved and reveals substantial task-level variation. Its averages can guide a test plan, but they cannot replace one.