HR Briefing

HR Technology

Benchmark gains did not convert to output in Claude model study

A Harvard study of 297 developers found a 1.4-point merge-rate gain from a higher-scoring AI model — statistically insignificant evidence that benchmarks predict output.

Downgrading From Claude Fable Barely Affected Dev Productivity During Ban
Downgrading From Claude Fable Barely Affected Dev Productivity During Bankonkapo / pixabay

Tom Whitfield

· 4 min read

A Harvard University study found that access to a higher-scoring AI coding model raised the proportion of accepted code changes by just 1.4 percentage points — a difference the researchers call statistically insignificant. The finding, published as 'Higher-Scoring Coding Models Did Not Get More Work Done', challenges a core assumption behind how employers buy AI tooling: that better benchmark scores deliver better work.

The study exploited an unusual natural experiment. A 19-day White House ban on Claude Fable, imposed in June, forced developers submitting pull requests to GitHub to downgrade to less capable versions of Claude. Because all Claude versions co-sign pull requests, and that signature is searchable through the GitHub API, the researchers could track output before, during and after the ban without surveying anyone.

What did the researchers measure?

The main group contained 297 developers who used Fable 5 during its three days of availability before the suspension. A control group of 8,908 developers continued using the older Opus 4.8 model with no recorded use of Fable 5. Pull requests were divided into four periods: before Fable 5's June 9 release; its three days of availability; the 19-day suspension; and the period after access returned on July 1.

The primary metric was the proportion of submitted code changes accepted into GitHub projects within seven days. Additional checks covered PRs merged within an hour, PRs closed without merging, comment counts, subsequently reversed changes, and failures in GitHub's automated checks. Data was collected between November 2025 and October 2026, with at least four weeks allowed for the latest submissions to be merged.

Fable 5 had scored nearly 11 points higher than Opus 4.8 on the SWE-Bench Pro coding benchmark. That advantage did not survive contact with actual work.

What did the numbers show?

The headline results were small and, in most cases, statistically indistinguishable from noise:

  • Access to Fable 5 changed the seven-day merge rate by only 1.4 percentage points, a change the paper deems not statistically significant.
  • Across five of the eight Claude model upgrades in 2026, covering 1,208 developers and more than 15,000 code submissions, the merge rate did not rise; it fell slightly, by an estimated 0.9 percentage points.
  • Developers on newer models completed 2.4% more accepted submissions per week — too small to establish a reliable improvement.
  • Even where benchmark scores rose by between 5 and 18 points, no significant productivity gains appeared.
  • Failure rates in GitHub's automated code checks remained essentially unchanged.

The paper states plainly: 'Results show that having access to Fable 5 changed the merge rate by only 1.4 percentage points, and the change was not statistically significant.' It adds: 'These results show that gains on current coding benchmark did not really translate into more finished work.'

What are the study's limits?

The researchers acknowledge several problems, and they matter for any HR or procurement leader quoting this paper. GitHub merges are an imperfect measure of productivity. Early adopters of new models may simply be more experienced developers, so the model and the person cannot be cleanly separated. Users may also assign harder tasks to stronger models, which take longer and are less likely to merge — making the strongest model look less productive on almost any measure.

The paper also contends that prior studies were skewed because most code changes came from a small number of projects, each with its own rules for accepting contributions. A related structural problem affects the whole field: benchmark test data tends to enter the training data of the next model, making older tests meaningless. The authors' own cited example, a vision-model challenge involving a pelican on a bicycle, is likely compromised because its data now appears in training sets.

Why does the benchmark gap persist?

The study's central hypothesis is restrained: 'The main reason Fable 5's gains on benchmarks did not translate to real-work productivity is that the tasks developers give agents are already within what the older model can do.' In plain words, everyday tasks may not need the newest model.

The authors warn: 'If regulators and buyers judge how much stronger a model is by its benchmark, the difference they rely on will likely not show up in user productivity.' For employers negotiating AI tooling contracts, that is the operative sentence. The findings suggest developers optimize to deadlines regardless of the tool available — so paying for the top tier may buy capability that routine work never exercises.

Source: unite.ai

Share this article:

More from Tom Whitfield

Tom Whitfield

Show full bio

Market editor covering money, funding and market shifts around Recruiting and talent acquisition, learning and development at HR Briefing.

287 articles

© 2026 HR Briefing. All rights reserved.