Dataset·Ledger

AI Productivity Evidence

Name: AI Productivity Evidence
Creator: Hermosa AI
Published: 2026-10-17
License: https://creativecommons.org/licenses/by/4.0/

Everyone quotes a number for what AI does to software production, and almost nobody quotes the denominator. This ledger holds one row per study, tiered by what its method can actually support, with the population, the metric, the quantity it divides by, the effect as published, and who paid for it. Browse it below, or take the raw files.

Snapshot: 2026-10-17
Coverage: 14studies·9independent of the seller·4trials and experiments·4null or mixed results
Download

About this dataset

[show]

Unit: One row = one study, measurement or audit of AI-assisted software production, with its population, sample size, metric, denominator, finding, effect, data availability and conflict of interest.
Evidence tier: Randomised trial, field experiment, independent measurement, platform telemetry, audit, survey, vendor study, or corporate self-report. The tier is the point: a trial and a vendor survey can report the same number and mean entirely different things. A practitioner survey stays a survey however large its sample.
Denominator: Required on every row. Percent of code written, percent of pull requests opened by an agent, and percent of merged pull requests authored by an agent are different quantities that are routinely reported as if interchangeable. A figure whose denominator cannot be established from the primary document is not stored.
Direction: How the measured metric moved, not whether the outcome was good. A study finding that completion time rose is recorded as a rise, and what that means for productivity is stated in the finding.
Editions: Annual instruments are kept as separate rows per edition, never overwritten, because the changes between editions carry the information. Where question wording changed, the caveat says so, so nobody plots a false time series.
Sources: Each row is verified against the primary document: the paper, the report, the dataset card or the audit itself, not a secondary summary. Where only a snippet was reachable, the row says so.
Caveat: Self-report is weak evidence here. The one randomised trial in the ledger measured developers believing they were faster while being measurably slower, so treat every survey-derived productivity figure with that in mind.
License: Free to use, distribute, and reproduce with attribution to Hermosa Labs LLC (CC BY 4.0).
Cite: Hermosa Labs LLC, “AI Productivity Evidence,” 2026-10-17.

The ledger

View

Evidence tier

Direction

Run by

14 of 14 studies

AI Productivity Evidence carries no numeric fields -- showing its fact columns instead, and no median footer (a median of text is not a median).

#
1	60 million Copilot code reviews and countingGitHub [Metric rose]	2026-03-05	GitHub	?	?
2	A year of AI disclosure in critical packagesAndrew Nesbitt (critical-ai-scan) [Metric rose]	2026-08-06	Andrew Nesbitt (critical-ai-scan)	589,798 non-merge commits in the year to 2026-07-29	?
3	AIDev: a dataset of agent-authored pull requests on GitHub (live version v4)SAIL Lab, Queen's University (Hao Li, Haoxiang Zhang, Ahmed E. Hassan) [Metric rose]	2026-02	SAIL Lab, Queen's University (Hao Li, Haoxiang Zhang, Ahmed E. Hassan)	Live v4 (AIDev-2.7M, cutoff Nov 2025): 2,743,854 PRs across 327,477 repositories and 159,056 human developers. Paper edition (arXiv:2602.09185): 932,791 PRs, 116,211 repositories, 72,189 developers, five agents, cutoff 2025-08-01.	?
4	Echoes of AI: Investigating the Downstream Effects of AI Assistants on Software MaintainabilityBorg (CodeScene; Lund University), Hewett, Graham and Kini (Equal Experts), Farley (Continuous Delivery), Hagatulah, Couderc and Soderberg (Lund University); ICSME 2025 registered report, arXiv preprint [Null result]	?	Borg (CodeScene; Lund University), Hewett, Graham and Kini (Equal Experts), Farley (Continuous Delivery), Hagatulah, Couderc and Soderberg (Lund University); ICSME 2025 registered report, arXiv preprint	151 participants in total. Phase 1: 39 valid AI-developed solutions and 37 valid non-AI solutions. Phase 2 (the randomised part): 75 valid solutions (40 evolving AI-developed code, 35 evolving non-AI code) out of 225 assigned.	2025-07-01 (arXiv v1; v3 revised 2026-02-26)
5	GitHub Octoverse 2025GitHub [Metric rose]	2025-10-28	GitHub	180 million-plus developers on the platform (a population count, not a sample)	?
6	GitLab on migrating its own rate-limiting system with AI agents: 17 of 121 keys missed and a production incidentGitLab (Sam Wiskow, GitLab blog) [Mixed]	2026-07-08	GitLab (Sam Wiskow, GitLab blog)	121 rate limit keys in Gitlab::ApplicationRateLimiter; 14 numbered specs; 'somewhere north of 30' merge requests	?
7	IT Systems Annual Assessment: software factory use among major DOD business programsUnited States Government Accountability Office [Metric rose]	2026-09	United States Government Accountability Office	10 programs (2026); 11 (2025); 10 (2024)	?
8	Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer ProductivityMETR (Model Evaluation & Threat Research); Becker, Rush, Barnes, Rein [Metric rose]	2025-07-10	METR (Model Evaluation & Threat Research); Becker, Rush, Barnes, Rein	16 developers; 246 issues (136 randomised to AI-allowed, 110 to AI-disallowed); about 2.0 hours per issue on average	?
9	PRarena: public GitHub pull requests attributable to six coding agents (data.csv snapshot 2026-10-17)Albert Avetisian (aavetis/PRarena) [Metric rose]	2026-10-17	Albert Avetisian (aavetis/PRarena)	Last observation 2026-10-17 15:27:03 (first observation 2025-05-26 14:21:34). Cumulative counts at that observation: 12,590,460 agent PRs, 10,205,344 merged. The brief states the file holds 2,433 observations; not independently counted here.	?
10	Stack Overflow Developer Survey 2025: AI sectionStack Overflow [Mixed]	?	Stack Overflow	49,000+ responses from 177 countries overall; the AI usage question had 33,662 responses (26,004 professional developers); the trust question about 33,000	2025
11	The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software DevelopersCui (Princeton), Demirer (MIT), Jaffe (Microsoft), Musolff (Wharton), Peng (Microsoft), Salz (MIT); preprint on SSRN and Microsoft Research, listed under Management Science DOI 10.1287/mnsc.2025.00535 (journal page not read) [Metric rose]	?	Cui (Princeton), Demirer (MIT), Jaffe (Microsoft), Musolff (Wharton), Peng (Microsoft), Salz (MIT); preprint on SSRN and Microsoft Research, listed under Management Science DOI 10.1287/mnsc.2025.00535 (journal page not read)	4,867 developers pooled per the abstract; enrolled per Section 2.2: 1,746 at Microsoft, 320 at Accenture, 3,054 at the anonymous company (these sum to 5,120, and the draft I read does not reconcile the two totals)	2025
12	Thoughts On A Month With DevinAnswer.AI (Hamel Husain, Isaac Flath, Johno Whitaker) [Level, no comparator]	2025-01-08	Answer.AI (Hamel Husain, Isaac Flath, Johno Whitaker)	20 tasks	?
13	We are Changing our Developer Productivity Experiment Design (METR follow-up on late-2025 tools)METR (Model Evaluation & Threat Research) [Null result]	2026-02-24	METR (Model Evaluation & Threat Research)	57 developers, 143 repositories, 800+ tasks (AI-allowed versus AI-disallowed split not given on the page)	?
14	Zalando: commit-level cyclomatic complexity inflects upward as coding agents arriveZalando SE (Bartosz Ocytko, Executive Principal Engineer, Zalando engineering blog) [Metric rose]	2026-08-14	Zalando SE (Bartosz Ocytko, Executive Principal Engineer, Zalando engineering blog)	?	?

AI Productivity Evidence

Snapshot

2026-10-17

Coverage

14studies·9independent of the seller·4trials and experiments·4null or mixed results

Download

60 million Copilot code reviews and countingGitHub

[Metric rose]

2026-03-05

GitHub

A year of AI disclosure in critical packagesAndrew Nesbitt (critical-ai-scan)

[Metric rose]

2026-08-06

Andrew Nesbitt (critical-ai-scan)

589,798 non-merge commits in the year to 2026-07-29

AIDev: a dataset of agent-authored pull requests on GitHub (live version v4)SAIL Lab, Queen's University (Hao Li, Haoxiang Zhang, Ahmed E. Hassan)

[Metric rose]

2026-02

SAIL Lab, Queen's University (Hao Li, Haoxiang Zhang, Ahmed E. Hassan)

Live v4 (AIDev-2.7M, cutoff Nov 2025): 2,743,854 PRs across 327,477 repositories and 159,056 human developers. Paper edition (arXiv:2602.09185): 932,791 PRs, 116,211 repositories, 72,189 developers, five agents, cutoff 2025-08-01.

Echoes of AI: Investigating the Downstream Effects of AI Assistants on Software MaintainabilityBorg (CodeScene; Lund University), Hewett, Graham and Kini (Equal Experts), Farley (Continuous Delivery), Hagatulah, Couderc and Soderberg (Lund University); ICSME 2025 registered report, arXiv preprint

[Null result]

Borg (CodeScene; Lund University), Hewett, Graham and Kini (Equal Experts), Farley (Continuous Delivery), Hagatulah, Couderc and Soderberg (Lund University); ICSME 2025 registered report, arXiv preprint

151 participants in total. Phase 1: 39 valid AI-developed solutions and 37 valid non-AI solutions. Phase 2 (the randomised part): 75 valid solutions (40 evolving AI-developed code, 35 evolving non-AI code) out of 225 assigned.

2025-07-01 (arXiv v1; v3 revised 2026-02-26)

GitHub Octoverse 2025GitHub

[Metric rose]

2025-10-28

GitHub

180 million-plus developers on the platform (a population count, not a sample)

GitLab on migrating its own rate-limiting system with AI agents: 17 of 121 keys missed and a production incidentGitLab (Sam Wiskow, GitLab blog)

[Mixed]

2026-07-08

GitLab (Sam Wiskow, GitLab blog)

121 rate limit keys in Gitlab::ApplicationRateLimiter; 14 numbered specs; 'somewhere north of 30' merge requests

IT Systems Annual Assessment: software factory use among major DOD business programsUnited States Government Accountability Office

[Metric rose]

2026-09

United States Government Accountability Office

10 programs (2026); 11 (2025); 10 (2024)

Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer ProductivityMETR (Model Evaluation & Threat Research); Becker, Rush, Barnes, Rein

[Metric rose]

2025-07-10

METR (Model Evaluation & Threat Research); Becker, Rush, Barnes, Rein

16 developers; 246 issues (136 randomised to AI-allowed, 110 to AI-disallowed); about 2.0 hours per issue on average

PRarena: public GitHub pull requests attributable to six coding agents (data.csv snapshot 2026-10-17)Albert Avetisian (aavetis/PRarena)

[Metric rose]

2026-10-17

Albert Avetisian (aavetis/PRarena)

Last observation 2026-10-17 15:27:03 (first observation 2025-05-26 14:21:34). Cumulative counts at that observation: 12,590,460 agent PRs, 10,205,344 merged. The brief states the file holds 2,433 observations; not independently counted here.

Stack Overflow Developer Survey 2025: AI sectionStack Overflow

[Mixed]

Stack Overflow

49,000+ responses from 177 countries overall; the AI usage question had 33,662 responses (26,004 professional developers); the trust question about 33,000

2025

The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software DevelopersCui (Princeton), Demirer (MIT), Jaffe (Microsoft), Musolff (Wharton), Peng (Microsoft), Salz (MIT); preprint on SSRN and Microsoft Research, listed under Management Science DOI 10.1287/mnsc.2025.00535 (journal page not read)

[Metric rose]

Cui (Princeton), Demirer (MIT), Jaffe (Microsoft), Musolff (Wharton), Peng (Microsoft), Salz (MIT); preprint on SSRN and Microsoft Research, listed under Management Science DOI 10.1287/mnsc.2025.00535 (journal page not read)

4,867 developers pooled per the abstract; enrolled per Section 2.2: 1,746 at Microsoft, 320 at Accenture, 3,054 at the anonymous company (these sum to 5,120, and the draft I read does not reconcile the two totals)

2025

Thoughts On A Month With DevinAnswer.AI (Hamel Husain, Isaac Flath, Johno Whitaker)

[Level, no comparator]

2025-01-08

Answer.AI (Hamel Husain, Isaac Flath, Johno Whitaker)

20 tasks

We are Changing our Developer Productivity Experiment Design (METR follow-up on late-2025 tools)METR (Model Evaluation & Threat Research)

[Null result]

2026-02-24

METR (Model Evaluation & Threat Research)

57 developers, 143 repositories, 800+ tasks (AI-allowed versus AI-disallowed split not given on the page)

Zalando: commit-level cyclomatic complexity inflects upward as coding agents arriveZalando SE (Bartosz Ocytko, Executive Principal Engineer, Zalando engineering blog)

[Metric rose]

2026-08-14

Zalando SE (Bartosz Ocytko, Executive Principal Engineer, Zalando engineering blog)