Hugging Face Blog·· 2026-08-13
What We Learned by Reproducing 2,200 papers from ICML
What We Learned by Reproducing 2,200 papers from ICML
AI summary
Hugging Face ran the ICML 2026 Open Reproduction Challenge from 15 July to 2 August. Using coding agents such as Claude Code, Codex and Cursor, 1,221 community members reproduced papers and published 6,816 Trackio logs covering 2,226 papers, around a third of the conference total.
Selection record
Threshold 60Official, first-handFirst 71Second 74
AdmittedSum of both 145 ≥ twice the threshold 120
- Source tier
- Official, first-hand; this tier's threshold is 60
- Pre-filter
- passed：复现ICML论文，用编码Agent做AI研究实验
- Why it was chosen
- Reproductions by over 1,200 community members and coding agents provide verifiable reproduction rates and falsification examples.
A model scores each item twice, independently, against one written standard, out of 100. An item is admitted only when the two scores add up to twice the threshold. The threshold is set per source tier.
Source: Hugging Face Blog · huggingface.co