Hugging Face reported results on August 13 from a community project that used coding agents to check research papers accepted at ICML 2026. More than 1,200 participants produced 6,816 public experiment logbooks covering 2,226 papers, about a third of the conference. The July 15 to August 2 challenge focused on reproducing specific claims rather than simply summarizing each paper.
Participants used tools such as Claude Code, Codex, and Cursor to write code, run experiments, and record outcomes. An automated judge then classified individual claims as verified, falsified, supported only by smaller toy experiments, or inconclusive. The logbooks retained code and artifacts so others could inspect the evidence.
At least one claim was independently verified in 1,103 papers. Only 266 papers had every extracted claim verified. Another 496 papers had at least one disputed or falsified claim. In 242 cases, different reproduction teams disagreed about the same claims.
Some reported problems survived closer scrutiny. One paper's published evaluation had counted large amounts of padding as meaningful tokens, changing the interpretation of its quality result. Other alleged failures turned out to be mistakes by the reproduction agents, including comparisons that used incompatible units.
Hugging Face's organizers manually rechecked formal falsification claims and contacted paper authors. Missing datasets, inconsistent experiment settings and incorrect agent calculations can make a legitimate result look irreproducible. The project demonstrates a way to audit research at scale, not that every paper without a successful reproduction was wrong.