Ai2 study: LLMs are unreliable at judging the novelty of research ideas
What happened
On 28 October 2026, Ai2 released the study "Old Ideas, Novel Problems", testing how reliably large language models judge the novelty of scientific research ideas, and open-sourced the benchmark, data and code alongside it. The study builds a human idea pool from ICLR 2026 reviews, containing 154 high-novelty and 145 low-novelty ideas, plus 154 LLM-generated ideas. The report did not disclose details of the test results, noting only that the study focuses on whether LLM novelty judgements are reliable.
Written by AI from the reports · updated 4 h ago
Timeline
Follow the reports to see the event from each side.
- Hugging Face BlogCan we trust LLMs to judge whether a research idea is novel?
Today we’re releasing Old Ideas, Novel Problems, a new study of how reliably large language models can evaluate the novelty of scientific research ideas. Alongside the paper, we’re publishing a benchmark, data, and code for testing novelty judges, plus an automated pipeline researchers can use to build new evaluation sets.
Heat of this event
Current heat 9·Peak in the comparable range 9（Oct 28 03:00）·Change over 24 hours in the comparable range –
The trend compares only the same accounts observed without a break, so its range can be narrower than the current heat count. Move the pointer or tap the chart to see each hour; with a keyboard, use the left and right arrow keys.