Hugging Face Blog·· 8 h ago
Can we trust LLMs to judge whether a research idea is novel?
Can we trust LLMs to judge whether a research idea is novel?
AI summary
Today we’re releasing Old Ideas, Novel Problems, a new study of how reliably large language models can evaluate the novelty of scientific research ideas. Alongside the paper, we’re publishing a benchmark, data, and code for testing novelty judges, plus an automated pipeline researchers can use to build new evaluation sets.
Selection record
Threshold 60Official, first-handFirst 58Second 47
Not admittedSum of both 105 < twice the threshold 120
- Source tier
- Official, first-hand; this tier's threshold is 60
- Pre-filter
- passed：研究LLM评估科研想法新颖性并发布基准
A model scores each item twice, independently, against one written standard, out of 100. An item is admitted only when the two scores add up to twice the threshold. The threshold is set per source tier.
Source: Hugging Face Blog · huggingface.co