Hume released Real World VoiceEQ, a speech evaluation benchmark covering over 40 proprietary and open-source voice models, more than 15 evaluation dimensions and over 60 metrics across ASR, TTS, S2S and speech understanding.
Import AI 464 reports Fable wrote what maintainers called the first genuine and fastest megakernel on KernelBench-Mega. CUDA on RTX PRO 6000 Blackwell achieved 18.71x acceleration, versus 14.4x for Claude Opus 4.8, 11.14x for GLM-5.2 and 4.34x for GPT 5.5.
Hugging Face released ScarfBench (Self-Contained Application Refactoring Benchmark) to evaluate AI agents migrating enterprise Java applications between Spring, Jakarta EE and Quarkus.
Every Eval Ever (EEE) and Hugging Face Community Evals are now interoperable, allowing cross-publication of evaluations with links to complete records. EEE stores around 229,000 results across more than 22,000 models and 2,200 benchmarks from 31 reporting formats.
The UK AISI alignment team and Timaeus co-founded nonprofit Sequent, arguing alignment research is unprepared for superintelligence. It plans 40–80 full-time staff within two to three years and initial fundraising of $100–150 million.
Learn how our WeatherNext AI model help forecasters give communities unprecedented time to prepare ahead of the historic Hurricane Melissa.
The first step towards recursive self improvement
Google released Vantage, a research experiment using generative AI simulated conversations to assess future skills such as problem-solving and collaboration in school and university students, with English registration open on Google Labs. An Executive LLM dynamically guides dialogue and an AI Evaluator scores against rubrics.
Google Research released ConvApparel, with over 4,000 human–machine multi-turn clothing-shopping conversations, nearly 15,000 turns, to quantify realism gaps between LLM user simulators and real people.
Google Research proposed situational judgement tests (SJT) adapted from psychological questionnaires such as IRI and ERQ to evaluate alignment between LLM behaviours and human consensus in real user–assistant scenarios.
Google Research studied trade-offs between evaluation item counts and annotators per item in Forest vs Tree: The (N, K) Trade-off in Reproducible ML Evaluation. Common counts of one, three or five annotators are often insufficient; reflecting human disagreement frequently requires more than ten.
Google and Cornell published a PNAS study asking GPT-4o, Perplexity, Claude 3.5, Gemini Advanced Pro 1.5, NotebookLM and a custom RAG system 67 expert superconductivity questions. Twelve international experts blindly assessed balance, comprehensiveness, concision, evidence, image relevance and qualitative feedback.
Mistral proposes judge LLMs scoring generator responses on numerical, binary or qualitative scales, then taking weighted averages across evaluation datasets to assess RAG systems.