Harvey Legal Agent Benchmark
Vals AI / Harvey / Professional work · Process legal materials and deliver reviewable files such as Word, Excel and slides.
What it measures, and how
The Harvey private test set, six file tools and docx, xlsx and pptx skills are fixed, with the internet disabled; the mean whole-task pass rate of two judges is used, not a single-condition pass rate. The 11.25% of Fable 5 that includes an Opus 4.8 fallback cannot be attributed to a single model; the official figure separately lists 10.42% without fallback, which must be reviewed under the unified configuration policy.
How this evidence is used
Under observation: complete legal work evidence is being prioritised; stable score exports, configuration and re-display boundaries are pending, so only the source is recorded for now, with no automatic collection or scoring.
Limits and data attribution
Model fallbacks and judge conditions affect comparability; a roughly 90% pass rate on individual conditions does not mean the whole task was completed. Multiple specialist items under the same Vals cannot be treated as several independent organisations, nor can total weight be increased by the number of leaderboards.
Data licence: Official public scores are available for viewing; Harvey's open question bank and code licence do not automatically cover Vals private test scores, and the boundaries for re-displaying leaderboard data are unconfirmed.
Scores published by Vals AI / Harvey. Raw scores and the News consensus score use different scales and cannot be added directly.