Qwen3.8 27B English word addition benchmark

On 5 October 2026, Simon Willison published a benchmark using 5,070 cases with reasoning disabled to test whether the local Qwen3.8-27B-Q4_K_M.gguf model could add positive integers written as English words and output exact results. It reports a numerical accuracy of 23.57% and a format compliance rate of 96.17%; accuracy fell from 97.04% for one- to three-digit numbers to 6.44% for ten- to thirteen-digit numbers as digit count increased; with medium reasoning enabled, it answered 167 of 169 paired tests correctly.

Written by AI from the reports · updated 11 d ago