Hugging Face Blog·· 4 h ago
How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications
How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications
AI summary
Models specialized for a domain or a task by fine-tuning general open-source models, such as Qwen and Gemma, inherit a vocabulary built for many domains and general applications. In an essay grader we run in production, that vocabulary split the text the model writes most into many tokens.
Selection record
Threshold 60Official, first-handFirst 62Second 58
AdmittedSum of both 120 ≥ twice the threshold 120
- Source tier
- Official, first-hand; this tier's threshold is 60
- Pre-filter
- passed：专讲SLM词表优化降推理成本
- Why it was chosen
- Gives the method for rebuilding a vocabulary around token cost rather than frequency, plus measured figures, to judge how much cost specialisation can cut for small models.
A model scores each item twice, independently, against one written standard, out of 100. An item is admitted only when the two scores add up to twice the threshold. The threshold is set per source tier.
Source: Hugging Face Blog · huggingface.co