Hugging Face Blog·· 22 h ago
180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers
180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers
AI summary
Streaming MoE experts directly from SSD gets within 6% of Strata on token generation.
Selection record
Threshold 60Official, first-handFirst 58Second 58
Not admittedSum of both 116 < twice the threshold 120
- Source tier
- Official, first-hand; this tier's threshold is 60
- Pre-filter
- passed:MoE推理优化与专家流式加载技术实践
A model scores each item twice, independently, against one written standard, out of 100. An item is admitted only when the two scores add up to twice the threshold. The threshold is set per source tier.
Source: Hugging Face Blog · huggingface.co