Skip to content

Hugging Face shows streaming MoE expert inference with Transformers

What happened

AI digest

On 18 November 2026, the Hugging Face blog published a post on implementing streaming loading of MoE experts with Transformers. On a MacBook M5 with 32GB of unified memory, the author ran the Qwen3.8-Flash-Next model, whose FP8 experts total 121GB. In testing, short decoding speed was 13.9 tok/s, about 6% slower than Strata's 14.8 tok/s; prefill speed was 228 tok/s, slightly faster than Strata's 220 tok/s. The report does not mention whether the approach had earlier versions or prior progress.

Written by AI from the reports · updated 2 h ago

Timeline

Follow the reports to see the event from each side.

Nov 18
  1. Hugging Face Blog
    180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers

    Streaming MoE experts directly from SSD gets within 6% of Strata on token generation.

Heat of this event

Not enough continuous observations yet to draw a trend.