PRX Part 4: Our Data Strategy

Hugging Face's fourth PRX instalment details its data strategy: mixing public and internal pretraining datasets, regenerating long image captions with a VLM and converting them for streaming training. It uses Lance for construction and filtering and MDS for streaming. After switching to Qwen3-VL, text latents are computed during training, with measured throughput loss of around 3–4%, or roughly one extra day for 30 days of training.