PRX Part 4: Our Data Strategy
Hugging Face's fourth PRX instalment details its data strategy: mixing public and internal pretraining datasets, regenerating long image captions with a VLM and converting them for streaming training. It uses Lance for construction and filtering and MDS for streaming.