Hugging Face Blog·· 11 h ago
Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
AI summary
Native-script data for 22 languages, romanized data for twelve, and a reusable GPU-accelerated pipeline for translation, quality assessment, and repair. We're excited to introduce Nemotron-pretraining-Indic-v1 approximately 2.8 trillion tokens across 22 Indian languages, releasing soon alongside the open, GPU-accelerated data generation pipeline used to create it.
Selection record
Threshold 60Official, first-handFirst 58Second 58
Not admittedSum of both 116 < twice the threshold 120
- Source tier
- Official, first-hand; this tier's threshold is 60
- Pre-filter
- passed：介绍印地语预训练语料与数据生成管线
A model scores each item twice, independently, against one written standard, out of 100. An item is admitted only when the two scores add up to twice the threshold. The threshold is set per source tier.
Source: Hugging Face Blog · huggingface.co