Pre-Training Data Sources & Token Budgets: From Common Crawl to 36T Tokens

Pre-Training Data Sources & Token Budgets: From Common Crawl to 36T Tokens

· 266 words · 2 minutes reading time

Pre-Training Data Sources & Token Budgets

TL;DR

  • 2026 frontier corpora span 14.8T (DeepSeek-V3) → 36T (Qwen 3) tokens.
  • The web is the substrate — Common Crawl + FineWeb (15T) + DCLM form the open-data backbone; private corpora add books, code (StackV2 ~3T), PDFs (Qwen 3 used Qwen2.5-VL OCR), and multilingual scrapes.
  • Mix recipes have shifted: ~5–30% of frontier tokens are now synthetic (textbooks, Q&A, code) generated by stronger teacher models.

Key facts

CorpusTokensSourceUsed by
FineWeb15T96 Common Crawl snapshotsopen replications
FineWeb-Edu1.3Tclassifier-filtered subset of FineWebopen models
DCLM-baseline3.8TCommon Crawl + DCLM-pool filterDCLM-7B
Nemotron-CC6.3T uniqueCC + classifier ensembling + syntheticNVIDIA models
Ultra-FineWeb-en~1TfastText-filtered FineWebMiniCPM
StackV2~900B (3T expanded)GitHubcode mixes
Llama 3.1 pre-train15.6Tmixed, undisclosedLlama 3.1 405B
DeepSeek-V3 pre-train14.8T"diverse and high-quality"DeepSeek-V3
Qwen 3 pre-train~36Tweb + PDFs + syntheticQwen 3 (119 languages)
Llama 4 pre-train>30T multimodaltext + image + videoLlama 4 family

How big is the frontier corpus and why?

Qwen 3's 36T tokens is roughly 2× Qwen 2.5's 18T — the team writes: "we collected twice as many pre-training tokens — covering three times more languages." This pushes deep into the overtraining regime (article 21).

Llama 3.1 trained on 15.6T tokens for the 405B model, which Meta describes as roughly compute-optimal under their FLOP budget of 3.8×10²⁵.

DeepSeek-V3's 14.8T is smaller — appropriate for a 37B-active MoE.

Where do the tokens come from?

Series

LLM Pre-Training 2026

LLM Pre-Training in 2026: The Frontier in Numbers

The state of frontier LLM pre-training in 2026 — token counts, parameter counts, cluster sizes, costs, and what it all means for CTOs and ML leads.

2026 大模型预训练：前沿数据全景

Pre-Training Data Sources & Token Budgets: From Common Crawl to 36T Tokens

Where 2026 frontier LLMs get their pre-training data — Common Crawl, FineWeb, DCLM, StackV2, multilingual, PDFs — and how Qwen 3, DeepSeek-V3, Llama 3.1 sized their corpora.

Quality Filtering for LLM Pre-Training: FineWeb-Edu, DCLM, Nemotron-CC, Ultra-FineWeb

How frontier labs filter trillions of web tokens — heuristic, perplexity-based, and classifier-based filtering, with concrete recipes from FineWeb-Edu and DCLM.

Mixture-of-Experts Design: DeepSeek-V3, Qwen 3, Llama 4 Compared

Fine-grained experts, shared experts, auxiliary-loss-free routing — the modern MoE recipe in 2026, with side-by-side comparison of DeepSeek-V3, Qwen 3, Llama 4.

Multi-head Latent Attention (MLA): The KV-Cache Compression Behind DeepSeek-V3

How DeepSeek's Multi-head Latent Attention compresses the KV cache via low-rank projections + decoupled RoPE, achieving large memory reductions versus MHA at equal or better quality.

Frontier AI Training Hardware in 2026: H100, H200, GB200 NVL72, TPU Ironwood, Trainium2/3

Side-by-side specifications of every chip and pod system used at the 2026 LLM frontier, with primary-source numbers from NVIDIA, Google Cloud, and AWS.

Cluster Scale 2026: Colossus, Stargate, Project Rainier

From 16K to 1M-GPU systems — how xAI built Colossus in 122 days, how AWS deployed 500K Trainium2 chips for Anthropic, and what OpenAI's $500B Stargate commitment actually covers.

FP8 Mixed-Precision Training

Optimizers in 2026: AdamW, Muon, Shampoo/SOAP

Training Cost Economics 2026

Energy, Power & Capex