2026 大模型预训练：前沿数据全景

2026 大模型预训练：前沿数据全景

· 263 words · 2 minutes reading time

2026 大模型预训练：前沿数据全景

TL;DR

  • 2026 年前沿模型预训练数据量 14.8T–36T tokens，参数量 17B–671B（MoE 激活 37B–288B），训练集群 1.6 万到 20 万 + 加速卡。
  • 单次训练运行成本从 DeepSeek-V3 的 556 万美元（H800 租用价） 到 Grok 4 约 4.9 亿美元（Epoch AI 中位数）；OpenAI 2025 年 R&D 算力预算约 90 亿美元。
  • 架构已收敛：细粒度 MoE + MLA/GQA + RoPE + RMSNorm + SwiGLU，FP8 混合精度经 DeepSeek-V3 大规模验证。

关键事实表

（同英文版表格——保留原始数字与单位）

2024 → 2026 的五大变化

  1. Token 增加，单 token FLOP 下降（Qwen 3 36T = 推理感知过训练）；
  2. MoE 成为前沿默认（Llama 3.1 405B 是唯一稠密模型）；
  3. FP8 进入生产（DeepSeek-V3 全张量 E4M3）；
  4. 集群突破 20 万 GPU（Colossus、Rainier、Stargate）；
  5. 瓶颈从硅片变为电力（四大厂 2026 资本支出 ~7,250 亿美元）。

常见问题（与英文版一致）

参考文献 / 延伸阅读

（与英文版同——保留 arXiv 编号）

Series

LLM Pre-Training 2026

LLM Pre-Training in 2026: The Frontier in Numbers

The state of frontier LLM pre-training in 2026 — token counts, parameter counts, cluster sizes, costs, and what it all means for CTOs and ML leads.

2026 大模型预训练：前沿数据全景

Pre-Training Data Sources & Token Budgets: From Common Crawl to 36T Tokens

Where 2026 frontier LLMs get their pre-training data — Common Crawl, FineWeb, DCLM, StackV2, multilingual, PDFs — and how Qwen 3, DeepSeek-V3, Llama 3.1 sized their corpora.

Quality Filtering for LLM Pre-Training: FineWeb-Edu, DCLM, Nemotron-CC, Ultra-FineWeb

How frontier labs filter trillions of web tokens — heuristic, perplexity-based, and classifier-based filtering, with concrete recipes from FineWeb-Edu and DCLM.

Mixture-of-Experts Design: DeepSeek-V3, Qwen 3, Llama 4 Compared

Fine-grained experts, shared experts, auxiliary-loss-free routing — the modern MoE recipe in 2026, with side-by-side comparison of DeepSeek-V3, Qwen 3, Llama 4.

Multi-head Latent Attention (MLA): The KV-Cache Compression Behind DeepSeek-V3

How DeepSeek's Multi-head Latent Attention compresses the KV cache via low-rank projections + decoupled RoPE, achieving large memory reductions versus MHA at equal or better quality.

Frontier AI Training Hardware in 2026: H100, H200, GB200 NVL72, TPU Ironwood, Trainium2/3

Side-by-side specifications of every chip and pod system used at the 2026 LLM frontier, with primary-source numbers from NVIDIA, Google Cloud, and AWS.

Cluster Scale 2026: Colossus, Stargate, Project Rainier

From 16K to 1M-GPU systems — how xAI built Colossus in 122 days, how AWS deployed 500K Trainium2 chips for Anthropic, and what OpenAI's $500B Stargate commitment actually covers.

FP8 Mixed-Precision Training

Optimizers in 2026: AdamW, Muon, Shampoo/SOAP

Training Cost Economics 2026

Energy, Power & Capex