Quality Filtering for LLM Pre-Training: FineWeb-Edu, DCLM, Nemotron-CC, Ultra-FineWeb

2026-05-25T00:00:00+00:00

Quality Filtering: FineWeb-Edu, DCLM, Ultra-FineWeb</h1>

TL;DR</h2>

Classifier-based filtering is the single biggest open-data lever</strong>: FineWeb-Edu yields ~10× the MMLU lift of unfiltered FineWeb.</li>
Two recipes dominate in 2026: (1)</strong> train a small classifier on LLM-rated documents (FineWeb-Edu); (2)</strong> use fastText classifiers + perplexity (DCLM, Ultra-FineWeb).</li>
Nemotron-CC pushes the frontier further: 4× more unique real tokens than DCLM</strong>, with +5 MMLU / +3.1 ARC-Challenge over Llama 3.1 8B at 15T tokens</strong>.</li> </ul>
Key facts</h2>
Pipeline</th> Token output</th> Filter type</th> Reference</th></tr></thead>

FineWeb-Edu</td> 1.3T</strong> from FineWeb 15T</td> Llama-3-70B-rated → linear classifier</td> Penedo et al., arXiv:2406.17557</td></tr>
DCLM-baseline</td> 3.8T</td> fastText + perplexity + heuristic</td> Li et al., 2024</td></tr>
Nemotron-CC</td> 6.3T unique</strong></td> classifier ensemble + synthetic rephrase</td> Su et al., arXiv:2412.02595</td></tr>
Ultra-FineWeb-en</td> ~1T</td> fastText (dim 256, n-gram 3)</td> Wang et al., 2025</td></tr> </tbody></table>
How does classifier-based filtering work?</h2>

Take ~500K documents from your corpus.</li>
Score each with a strong LLM (FineWeb-Edu uses Llama-3-70B-Instruct rating 0–5 on "educational value").</li>
Train a small classifier (FineWeb-Edu: linear head on Snowflake-arctic-embed-m embeddings).</li>
Apply to your full corpus. Cost: 6,000 H100-hours for 15T tokens.</strong></li> </ol>
Why does it work?</h2>
FineWeb-Edu's threshold = 3 keeps ~8.7% of tokens but lifts MMLU by ~3 points and ARC by ~5 points vs full FineWeb.</p>
Code (FineWeb-Edu classifier inference, simplified)</h2>

Training Cost Economics 2026

2026-05-25T00:00:00+00:00

Training Cost Economics 2026</h1>
TL;DR</h2>

DeepSeek-V3 final pre-training</strong>: 2.788M H800-hours = $5.576M at $2/hr rental</strong> (paper, Table 1). Excludes R&D, salaries, ~$1B of owned H800 hardware, ablations, post-training.</li>
Llama 3.1 405B</strong>: ~$170M est., 3.8e25 FLOP on 16K H100s over ~54 days.</li>
Grok 4 median</strong>: $490M</strong> (Epoch AI, 2025) — two independent methods (H100 rental + amortized hardware+power) both yielded ~$490M.</li>
GPT-5 total compute</strong>: ~5e25 FLOP</strong> (Epoch AI estimate; less than GPT-4.5 at >1e26).</li>
OpenAI 2024 cloud spend</strong>: $7B (</del>$5B R&D + ~$2B inference); 2025 projected ~$9B R&D</strong>.</li> </ul>
What is and isn't in the headlines</h2>
Cost category</th> DeepSeek-V3 $5.5M</th> Llama 3.1 $170M</th> GPT-5 (Epoch est.)</th></tr></thead>

Final pre-training run</td> ✅</td> ✅</td> ✅</td></tr>
Hardware capex</td> ❌ (rental price)</td> partial</td> ❌</td></tr>
Ablations / experiments</td> ❌</td> ❌</td> ❌</td></tr>
R&D salaries</td> ❌</td> ❌</td> ❌</td></tr>
Failed runs</td> ❌</td> ❌</td> ❌</td></tr> </tbody></table>
The CFO interpretation</h2>
Headline numbers understate true cost by 5–20×</strong>. Use Epoch AI's amortized-hardware methodology for "all-in" comparisons.</p>
References</h2>

arXiv:2412.19437; arXiv:2407.21783; epoch.ai/gradient-updates/why-gpt5-used-less-training-compute-than-gpt45-but-gpt6-probably-wont; epoch.ai/data-insights/grok-4-training-resources; epoch.ai/data-insights/openai-compute-spend</li> </ul>

2026 大模型预训练：前沿数据全景

2026-05-24T00:00:00+00:00

2026 大模型预训练：前沿数据全景</h1>
TL;DR</h2>

2026 年前沿模型预训练数据量 14.8T–36T tokens</strong>，参数量 17B–671B（MoE 激活 37B–288B）</strong>，训练集群 1.6 万到 20 万 + 加速卡</strong>。</li>
单次训练运行成本从 DeepSeek-V3 的 556 万美元（H800 租用价）</strong> 到 Grok 4 约 4.9 亿美元（Epoch AI 中位数）</strong>；OpenAI 2025 年 R&D 算力预算约 90 亿美元</strong>。</li>
架构已收敛：细粒度 MoE + MLA/GQA + RoPE + RMSNorm + SwiGLU</strong>，FP8 混合精度</strong>经 DeepSeek-V3 大规模验证。</li> </ul>
关键事实表</h2>
（同英文版表格——保留原始数字与单位）</p>
2024 → 2026 的五大变化</h2>

Token 增加，单 token FLOP 下降（Qwen 3 36T = 推理感知过训练）；</li>
MoE 成为前沿默认（Llama 3.1 405B 是唯一稠密模型）；</li>
FP8 进入生产（DeepSeek-V3 全张量 E4M3）；</li>
集群突破 20 万 GPU（Colossus、Rainier、Stargate）；</li>
瓶颈从硅片变为电力（四大厂 2026 资本支出 ~7,250 亿美元）。</li> </ol>
常见问题（与英文版一致）</h2>
参考文献 / 延伸阅读</h2>
（与英文版同——保留 arXiv 编号）</p>

LLM Pre-Training in 2026: The Frontier in Numbers

2026-05-24T00:00:00+00:00

LLM Pre-Training in 2026: The Frontier in Numbers</h1>
TL;DR</h2>

Frontier 2026 LLMs are pre-trained on 14.8T–36T tokens</strong> with 17B–671B parameters</strong> (37B–288B active for MoE), on clusters of 16K to 200K+ accelerators</strong>.</li>
A single final training run now costs $5.6M (DeepSeek-V3 rental cost) to ~$490M (Grok 4 median, Epoch AI)</strong>, while OpenAI's 2025 R&D</em> compute is projected at ~$9B</strong>.</li>
Architecture has converged: fine-grained MoE + MLA or GQA + RoPE + RMSNorm + SwiGLU</strong>, with FP8 mixed precision</strong> now production-validated by DeepSeek-V3.</li> </ul>
Key facts (citable)</h2>
Lab / Model</th> Total params</th> Active</th> Pre-train tokens</th> Hardware</th> Headline cost</th></tr></thead>

DeepSeek-V3</td> 671B</td> 37B</td> 14.8T</td> 2,048 H800</td> 2.788M H800-hours = $5.576M</strong> rental</td></tr>
Llama 3.1 405B</td> 405B (dense)</td> 405B</td> 15.6T</strong></td> 16,384 H100</strong></td> ~$170M est. (3.8e25 FLOP)</td></tr>
Qwen 3 235B-A22B</td> 235B</td> 22B</td> 36T</strong></td> undisclosed</td> —</td></tr>
Llama 4 Maverick</td> 400B</td> 17B (128 experts)</td> >30T multimodal</td> undisclosed</td> —</td></tr>
Llama 4 Behemoth</td> ~2T</td> 288B (16 experts)</td> undisclosed</td> undisclosed</td> —</td></tr>
GPT-5 (Epoch AI estimate)</td> undisclosed</td> undisclosed</td> "≥30T likely"</td> undisclosed</td> ~5e25 FLOP</strong> total</td></tr> </tbody></table>

Glossary — "active parameters."</strong> In MoE, "active" is the per-token compute footprint: e.g., DeepSeek-V3 routes each token through 37B of its 671B parameters.</p> </blockquote>
What changed from 2024 to 2026?</h2>
1) Tokens went up; FLOPs per token went down</h3>
Qwen 3's 36T-token</strong> corpus is more than 2× the Chinchilla-optimal budget for a 235B-active model, reflecting the new economics of inference-aware overtraining</em> (covered in article 21).</p>
2) MoE became the default at the frontier</h3>
DeepSeek-V3 (671B/37B), Qwen 3 235B-A22B, Llama 4 Maverick (400B/17B), Llama 4 Behemoth (2T/288B). The only dense frontier holdout is Llama 3.1 405B</strong>.</p>
3) FP8 is no longer experimental</h3>
DeepSeek-V3 trained the full</em> model in FP8 with E4M3 on every tensor (Section 3.3 of the V3 paper), using 1×128 activation tiles and 128×128 weight blocks to control outliers.</p>
4) Cluster scale broke the 200K-GPU barrier</h3>
xAI Colossus</strong> in Memphis went from 100K H100 to 200K H100/H200 in 92 days, then added 30K GB200. AWS Project Rainier</strong> (~500K Trainium2, scaling to 1M) hosts Anthropic Claude training. OpenAI Stargate</strong> is a $500B, 10-GW commitment, with ~7 GW under contract as of late 2025.</p>
5) Power, not silicon, is now the bottleneck</h3>
The big four hyperscalers' combined 2026 AI capex is ~$725B</strong> (Microsoft $190B, Amazon ~$200B, Alphabet up to $190B, Meta up to $145B per company Q1 2026 guidance).</p>
How to read this series</h2>
Each article is self-contained but cross-linked. We recommend:</p>

Multi-head Latent Attention (MLA): The KV-Cache Compression Behind DeepSeek-V3

2026-05-23T00:00:00+00:00

Multi-head Latent Attention (MLA)</h1>
TL;DR</h2>

MLA, introduced in DeepSeek-V2</strong> (and refined in V3), compresses each token's KV state into a low-rank latent vector</strong> of dimension d_c much smaller than 2·d_h·H, dramatically shrinking the KV cache per token.</li>
Unlike GQA (which shares heads), MLA stores a compressed latent and re-projects to full K/V at attention time</strong>.</li>
A decoupled RoPE</strong> branch carries positional information on a small dedicated subspace, so the latent cache remains position-agnostic and reusable.</li>
DeepSeek-V2 ablations show MLA matches or slightly beats MHA</strong> at a fraction of the KV memory.</li> </ul>
Key facts</h2>
Variant</th> Cache per token (elements)</th> Quality vs MHA</th></tr></thead>

MHA</td> 2 · d_h · H · L</td> baseline</td></tr>
GQA (g groups)</td> 2 · d_h · g · L</td> usually slightly worse</td></tr>
MLA</strong></td> (d_c + d_r) · L</strong></td> matches or beats MHA</td></tr> </tbody></table>
How does MLA work mathematically?</h2>
For input (x_n \in \mathbb{R}^d), per layer:</p>

Down-project to latent</strong>: (c^{KV}_n = W^{DKV} x_n \in \mathbb{R}^{d_c}) ← cached</li>
Up-project for attention</strong>: (K = W^{UK} c^{KV}), (V = W^{UV} c^{KV})</li>
Decoupled RoPE</strong>: a small per-head subspace (k^{rope}_n = \mathrm{RoPE}(W^{KR} x_n)) is computed separately and concatenated to K for the attention dot product (also cached, dimension d_r).</li> </ol>
Queries are similarly down/up-projected; query compression is used only during training.</p>
Why decouple RoPE?</h2>
If you applied RoPE to the latent</em> (c^{KV}) directly, the up-projection (W^{UK}) would no longer commute with rotation — you'd have to materialize K at every step at full size, killing the cache savings. The decoupled RoPE subspace solves this elegantly: position information lives on a small, separately cached vector.</p>
Code (MLA core, illustrative)</h2>

Optimizers in 2026: AdamW, Muon, Shampoo/SOAP

2026-05-23T00:00:00+00:00

Optimizers in 2026: AdamW, Muon, Shampoo/SOAP</h1>
TL;DR</h2>

AdamW</strong> remains the production default at frontier scale (Llama 3.1, Qwen 3, GPT-5-era).</li>
Muon</strong> (Jordan et al., 2024; Liu et al. arXiv:2502.16982) uses Newton–Schulz orthogonalization</strong> of momentum and shows ~2× compute efficiency vs AdamW under compute-optimal training</strong> in Moonshot's Moonlight scaling study (3B/16B MoE, 5.7T tokens).</li>
However, Wen et al. (arXiv:2512.05620) find the gap shrinks to ~1.4× at 400M–1.5B and ~1.1× at 1.2B once AdamW is well-tuned</strong>, raising the question of whether headline 2× gains transfer to multi-billion-scale frontier runs.</li> </ul>
Why Muon works</h2>
Orthogonalizing the per-matrix momentum update prevents the optimizer from collapsing into a few dominant directions. Newton–Schulz makes the operation ~5–10× cheaper than full SVD, with negligible per-step overhead at GPU scale.</p>
Recommendation</h2>

Frontier dense / MoE runs</strong>: AdamW remains lowest-risk; switch to Muon only with a μP-style HP sweep at small scale.</li>
Pre-1B research</strong>: Muon's 1.4–2× gain is real and easy to pick up.</li>
SOAP</strong>: a Shampoo variant; competitive with Muon at 100M–1B but heavier compute.</li> </ul>
References</h2>

Jordan et al. 2024 (Muon original); Liu et al. arXiv:2502.16982; Wen et al. arXiv:2512.05620; Du & Su arXiv:2604.01472 (Newton-Muon).</li> </ul>

Frontier AI Training Hardware in 2026: H100, H200, GB200 NVL72, TPU Ironwood, Trainium2/3

2026-05-22T00:00:00+00:00

Frontier AI Training Hardware in 2026</h1>
TL;DR</h2>

NVIDIA</strong> ships the most-deployed training chips: H100 (700 W, 3.35 TB/s HBM3, 989 BF16 dense / 1,979 FP8 dense TFLOPS), H200 (141 GB HBM3e, 4.8 TB/s, same compute), and Blackwell B200 / GB200 NVL72 (per-GPU 5 PFLOPS FP8 dense / 10 PFLOPS sparse; rack 1.44 EFLOPS FP4 sparse).</li>
Google's Ironwood (TPU v7)</strong> delivers 4,614 TFLOPS FP8 per chip</strong>, 192 GB HBM3e, and 42.5 EFLOPS FP8 per 9,216-chip pod</strong> — roughly closing the gap to NVIDIA on a per-chip basis.</li>
AWS Trainium2</strong> = 1.3 PFLOPS FP8 dense / chip; UltraServer = 64 chips / 83.2 PFLOPS / 6 TB HBM. Trainium3 (3 nm, 2026)</strong> doubles per-chip compute to 2.52 PFLOPS FP8</strong> with 144 GB HBM3e.</li> </ul>
Master comparison table (primary sources)</h2>
Chip</th> Dense FP8 (TFLOPS)</th> HBM</th> HBM BW</th> Interconnect</th> TDP</th></tr></thead>

H100 SXM5</td> 1,979</strong></td> 80 GB HBM3</td> 3.35 TB/s</td> NVLink 4: 900 GB/s</td> 700 W</td></tr>
H200 SXM</td> 1,979</strong></td> 141 GB HBM3e</strong></td> 4.8 TB/s</strong></td> NVLink 4: 900 GB/s</td> 700 W</td></tr>
B200 (HGX)</td> 4,500</strong> (dense)</td> 180 GB HBM3e</td> 7.7 TB/s</td> NVLink 5: 1.8 TB/s</td> ~1,000 W</td></tr>
GB200 GPU</td> 5,000</strong> (dense)</td> 186 GB HBM3e</strong></td> 8 TB/s</td> NVLink 5: 1.8 TB/s</td> up to 1,200 W</td></tr>
TPU v5p</td> (BF16) 459 TFLOPS</td> 95 GB HBM2e</td> 2.76 TB/s</td> ICI 4,800 Gbps/chip, 8,960-chip pod</td> —</td></tr>
TPU v6e Trillium</td> (BF16) ~918 TFLOPS</td> 32 GB HBM</td> ~1.6 TB/s</td> 256-chip pod</td> —</td></tr>
TPU v7 Ironwood</strong></td> 4,614</strong></td> 192 GB HBM3e</strong></td> 7.37 TB/s</strong></td> 9,216-chip pod</td> ~1 kW</td></tr>
Trainium2</td> 1,300</strong></td> 96 GB HBM</td> ~2.9 TB/s</td> NeuronLink, 64-chip UltraServer</td> —</td></tr>
Trainium3 (3 nm)</td> 2,520 (MXFP8)</strong></td> 144 GB HBM3e</strong></td> 4.9 TB/s</strong></td> NeuronLink v4, 144-chip UltraServer</td> —</td></tr> </tbody></table>
Pod / rack totals</h3>
System</th> Aggregate FP8 (dense)</th> Aggregate HBM</th> Aggregate interconnect</th></tr></thead>

GB200 NVL72</td> 72 × 5 PF = 360 PFLOPS dense</strong> (720 PF sparse)</td> ~13.4 TB HBM3e</td> 130 TB/s NVLink</strong></td></tr>
TPU v7 Ironwood pod</td> ~42.5 EFLOPS FP8</strong></td> 9,216 × 192 GB = ~1.7 PB HBM3e</strong></td> 3D-torus ICI</td></tr>
Trn2 UltraServer</td> 64 × 1.3 = 83.2 PFLOPS FP8</strong></td> 6 TB HBM</strong></td> 185 TB/s</td></tr>
Trn3 UltraServer</td> 144 × 2.52 = 362 PFLOPS MXFP8</strong></td> 20.7 TB HBM3e</strong></td> 706 TB/s</td></tr> </tbody></table>
Sources (verbatim quotes)</h2>

H200</strong>: "the NVIDIA H200 is the first GPU to offer 141 gigabytes (GB) of HBM3e memory at 4.8 terabytes per second (TB/s)" (nvidia.com/en-us/data-center/h200/).</li>
GB200 NVL72</strong>: "72 NVIDIA Blackwell GPUs interconnected by the largest NVIDIA NVLink domain ever offered, NVLink Switch System provides 130 terabytes per second (TB/s) of low-latency GPU communications" (nvidia.com/en-us/data-center/gb200-nvl72/).</li>
TPU v5p</strong>: "Each TPU v5p pod composes together 8,960 chips over our highest-bandwidth inter-chip interconnect (ICI) at 4,800 Gbps/chip in a 3D torus topology" (cloud.google.com).</li>
TPU Ironwood</strong>: "TPU7x is the first release within the Ironwood family… With a 9,216-chip footprint per Pod… Each chip is equipped with 192 GB of HBM, with bandwidth of approximately 7.37 TB/s" (docs.cloud.google.com/tpu/docs/tpu7x).</li>
Trainium2</strong>: "Trn2 instances feature 16 Trainium2 chips… up to 20.8 FP8 petaflops of compute. Trn2 UltraServers extend NeuronLink connectivity to 64 Trainium2 chips… up to 83.2 FP8 petaflops of compute" (aws.amazon.com/ec2/instance-types/trn2/).</li>
Trainium3</strong>: "AWS Trainium3 chip provides 2x higher compute performance to 2.52 petaflops (PFLOPs) of FP8 compute, increases the memory capacity by 1.5x and bandwidth by 1.7x over Trainium2 to 144 GB of HBM3e memory, and 4.9 TB/s of memory bandwidth" (aws.amazon.com/ai/machine-learning/trainium/).</li> </ul>
FAQ</h2>
Q: How does Ironwood compare to GB200?</strong> Per-chip FP8: Ironwood ~4.6 PF vs GB200 5 PF dense — essentially equal. Pod scale: 9,216-chip Ironwood pod = 42.5 EFLOPS vs 72-chip NVL72 rack = 360 PFLOPS dense. Google's scale-up unit is now ~100× larger than NVIDIA's.</p>
Q: When does FP4 become production?</strong> GB200 supports it; DeepSeek's V3 paper and the SemiAnalysis Trainium3 piece both note FP4 as the next step.</p>
Q: Why ~1 kW chips?</strong> HBM stacks + tensor cores at higher clock — power density driving the gigawatt data-center wave.</p>
References</h2>

NVIDIA HGX B200 OEM datasheet; nvidia.com/en-us/data-center/{h100,h200,gb200-nvl72}/</li>
developer.nvidia.com/blog/nvidia-gb200-nvl72-delivers-trillion-parameter-llm-training-and-real-time-inference/</li>
cloud.google.com/blog/products/compute/{introducing-trillium-6th-gen-tpus,ironwood-tpus-and-new-axion-based-vms-for-your-ai-workloads}</li>
aws.amazon.com/ec2/instance-types/{trn2,trn3}/</li>
SemiAnalysis: TPUv7 deep-dive; AWS Trainium3 deep-dive.</li> </ul>
Further reading</h2>
→ Article 13 (Cluster scale) · Article 14 (5D parallelism) · Article 15 (FP8)</p>

Mixture-of-Experts Design: DeepSeek-V3, Qwen 3, Llama 4 Compared

2026-05-22T00:00:00+00:00

Mixture-of-Experts Design in 2026</h1>
TL;DR</h2>

Fine-grained MoE + shared expert + auxiliary-loss-free routing</strong> is the de-facto 2026 recipe (DeepSeek-V3, Llama 4, Qwen 3 MoE variants).</li>
Sparsity ratios have widened: DeepSeek-V3 activates 5.5% (37B/671B); Llama 4 Maverick activates 4.25% (17B/400B); Behemoth activates 14% (288B/2T)</strong>.</li>
Auxiliary-loss-free balancing (per-expert bias, updated each step) replaces classic load-balancing loss — better quality, same balance.</li> </ul>
Key facts</h2>
Model</th> Total</th> Active</th> Experts</th> Top-k</th> Shared</th> Routing</th></tr></thead>

DeepSeek-V3</strong></td> 671B</td> 37B</td> 256 routed + 1 shared (per MoE layer)</td> 8</td> yes</td> aux-loss-free</strong></td></tr>
Qwen 3 235B-A22B</strong></td> 235B</td> 22B</td> 128</td> 8</td> yes</td> softmax + small aux</td></tr>
Qwen 3 30B-A3B</strong></td> 30B</td> 3B</td> 128</td> 8</td> yes</td> softmax + small aux</td></tr>
Llama 4 Scout</strong></td> 109B</td> 17B</td> 16</td> 1</td> yes</td> softmax</td></tr>
Llama 4 Maverick</strong></td> 400B</td> 17B</td> 128</td> 1</td> yes</td> softmax</td></tr>
Llama 4 Behemoth</strong></td> ~2T</td> 288B</td> 16</td> 1</td> yes</td> softmax</td></tr> </tbody></table>
How does auxiliary-loss-free routing work?</h2>
Classic MoE adds an auxiliary loss penalizing expert imbalance. DeepSeek-V3 instead maintains a per-expert bias</strong> (b_e) that nudges the routing logits each step:</p>

Energy, Power & Capex

2026-05-21T00:00:00+00:00

Energy, Power & Capex: Stargate, the Big-4, and the Gigawatt Era</h1>
TL;DR</h2>

Big-4 hyperscaler 2026 AI capex = ~$725B combined</strong> (Microsoft $190B; Amazon ~$200B; Alphabet up to $190B; Meta up to $145B per Q1 2026 guidance).</li>
Stargate</strong> = $500B / 10 GW by 2029; ~7 GW under contract by end-2025.</li>
xAI Colossus</strong> + Colossus 2 target gigawatt-scale, including ~1.1+ GW of Solaris-supplied gas turbines (Memphis + Mississippi).</li>
A 1 GW AI data center consumes power equivalent to ~750,000 US homes; grid interconnect, not chips, is now the bottleneck</strong>.</li> </ul>
The number to remember</h2>
$725 billion</strong> — combined 2026 AI capex of MSFT, AMZN, GOOGL, META.</p>
Why power is the bottleneck</h2>

Per-chip TDP: 700 W (H100) → 1,200 W (GB200) → ~1 kW (Ironwood).</li>
100K GB200 = ~120 MW just at the silicon; double with cooling+networking.</li>
Permitting a new 1 GW interconnect in the US takes 3–7 years.</li> </ul>
Strategic implications for CTOs</h2>

Lock multi-year power PPAs early</strong> (Anthropic's $100B Amazon deal includes capacity reservations).</li>
Site selection now matters more than chip selection</strong> — ERCOT, PJM, Pacific NW where excess capacity exists.</li>
Co-locate with renewables + batteries</strong> (Tesla Megapacks at Colossus, SB Energy at Stargate Milam).</li> </ol>
References</h2>

Statista / Tom's Hardware (May 2026) on Big-4 capex</li>
openai.com Stargate announcements; anthropic.com/news/anthropic-amazon-compute</li>
SemiAnalysis Colossus 2 piece</li> </ul>

FP8 Mixed-Precision Training

2026-05-21T00:00:00+00:00

FP8 Mixed-Precision Training (DeepSeek-V3 recipe)</h1>
TL;DR</h2>

DeepSeek-V3 is the first extremely large model (671B/37B-active) to validate end-to-end FP8 pre-training</strong> at scale.</li>
The team uses E4M3 on all tensors</strong> (not the NVIDIA hybrid E4M3/E5M2), enabled by fine-grained 1×128 activation tiles and 128×128 weight blocks</strong>.</li>
Optimizer state uses BF16 instead of FP32</strong>, saving memory; accumulation uses promoted FP32 (NVIDIA CUTLASS pattern).</li> </ul>
Recipe (verbatim from DeepSeek-V3, §3.3)</h2>

"we adopt the E4M3 format on all tensors for higher precision. We attribute the feasibility of this approach to our fine-grained quantization strategy, i.e., tile and block-wise scaling." "we calculate the maximum absolute value online for each 1x128 activation tile or 128x128 weight block."</p> </blockquote>
Why it works</h2>
Tile/block-wise scaling shares the exponent across small element groups, capturing dynamic range that tensor-wide scaling can't. Outliers in any single element don't blow up the global scale.</p>
FAQ</h2>
Q: Is FP4 next?</strong> Yes — Blackwell supports it; DeepSeek's hardware companion paper (arXiv:2505.09343) discusses low-precision directions. Q: Does FP8 hurt loss?</strong> DeepSeek-V3 reports <0.25% loss perturbation</strong> vs BF16 — negligible at scale.</p>
References</h2>

arXiv:2412.19437 (V3); arXiv:2505.09343 (V3/R1 hardware insights); Colfax: "DeepSeek-R1 and FP8 Mixed-Precision Training"</li> </ul>

Cluster Scale 2026: Colossus, Stargate, Project Rainier

2026-05-20T00:00:00+00:00

Cluster Scale 2026</h1>
TL;DR</h2>

xAI Colossus 1</strong>: ~200,000 GPUs (150K H100 + 50K H200) plus ~30K GB200, ) built in a former Memphis Electrolux plant; first 100K phase deployed in 122 days</strong>, doubled in another 92.</li>
AWS Project Rainier</strong>: ~500,000 Trainium2 chips across multiple US data centers (Indiana flagship), scaling to >1M; Anthropic exclusive tenant</strong>; ~5× the compute Anthropic used for prior Claude generations.</li>
OpenAI Stargate</strong>: a $500B, 10-GW</strong> commitment over four years; ~7 GW under contract by end-2025</strong> (Abilene flagship live, Texas / NM / Ohio / WI sites; Stargate UAE).</li> </ul>
Key facts</h2>
Cluster</th> Chips</th> Type</th> Power</th> Tenant</th> Status</th></tr></thead>

Colossus 1</td> 200K + 30K GB200 )</td> NVIDIA H100/H200 + GB200</td> ~250 MW + gas turbines</td> xAI (Grok)</td> Live</td></tr>
Colossus 2 (Mississippi)</td> target 550K+ Blackwell</td> NVIDIA GB200/GB300</td> gigawatt-scale</td> xAI</td> Construction 2026</td></tr>
Project Rainier</strong></td> ~500K Trainium2</strong> (→1M)</td> AWS Trainium2 → Trainium3</td> multi-site</td> Anthropic</td> Live, expanding</td></tr>
Stargate Abilene</strong></td> NVIDIA GB200 racks</td> NVIDIA</td> ~1 GW potential</td> OpenAI</td> Live</td></tr>
Stargate total</td> "2M chips" plan</td> NVIDIA + custom</td> 10 GW by 2029</strong></td> OpenAI</td> $500B commitment</td></tr> </tbody></table>
How did xAI build 200K GPUs in 214 days?</h2>
xAI took an abandoned 785,000-sq-ft Electrolux plant, brought in 14 mobile gas turbines for power, and used Supermicro liquid-cooled racks (64 GPUs/rack, 1,500 racks for the 100K phase). Networking: Nvidia Spectrum-X Ethernet (not InfiniBand)</strong> with BlueField-3 SuperNICs at 400 Gb/s/GPU and SN5600 (51.2 Tb/s) top-of-rack switches.</p>
What is Project Rainier actually?</h2>
AWS describes Rainier as an "EC2 UltraCluster of Trainium2 UltraServers</strong>." Each UltraServer = 4 servers × 16 chips = 64 Trainium2 chips connected by NeuronLink (blue cables); UltraServers tile via Elastic Fabric Adapter (yellow cables). Anthropic: "we currently use over one million Trainium2 chips to train and serve Claude."</p>
What is Stargate, concretely?</h2>
Stargate is OpenAI's umbrella for all</strong> future infrastructure (not a single facility). Sites announced through Q1 2026:</p>
Site</th> Status</th> GW</th></tr></thead>

Abilene, TX (flagship)</td> Live (GB200 racks since Jun 2025)</td> up to ~1</td></tr>
Shackelford County, TX</td> Construction</td> 1.4</td></tr>
Doña Ana County, NM</td> Construction</td> —</td></tr>
Lordstown, OH (SoftBank)</td> Construction</td> —</td></tr>
Milam County, TX (SB Energy)</td> 2026</td> 1.2</td></tr>
Stargate Michigan ("The Barn")</td> 2026 construction</td> 1.0</td></tr>
Stargate UAE</td> 2026 opening</td> —</td></tr>
Stargate Norway / UK / Argentina</td> Planned</td> —</td></tr> </tbody></table>
Plus contracts: $300B Oracle (5 years), $100B Nvidia equity-for-compute, $90B AMD warrant deal, $250B Microsoft Azure, $350B Broadcom custom silicon, AWS $50B Trainium.</p>
FAQ</h2>
Q: Are these single-fabric clusters or "logical" clusters?</strong> Colossus 1 is split across four 25,000-GPU halls</strong>; analysts believe they were not designed as one HPL system. Rainier is multi-data-center</strong> with EFA cross-site. Stargate Abilene is single-site.</p>
Q: Power per GPU?</strong> H100 ≈ 700 W, GB200 up to 1,200 W. A 100K GB200 system = ~120 MW just for chips, doubling with cooling + networking.</p>
Q: 1M GPUs — feasible?</strong> xAI's Colossus 2 roadmap, OpenAI's Stargate 10 GW, and Anthropic's "≥5 GW" AWS expansion all imply ~$725B aggregate 2026 hyperscaler capex.</p>
References</h2>

xAI Colossus: x.ai/colossus; SemiAnalysis "Colossus 2"; en.wikipedia.org/wiki/Colossus_(supercomputer)</li>
AWS Rainier: aboutamazon.com/news/aws/aws-project-rainier-ai-trainium-chips-compute-cluster; anthropic.com/news/anthropic-amazon-compute</li>
Stargate: openai.com/index/announcing-the-stargate-project/; openai.com/index/five-new-stargate-sites/</li> </ul>
Further reading</h2>
→ Article 12 · Article 16 (reliability) · Article 24 (energy & capex)</p>

Pre-Training Data Sources & Token Budgets: From Common Crawl to 36T Tokens

2026-05-20T00:00:00+00:00

Pre-Training Data Sources & Token Budgets</h1>
TL;DR</h2>

2026 frontier corpora span 14.8T (DeepSeek-V3) → 36T (Qwen 3)</strong> tokens.</li>
The web is the substrate — Common Crawl + FineWeb (15T) + DCLM</strong> form the open-data backbone; private corpora add books, code (StackV2 ~3T), PDFs (Qwen 3 used Qwen2.5-VL OCR), and multilingual scrapes.</li>
Mix recipes have shifted: ~5–30% of frontier tokens are now synthetic</strong> (textbooks, Q&A, code) generated by stronger teacher models.</li> </ul>
Key facts</h2>
Corpus</th> Tokens</th> Source</th> Used by</th></tr></thead>

FineWeb</td> 15T</strong></td> 96 Common Crawl snapshots</td> open replications</td></tr>
FineWeb-Edu</td> 1.3T</strong></td> classifier-filtered subset of FineWeb</td> open models</td></tr>
DCLM-baseline</td> 3.8T</strong></td> Common Crawl + DCLM-pool filter</td> DCLM-7B</td></tr>
Nemotron-CC</td> 6.3T unique</strong></td> CC + classifier ensembling + synthetic</td> NVIDIA models</td></tr>
Ultra-FineWeb-en</td> ~1T</strong></td> fastText-filtered FineWeb</td> MiniCPM</td></tr>
StackV2</td> ~900B (3T expanded)</td> GitHub</td> code mixes</td></tr>
Llama 3.1 pre-train</td> 15.6T</strong></td> mixed, undisclosed</td> Llama 3.1 405B</td></tr>
DeepSeek-V3 pre-train</td> 14.8T</strong></td> "diverse and high-quality"</td> DeepSeek-V3</td></tr>
Qwen 3 pre-train</td> ~36T</strong></td> web + PDFs + synthetic</td> Qwen 3 (119 languages)</td></tr>
Llama 4 pre-train</td> >30T</strong> multimodal</td> text + image + video</td> Llama 4 family</td></tr> </tbody></table>
How big is the frontier corpus and why?</h2>
Qwen 3's 36T tokens is roughly 2× Qwen 2.5's 18T</strong> — the team writes: "we collected twice as many pre-training tokens — covering three times more languages." This pushes deep into the overtraining regime</strong> (article 21).</p>
Llama 3.1 trained on 15.6T tokens for the 405B model</strong>, which Meta describes as roughly compute-optimal under their FLOP budget of 3.8×10²⁵.</p>
DeepSeek-V3's 14.8T is smaller — appropriate for a 37B-active MoE.</p>
Where do the tokens come from?</h2>

About

2026-05-08T00:00:00+00:00

What We Do</h2>

Why 6AI6</h2>
AI-native delivery.</strong> We embed generative AI and agentic workflows into our engineering process, achieving measurable per-engineer productivity gains that translate directly into faster delivery, lower cost, and better margins for clients.</p>
Mixed-team depth.</strong> Enterprise systems, mobile apps, game products, and AI pipelines operate on very different rhythms. We have led all of them simultaneously — and know how to staff, coordinate, and quality-control across each.</p>
Engineering discipline.</strong> CI/CD, automated testing, code quality gates, data security, and GDPR-aware delivery practices are built into our process — not bolted on at the end.</p>

Location</h2>

Company Profile

2026-05-08T00:00:00+00:00

6AI6 · Company Profile</h1>
AI-Native Company</strong></p>

Company Overview</h2>

Core Services</h2>

Delivery Capabilities</h2>
Capability</th> Detail</th></tr></thead>

Architecture</strong></td> High-concurrency, scalable, maintainable systems; microservices, cloud-native, serverless</td></tr>
Engineering</strong></td> CI/CD pipelines, automated testing, version management, canary releases, code quality standards</td></tr>
AI-Powered Development</strong></td> AI-assisted coding, agentic engineering workflows, prompt/skill asset libraries — 80%+ per-engineer efficiency gains</strong></td></tr>
Security & Compliance</strong></td> OWASP practices, data security, privacy de-identification, GDPR-aware delivery</td></tr>
Cross-border Delivery</strong></td> European client communication standards, overseas release processes, multilingual teams</td></tr>
Project Management</strong></td> Milestone-based delivery, risk tracking, transparent progress reporting, acceptance testing</td></tr> </tbody></table>

Technology Stack</h2>

Why Partner with 6AI6</h2>

Contact</h2>

Search

2026-01-14T00:00:00+00:00

Attributions

2021-06-04T00:00:00+00:00

Content management system</h2>
Zola</a> is my choice for managing content and creating a static website from it. It is your one-stop static site engine.</p>
Cascading style sheets</h2>
For styling, I use the Tailwind CSS</a> framework by Adam Wathan.</p>
Artwork</h2>
I use free artwork on this site:</p>

Free icons (MIT License) from

Heroicons</a> by Refactoring UI Inc.</li> </ul> </li>
Free images under the Unsplash License</a> from

冬城</a></li>
Abel Pérez</a></li>
Alexander Ant</a></li>
Alexander Grey</a></li>
Andrew Buchanan</li>
Andrew Stutesman</a></li>
Brett Jordan</a></li>
Christian Stahl</a></li>
Clint Adair</a></li>
Dan Gold</a></li>
Dave Lowe</a></li>
David Clode</a></li>
Hasan Almasi</a></li>
Jason Rosewell</a></li>
Jehyun Sung</a></li>
Katherine Fleitas</a></li>
Lionel Hesry</a></li>
Marcelo Marques</a></li>
Marian Kroell</li>
Marko Blažević</a></li>
Markus Spiske</a></li>
Markus Winkler</a></li>
Max Williams</a></li>
Mohammad Ali Berenji</li>
Nathan Hulsey</a></li>
Nick Fewings</a></li>
Olav Tvedt</a></li>
Omar Flores</li>
Pawel Czerwinski</a></li>
Shubham Dhage</a></li>
Sophie Vinetlouis</li>
Steve Johnson</a></li>
Sydney Rae</a></li>
Tim Marshall</a></li>
Tim Mossholder</a></li>
Vitalik Vynarchyk</a></li>
Vivaan Trivedii</a></li> </ul> </li> </ul>
Some Unsplash profiles are no longer available.</p>
Libraries and assets</h2>

KaTeX</a> (MIT License) for math rendering.</li>
Mermaid</a> (MIT License) for diagrams.</li>
Howler.js</a> (MIT License) for the custom audio player.</li>
MiniSearch</a> (MIT License) for client-side search.</li>
Artalk</a> (MIT License) for optional comments.</li>
Giallo</a> (EUPL-1.2 License) for syntax highlighting.</li> </ul>

Terms

2020-01-08T00:00:00+00:00

Last updated: February 9, 2026</em></p>
General character of the website</h2>
This website is a private, non-commercial blog</strong> with mostly informational content. It can be used without registration</strong> and without providing personal data</strong>.</p>
The following notes are for transparency and clarity. They are not general terms and conditions</strong> and not a contract</strong> in a legal sense.</p>
Content and opinions</h2>
The content published on this website reflects my personal views and experiences</strong>. It is not professional advice</strong>, in particular not legal, tax, medical, or financial advice.</p>
I aim to provide correct and current information but make no warranties</strong> regarding accuracy, completeness, or suitability for a specific purpose.</p>
Liability</h2>
Liability for damages resulting from the use or non-use of the information provided on this website is excluded, unless caused by intent or gross negligence</strong>.</p>
Mandatory statutory liability remains unaffected, especially for injury to life, body, or health.</p>
External links</h2>
This website may contain links to external websites of third parties. I have no influence on their content.</p>
The provider or operator of the linked pages is always responsible. At the time of linking, no illegal content was apparent. If legal violations become known, the links will be removed promptly.</p>
Copyright and use of content</h2>
The content and works published on this website are subject to German copyright law.</p>
Unless stated otherwise, copying and redistribution</strong> are permitted as long as the author's attribution notice remains</strong> and the use is not commercial.</p>
Any further use requires my prior consent.</p>
Privacy</h2> Information about the processing of personal data and browser storage can be found in the privacy policy</a>.</p>
Applicable law</h2>
These notes and the use of this website are governed by the laws of the Federal Republic of Germany.</p>
Closing note</h2>
If any part of these notes is or becomes invalid, the remaining content remains effective.</p>

Imprint

2019-03-11T00:00:00+00:00

6AI6</em></p> </blockquote>
Copyright 6AI6.</p>
Liability for content</h2>
As a service provider, I am responsible under the general laws for my own content on these pages. However, I am not obliged to monitor transmitted or stored third-party information or to investigate circumstances that indicate illegal activity. Obligations to remove or block the use of information under the general laws remain unaffected. Liability is only possible from the time of knowledge of a specific infringement. If I become aware of violations, I will remove the content immediately.</p>
Liability for links</h2>
This website contains links to external websites of third parties. I have no influence on their content. The provider or operator of the linked pages is always responsible. At the time of linking, no illegal content was apparent. Permanent monitoring of the content of the linked pages is not reasonable without concrete evidence of an infringement. If I become aware of violations, I will remove such links immediately.</p>
Copyright</h2>
The content and works created by me on these pages are subject to copyright law. Reproduction, editing, distribution, and any kind of use outside the limits of copyright law require my written consent. Downloads and copies of this page are permitted only for private, non-commercial use. If the content on this page was not created by me, the copyrights of third parties are respected and such content is marked. If you notice a copyright infringement, please inform me. If violations become known, I will remove such content immediately.</p>
Attributions for third-party assets used on this site are listed here</a>.</p>

Privacy Policy

2019-03-11T00:00:00+00:00

General notes</h2>
Protecting your personal data matters. This website is a static information site</strong> and can be used without providing personal data</strong>.</p>
I do not</strong> actively collect personal data unless optional features are used, in particular:</p>

no contact forms</li>
no tracking</li>
no analytics or marketing services</li>
no comments or user accounts without explicit consent</li> </ul>
Responsible controller</h2>
The responsible controller for data processing on this website is the person named in the imprint</a>.</p>
Hosting</h2>
This website is hosted by an internet hosting provider</strong>.</p>
The web server stores information in its log files, some of which your browser transmits:</p>

IP address</li>
date and time of the request</li>
requested page</li>
HTTP status code</li>
referrer URL</li>
browser and operating system information</li> </ul>
This data is used only for technical operation</strong>, security</strong>, and error analysis</strong>.</p>
Legal basis:</strong> Art. 6(1)(f) GDPR
Legitimate interest:</strong> secure and stable operation of the website</p>
The log data are stored separately and not used for profiling.</p>
Data processing</h2>
I have signed a data processing agreement with the hosting provider under Art. 28 GDPR.</p>
Cookies and browser data storage</h2>
This website does not use cookies.</p>
Browser storage is used in these cases:</p>

After an active click on the theme toggle</strong>, to store the display setting (`light</code> or dark</code>) for this session in the browser (sessionStorage</code>). Legal basis: Section 25(2) No. 2 TDDDG.</li>`
When using the comment function, to store user-provided data and technical session information for later comments in the browser, e.g. username/nickname, email address, optional website (localStorage</code>). Legal basis: Section 25(1) TDDDG and Art. 7 GDPR.</li> </ul> The data storage takes place exclusively locally in the browser. Entries can be deleted there at any time.</p> External content and services</h2> This website does not</strong> embed external content (e.g., YouTube, Google Fonts, tracking scripts). All resources (scripts, images, etc.) are served locally.</p> SSL/TLS encryption</h2> This website uses SSL/TLS encryption. Data transmitted during a page request cannot be read by third parties.</p> Rights of data subjects</h2> You have the right to:</p> access to stored personal data</li> correction or deletion</li> data portability where applicable</li> restriction of processing</li> objection to processing</li> </ul> These rights mainly apply to technically necessary server log data. If you use the comment function, the data you enter (e.g. name/nickname, email address, optional website) and the comment content may also be processed. In that case, the same rights apply. On request, a comment can be deleted or anonymized unless statutory retention obligations prevent this.</p> Requests can be sent via the contact address in the imprint.</p> Supervisory authority and right to file a complaint</h2> You have the right to file a complaint with a data protection supervisory authority. The competent authority is the authority of your federal state of residence or the seat of the responsible controller.</p>