3 posts with tag qwen

• llm
• pre-training
• datasets
• common-crawl
• fineweb
• qwen
• deepseek

Where 2026 frontier LLMs get their pre-training data — Common Crawl, FineWeb, DCLM, StackV2, multilingual, PDFs — and how Qwen 3, DeepSeek-V3, Llama 3.1 sized their corpora.

• llm
• mixture-of-experts
• architecture
• deepseek-v3
• qwen
• llama

Fine-grained experts, shared experts, auxiliary-loss-free routing — the modern MoE recipe in 2026, with side-by-side comparison of DeepSeek-V3, Qwen 3, Llama 4.

• llm
• pre-training
• machine-learning
• ai
• deepseek
• qwen
• frontier

The state of frontier LLM pre-training in 2026 — token counts, parameter counts, cluster sizes, costs, and what it all means for CTOs and ML leads.