2 posts with tag architecture

• llm
• mixture-of-experts
• architecture
• deepseek-v3
• qwen
• llama

Fine-grained experts, shared experts, auxiliary-loss-free routing — the modern MoE recipe in 2026, with side-by-side comparison of DeepSeek-V3, Qwen 3, Llama 4.

• llm
• attention
• architecture
• deepseek-v3
• kv-cache
• mla

How DeepSeek's Multi-head Latent Attention compresses the KV cache via low-rank projections + decoupled RoPE, achieving large memory reductions versus MHA at equal or better quality.