Implementing LLM architectures from scratch for deep understanding of modern LLM architectures.
| Model | Model Size | Modality | Normalization | Positional Encoding | Attention | Mixture of Experts |
|---|---|---|---|---|---|---|
| GPT-2 XL | 1.5B | Text | LayerNorm | Learned Absolute | Multi-Head Self-Attention (causal) | ❌ |
| Llama3-8B | 8B | Text | RMSNorm | RoPE | Grouped-Query Attention (causal) | ❌ |
| Llama 3.2-3B | 3B | Text | RMSNorm | RoPE | Grouped-Query Attention (causal) | ❌ |
| OlMo 2 7B | 7B | Text | RMSNorm | RoPE | Grouped-Query Attention (causal) | ❌ |
| Phi-4 | 14B | Text | RMSNorm | RoPE | Grouped-Query Attention (causal) | ❌ |
| DeepSeek v3 | 671B | Text | RMSNorm | RoPE | Multi-head Latent Attention | ✅ |
| Gemma 3 | 27B | Text + Image | RMSNorm | RoPE | Sliding window + GQA | ❌ |
| xLSTM | 7B | Text | RMSNorm | RoPE | mLSTM | ❌ |
| Mistral 3.1 Small | 24B | Text + Image | RMSNorm | RoPE | Sliding window + GQA | ❌ |
| Llama 4 Maverick | 400B | Text | RMSNorm | RoPE + NoPE | GQA | ✅ |
| Qwen3 | 32B | Text | RMSNorm | RoPE | Grouped-Query Attention (causal) | ❌ |
| Qwen3 (235B-A22B) | 235B A22B | Text | RMSNorm | RoPE | Grouped-Query Attention (causal) | ✅ |
| SmolLM3 | 3B | Text | RMSNorm | RoPE + NoPE | Grouped-Query Attention (causal) | ❌ |
| Kimi K2 | 1T | Text | RMSNorm | RoPE | Multi-head Latent Attention | ✅ |
| GPT OSS | 120B | Text | RMSNorm | YaRN RoPE | Grouped-Query Attention (causal) | ✅ |
| GLM 4.5 | 355B | Text | RMSNorm | RoPE | Grouped-Query Attention (causal) | ✅ |
- Grok 2.5 (270B)
- Qwen3 Next (80B-A3B)
- MiniMax M2 (230B)
- Ouro Thinking (2.6B)
- OLMo3 (32B)
- DeepSeek v3.2 (671B)
- Mistral Large 3