Qwen3 0.6B From Scratch

Overview

A 0.6B decoder-only transformer written without hiding behind high-level wrappers. 28 layers, 1024-dim embeddings, 151K vocab. Implements Grouped Query Attention, RoPE, SwiGLU FFN, and pre-norm RMSNorm so every piece of a modern LLM is visible and inspectable.

Highlights

• 28 layers, 1024-dim, 151K vocab — no black-box modules
• GQA: 16 query heads sharing 8 KV heads for cheaper inference
• RoPE base 1M, SwiGLU ~3× expansion, bias-free linears
• Pre-norm RMSNorm plus Q/K norm for training stability

Stack

PyTorch

Raw tensor attention, FFN, norms

RoPE

Rotary positions for long context

GQA

Grouped query attention

SwiGLU

Gated feed-forward blocks

RMSNorm

Pre-norm without mean centering