Overview
What this is
The compact read before the technical details.
A ground-up implementation of Qwen3 0.6B, trained on the fineEDU dataset to explore LLM internals. Built in PyTorch from raw tensors — tokenization, embeddings, rotary positional encodings, grouped-query attention, and the full transformer stack. Trained to completion to validate the architecture and understand every component of a modern decoder-only language model.
Capabilities
What it actually does
The useful parts, pulled out of the paragraph wall.
Ground-up implementation of a 0.6B-parameter Qwen3 transformer — 28 layers, 1024-dim embeddings, 151K vocabulary — with no high-level nn.Module abstractions hiding the internals
Implements Grouped Query Attention (GQA) with 16 query heads and 8 KV heads, enabling efficient KV caching for autoregressive inference without the full multi-head attention memory cost
Includes RoPE (Rotary Position Encoding) with a base frequency of 1M for long-range dependency modeling, plus SwiGLU FFN with ~3x expansion and bias-free linear layers
Pre-norm architecture with RMSNorm applied before attention and FFN sub-layers, plus additional Q/K normalization for training stability at scale
Implementation
Technology with jobs attached
Names are less useful than responsibilities. This is what each piece is doing.
PyTorch
Implementation framework — raw tensor operations for attention, normalization, positional encoding, and forward pass
RoPE
Rotary position encoding applied to Q/K vectors for extrapolation-friendly sequence modeling up to 4096 tokens
GQA
Grouped Query Attention — 16 Q heads share 8 KV heads, reducing memory bandwidth during inference
SwiGLU FFN
Gated feed-forward network with SiLU activation, expanding 1024→3072→1024 per transformer block
RMSNorm
Root Mean Square normalization replacing LayerNorm for improved training stability without mean centering