The QA-SFT checkpoint, further trained with DPO against a frozen reference copy of itself, so it learns to prefer the better answer directly from preferences.

Trained on Modal L4. Judged correctness 0.054 [0.038–0.071] — DPO regressed the 125M below its own SFT (paired win-rate just 33% vs SFT): answers collapsed to ~2 words and fabrication rose to 31%. A cautionary result — direct preference optimization destabilized the smallest model.

Architecture

Class	LlamaForCausalLM (from scratch)
Layers	12
Hidden size	768
Attention	12 heads · dim 64 · full MHA
Feed-forward	SwiGLU · inner 3,072
Positional	RoPE · θ 10,000
Norm	RMSNorm · ε 1e-5
Context	1,024 tokens
Vocabulary	16,384 · byte-level BPE
Embeddings	tied input/output

Training

Init from	slm-125m QA-SFT
Method	DPO (full fine-tune)
Trainable params	125.8M — all
Training data	500 preference pairs
Training tokens	344K (chosen + rejected)
Optimizer steps	375

What this model cost to build

$71.84 total Modal usage

the full lineage. Unlike the 500M and Gemma sites, this base was pretrained from scratch here, so its cost is part of the total rather than someone else's.

Stage	Detail	Cost
125M pretraining (4 legs)	v1 + extended + e2 + e4 · 27.5 h on A100-40GB	$70.14
QA supervised fine-tune	the checkpoint DPO starts from · 9.7 min on L4	$0.20
DPO alignment	375 steps · 1.0 min on L4	$0.02
Evaluation (shared)	13 versions on 500 held-out questions, this model's share	$1.48
Total		$71.84

Figures are Modal GPU usage (time × rate) across this model's lineage; shared datasets are charged at this model's share. Whether base pretraining is included is stated above — it is for the models pretrained here, and excluded for imported bases. Evaluation-derived metrics come from an independent blind-judge harness on a frozen, decontaminated held-out set. Serving is billed separately and scales to zero.