Common knowledge distillation slows factual recall during mid-training while boosting reasoning.

Smaller language models learn from larger teacher models through a technique called knowledge distillation. Researchers wanted to know if this learning process works the same way across different training stages. They ran experiments during mid-training, an intermediate step where models train on curated texts. They discovered that while distilling from a teacher during initial pre-training improves both reasoning and factual recall, doing it during mid-training actually slows down factual recall. This happens because teachers show more confidence on procedural steps than on fact-heavy data, while students learn facts earlier on. To fix this problem, the team created Switch Distillation. This method checks the teacher's certainty token by token. If the teacher is confident, the student learns from the teacher; otherwise, it learns from standard text prediction. This approach preserves factual recall while continuing to boost reasoning and knowledge.

Why it matters

Smaller language models are cheaper to run but often less capable. This research shows a way to train them so they reason well without losing factual knowledge, making AI tools more efficient and reliable.

Questions answered