The right teacher for each task isn't always the domain expert.

Modern large language models often train on specific subjects, but combining several specialized models into a single student model is hard. Older methods let topic tags choose the teacher model. Yet, an expert model can still make mistakes on specific questions, while an outside model might get them right. Researchers created a method called Multi-Teacher Self-Distillation Policy Optimization to solve this. Instead of trusting subject labels, it checks each teacher's answer first. A teacher can only guide the student if its answer passes a verification test. The method also uses correct student answers as guides and combines all verified answers into a special context that only an internal helper model reads during training. Testing showed strong results. For a model called Qwen3-8B, the method boosted its weakest subject score by 14.79 points and shrank its performance gap across subjects by 74.7%, achieving a better balance than using fixed subject teachers.

Why it matters

This helps developers build single AI models that perform well across different topics, making them more versatile and reliable for users by ensuring the best advice is used for each specific problem.

Questions answered