Comparing tokens to a moving average instead of absolute magnitude sharpens expert routing and improves accuracy.

Some AI models use a Mixture-of-Experts setup, where smaller sub-networks, called experts, handle different pieces of data. Usually, a router decides which expert gets each token, meaning each piece of text. But these decisions rely on raw representations that share too much background information across all tokens, which keeps experts from truly specializing. To fix this, researchers built the Contrastive Routing Mechanism, or CoRM. Instead of looking at raw token size alone, CoRM compares each token against a running average of the layer's states. It scores each expert on the gap between the token and that shared baseline. This focuses the routing decisions, aligns expert boundaries with linguistic structure, and boosts reasoning accuracy across nine benchmarks while adding very little computational cost.

Why it matters

This improves language models used in AI assistants, making them more accurate at reasoning tasks without significant extra computing cost. It benefits anyone relying on AI for problem-solving by providing more reliable outputs.

Questions answered