Permutation symmetries can both hide and remove malware in large language models.

Because large language models are hard to train and widely used, attackers can hide malicious payloads directly inside model weights. This threat is called stegomalware. Researchers studied behavior-preserving symmetries, which are ways to shuffle an AI model's internal parts without breaking how it works. By swapping these parts around, defenders can scramble hidden data, but attackers can also use the same shuffling to hide malware. The researchers discovered that defenders can choose permutations that displace all model parameters. This fixes an issue with older defenses, which left a significant percentage of weights untouched in large language models. They also showed attackers can use these symmetries to hide malware losslessly without retraining or needing payload-specific extraction scripts. Although shuffling can accumulate numerical errors in practice, tests showed that performance loss for both attack and defense is minimal.

Why it matters

If you use a large language model, its weights could secretly contain malicious code. This paper shows that the same mathematical property that lets attackers hide that code can also be used to scrub it out, with almost no harm to the model's performance. That means safer deployment of LLMs, but also a new threat to watch for.

Questions answered