Top security papers average over five benchmarking flaws each.

Microarchitectural side-channel attacks steal private secrets by watching physical clues like timing or memory use inside computer chips. To show that a new attack is dangerous, scientists must measure how well it works. However, small shifts in hardware can drastically change these tests, and researchers often copy outdated standards or directly reuse raw numbers from older studies. To see how reliable these tests really are, the authors examined eighty-three attack papers from top computer conferences published between 2014 and 2024. They identified 19 repeated benchmarking flaws that hurt how complete, relevant, sound, and reproducible these studies are. Common mistakes include unfair or missing comparisons, failing to release code or research materials, and skipping tests on key attack traits. The authors found that papers averaged 5.5 flaws each. Finally, they provide a guide detailing the key properties needed to evaluate future attacks properly.

Why it matters

This matters because security researchers depend on benchmarks to compare attacks and trust results. Flawed tests make it hard to know if new methods truly improve protection, potentially slowing advances in keeping computers safe.

Questions answered