Most verifier errors come from extra spaces or periods, not complex issues.

Computers learning math use automated verifiers to check whether answers are correct. These grading programs turn text answers into a simple pass or fail reward. However, grading code often marks correct answers wrong. To see why, researchers tested four popular verifiers across 307,420 verdicts. They did this by rewriting correct answers into equivalent forms that mean the exact same mathematical thing. Because the meaning never changes, any rejection proves the checker made a mistake without needing human review. The researchers found huge problems. First, verifier accuracy on identical inputs swings wildly, from 53.8% to 95.2%. Second, most mistakes are surprisingly silly: for the default LaTeX setup, whitespace and punctuation cause 93.0% of valid failures, dominated by a simple trailing period or newline. Finally, verifiers fail for opposite reasons, and one setup accepts totally wrong, off by one numbers once values reach 10^4 or higher.

Why it matters

This matters for researchers training AI, as small formatting errors can invalidate correct answers, skewing results and wasting resources.

Questions answered