67% of material changes to AI safety frameworks are not disclosed.

Leading artificial intelligence companies release safety frameworks. These documents promise how they will test if their models are dangerous. Governments in the European Union and California now use these frameworks to hold developers accountable when policies change. However, rules do not require companies to explain clearly what they changed. To study this, researchers tracked every public update across twelve companies. They compared the actual policy updates against what the companies publicly announced, tracking 710 specific commitments across twelve pairs of versions. They found that companies silently make most of their important changes. Across these updates, 67% of material changes were never disclosed in developer announcements under a strict standard. Even worse, 77% of traced changes weakened or erased existing safety promises. Companies also hid rollbacks far more often than new protections. The researchers argue that laws must force companies to list every single change, not just explain why they made updates.

Why it matters

When AI developers promise to keep their systems safe, regulators and the public rely on those promises being kept. This study shows that two-thirds of changes to those promises are not reported. Without knowing what changed, oversight is impossible.

Questions answered