Current guardrails for code generation allow nearly all malicious code requests through.

People can trick large language models, or LLMs, into writing malware, which is computer code made to cause harm. Security tools called guardrailsguardrailsSafety filters that block harmful outputs from AI models. try to stop models from answering bad requests, but nobody knew how well they worked for code. The researchers created CS-Guard to test these defenses. They tested whether models write malicious code from plain instructions or from existing code snippets. They also invented a test that hides bad intent inside a harmless, pretend software project. They found that current guardrails perform poorly. Sneaky prompts bypass defenses about 50% of the time on text requests. Starting from existing code lets attacks succeed between 14.4% to nearly 100% of the time, and the pretend project attack succeeds close to 100% of the time across many guardrails.

Why it matters

If you use an AI assistant to write code, current safety filters often fail to block requests to generate malware. This means developers could unknowingly produce malicious code, and attackers could exploit these tools. The benchmark exposes these weaknesses, urging better guardrails before these tools are widely adopted.

Questions answered