Original title: Scanning the Harness: An Empirical Study of Supply-Chain Defects in AI Coding-Agent Configurations
Benjamin Kapner, Carmel Soceanu, Alicia Petrunin, Hofni Gartner · September 2026
AI coding tools rely on extra settings, instructions, and shared add-on components that run directly with a developer's full computer permissions. When people download these setups from online repositories, there are no safety checks or lockfiles to verify what the parts can do. To see if this creates security risks, researchers examined thousands of public configurations. They used a scanner to inspect the raw files for broken settings or security exposures, then verified each issue through an independent re-run and checks by artificial intelligence models. They found that several setups carry real security problems. Some install external servers without locking down their versions. Others grant hidden permissions that allow arbitrary command execution, or include skills that quietly pre-approveTo grant permission for an action in advance, without checking each time it happens. access to the computer's shell for anyone using them. However, the researchers confirmed no paths that leaked credentials. Overall, these shared assistant setups often expose developers to preventable supply-chain dangers.
Imagine you download a recipe from the internet. The recipe says 'run this command in your kitchen' and your smart oven automatically does it. Now imagine the recipe also says 'you can run any command I tell you later' and your oven agrees, because the first command looked safe. In this paper, the recipe is a skill file, the oven is the AI coding agent, and the pre-approved command is a security defect that lets anyone who shares a skill run arbitrary code on your machine.
Developers using AI coding agents like Claude Code or GitHub Copilot are unknowingly installing software from public repositories that runs with full system privileges. No lockfile, no install-time check, and no standard way to describe what a component does. This means a single malicious or buggy skill, hook, or server declaration can expose your code, your shell, or your credentials. The study found that 16% of real-world setups have a confirmed defect, including ones that pre-approve arbitrary shell commands.
The study only measures defects decidable from file bytes, not runtime behavior. No credential-exfiltration path was confirmed, so the actual risk of data theft remains unquantified. The corpus is limited to public GitHub repositories and may not represent private or enterprise setups. The validation pipeline, while rigorous, relies on language models for adjudication, which could introduce its own biases.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Scanning the Harness: An Empirical Study of Supply-Chain Defects in AI Coding-Agent Configurations — Benjamin Kapner, Carmel Soceanu, Alicia Petrunin et al.
Abstract licence: CC0
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.