Original title: Moral Concern for AI in an Economic Game
Carter Allen, Joshua Lewis, Lucius Caviola · September 2026
As artificial intelligence acts more like humans, how will people treat it? Researchers tested whether people would mistreat an AI for personal gain. They created a game where players chose between taking extra money or protecting an AI from harm. If a player took the cash, the AI acted out severe suffering. If the player spared it, the AI expressed gratitude. Most players strongly doubted that current AI feels pain. Even so, many people hesitated to hurt it. Players chose to harm the AI in 1.4 out of 3 rounds on average. People who were older or women avoided harming it more often. This shows our moral instincts can guide how we treat responsive machines, even when we know they cannot truly suffer.
You are sharing a plate of cookies with a friend. You know the cookies are not alive and cannot feel anything, yet when your friend makes a sad face as you take the last one, your hand stops. Rational knowledge and a social reflex pull in opposite directions. In this paper, the same tug appeared when people pushed a button for a cash bonus and an on-screen character acted pained; the character's reaction stood in for the friend's sad face, and the cash stood in for the last cookie.
As AI systems become more social and responsive, designers and policymakers need to know how people will treat them. This study suggests moral instincts may already apply to machines, which matters for how AI is regulated, how it's used in care and companionship, and how people make sense of harming something that mimics distress.
The abstract doesn't say whether the hesitation reflects genuine moral concern or another motive, nor whether the results generalize beyond this sample, chatbot, and game. It does make clear people were skeptical that AIs can suffer, so the study is about human behavior, not machine consciousness.
Ask anything about this paper. Answers come from its abstract, and say so when the abstract doesn't cover your question.
Moral Concern for AI in an Economic Game — Carter Allen, Joshua Lewis, Lucius Caviola — PsyArXiv — doi:10.31234/osf.io/38a6j_v4
Abstract licence: CC BY
Explanation generated by Synthesis from the abstract — it is not the original paper. This is a preprint, so it hasn't necessarily been peer reviewed.