06/19/2026
Earlier this year, Anthropic released a “constitution” for Claude, its large language model and flagship product; Anthropic CEO Dario Amodei has said “we’re open to the idea” that AI could be conscious; and Anthropic’s in-house philosopher, Amanda Askell, said in an interview, “I want Claude to be very happy.” https://theatln.tc/AZ0GeP61
“It’s enough to make you wonder: Should we seriously consider the possibility that Claude, or any large language model, might be conscious? And if it has feelings, is it capable of receiving moral instruction?” Ted Chiang asks. “Absolutely not.”
“LLM conversations are cleverly disguised examples of sentence continuation,” Chiang writes. Perhaps the most fruitful way to understand Claude’s constitution “is as an 83-page character sheet for a role-playing game. LLMs can generate dialogue for Julius Caesar because many books about him exist in the training data those models used. Claude’s constitution serves a similar role for delineating the helpful-chatbot character that customers interact with when they’re using Anthropic’s products.”
“The result is a sentence-continuation machine that is likelier to emit sentences resembling those that a thoughtful, moral person could utter,” Chiang continues. “However, for all the times that ‘honesty’ is mentioned in Claude’s constitution, I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences including any sentences using first-person pronouns.”
“Whenever a person delegates a decision to an LLM, they are trying to off-load accountability for that decision, and if a company that sells an LLM portrays the product as having a moral center, it is offering a way for its customers to abdicate their responsibilities,” Chiang writes. “Off-loading tasks such as writing code might result in cognitive atrophy over the long term, and that is problematic in itself, but off-loading ethical decisions will result in an atrophy of moral reasoning, which is worse.”
“It’s fortunate that LLMs are not conscious,” Chiang continues, “or else the actions of the big AI firms would be even more scandalous than they already are.”
