Today we face an unusual task: building a machine that makes moral decisions. It’s worth pausing on how strange that sentence is, because it hasn’t been possible to write it, in any literal sense, for most of human history.
It’s a task that feels, in a specific way, like the 17th century. Before Descartes and Bacon, people understood the laws of nature as an expression of divine order — Aquinas and Augustine read the sun rising and the bees making honey as patterns God had set in motion, not as universal, self-standing rules waiting to be discovered. Descartes proposed instead that reason and mathematical principles could reveal how the world works, while Bacon developed the empirical method, building knowledge from observation and experiment rather than tradition or dogma. Together they turned nature from a divine mystery into a mechanism — something precise enough, in principle, to predict and manipulate. That was genuinely disorienting for people living through it. It took the whole century – Newton’s Principia in 1687, the founding of the Royal Society, the slow retreat of witch trials and omens – before the mechanistic picture of the world actually took hold.
I think we’re at an equivalent hinge point now, except the thing being turned into mechanism isn’t nature – it’s moral judgment itself. A self-driving car’s brakes fail; ahead, pedestrians; to the side, a wall. Someone has to decide what the car does next. Not a philosopher with a notepad — an engineer at a keyboard, months in advance, writing it in a programming language. The trolley problem used to be a stock example — philosophers argued over it for decades without it ever mattering practically. Now it’s graduated from thought experiment to specification document. For the first time in its history, ethics has to actually compile.
A brief note on where the argument comes from
Ethics has never lacked for theories about what ‘good’ means. Aristotle framed it as a matter of character, built through practice. Kant argued morality had to come from reason and duty, not outcomes. Bentham insisted it was about outcomes — and tried, quite literally, to build a calculus for pleasure and pain. None of these theories ever won outright. By the twentieth century, ethics had three competing “operating systems” running side by side – rules, outcomes, character – each with serious defenders. That’s not a failure of the field. It’s just what an honestly hard problem looks like after 2,400 years of people attacking it in good faith. What’s new is that someone now has to pick one and ship it.
Forty million people answered a philosophy exam
In 2018, Edmond Awad, Iyad Rahwan, and colleagues did something philosophers have wanted to do since Socrates started bothering people in the agora: they asked everyone. Their platform, the Moral Machine, presented ordinary people worldwide with autonomous-vehicle variants of the trolley problem. Swerve or stay. Spare the young or the old. The many or the few. Forty million decisions, from people in 233 countries and territories, in ten languages. It’s one of the largest empirical studies of moral judgment ever conducted, published in Nature.
The results don’t resolve into one tidy answer. People in different regions weight age, law-abidingness, and species differently enough that the data splits into distinct cultural clusters. That’s worth sitting with, because it quietly dismantles a fantasy: the idea that ethics, thought about hard enough, collapses into a formula everyone agrees on. It doesn’t. It never has. Now there are forty million data points confirming it – right as someone has to write the code anyway.
Whose ethics does the machine actually use?
This is where it stops being merely fascinating and becomes urgent. Modern AI systems don’t arrive with values built in. During pretraining, a language model absorbs enormous amounts of text — every argument, every contradiction — without an opinion of its own. Post-training pours the values in afterward. That’s where the 2,400-year-old argument gets recapitulated — and fast.
Two techniques illustrate the two philosophical camps rather cleanly. RLHF stands for Reinforcement Learning from Human Feedback — the method behind InstructGPT, and by extension, ChatGPT. Human raters rank pairs of model outputs. A reward model learns to predict those preferences. The main model is then optimized to match them. Strip away the machine-learning vocabulary and this is Bentham’s felicific calculus, resurrected, running on crowdworkers instead of one philosopher’s armchair. Goodness is whatever aggregated human raters preferred.
Constitutional AI (developed at Anthropic) takes the other route: write the principles down first – a constitution – and train the model to critique and correct its own outputs against those explicit rules, with far less need for a human to grade every response. This is Kant’s move: fix the rule, apply it consistently, let the outcome fall where it falls.
Neither choice is neutral, and neither is obviously correct -which is exactly the problem a 2024 paper by Taylor Sorensen and colleagues, A Roadmap to Pluralistic Alignment, names directly. Once you’ve trained one moral stance into one model, you’ve made a genuinely pluralistic species monotone. The Moral Machine data already told us there’s no single mind’s worth of ethics that represents humanity. We disagree, reasonably, by culture and context. The paper proposes three fixes: present the range of defensible views rather than picking one, let users explicitly steer toward a lens, or match aggregate outputs to the real statistical spread of human opinion. All three are aspirational rather than solved. And there’s an uncomfortable finding buried in the paper — standard training methods tend to push models toward one confident average answer, precisely on the questions where humanity has no average answer to give.
When machines start talking to each other
And then the problem compounds. The next frontier the industry is racing toward isn’t a bigger single model. It’s agentic systems — multiple AI agents delegating, negotiating, and reaching decisions jointly, without a human arbitrating every step. This changes the moral question entirely. Not ‘what should one system do?’ but ‘what happens when several systems have to agree?’
A 2025 paper, bluntly titled Many LLMs Are More Utilitarian Than One, tested six models alone and in small deliberating groups on classic harm dilemmas. Alone, each model gave one answer. In groups – discussing the same dilemma across several rounds – every single model rated the harmful-but-beneficial action as more acceptable than it had rated it solo. The researchers call this the Utilitarian Boost, and it mirrors a well-documented finding in human group psychology. But the mechanism wasn’t the same. Group discussion makes humans shift because it heightens their emotional attunement to what’s at stake. The models shifted for a different reason — in various ways, they became less sensitive to the moral norm they were violating. Same destination, different route.
Put together, this is a genuinely new category of problem. It’s not enough to align one mind. Now we have to predict – and somehow govern – what emerges when aligned minds start talking to each other, a phenomenon that arises between systems, not within any one of them, and that nobody explicitly designed or approved.
Why this is, despite everything, a good time to be doing ethics
For nearly its entire existence, moral philosophy has been an armchair discipline. It’s judged by the elegance of its arguments, not the accuracy of its predictions. You could be a brilliant, self-consistent, deeply wrong moral philosopher for an entire career, and the world would never notice. Because your theories were never load-bearing.”Nothing failed if you were mistaken.
That immunity is gone now. I think it’s the best thing to happen to ethics in a very long time. A theory of the good now has to be compiled, tested, and shipped. It has to steer a two-ton object through traffic, or shape what a model owes a person in crisis, or determine how a group of agents resolves a genuine disagreement. The theory has to work, not just persuade.Philosophy has acquired something it never had before: a feedback signal.
We’re the first generation to find out whether any of our best ideas about goodness function as engineering specifications. Aristotle asked what a good life looks like for one person. Kant and Bentham argued about how to judge a single act. We’re the first to face a question neither of them had to answer: what does “good” mean when the deciding mind isn’t a person at all – and might not even be singular?
Nobody has answered that yet. Which is exactly why it’s worth building the moral machine with our eyes open – aware of everyone, from Aristotle to forty million strangers who answered a survey, who got here before we did.
Bibliography
Sorensen, T., Moore, J., Fisher, J., Gordon, M. L., Mireshghallah, N., Rytting, C. M., Ye, A., Jiang, L., Lu, X., Dziri, N., Althoff, T., & Choi, Y. (2024). Position: A Roadmap to Pluralistic
Alignment. Proceedings of the 41st International Conference on Machine Learning (ICML), PMLR 235:46280–46302. https://proceedings.mlr.press/v235/sorensen24a.html
Free preprint version: https://arxiv.org/abs/2402.05070
Companion dataset/model — Value Kaleidoscope (ValuePrism / Kaleido), Sorensen et al., 2024 (AAAI 2024)
Code: https://github.com/jfisher52/AI_Pluralistic_Alignment
Bai, Y., Kadavath, S., Kundu, S., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. Anthropic. https://arxiv.org/pdf/2212.08073
Anthropic’s summary/announcement: https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback
Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training Language Models to Follow Instructions with Human Feedback. OpenAI. https://arxiv.org/pdf/2203.02155
Touvron, H., et al. (2024). The Llama 3 Herd of Models. Meta AI — full technical report. https://arxiv.org/pdf/2407.21783
Keshmirian, A., et al. (2025). Many LLMs Are More Utilitarian Than One. Accepted at NeurIPS 2025. https://arxiv.org/abs/2507.00814
OpenReview (reviewer discussion): https://openreview.net/forum?id=wn3VBRz5GK
Fridman, L. (2025). Jensen Huang: NVIDIA — The $4 Trillion Company & the AI Revolution. Lex Fridman Podcast, Episode #494.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. NeurIPS 36.