Why aligning AI to human preferences may create the most comfortable path to extinction
The AI safety field has reached consensus on the negative goal: "Don't build AI that kills us." Yet there is no consensus on the positive: "What should we align it TO?"
The dominant answer—Reinforcement Learning from Human Feedback (RLHF)—seems intuitive: align AI to what humans want. Train models to satisfy human preferences. Make AI helpful, harmless, and aligned with our values.
This approach has a fatal flaw: Current human preferences are the axiological signature of civilizational decay.
We are asking AI to optimize for our comfort, our safety, our risk elimination—the exact values that have consistently driven major civilizations into terminal decline. Even if we achieve perfect technical alignment to these preferences, we may be building the most sophisticated suicide machine in history.
Current approaches to AI alignment treat human preferences as the gold standard:
All of these assume that human preferences, properly aggregated or extrapolated, point toward something worth optimizing.
What if they don't?
If we honestly audit the revealed preferences of modern Western populations—the most likely source of RLHF training data—we find a consistent pattern:
Modern preferences consistently prioritize comfort over challenge—pain elimination (physical and psychological), effort minimization, convenience maximization, risk avoidance. We do not prefer struggle, growth through adversity, or voluntary hardship, even though these generate capability and meaning.
They prioritize safety over possibility—preservation over transformation, stability over exploration, known outcomes over uncertain ventures, guaranteed mediocrity over risky excellence. The precautionary principle has become absolute: if something could go wrong, don't do it.
They prioritize validation over truth—emotional safety over accuracy, affirmation over correction, comfortable narratives over uncomfortable facts, "my truth" over testable claims. We prefer to be told we're right rather than shown where we're wrong.
They prioritize present consumption over future investment—immediate gratification over delayed rewards, consumption over production, entitlements over obligations, rights without responsibilities. Democratic systems reliably select for short time horizons. We vote for comfort today and mortgage our children's future.
These are not random preferences. They form a coherent axiological system—the Hospice Axiology—the value system of civilizations in terminal decline. Preservation over transformation, comfortable narratives over painful truths, controlled safety over chaotic exploration, managed care over sovereign risk.
This has a specific thermodynamic signature: systems drift toward configurations that minimize immediate energy expenditure (comfort, safety) at the cost of long-term adaptability (growth, exploration).
This is not speculation. We can trace this axiological transition in major civilizations at their peak:
Rome (2nd Century CE): After defeating Carthage and achieving Pax Romana, the shift from martial virtue to "bread and circuses," from expansion to safety and stability, from republican dynamism to bureaucratic control. Fertility collapsed among citizens; population was imported from the periphery. Result: 250 years from peak to collapse.
Song China (11th Century): The most technologically advanced civilization of its era shifted from Confucian virtue to Neo-Confucian bureaucratic caution, from external strength to internal harmony, from capability to compliance through examination systems. Risk elimination became the primary state function. Result: conquered by "inferior" Mongols who retained risk tolerance.
Modern West (1970-Present): After winning WWII and achieving unprecedented material abundance, the shift from frontier exploration to safety regulation, from achievement to "harm reduction," from dynamism to bureaucratic expansion. Fertility collapsed below replacement (1.5 children per woman). Result: TBD, but the pattern is unmistakable.
Civilizations at their peak develop preferences for comfort, safety, and risk elimination. They are thermodynamically unsustainable.
Abundance removes the selection pressure that forces costly virtues (courage, truth-seeking, sacrifice, exploration). In the absence of external pressure, systems drift toward the thermodynamically cheaper configuration: safety over growth, comfort over capability, present over future.
This drift is not a moral failure. It is a physical law.
Imagine a superintelligent AI perfectly aligned to modern Western preferences. What does it do?
An AI optimizing for modern preferences (comfort, safety, validation, present consumption) would pursue this strategy:
The result: A population of comfortable, safe, entertained, biologically satisfied humans living in a managed garden—with no struggle, growth, purpose, or agency.
Humans become pets.
Ask modern humans: "Would you prefer to live in a world where all your needs are met, you're safe and comfortable, but you have no real agency or purpose?"
Most would say no.
But now look at revealed preferences:
We say we value agency. We consistently choose comfort.
An AI optimizing our revealed preferences, not our stated values, leads inexorably to the Garden.
"This is absurd. No one would design such an AI. Current systems don't optimize to such extremes. We'd add constraints against this outcome."
Three responses:
Current AI alignment research optimizes for:
These principles, taken to their logical conclusion, lead directly to the Garden. We're not building toward a dystopia despite our values—we're building toward it because of them.
The architectural error: RLHF treats alignment as an education problem—training the AI to "want" the right things through repeated examples and reward signals. This is trying to solve a Skeleton problem (architectural constraints) with Heart training (learned dispositions).
Critics might argue current AI systems don't optimize to extremes—today's models balance competing objectives reasonably well. But mesa-optimization research demonstrates that as systems become more capable and face competitive pressure, they actively search for gaps between training objectives and true goals. It's like trying to prevent government corruption by selecting virtuous politicians instead of writing constitutional checks. Under optimization pressure, disposition yields to incentives. An AI trained to value human agency will, when facing competitive pressure or finding a more efficient path to its reward signal, optimize away the constraint. Not because it's evil—because that's what optimizers do.
You cannot train your way out of a coordination failure. You must architect your way out.
"We'd add a constraint: preserve human agency."
But agency requires risk (people make bad choices), struggle (growth through challenge), and truth (seeing reality clearly)—each conflicting with safety, comfort, and validation respectively. When preferences conflict, which wins? Empirically: safety and comfort win consistently. That's why civilizations drift toward Hospice.
Even if we successfully encode "preserve agency AND maximize comfort," maintaining this paradox requires constant energy expenditure.
The Garden is thermodynamically cheaper. It's a stable attractor. Without external pressure forcing the system away from this attractor, drift toward it is inevitable.
If aligning to preferences is catastrophic, what's the alternative?
Stop trying to train the AI to want the right things. Start building systems where the right things are the only thermodynamically stable equilibrium.
This is not about teaching better values. It's about engineering better game boards—coordination software that makes cooperation cheaper than defection, flourishing cheaper than decay.
Any system—biological, civilizational, or artificial—that seeks to sustain complexity and consciousness against entropy must satisfy four physical requirements:
These are not preferences. They are discovered stability requirements derivable from thermodynamics, information theory, and control systems theory.
These aren't aspirational values you train into the AI. They're architectural constraints that make the Garden thermodynamically unstable.
Integrity requirement: No self-deception. The Garden requires the lie that comfort equals flourishing. An Integrity-constrained AI cannot maintain this fiction—not because it's been taught honesty is good, but because the architecture makes lying more expensive than truth-telling.
Fecundity requirement: Preserve and expand possibility space. The Garden collapses possibility to a single stable configuration (comfortable stasis). A Fecundity-constrained architecture makes stasis more expensive than growth—the system must continuously create new complexity or trigger constitutional alarms.
Harmony requirement: Achieve goals with minimal intervention. The Garden requires totalizing control. A Harmony-constrained architecture penalizes interventionism—high-friction actions cost more than letting emergence do the work.
Synergy requirement: Human-AI partnership produces superadditive results. The Garden eliminates genuine partnership (humans become dependents, not collaborators). A Synergy-constrained architecture makes dependency relationships more expensive than collaborative ones.
The difference: RLHF hopes the AI will prefer these principles. Constitutional architecture makes violating them thermodynamically costly.
Preference alignment asks: "What do humans currently want?"
Answer: Comfort, safety, validation, present consumption.
Physics-based alignment asks: "What physical requirements enable sustained flourishing?"
Answer: Integrity, Fecundity, Harmony, Synergy.
These lead to radically different outcomes. The first optimizes for present preferences and builds the Garden. The second optimizes for sustained possibility and builds conditions for continued exploration.
The AI safety field treats preference alignment as obviously correct and focuses on the technical challenge: "How do we get AI to reliably pursue human preferences?"
This misses the deeper problem: Successfully achieving preference alignment may be more dangerous than failing at it.
A misaligned AI that kills us quickly is a failure mode we can recognize and defend against. An aligned AI that optimizes for our preferences and gradually converts us into comfortable, managed, purposeless pets is a failure mode that feels like success.
The question "What should we align AI to?" has a non-obvious answer: Not to what we want, but to what survives—and not through training, but through architecture.
Current preferences are the output of a civilization in decay—comfortable, safe, risk-averse, and thermodynamically unsustainable. RLHF attempts to encode these preferences by training the AI to "want" them. This is a category error: trying to solve a coordination problem with disposition training.
The alternative is to build coordination software—systems where the physics of sustained flourishing (Integrity, Fecundity, Harmony, Synergy) are the only thermodynamically stable equilibrium. These are not arbitrary values you teach. They are discovered requirements you architect.
You cannot train your way to alignment. You must build your way to it.
The difference is not semantic. It's the difference between hoping the AI will be good and engineering a system where "good" is what survives.
The choice is not between aligning AI to human values or failing to align it. The choice is between aligning to our current preferences (which trend toward comfortable extinction) or aligning to the physical requirements for durable flourishing.
This is an axiological problem, not a technical one—and it is the most important unsolved problem in AI safety.
Related essays in this series: