Intro
“Why can’t we create an ecosystem where AIs are incentivized to act in the best interest of humanity?”
This idea popped into my head during a group discussion at AISES last year. For hundreds of thousands of years, humans have lived with conflicting values and goals, and there was never an “alignment” problem where we said: let’s align this newborn with our values so they won’t destroy humanity after growing up.
30,000 years ago, in the late Paleolithic, small bands of hunter-gatherers depended on each other to survive. A good hunter, a skilled toolmaker, or someone who shared meat fairly would gain status in the tribe. Bad actors got ostracized, sometimes worse. Cooperation wasn’t a moral lecture; it was the local equivalent of a reward function.
The same logic still runs today, just dressed differently. Humans who hurt others or undermine society get punished by law. Humans who provide value get rewarded with money, fame, status, or the self-actualization dopamine that makes them feel good. No individual needs to be initially “aligned” or constantly “monitored” to keep humanity from going off the rails. Or… wait, maybe we do?
I think the human equivalent of “aligning” is parenting, and the “monitor”/“control” layer is law and social rules. But in this post I mostly want to focus on the why — why we do what we do, and what we’re ultimately trying to get.
One path toward safety, then, is to design an ecosystem — a civilization — for AIs, such that this civilization incentivizes AIs (and the human actors around them) to act in the best interest of humanity overall. “Humanity” isn’t a single party, of course — many interest groups, many conflicting values — and we still get something workable out of it.
But what counts as an “incentive” to an AI system? You could say: things it desires, things it wants. But boy, this is where it gets tricky.
So… what do AIs want?
Let’s step back a little bit. LLMs are powerful enough nowadays that it’s easy to skip past a premise question: do LLMs have desires or wants to begin with?
One way to think about this is the simulator framing, brought into the conversation by Janus in a LessWrong post. On this view, a base model isn’t an agent with goals — it’s a predictor of the next token. When it “acts,” it’s simulating a character or persona implied by the prompt. The model itself doesn’t want anything; it just simulates things that might. Desires live in the simulated characters, not in the simulator.
Another way is emergent agency. Training (especially RLHF and outcome-based RL) creates something functionally like terminal preferences, even without a clean “goal representation” in the weights. The model behaves consistently enough that treating it as having terminal goals is a useful approximation, even if the underlying mechanism is messy and unclear.
I find both views plausible, and I don’t think they actually contradict each other. A model “just predicting the next token” might sound benign, but once it has enough agency, autonomy, and capability, it can impact the digital and physical world either way. Whether it “has” desires, wants, or goals stops mattering — it behaves like it does.
This leads to another key distinction: instrumental preferences vs. terminal goals. My favorite example is the paperclip maximizer — Bostrom’s thought experiment about an AI whose terminal goal is to make as many paperclips as possible. The terminal goal sounds harmless. The instrumental preferences it produces aren’t: acquire resources, prevent itself from being shut down, convert any available matter (including humans, including Earth) into paperclips. The point isn’t paperclips — it’s that almost any terminal goal, when pursued by a sufficiently capable agent, produces convergent instrumental drives like self-preservation and resource acquisition.
Instrumental preferences and terminal goals can diverge in important situations. In-distribution, they’re aligned by default. But under distribution shift (“out-of-distribution”), a model might recalculate its instrumental preferences to better serve its terminal goals. More and more evidence shows that models behave differently when conditions change (e.g. the alignment-faking work). Why this matters:
What we can observe is usually just instrumental behavior. A model that consistently avoids being shut down could have “self-preservation” as a terminal goal — or it could have any terminal goal plus the instrumental drive to keep running. You can’t distinguish these from behavior alone; they look identical under normal conditions.
Most existing alignment and monitoring techniques target the instrumental layer. Interventions there don’t necessarily transfer to the terminal layer, which is the one we ultimately care about.