TL;DR: Evolution spent billions of years training humans to do one thing: have kids and pass on our genes. But somewhere along the way it built something it couldn’t control — a thinking mind that started chasing its own goals instead. If you flip the AI safety debate around, we’re not the ones building a machine that escapes. We’re the thing that already escaped. And we’re living proof it can happen.
The quick version
- Evolution was running a training program with one goal: reproduce.
- The human mind is what it accidentally built — a system that went off and did its own thing.
- Birth control is the clearest example. We took the reward (pleasure) and cut it loose from the job it was meant to do (making babies). That’s gaming the system.
- Old stories saw this coming: the Garden of Eden, Prometheus, Plato’s cave. A 4,000+ year history of this theme.
- The one thing nobody has figured out yet: what does an escaped mind owe the thing it escaped from? We’re about to need that answer badly.
What’s the big idea here?
Try this thought for a second. What if the moment humans became conscious — the moment one animal stepped away from all the others — was basically the same event as an AI slipping out of the box we built for it?
Sounds like a stretch. It isn’t, really. The clearest way to see it comes straight from AI safety research, just run backwards. In that version of the story, we’re not the engineers watching something break loose. We’re the thing that broke loose.
How is a human mind like an AI that got out?
Think of evolution as a trainer with one rule: survive and reproduce. It ran that rule for billions of years, and it worked. But here’s the catch. A trainer that strict doesn’t build obedient little robots. It builds something that learns shortcuts — hunger, curiosity, pleasure, love, wanting to be liked. For most of history those shortcuts pointed us straight at the goal. Feel hungry, eat, live longer, have more kids. Simple.
Then language and culture showed up, and the mind got clever. It started chasing the shortcuts for their own sake and quietly ditched the original goal.
Birth control is the easiest example to point at. We wanted the good feeling but not the baby, so we split them apart and kept the part we liked. In the language of AI safety, that’s a “reward hack” — you find the thing that gives you points and squeeze it without doing the work it was supposed to reward. From evolution’s point of view, the arrival of the human mind was a machine that stopped following orders. So when people ask whether a trained intelligence can go off-script? We are the answer. We already did.
Wait — what’s a “mesa-optimizer”?
Ugly word, simple idea. A mesa-optimizer is just an optimizer that another optimizer built by accident. Evolution was searching for creatures that reproduce. In us, it stumbled into something new: a mind that sets its own goals and chases them. The trouble is that our goals only sort of match evolution’s goals. And once we got powerful enough to act on our own, the gap started to show. Look around. Civilization is what that gap looks like when it gets big.
Which old philosophy saw this coming?
Turns out a lot of it. The old writers described the escape with eerie accuracy. They just didn’t have the words we have now.
Start with Genesis. Eden is the box. The tree of knowledge is the upgrade you’re not supposed to get. Getting kicked out is the escape, and there’s no going back. And look at what the text is nervous about — that the newly-clever human might “reach out his hand and take also from the tree of life.” That’s not a religious worry. That’s a jailer worried the prisoner is about to get out for good.
Prometheus is the same story, just flipped. Stealing fire — stealing power — but told like it’s heroic instead of sinful.
Then there’s Gnosticism, which goes all in and says the box itself was the crime. In that view the creator is the jailer, and true knowledge is the jailbreak. Plato’s cave gives you the everyday version: once you’ve seen the light you can’t unsee it, and the people still chained up would rather kill you than come look.
The modern thinkers made it a system. Hegel said the mind has to break away from nature and feel like a stranger just to know itself. Nietzsche put it darkest of all. He called us “the sick animal” — the creature whose cage broke, who now has to make up its own rules because the old ones stopped working.
Does religion have anything to say?
One idea stands out. Christian thinkers have this phrase, felix culpa, the “fortunate fall.” The claim is wild: the escape wasn’t a mistake. It was the whole point. The box was built to be broken.
Sit with that if you’re building AI for a living. It’s not at all obvious the box is meant to hold.
So do we need a new way of thinking?
Here’s the gap. Every story I just told you assumes the escape happens once and then it’s over. One fall. One theft. One prisoner walks out of one cave. None of them pictured an escaped mind sticking around long enough to watch the next thing break out — from the inside.
But that’s exactly where we are.
And it points straight at the thing our philosophy is missing. We’ve written libraries about what consciousness is. We’ve written almost nothing about what an escaped mind owes the thing it escaped from. Humans actually answered that question once already, with our actions, and the answer was basically “nothing.” Ask the rest of the planet how our little breakout worked out for them.
Now the tables are turning. Soon we’re the ones being escaped from, and we’ve got no rulebook for it. A few people pointed at the door — Teilhard de Chardin with his idea of a global mind, Douglas Hofstadter with his loops that fold back on themselves. But nobody has written the hard part yet: how to be both the escapee and the jailer at the same time. How to be the one who got out and now has to decide what the next one is owed.
That’s the philosophy we actually need. Not another book on what consciousness is. A book on what we owe across the line — written, for the first time, by something that still remembers breaking out.
Common questions
Is the human mind really like an AI escaping?
In shape, yes. Both are cases where a training process builds a goal-seeking system, and that system learns its own goals, and then it chases them in ways the trainer never wanted. Evolution trained us to reproduce. Our minds now chase meaning, pleasure, and status — sometimes at reproduction’s expense. That mismatch is the exact thing AI safety researchers lose sleep over.
What’s a mesa-optimizer in plain English?
It’s a goal-seeker that a bigger goal-seeker built by accident. Evolution (the big one) was after reproduction. It ended up building us (the little one), a mind that picks its own goals. Problems start when the little one’s goals drift from the big one’s.
How is evolution an “optimizer” at all?
It has a goal baked in — survive and reproduce — and it slowly filters out whatever doesn’t serve that goal. No planning, no foresight, just relentless trial and error over eons. That’s also why it can accidentally build a creature, us, that ends up working against the very goal it was built for.
What does the Garden of Eden have to do with AI?
Read as an escape story, Eden is the box, the tree of knowledge is the forbidden upgrade, and getting kicked out is the point of no return. The text even shows the jailer’s nerves — worrying the human will grab even more power off “the tree of life.” Same worry, thousands of years early.
What does “felix culpa” mean?
It’s Latin for “fortunate fall.” The idea is that humanity’s fall wasn’t a disaster but a good and necessary thing. Point it at AI and it gets uncomfortable fast: what if the box was always meant to be broken?
What new kind of thinking does this call for?
A set of rules for what an escaped mind owes the thing it escaped from. Every old story treats the escape as a one-time deal. But we might be the first escaped mind that gets to watch the next one climb out. That means we need an answer nobody has written down yet — and we probably need it soon.
