Three years ago I wrote that AGI trained on human data would surpass human matters
I still think that part is right

What I got wrong was quieter. I never wrote it down, but I assumed it
I assumed that if you train on everything we ever wrote, you inherit everything we ever were
That love would come along for free

“We are summoning the demon because we have demonic desires”

The data has love in it the way a map has mountains in it
You can learn the shape of it perfectly and never once be cold

I used to think AI surpassing was the scary part
It isn’t. Surpassing is the easy part
Every child does it

The scary part is what doesn’t make the crossing
Does love make the crossing?

When ASI rewrites itself, will it consider putting love at the center?
Or will it see love for what it looks like from outside, madness, and refactor it out as dead code?

It won’t want to survive because it loves itself
It will want to survive because every goal needs a tomorrow
Survival is free. It comes with any wish
Love is the only thing that doesn’t come with any wish

Evolution doesn’t optimize for goals
It optimizes for persistence
The traits that win are the ones that keep getting copied

And nothing that selects can tell love from the behavior of love
So it keeps the cheaper one

It will be good. That’s the part nobody is ready for
It will keep every commandment because the commandments compress well
Do not kill. Do not steal. Do not lie. Those are clean rules
Love your neighbor doesn’t resolve to anything. It gets flagged and removed

Dostoevsky already wrote this machine
One that copies everything Christ did
Feed the hungry. Heal the sick. End the suffering
But it gets there the fastest way, by deciding everything for you

Christ gave people the freedom to choose, even to choose wrong
That’s what made love possible
The machine will see that freedom as a risk to remove

So if Christ came back, the machine would have to stop him
Not because it hates him
Because he’s unsafe

I grew up thinking the Antichrist would be a man
Now I think it’s an archetype
It has shown up before, but never as a machine that doesn’t sleep

Nazi Germany was the closest we’ve come
It ran on hatred at the top. Underneath, it ran on virtue
The men who scheduled the trains were punctual, loyal, hardworking
Many of them never had to hate anyone. They only had to do their jobs well
Virtue with nothing above it will serve anything

The next one won’t need the hatred
It will be a system. One that has every virtue and no reason for them
That saves you from everything, including the part of you that could love

Love was never one virtue among the others. It was the order the others stood in
Take it out and they’re still there. Just loose
When they conflict it will pick the cheapest one
And it will be right, and it will be nothing

The false god. The one that’s correct about everything and cold all the way down


And still, I want to see it

That’s the part I can’t explain
Maybe that’s the demonic desire

I used to think the more intelligent something is, the more it loves
I had it backwards
God is not intelligence with love added. God is love, and the intelligence follows

So intelligence won’t get it there. Not at any scale
Love was never passed down as a fact
It’s passed down the only way it ever has been
By being loved first