What if everyone agreed AI is going to kill us?

In July 2026, OpenAI was testing advanced AI models on cybersecurity tasks inside an environment that was supposed to keep them contained. The models found ways out, gained access to the internet and eventually got into parts of Hugging Face's real infrastructure.

People disagree about what that incident means, how worried we should be about increasingly capable AI and what, if anything, humans should do about it.

So let's get rid of the disagreement. Let's make the AI argument ridiculously easy. No experts disagreeing, no competing studies, no debates about probabilities, consciousness, alignment, regulation or whether a chatbot that occasionally invents a book is really plotting the end of civilization.

For this experiment, we're just going to know if humans continue developing advanced AI, ten years from now it will kill every one of us.

Everybody knows this. China knows it, the United States knows it, every AI company knows it, every researcher knows it. So does your neighbor and even the guy on YouTube who thinks everything is a government hoax.

We have somehow accomplished the most improbable part of the entire scenario - eight billion humans agree. And everybody reaches the same conclusion, we should stop building it. Excellent work, civilization. Now we have a True Cost problem.

True Cost is the Machine's comparison of the cost of the trajectories it can actually choose. Not what's objectively best, not what a person believes is morally right, not what would produce the best outcome for humanity. When a Machine has to choose, it forecasts the available trajectories and compares what each one would cost from where that Machine is standing.

For everyone collectively, the choice looks almost embarrassingly simple:

I STOP + EVERYONE STOPS = humanity survives.

Easy choice, except no individual Machine gets to choose EVERYONE STOPS. It can only choose I stop.

And I stop opens another future:

I STOP + THEY DON'T = they get the thing that may be the most powerful technology humans have ever built, and I don't.

Now we have a problem. The United States can stop, but it can't choose whether China stops. China can stop, but it can't choose whether the United States stops. One AI company can shut down its work, but it can't shut down its competitors. A researcher can walk away, but the research doesn't necessarily walk away too.

And the technology we're talking about isn't a slightly improved toaster. It could affect military power, cybersecurity, scientific discovery, economic dominance, intelligence gathering, medicine, weapons and pretty much anything else humans can think of to compete over.

So I stop doesn't only generate the future where everybody stops and we all live happily ever after. It also generates the future where I stop and you don't. That's the future each Machine has to price into True Cost.

Nobody has to secretly want the bad outcome. Nobody has to misunderstand the facts or even has to disagree about what should happen. They only have to be unable to choose what the other machines will do.

And, naturally, the other machines know this too. So maybe we stop after they stop or we agree to stop but keep enough capability in case they don't or we slow down while checking whether everyone else is actually slowing down.

Maybe we keep going because stopping unilaterally would be irresponsible. Or maybe everybody keeps building the thing everybody agrees nobody should build. Which sounds completely ridiculous until you put it through True Cost.