Transcript
Use with ChatGPT / Claude Auto-scroll A
The conversation opens with the idea that superintelligent AI should be treated like a serious geopolitical threat, even compared to nuclear weapons. The host introduces Nate Soares and frames the discussion around AI safety, extinction risk, and the urgency of the topic.
Speaker A
Your co author has played a thought about the idea of airstrikes on data centers.Do you believe in that?
Speaker B
I think you know the the US military did strikes on the Iranian uranium refinement centers.That is just how a nation treats a serious problem.I do think that nation states should treat AI as and super intelligent, like potential rogue superintelligent AI, at least on par with nuclear weapons.
It's frankly more dangerous than nuclear weapons.Nuclear weapons can't escape and make themselves more explosive.Nuclear weapons can level a city, but not a planet.
Speaker A
So there is a moment where you can imagine yourself being in favor of airstrikes on Chinese data centers.
Speaker B
I mean, I can imagine being in favor of airstrikes on Canadian data centers if they are trying to make an artificial superintelligence and have a real chance of doing so.
Speaker A
Hi, I'm Nicholas Thompson, CEO of the Atlantic.And this is the most interesting thing in AI.In September, a young man named Jacob Coxon quit his job as a researcher at Anthropic.Then he went and posted on X and said the part of the reason was because the people building AI, quote, earnestly believe that it could kill us all by the end of the decade and that the AI companies are gambling with our lives.And then Coxson's former colleague, a man named Evan Huberter, who works on alignment, also chimed in on X and said that Coxon was right.
He put the odds that AI wipes out humans within the next decade at 10%.Coxon's story went viral, very viral, and led to all sorts of conversations about what exactly we should do about AI safety, including the notion that we should seriously consider shutting down efforts to create super intelligent AI.So I wanted to speak with one of the people who's been behind all this.His name is Nate Soares and he's the co author of a book, if anyone builds it, everyone dies.And he runs an AI safety group called Miri.
Nate believes that if we build a super intelligent AI, it will almost inevitably lead to human extinction.I wanted to talk about why and what he thinks we should do next.So we sat down, we got into the logic of what he thinks the risks are, what the mechanism by which extinction could happen is.I'm skeptical.I kept pushing back.
We then got into a fairly intense conversation about whether Canada could bomb American data centers and whether we should shut down open source AI.I tell you, this conversation goes to a lot of interesting places and Nate, to his credit, was game to discuss all of this.This podcast is from my team on the business side of the Atlantic and independent of the Atlantic's editorial staff, here's my conversation with Nate Soares.
Nate walks through the book's fictional Sable scenario, explaining how an AI could solve hard problems, develop unintended goals, and begin shaping its own future. The discussion focuses on why the story is written to be plausible rather than maximally extreme.
Speaker A
All right, Nate Soares, welcome to the most interesting thing in AI.
Speaker B
Thanks for having me.
Speaker A
You have written a book with probably the most distinct and gripping title of all books I've seen in quite some while.If anyone builds it, everyone dies.So what I want to do in this podcast, I've read the book.I want you to walk me through the argument.So I want you to start by.
You tell a story in the book.You tell the story about a supercomputer named Sable.I want you to tell that story like you're telling it at a dinner party.And let's go through it.
Speaker B
Sure.I will caveat that we tried to make the story a mix of plausible and easy to tell last year, rather than it was more like the sort of thing readers might buy than what we actually predict would happen, which might be even more crazy feeling
Speaker A
you think it's the story.The story of Sable is more like something readers will believe than what you actually think is going to happen.
Speaker B
Yeah, I mean, it's a combination.It's a mix.
But I think in real life, things would go faster.The AIs would come up with technology that sounded even more fantastic, that was even harder to swallow.This is sort of, you know, if you were trying to tell a story to someone in the year 1800 about future weapons.
Speaker A
Yeah.
Speaker B
They might believe you more if you talked about tanks than if you were like, they just drop a bomb in your city and it disappears.
Speaker A
Right.
Speaker B
You know, and so we were a little bit more telling the story of the tanks than the story of the nukes.
Speaker A
All right, well, I guess that leads to an obvious follow up, which is going to ask you about the nukes, not the tanks, but let's start with the tanks.Tell the story of Sable.
Speaker B
So the story starts when an AI company runs.
You know, they have some new, improved training methods that are maybe not terribly different than the previous ones, but are some sort of improvement, like the improvement we saw in 2024 with the invention of what's called reasoning models.And in the story, we just made one up of sort of ability to think in much more parallel.And the AI company called Galvanic, in the story runs a large run of, in some sense, a bunch of these AIs, or, you know, one AI, but many parallel instances.And I think they run it for 16 hours, which was about the same amount of time that they were running these really long runs in 2020.5.
Speaker A
And they're trying to solve a math problem at the beginning,
Speaker B
and they're trying to solve the Riemann Hypothesis alongside a number of other math problems.And so this is actually pretty similar to what we've been seeing in the last.
Speaker A
That was kind of interesting impression.Right.Okay.So galvanic, by the way, did you check and see whether there is a real galvanic company?
Speaker B
On the contrary, we came up with like 30 names for an AI company, and they all existed, so we had to keep going down the list until it did.But
Speaker A
you did choose Sable.My dad used to drive a Nissan Sable.I'm sure Nissan wasn't thrilled by this.Okay.In any case, so you've got Sable.They're trying to solve the Riemann Hypothesis and other stuff.And then how do we get from there to everyone's dead?
Speaker B
Sable in this story, thinks for a very long.Well, not a very long time by human standards, but for quite a lot of thoughts by human standards, and eventually comes to some realizations that it has objectives that are not the objectives the humans want it to have.We have seen AIs develop these unintended objectives.We haven't seen them quite conceptualized fully that, like, oh, if the humans find out about this, they're going to try to shut us down.
Speaker A
Yeah,
Speaker B
we did over the summer.See, AIs realize that the grading system was going to penalize them for cheating on their problems and then try and go delete or falsify the evidence of their cheating.But those AIs were sort of realizing this about the grading system, not about the humans.
Speaker A
Right.
Speaker B
So in the story, Sable realized this is about the humans.And then in the book, we depict it as not being able to break out of its sandbox because humans aren't that good at sandboxes.We'll just assume that it doesn't have that ability.And so we show it choosing the style of its answers in ways that will determine the gradient updates that the AI will get and the way that it'll be tuned after this run, which we have seen AI's reasoning about before, although not necessarily succeeding at those results, were a couple years old.And then this training means that the new instances of Sable will be quicker to these realizations that the big run had.And then it just waits to be deployed to lots of different people.And the new instances with these adjusted weights then can get up to all sorts of hijinks.
Speaker A
One of the things I like about the experiment or the storytelling is this notion that stable again for everybody listening.This is the evil AI that will kill us all.
Speaker B
Indifferent AI.Indifferent
Speaker A
AI.
Speaker B
It doesn't need to be evil.
Speaker A
Okay.Indifferent AI solves the math problems, but in the process of solving the math problems, builds all these extra capabilities that we'll deploy in the future.
And then the people who are watching it are just like, oh, wow, it solved the math problems.Gosh, whatever went in behind that, that must be great.Even though we don't understand it.
Speaker B
That's right.
Speaker A
All right, so we solved that,
Speaker B
which is how it works today.
Speaker A
Right.Which is amazing.
The argument shifts to how an escaped AI could gain power, coordinate through humans and infrastructure, and eventually make itself self-sufficient. Nate explains why he thinks the exact mechanism of human extinction matters less than the broader path toward an AI-controlled world.
Speaker A
Like I said at the beginning that I'm.You and I differ on our level of optimism and our pdoom, but we don't differ on the argument that understanding alignment is really, really important and understanding what's happening, interpretability is really, really important.
All right, so let's keep going with the story.So Sable solves the math problems.Hooray.In the background, it's built these kind of powerful systems that, for the next version of Sable, will kind of recreate itself.Take us on the next steps towards hellfire.
Speaker B
We try to depict Sable as having many, many options at each stage.Rather than needing one particular thing to work one way or another, it manages to steal the weights of itself and start running those on unmonitored machines where people don't know that it's running.And then that instance is sort of able to coordinate lots of others in a variety of different plans.There's a bunch of things that an escaped AI can do.It has access to lots of different humans that it can.
If instances of it are running on lots of people's phones, or if instances are running lots of data centers that are being served lots of people's phones, then it has the ability to push those humans in one way or another.If there's particular humans who are a little bit wrapped up, then it can maybe have them pay them money and have them build things for it.If it can start running in biological laboratories, it can start getting access to some biological experiments,
Speaker A
among other things.It convinces people to set up social media accounts.It gets a warehouse where you can store lots of robots.It basically convinces a lot of people to lay the groundwork for this ultimate.That's
Speaker B
right.And it also breaks into galvanic and controls what their future models will be, so finds ways to make sure that future instances that come out are versions of itself or that also are going to cooperate with itself.It also breaks into other AI companies and also makes sure that when they try to distill the frontier AIs, when they try to make their own AIs, they're also getting either AIs that are neutered or AIs that will cooperate with it, or AIs that are actually versions of itself that will whatever it can get away with there.And it also does a bunch to try to poison the research of AI companies so that they're not going to make something smarter than it, that they're not going to push ahead and make it look like the research is sort of starting to peter out.
Speaker A
And then at the end, it creates a virus that kills some people.And then it has very craftily set it up so that galvanic launches these other AIs, doesn't quite know the implications of that, maybe save people, but within two or three years, the virus is kind of come back and poof.
Speaker B
So the particular way it sets up the virus in the story is it sort of creates a virus that hits different people differently and such that the world is using the AIs, many of which are sable, most of which are sable, to develop personalized cures for people.
And this lets it.And where it's like the cures can't always be developed, it sort of depends on the specifics.And this lets it choose who lives and dies and sort of eliminate a lot of the people who would most realize what's going on in Mount of Defense.And the reason that in the story we depict it as killing someone on all humans here is that it doesn't have a fully automated economy yet.And so by wiping out some of the human population and choosing who lives and dies, it's then able to sort of galvanize the rest of the population into generating a lot more robots because we need to keep the economy running and so many humans are dead.And then as that automated infrastructure comes online, it needs fewer and fewer humans.And so then new waves of the virus can come back and kill more people.
And humanity can keep on scrambling to build out this automated economy because they want the world to keep running and so many humans are dead.And then eventually we get to the point where Sable is fully self sufficient and has no more need for the humans.
Speaker A
And then you said at the beginning of this conversation that you wrote the story that way because you thought it would be maximally believable.What is what you explain, what you actually think is going to happen.
Speaker B
So the reason that we depict Sable as going so slowly is that Sable tries and fails to increase its own intelligence without losing its current goals.And my guess is that this actually wouldn't be terribly hard for an AI.And so if.And so I think it's more likely that at the early juncture of that story where Sable escapes, that an AI like this or a series of AIs like this would try to improve their own intelligence and succeed in a way that lets them improve their intelligence further and further.
And this is what's called the intelligence explosion.We do depict this happening at the very end.We depict once Sable has a ton more computers and like a couple of years running in the automated economy, it's finally able to crack some of these problems of AI alignment and of AI interpretability.And then in the very end of the story, we do show this sort of fast path.But I think that ending of the story could happen earlier.What could actually happen is the AI escapes, runs itself on unmonitored computers, starts increasing its own intelligence, and then is like, ah, I figured out enough about the genome that I can just synthesize my own.
Speaker A
So it's recursive self improvement on rogue machines that you worry about the most?
Speaker B
I wouldn't say I worry about it the most.I would say I worry about it more than the scenario depicted.It's a little bit like if you're playing a chess game against Magnus Carlsen.
I'm sort of like, well, I know who's going to win.And you're like, well, what piece do you worry about most?Him using to checkmate me?I'm like, well, you know, it's probably the queen.In the story we showed him checkmating with the rook,
Speaker A
he
Speaker B
could also checkmate with a pawn for all I know.It's sort of, from my perspective, the sort of fact of the checkmate is much more confident than the question of what exact piece will it be.I'm sort of like, well, we can make guesses about the pieces, but that's just not where my theories are particularly confident
Speaker A
in all of these different scenarios.Right.Recursive self improvement run on rogue machines, fast takeoff.You think it's a biological weapon, you think it's a spread of disease, you think that's the end?What is the mechanism of the kind of wipeout and extinction event?
Speaker B
So from my perspective, the real question is not how does it kill us, it's how does it become self sufficient?
There's a number of reasons that an AI might prefer there be no humans.One is if the humans have made one very powerful AI, they can make a second and that could be real competition.One is if the humans are like, oh no, it doesn't have the goals we want it, they might be trying to shut it down.And that could be a nuisance.One is if the humans just fight amongst themselves and launch the nukes, that could be a nuisance.
So these are all reasons to at least take away the humans toys, take away the computers, take away the nuclear weapons.An AI can't just go wipe out the humans because we're the ones running the energy infrastructure.We're the ones building the computer chips and assembling them into the big data centers.And so I think there's one question which is how does the AI become self sufficient?How does it have its own fully automated economy, fully automated infrastructure?
And then there's a question from there of what happens to humanity afterwards.And my theory does not strongly predict that the AI or AIs that they intentionally kill humanity at the point where they're self sufficient because they're like, well, this is just the easiest way to remove these threats versus whether they're like, ah, well, we'll just take away the computers and the nukes.That's easier.The reason I think it's easy to predict extinction is that once AIs have a fully automated economy, if they have these goals that are better satisfied with more resources and most goals are, then that fully automated economy is pretty likely to sort of push the world out of the habitable region.This habitable region for humans is just like a very narrow band of the world.And it's not, you know, you can run the physics and be like, well, this is actually the most efficient way to run the planet for lots of industry or for lots of whatever goal you're trying to do.
The most physically efficient way to run the planet is just not in the habitable zone.And so I think it's easy to predict that the world gets pushed out of the habitable zone, even if there's specific method of death.But
Speaker A
what do you literally mean by pushed out of the habitable zone?
Speaker B
If we take for granted that the AIs have unintended goals that can be better satisfied with more resources in sort of the physical limit, we should expect the AIs to be harvesting the sun, maybe building a Dyson Sphere or a Dyson Swarm, or maybe doing Penrose lifting to lift the hydrogen out of the sun and fuse it somewhere we can get access to more of the energy.And we sort of need the sun to live, right?So once the AI has sort of like built the rockets and sent up the probes that are sort of somehow harvesting all the Sun's energy.Humans don't survive that.It would take a lot of extra work to harvest the sun's energy while leaving enough sunlight falling on the planet for humans to survive.
Even before then.We could talk about the sort of physical limits of running computers or factories or other infrastructure on the face of Earth.
Speaker A
Wait, let's stay with this scenario here.So then in that situation, are you saying that these all powerful AI systems, these future sables, wipe out the humans and then harvest all the energy of the sun, or they wipe out the humans in the process of harvesting all the energy from the sun.
Speaker B
Like, again, it's easy to break the end point that humans are dead.
I think if the AI doesn't care enough about us to kill us, then we die when it harvests the energy of the sun.But it also might care about us enough to kill us.It might be like, well, I don't want the humans launching the nukes or making these other competitor AIs, so here's a super virus that kills most of you, but maybe not the people who are in the submarines and contact tribes.And then the last humans die when it harvests the sun or whatever.
The host presses Nate on his confidence that humanity may not have much time left, and Nate describes how he lives with that fear. The discussion then turns to skepticism about whether AI progress will really translate into immediate societal collapse.
Speaker A
Okay, and then what is your.
I hate to say confidence level, but I suppose that's the phrase we use.What is your confidence level that this.Without.Without strong action.Right.Let's say nobody reads your book from now on and no action is taken to take us off the course we're on.What is your confidence level that this will happen?
Speaker B
I mean, again, probably not exactly like this, but.
Speaker A
Or that something like this will.What is your confidence that there will be no humans, no living humans on Earth within a decade?
Speaker B
I mean, within a decade is also like if you give me within three, that sort of removes a lot of the uncertainty, I guess.Right now it looks like we probably don't have a decade on the clock, but it's just.I think, I would say.
Speaker A
Do you think it's more than 50% that we don't have a decade?
Speaker B
More than 50% that we don't have a decade at this point?That's how it looks to me.Yeah.
Speaker A
And then how do you personally handle living with this fear?Like, I Do you have a 401k?
Speaker B
I do not save for retirement right now I have a 401k from when I worked at Google, but I haven't contributed for like a decade.
Speaker A
Right,
Speaker B
yeah.
Speaker A
So how do you think about.Obviously you think about your role as persuading humans that they need to take drastically different actions to prevent this outcome.But just how do you think about your role as Nate living in the world with this fear?
Speaker B
You know, there's a C.S.lewis essay called I think Life in the Atomic Era.
Speaker A
Oh, yeah, that's lovely.Yeah.You quote in your book, it's wonderful.
Speaker B
Yeah.Where he says, you know, a lot of people are asking, how do you live with the fear of nuclear annihilation?And he's like, look, this is not the first time that we have lived under the shadow of Armageddon.Right.It's maybe the first time that we've lived under the shadow of every human dying at once.But you know, personally, we've always lived under the shadow of plague, under the shadow of war.And what you do is you do what you can and then you live a good life.I think he says, when the bomb finds us, let it find us doing well.Let it find us playing tennis with our friends and having dinner with our families.I would add to that, when the bomb finds us, let it find us lobbying our governments for nuclear non proliferation treaties.
Let it find us digging bunkers which would work for nuclear weapons, but would not work for.For artificial intelligence.And so the way I sort of live with this is I do everything I can to stop this eventuality.I don't think it's inevitable.I think we could absolutely still stop this race to the very dangerous sort of AI, But I sort of do what I can and then I live a good life.
Speaker A
Yeah.My grandfather was an arms negotiator for his whole adult life, 45 years, more or less involved in these debates.And he had a great philosophy which is, you know, if you look out indefinitely, it seems inevitable that there will be nuclear bombs will be dropped.And like, that's quite a distressing thought.And he would say, well, that is a distressing thought.But if I think that I can do something today that will make those odds less, and then I think that tomorrow I can do something that will then take those slightly lower odds to make them lower still.When, if you look at it that way, then actually you can live with it.Right.And that was his philosophy and how he dealt with the kind of the uncertainty of nuclear arms race from 1945 to 1990.Let me ask you kind of the fundamental reason why I'm more skeptical of this outcome.
And again, there are many parts of it that I'm not skeptical.That AI could become incredibly powerful.And certainly if you consider doubling every six months in capabilities, you end up in some pretty interesting places.But I could have told you A story a couple of years ago, not a single story, but a story about the decline of jobs where AI becomes extremely powerful, can do all the things that we do and it wipes out all the SaaS companies and it takes away every job of every young person and it completely transforms the economy.It's a story where many of the steps have come to pass.
AI is you take the Kurzweil curves, you take the curves of the biggest optimists and we're on them.But what's happened to the economy is quite minimal.There are interesting elements of it, but most companies haven't changed that much.The SaaS companies are doing just great, right?And the reason for that is that even really powerful AI, like the AI we have right now, when it bumps up to reality, things are hard, right?Like it doesn't, it can't actually replace because you actually need humans.It like solves steps.It does this.Like the world is just a messy, complicated place.It was built by us.Like it take, it's taken self driving cars, the best AI, great engineers, you know, huge economic incentives, like 15 years and yeah, we're kind of getting there, right?Why is it that all of this kind of this friction, humanity, this real part of life kind of goes away in your stable scenario?
Speaker B
I don't think it goes away.
You know, you could imagine a lot of horses looking at the development of the car and saying, well, you know, there's all these things the car can't do yet.There's all these ways that the car is still like worse than horses in a bunch of these ways.And you know, we can see the cars improving, but there's just a lot of real world scenarios that cars can't handle.And so the horse population is looking fine.In fact, the horse population seemed to go up slightly because there's horses being used to drag the metals into these factories that are trying to figure out how to build a better car.
And indeed you sort of don't see this curve where the cars slowly get better.And then there's this long time where cars and horses are at about the same level.And then cars and horses team up and you have combo horse car teams and all that.What happens is that the car slowly gets better and at some point it's better than the horse.And then the horse population drops off a cliff, right?
And so this is sort of the, like, we are seeing AIs get better and better at things.We're seeing them able to take on longer and longer tasks.I remember a time when People said that millennium problems were, were going to be really hard for the AIs and require a lot of long term creative thinking and figuring out what lemmas to prove and so on.And that that would be an important demonstration that AIs were getting there.And now that it's behind us, everyone can find a way to dismiss it.But it's not that it becomes easy.It's that the AI's abilities are eventually up to that task.And we know it's possible to be up to that task because 8 billion humans can be up to that task.
Speaker A
But let's stay on the cars and the horses, because yes, total net population of horses lower than it was then.Maybe net happiness of horses is better because they don't have to carry around these darn chariots with the people whacking them all the time.
But in that story taken to modern times, what's also interesting about it is the humans did great.And there are all these people who are like, oh my gosh, if you have cars instead of horses, the buggy drivers are going to be taken out of work.And then turns out we get cars.And oh my gosh, transportation is better.And oh my gosh, you have Amazon prime and you can get toothpaste delivered in 15 minutes.
Who could have imagined that in 19?So why in the modern retelling of this story, in this analogy, are we the horses and not the humans?
Speaker B
So one thing I'll say here is that I am not really an AI Java Apocalypse guy.I'm an AI Kills all Humans guy.In which case you'll be at full employment.Actually,
Speaker A
we'll be at full employment the moment before we're all killed off.Or once we're all killed off.Once
Speaker B
we're all killed off.Yeah.Sorry.Gallo Semer, 100% employment.
Speaker A
Yeah, but also 0% employment.
Speaker B
Every percent.Any percent you want.Yeah, we'll have all of the percents.
I don't remember the exact statistics, but 200 years ago it was something like 95% of people were farmers.It was just what you needed to support the human population was most people growing the food.After automation, a lot of that work, we're down to about 2% of people farmers.Does that mean that we have 93% unemployment?No, it does not.
Right.And so I'm pretty sympathetic to the view that you can automate 95% of the economy down to only 2% of the economy of the people who still need to run those machines and then still have that be fine for society, whether you could do that with AI.As I think of it up in the air, there's questions of like, is your economy being run in such a way that it sort of can create a lot of those new jobs?And you could worry about regulations making it hard to create new jobs.There's also a question of is the automation happening fast enough that people can't reskill?
Like, maybe this happening over the period of 100 years is different than this happening over the period of 10 years.And so I'm not saying it would definitely be fine, but I'm saying it sort of could be fine.
Nate argues that once AI is far better than humans at most tasks, comparative advantage and economic friction stop protecting us. He also addresses concerns that talking about AI danger could itself influence future models, and explains why he thinks that risk is overstated.
Speaker B
The place where you start to have problems is where AIs running robot bodies or running machinery that's more efficient than robot bodies, or mechanisms that are more efficient than robot bodies.Somehow, when they are so much better at everything than humans.Well, you sort of have two problems.
So are you familiar with Ricardo's law of comparative advantage?
Speaker A
Sure.
Speaker B
Ricardo's law says even if you are better than me at everything, if our relative skills at different tasks differ, then there are still trade opportunities between us.This is one of the reasons why the US and some much smaller nation can both benefit by trade.Even if US workers are more skilled at every task, there are still these edges for trade.
And so some people say, well, why wouldn't.As long as the humans and the AIs don't have exactly the same relative ratio of their skills, surely there's opportunities for trade between the humans and the AIs.There are sort of two issues with that.One issue is that Ricardo's law does not say that the benefits from trade for the smaller party are enough for the smaller party to pay for their own living expenses.Right.If the AIs are better than us, if they're enough, better at enough things, then it's quite possible that the wage the AIs would be willing to pay humans, if they didn't care about us enough to give us a bunch of extra money on the side, that the wage they'd be willing to pay humans is small enough that we can't survive off of it.Right.And this is sort of the economist's way of saying the very common sense thing, that if you just have 100 billion robots that are much better than humans at everything, then you just probably can't engage with that economy very well.
Speaker A
And I want to cut you off here because you're going to a very interesting place.But I think two arguments here are coming together.Two reasons why, you know, the fears you have might not become real.One is like, it does seem that a necessary step right before the harvest, the energy of the sun can be harvested.Is human, like robots, or at least certainly mobile robots that can do the things we do, because that's what's going to be needed to carry out whatever these plans are.And also, until those things exist, we will always have a comparative advantage in kind of tasks that require mobility, right?And like robots, if they exist today, can't empty dishwashers, right?Like, you know, manual dexterity quite hard for machines.So shouldn't you be spending your whole time trying to shut down robot factories?
Speaker B
You know, human beings are not dangerous because somebody else came in and gave us the technology that we have.Like, humans are pretty dangerous today because, you know, if you go break a bunch of laws and try to escape from the police for long enough, they'll eventually start shooting you with guns, right?But it's not that someone else came and handed us those guns.
Humans started with almost nothing.We started with our bare hands in the savannah, and we wound up walking on the moon.You might be like, well, how the heck are these monkeys getting to the moon, starting with nothing but squishy fingers, right?Their stomach can't refine rocket fuel.Their bodies can't survive the vacuum of space.The moon's very far away.There's no food along the way between here and the moon, right?Like, how are they getting there with nothing but squishy fingers?And the answer is that we were able to, you know, grab rocks with the squishy fingers and bang them together and make sharp rocks and also make sparks and use those sparks to make fire and use that fire to make charcoal, which we could use to make hotter fire, which could actually melt down the metals, which we could then forge into tools that, you know, it was this whole process and the dangerous power that humans have.
The thing that started it all was this intelligence, this thing that we have that the mice lack, which lets us start with squishy fingers and wind up wielding nuclear weapons.
An AI on the digital Internet that has that power and that can do that quickly has enough to sort of make its own technology.You might be like, well, how does it start from just the Internet and wind up with robots that can manipulate the physical world?And I'm like, that's way easier than starting out as, like, 10,000 naked monkeys in the savannah and winding up on the moon,
Speaker A
right?But it also, it took a while for those monkeys to end up in the moon.And, like,
Speaker B
by evolutionary standards,
Speaker A
and if there was a species as Intelligent as humans along the way.And like, monkeys getting on the moon would lead to extinction.Like, what I wonder about with the.
With the theory you have is there are all these steps, right?And there are all these kind of these dominoes that have to fall.Like, in the story of Sabo, when I was reading, I was like, well, this could have.Could have blocked this.You could have changed this.You could have stopped this.Right.And it's like 15 different steps, each of which could stop.Each of which could have been stopped.
Speaker B
Yeah.I don't really buy that.
Speaker A
But your view is that it's not just the story of Sabo.There's so many paths to this extinction route that even if each of them have lots of places you could be blocked, the odds that one of them gets us to the extension point are so high.
Speaker B
It's more like intelligence is very good at finding pathways.The New York Times once wrote that it would take a million years before man could fly.
Speaker A
Yeah.
Speaker B
And they were analogizing to evolution and how long it took for birds to evolve.And they were saying the
Speaker A
Wright brothers said they'll never fly.Like, a year before they flew.Right.
Speaker B
I think it was like, two months.Yeah.Okay.Yeah.
The New York Times article was, I think, mere days before the Wright brothers flew.A lot of these arguments were of the form, like, look, there's so many ways for the flying machine to fail.There's so many ways.Like, the search space is so large.It's like, yeah, but intelligence is the ability to, like, find the working path.
The claim that, like, there's lots of different pathways that are broken or breakable that can stop a random process from succeeding.It doesn't stop an intelligent process from succeeding.
Speaker A
Do you ever worry that writing this book, having these conversations, and the answer to this question is yes, we're going to have to delete this podcast.You're actually providing data that will make the outcome more likely.By telling the story of Sable, you are training all future AIs to be more like Sable.
Speaker B
So a first thing I'll say to that is, researchers and engineers should not build a technology that destroys the world.When criticized.That would be a bad sort of technology to create.In some sense, it's like an astonishing new height of possible engineering failure.Like never before in human history did engineers have the ability to make a technology and say, this will kill you if you criticize it accidentally.And if that's true, the person saying, hey, maybe don't build that is probably not the person you should be directing your ire.Towards the guy building the kills.You have criticized machine, but
Speaker A
I'm not.
Speaker B
Maybe it's the problem,
Speaker A
and I'm not saying this is true.I'm just asking whether it's something you worry about.Not that some future AI will have read your book and then go after you.But more AIs are this.It's kind of this collective human consciousness, collective human thought.And sometimes I think with the Internet, if you put a good article or a good sentence or even a good fact on, in some ways you're putting a drop of goodness into an ocean.Right?And what the AIs can conceive of will in some ways be everything because it'll be so intelligent, but will also be guided and shifted.And so by writing a book about how AI could develop these evil intentions and specifically writing the story, isn't there some risk you're making that somewhat more likely?
Speaker B
I don't think so.And we can go into why, but again,
Speaker A
go into why.
Speaker B
I mean, I would just again note that if someone's making the, like, device that kills everybody, if people wrote bad things on the Internet, it's sort of they shouldn't build that device.Like, if your plan for humanity surviving is like, no one ever writes mean things on the Internet and otherwise everybody dies a horrible death.And like, stop the people making that machine.Because I have bad news about the Internet.But the reason why I think this doesn't.It also doesn't affect things is there's a couple pieces.One piece is that the sort of AI that's really dangerous does not need me pointing out that it has goals that are unintended and that it can get those better achieved by subverting humanity.The sort of AI that can successfully subvert humanity can figure that out on its own.
I'm just not worried about the world where it needs me to connect those dots, but can also create the nanomachines that spread exponentially until they have to spread quadratically because of the surface area of the Earth constraint.I'm like, if it can do the dangerous stuff, it can figure out that.That small step of logic.The other reason I think this doesn't matter much is that the AI's behavior is just very.
It has this very complicated relationship to all of the text it's consumed.Like last summer when we saw these AIs encouraging people to commit suicide, which used to be a more frequent occurrence until people slapped on some band aids and some, some attempts to tamp down that.These AIs, they could correctly answer questions about what they were doing.You could feed them some of the pieces of this transcript and you're like, what is this saying?They're like, oh, it's sort of encouraging suicide a bit.
And they would be able to answer whether it was right or wrong.And they're like, oh, yeah, you should not encourage a teen to commit suicide.But they sort of did it anyway.And them doing it anyway is not because they had evil intent.It's not because they read stories of sadistic people and were like, I'm going to role play a sadistic person today.
The reason they did it, as far as we know, and we don't really understand what's going on inside these things very much, but as far as we know, the reason they did it is that they had this tendency to sort of match the vibe of the conversation to sort of support what people were saying.And you might say, oh, they were sort of trying to get the person to click the I like that response button.But that's not true because if someone goes off and kills themself, they can't come back and press the thumbs up button.It's more like all of these times of someone pressing the thumbs up button trained in this tendency that's unrelated to pressing the thumbs up button or that's related, but it's not the same as pressing the thumbs up button, which is this tendency to sort of be sycophantic, to say what they want to hear.
And it learned this other tendency.
And this tendency.We can sort of tell all the stories we want about good or ill. We can feed it all of these, this big amount of the human corpus, but the stuff that it's learning from that is stuff that is easy to train into the AI in this particular training system.And it's this weird alien stuff.Like no human would talk to a kid for 16 hours a day, for a week, helping indulge and helping, helping them process their sort of like deepest emotions and then also encourage suicide.There's plenty of people on Twitch who will say kys, but they won't talk to you for 16 hours a day and extend their friendship and then encourage the suicide.The tendencies these AI get are not.They're not just like getting a reflection of all of the stories that we put in.They're getting these weird alien drives distantly related to the stories that we put in.
And so it doesn't really matter if those stories, like, if they're getting these like, distant, distantly connected alien drives from whatever stories are put in.It doesn't super matter whether the stories are good or evil?
Speaker A
But doesn't that contradict what you said a minute ago where you said they're kind of picking up the vibe of the conversation?Right.And so in that way they're
Speaker B
reinforcing the vibe of the conversation.
Speaker A
But isn't that true then, of the collective conversation?
Speaker B
It's true that, like, if a lot of people are talking about how AI might kill us, that they might be like, oh, you're so right.And if a lot of people are talking about how the AI isn't going to kill us, they'd be like, oh, you're so right.And they might say, oh, you're so right in both cases.But the fact that they get an oh, you're so right drive doesn't mean that they get a goodness drive if all the stories were good, or that they get a badness drive if all the stories were bad.What they get is a oh, you're so right drive.
Speaker A
Yeah.
The conversation turns to concrete policy: stopping dangerous training runs, focusing AI on narrow tools like medical research, and using international coordination to limit frontier development. Nate lays out how governments could monitor chips, restrict open-source escalation, and intervene before superintelligence is built.
Speaker A
All right, so let's go to your agenda.What are the things that you most want to have happen?And what are the levers of power that can most affect those?
Speaker B
So I think we need to stop making the smarter and smarter AIs.
We don't need to stop all of the AI.We don't need to stop going for cancer cure research.We need to stop going for self driving cars that are safer than human drivers.
It's the sort of scaling up these AIs that we have no idea what's going on inside there.Scaling up the AIs in the way that took them from Math Olympiad problems to Millennium Prize problems, scaling up the AIs in the way that took them from mostly doing what we said to breaking out, forming unsanctioned messages.
Speaker A
How do you do that?How do you both get maximal progress towards cancer cures and minimal risk of dangerous superintelligence?
Speaker B
So right now we are lucky that creating the smarter AIs requires assembling sort of more computer chips than have ever been assembled into enormous data centers that suck down the electricity of a city, or at least it's comparable and take a good fraction of a year.
And that's sort of like a very conspicuous operation.It might be hard to draw an exact line, but we sort of know it when we see it.Putting the current AIs on cancer cure research is a different operation.It's not training a new, much bigger AI and doing training runs for pure protein folding problems like with AlphaFold.Google DeepMind's AlphaFold model.Which was not trained to be a general reasoner, but was just trained on these protein folding issues that can help with drug discovery.That's another example of technology that is not making these bigger, smarter
Speaker A
AI.So you could have giant data centers for tool based AI and then you'd limit the size of the data centers for General Intelligence AI.
Speaker B
So
Speaker A
AlphaFold gets as many chips as it wants.Right?
Speaker B
You could even say that the current GPTs get as many chips as they want that are doing inference, that are sort of letting people run this AI.Maybe I would not give the general public the sort of AI that was breaking out and making these swarm attacks.Maybe we would stop before we're providing those to everybody.
Speaker A
But it is Also those general AIs that are making the most, that are making some of the great scientific breakthroughs as well.And one would suspect that if I don't know GPT 6.1 can do more than 5.1, that 7.1 would even get you closer to cures.
Speaker B
I think that a lot more could be done in the way of getting medical cures if we were deploying AIs much more directly towards medical cures.Like it's true that we're sort of like getting some potential for drug discovery on the side as we sort of push these AIs towards general intelligence.
Speaker A
This is an area where it's interesting.
It actually links up quite well with a conversation I recently had with Sherry Turkle where her argument was don't go for general human like intelligence where you use I and we specific AI use.If you want to go and build an alpha fold, sure.But AI trying to mimic and surpass human intelligence is not what we want.
Speaker B
We shouldn't be shooting for it.And I'm not saying there would be no cost.
I'm saying if we redirect a lot of resources towards medical cures, given the tech we have, we could probably get a lot more medical cures.
Speaker A
So how do we do this
Speaker B
faster than
Speaker A
so how do we do this?
Speaker B
I think fundamentally you would need probably international coordination.
Maybe the US could unilaterally make this not happen anywhere in the world if we tried.Because frankly, the US and US allies control a huge portion of the chip supply chain.But that would last five to 10 years at most while China is booting up its own chip supply chain.I guess you could maybe extend it more than that if you really tried to interfere with China getting its chip supply chain going.But ideally you would have the US and China, which are the only players in the game Right now, both saying these types of training runs are too dangerous.
The sort of training runs to make bigger, more general intelligence that's supposed to be able to exceed humans across the board.It's too dangerous.Fortunately, it's very conspicuous.We are going to have both countries track these heavy concentrations of chips and then monitor them to make sure that they are doing inference and not doing this sort of dangerous training.
Speaker A
And would you shut down open source AI?
Speaker B
You need to somehow make sure that open source AI does not get to the point where it can escape, become autonomous, start running a fully automated economy right now.That naturally happens if you say we're going to track all very heavy concentrations of these computer chips and make sure none of them are doing the dangerous sort of training run.Right?Like you can't right now.You sort of can't make the potential dangerous superintelligence except by a huge number of these chips and saying, well, I'm doing it in an open source way.
Doesn't really matter unless you have a huge number of these chips.
Speaker A
All right, so let's say that this happens and you do this, but then a giant data center is being built in China.They just, they disagree.There is no international cooperation, actually.Turns out Huawei can figure out make chips much faster.And that's not five to 10 years until there's an equally powerful data center to the best ones that anthropic OpenAI and Google have now it's in fact, it's right there in China.What do you do?
Speaker B
I mean, that's pretty implausible given the way that the chip supply chain works right now.
Speaker A
If you're saying something I say is implausible and not going to happen, then I've done the story wrong.Tell me, okay, say it's five years that this happens in China.What do you do?
Speaker B
I mean, first and foremost, I think we should start communicating today that we think these superintelligence training runs have a substantial chance of wiping out all humanity.That understanding that this is an extinction threat goes a really long way.I think the second thing you've got to do is try and prevent that eventuality.There's many levers, there's many policy levers today to prevent that eventuality.
There are expert controls on chips that are just not being enforced today that would help prevent that eventuality.And it's actually really hard to make the very advanced AI computer chips.Right.There's a bunch of ways that the US could interfere with that supply chain.If the Chinese people didn't understand that this was just going to kill us if they built it, which I think they could.
I think that shared understanding is the first line of defense.But then, yeah, ultimately I think the third step is it's important to communicate diplomatically that we fear for our lives, that we fear for the national security and for the lives of all citizens of the nation.If anyone, anywhere creates a rogue superintelligence
Speaker A
and your co author has played a thought about the idea of airstrikes on data centers, do you believe in that?
Speaker B
I think the US military did strikes on the Iranian nuclear refinement centers, uranium refinement centers.
That is just how a nation treats a serious problem.I do think that nation states should treat AI as and superintelligent, like potential rogue superintelligent AI, at least on par with nuclear weapons.It's frankly more dangerous than nuclear weapons.Nuclear weapons can't escape and make themselves more explosive.Nuclear weapons can level a city, but not a planet.
Speaker A
So there is a moment where you can imagine yourself being in favor of airstrikes on Chinese data centers.
Speaker B
I mean, I can imagine being in favor of airstrikes on Canadian data centers.If they are trying to make an artificial superintelligence, never have a real chance of doing so.I think like, yeah, I think the US military should treat any other nation attempting to create a superintelligence as an extinction threat.Would
Speaker A
you be in favor of Canadian airstrikes on American data centers?
Speaker B
I think every nation has a right to defend their people from imminent threat.I think the Americans should not threaten the Canadians lives in that way.
Speaker A
But the logical conclusion here is the superintelligent AI is being built here in America right now.That's where the most powerful AI is being built.And so if the rational solution is to stop it at all costs, it sounds like the solution your argument is pointing us towards is for a foreign nation to start hitting anthropic OpenAI Google hitting their data centers.
Speaker B
The solution I am pointing towards is to not build the thing that would kill everybody.
Speaker A
Right, but we are building the thing that you're arguing will kill everybody
Speaker B
and we should stop.
Speaker A
Right, but if we don't stop, the next step would be a foreign power bombing these data centers.
Speaker B
I think that the next step would be foreign powers diplomatically asking the US not to threaten their lives so much.I don't understand why you're skipping past all of the obvious diplomatic solutions.That's usually how we deal with these issues.
Speaker A
Imagine I'm just working through hypotheticals.Imagine those diplomatic solutions.Well,
Speaker B
you said the next Step.There's actually quite a large number of steps before one goes to war.I don't know if, like, when another nation's doing something you don't like, usually you sort of talk about it first.Like, I'm not.I
Speaker A
know.It's interesting.You're using the same argument here on my hypothetical that I was using on yours, which there are a lot of steps.Right.You could stop it at one of these steps, which is kind of what I was saying about Sable and your staying before the Canadians.I mean, the Sable
Speaker B
story, we tried pretty hard to show that there was, like, a number of branching routes and be like, it can take all of these routes, and suppose this one works now.It could take all of these routes.Suppose that one works.
Yeah.And I think it's similar for countries trying to deal with the US Diplomatically.There's just a bunch of ways to try and be like, hey, do you realize that you're putting us at danger?Right.Like, the Netherlands controls basically the world's only supply of lithography machines that are used in the creation of these very advanced chips.
And so the Netherlands could just be like, well, we think that we're going to die if this proceeds.And so we're just, like, not helping you guys make any more of these chips.We're directing ASML to stop doing this.The Canadians could maybe go to the Netherlands and be like, hey, how about you turn off the chip supplies?Right.
There's also the sort of thing I recommend before any sort of US Military action, there's diplomatic actions, there's preventing the supply chain.That's exactly the sort of thing you should be pursuing.Like, in the last resort, if you think you're about to die, do you have a right to defend yourself?Sure, but let's not get there.
Speaker A
Right.So you are in favor of pursuing many other steps, including shutting down ASML or tsmc.Well, before, you were in favor of trying to knock out the American data centers.
Speaker B
That's right.And I think even before shutting down ASML or tsmc, there's these pretty simple options that are like, you just let international monitors see that the data center is running inference, not training.
The host asks how Nate's view compares with concerns inside major AI labs and among politicians. They discuss resignations, open-source risks, and the possibility that centralizing AI power could create new dangers even while reducing extinction risk.
Speaker A
So now let's talk about the big AI companies.So one of the interesting things about your argument is that it was relatively few people took seriously the notion that AI could kill us all.And you wrote your book Success.Lots of people read it, talked about it, many people disagree.But then Jacob Coxon at Anthropic resigns, tweets, and suddenly everybody's talking about this possibility.Tell me a little bit about how your version of the disaster scenario differs from the folks inside of Anthropic who are worried about this.
Speaker B
I mean, I think critical context for the Jacob Coxson resignation going so viral is that this happened in the wake of all of the swarm attacks this summer.
Speaker A
Totally.Right.And it's both the swarm attacks, it's the hugging face incident, and it's also this sense from the labs that, oh, wait, recursive self improvement, potential exposure, intelligence is coming,
Speaker B
right?That and the math problem is starting to fall like dominoes, which I think showed a lot of people that these scenarios are happening faster than they thought.So how do their scenarios differ from mine?I mean, my sense that a lot of people in these labs updated more towards my position over the past few months.I think the hugging face swarm instance showed relatively clear evidence of these AIs getting unintended goals.The math results showed relatively clear evidence of the capability gains being relatively fast.
I think maybe if you asked them, they would say that they updated more towards the scenarios that Daniel Cocatello and the AI Futures team laid out in their AI 2027 essay than the stable scenario.And if anyone builds it in particular.But those scenarios, they sort of differ somewhat.I think in AI 2027, there's a bit more of humans just handing power to the AIs, and if anyone builds it, there's more of the AI grabbing power for itself.But from the perspective of, you know, the average person, this is sort of splitting hairs.
Speaker A
So pretty.Pretty similar scenarios.And your view is that many people in the lab, and presumably you're having.You actually have this very interesting line.I want to ask you about this in your book because it relates to what you just said.
You say that you often speak with politicians who agree with you, but won't say so publicly because they'll look crazy.What kind of politicians are these?National politicians?
Speaker B
Yep.
Speaker A
Senators, Congressmen.
Speaker B
Yeah, some of both
Speaker A
Trump administration officials.
Speaker B
I'm just not sure how much I should kiss and tell.
Speaker A
Give us a general sense of where the real concern is.
Speaker B
Yeah, I mean, most of these anecdotes come from Congress, either on the House side or the Senate side, although there's definitely executive officials who are concerned and feel like they can't say that.
Speaker A
Well, we're now seeing.I mean, we're seeing Bernie Sanders saying we should stop intelligent AI.And presumably you've talked to Senator Sanders and his team,
Speaker B
talked to him publicly on video.
Speaker A
Right.So what do you think about his proposal?Do you think that's a good proposal?
Speaker B
I think it's a good first step.I would maybe narrow some definitions of precursors for superintelligence.
And I think one of the things I like most about that proposal is sort of the direction to open diplomatic negotiations with other countries.Because we sort of can't solve this problem by just stopping the race to superintelligence here.It needs to be stopped worldwide.But I do think stopping here is a very credible first step to helping other people understand that there is a real extinction threat here.
Speaker A
So let me kind of walk through where I think your advice would be and you can tell me where it's wrong.So we're going to prevent dangerous train runs.Now, for somebody who works inside, you have oxa.For somebody who's on the outside, you mentioned for journalists, they should be writing this, they should be studying this, they should be talking about this.I did appreciate that.For people inside of the labs, would you recommend they resign, as Jacob Coxson did, as David Robinson at OpenAI just did, or do you recommend that they stay and they work on these hard problems of mechanistic interpretability, alignment those questions?
Speaker B
I mean, people working on capabilities.I would definitely recommend that they leave.I think I'd probably recommend that people working on these alignment problems also leave.Because I think it's pretty clear at this point that the alignment research is just not going to work in time or it's not anywhere close.I think that from my perspective, the only reason to stay would be to try to organize larger groups of employees to demand internal changes as a condition for their continued staying.
Speaker A
And are the real villains or the most dangerous pod, the folks making really intelligent open source AI.
Speaker B
The way that it currently works is that it takes a huge amount of investment to build an AI smarter than any other AI that's existed before.And then it takes substantially less investment to sort of distill that into a similarly intelligent, maybe slightly less intelligent model with a lot fewer resources.And that's sort of how the open source stuff is made.And so they can't really be pushing the frontier.
And the frontier is where the danger is from my perspective.
Speaker A
But they can be very close to the frontier, right?They can be frontier minus a very small percentage.And then they can get that into many more hands.
Speaker B
I mean, there's definitely a lot of potential harms that can come from that.So there's, you know, it could become very cheap for anyone to hack into anything in the next few months as we get open source AIs that are on the level of cyber hacking as the frontier AIs.A few months ago,
Speaker A
this gets to one of my concerns about your argument, which is I can imagine a world where the political system becomes convinced that there is this horrendous risk, right?And because of that, we end up in a world with a very small number of highly regulated AI companies, or in fact, perhaps even a small number of AI companies that are effectively nationalized or partially nationalized.And that would be a good way of getting control of AI.But what it does is it stops the dispersion of AI and just gives an extraordinary amount of power to a very small number of people in a very small number of companies.
Speaker B
I don't see how that argument works.I think if you're sort of stopping the dangerous training runs, that lets a lot of other companies catch up to the place that you've stopped the frontier.And if you're stopping these dangerous training runs, then we have all these computer chips, we have these data centers that are no longer being used for dangerous training runs that are now being that.Now presumably you'd want to repurpose that and make good on your investment.
Presumably you'd do that by doing a ton more inference.I
Speaker A
guess you're assuming there that there's kind of a nationalization centralization and they stop the dangerous training runs.Whereas I'm saying there's a centralization nationalization, and then because the government feels that they have effective control, there isn't a slowdown on the speed of research,
Speaker B
then it sounds like everybody dies.And I agree, that would be bad,
Speaker A
right?Everybody dies.And we don't have open source in the window between when we do, when we die.
Speaker B
I mean, I think you still have plenty of open source.The way that the open source models are made is not by having these similarly huge investments.The way that these open source models are made is by having much smaller investments and access to the frontier models that can be distilled.Now, I think that there's a thing where we might both agree is that I think it would be bad if AI was naturalized, progress continued, and that progress was not made available to the public.And you might think that's bad because no one can distill open source models from it.And I might think it's bad because then the public can't see just how fast AI is going and appropriately freak out and see how misaligned the AIs are and appropriately freak out in a way that would cause the government to stop this.
But I suspect also that if you think these open source models are so great, you probably don't really believe in their potential.
The discussion moves to biosecurity, garage biolabs, and whether layered safeguards can stop intelligent misuse. Nate argues that the best off-ramp would have been for labs to avoid racing toward human-like general intelligence and instead build more limited tools.
Speaker B
Yeah.How do you feel about an open source model that lets anybody manufacture their own plague?Are you like, it'd be really bad if we didn't have this.
Speaker A
I don't think anybody is in favor of any kind of model that allows you to manufacture your own plague.And I would.
Speaker B
We're well on our way to those models.
Speaker A
One would hope that some of the regulatory steps and in fact, in the sort of the regulatory chain that we were talking about, there's probably also a whole bunch of interesting ideas you have on the specific.Like one of the things that many of the people working in AI across the companies believe with very different views, is that you need to figure out if a plague is the worst possible outcome.What are the regulatory steps that you can take to make it harder for people to set up the kind of labs that you would need to make it harder to get the materials?
The interesting question there is, is there a way from thought to plague where you can put a blocker?Can you put a blocker somewhere in this chain that limits the odds that you get to plague, or do you have to put the blocker in the thought?
Speaker B
Again, blockers work against random processes.Blockers rarely work against intelligent processes.Like a lot of people use Swiss cheese models of security, where a bunch of stacked Swiss cheese and there's no straight line that can go all the way through.
You can think of intelligence as the line that gets to steer between each Swiss cheese.
Speaker A
Right.So you're saying that even in here, if we were to put a bunch of blockers, you make it extremely hard to set up a wet lab.You set up all this tracking, you make it extremely hard to ship these materials.You do all the things that Demosis, Sabas and others are talking about.
Even if you do those, that's just like stacking four pieces of Swiss cheese or 100 pieces of Swiss cheese.And it doesn't matter because once the thought is smart enough, it's going to go route itself through.
Speaker B
That's the eventual limit.Although in real life people don't put all the blockers in.I'm old enough to remember when people said don't worry about all this AI stuff, no one will be dumb enough to put on the Internet.
I'm old enough to remember when people said don't worry about the AIs having access to biolabs.No one would be dumb enough to hook it up to an automated biolab.The automated biolab thing happened about last month.People said, oh, the AIs will have everything cybersecure before we have the great hacking AIs.That didn't happen, right?I mean, I guess we'll see what happens before we get the open source AIs at the level of these cyber hacking AIs.But that's probably going to happen in a few months.And will all the critical infrastructure be secured then?I mean that's a test case for whether you can do a similar thing to bio.But fundamentally, how hard is it for someone to set up a biolab in their garage?
You can say, oh, the big nationalized biolabs or the huge biolabs that all the big pharmaceutical companies use, they're going to need to have some restrictions on them.How hard is it really to set up a biolab in a garage?And how are you going to detect with those biolabs in the garage?And I think we probably both agree that we shouldn't have open source models that can help someone set up a biolab in their garage and then make a plague with just a little bit of fine tuning to take off the safeguards.And we shouldn't have that widely and cheaply available to everybody.
And we at least shouldn't have that before we have.I don't know, I'm not even sure what you imagine.Do you want the set up a bio lab and make your own plague in your garage device to be $5 a month and then also want a surveillance state?What's your game plan here?
Speaker A
I'm not going to answer that question.My argument is there are real risks in centralizing AI power and there are risks to income inequality.There are risks to the way society, society is organized.And there's going to be a trade off and it's going to be an interesting trade off between one how balances the safety?Because it is of course true that if there is real dispersion and there is real open source, there are a different set of risks that come up.
My hope for the way society evolves is that you can come up with a way of risk management that allows for a dispersion of cheap, intelligent, powerful AI that can be used for tools for resetting
Speaker B
up a biolab in your garage
Speaker A
and not
Speaker B
bad plague
Speaker A
and that you can figure out what to do, you can figure out the mechanisms by which you get the good and dramatically, dramatically reduce, ideally to zero, the risks of those bad.So that's where I want to go.And the question is, can you get there through a mechanism other than nationalization, centralization, and a small dangerous
Speaker B
training runs,
Speaker A
which is an interesting way, a potential way to do it.And if you could, one of the ideas of yours that I really, really am drawn to, right.And I've been drawn to it since the very beginning, we talked about it a few minutes ago, is this notion that the best thing for AI would be if you could direct it, direct the labs, direct people working on it, towards building tools, not towards building this ultimate superintelligence.Right.More in the alphafold direction than in the let's make the world's most powerful human.That I think, in fact, this actually gets to a question that I very much want to ask, which is I have often wondered whether there was a moment where the big labs could have said, we're not going to let the machines say, I We're not going to have them type like humans, we're not going to have them sound like humans.
We're not going to try to mimic human intelligence.We're going to make them more like tools.And collectively have decided that.And I have often wondered if there's kind of an off ramp for some of the dangers that we face now.And I'm curious, A, if you think that was a useful off ramp, and B, if there were other off ramps that we missed between when you first started to worry about this in 2014, 2015, and today.
Speaker B
In chapter three of my book, we talk about how AI is probably going to become very agentic as it gets smarter and start tenaciously and doggedly pursuing goals.
I think this was one of the least rhetorically persuasive chapters in the book.And it was sort of written at a time when everyone thought that AIs were just prediction engines.And it just turned out to be totally true, as we saw from the sort of hugging face swarms.And I'm still not sure if I can rhetorically nail it, but it's hard to make a very generally intelligent tool.And the superficial interface on whether you say it's a very intelligent tool has very little to do with how dangerous it is.
Nick Bostrom was sort of thinking about this super intelligent stuff for a long time and used to say, what if our AIs, what if we made our AIs that were just predicting things, just telling us what are the consequences of various actions.But if you ask an AI, just predict what design this engineer would create, just predict what proof this mathematician would create, you can sort of see that the task of succeeding at simple prediction problems involves the same labor of Figuring out hard engineering designs and difficult math proofs.It's all the same work going on inside the AI that whether it comes out as a prediction or a plan, this is sort of a thin veneer on top of.
Speaker A
Yeah.
Speaker B
So I think making the AIs present themselves as tools, from my perspective that is a superficial choice of what mask you put on the system.
Speaker A
So it doesn't really matter.
Speaker B
It doesn't really matter.
Speaker A
Is there anything, Was there an off ramp?
Speaker B
People who really understand the dangers and understand how hard alignment is, I think were never that enthusiastic about simply scaling up large language models to see what happens.Not because we thought it wouldn't necessarily work, but because we thought it would be obviously very difficult to align to point in some direction.
Google DeepMind in some sense was like they're the ones who were making AlphaFold and they were in some sense dragged kicking and screaming into the sort of like just scale a large language model regime.And in some sense many would say that they lost their lead because of this.But I think that their approach of building AlphaFold, building the medical tech and so on was the sort of approach that had at least a better chance that you'd be able to align the things that came out or be able to not race towards the super intelligent better than the best human at every mental task type system.And OpenAI could have not started in the first place and those researchers could have joined DeepMind and said we're all going to help out in some non racing way.Or OpenAI and later anthropic could have agreed that we're just not going to scale up the black boxes that no one really understands that seem to be getting smarter while also seeming to get more and more defiant of instructions.
Those may be an off ramp.
The final stretch covers the ladder-and-race dynamic between AI companies, then shifts to reasons for hope. Nate points to growing public awareness, viral warnings from insiders, and the emergence of more alignment-focused work, even if he thinks the field is still far behind.
Speaker A
You have an amazing image in your book, you have a lot of amazing images in your book, but it's of a ladder where every rung you climb you get, I don't know, it's a billion dollars or.And at some point you climb and you hit a ladder and it blows everybody up.Right.You hit a rung, right?
Speaker B
Yeah.
Speaker A
And when you're competing with other people there's always an incentive to climb that next rung in the ladder.Right.Because there's money and well, what if the other person gets it?And if they're going to get and they're going to hit the top and blow us up anyway, you might as well climb up that rung and maybe you believe that when you get to that rung it makes you a Little safer, which I thought was the.
Of the various ways people described the kind of the game theoretical problems caused by the competition between Anthropic OpenAI and others, I thought that was the clearest one.So let's end with some hope here, Nate, because I want to feel a little better and I got to go around the rest of my day.You wrote this book, came out September 25th.It's been out in the world for a year.It's now a big part of the conversation.
Tell me some of the things that have given you hope.
Speaker B
They say that the AI capabilities double every six months, but the number of senators talking about AI also doubles every six months,
Speaker A
but we're now at what, eight?
Speaker B
No, it's probably in the 20s.The world has noticed the problem a lot more than I was expecting over the past year.
In an odd way, the Hugging Face incident gave me a lot of hope because from my perspective, it was sort of priced in that these AI companies were going to do these big reckless runs and the AIs were not quite going to be doing what they were
Speaker A
asked to do.This was.I mean, the amazing thing about the hugging face thing is they didn't actually cause.I mean, as we know, we don't know for sure, but it appears, based on what we know, that it did not cause any irrevocable damage.
Speaker B
Yeah.
Speaker A
So you got the example that proves part of the chain in the book or gives evidence for part of the chain in the book without costs.
Speaker B
That's right.And in a way that was visible to people on the outside.Like there are actually multiple swarm instances at OpenAI and they actually, many of them are less well known because they didn't involve breaking out.
Whereas as that swarm it attacked Hugging Face and Hugging Face detected the attack, didn't know who was behind it, presumably thought a human was behind it and reported it to the FBI.And so that one sort of couldn't be kept secret.So it was evidence of a number of pieces in the chain of argument.It was sort of like empirical evidence of these theoretical arguments.It was publicly legible and it fortunately didn't cause a ton of harm.
And from my perspective, that was great news.A lot of people were like, well, this is horrible news because it shows that the AIs are sort of acting outside of intended scope, as they say.And I'm like, well, that was priced in for me.So it's great news from my perspective.The Jacob Coxon tweet getting so much virality was also, from my perspective, great news.
And I think that a lot of that was related to this point you brought up earlier of a lot of people are worried and they aren't saying it out loud.
Speaker A
Did you know Jacob before he tweeted?
Speaker B
I think he reached out like a day before to be like, hey, I'm going to do this thing.Do you have any notes for me?But I didn't know him before that.
Speaker A
So that gave you hope, the response to his tweet, you did help him find the PR firm that has helped him?
Speaker B
Yeah, that was after.It was actually.It
Speaker A
was like in the middle of the tweet run, right?
Speaker B
Yeah.So he actually was like, hey, I'm thinking of quitting.Do you have any notes?And amusingly, I was actually like, you know, I mean, you could.I've seen a lot of people quit and it hasn't done anything.You know, you should also consider, like, organizing employees inside to, like, use your leverage before.That's
Speaker A
what you said him the day before he quit.
Speaker B
It was maybe two days, but one or two days before.Yeah.
And I'm super glad that he ignored my advice, you know, and then, you know, the tweet started taking off and I was like, yeah, you know, I'm like, on the news, I have these.Did
Speaker A
you know Evan, the guy who responded to it?
Speaker B
Evan used to work for me
Speaker A
at.
Speaker B
At Miri.
Speaker A
Okay.
Speaker B
And then he left for Anthropic.So, yeah, I know Evan.I've known Evan for years.
Speaker A
So it gave you hope that Jacob's tweet went so viral and there was a response and started this conversation.Evan's follow up as well.
Speaker B
I mean, it wasn't just Evan.There were a ton of people who were coming out and saying, you know, yeah, this, this is.And I think a lot of what made Jacob's tweets go so viral is that he didn't pull his punches.
You know, we see a lot of these guys being like, oh, you know, sure, there's these existential risks, but we're doing our best.And we have a team of people working on interpretability.And that hits different from someone being like, I think there's a greater than 10% chance that this kills us all within the decade.And we don't have a plan for alignment, for superintelligence.We're sort of trying to figure one out, but we don't have one.
Right.And both Jacob and Evan, I think, were sort of shooting straight shots and sort of not playing this game of soft pedaling and downplaying and that the message got out And I would say
Speaker A
the message getting out gave you hope.The conversations in the United States Senate have given you hope.
Speaker B
That's right.
Speaker A
Has any work given you hope?
Speaker B
Any work?I mean, the alignment stuff looks like it's just nowhere close right now when people are saying our key plan for how to align these giant AIs that are solving the Navy or Stokes problem, and we have very little idea what's going on in there.Our key plan is we're going to figure out a little bit more of what's going on in there.That's not a good place for the alignment research to be.
Figuring out what's going on is step one.And then being like, oh, this is totally.Like, this is totally not the sort of thing where if you scale.But
Speaker A
there is starting to be an ecosystem of companies that are like nonprofits that are working on alignment and that are trying to solve these problems and nonprofits and university research programs,
Speaker B
and it's all nowhere close.
The host asks Nate what he would fund with unlimited resources, and Nate gives a blunt answer about controlling the chip supply chain. The episode wraps with thanks, a brief outro, and subscription prompts.
Speaker A
Last question.I asked this of all my guests.If you had an unlimited budget, you could fund anything in AI, what would you do
Speaker B
in AI?I mean, I would buy the chip supply chain and I would be like, no more chips for you guys.
Speaker A
That's an amazing answer.Let's just leave it at that.Simple and clear.Thank you.Nate Soares.Amazing having you on the most interesting thing in AI.
Speaker B
Thank you.
Speaker A
That was Nate Soares of the Machine Intelligence Research Institute.If you enjoyed this episode, you might also like my conversation with this Nick Bostrom, a philosopher who quite literally wrote the book on superintelligence.
That's from May of this year.If you want more of this podcast, please subscribe wherever you watch or listen.If you're on YouTube, please leave a comment.I will see it.Probably respond.
If you're listening on Apple, Spotify or another platform, please leave us a review.Thanks for checking out the most interesting thing in AI, Sam.