caption

Max Tegmark: The Case for Halting AI Development | Lex Fridman Podcast #371

– A lot of people have said for many years that there will come a time when we want to pause a little bit. That time is now. – The following is a
conversation with Max Tegmark, his third time in the podcast. In fact, his first appearance was episode number one of this very podcast. He is a physicist and artificial intelligence
researcher at MIT, co-founder of Future of Life Institute, and Author of "Life 3.0: Being Human in the Age of
Artificial Intelligence." Most recently, he's a key figure in
spearheading the open letter calling for a six-month
pause on giant AI experiments like training GPT-4. The letter reads, "We're calling for a pause on training of models larger than
GPT-4 for six months.

This does not imply a pause
or ban on all AI research and development or the use of systems that have already been
placed in the market. Our call is specific and addresses a very small pool of actors who possesses this capability." The letter has been signed
by over 50,000 individuals, including 1800 CEOs and
over 1500 professors. Signatories include Yoshua Bengio, Stuart Russell, Elon Musk, Steve Wozniak, Yuval Noah Harari, Andrew Yang, and many others. This is a defining moment in the history of human civilization, where the balance of power between human and AI begins to shift, and Max's mind and his voice is one of the most valuable and powerful in a time like this.

His support, his wisdom, his friendship, has been a gift I'm forever
deeply grateful for. This is the Lex Fridman podcast. To support it, please
check out our sponsors in the description. And now, dear friends, here's Max Tegmark. You were the first ever
guest on this podcast, episode number one. So first of all, Max, I just have to say, thank you for giving me a chance. Thank you for starting this journey, and it's been an incredible journey, just thank you for sitting down with me and just acting like I'm
somebody who matters, that I'm somebody who's
interesting to talk to. And thank you for doing it. That meant a lot. – And thanks to you for putting your heart and soul into this. I know when you delve
into controversial topics, it's inevitable to get hit by what Hamlet talks about "The slings and arrows," and stuff. And I really admire this. It's in an era, you know, where YouTube videos are too long, and now it has to be
like a 20-minute TikTok, 20-second TikTok clip.

It's just so refreshing to see you going exactly against all of the advice and doing these really long form things, and the people appreciate it, you know. Reality is nuanced, and thanks
for sharing it that way. – So let me ask you again, the first question I've
ever asked on this podcast, episode number one, talking to you. Do you think there's intelligent life out there in the universe? Let's revisit that question. Do you have any updates? What's your view when you
look out to the stars? – So, when we look out to the stars, if you define our universe the
way most astrophysicists do, not as all of space, but the spherical region of space that we can see with our telescopes from which light has the time to reach us, since our Big Bang. I'm in the minority. I estimate that we are the only life in this spherical volume that has invented internet, the radio, has gotten
to our level of tech.

And if that's true, then it puts a lot of responsibility on us to not mess this one up. Because if it's true, it means that life is quite rare. And we are stewards of this one spark of advanced consciousness, which if we nurture it and help it grow, eventually life can spread from here, out into much of our universe, and we can have this just amazing future. Whereas, if we instead are reckless with the technology we build and just snuff it out due to stupidity or in-fighting, then, maybe the rest of cosmic
history in our universe is just gonna be playing
for empty benches. But I do think that we are actually very likely to get visited by aliens, alien intelligence quite soon. But I think we are gonna be building that alien intelligence. – So we're going to give birth to an intelligent alien civilization, unlike anything that human, the evolution here on
earth was able to create in terms of the path, the biological path it took. – Yeah, and it's gonna be
much more alien than a cat, or even the most exotic animal
on the planet right now, because it will not have been created through the usual Darwinian competition where it necessarily cares
about self-preservation, that is afraid of death, any of those things.

The space of alien
minds that you can build is just so much faster than
what evolution will give you. And with that also comes
a great responsibility, for us to make sure that
the kind of minds we create are the kind of minds
that it's good to create. Minds that will share our values and be good for humanity and life. And also don't create
minds that don't suffer. – Do you try to visualize the full space of alien minds that AI could be? Do you try to consider all the different kinds of intelligences, instead of generalizing
what humans are able to do to the full spectrum of
what intelligent creatures, entities could do? – I try, but I would say I fail, I mean, it's very difficult for human mind to really grapple with
something so completely alien. Even for us, right? If we just try to
imagine how would it feel if we were completely indifferent towards death or individuality? Even if you just imagine that for example, you could just copy my knowledge
of how to speak Swedish, (fingers snapping) boom,
now you can speak Swedish, and you could copy any
of my cool experiences, and then you could delete the ones you didn't like in your own life, just like that.

It would already change quite a lot about how you feel as
a human being, right? You probably spend less
effort studying things if you just copy them, and you might be less afraid of death, because if the plane
you're on starts to crash, you'd just be like, "Oh shucks, I haven't backed my
brain up for four hours, (Lex laughs) so I'm gonna lose this, all this wonderful
experiences of this flight." We might also start feeling more, like compassionate maybe with other people if we can so readily share
each other's experiences and our knowledge, and
feel more like a hivemind.

It's very hard though. I really feel very humble about this to grapple with it, how it might actually feel. The one thing which is so obvious though, which, I think is just
really worth reflecting on, is because the mind space
of possible intelligences is so different from ours, it's very dangerous if we assume they're gonna be like us, or anything like us. – Well there's, the entirety of human written history has been through poetry, through novels, been trying to describe
through philosophy, trying to describe the human condition and what's entailed in it. Like, just like you said, fear of death and all
those kinds of things, what is love, and all of that changes.

– [Max] Yeah. – If you have a different
kind of intelligence. Like all of it, the entirety of all those poems, they're trying to sneak up to what the hell it means to be human. All of that changes. How AI concerns and existential crises
that AI experiences, how that clashes with the
human existential crisis, the human condition. – [Max] Yeah. – That's hard to fathom, hard to predict. – It's hard, but it's
fascinating to think about also. Even in the best case scenario, where we don't lose control over the ever more powerful AI that we're building to other humans whose goals we think are horrible, and where we don't lose
control to the machines, and AI provides the things we want.

Even then, you get into the questions you touched here, you know, maybe it's the struggle that it's actually hard to do things is part of the things that
gives us meaning as well, right? So for example, I found it so shocking that this new Microsoft GPT-4 commercial that they put together, has this woman talking about, showing this demo how she's gonna give a graduation speech to
her beloved daughter. And she asks GPT-4 to write it. It was frigging 200 words or so. If I realized that my
parents couldn't be bothered struggling a little
bit to write 200 words, and outsource that to their computer, I would feel really offended, actually. And so I wonder if eliminating too much of the struggle from our existence, do you think that would also take away a little bit of what- – it means to be human? Yeah. – [Max] Yeah. – We can't even predict. I had somebody mentioned
to me that they use, they started using ChatGPT
with the 3.5 and now 4.0, to write what they
really feel to a person, and they have a temper issue, and they're basically
trying to get ChatGPT to rewrite it in a nicer way.

To get the point across, but rewrite it in a nicer way. So we're even removing the inner asshole from our communication. So I don't, you know, there's some positive aspects of that, but mostly it's just the transformation of how humans communicate. And it's scary because so much of our society is based on this glue of communication. And if we're now using AI as
the medium of communication that does the language for us, so much of the emotion that's
laden in human communication, and so much of the intent, that's going to be handled by, outsourced to AI, how does that change everything? How does that change the internal state of how we feel about other human beings? What makes us lonely? What makes us excited? What makes us afraid? How we fall in love? All that kind of stuff.

– Yeah. For me personally, I have to confess, the challenge is one of the things that really makes my life feel meaningful, you know? If I go hiking mountain
with my wife, Meia, I don't wanna just press a
button and be at the top, I want to struggle and
come up there sweaty, and feel, "Wow, we did this," in the same way. I want to constantly work on myself to become a better person. If I say something in anger that I regret, I want to go back and really work on myself rather than just tell an AI, from now on, always filter what I write so I don't have to work on myself, 'cause then I'm not growing. – Yeah, but then again, it could be like with chess, and AI, once it significantly, obviously supersedes the
performance of humans, it will live in its own world, and provide maybe a flourishing
civilizations for humans.

But we humans will
continue hiking mountains, and playing our games, even though AI is so much smarter, so much stronger, so much superior in every single way, just like with chess. – [Max] Yeah. – So that, I mean, that's one possible
hopeful trajectory here, is that humans will continue to human, and AI will just be a kind of, a medium that enables the
human experience to flourish.

– Yeah, I would phrase that
as rebranding ourselves from Homo sapiens to Homo sentiens. You know, right now, if it's sapiens, the ability to be intelligent, we've even put it in our species' name. So we're branding
ourselves as the smartest information processing
entity on the planet. That's clearly gonna change
if AI continues ahead. So maybe we should focus
on the experience instead, the subjective experience that we have, Homo sentiens, and say that's
what's really valuable, the love, the connection,
the other things, and get off our high horses, and get rid of this hubris that, you know, only we can do integrals.

– So consciousness, the subjective experience
is a fundamental value to what it means to be human. Make that the priority. – That feels like a
hopeful direction to me. But that also requires more compassion, not just towards other humans, because they happen to be
the smartest on the planet, but also towards all our
other fellow creatures on this planet. I personally feel right now, we're treating a lot of farm
animals horribly, for example. And the excuse we're using is, "Oh, they're not as smart as us." But if we admit that we're not that smart in the grand scheme of things either, in the post-AI epoch, you know, then surely, we should value the subjective experience of a cow also. – Well, allow me to
briefly look at the book, which at this point is becoming
more and more visionary that you've written, I
guess over five years ago, "Life 3.0." So first of all, 3.0, what's 1.0, what's 2.0, What's 3.0? and how's that vision sort of evolve, the vision in the book evolve to today.

– Life 1.0 is really dumb like bacteria, and that it can't actually
learn anything at all during their lifetime. The learning just comes
from this genetic process from one generation to the next. Life 2.0 is us and other
animals which have brains which can learn during
their lifetime a great deal. Right so, and you know, you were born without being able to speak English, and at some point you decided, "Hey, I wanna upgrade my software, and so let's install an
English-speaking module." So you did. And Life 3.0, which does not exist yet, cannot replace not only its
software the way we can, but also it's hardware. And that's where we're
heading towards at high speed. We're already maybe 2.1 because we can, you know, put in an artificial knee, pacemaker, et cetera, et cetera.

And if Neuralink and
other companies succeed, it will be life 2.2, et cetera. But the companies trying to build AGI, or trying to make is of course, full 3.0, and you can put that intelligence in something that also has no, biological basis whatsoever. – So less constraints
and more capabilities, just like the leap from 1.0 to 2.0. There is nevertheless, you speaking so harshly about bacteria, so disrespectfully about bacteria, there is still the same
kind of magic there that permeates life 2.0 and 3.0. It seems like maybe the
thing that's truly powerful about life, intelligence,
and consciousness, was already there in 1.0.

Is it possible? – I think we should be
humble and not be so quick to make everything binary and say either it's there or it's not. Clearly there's a great spectrum and there is even controversy about whether some unicellular
organisms like amoebas can maybe learn a little
bit, you know, after all. So apologies if I offended
any bacteria here. (laughs) It wasn't my intent. It was more that I wanted to talk up how cool it is to actually have a brain. – [Lex] Yeah. – Where you can learn
dramatically within your lifetime. – [Lex] Typical human. – And the higher up you
get from 1.0 2.0 to 3.0, the more you become the
captain of your own ship, the master of your own destiny. And the less you become a slave to whatever evolution gave you, right? By upgrading your software, we can be so different
from previous generations and even from our parents, much more so than even a bacterium, you know, no offense to them.

And if you can also swap out your hardware and take any physical
form you want, of course, it's really, the sky's the limit. – Yeah, so the, it accelerates the rate
at which you can perform the computation that
determines your destiny. – Yeah, and I think it's worth commenting a bit on what "you" means in this context. Also, if you swap things out a lot, right? This is controversial, but my, current understanding is that, you know, life is best thought of not as a bag of meat, or even a bag of elementary particles, but rather as a system which
can process information and retain its own complexity, even though nature is
always trying to mess it up, so, it's all about information processing. And that makes it a lot like something like a wave in the ocean, which is not, it's water molecules, right? The water molecules bob up and down, but the wave moves forward, it's an information pattern
in the same way you, Lex, you're not the same atoms as during the first, – Time we talked, yeah.
– Interview you did with me, you've swapped out most of them, but it's still you.

And the information
pattern is still there, and if you could swap out your arms, and like whatever, you can still have this
kind of continuity, it becomes much more sophisticated sort of way forward in time where the information lives on. I lost both of my parents
since our last podcast, and it actually gives me a lot of solace that this way of thinking about them, they haven't entirely died because a lot of mommy and daddy's, sorry, I'm getting a
little emotional here, but a lot of their values, and ideas, and even jokes and so on, they haven't gone away, right? Some of them live on, I can carry on some of them, and they also live on a
in a lot of other people.

So in this sense, even with life 2.0, we can to some extent, already transcend our
physical bodies and our death. And particularly if you can
share your own information, your own ideas with many others like you do in your podcast, then you know, that's the closest immortality we can get with our bio bodies. – You carry a little bit of
them in you in some sense. – [Max] Yeah, yeah. – Do you miss them? Do you miss your mom and dad? – Of course, of course.

– What did you learn about life from them? If we can take a bit of a tangent. – Oh, so many things. For starters, my fascination for math and the physical
mysteries of our universe, I got a lot of that from my dad. But I think my obsession
for really big questions, and consciousness, and so on, that actually came mostly from my mom and what I got from both of them, which is very core part
of really who I am, I think is this, just feeling comfortable with, not buying into what
everybody else is saying, just doing what I think is right. They both very much just, you know, did their own thing, and sometimes they got flak for it and they did it anyway.

– That's why you've always
been in an inspiration to me. That you're at the top of your field and you're still willing to tackle the big
questions in your own way. You're one of one of the people that represents MIT best to me, you've always been an inspiration in that. So it's good to hear that you got that from your mom and dad. – Yeah, you're too kind. But yeah, I mean, the good reason to do science is because you're really curious, and you wanna figure out the truth. If you think, this is how it is and everyone else says, "No, no, that's bullshit,
and it's that way," you know, You stick with what you think is true, and even if everybody else
keeps thinking it's bullshit, there's a certain, I always root for the underdog, (Lex laughs)
when I watch movies. And my dad once, one time for example, when I wrote one of my
craziest papers ever, talking about our universe
ultimately being mathematical, which we're not gonna get into today, I got this email from a quite
famous professor saying, "This is not only all bullshit, but it's gonna ruin your career.

You should stop doing this kind of stuff." I sent it to my dad. Do you know what he said? – [Lex] (laughs) What he say? – He replied with a quote from Dante. (Lex laughing) (Max speaking in Italian) "Follow your own path
and let the people talk." (Both laughing) Go dad! – [Lex] Yeah. – This is the kind of thing, you know, he's dead, but that attitude is not. – How did losing them as a man, as a human being change you? How did it expand your
thinking about the world? How did it expand your thinking about, you know, this thing we're talking about, which is humans creating another living, sentient perhaps, being? – I think it, mainly do two things.

One of them just going
through all their stuff after they had passed away and so on, just drove home to me how
important it is to ask ourselves, why are we doing this things we do? Because it's inevitable
that you look at some things they spent an enormous time on and you ask in hindsight, would they really have
spent so much time on this? Would they have done something that was more meaningful? So I've been looking more
in my life now and asking, you know, why am I doing what I'm doing? And I feel, it should either be something
I really enjoy doing, or it should be something that I find really, really meaningful
because it helps humanity, and if it's in none of
those two categories, maybe I should spend less
time on it, you know.

The other thing is, dealing with death up in person like this, it's actually made me less afraid of, even less afraid of
other people telling me that I'm an idiot, you know, which happens regularly, and just live my life,
do my thing, you know? And it's made it a
little bit easier for me to focus on what I feel
is really important. – What about fear of your own death? Has it made it more real that this is something that happens? – Yeah, it's made it extremely real, and you know, I'm next in line in our family now, right? It's me and my younger brother, but, they both handled it with such dignity, that was a true inspiration also.

They never complained about things, and you know, when you're old and your body starts falling apart, it's more and more to complain about, they looked at what could they
still do that was meaningful, and they focused on that rather than wasting time talking about, or even thinking much about things they were disappointed in. I think anyone can make
themselves depressed if they start their morning by
making a list of grievances. Whereas if you start your day
when the little meditation and just the things you're grateful for, you basically choose to be a happy person. – Because you only have
a finite number of days, we should spend them, – [Max] Make it count. – Being grateful. – [Max] Yeah. – Well you do happen to
be working on a thing which seems to have potentially, some of the greatest impact
on human civilization of anything humans have ever created, which is artificial intelligence.

This is, on the both
detailed technical level, and on the high philosophical
level you work on. So you've mentioned to me that there's an open letter
that you're working on. – It's actually going live in a few hours. (Lex laughing) So I've been having late
nights and early mornings. It's been very exciting, actually. In short, have you seen, "Don't Look Up," the film? – Yes, yes. – I don't want to be the movie spoiler for anyone watching
this who hasn't seen it. But if you're watching this, you haven't seen it, watch it, because we
are actually acting out, it's life imitating art.

Humanity is doing exactly that right now, except it's an asteroid that
we are building ourselves. Almost nobody is talking about it. People are squabbling across the planet about all sorts of things, which seem very minor
compared to the asteroid that's about to hit us, right? Most politicians don't
even this on the radar, they think maybe in 100 years or whatever. Right now we're at a fork on the road. This is the most important
fork that humanity has reached in it's over 100,000 years on this planet.

We're building effectively a new species that's smarter than us, it doesn't look so much like a species yet 'cause it's mostly not embodied in robots. But that's the technicality
which will soon be changed. And this arrival of of
artificial general intelligence that can do all our jobs as well as us, and probably shortly
thereafter, superintelligence, which greatly exceeds
our cognitive abilities. It's gonna either be the best thing ever to happen to humanity or the worst. I'm really quite confident that there is not that
much middle ground there. – But it would be
fundamentally transformative to human civilization. – Of course, utterly and totally. Again, we'd branded
ourselves as Homo sapiens 'cause it seemed like the basic thing, we're the king of the
castle on this planet, we're the smart ones, we can control everything else, this could very easily change. We're certainly not gonna be the smartest on the planet for very long if AI, unless AI progress just halts, and we can talk more about
why I think that's true 'cause it's controversial.

And then we can also talk about reasons we might think it's
gonna be the best thing ever, and the reason we think it's going to be the end of humanity, which is of course, super controversial. But what I think we can, anyone who's working on advanced AI can agree on is, it's much like the film "Don't Look Up," in that it's just really comical how little serious public
debate there is about it, given how huge it is. – So what we're talking
about is a development, of currently, things like GPT-4, and the signs it's showing
of rapid improvement that may, in the near
term lead to development of superintelligent AGI, AI, general AI systems, and what kind of impact
that has on society.

– [Max] Exactly. – When that thing achieves
general human-level intelligence, and then beyond that, general superhuman level intelligence. There's a lot of
questions to explore here. So one, you mentioned halt. Is that the content of the letter? is to suggest that maybe we should pause the development of these systems. – Exactly, so this is very controversial, from when we talked the first time, we talked about how I was involved in starting the Future of Life Institute, and we worked very hard on 2014, 2015, was the mainstream AI safety. The idea that there even could be risks and that you could do things about them. Before then, a lot of people thought it was just really kooky
to even talk about it. And a lot of AI researchers felt, worried that this was too flaky, and could be bad for funding, and that the people had
talked about it were just not, didn't understand AI. I'm very, very happy with how that's gone, and that now, you know, it's completely mainstream, you go in any AI conference, and people talk about AI safety, and it's a nerdy technical
field full of equations and blah-blah.

– [Lex] Yes. – As it should be, but there is this other thing, which has been quite taboo up until now, calling for slowdown. So what, we've constantly been
saying, including myself, I've been biting my tongue a lot, you know, is that, we don't need to slow down AI development. We just need to win this race, the wisdom race between
the growing power of the AI and the growing wisdom
with which we manage it. And rather than trying to slow down AI, let's just try to accelerate the wisdom, do all this technical work to figure out how you can actually ensure
that your powerful AI is gonna do what you want it to do. And have society adapt also with incentives and regulations so that these things get put to good use. Sadly, that didn't pan out. The progress on technical AI capabilities has gone a lot faster
than many people thought back when we started this in 2014.

Turned out to be easier to build really advanced AI than we thought. And on the other side, it's gone much slower than we hoped with getting policymakers and others to actually put incentives
in place to make, steer this in the good directions, maybe we should unpack it and talk a little bit about each, so. – [Lex] Yeah. – Why did it go faster than
a lot of people thought? In hindsight, it's exactly
like building flying machines. People spent a lot of time wondering about how do birds fly, you know. And that turned out to be really hard. Have you seen the TED
Talk with a flying bird? – Like a flying robotic bird? – Yeah, it flies around the audience, but it took 100 years longer to figure out how to do that than for the Wright brothers
to build the first airplane because it turned out there
was a much easier way to fly. And evolution picked
a more complicated one because it had its hands tied. It could only build a machine
that could assemble itself, which the Wright brothers
didn't care about that, they could only build a machine that use only the most common atoms
in the periodic table, Wright Brothers didn't care about that, they could use steel, iron atoms, and it had to be built to repair itself, and it also had to be
incredibly fuel efficient, you know, a lot of birds use less than half the fuel of a remote-controlled plane
flying the same distance, For humans, just throw
a little more money, put a little more fuel in it, and there you go, 100 years earlier.

That's exactly what's happening now with these large language models. The brain is incredibly complicated. Many people made the mistake, you're thinking we have to
figure out how the brain does human-level AI first before we could build in the machine, that was completely wrong. You can take an incredibly simple computational system called
a transformer network and just train it to do
something incredibly dumb. Just read a gigantic amount of text and try to predict the next word. And it turns out, if you just throw a ton of compute at that and a ton of data, it gets to be frighteningly
good like GPT-4, which I've been playing with so much since it came out, right? And there's still some debate about whether that can get you all the way to full human level or not, but yeah, we can come back
to the details of that and how you might get the human-level AI even if large language models don't.

– Can you briefly, if it's just a small tangent, comment on your feelings about GPT-4? So just that you're impressed
by this rate of progress, but where is it? Can GPT-4 reason? What are like the intuitions? What are human interpretable
words you can assign to the capabilities of GPT-4 that makes you so damn impressed with it? – I'm both very excited
about it and terrified.

It's interesting mixture
of emotions. (laughs) – All the best things in life
include those two somehow. – Yeah, it can absolutely reason, anyone who hasn't played with it, I highly recommend doing
that before dissing it. It can do quite remarkable reasoning. I've had to do a lot of things, which I realized I couldn't
do that myself that well even, and it obviously does it dramatically faster than we do too, when you watch it type, and it's doing that well, servicing a massive number of
other humans at the same time. The same time, it cannot reason as well as a human can on some tasks, it's obviously the limitations
from its architecture. You know, we have in our heads, what in geek-speak is called
a recurrent neural network. There are loops, information can go from this neuron, to this neuron, to this neuron, and then back to this one, you can like ruminate on
something for a while, you can self-reflect a lot. These large language models, they cannot, like GPT-4. It's a so-called transformer where it's just like a one-way street of information, basically.

In geek-speak, it's called a
feed-forward neural network. And it's only so deep, so it can only do logic
that's that many steps and that deep, and it's not, so you can create problems
which it will fail to solve, you know, for that reason. But the fact that it
can do so amazing things with this incredibly simple
architecture already, is quite stunning, and what we see in my lab at MIT when we look inside large language models to try to figure out how they're doing it, which, that's the key core
focus of our research, it's called mechanistic
interpretability in geek-speak.

You know, you have this machine
that does something smart, you try to reverse engineer it, and see how does it do it. I think of it also as
artificial neuroscience, (Lex laughs)
'Cause that's exactly – I love it.
– what neuroscientists do with actual brains. But here you have the
advantage that you can, you don't have to worry
about measurement errors. You can see what every
neuron is doing all the time, and a recurrent thing
we see again and again, there's been a number of beautiful papers quite recently by a lot of researchers, and some of 'em are
here even in this area, is where when they figure
out how something is done, you can say, "Oh man, that's
such a dumb way of doing it." And you read immediately
see how it can be improved.

Like for example, there was this beautiful paper recently where they figured out how a large language model
stores certain facts, like Eiffel Tower is in Paris, and they figured out
exactly how it's stored and the proof of that they understood it was they could edit it. They changed some synapses in it, and then they asked it,
Where's the Eiffel Tower?" And it said, "It's in Rome." And then they asked,
"How do you get there? Oh, how do you get there from Germany?" "Oh, you take this train, the Roma Termini train station, and this and that," "And what might you see
if you're in front of it?" "Oh, you might see the Colosseum." So they had edited, – So they literally moved it to Rome.

– But the way that it's
storing this information, it's incredibly dumb, if any fellow nerds listening to this, there was a big matrix, and roughly speaking, there are certain row and column vectors which encode these things, and they correspond very hand-wavingly to principle components and it would be much more
efficient for as far as matrix, just store in the database, you know and, and everything so far, we've figured out how these things do are ways where you can see
it can easily be improved. And the fact that this
particular architecture has some roadblocks built into it is in no way gonna
prevent crafty researchers from quickly finding workarounds and making other kinds of architectures sort of go all the way, so. In short, it's turned
out to be a lot easier to build close to human
intelligence than we thought, and that means our runway as a species to get our shit together has has shortened.

– And it seems like the scary thing about the effectiveness
of large language models, so Sam Altman, I've recently
had conversation with, and he really showed that
the leap from GPT-3 to GPT-4 has to do with just a bunch of hacks, a bunch of little explorations
with smart researchers doing a few little fixes here and there. It's not some fundamental leap and transformation in the architecture. – And more data and more compute. – And more data and compute, but he said the big leaps has to do with not the data and the compute, but just learning this new discipline, just like you said. So researchers are going to
look at these architectures and there might be big
leaps where you realize, "Wait, why are we doing
this in this dumb way?" And all of a sudden this
model is 10x smarter. And that that can happen on any one day, on any one Tuesday or Wednesday afternoon. And then all of a sudden you have a system that's 10x smarter.

It seems like it's such a new discipline, it's such a new, like we understand so little about why this thing works so damn well, that the linear improvement of compute, or exponential, but the steady improvement of compute, steady improvement of the data may not be the thing that
even leads to the next leap. It could be a surprise little
hack that improves everything. – Or a lot of little leaps here and there because so much of this
is out in the open also, so many smart people are looking at this and trying to figure out
little leaps here and there, and it becomes this sort
of collective race where, a lot of people feel, "If I don't take the
leap someone else will," and it is actually very crucial
for the other part of it, why do we wanna slow this down? So again, what this open
letter is calling for is just pausing all training of systems that are more powerful
than GPT-4 for six months. Just give a chance for the labs to coordinate
a bit on safety, and for society to adapt, give the right incentives to the labs.

'cause I, you know, you've interviewed a lot of
these people who lead these labs and you know just as well as I do that they're good people, they're idealistic people. They're doing this first and foremost because they believe that AI has a huge potential to help humanity. But at the same time they are trapped in this horrible race to the bottom. Have you read "Meditations on Moloch" by Scott Alexander? – [Lex] Yes. – Yeah, it's a beautiful
essay on this poem by Ginsberg where he interprets it as
being about this monster. It's this game theory
monster that pits people against each other in
this race to the bottom where everybody ultimately loses. And the evil thing about this monster is even though everybody
sees it and understands, they still can't get
out of the race, right? A good fraction of all the
bad things that we humans do are caused by Moloch. And I like Scott Alexander's
naming of the monster. So we can, we humans can think of it as a thing.

If you look at why do we have overfishing, why do we have more generally, the tragedy of the commons. Why is it that, so Liv Boeree, I don't know if you've
had her on your podcast. – Mhm, yeah. She's become a friend, yeah. – Great, she made this
awesome point recently that beauty filters that a lot of female influencers feel pressure to use, are exactly Moloch in action again. First, nobody was using them, and people saw them
just the way they were, and then some of 'em started using it, and becoming ever more plastic fantastic, and then the other ones
that weren't using it started to realize that, if they wanna to keep
their their market share, they have to start using it too. And then you're in a situation
where they're all using it, and none of them has any more market share or less than before. So nobody gained anything, everybody lost, and they have to keep becoming ever more plastic fantastic also, right? But nobody can go back to the old way because it's just too costly, right? Moloch is everywhere, and Moloch is not a new
arrival on the scene either.

We humans have developed a lot
of collaboration mechanisms to help us fight back against Moloch through various kinds of
constructive collaboration. The Soviet Union and the United States did sign a number of arms control treaties against Moloch who is trying to stoke them into unnecessarily risky
nuclear arms races, et cetera, et cetera. And this is exactly what's
happening on the AI front. This time it's a little bit geopolitics, but it's mostly money, where there's just so
much commercial pressure.

You know, if you take any of these leaders of the top tech companies, if they just say, you know, "This is too risky, I want
to pause for six months." They're gonna get a lot of pressure from shareholders and others. They're like, "Well
you know, if you pause, but those guys don't pause. We don't wanna get our lunch eaten." – [Lex] Yeah. – And shareholders even have the power to replace the executives
in the worst case, right? So we did this open letter
because we want to help these idealistic tech executives to do what their heart tells them, by providing enough public
pressure on the whole sector.

Just pause, so that they can all pause in a coordinated fashion. And I think without the public pressure, none of them can do it alone. Push back against their shareholders no matter how goodhearted they are, 'cause Moloch is a really powerful foe. – So the idea is to, for the major developers
of AI systems like this, so we're talking about Microsoft, Google, Meta, and anyone else. – Well OpenAI is very
close with Microsoft now, – With Microsoft, right, yeah.
– of course, – And there there are
plenty of smaller players. for example, Anthropic
is is very impressive, there's Conjecture, there's many, many, many players, I don't wanna make a long list that sort of leave anyone out. And for that reason, it's so important that
some coordination happens, that there's external
pressure on all of them, saying, "You all need the pause." 'Cause then, the people, the researchers in there
at these organizations, the leaders who wanna
slow down a little bit, they can say to their
shareholders, you know, "Everybody's slowing down
because of this pressure and it's the right thing to do." – Have you seen in history, there examples what it's possible to pause the Moloch?
– Yes, absolutely.

And even like human cloning for example, you could make so much
money on human cloning. Why aren't we doing it? Because biologists thought hard about this and felt like this is way too risky, they got together in the
seventies in Asilomar, and decided even to stop a lot more stuff, also just editing the
human germline, right? Gene editing that goes
in to our offspring, and decided, "Let's not do this because it's too unpredictable
what it's gonna lead to," we could lose control over
what happens to our species," so they paused.

There was a ton of money to be made there, So it's very doable, but you need a public awareness
of what the risks are, and the broader community
coming in and saying, "Hey, let's slow down." And you know, another
common pushback I get today, is that we can't stop in
the West because China. And in China undoubtedly, they also get told, "We can't
slow down because the West," because both sides think
they're the good guy. – [Lex] Yeah. – But look at human cloning, you know? Did China forge ahead with human cloning? There's been exactly one human cloning that's actually been done that I know of. It was done by a Chinese guy. Do you know where he is now? – [Lex] Where? – In jail. And you know who put him there? – [Lex] Who? – Chinese government.

Not because Westerners said, "China look, this is…" No the Chinese government put him there 'cause they also felt, they like control, the Chinese government. If anything, maybe they're
even more concerned about having control than Western governments, have no incentive of just losing control over where everything is going, and you can also see the Ernie Bot that was released by, I believe, Baidu recently, they got a lot of pushback
from the government and had to rein it in,
you know, in a big way.

I think once this basic message comes out that this isn't an arms
race, it's a suicide race, where everybody loses if anybody's AI goes out of control, it really changes the whole dynamic. It's not, and I'll say this again 'cause this is this very basic point I think a lot of people get wrong. Because a lot of people
dismiss the whole idea that AI can really get very superhuman because they think there's something really magical about intelligence such that it can only
exist in human minds, you know, because they believe that, they think it's gonna kind
of get to just more or less "GPT-4 plus plus," and then that's it.

They don't see it as a suicide race. They think whoever gets that first, they're gonna control the world, they're gonna win. That's not how it's gonna be. And we can talk again about
the scientific arguments from why it's not gonna stop there. But the way it's gonna be, is if anybody completely loses control and you know, you don't care if someone manages to take over the world who really doesn't share your goals, you probably don't really
even care very much about what nationality they have, you're not gonna like it
much worse than today. If you live in Orwellian dystopia, what do you care who's created it, right? And if someone, if it goes farther, and we just lose control
even to the machines, so that it's not us versus them, it's us versus it. What do you care who created
this unaligned entity which has goals different
from humans, ultimately? And we get marginalized,
we get made obsolete, we get replaced.

That's what I mean when I
say it's a suicide race, it's kind of like we're
rushing towards this cliff, but the closer the cliff we get, the more scenic the views are, and the more money there is there, and the more, so we keep going, but we have to also stop
at some point, right? Quit while we're ahead, And it's, it's a suicide race which cannot be won, but the way to really benefit from it is, to continue developing awesome
AI a little bit slower. So we make it safe, make sure it does the
things that humans want, and create a condition
where everybody wins.

The technology has shown us that, you know, geopolitics
and politics in general is not a zero sum game at all. – So there is some rate of
development that will lead us as a human species to
lose control of this thing. And the hope you have is that there's some
lower level of development which will not allow us to lose control. This is an interesting thought you have about losing control, so if you have somebody, if you are somebody like Sundar Pichai or Sam Altman at the head
of a company like this, you're saying if they develop an AGI, they too will lose control of it.

So no one person can maintain control, no group of individuals
can maintain control. – If it's created very, very soon and is a big black box
that we don't understand like the large language models, yeah. Then I'm very confident
they're gonna lose control. But this isn't just me
saying it, you know, Sam Altman and Demis Hassabis have both said, they themselves acknowledge that, you know, there's really
great risks for this and they want slow down once
they feel it gets scary. But it's clear that they're stuck in this, again, Moloch is forcing
them to go a little faster than they're comfortable with because of pressure from, just commercial pressures, right? To get a bit optimistic here, of course, this is a problem
that can be ultimately solved. To win this wisdom race, it's clear that what we
hope that was gonna happen hasn't happened. The capability progress has gone faster than a lot of people thought, and the progress in the public sphere of policy making and so on, has gone slower than we thought.

Even the technical AI
safety has gone slower. A lot of the technical safety research was kind of banking on
that large language models and other poorly understood systems couldn't get us all the way. That you had to build more
of a kind of intelligence that you could understand. Maybe it could prove itself safe, you know, things like this, and I'm quite confident
that this can be done so we can reap all the benefits, but we cannot do it as quickly as this out of control
express train we are on now is gonna get to AGI. That's why we need a
little more time, I feel. – Is there something to be said, well like Sam Altman talked about, which is while we're in the pre-AGI stage, to release often and as
transparently as possible to learn a lot. So as opposed to being extremely cautious, release a lot, don't invest in a closed development where you focus on the AI safety. While it's somewhat "dumb," quote-unquote, release as often as possible. And as you start to see signs
of human-level intelligence and or superhuman level intelligence, then you put a halt on it.

Well what a lot of safety researchers have been saying for many years is that the most dangerous
things you can do with an AI is first of all
teach it to write code. – [Lex] Yeah. – Because that's the first step towards recursive self-improvement, which can take it from
AGI to much higher levels. Okay? Oops, we've done that. And another thing high risk is connect it to the internet, let it go to websites, download stuff on its
own and talk to people.

Oops, we've done that already. You know Eliezer Yudkowsky, you said you interviewed
him recently, right? – [Lex] Yes, yep. – So he had this tweet
recently which said, gave me one of the best laughs in a while, where he is like, "Hey, people used to
make fun of me and say, 'You're so stupid, Eliezer.' 'Cause you're saying you have to worry of obviously developers once they get to like really strong AI, first thing you're gonna do is like, never connect it to the internet, keep it in a box.

Where you know, you can
really study it safe." So he had written it in
the like in the meme form so it's like "Then," and then that, and then, "Now." (Lex laughing) "LOL, let's make a chatbot." (both laughing) – [Lex] Yeah, yeah, yeah. – And the third thing is Stuart Russell. – [Lex] Yeah. – You know, amazing AI researcher. He has argued for a while
that we should never teach AI anything about humans. Above all, we should never let it learn about human psychology and
how you manipulate humans.

That's the most dangerous kind
of knowledge you can give it. Yeah, you can teach it
all it needs to know about how to cure cancer
and stuff like that. But don't let it read
Daniel Kahneman's book about cognitive biases and all that. And then oops, "LOL, you know, let's invent social media recommender algorithms
which do exactly that." They get so good at knowing
us and pressing our buttons that we are starting to create a world now where we just have ever more hatred, 'cause they've figured
out that these algorithms, not for out of evil, but just to make money on advertising, that the best way to get more engagement, the euphemism, get people glued to their
little rectangles, right? Is just to make them pissed off. – Well that's really interesting that a large AI system that's
doing the recommender system kind of task on social media, is basically just studying human beings because it's a bunch of
us rats giving it signal, nonstop signal. It'll show a thing and
then we give signal, and whether we spread that
thing, we like that thing, that thing increases our engagement, gets us to return to the platform, and it has that on the scale of hundreds of millions
of people constantly.

So it's just learning, and
learning, and learning, and presumably if the number of parameters in the neural network
that's doing the learning, and more end to end the learning is, the more it's able to
just basically encode how to manipulate human behavior. – [Max] Exactly. – How to control humans at scale. – Exactly, and that is
not something you think is in humanity's interest. And right now it's mainly letting some humans manipulate other
humans for profit and power, which already caused a lot of damage, and then eventually that's a sort of skill that can make AI persuade
humans to let them escape whatever safety precautions we had put, you know, there was a really nice article in the New York Times
recently by Yuval Noah Harari and two co-authors
including Tristan Harris from "The Social Dilemma," and we have this phrase in there I love, It said, "Humanity's first
contact with advanced AI was social media." And we lost that one.

We now live in a country where there's much more hate in the world where there's much more hate, in fact. And in our democracy than
we're having this conversation, and people can't even agree on who won the last election, you know. And we humans often point fingers at other humans and say it's their fault, but it's really Moloch
in these AI algorithms. We got the algorithms and then Moloch pitted the social media
companies against each other so nobody could have a
less creepy algorithm 'cause then they would lose out on revenue to the other company. – Is there any way to win that battle back if we just linger on this one battle that we've lost in terms of social media, is it possible to redesign social media, this very medium in which
we use as a civilization to communicate with each other, to have these kinds of conversation, to have discourse, to try to figure out how to solve the biggest problems in the world, whether that's nuclear war
or the development of AGI.

Is is it possible to do
social media correctly? – I think it's not only
possible, but it's necessary. Who are we kidding? That we're gonna be able to
solve all these other challenges if we can't even have a
conversation with each other? It's constructive. The whole idea, the key idea of democracy is that you get a bunch of people together and they have a real conversation. The ones you try to foster on this podcast where you respectfully listen
to people you disagree with. And you realize actually, you know, there are some things actually some common ground we have and let's, we both agree, let's not
have any nuclear wars, let's not do that, et cetera, et cetera. We're kidding ourselves that
thinking we can face off the second contact with
ever more powerful AI that's happening now with these large language
models if we can't even have a functional conversation
in the public space.

That's why I started the
Improve The News project, improvethenews.org. But I'm an optimist fundamentally, in that there is a lot of
intrinsic goodness in people. And that what makes the difference between someone doing
good things for humanity and bad things is not some
sort of fairytale thing, that this person was
born with the evil gene and this one is born with the good gene. No, I think it's whether we put, whether people find
themselves in situations that bring out the best in them or that bring out the worst in them. And I feel we're building an internet and a society that brings out the worst. – But it doesn't have to be that way. – [Max] No, it does not. – It's possible to create incentives and also create incentives
that make money. That both make money and
bring out the best in people. – I mean, in the long term, it's not a good investment
for anyone, you know, to have a nuclear war, for example.

And you know, is it a good investment for humanity if we just ultimately replace
all humans by machines, and then we're so
obsolete that eventually, there are no humans left? Well, it depends guess
how you do the math, But I would say by any
reasonable economic standard, if you look at the future income of humans and there aren't any, you know, that's not a good investment. Moreover, like why can't we have a little bit of pride
in our species, damn it? You know, why should we just build another species that gets rid of us? If we were Neanderthals, would we really consider it a smart move if we had really advanced
biotech to build Homo sapiens? You know, you might say, "Hey Max, you know, yeah, let's build, these Homo sapiens, they're
gonna be smarter than us, maybe they can help us, defend us better against predators and help fix up our
caves, make them nicer, we'll control 'em undoubtedly, you know?" So then they build a couple, a little baby girl, little baby boy.

They either, and then you have some wise
old Neanderthal elder is like, "Hmm, I'm scared that we're
opening a Pandora's box here and that we're gonna
get outsmarted by these super Neanderthal intelligences, and there won't be any Neanderthals left." But then you have a bunch of
others in the cave, right? "You're such a Luddite scaremonger. Of course, they're gonna
want to keep us around 'cause we are their creators, and, you know, the smarter, I think the smarter they get, the nicer they're gonna get, they're gonna leave us.

They're gonna want us around
and it's gonna be fine, and besides look at these
babies, they're so cute. Clearly they're totally harmless." Those babies are exactly GPT-4. It's not, I wanna be clear, it's not GPT-4 that's terrifying. It's that GPT-4 is a baby technology, you know, and Microsoft even
had a paper recently out, titled something like, "Sparkles of AGI." Well they were basically
saying this is baby AI, like these little Neanderthal babies, and it's gonna grow up. There's gonna be other
systems from the same company, from other companies, they'll be way more powerful, but they're gonna take all the things, ideas from these babies
and before we know it, we're gonna be like
those last Neanderthals who were pretty disappointed when they realized that
they were getting replaced. – Well, this interesting point you make, which is of programming, it's entirely possible that GPT-4 is already the kind of system that can change everything
by writing programs. – Yeah, it's because it's life 2.0, the systems I'm afraid
of are gonna look nothing like a large language
model, and they're not, but once it gets, once it or other people figure out a way of using this tech to make
much better tech, right? It's just constantly
replacing its software.

And from everything that we've seen about how these work under the hood, they're like the minimum
viable intelligence. They do everything, you know, the dumbest way that still works, sort of. – [Lex] Yeah. – And so they're life 3.0, except when they replace their software, it's a lot faster than when
you decide to learn Swedish. Poof. (fingers snapping) And moreover, they think
a lot faster than us too. So when, you know, we don't think, have one logical step every nanosecond or few, or so, the way they do, and we can't also just
suddenly scale up our hardware massively in the cloud 'cause
we're so limited, right? So they are, and they are also life, can soon become a little
bit more like life 3.0 in that if they need more hardware, hey, just rent it in the cloud, you know? "How do you pay for it?" "Well, with all the services you provide." – And what we haven't seen yet, which could change a lot, is entire software systems.

So right now programming is
done sort of in bits and pieces as an assistant tool to humans. But I do a lot of programming and with the kind of stuff
that GPT-4 is able to do, I mean, it's replacing a lot
what I'm able to do, right? You still need a human in the loop to kind of manage the design of things, manage like, what are the prompts that generate the kind of stuff to do some basic adjustment of the codes, do some debugging, but if it's possible
to add on top of GPT-4, kind of a feedback loop of self-debugging, improving the code, and then you launch that
system onto the wild on the internet because
everything is connected, and have it do things, have it interact with humans
and then get that feedback, now you have this giant
ecosystem of humans. That's one of the things that Elon Musk recently sort of tweeted as a case why everyone
needs to pay $7 or whatever for Twitter, – [Max] To make sure they're real.

– Make sure they're real, we're now going to be living in a world where the bots are getting smarter, and smarter, and smarter
to a degree where, you can't tell the difference between a human and a bot. – [Max] That's right. – And now you can have
bots outnumber humans by 1 million to one. Which is why he's making a
case why you have to pay. To prove you're human, which is one of the only
mechanisms to prove, which is depressing. – And yeah, I feel we have to remember, as individuals, we
should from time to time, ask ourselves why are we
doing what we're doing, right? And as a species, we need to do that too. So if we're building, as you say, machines that are outnumbering us, and more and more outsmarting us, and replacing us on the job market, not just for the dangerous
and and boring tasks, but also for writing poems and doing art, and things that a lot of
people find really meaningful, we gotta ask ourselves, why? Why are we doing this? The answer is Moloch is
tricking us into doing it.

And it's such a clever trick that even though we see the trick, we still have no choice
but to fall for it, right? And also, thing you said about you using co-pilot AI tools to program faster, how many, what factor faster would
you say you code now? Does it go twice as fast? Or, – I don't really, because it's such a new tool. – [Max] Yeah. – I don't know if speed
is significantly improved, but it feels like I'm a year away from being 5 to 10 times faster.

– So if that's typical for programmers, then you're already seeing another kind of recursive self-improvement, right? Because previously, like a major generation of
improvement of the codes would happen on the
human R and D time scale. And now if that's five times shorter, then it's gonna take five times less time than it otherwise would to develop the next level of these tools, and so on. So this is exactly the sort of beginning of an intelligence explosion. There can be humans in the
loop a lot in the early stages, and then eventually humans
are needed less and less and the machines can
more kind of go alone.

But what you said there
is just an exact example of these sort of things. Another thing which, I was kind of lying on my
psychiatrist imagining, I'm on a psychiatrist couch here saying, "Well what are my fears
that people would do with AI systems?" So I mentioned three
that I had fears about many years ago, that they would do, namely teach it to code, connect it to the internet, and teach it to manipulate humans. A fourth one is building an API, (Lex chuckles) where code can control this
super powerful thing, right? That's very unfortunate because one thing that systems like GPT-4
have going for them is that they are an oracle in the sense that they just answer questions. There's no robot connected to GPT-4. GPT-4 can't go and do stock trading based on its thinking.

It is not an agent, and an intelligent agent is something that takes in information from the world, processes it, to figure out what action to take based on its goals that it has, and then does something back on the world. But once you have an API for, for example, GPT-4, nothing stops Joe Schmoe and a lot of other people
from building real agents, which just keep making calls somewhere in some inner loop somewhere to these powerful oracle systems, which makes themselves much more powerful. That's another kind of
unfortunate development, which I think we would've
been better off delaying. I don't wanna pick on
any particular companies, I think they're all under a
lot of pressure to make money. – [Lex] Yeah. – And again, the reason we're
we're calling for this pause is to give them all cover to do what they know is the right thing, just slow down a little bit at this point.

But everything we've talked about, I hope we'll make it clear
to people watching this, you know, why these sort
of human-level tools can cause a gradual acceleration. You keep using yesterday's technology to build tomorrow's technology. And when you do that over and over again, you naturally get an explosion. You know, that's the definition of an explosion in science, right? If you have two people, and they fall in love, now you have four people, and then they can make more babies, and now you have eight people, and then you have 16, 32, 64, et cetera. We call that a population explosion where it's just that each, if it's instead free neutrons
in a nuclear reaction that if each one can make more than one, then you get an
exponential growth in that, we call it a nuclear explosion. All explosions are like that, and an intelligence explosion, it's just exactly the same principle, that some amount of intelligence can make more intelligence than that, and then repeat. You always get exponentials. – What's your intuition why it does, you mentioned there's
some technical reasons why it doesn't stop at a certain point.

What's your intuition? And do you have any
intuition why it might stop? – It's obviously gonna stop when it bumps up against
the laws of physics. There are some things you just can't do no matter how smart you are, right? – Allegedly. 'Cause we don't know all the full laws of physics yet, right? – Seth Lloyd wrote a really cool paper on the physical limits on
computation, for example. If you make it, put too much energy into it and the finite space will
turn into a black hole, you can't move information around faster than the speed of
light, stuff like that. But it's hard to store way more than a modest number
of bits per atom, et cetera.

But, you know, those limits are just astronomically above, like 30 orders of magnitude
above where we are now. So, you know. Bigger difference, bigger
jump in intelligence than if you go from ant to a human. I think, of course what we want to do is have a controlled thing, in a nuclear reactor you put moderators in to make sure exactly it doesn't blow up out of control, right? When we do, experiments with biology
and cells and so on, you know, we also try to make sure it doesn't get out of control.

We can do this with AI too. The thing is, we haven't succeeded yet. And Moloch is exactly doing the opposite. Just fueling, just egging everybody on, "Faster, faster, faster, or the other company is
gonna catch up with you, or the other country is
gonna catch up with you." We have to want to stop, and I don't believe in just asking people to look into their hearts
and do the right thing. It's easier for others to say that, but like, if you are in this situation where your company is gonna get screwed by other companies that are not stopping, you're putting people in
a very hard situation, the right thing to do is change the whole
incentive structure instead.

And this is not an old, maybe I should say one
more thing about this, 'cause Moloch has been around as humanity's number
one or number two enemy since the beginning of civilization. And we came up with some
really cool countermeasures. Like first of all, already over 100,000 years ago, evolution realized that
it was very unhelpful that people kept killing
each other all the time. So it genetically gave us compassion and made it so that, like if you get two drunk dudes getting into a pointless bar fight, they might give each other black eyes, but they have a lot of inhibition towards just killing each other.

That's a, And similarly, if you find
a baby lying on the street, when you go out for your
morning jog tomorrow, you're gonna stop and pick it up, right? Even though it maybe make you
late for your next podcast. So evolution gave us these genes that make our own egoistic incentives more aligned with what's good for the greater group
we're part of, right? And then as we got a
bit more sophisticated and developed language, we invented gossip, which is also a fantastic
anti-Moloch, right? 'Cause now, it really discourages
liars, moochers, cheaters, because their own incentive
now is not to do this because word quickly gets around and then suddenly people
aren't gonna invite them to their dinners anymore or trust them. And then when we got
still more sophisticated in bigger societies, you know, we invented the legal system where even strangers who
couldn't rely on gossip and things like this
would treat each other, would have an incentive.

Now those guys in the bar fights, even if someone is so drunk that he actually wants
to kill the other guy, he also has a little thought
in the back of his head that, you know, "Do I really wanna
spend the next 10 years eating like really crappy
food in a small room? I'm just gonna chill out," you know? And we similarly have tried to give these incentives to our corporations by having regulation and
all sorts of oversight so that their incentives are
aligned with the greater good. We tried really hard, and the big problem
that we're failing now, is not that we haven't tried before, but it's just that the tech is growing, is developing much faster than the regulators been
able to keep up, right? So regulators, it's kind of comical that
the European Union right now is doing this AI act, right? And in the beginning they had
a little opt-out exception that GPT-4 would be completely
excluded from regulation.

Brilliant idea. – What's the logic behind that? – Some lobbyists pushed
successfully for this? So we were actually quite involved with the Future of Life Institute, Mark Brakel, Risto Uuk, Anthony Aguirre, and others, you know, we're quite
involved with talking to, educating various people
involved in this process about these general-purpose
AI models coming, and pointing out that they
would become the laughing stock if they didn't put it in. So the French started pushing for it, it got put in to the draft, and it looked like all was good, and then there was a huge
counter push from lobbyists. Yeah, there were more
lobbyists in Brussels from tech companies than from
oil companies, for example. And it looked like it might, this was gonna maybe get taken out again.

And now GPT-4 happened, and I think it's gonna stay in. But this just shows, you know, Moloch can be defeated. But the challenge we're
facing is that the tech is generally much faster than
what the policymakers are, and a lot of the policymakers also don't have a tech background, so it's, you know, we really need to work
hard to educate them on what's taking place here. So we're getting this situation where the first kind of, so I define artificial intelligence just as non-biological
intelligence, right? And by that definition, a company, a corporation is
also an artificial intelligence because the corporation isn't its humans, it's a system. If its CEO decides, if a CEO of a tobacco
company decides one morning that she or he doesn't wanna
sell cigarettes anymore, they'll just put another CEO in there. It's not enough to align the incentives of individual people or align individual computers'
incentives to their owners, which is what technically,
AI safety research is about.

You also have to align the
incentives of corporations with the greater good. And some corporations have
gotten so big and so powerful very quickly that in many cases, their lobbyists instead
align the regulators to what they want rather
than the other way round. It's a classic regulatory capture. – Right, is the thing that
the slowdown hopes to achieve is give enough time to
regulators to catch up, or enough time to the companies themselves to breathe and understand how to do AI safety correctly? – I think both, but I think that the vision, the path to success I see is first you give a breather actually to the people in these companies, their leadership who wants
to do the right thing, and they all have safety teams and so on, on their companies, give them a chance to get
together with the other companies, and the outside pressure can
also help catalyze that, right? And work out what is it that's, what are the reasonable
safety requirements one should put on future systems
before they get rolled out.

There are a lot of people also in academia and elsewhere outside of these companies who can be brought into this and have a lot of very good ideas. And then I think it's very
realistic that within six months, you can get these people coming up, so here's a white paper, here's what we all think it's reasonable. You know, you didn't, just because cars killed a lot of people, you didn't ban cars, but they got together a bunch of people and decided, you know, in order to be allowed to sell a car, it has to have a seatbelt in it. They're the analogous things that you can start requiring
a future AI systems so that they are safe. And once this heavy lifting, this intellectual work has been done by experts in the field,
which can be done quickly, I think it's going to be quite easy to get policymakers to see, yeah, this is a good idea.

And it's, you know, for the companies to fight Moloch, they want, and I believe Sam Altman has explicitly called for this, they want the regulators
to actually adopt it so that their competition is gonna abide by it too, right? You don't want, you don't want to be
enacting all these principles and then you abide by them, and then there's this one little company that doesn't sign onto it and then now they can
gradually overtake you. Then the companies will get, be able to sleep secure knowing that everybody's playing
by the same rules. – So do you think it's
possible to develop guardrails that keep the systems from basically damaging
irreparably humanity, while still enabling sort
of the capitalist-fueled competition between companies as they develop how to best
make money with this AI? You think there's a
balancing that's possible? – Absolutely, I mean, we've seen that in many other sectors where you've had the free market produce quite good things without causing particular harm. When the guardrails are there
and they work, you know, capitalism is a very
good way of optimizing for just getting the same
things done more efficiently.

But it was good, you know, and like in hindsight,
and I never met anyone, even on parties way over on the right, in any country who think it was a bad, thinks it was a terrible idea to ban child labor, for example. – Yeah, but it seems like
this particular technology has gotten so good so fast, become powerful to a
degree where you could see in the near term, the ability to make a lot of money.

– [Max] Yeah. – And to put guardrails, to develop guardrails quickly
in that kind of context seems to be tricky. It's not similar to cars or child labor, it seems like the opportunity
to make a lot of money here very quickly is right here before us. – So again, there's this cliff. – Yeah, it gets quite scenic, (laughs) – [Max] The closer to the cliff you go, – Yeah. – The more money there is, the more gold ingots
there are on the ground you can pick up or whatever, if you want to drive there very fast, but it's not in anyone's incentive that we go over the cliff and it's not like
everybody's in the wrong car. All the cars are connected
together with a chain. So if anyone goes over, they'll start dragging
the others down too. And so ultimately it's in the selfish interests also of the
people in the companies to slow down when you just start seeing the contours of the cliff
there in front of you, right? And the problem is that, even though the people who
are building the technology, and the CEOs, they really get it, the shareholders and
these other market forces, they are people who don't honestly, understand that the cliff is there, they usually don't.

Leave a Reply

Your email address will not be published. Required fields are marked *