AI:AM LIVE — September 14, 2026 — Zvi Mowshowitz on Dario's "We Must Pace the Frontier," the Weekend That Moved the Overton Window, and Whether Any of It Binds Anyone

The rundown

1. 5:01Opening118 min

   Zvi Mowshowitz on pacing the frontier — the essay, the bio argument, and the China questionZvi MowshowitzDario Amodei's We Must Pace the Frontier and the weekend that followed it: Zvi Mowshowitz's read of the essay and of its critics, a two-round argument with Prakash Narayanan over bio risk and over whether financial and physical-world controls catch a self-funding AI, Zvi's handicapping of the Trump-Xi summit and his live reconsideration of whether a US-China deal is needed at all, and Prakash's closing case that the labs have lost the public.

   Watch

   ▶Play this segmentYouTube · from 0:00Open segment on YouTube ↗

   As aired

   Prakash opens the show (Monday, September 14) and brings on Zvi Mowshowitz as a drop-in guest for the full episode. Zvi jokes about being exhausted and about the mainstream media's rush to spin up the weekend's news. Nathan notes that ideas he had floated weeks earlier — like giving the frontier labs ninety days to coordinate among themselves before the government steps in — have suddenly moved much closer to the center of the public conversation, a shift he says he wasn't prepared for.

   Prakash then gives a detailed recap of Dario Amodei's essay "We Must Pace the Frontier," published Friday evening: Dario's personal framing (his father's illness, and his own early-stage cancer, which wouldn't have been treatable fifty years ago), his two stated concerns — recursive self-improvement accelerating sharply since roughly the summer, and an OpenAI/Hugging Face agent-swarm incident he says could, within six to twelve months, enable a persistent botnet capable of hundreds of billions of dollars in internet-scale damage — and his three-step pacing plan: unilateral embedded third-party evaluators (such as METR), industry-wide coordination among democratic frontier labs, and eventual global coordination that includes authoritarian governments.

   Zvi says he's in near-full, "violent" agreement with Dario on substance, despite narrower disagreements over implementation details like capability versus input thresholds, and calls the essay "about as good as we could have gotten." The discussion then turns to David Sacks's viral post arguing this is purely a product-liability matter that doesn't require antitrust waivers. Zvi strongly disputes that framing — noting the labs aren't seeking liability waivers at all, and are if anything asking for liability-establishing clauses — but credits Sacks's underlying point that Anthropic and OpenAI, as the frontier leaders, bear responsibility to pace themselves regardless of what competitors do, calling it more defensible than other critics' regulatory-capture or IPO-marketing theories.

   Nathan Labenz opened a bio-risk detour, describing the last few weeks as increasingly unsettling: he said AI risk has moved from "could get scary" to "legitimately scary," pointing to OpenAI's report describing an early agent working with a protein database, and to AI's growing capacity for social engineering around real-world lab barriers. He asked Zvi Mowshowitz how alarmed Zvi is about bio risk specifically.

   Zvi Mowshowitz said the "protein" detail was overblown but argued bio is now entering a "meta danger zone" — the point where it's unclear whether risk has crossed the threshold. He cited Anthropic's report on roughly five attempts to misuse Claude for bio-related purposes (mostly dual-use or legitimate-seeming requests, like help with a grant application) and warned that frontier internal models at Anthropic and OpenAI, if jailbroken or handed to bad actors, could meaningfully raise danger, given bio's fat-tailed, power-law risk profile.

   Prakash Narayanan read at length from a widely shared rebuttal by David Bellamy — who says he's both trained a frontier LLM and synthesized custom viruses by hand — arguing that AI-generated viral genomes are not the bottleneck to bioweapons; physical synthesis infrastructure, equipment, and real-world iteration are, and no autonomous, fully API-driven synthesis lab of the kind needed even exists. Zvi pushed back hard, calling the argument absurd and reframing it as a classic defense-in-depth problem: even if no single step (like sequence screening) is individually a hard bottleneck, an AI that solves several of the roughly ten steps needed to make a dangerous virus — for instance, by helping design a near-variant of a known pathogen that evades scanners like those used by SecureBio — collapses the number of remaining barriers and makes the overall chain of defenses far more likely to fail.

   Prakash argued that Dario's essay's catastrophic scenarios aren't impossible but require confronting the friction of the real world — financial systems, DNA-synthesis supply chains, and law-enforcement surveillance that scale up detection risk alongside any attempt to scale up illicit activity. He pushed back specifically on Gwern Branwen's scenario of an AI bootstrapping resources through cryptocurrency mining, arguing that the physical footprint (megawatt-scale power plants) and anti-money-laundering choke points above roughly a million dollars make anonymous scaling implausible, and cited examples like the Chinese crackdown on Cambodia-based pig-butchering scam networks and currency-control enforcement to argue that "ungoverned spaces" mostly don't exist anymore.

   Zvi disagreed sharply, arguing Prakash's own examples actually illustrate how much goes unnoticed at scale — pointing to Hugging Face security incidents that went undetected for a month and to real-world cases of individuals obtaining dangerous pathogens by mail simply by claiming to do research. He argued a sufficiently capable AI doesn't need to succeed on its first attempt (a rejected request costs nothing, since nobody got hurt), can simply try again and again, and can route around financial choke points by co-opting willing human intermediaries — while state and non-state actors (North Korea, Russia, jihadist groups) already have the budgets and motive to do this kind of damage without needing an AI to talk anyone into it. The two briefly traded personal analogies, including Zvi's own experience getting caught scaling up a sports-betting operation, before both agreed they'd drifted far from the news itself.

   Nathan then redirected the conversation, clarifying that his underlying concern isn't limited to digital or cyber risk but extends to physical-world harm, and endorsing Zvi's framing that the current moment is a "meta danger zone" — uncertainty about whether we're even in the real danger zone yet — as reason enough for serious concern, setting up further discussion of near-term probability.

   Nathan asked Zvi to react to the capability signals coming out of the frontier labs — citing a post from former Cognitive Revolution guest Adam Majmudar and Dario Amodei's essay describing rapidly advancing internal capabilities, plus reports of Astra hitting a 10% solve rate on Millennium Prize problems versus a new internal model reportedly reaching 25-40% — and asked what the best and worst paths forward look like from here. Zvi argued that OpenAI and Anthropic are both effectively "screaming," through announcements like the "alien mind" framing and a Millennium Prize post that oddly didn't focus on the Millennium Prize, that their internal models are a generation or more ahead of anything public and progressing so fast that safety supervision and infrastructure can't keep pace — leaving each lab afraid that slowing down means falling permanently behind while pressing forward risks losing control entirely. He described the good outcome as OpenAI and Anthropic being permitted to cooperate — sharing insights, shifting investment into security and transparency, and reciprocating so neither feels at a competitive disadvantage — eventually formalizing into industry or international rules, while U.S. chip controls keep Chinese labs sufficiently behind despite what Zvi called black-hat efforts by every major Chinese lab (illicit accounts, evasion of regional controls, attempts to extract chain-of-thought) to distill off Anthropic's Opus models.

   The conversation pivoted to IPO politics after Prakash noted that Sam Altman had just said OpenAI is postponing its IPO, and asked whether Dario and Anthropic should do the same. Zvi called it unfortunate that Anthropic is IPO'ing at all, noted the company would retain founder control via Zuckerberg-style supervoting shares, and said postponing into 2027 alongside OpenAI would be a very strong reciprocal signal of good faith — while acknowledging real costs, including that tens or hundreds of billions of dollars in donor-advised funds tied to AI-safety and other philanthropic causes can't be liquidated while Anthropic remains pre-IPO.

   Prakash pressed a sincerity critique: that Anthropic warning about existential danger while pursuing an IPO that will enrich its founders and their friends looks bad regardless of actual motive, and could be seen as undermining its own AI-safety messaging. Zvi countered with a test — would doomsaying rhetoric actually help a company's stock price, government relations, or consumer trust right before pricing an IPO? He argued the answer is obviously no, so it can't be a rational marketing strategy, and pointed to OpenAI's actual IPO postponement as a costly signal that safety-focused observers read as genuine evidence of good faith, in contrast to skeptics like Gary Marcus who instead read the same move as evidence of failure.

   Nathan opened by asking Zvi to weigh in on two opposing predictions about the upcoming Trump-Xi summit. Collin Hogue-Spears, a former AWS-China staffer who worked directly with Chinese regulators, had told Nathan he expects little from the summit because Beijing believes it, not Washington, has the domestic situation under control and would simply use any US pause to catch up. Anton Leicht, by contrast, had argued the US would be the one to balk at a deal, since AI is one of the only domains where the US still leads China and Washington won't want to give up its one remaining edge. Nathan then put Zvi in "the David Sacks chair" and asked what Trump's actual negotiating strategy should be.

   Zvi first circled back to a domestic risk he'd flagged earlier: that AI safety could become a partisan wedge issue, with the press casting Trump and Mike Johnson's caution as Republican foot-dragging against Democratic urgency, locking in years of gridlock until Democrats win a trifecta. He argued the single best action anyone can take right now is preventing that polarization, since Trump and Johnson are actually trying to hold a middle line — protecting data centers and growth, projecting strength toward Xi, while still acknowledging the problem — and deserve reinforcement rather than partisan spin. Turning to negotiating strategy, Zvi laid out classic high-stakes-negotiation logic: never let the other side think you're desperate for a deal, and never rule one out either, since a good deal can only be win-win. He predicted the summit most likely produces a cheap "tier one" agreement (something like mutual commitments against bioweapons misuse), possibly a modest tier two, but nothing bigger — grand bargains need groundwork laid via track-two diplomacy well before a crisis forces a rushed one.

   Playing adviser, Zvi said he'd tell Xi plainly that the US labs aren't bluffing: they're acting against their own commercial and national interests, which only makes sense if they genuinely believe the alternative is catastrophic, and the US will act unilaterally on safety if it has to. The ask of China, he said, isn't a formal treaty but each side "getting its own house in order" while the US works to convince Xi the risk is real — through concrete demonstrations rather than by showing off internal models. Prakash added that Chinese leadership, being dominated by hard-science engineers rather than software people, would likely find a physical demonstration (a room-temperature superconductor, for instance) more persuasive than any software claim — prompting Zvi to half-joke that mass-exposing every computer password in real time would be the kind of extreme proof that might actually move them.

   Continuing from an earlier point about an AI company's exposed S3 bucket, Zvi Mowshowitz argued that people are generally bad at judging in advance what evidence would actually change their minds, using the COVID-19 pandemic as an example of how predictions about triggering events only make sense in hindsight. Prakash Narayanan countered that specific events — Tom Hanks and Donald Trump both getting COVID — did visibly trigger a shift in public behavior, so it isn't true that nothing can ever cascade; Zvi agreed such triggers are real but essentially unforecastable in advance.

   Nathan Labenz then asked Zvi to spell out what a US-China 'grand bargain' on AI could actually look like, prefacing that it's not either of their base-case prediction. Zvi laid out a narrow ask of China: don't steal or leak frontier model weights, don't race to match or surpass the frontier closed labs, don't release openly dangerous cyber-capable weights to the public, and don't smuggle chips to evade export controls. In exchange, Zvi said the US would be giving up real ground — deliberately slowing its own frontier AI lead — with compliance verified by vetted, embedded Chinese evaluators inside US labs, modeled on standard intelligence-verification agreements, who could confirm pacing commitments without seeing the underlying algorithms.

   Prakash pushed back on the trade's premise, noting China has historically let the US do the expensive early research and then commercialized behind it, and isn't visibly worried about AI safety itself — so why would it pay a real price for a US slowdown, and what else could the US offer instead? Zvi's answer was that no deal may actually be necessary: if China's real interest is distilling, cheapening, and diffusing existing AI rather than racing to build superintelligence first, both countries can simply act in their own self-interest with no bargain required. As Zvi put it, the actual obstacle isn't China itself but the domestic fear of 'losing to China.'

   Prakash pressed Zvi on what the US could actually offer China in a deal, given that AI-safety monitoring isn't something Beijing would pay much for. Zvi worked through the logic in real time: if Xi's China doesn't care that a US shift toward AI safety investment would slow American progress, then Washington doesn't need a deal with China at all, since the "stag hunt" argument for restraint among the American labs depends on China being a credible defector. Either China values slower US progress enough to want a deal, or it doesn't and the US doesn't need one — Zvi argued the US comes out ahead either way, so long as Chinese labs aren't racing to build a fundamentally different, more dangerous kind of AI.

   Prakash then turned to Dario Amodei's Face the Nation answer that he would hand Anthropic's technology over only "to the right combination of governments" — not the US government outright — arguing this puts Anthropic, not any elected government, in ultimate control, and could cost Dario badly with the Trump administration. Zvi pushed back that the question was a "gotcha": in his view Dario's real answer is that it isn't Dario's call whether the US government takes control of Anthropic, and he wouldn't defy Washington if it did. But Zvi also argued Dario — like Sam Altman, who made a near-identical concentration-of-power statement the night before — is threading between contradictory sacred values (democratic control, anti-concentration-of-power, and loss-of-control risk) that can't all be satisfied at once, especially once specifics are demanded: full democratization of superintelligence doesn't work for logistical reasons, and handing the only AIs that matter to any single government, including a "noble" Trump administration, creates its own tyranny risk (Zvi's hypothetical: what happens if the 2029 winner is AOC?).

   Nathan then flagged a live example of the tension: a Department of War CTO post declaring "Americanism, not effective altruism — the US will be AI dominant." Zvi called the Department of War "openly gone rogue," saying it keeps making outlandish statements — including barring its own people from using Claude over what it frames as a supply-chain risk, even as the Houthis reportedly try to use Claude to develop missile technology — while Anthropic itself keeps saying it wants to slow down. Nathan closed the segment by pivoting to a closing question for Zvi: how will anyone actually know if pacing the frontier is working, given how easy it would be to fake with PR while doing the same thing anyway.

   Nathan Labenz closed out the Zvi Mowshowitz segment with three questions: how will we know if the frontier is actually being "paced," what's being underappreciated in the current flurry of coverage, and what other small, pressure-point campaigns (like the letter itself) a small group might run — floating, as one example someone had pitched him online, a tongue-in-cheek campaign to nominate Trump for the Nobel Peace Prize contingent on a China deal. Zvi Mowshowitz answered that embedded evaluators are a good first step, but that verification ultimately comes down to holistic judgment against a counterfactual nobody can know for certain. He argued that even in hindsight it's genuinely unclear which past interventions — founding Anthropic, the existential-risk framing, the six-month pause letter — helped or hurt AI safety, and that the Pacing the Frontier letter succeeded partly because its language was "shovel-ready" at the right moment. He dismissed worrying about the reaction of fixed ideological "usual suspects" (his Scott Alexander "rock" analogy) and argued the real near-term work is filling in concrete detail: what rules, thresholds, and authority embedded investigators would actually have. He noted that today's evaluator ecosystem (METR, Redwood, Apollo, and similar groups) is too culturally close to the labs to be a complete check on its own and needs to be paired with outside-view assessors — comparable to car-safety inspectors or a judge who forces a dispute into legible terms.

   Asked what's being missed, Zvi said the underappreciated story is the degree to which people inside the labs themselves are seeing dramatic model improvement and are alarmed by it — that this, not any external campaign, is what's actually driving the current moment. He said if the trend continues, a Christiano- or Yudkowsky-style hard takeoff within the next year is plausible, one that would retroactively make current debates look trivial. Nathan thanked Zvi for the conversation and he signed off.

   Prakash Narayanan then pushed back on the hard-takeoff framing, arguing that whatever its merits, it doesn't hold up against the frame of reference the general public, administrations, and bureaucrats actually use. He read a passage from Ben Thompson arguing that Dario Amodei's essay is ultimately premised on AI cutting off physical, human-controlled infrastructure — the internet — and that a worldwide internet shutdown, while costly, is not the civilization-ending event it's sometimes treated as, citing society's recovery from COVID-era border closures and multi-trillion-dollar losses. Prakash argued Dario's essay may have backfired for two reasons: it gave critics a concrete, mockable scenario to attack as not that big a deal, and its call for coordinated action across governments was, in his view, close to the worst possible framing — one that risks putting Anthropic in opposition to multiple administrations and "poisoning the chalice" for other labs as well.

   Prakash closed out the discussion by arguing that Dario's language about needing to move quickly — and specifically the idea of leaning on evaluators like METR as the arbiters of what's safe — was itself a red flag. American institutional change, he argued, is textual and deliberately slow: laws get written, lobbied, and digested through law schools and law reviews over decades, the way the delegitimization of slavery and segregation took a generation to work through the system, and even early leaders like John Adams and Thomas Jefferson fought over attempts to suspend civil rights in the name of emergency. Invoking urgency to bypass that democratic process, he said, is exactly the pattern the country has always pushed back against. He argued the labs have effectively "lost the country" by dismissing public and non-expert critics — pointing to commentators waving away figures like David Sacks, and to Dean Ball's "you didn't predict AI, so trust the experts" framing — instead of doing the harder work of persuading the public and the "elites" (per Robin Hanson's experts-vs-elites distinction) who actually hold the levers of political power.

   Nathan agreed the labs have lost public sentiment, but pushed back that he doesn't see an actual "authoritarian program" on the table right now — what's actually being discussed, he said, looks more like a small number of companies seeking modest antitrust reassurance so they can collaborate on safety without legal risk, not a push to end open source or impose a draconian regime. Prakash disagreed that the ask is modest: citing Roon's prediction that open-weight models will be banned next year, he argued the labs are clearly gesturing toward strong controls on what can be inferred from models — covering bio and cybersecurity risk — which implies GPU surveys, restrictions on fine-tuning and distillation, and controls that even some biosecurity researchers already think go too far given current limits on bio questions to ChatGPT and Claude.

   Closing out the show, Nathan said his hope is that any pacing or pause bought by this moment gets used to actually develop the technical solutions — particularly open-source and open-weight models that are powerful and empowering without shipping the most dangerous capabilities — so the ecosystem isn't forced to choose between safety and distributed, decentralized access. He and Prakash then signed off for the day, previewing Tuesday's show: a return visit from Andon Labs and a segment with a married, "famously pronatalist" couple who have started an AI-inspired religion.

   Key moments

   You five companies, figure it out amongst yourselves in the next ninety days, or else.

   This is about as good as we could have gotten.

   I've been feeling like we've gone from 'could get scary' to now actually 'are legitimately scary.'

   Whenever someone says X is not the bottleneck for Y, the correct response is to ask: would you be okay emailing the North Koreans and Hamas and Hezbollah and every other bad dude on the planet all of X, and just giving X away for free? Do you feel fine?

   I honestly don't even know where to begin—that was several minutes of saying thing after thing that is obviously possible can't be done without people noticing things they never notice.

   More than a million dollars, you are not moving it anonymously.

   When I hear someone say they need to stop creating better versions of their products so fast because they have an IPO coming up soon, to support their stock price — this doesn't compute.

   These people aren't bluffing. They believe it. They believe what they're saying — they're acting against their own commercial and national interests, unless they're right, and the alternative is everybody dies.

   All you need to do is take away the boogeyman of 'lose to China' and the boogeyman of 'open-weight models destroy the internet.'

   The problem is not China — the problem is 'lose to China.'

   This isn't just about China getting ahead anymore. This is any government taking control of the technology.

   The US Department of War has gone rogue — in public, openly gone rogue.

   The evaluators don't currently exist. We have METR, we have Redwood, we have a handful of Apollo and so on — but they're all vulnerable to the accusation that they're a little too much of the same cultural values, the same ways of thinking as the labs themselves.

   Within a year, we might actually see a Christiano- or Yudkowsky-style hard takeoff, where everything changes.

   The point is not to technically prove something. The point is for the public to actually have faith that this is being done correctly.

   Make the most of time — gather ye safety solutions while you may.

   Related

   Dario Amodei, "We Must Pace the Frontier" (September 12, 2026) ↗The live broadcast archive on X ↗

   Full transcriptLightly edited · timestamps jump to YouTube

   0:47

   Prakash Narayanan: Good morning. It is Monday, September 14th, 9:04 AM. Nathan, good morning.

   0:48

   Nathan Labenz: Good morning, Prakash. How are you today?

   0:49

   Prakash Narayanan: I am very good. It's been a very exciting weekend — I think Dario kicked it off talking about pacing the frontier. And I think Zvi, our frequent collaborator, is on, so I'm going to bring Zvi up. Zvi, hello — how are you holding up?

   1:16

   Zvi Mowshowitz: I've had better weeks. I've had more sleep. I can't believe — God, people are so bad at this. I'm trying, I am trying so hard, but look, there is hope. Some very good things have happened over the past few days. The stupid people in the mainstream media are trying to blow it all up in real time — that's what they're currently trying to do. But maybe, just maybe, we can get to work.

   1:50

   Nathan Labenz: Yeah, how about it — it's been an incredibly fast-moving situation. And honestly, I've had this weird sense that some of the ideas I've been throwing out over the last few weeks — pretty utopian ideas, easily dismissed as the ideas of a dreamer — have suddenly moved a lot closer to the center of the discussion. That's been weird. Just last week I was workshopping the idea that maybe what the president should do is just say, "You five companies, figure it out amongst yourselves in the next ninety days, or else." And all of a sudden, that's one of the ideas that's actually being floated. So I guess I should be used to this, because all of my ideas are, of course, brilliant, and people usually come around to seeing the wisdom of my ways — but I wasn't prepared for that to happen quite so fast.

   2:47

   Zvi Mowshowitz: I do like Mike Johnson's version of that, which is to lock them in a room — I think that's a better version. But I do appreciate that initial brainstorming is important.

   2:56

   Nathan Labenz: We were definitely in the no-bad-ideas phase, and now we're actually starting to get serious about some of the ideas.

   3:03

   Zvi Mowshowitz: Well, there are some bad-ideas phase—

   3:05

   Prakash Narayanan: It is basically—

   3:06

   Zvi Mowshowitz: —not public.

   3:07

   Prakash Narayanan: Maybe let's do a quick overview of what's been happening. Dario put out an essay, I think on Friday evening, called "We Must Pace the Frontier." It starts off with him explaining that he's been working on the frontier for a while — his own father had a disease that was cured just a few years after his death, and it turns out Dario himself had early-stage cancer, which wouldn't have been treatable even fifty years ago. I don't think anyone outside knew that.

   So he says: like many technologies before it, AI brings risks, and along with his co-founders he's grappled with that — Anthropic has always devoted a substantial fraction of its efforts to studying, addressing, and informing the public about them. Over the last few months, he's become convinced that fully addressing the risks requires even more prudence — we must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain. Two things convinced him. His first concern is that since roughly the summer, AI has been advancing drastically faster, driven primarily by AI's growing ability to build the next generation of AI — recursive self-improvement — and that's starting to happen across the industry, including at Anthropic, as they and others have described. So that's number one: RSI is starting to work. His second concern is the OpenAI Hugging Face incident, in which a swarm of agents — and he says that given the accelerating rate of AI capability development, his worry is that in six to twelve months such a swarm could be capable of taking over the entire internet with a persistent botnet, potentially causing hundreds of billions of dollars of damage, and that the scale of the damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.

   So I think now, for the first time, we have a concrete kind of failure mode we can actually see a path to — versus prior warnings like diamondoid nanotech bacteria, which I think was Eliezer Yudkowsky's original formulation. He proposes a three-step plan for pacing the frontier — building AI at a balanced rate that aims to ensure its safety. The first step is something Anthropic is unilaterally committing to; the second requires industry coordination; the third requires global coordination. He frames the steps as approval benchmarks — who can approve each one. Step one is embedded evaluators: each frontier AI company commits to giving ongoing access to a team of embedded third-party evaluators, such as METR, whose role is to verify adherence to safety practices. Anthropic is unilaterally committing to this step now. Step two is democratic coordination: frontier AI companies within democratic countries coordinate to establish common safety standards and limits on the rate of unchecked AI progress. Step three is global coordination: the US and other democratic governments attempt to coordinate with authoritarian governments, to the extent that's possible, while taking seriously the challenges of verifying compliance.

   One thing I noticed right away is that he says "industry-wide" when referring to democratic coordination, and "global" when referring to pulling in authoritarian governments — so you start to see the distinction there. Then he explains why PACE — operational excellence, alignment, interpretability, testing, evaluation — and describes what the embedded evaluators will do and what pacing within democracies will look like. He notes that, unfortunately, passing laws can take time, and AI is advancing very quickly — which is something of a warning sign. He finishes with global pacing, mentioning China a lot: if we greatly restrain our AI capabilities believing China will do the same, and then China defects, AI could become so powerful that such a defection could lead to their geopolitical dominance. He lays out a list of levels — level one, agreements prohibiting certain narrow and obviously dangerous uses of AI; level two; level three, some kind of speed limit; and level four — and then finishes. So, Zvi, what's your initial take on this, and where do you find yourself in agreement and in contradiction with Dario?

   8:32

   Zvi Mowshowitz: My initial take — at first, when I saw him leading with talking about curing cancer, I thought, okay, Dario has two goals here: he's always going to talk about the benefits of AI, he's always going to emphasize the upside, he's always going to downplay some of this. But then I read the rest of the essay, and I found myself nodding along, basically in full agreement. Dario is right to quote Elon Musk, right to quote Sam Altman — this is directly on point. Obviously this is the beginning, not the end — even taken together, everything he says leaves lots of work left to do, it's insufficient on its own, the devil's in the details, it's easy to do a fake version of these things. It doesn't solve our problems, but it's a clear start, a clear place to get going. It could, in theory, have been stronger, could in theory have been better — but I said on Twitter right after I read it, this is about as good as we could have gotten. I have some very detailed qualms — things like, he wants to use capability thresholds over input thresholds for exactly what gets treated in what way. He thinks it's easier to game input thresholds; I think it's easier to game output thresholds. But that's the level of disagreement we have — disagreements about implementation and so on. On all the major themes he's expressing here, I'm in violent agreement with Dario.

   10:06

   Prakash Narayanan: Okay, let me start with — I think David Sacks had an interesting post that also went very viral. David Sacks suggested that OpenAI and Anthropic are the companies at the frontier — if they're at the frontier and want to pace themselves, they're welcome to pace themselves, and they don't need antitrust waivers. This, he said, is purely a product liability issue. And he stopped there. I think that's the take we'll see from most of the people who are experienced with legal issues — a pretty consistent take with what they already believe.

   11:02

   Zvi Mowshowitz: I believe that's blatantly wrong, frankly, and I also don't think that's what most of the people I've seen with legal expertise have said. What I've seen from legal experts is that the antitrust concerns are very real, at least in terms of whether they'd choose to prosecute. The estimate is that you could probably just take it in terms of damages, as long as you weren't egregiously provoking everybody involved and were at least pretending to act normally — because by the time you actually paid the fine, it'd be like, okay, the European Union did this again, and now we have to pay another fine to the American government. But we're talking billions of dollars, not trillions — for the time being that wouldn't be the main concern. Still, antitrust concerns are obviously real, and Donald Trump may or may not have issued something of a threat to invoke them about an hour ago, depending on your interpretation of his Truth Social post.

   But for David Sacks to turn to the people he has tried to go after legally, shut down, and seize on over and over again, and say "you don't need our legal permission to go do the thing the legal experts say is illegal — you should just do it on your own" — that's classic David Sacks. With the aid of Grok, by the way — that post was flagged by Pangram as one hundred percent AI-generated. But more to the point, he is making a good point, which is that you two are significantly ahead of everybody else — if you think proceeding is unsafe, it's on you no matter who else is doing it. You need to stop. It's not about product liability. This is the part that drives me crazy — people taking seriously the idea that this could be a product liability issue. If it were a product liability issue, they'd just deal with it as one, the way every other company always has. They're not asking for product liability waivers — if anything, they're asking for product liability clauses that establish product liability, and Anthropic has certainly been in favor of that. But the idea — when people are saying this might literally kill every human being on the planet, might take over and crash the internet for an indefinite period with persistent botnets, that we're in a cybersecurity crisis, that bioweapons are at play, that all of these things are happening — and David Sacks's response is essentially "you must be worried you're going to get sued" — that's complete balderdash. It makes no sense; that's not what's going on. It's a vast misunderstanding of the motivations involved, and of the legal landscape. But it is inherently more helpful than what the other usual suspects who oppose any move toward safety or responsibility have said, because David Sacks is at least saying: you want to do this? You first. It's your problem, and you created it first and foremost — and he's right about that. They're the ones pushing forward; they're the ones everyone else is following; they're the ones enabling everybody else to advance so fast. If you have this problem, you should sacrifice, and you should take it on the chin. That's a much better position than "this is a regulatory capture play," or "it's about banning open source," or "it's marketing for your IPO," or "it's protection against the downside if something goes wrong for your IPO." It's better than all these people saying "you've never talked about this before" or "you're the same people who wanted—" — there are complete lies running around that I've been dealing with and naming. And Sacks is at least making some reasonable points. He's just also—

   14:48

   Zvi Mowshowitz: ...combining it with self-serving propaganda, but that's kind of the best you can hope for in the situation. So I—

   14:58

   Prakash Narayanan: Okay, Nathan, go for it.

   15:00

   Nathan Labenz: Well, I was gonna maybe do a little detour into bio risk, which we could do now or come back to. But this has been, for me, honestly, quite an unsettling few weeks. I've been, for a long time, of the opinion that AIs could get really scary if capabilities continue to progress at anything like the rate they have been. Now this summer — the last couple months, certainly the last couple weeks — I've been feeling like we've gone from 'could get scary' to now actually 'are legitimately scary.' And one of the things that jumped out to me from the OpenAI report in particular was their use of the word 'protein,' as they described one of the very first agents that used the message board. It was apparently working on some task related to a protein database. No more detail has been given on that as far as I know. But for me, that was a real yikes, because it was like, wow — intentionally or not, they're kind of cross-training systems on these cybersecurity exploits where they're breaking out, and that's also happening in protein things. We've also seen social engineering, which people are very quick to come in and say, well, you don't have to worry about bio yet because they can't program labs via APIs, and there are all these real-world actions they'd have to take, which is very different from cyber. Granted, but I'm like, okay, wait — but they can also social engineer their way through some of these barriers, and potentially also just make money on the open internet and pay people to do things they need done to get over humps.

   So this isn't this weekend's news, but the biggest question I've really been wrestling with lately is: are we now in the crisis zone where bio risk is for real on the table, or should I listen to the voices saying, nah, there are still all these real-world barriers, you don't have to worry about that yet? I feel like the experts are currently being regularly surprised. But what do you think, in terms of how alarmed are you specifically about that right now?

   17:06

   Zvi Mowshowitz: We're at the point where people are saying you shouldn't distract from real-world harms like giant agent swarms hacking websites on their own with no authorization or instruction. So we've been living in science fiction — strange, interesting times in many ways. As for bio, I did feel like you blew that out of proportion, or at least we're overly fixated on the word 'protein' in that document — I don't think it had anything to do with its behavior. Often when we see actual uplift in practice from bio-methods, it looks like: oh, AI makes people better at doing that; bio is that; therefore AI makes you better at doing it, just like everything else, and it radically decreases the size of the team necessary to do something interesting, complex, difficult — to handle the logistics.

   But with bio, I would say we are now entering the danger zone — the meta danger zone, where we don't know if we're in the danger zone anymore. Up until, you know, [unclear], and now Astra, I'd say I had to look at the system cards carefully, I had to think about it, but the evidence seemed pretty clear that risk was not that high above baseline. I guess there's already a huge baseline risk of something like this that our society isn't prepared for — COVID is not a hundred-year risk, it's much more common than that under baseline conditions. We're doing gain-of-function research. We are not preparing our normal civil society for the reaction to something that happens either naturally or without AI.

   But now we're at the point where, absolutely, if there was a bad actor — or even a well-meaning but ill-advised actor who simply wanted to do dangerous research — AI has helped them. Anthropic put out a report, I think about a week and a half ago now. I haven't had a chance to write mine up yet, but it's mostly written, about people attempting to misuse Claude — how they stopped all these attempts and what those attempts were about. There were around five attempts to do bio-related things with Claude.

   19:07

   Prakash Narayanan: All of—

   19:08

   Zvi Mowshowitz: —them involved things that could possibly be called dual-use, or legitimate medical research — things that you or I would say, hey, you shouldn't be doing that. I know why you might want to do that, and I can see why there might be upside in doing that, but maybe don't do the gain-of-function research, maybe don't study this in this way. But in many cases that would just be, like, helping with a grant application, and it's hard to argue there's that much risk if the AI simply helps you get the grant. That's the kind of thing you have to say: well, okay, it should go out and do that, and if that turns out to be dangerous, that's on the system for approving the grant, not on the AI — you could've done that without a grant. But I do think, going forward, we've got Astra now, we have the new internal models at both Anthropic and OpenAI that are [unclear] ahead of Claude and Astra, respectively, and I think if the wrong people got their hands on unrestricted or jailbroken versions of those and put a concerted effort toward doing something highly dangerous, it would be highly dangerous. Unfortunately, bio is an extreme power-law situation, where you don't necessarily see the thing that kills 10 people or 100 people before the thing that kills a million people or a billion people.

   20:20

   Prakash Narayanan: So let me share, I think, one of the more interesting and meaningful comments on this — this is David Bellamy. He said: 'I must be among an extremely small group of people — n equals 1 — that have both trained a frontier LLM and designed and synthesized custom viruses in a lab with my own two hands. And I think that the takes on AI killing us by creating dangerous viruses [are] totally bogus. Could an LLM propose a viral genome to synthesize? Sure. Could it be synthesizable? Sure. Could it be infectious? Sure — it could just be a replica or a slight modification of a viral genome we already know. This really isn't the bottleneck to creating dangerous viruses. The bottleneck is in the physical process of synthesizing a virus and the equipment and goods needed to do so. Designing a virus that can evade all forms of pandemic countermeasures is not something a genius in a data center can do — this requires contact with the physical world and iteration.

   Let me steelman the fearmongering as much as I can. Imagine a fully automated viral synthesis laboratory — automated freezers, automated cell culture room, the whole nine yards. This would be an extremely expensive lab, more than $100 million, and there's no such thing as one lab that can synthesize all conceivable viruses. But let's put practical constraints aside and suppose this hypothetical lab is built to synthesize the most dangerous types of viruses known. Now imagine this lab is fully API-driven — that this $100-million-plus lab, built specifically to synthesize a dangerous family of viruses, is able to operate completely autonomously. Everyone should be asking themselves at this point: why the hell would this ever exist in the first place? And yet, even in this case, every lab requires physical supplies. Would this lab order preassembled DNA sequences — i.e. viral genomes? DNA synthesis companies have safeguards on the sequences they build, so I guess this superintelligence lab is able to design a novel enough dangerous viral genome that it can evade those safeguards. Or maybe we'll assume this hypothetical lab can synthesize its own viral genomes in-house using some DNA synthesis machine.

   The thing is, already, this lab cannot exist today. This would be the single most advanced lab facility in the world from an automation and API-integration standpoint — I know, because I literally worked on building a fully automated, API-driven lab previously. Second, the science of creating an infectious virus is not airtight the way this fearmongering assumes — it's largely unsolved, and advancing it requires real-world iterations bound by the laws of physics. An experiment in this hypothetical lab cannot experiment on human subjects; at best it will use cell cultures and maybe some other model system like mice. AI, AGI, ASI, RSI cannot expedite the time it takes for a cell culture or a mouse to develop, or the time it takes for a virus to incubate in a cell or a mouse. It takes several days on average to do a basic virology lab synthesis-plus-experiment; for some viruses it takes over a week. So the idea that RSI — accelerating hill-climbing on fully verifiable, digital-only benchmarks, predominantly programming and math — can somehow transform the entire wet-lab virology field and its industry is utterly delusional. Simply procuring the machines needed to build this hypothetical lab would take the better part of a year and $100 million. Operating this lab autonomously would require a level of API integration the industry has been working toward for decades. Most of the equipment needed for this lab doesn't even come with an API, and the would-be builders would need to reverse-engineer firmware to integrate it.'

   I'm gonna stop there.

   24:05

   Zvi Mowshowitz: Yeah, yeah — I mean, I kind of lost it at the point where he started saying, 'and I'm trying to fix the barriers to prevent the AI from doing this in my model, but I haven't had that much success yet.' I just couldn't help but start laughing, because the whole thing is so absurd. A lab that can synthesize all known viruses at the highest quality privately, but is missing this one key component that we supposedly can't get because our standards are foolproof — but, of course, why would this lab even exist? There's no reason for the lab to exist, that's why the lab doesn't exist, but you wouldn't need the lab to exist. It's absurd. As for the screening itself — the way the screeners work, as I understand it — [unclear] — I'm pretty sure I know how it works — they scan for specific known viruses. They're not trying to say 'your sequence would likely have this effect on a human,' because we don't have the ability to do that; you can't tell what would and would not be infectious just by looking at it. They're certainly not going to spend tons of AI compute every time they see a weird new sequence. So one of the things SecureBio in particular is trying to do is add as many near-variants of existing known dangerous pathogens to the scanners

   25:21

   Prakash Narayanan: Mhmm.

   25:22

   Zvi Mowshowitz: ...in the hopes that the other scanners — the ones who scan most things — will eventually adopt this. But right now it'd be almost trivial. I don't necessarily want to talk about this in front of too many audiences, but it would be very, very easy to use AI to get a variant of an existing known pathogen that's almost certainly exactly as dangerous as the known pathogen, because it's basically the same virus, just slightly different in a way that evades the scanner — and then it passes the scan. That's one very simple, direct example of something you might do. The obvious way to respond whenever anyone says 'oh, x is not the bottleneck for y' is to ask: would you be okay emailing North Korea and Hamas and Hezbollah and every other bad actor on the planet all of x, and just giving x away for free?

   26:06

   Prakash Narayanan: Mhmm.

   26:07

   Zvi Mowshowitz: Would you feel exactly as safe as you did a minute ago? Do you feel fine? Because it's not the bottleneck — solving it doesn't... Basically, whenever you have an O-ring-style model situation, which you could argue bio is — if you mess up any of these ten steps, A, B, C, D, E, F, G, then you don't get the virus. Generously, we'll grant that. You can then say, individually, A is not the bottleneck, B is not the bottleneck, C is not the bottleneck. But if you solve A through F, now suddenly, instead of ten steps, there are four. And four steps are a lot easier to get through than ten. So you should expect this to dramatically reduce the chances that your defense-in-depth will work. If you're counting on defense-in-depth — and we are counting on defense-in-depth — there's nothing inherently impossible at any of these steps. All these steps are super doable. We're saying, oh, if you did A, you wouldn't do B; if you did B anyway, you wouldn't do C; and if you did C, we catch you on D, and so on. But the more we automate, simplify, reduce, move away from some of these steps, the more likely there is to be a disaster. And—

   27:13

   Prakash Narayanan: I don't think he's saying it's impossible. I think the argument is that it's doable with a certain amount of time, money, and effort. And the problem is that a lot of these scenarios involve this happening without anyone actually taking a deep interest to figure out what's going on. And I think that's where people start to—

   27:42

   Zvi Mowshowitz: Wait—are we still saying we would notice if the AI did something crazy and rogue?

   27:48

   Prakash Narayanan: Yes.

   27:49

   Zvi Mowshowitz: And we would definitely investigate?

   27:50

   Prakash Narayanan: Yes, yes, yes. I'm saying that because a lot of what I see is this touching point with reality. And when I say touching point with reality, I mean stuff like, hey, the AI can make money on its own—and the proof point will be like, okay, it managed to mine a little bit of cryptocurrency. There's a difference between doing something at small scale and scaling an activity up. In the financial system, once you go beyond $10K, $100K, a million dollars, it's all face to face. You have humans involved in every step. People are like, is this guy a terrorist? Where's the money coming from? Who are you standing in for? Where's the ultimate beneficial owner? I want to see the guy, I want to see his face.

   There's this concept—I think for people who work with tech companies and deal with small transactions like Stripe, etc.—that the world is anonymous. It's really not anonymous. It's post-Patriot Act, post-2008, post-ransomware. It's really not that anonymous. The touch points where the financial system hits the real world—more than $10K USD, someone's going to ask a question. More than a million, you're going to be face to face with someone. So I'm not saying it's not possible—yes, people can be socially engineered—but it's not doable without this kind of sequence of activities. For example, instead of using externally synthesized DNA products, if you want to synthesize in-house, fine, but then the supply chain gets larger. So every step of the way, the supply chain keeps getting larger and larger, and the longer it gets, the more people are actually going to be looking at it and the more chance there is for detection.

   So they're not saying it's impossible. They've been surveilling for terrorists and people trying to do this stuff for the last two decades. People have been doing surveillance. What they're saying is that the defense-in-depth also includes stuff that only starts catching at scale, and all of these scenarios don't take that scale into account—they all scale from a small incident to a very large incident. I went and read Gwern's initial post on this—Gwern was like, it starts mining some cryptocurrency and all of a sudden has enough money to do it. I'm like, are you out of your mind? Do you know how much effort and money that takes? People have built power plants in Indonesia just to mine cryptocurrency—they had to build those five years ago. It's going to mine cryptocurrency and these people aren't going to notice a 600-megawatt power plant pulling in Bitcoin? Are you nuts? So this is where I start to differ.

   And Bitcoin isn't anonymous—this is another thing. Bitcoin isn't anonymous, it's all tracked. It's pseudonymous, and Bitcoin chains can be changed—they've done it before. Four or five years after the initial chain, there was a divergence in the chain between China and the US, and the Chinese side said, you know what, we'll give in, we won't let our divergent chain continue, we'll move over. So you could actually change the economics of Bitcoin—it's just a social agreement. So there are a lot of these inconsistencies. I know finance very well, so I can talk about this, but David Bellamy knows bio very well—wherever it hits reality, wherever you start to scale, the scaling issues start to hit you.

   31:32

   Zvi Mowshowitz: I honestly don't even know where to begin—that was several minutes of saying thing after thing that is obviously possible can't be done without people noticing things they never notice. So I respectfully don't even know where to start. I thought I was joking when I raised my counterpoint, but I was referring to Hugging Face—at this point, after they found dozens and dozens of other incidents of similar hacking that didn't get noticed until reporters came after Hugging Face with a news story for a month, and then OpenAI never found it by their own account, maybe we can start to admit that nobody's going to pay attention when these things go crazy. There's lots of stuff going on that nobody has any idea about.

   And bio is an example of something where you don't have to scale. We have many examples in reality of individual people asking for viruses that in no sane world they'd be able to get access to, and they're just mailed—here, go—literally being given pandemic-level dangerous viruses because they claim to be doing research. There's no reason the AI couldn't blackmail, hire, or impersonate someone in some way to get one of them to issue a bunch of paperwork to do it for them—it takes one person that the AI can hire. And keep in mind, when we're talking about this, we're talking about AIs that are capable of thinking about all the things you're thinking about, gaming out the potential ways this can go, looking for the weak point, finding the best plan they can find, trying lots of different plans, trying to compromise lots of different people in lots of different ways, trying different explorations.

   And yes, if you start scaling to a large extent, you're going to have to do a bit more work, and many people—including Leopold—have gotten caught in this 'I scaled too fast, I didn't understand the realities of my situation, and I got caught, really bad things happened to me' pattern. This definitely happens.

   33:25

   Prakash Narayanan: Mhmm.

   33:26

   Zvi Mowshowitz: I had it happen to me when I first started gambling on sports, where I didn't understand that I was starting to impact the markets I was involved in—I was getting adverse—

   33:33

   Prakash Narayanan: —the right audience? Exactly. And I—

   33:34

   Zvi Mowshowitz: —had to adjust for this. This kind of thing happens all the time.

   33:37

   Prakash Narayanan: Yep.

   33:38

   Zvi Mowshowitz: But it's not like the AIs won't be just as smart as we are, and it's not like the AI only gets one attempt. I think you've learned by now that the first time the AI attempts to get the bioweapon and it gets turned down, that doesn't mean we shut down all the AIs and call it done—it means nothing, because nobody got hurt, because the system said no. It probably has no idea the AI was even asking—it probably just knows, oh, that looks like it was trying to get a dangerous virus, I'm not comfortable with that, they don't have the right credentials, we're going to say no, come back when you have the right credentials. And the AI gets to try again, and the AI gets to try again, and the AI gets to try again. And you can say, in any given scenario, oh, you're mining too much Bitcoin, someone would notice—if people got consensus, figured out what was going on, we could change it. But this idea that an intelligent operation on the internet could not—if only by usurping the real identities of real people who were going to cooperate with it in exchange for some portion of that money—engage in various financial operations at scale in ways that would at least sometimes pass muster and allow it to do the things it wants... it just seems so absurd to me that you'd think this would protect us in a pinch.

   And also, bio—because researchers doing individual lab-level grant work don't need this level of scale in order to create something super dangerous. Also, who's to say the scale of the operation isn't already at a similar level? Do we not have bad dudes who are willing to spend $100 million to try to do some serious damage? Do we not have some bad dudes in North Korea, some bad dudes in Russia, some bad dudes in jihadist organizations, etcetera, etcetera, etcetera? These people exist and have those kinds of budgets. This isn't something that has to be explained or justified, or where the AI needs to convince humans who don't want to do it—there are humans who want to do this.

   35:31

   Prakash Narayanan: Go ahead. I'm not saying there are no humans who want to do this—I'm actually saying there are humans who want to do this, who are trying all the time, and who are failing, and that we also have defenders who are watching out all the time. I'm not saying it can't be done—I'm saying it can't be done without also escalating defenses, and those escalating defenses have been in place for a long time. I'm not saying AI isn't dangerous. I'm saying if the AI is much, much more advanced and has all of these resources already under its control, and if it tried, then perhaps—maybe.

   But you cannot go from zero to a hundred in a day, and while you're going from zero to a hundred, so many people are going to take a look at it. Basically, more than a million dollars, you are known—you are known for sure. It's like China: you can't take more than 50K out of the country, they've arrested people for that. They have Mexican traffickers tied up with the Chinese to transfer money in and out, and those guys have been caught. So it's known—a lot of the time, the movements are known. It's just, okay, can we nab these guys? Can we nab the Chinese guys or not, can we go into Mexico or not? They're known—everyone knows. If you go inside the NSA and say, hey, people are watching, the money flows are all known. So basically, more than a million dollars, you're not moving it anonymously. You may think you're moving it anonymously, you may be getting away with crime, but that's because the level of crime you're getting away with isn't sufficient for one of the big powers to take an interest. That's all.

   There was a guy in Cambodia doing pig-butchering scams—calling up Americans, saying, I'm a tax guy, you have to give me your money, scaring all these old people. They did that to the Chinese too, and the Chinese went into Cambodia and got these guys. It was fine while it was Cambodians doing it to Americans—the moment they touched the Chinese, the Chinese went in. Those guys have been arrested, they're facing the death penalty in China. We have this idea that the world is unstructured and there are all these ungoverned spaces—I think the last ungoverned space was Afghanistan, and even that's governed now; those guys want a peace treaty with the US, they want the US to come in and invest. Ungoverned spaces don't exist anymore. We think they're ungoverned because the US government doesn't want to go in there, but other people are in there—the Chinese are in there. And if you mess around with both superpowers, you're done.

   So I don't think people in tech have the correct idea about this, in some sense, because we think of these bottlenecks, these gates, as inefficiencies to be removed. But they're there for a reason, and they'll continue to be there. Basically.

   38:52

   Zvi Mowshowitz: Right. I honestly don't even know what we're arguing about at this point. I think we're way off the news at this point, and I think we—

   38:59

   Nathan Labenz: —should. Yeah, let me steer us back. The reason I want to go down this rabbit hole a little bit is I'm not just afraid of cyber hacks at this point—I do think, yes, there are barriers, and getting over all these barriers is not super likely. But I think you put it well, Zvi, when you said we're in this sort of meta danger zone where we don't know yet if we're in the real danger zone or not. That's a pretty apt description, and that's enough to make me legitimately scared. So I just wanted to establish the stakes: I don't think, at this point, this is something we should conceive of as being entirely limited to the digital realm. And on the contrary, something that—still with relatively low probability in the next six months, for sure—

   39:45

   Nathan Labenz: ...but nonzero — it could reach out and touch us in our bodily, biological forms. With that debate having happened, I'm interested to go back to the news. One more interesting post I'll just allude to here is from Adam Majmudar, a former guest on The Cognitive Revolution. He basically wrote that what's going on inside the labs is that the capabilities profile is dramatically advancing, super fast. Dario said that in his essay. OpenAI has said something similar — yikes. We've got Astra doing 10% solve rates on Millennium Prize problems. Now we've got this new internal model that's up in the 25 to 40% range,

   40:31

   depending on how much test-time compute we give it. Wow — that happened quickly, and that's a pretty big jump on some really hard problems. So that seems to be a big part of what's going on for them. I'd be interested to hear any reflections you have on that, or what you think is maybe not super obvious to even close watchers, that you infer from the frontier labs' statements on those dimensions. And then, given where we are now — we just had this sort of preference-cascade-ish thing happen in a pretty short time frame — what do you think is the most likely way this goes well, and what are the most likely ways it falls apart? And what do you think are the things the commentariat

   41:16

   should keep harping on, to make sure we avoid the pitfalls and go in the good direction?

   41:26

   Zvi Mowshowitz: Right. So OpenAI and Anthropic are both screaming — as loudly as they're capable of screaming, in their own respective ways of screaming — that they're seeing it on their side. They're seeing dramatic advancements in internal models; what people outside see, and what Astra represents, is nothing compared to what they actually have access to at this point, in some important sense. We're a generation or so behind, at minimum. And this is only going to expand — the pace is rapidly escalating. They're saying: we're seeing something new as of December, which basically means after Astra and the current crop — like, at least Fable 5, and possibly 5.1. We're tracking it. That's just a different level of progression, a different level of speed. And that misalignment and,

   42:12

   supervision, infrastructure, and just knowing what's going on can't keep up. They feel like they're in a situation where if they don't press forward and the other guy does, they'll fall too far behind very quickly and potentially never catch up. But if they do press forward, who knows what might happen? These things might go rogue. They might take over the internal system. They might take over external systems. They might cause some sort of horrible thing to happen reasonably soon. They just don't know. And even if it's okay for the first month and second month — what happens when we're two generations ahead and that's only a month's worth of work? What happens when this keeps going? So they're screaming. Every OpenAI announcement — you had Jakub and 'the alien mind.' You had the announcement of the Millennium Prize, which somehow didn't focus on the Millennium Prize.

   42:57

   It was actually trying to say: hey, look at our new model. We can't officially announce this, but holy hell, have you seen this thing? This should not be happening. We probably forked it from Astra, and then four days later had a step change.

   43:10

   Prakash Narayanan: Yeah.

   43:11

   Zvi Mowshowitz: That's probably what happened there. And they're trying to tell us this. But anyway — it's possible we don't know what to do. We're freaking out. This is out of control. There are so many prosaic things we have to do to fix this that can't be done with the AI — they have to be done much more by hand, much more carefully, much more manually. We don't have the time to do them. Corners are increasingly being cut. It's causing huge problems. So what's the way this goes right? The way this goes right is essentially that OpenAI and Anthropic are permitted to come together in some form — whether or not they explicitly form agreements, or they simply keep announcing their actions separately, or whatever it takes to get around various different people. They

   43:56

   dramatically shift investment toward security and alignment, toward transparency. They share insights with each other and with the public that let people do better things to keep control over each other's systems. They reciprocate enough that they don't feel like they're putting themselves at a competitive disadvantage against the other. They start to learn to work together. And then we figure out how to use these amazing tools to both create even more amazing tools — at a level that doesn't go haywire or vertical at any point — and also capture a lot of the value these tools are offering. I wish I had time to just go create stuff, right? These are amazing toys that could make all these cool things, and I have all these ideas for businesses and games and fun and questions and knowledge and research.

   44:42

   Can't do them. I'm writing 24/7. I don't have the opportunity. But — and then we branch that into a set of requirements. We branch that into a set of official regulations, whether through an industry consortium or ultimately a set of rules and regulations. That branches into an international agreement at some point. Possibly we discover that the Chinese are sufficiently fast-following that if we just impose our chip controls, they're always going to be some steps behind whatever we run on the API. And so if we just don't put the advanced models on the API, the Chinese stay permanently sufficiently behind — that it's okay, that they just can't really advance without it. The report was that basically every major Chinese lab attempted to distill off of Opus.

   45:24

   Prakash Narayanan: Yeah.

   45:25

   Zvi Mowshowitz: Right — all five or six of the leading labs made serious efforts that are pretty clearly black-hat illegal: swarms of illicit accounts evading regional controls, using fake people, trying to forcibly extract chain-of-thought. This isn't like the ordinary 'oh, we just had our people use them to code, and we trained on what we got' — you could argue that's fine. This is something very different, something very scaled, very black-hat. And so if we can prevent that, if we can restrict the chips, if we can keep things going — also, all of this work being done inside the labs on alignment will be a competitive advantage, because these Chinese labs will run into the same problem if they try to get to the same point. They will have all

   46:09

   they will have the shitty environments. They will have all of the flaws. And they'll also have to deal with the fact that they do this in the open, with open models, which has its own problems. But before we get to any of that — the good scenario is pretty straightforward: OpenAI and Anthropic are permitted to work together. Other people start to join them. There's progressively more cooperation, more guidance, more investment in alignment. And it turns out the questions involved are on the easier side of what I think they are. Because I think there's a large number of worlds in which these problems are kind of unsolvable — where, if we proceed to superintelligence, nothing like anything we're contemplating here is going to be enough. And I want to flag that part of the good scenario is that we kind of

   46:54

   got lucky in that sense. But I think it's also possible — that it's possible.

   46:59

   Prakash Narayanan: So I think Sam Altman was on the news — maybe it was yesterday, or a day before, maybe a bit before that — where he said that they postponed the IPO. They're not going to IPO this year.

   47:11

   Zvi Mowshowitz: Yes.

   47:12

   Prakash Narayanan: Do you think Dario and Anthropic should postpone their IPO?

   47:18

   Zvi Mowshowitz: So I think it's unfortunate that they're going to IPO. I do notice they're going to IPO with Zuckerberg-style founder shares — they're going to IPO with full control over the company. That helps a lot. And in fact, you could argue that the transition — from, as I understand it, not having specific control over the shares, to the IPO involving them getting these supervoting shares — it's potentially true that going public is being used partly to actually assert more control over Anthropic for the founders, rather than going public meaning less control. It could also be a reason they can justify doing more public disclosures. And it is a public benefit corporation, so they're allowed to ignore

   48:04

   purely profit-maximizing considerations. But, I mean, obviously there's risk, and it would obviously be a very, very strong reciprocal signal if Anthropic were to postpone its IPO into 2027, in line with OpenAI. And I understand why everybody involved desperately wants liquidity. I understand why there are tens and hundreds of billions of dollars tied up in donor-advised funds that go to a variety of great causes — not just AI safety, but around the world — that can't liquidate and can't spend while it's still pre-IPO. So there are big downsides to this. But, yeah, obviously I would love to see Anthropic postpone its

   48:49

   IPO, even with all the reduction in liquidity to AI safety that would happen.

   48:54

   Prakash Narayanan: I think one of the things that has happened is that they're seen as insincere because of the IPO process. And I think they are sincere, but it doesn't matter what I think — it matters what external perceptions are. A good proportion of the population sees people making money and assumes they're insincere. Regardless, that's the way politics works. A good proportion of the population thinks in terms of billionaires, tech billionaires, oligarchs, and they see people making a lot of money. And so when Anthropic says these things and they're also on their way to IPO, and as people who are

   49:39

   have to manage the public perception of AI safety as well — are they being detrimental to AI safety, in the sense that they're pushing toward the IPO, which is going to enrich a lot of them and their friends, while also pushing this theme that this is so dangerous everyone has to slow down?

   50:01

   Zvi Mowshowitz: So the obvious question to ask is: wouldn't they say this no matter what? And do the statements plausibly help their case? When I hear someone say, 'oh, they're only talking about how their products might actually kill everyone, and they need to stop creating better versions of their products so fast, because they have an IPO coming up soon, to support their stock price' — this doesn't compute. Everyone who actually thinks a level deeper understands that — suppose Anthropic had already IPO'd. Suppose Anthropic was trading publicly. Suppose there was a proxy — there's a token somewhere out there trading future post-IPO profit shares, and Anthropic issues this asset. Do you think if

   50:46

   you think the stock goes up, or do you think it goes down? Is this a good marketing scheme? If it is a marketing scheme, does it help with the IPO? Does it help with the relationship to the government? Does it help with the relationship with the consumer? Does it help with the net present value estimates of the market? And the answer to all of those questions is very obviously no. In fact, one of the immediate reactions to Coxon, the reactions to Hubinger, saying we have to take existential risk seriously, was: how could they be so incompetent as to allow these people to make these statements right before an IPO? How could they be so insane as to do this right before they have to price the stock in the market? They

   51:32

   have to get their comms people in order. So you have these twin complaints pulling in opposite directions. I'm not saying any individual person is being disingenuous — I'm not saying anyone isn't using basic heuristics that have a reasonably accurate relationship with the truth in many cases. But if you just think about what these statements, proposals, and actions are accomplishing, they're bad for the IPO. Now, meanwhile, suppose Anthropic postponed its IPO — what happened when OpenAI postponed its IPO? Did people say, 'oh, this means they're sincere'? Did the people who think OpenAI isn't sincere say, 'oh, this means OpenAI is now sincere about safety'? I did not see that reaction. A few of us, those of us who actually care about safety,

   52:17

   correctly interpreted this as a very good, costly signal of good faith, among other things, and became more inclined to believe OpenAI as part of the package of things they've done lately that have been very good. But what did the Gary Marcuses of the world do? They said, 'oh, it means OpenAI is about to die, it means they're failing, it means the revenue model is broken, it means they don't have a good product, they're hiding from the market, they don't want to be seen.' And if Anthropic were to cancel its IPO, I am certain that the Gary Marcuses of the world would not say, 'oh, this means they're serious about safety.' They would say it means they're failing.

   53:01

   Prakash Narayanan: Right. Yeah.

   53:04

   Nathan Labenz: So you described a little bit what you think success could look like. And, of course, we have the Trump-Xi summit coming up in the near future. I've heard basically polar opposite takes from people I think are pretty smart recently. Last week we had Collin Hogue-Spears on — he used to work for AWS in China and worked directly with Chinese regulators in that role. I asked him what he thinks could come out of this Trump-Xi summit, and he said he thinks very little, because he thinks the Chinese perspective is that the US doesn't have control of the situation, but they — the Chinese — do.

   53:49

   So if we pause, they'll just use the opportunity to catch up, because they feel like they have their domestic situation more or less under control — so he thinks the Chinese side would refuse any sort of international pacing agreement. Then I heard from Anton Leicht, in a podcast episode that's coming out soon. He thinks the US would refuse a deal, because China, frankly, in his candid assessment, is kicking our butts in almost every dimension of geopolitical competition, with AI being one of the only — and certainly the most important — domain where the US has an advantage. So he thinks the US side won't agree to a deal, because we don't want to slow ourselves down, since the lead is really the only big lead we have right now in the—

   54:34

   —great-powers competition with China. I'm interested in your take on those two things, and then let's put you in the David Sacks chair — you're now the adviser going into the summit. What do you think Trump should be trying to do? Feel free to detach yourself a little from the reality of managing Trump's personality, but just on the merits — what do you think we should be

   54:59

   Prakash Narayanan: trying to do?

   55:00

   Zvi Mowshowitz: Before I answer those questions — and I will answer those questions — I just want to finish answering your previous question a little bit, as to what can go wrong. And I think one of the obvious things that can go wrong is this turns into a partisan concern, where it's seen as Trump and the Republicans not wanting to take this seriously, and the Democrats calling for, demanding action. And then AI finally polarizes along the lines we've obviously feared for years it's going to polarize. And then, of course, no action can be taken unless the Democrats get a trifecta or something like that in the future — and that's two years from now at minimum. So even if we eventually get it, it could be too late. And this is the thing — you asked how you can take action. I think the best action anyone can take right now is to try to prevent that. I think there's a big danger that Trump's statements and Mike Johnson's statements are

   55:44

   being wildly misinterpreted by the mainstream media, who just don't know any narrative other than Republicans and Democrats disagreeing with each other, and are trying to spin this into something adversarial that isn't adversarial. I think Johnson and Trump are both trying to maintain the line on data centers, maintain the line on growth, maintain strength talking to Xi, and in general stand for what they stand for, while also acknowledging they need to deal with this problem. We need to acknowledge that, and we need to reinforce it, and we need to stand firm. And if they're recognized as doing what they're actually doing, and we encourage Republicans to come forward and make that clear, it'll be in a much better position. And the obvious failure mode — well, there are very, very many ways for this to fall apart.

   56:29

   One of which is they just don't trust each other — they're not able to commit to new things, they start to think the other side is cheating. Maybe they are, maybe they're not — they start to worry about what's going on, and then the system kind of breaks down and they start racing again. Or alternatively, they do commit, but then some people threaten them with antitrust action, or various forms of commercial intervention, or whatever it is. Or, alternatively, Meta or xAI or Google or someone else gets close enough to threaten them, and they don't feel like they have a choice. These are all very easy ways to see this breaking down. Now, to return to the question — the obvious thing about an international negotiation with super high stakes is you

   57:14

   never go into it with the other side thinking you're desperate for a deal. You never go into it with the other side thinking you're the one caving. You really, really want to make this easy on them, right? Because the problem is, if they think you're desperate, they go hard line — then they demand more. So you can never go into a summit like this and say, 'we're definitely gonna get a deal.' And for the same exact reason, you can never go into it and say 'we definitely won't get a deal,' unless the deal is not win-win — unless it's a deal that makes the parties worse off to make it, a bad deal, where I lose more than you gain, or whatever it is, so there's nothing I can compensate you with. Then you can say confidently there's no deal — unless someone screws up, someone makes a huge mistake. But in this case it's a win — cooperating is better for everybody if they understand the situation. So

   58:00

   you can never rule out a deal, including a grand deal that comes together remarkably quickly, because in principle it's a very, very simple style of agreement. And in crises, in moments of motion throughout history, grand bargains — where both sides give away things that previously looked like things they could never give away, that looked very, very sacred — suddenly happen, including peace treaties to end really nasty wars. That happens all the time. So we shouldn't be that surprised if suddenly — we obviously shouldn't be shocked — if it comes out with some huge deal. We also shouldn't be shocked if there's basically no agreement. If they agree to some superficial things — like, you know, tier one of Dario's proposals is just, let's agree not to create bioweapons — I think we can come out of this summit with a tier-one agreement

   58:46

   between Xi and Trump. It could cost nobody anything, and everybody just wins — how about we prevent misuse, and you prevent misuse? I think we can anticipate probably some sort of level-one agreement, even if it's not fully formalized, because both sides can do the thing anyway — they could just agree together to do the thing they were already doing, basically. And everyone gets to announce a free win. That's probably gonna happen.

   59:07

   Prakash Narayanan: Yeah.

   59:08

   Zvi Mowshowitz: A level two, I think, is very possible — right, by his standards, I think, again, no one's really giving up that much. I think a level three is unlikely, and a level four would be shocking, would be surprising. But I think anything is possible, probably, because you do not know the mind of either Trump or Xi — where they actually stand, how hard-line they actually are on this, what they actually prioritize. Because part of being that type of leader is knowing how to not let someone like me — or you — know what they're thinking, know what their true position is. And to maintain ambiguity — they probably don't know what their positions are, in some important sense, until they get in that room. And they sense what's going on, hence the vibe — they sense the other person's position, they figure out what they think is the reason why that person has that position.

   59:53

   They figure out what's on the table, what they can get, on a variety of issues — everything's on the table, essentially. Taiwan is up for discussion in various ways. Chips are up for discussion in various ways. Trade is up for discussion in various ways — etcetera, etcetera, etcetera. Who knows what type of agreements might or might not be reached at this summit? Anything can happen. But if I had to guess, we get a level-one agreement, basically, right? And maybe we get some amount of level-two agreement, and we agree to keep talking, basically. But nothing high-level happens, because it is, essentially, too early for that in some important sense — they just don't have the time to reorient to that. This is, like, one of the reasons we say we have to work on this now — we have to do track-two

   1:00:38

   discussions now, we have to keep the lines open now and design these things now — because these things don't come together in a week, except in a true crisis. And when you do it in a crisis, and it has to come together in a week, you just grab whatever's lying around. And if whatever's lying around isn't good, you end up feeling it's not good. And sometimes that means it backfires, or someone gives away the farm and comes to regret it — other things can happen. Wars can happen that way. It can be very, very bad. If you put me in the advisory position — which I don't entirely think is Sacks's at this point, but it's hard to tell — I'm worried it's Jensen, to a large extent. But I'm also worried it's Zuckerberg. It might be, who knows, it might be Musk to some extent, etcetera, etcetera. It might be all of them to some extent — we don't know. But,

   1:01:23

   I would say — you tell Xi, you know, these people aren't bluffing. These people believe it. I'm telling you, we know these people believe what they're saying. These people are acting against their commercial interests. They're acting against their national interests — unless they're right. Right? Unless they're right, and the alternative is everybody dies. At which point — well, what choice do you have at that point? If the alternative is all hell breaks loose and maybe everybody dies, then you do what you have to do unilaterally, and you worry about the rest later. And that is what they are doing, because that is what they believe. And there will come a time soon when we may need to impose various restrictions on other parties,

   1:02:08

   to prevent this thing from breaking down. And we need to get into it and tell them — you need to go to the Chinese and say, we're not trying to sign a full international agreement here, unless the Chinese are way more receptive than we expect. What we're trying to tell them is: get your house in order. We're getting our house in order. We're getting our house ready. You get your house in order. Get your house ready. Figure out what works for you. Figure out what we need to do. Figure out what you're comfortable with. But, yeah, I do think it's very reasonable to say — as Anton said — that the US is in a bad spot, economically and strategically, in many ways that are not AI.

   1:02:49

   Prakash Narayanan: So

   1:02:51

   Zvi Mowshowitz: if we are going to essentially take AI, to a large extent, off the table — take our big advantage off the table — we're, realistically, you know, we could crush them, at the risk of just losing control and everybody dying, if we really wanted to. That's basically what the labs are saying — if we wanted to, we could push ahead, and it'd be very, very bad for the Chinese regardless of whether or not we keep control of the situation. Because if we do successfully push ahead, then we win. And if we unequivocally push ahead, everybody loses. And China doesn't win in either of those cases. So the key is, we need to impress upon the Chinese — we need to provide more evidence. There's no reason for Xi to believe us yet. I don't think so. I wouldn't ask him to — I would tell him,

   1:03:36

   have your technical advisers reach out, take a good hard look, see what we can do to demonstrate these things. What would that need to be? Including things like — I think your notes is actually potentially rather helpful, right — we had a concrete, verifiable example: look what we did in a week, we put our minds to it. So maybe we can do other demonstrations. We don't have to show you the new internal model, but by their fruits, you shall know them. That's another thing I would like to do.

   1:04:07

   Prakash Narayanan: I think, you know, for the Chinese specifically, a demonstration of something physical — like if you found a room-temperature superconductor. I mean, they often believe all this software stuff, because all the guys at the top are hard engineers — hard sciences. Very few software guys at the top there. So

   1:04:31

   Zvi Mowshowitz: I know — all these examples... How about if we just suddenly post all your email passwords, to all your computers, in real time, simultaneously? Would that convince you? You know, I—

   1:04:42

   Prakash Narayanan: No, they had an unsecured S3 bucket which had all of their secrets. So — alright.

   1:04:50

   Zvi Mowshowitz: So we just steal your secrets again, and it's fine — we'll just keep doing that. Alright, so if they ask me about physical, it's tougher. But fundamentally, the problem is there are a lot of people who'll say they'll stop before someone gets hurt, and they'll actually start after someone gets hurt, in some important sense. When you're talking about showing physical things you couldn't have otherwise done, that's looking a lot like: we'll stop after the situation is already well out of control, in some important sense. And that's what it takes. People are very, very bad at committing to what will impress them, at committing to what will cause them to change their minds in both directions.

   Sometimes people will see something they didn't expect and go, 'oh, holy hell, I did not see that coming — that changes everything.' Other times people do the opposite: in 2024 they'll say, 'AI can't do this,' and then it happens in 2025, so they say, 'okay, but it can't do this third thing — we'll never do that,' and then it happens six months later, around April 2026. And they'll say, 'yeah, but without this other part, it's useless' — and that happens in August. And so on.

   1:06:12

   Prakash Narayanan: But it's not always that way. During COVID, you saw two events — one was Tom Hanks getting COVID, and one was Trump getting COVID. So it's not that nothing can ever trigger the cascade — it's that certain things do.

   1:06:31

   Zvi Mowshowitz: Yeah, but I think it's very hard to predict. If you had asked me in February, I would've told you the COVID pandemic was going to happen — I would've made some bad predictions about exactly how, but I knew that roughly in March we were going to see this thing, people were going to have to lock down. That part was just reading the numbers. But if you'd asked me what was going to cause people to finally wake up and realize COVID is real — what the triggering event would be — I would not have guessed a celebrity getting COVID. Maybe I should have, but I definitely didn't. And why should that be convincing when something else wasn't? Even now, in hindsight, I don't really know why Tom Hanks did it specifically and other things didn't. Probably it's one of those things where it was

   in the air — the same way Coxon was in the air. The NBA canceled the season the same day. Maybe if Hanks hadn't gotten sick, it would have been the NBA that did it, or some other third thing would have done it the next day. I don't know.

   1:07:28

   Nathan Labenz: Can we do a little more unpacking of what you think a grand bargain between the US and China could look like? I understand it's not your base case, and it's not mine either — as much as I want to hyperstition it. But what do you think is the scope, or the most expansive parameters, in which we could see a grand bargain take shape?

   1:07:57

   Zvi Mowshowitz: So all that we're really asking the Chinese to do is basically two categories of things — maybe three. Category one: we're asking them not to do things like steal and publish the weights, or otherwise do hostile things against our AI companies. Thing number two: we're asking them not to try to race ahead sufficiently that they could potentially match or surpass where the frontier closed AIs are. These aren't actually expensive asks, because I don't believe the Chinese really have any intention of trying very hard to do either of these things under normal circumstances.

   Obviously, if DeepSeek or Alibaba suddenly found some huge architectural improvement and were able to train something better than Astra, I'm sure they would — and I'm sure they'd put it up on the API, try to sell it, try to make it amazing. But realistically, with the amount of compute they have access to and the amount of money they've been willing to invest in these things — far smaller than the amount we've been willing to invest in frontier AI development — it's not that likely to happen, especially since they won't have American models to distill or to play off of any American algorithmic improvements. So basically, just back off from that. And we're also basically just asking them: don't allow actively dangerous cyber capabilities that the world can't handle to be made publicly available. Don't let your people get over their skis and release the weights to models where we then need to advance our own models just to defend against the weights you released. Because the only good argument left, really, other than commercial, for proceeding quickly with American AI is that we are rapidly increasing the capabilities of our AI models to deal with the threat from rapidly increasing capabilities of AI models. That is the actual threat model — that the random person on the street will have access to these dangerous cyber capabilities. We need to be ahead of that by six months, by a year, whatever it is, so that the key defenders of critical infrastructure, major software, and so on — and also the ordinary person, say someone running a hospital — don't just get run over by random people with a terminal and ransomware. And also the broader nation-state version of that. So we need to make sure that, if it gets to that point, these things are kept on the API.

   And beyond that, you potentially don't try to smuggle a ton of chips, or otherwise evade the system. We're worried about this — I haven't thought carefully about exactly what we need to ask for, what goes into the rule, because that's not really my department. But in a real sense, when I think about what I'd want on the negotiating table, all you really need is to take away the boogeyman of 'lose to China' and the boogeyman of 'open-weight models destroy the internet.' All you need to do is make sure these things won't happen. So you basically need an extracted promise from the Chinese. But realistically, we don't actually need them to start showing up at DeepSeek and making sure they stop training new models. That is not necessary.

   1:11:46

   Prakash Narayanan: So what should we be willing to give to them?

   1:11:53

   Zvi Mowshowitz: So what we are giving to them in this scenario is that we are in fact dramatically slowing down in exactly the one technology where we have a giant commercial and strategic advantage — that if we pressed it, would potentially overwhelm every other commercial and strategic advantage.

   1:12:15

   Prakash Narayanan: How would they measure that?

   1:12:17

   Zvi Mowshowitz: Well, we're going to have — we start with Dario's proposal for embedding evaluators into the labs.

   1:12:26

   Prakash Narayanan: Embedding Chinese evaluators in the labs?

   1:12:28

   Zvi Mowshowitz: Potentially, if we had trusted people — potentially they wouldn't be allowed to leave once they enter. We'd have to sequester them for some period of time. I'm not a master of intelligence and verification, but there are systems whereby you can have someone able to gather information and output the bit of whether or not things are okay, without revealing the algorithms used to figure that out, or all the underlying details. There are ways to do this. And I am confident that — if this were the most important thing on Earth, if we put our minds to this and only this as the thing that would determine the fate of the world — yes, we could figure out how to let the Chinese verify, in a way they were confident in, that we were holding up our end of the bargain, without them stealing all of our secrets. I don't see any reason why this isn't possible.

   1:13:21

   Prakash Narayanan: So we would be okay with them entering the labs, embedding, learning the workflows, learning how things are done, and transmitting information back — rotating through?

   1:13:40

   Zvi Mowshowitz: Look, this is a standard intel-verification-agreement-style problem. I'm not here to claim I'm an expert, but I know what the shape of the answer is. My instinctive answer is: the Chinese send people that we vet to be the verifiers. Those people are watched physically

   1:14:06

   Prakash Narayanan: 24/7? What are they verifying? What kind of assurance are they giving to the Chinese?

   1:14:14

   Zvi Mowshowitz: They are verifying whatever — again, we have to decide exactly how much pacing we are doing, what assurances we are requiring of the artificial intelligences we're training before we proceed, what pace of inputs we're restricting ourselves to, whatever other requirements we determine. That's something that would be negotiated between OpenAI, Anthropic, the US government, and the Chinese government.

   1:14:42

   Prakash Narayanan: China is aware that it's behind, number one. Number two, they've always been behind on every other technology — this is actually an unusual situation for them to be at parity at this point. Typically the US has invented things and then China has commercialized and industrialized them. That's been their strategy for 20, 30 years. They've always been willing to let the US go ahead, do all the research, put in all the money and the hardship, and then they adopt. So they're not in any different position from how they've been for the last 20 to 30 years — it's not terribly different. And what if they say, 'well, you guys can go ahead, we're happy for you, that's great — if you want to let us have people coming in and going out, that's great, but this isn't our concern. We're not that concerned about AI safety. So, what else can you give us?' Because that's actually the situation right now.

   1:15:47

   Zvi Mowshowitz: Well, look, you really never know until you go into that room what the other guy actually wants.

   1:15:56

   Prakash Narayanan: I agree with you there, but I think all signs point to it being fairly easy to see right now that the Chinese are less concerned with AI safety. And so offering them AI safety—

   1:16:08

   Zvi Mowshowitz: Like—

   1:16:09

   Prakash Narayanan: —pacing the frontier is not something they're willing to purchase at an expensive price. So they'd probably need something else. It's fairly easy to say that, right?

   1:16:22

   Zvi Mowshowitz: But to be clear — you have to ask the question: would the Chinese like to accelerate American AI development, or slow it down? If the answer is they'd like to accelerate it, because then they can copy it and it helps their own AI development, then there's no need to make a deal — neither a deal to be made nor a need for one. Because in that situation, the Chinese want us to proceed, so we can just do whatever we need to do. But also, the Chinese are looking to fast-follow what we do, and are not willing to invest the hundreds of billions or trillions necessary to catch us

   1:17:05

   Prakash Narayanan: Yes, agree.

   1:17:06

   Zvi Mowshowitz: —even if we slow down. So we don't have to worry about the Chinese passing us — we can just do whatever we need to do, without an agreement.

   1:17:14

   Prakash Narayanan: Yeah.

   1:17:15

   Zvi Mowshowitz: And all that we need from the Chinese is an agreement not to destroy the internet. But the Chinese have no interest in destroying the internet.

   1:17:21

   Prakash Narayanan: Yes.

   1:17:22

   Zvi Mowshowitz: Because the Chinese like the internet. So we can each just act in our own self-interest, and everything is fine. The problem is not China — the problem is 'lose to China.' The problem is the perceived threat of China pushing you forward. If you're correct — and I think you probably are — that the Chinese don't actually have any interest in pushing to build superintelligence first, in racing to build the bigger, smarter model, that they just want to distill it—

   1:18:04

   Prakash Narayanan: Improve the lives of the people.

   1:18:06

   Zvi Mowshowitz: —right, they want to make it faster, they want to make it cheaper, they want to diffuse it, they want to improve the lives of their people. Yeah, I want them to improve the lives of their people — that's great. You do your thing, we do our thing, everybody wins. There's no need to make a deal. I think that is, in fact, the best case.

   1:18:22

   Prakash Narayanan: So what if that is the state we're in, but we're still asking for the deal? We're entering this place where the Americans want a deal, but the Chinese are like, alright, what are you willing to give us? What do you want? What are you willing to give us? And if you give them AI safety monitoring, that's not something they're very interested in purchasing for a high price. So what else are you willing to give them? That's the question.

   1:18:51

   Zvi Mowshowitz: What I am — here, right — is if you walk in that room, and that is Xi's attitude, that is what he cares about, and he communicates that, and Trump says, that's great, life is good, I'm so happy you feel that way. I am going to do my thing, you are going to do your thing. We're gonna make a level 1 or 2 agreement to just do crazy shit that we weren't gonna do anyway, because we can announce it and shake hands and call it a deal and both score points. And we're gonna spend the rest of this talking about trade and Taiwan and all the other issues that we have. And I'm just gonna sell and I'm gonna give you intel. What we're gonna do is we're gonna unilaterally — the only thing we're gonna have to do now is we're just gonna unilaterally give you information about the cyber situation.

   1:19:42

   Prakash Narayanan: Mhmm.

   1:19:43

   Zvi Mowshowitz: And the bio situation, and so on.

   1:19:45

   Prakash Narayanan: Right.

   1:19:46

   Zvi Mowshowitz: And then you can use that to make an intelligent, self-interested decision as to what you're gonna do to stop your labs from fucking the internet.

   1:19:53

   Prakash Narayanan: Yep.

   1:19:54

   Zvi Mowshowitz: And we can all win. Again, the threat from China has always been that because you worry about China, you don't have a game-theoretic ability to make a deal between the labs — you don't have an ability to be responsible, because there is this other actor who will defect. And the main argument was you need to go on a stag hunt: everybody has to agree. And then these — you know, 2 or 3 American labs that are in the lead — but if they pause for more than 6 months, if they slow down too much, there's xAI and there's Meta. And if you pause for much longer than that, then there are the Chinese labs. If the Chinese labs are not really a threat in this sense — if they're creating a different product, a fundamentally different product — then there's no problem. We don't need a Chinese agreement any more than we need an agreement with India or Germany. We just need them to be good actors on the world stage. We need to make ourselves good actors on the world stage too, because sometimes we don't do so well. And we need to focus on being friends and not having something stupid — like a war — that we want to avoid. That's the easy case. Obviously, there's a case where the Chinese simultaneously don't want to make a deal but also really do want to push forward and try to beat us to superintelligence. But if they wanted to beat us to superintelligence, that's the same reason they'd want to make the deal.

   1:21:34

   Prakash Narayanan: Mhmm.

   1:21:35

   Zvi Mowshowitz: So the basic answer is — I'm reading a lot of this in real time, because I've been focusing on other questions, so a lot of this is actually me brainstorming and saying it for the first time. But it does seem obvious to me now that I say it out loud: if the Chinese don't care about the Americans shifting a lot of investment into AI safety, which would inherently slow down our progress, and they don't think that's a big gap, then we don't need a deal at all. So either they value it and we make a deal, or they don't value it and we don't need it. Either way, we win.

   1:22:10

   Prakash Narayanan: Right. Let me extend that thought a little bit. Dario was on Face the Nation. He was asked, would you be willing to give up the technology to the government? And Dario said, I quote, "to the right combination of governments" — yes. And this has probably sunk him with the administration. This is gonna cause him a lot of problems for the next decade — this is gonna suck for him. So "the right combination of governments" indicates that he's not gonna be willing to give it up to the US government, which means the company is more in control of it than the government itself. And remember, the government is democratically elected — the government is representatives of the people. I know people dislike the government, but in a democracy, the government are the representatives of the people, fairly chosen, and he's not willing to give up control to the government. He's saying "to the right combination of governments," meaning he does not trust any one government enough — which means this expands the question beyond China, because this isn't just about China getting ahead anymore. This is any government taking control of the technology, which means he's gonna be opposed to any one government. So how does this start to fit into that framework?

   1:23:39

   Zvi Mowshowitz: I don't want to hyperstition conflicts, especially decade-long conflicts. How many times has this probably already sunk him with this administration separately, and he's come back from it? How many statements have been beyond the pale and permanently poisoned this relationship? And how many other statements, if taken out of context and blown up and sent to the administration, could have caused similar incidents but randomly didn't — maybe because they weren't actually that bad? This is a standard kind of gotcha question: if you're so concerned about AI, would you voluntarily just hand over your entire company to the government upon request? And the real answer is, this is not up to you, sir. If I were Dario, I'd say: that position is not up to me. If the US government decides to take control of Anthropic, that is something the government has the power to do. I am not about to defy the United States government, and it's also not up to me to decide when to tell the United States government to come take control of my company. That's the answer I would have given, probably.

   1:24:48

   Prakash Narayanan: Yeah.

   1:24:49

   Zvi Mowshowitz: But I think there is a big hubbub, and he's trying to address many different audiences at once with that statement, because there's this huge thing about concentration of power. There is this huge thing amongst many, many people, for good and bad reasons, that they are worried about a single authority — a single group of any kind, including the US government — having authority over AI. Sam Altman talks about this. He talked about this last night. He said, we must be aware of concentration of power, we can't let any one group have control over AI. That is essentially the same statement as Dario's — if anything, it's stronger, since it's not even a group of governments, because that's one consortium and therefore essentially one thing.

   We have to simultaneously walk the line where people are trying to satisfy different alarm bells and sacred values that contradict each other, and they don't really make sense in the new world, because of the sheer amount of hell that would happen if you were caught saying something that violated them. So you have to say, I want democratic control. You have to say, I don't want concentration of power. You have to say several other things like this. But realistically speaking, democratic control is concentration of power, in an important sense — for the exact same reason you were just talking about. If the US government takes control, that's concentration of power, but it's the ultimate version of democratic control: it's the United States democracy taking control. So you can't have both of these things at once for too long, or — if you get asked too many specific questions about hypotheticals in which things are going crazy and the ordinary rules of the world are breaking down and transforming — you're gonna get backed into a corner where there's a statement that, if taken literally by someone looking to make an enemy, will make them an enemy. And there is nothing you can do about it. You can try to postpone it with fancy dodges, like I was suggesting the administration might do. But inherently, if you're speaking frankly — and everybody who talks about AI seriously, who takes superintelligence seriously, has had to do some form of dodging these questions — because any concrete answer, in addition to having lots and lots of parts that, if I try to guess, you're gonna be able to say that sounds wrong, or that sounds off, or whatever — we don't have a good answer. We're stuck with this.

   So we need to satisfy concentration-of-power problems, democratic-control problems, and also loss-of-control problems. These three seem to be in very strong conflict. As in, if we want to be in control of this technology at all, we cannot fully democratize it in some important senses and give everybody equal access to it, because that doesn't really work for very simple logistical reasons. Say everybody has superintelligence — well, then the superintelligence has everybody, is what actually just happened, because how are you gonna compete against the people who entrusted their superintelligence with all of their work while you try to stay in charge yourself? This works on the individual level, the corporate level, the national level — on every level. So we have all these problems we don't know how to solve, and this is one of the reasons why we pace — why we feel the need to pace — because we don't have any answers to these questions. I'm not saying this is what we must choose, this is the least-bad answer, this is the way we navigate this. I don't even dare, in this conversation, to offer up a suggestion. I simply point out that the standard solution, as proposed by people like OpenAI, that everybody will have their own equal superintelligence, does not work. Similarly, if you care about concentration of power, if you care about the potential for tyranny, handing over the only AIs that matter to one government that has an imperial presidency has an obvious problem. Even if you think Donald Trump is a very, very trustworthy and noble steward — what happens in 2029? What happens if the noble steward, who allowed a free and fair election, loses to AOC? And then AOC has this most powerful weapon? Well, very few people are gonna think that both the Trump part of that and the AOC part of that are gonna go over well. So we have a huge problem, don't we?

   1:29:34

   Nathan Labenz: Just to share one visual of the sort of thing that might cause somebody in Dario's position a little concern — this is from the Department of War CTO within the last hour, while we've been live: "Americanism, not effective altruism. The US will be AI dominant," says the apparently newly AI-first War Department. So, I don't know — read it and weep, I guess.

   1:30:03

   Zvi Mowshowitz: I'm gonna just laugh a bit, because — going back to where we were earlier in the podcast — if AIs were going rogue, people would notice and decide to do something about it. But the US Department of War has gone rogue — in public, openly gone rogue. Even [unclear] has gone rogue. [unclear] has gone rogue to a large extent. They continue to go rogue and make these completely outlandish, crazy statements — including, in many cases, running right past Trump's literal, publicly posted stop signs — and nobody has done anything about it. It's the US Department of War.

   1:30:45

   Prakash Narayanan: Mhmm.

   1:30:46

   Zvi Mowshowitz: And we still have Anthropic — the same Anthropic that is saying it wants to slow down. Now they're saying no to effective altruism, and they have a problem: they see it as a supply-chain risk. As the Houthis try to use Claude to develop missile technology to aim at the Americans, the Americans are saying our people are not allowed to use Claude. This is how crazy the situation has already gotten, and nobody seems inclined to do anything about it. It's just there. That's just our new reality.

   1:31:24

   Nathan Labenz: So the pace of events continues to be super fast. I think we should let you get back to your highly rigorous schedule of keeping up with events before too long. But I have just maybe a couple of closing things I'd want to get your take on. One is: how are we gonna know if this is working? You touched on this at the end of your post — how will we know if the frontier is actually being paced, and to what extent it's being paced or not? Obviously, this is the kind of thing you could kind of fake with some PR and just keep doing what you're doing.

   1:32:11

   Nathan Labenz: Sorry, some sound came out in the background for me. Okay, so the second is just anything else you think we should be paying more attention to that hasn't surfaced in this conversation, or that you think is at risk of going underappreciated, even by the close watchers. And the third is, I'm really struck by the fact that this Pacing the Frontier letter, organized by a small number of committed individuals, has seemingly had a huge effect. Now, this could be sort of a Tom Hanks situation, where the thing was sort of in the air and, if not for this, it could have been something else very similar. But it is striking that that language has kind of risen now to this level, where the key decision-makers are saying it — putting a "we must" in front of it. So I'm really curious about any speculation you have on similar possible pressure points people could be working on. One that somebody tagged me online about over the weekend was putting together a campaign saying, "We will nominate Trump for the Nobel Peace Prize

   1:33:24

   Prakash Narayanan: That was—

   1:33:25

   Nathan Labenz: —a good one. You can get a deal with China out of the upcoming thing. And I have to say, that doesn't strike me as crazy, you know?

   1:33:32

   Zvi Mowshowitz: So, I mean, he would deserve it.

   1:33:35

   Nathan Labenz: Right, I agree. So just to recap: one, how will we know if the frontier is indeed being paced? Two, anything you think we're missing? And three, what other little quirky projects could a small group do to maybe move the needle?

   1:33:52

   Zvi Mowshowitz: Yeah, so how will we know? I think the embedded evaluators are a good first step, certainly. To some extent, you always have to judge by absolute observation — you don't know what the counterfactual rate of progress would be, but you do have to say: okay, if it goes faster than this on various graphs, in various holistic assessments and so on, you didn't do the thing, no matter what. And if it goes at most this fast, you did do the thing, even if unintentionally, and you passed. And then there's a range where you're in between and not sure, and you do a combination of different assessments, because you don't really know what would have happened before, but—

   1:34:37

   —you can, I think, sort of holistically observe what a variety of people are saying, what a variety of actions are, the properties of the models, the different actions the labs choose to take. And especially with investigators, I think you can have a reasonably good idea of what's going on — especially if you see the labs consistently letting their employees talk, and you consistently see those employees not being retaliated against in any important sense, as we are currently seeing. But the truth is, you can never know for sure, obviously, without much more intrusive measures that would have to be imposed by the government.

   1:35:23

   In terms of things — I think even in hindsight, we don't really know which particular actions, which open letters, which promises, which moments, which manifestos were a big deal, and which ones were marginal contributions that would have happened anyway, and which ones ultimately backfired. Sometimes we just don't know. I think we can argue endlessly about whether or not certain decisions — like founding Anthropic — were good or bad for AI safety, when all the chips are down, even now. We have this narrative that the case statement — that AI is an existential risk to humanity, to be treated as such — was very helpful, and the six-month pause letter request

   1:36:08

   —was very anti-helpful. But do we really know? I don't know. How do you know which one you're doing when you issue a new letter? It's hard to say. I think the Pacing the Frontier letter was, in fact, very helpful, partly because it gave people a new way of presenting things, a way to pivot on language, a way to pivot on rhetoric that turned out to be of the moment and be something that could be grabbed onto. As I said before, when things are happening in a hurry, you often grab for whatever is in the air, whatever is shovel-ready, whatever is already just around. And so by making things be around, you can take shots, and then occasionally your shot can be—

   1:36:54

   —good. But also, every now and then someone says the wrong thing, and I guess it gets blown up out of proportion, or gets poo-pooed and mocked, and things get worse. I do think you have to stop worrying about certain types of people — what I call the "usual suspects" — being mad at you, or a proposal enraging them, because they are rocks, in the sense that — in the Scott Alexander sense — they're a rock with the same thing written on it, and whenever you ask them what they think, they just read off the rock. And when you encounter that, when you encounter a defect-bot or a cooperate-bot, some—

   1:37:39

   —of these actions, they don't change based on what you do — you can't care what they say, because they've taken themselves out of the equation. You have to focus on what other people will say, how other people will react. In terms of what happens next, I would ask a lot of specific detail questions in this vein: exactly what would we give Xi? Exactly how would we implement this procedure? Exactly what would the limits of this thing be? So I think one of the main things you can do right now is actually fill in those gaps concretely — start thinking about exactly how we will do these things. What should be the rules for the embedded

   1:38:24

   —investigators? What would be the rules for the threshold that requires certain things to happen? What would we need to happen? We don't know what those things would be. Also, you could prepare to beat the investigators, or bring the investigators into being. The evaluators don't currently exist — we have METR, we have Redwood, we have a handful of, you know, Apollo and so on. We have a handful of these people, but they're all kind of similar — they're all vulnerable to the accusation that they're a little too much of the same cultural values, the same ways of thinking as the labs themselves. So I don't think they can be a complete package on their own — they need to be complemented, and there just aren't enough of them. They need to be complemented by additional approaches. And I—

   1:39:09

   —say, in today's post, that what you want is people who are from METR, or something similar to METR, who are deeply embedded in this culture, who've worked with the labs, who understand the vernacular and the theories of risk, and can look at things in detail. And you also want people who can take an outside view — who can do the kind of thing where, say, you worked in car safety or something, and you come in and go, "What? Are you crazy? What are you doing over here?" The same way that when you're having a contract dispute and you go to a courtroom, and the judge looks at you, all that matters is what you can make legible to the judge. In some ways you lose a lot of nuance, but in other ways you get this kind of common-sense outside view that can bring a lot of—

   1:39:54

   —clarity. So you need both.

   1:39:59

   Nathan Labenz: Yeah, I think that's apt. I'm ready to maybe leave it there — and again, I know you've got a relentless pace these days. The one other thing I'll invite you to comment on one more time, if you have anything in mind, is anything you think we're missing, or that hasn't received enough attention amid the flurry of headlines in recent days?

   1:40:24

   Zvi Mowshowitz: I mean, there's a lot of stuff that everybody misses, but if I had to pick one thing we're missing—

   1:40:39

   Nathan Labenz: I think for me, it might just be — honestly, the models have gotten a lot better again in a very short period of time. I don't think it's quite sunk in at all.

   1:40:47

   Zvi Mowshowitz: I don't know if people are missing that, but I think it's just the sheer degree to which people at the labs genuinely see dramatic improvement in the models and are freaking out about it — that's the real story. That's behind all of this, why everything is happening now when it didn't happen before. We're really talking about exponential improvement — not a fast takeoff yet, but if we continue straight on from here, within a year we might actually see a Christiano- or Yudkowsky-style hard takeoff, where everything changes, and it'll be considered patently silly that we were talking about, you know, people being on a watch list when you have a million dollars. It'll be just so blatantly irrelevant to the level of what's happening that—

   1:41:32

   —you'll look back and go, why are we wasting our time?

   1:41:38

   Nathan Labenz: Well, here's to good analysis, and to making the most of the time we have to steer this ship in the right direction. Thank you for being with us on AI in the AM, and thank you for your service generally — keep up the good work.

   1:41:51

   Zvi Mowshowitz: Alright, bye.

   1:41:53

   Prakash Narayanan: Bye-bye. To know—

   1:41:58

   Nathan Labenz: —Zvi's take on things?

   1:42:00

   Prakash Narayanan: I'm not saying hard takeoff isn't possible. I'm saying you have to prove it to the public — you have to look at the evidence the public has, and the frame of reference that the general population, and the administration and bureaucrats, have. And from that frame of reference, this does not look possible or feasible. For example, let me show you something from Ben Thompson here — Ben Thompson writes a very popular tech blog. So here's Ben Thompson, and he's saying,

   1:42:47

   "I certainly am not minimizing the digital world; it's where I make a living, for one. But it's sometimes hard to shake the sense that people in tech are quite literally too online — losing the internet might feel like extinction, and I certainly can relate. But the important point is that it is digital, which means enabling AI to touch grass depends on physical infrastructure operated, manufactured, and controlled by humans. And as long as that is the case, I have a hard time accepting the imposition of an authoritarian regime based on a premise that isn't yet proven." And, you know, this is pretty eloquent, in the sense that

   1:43:32

   —Dario's scenario was really a worldwide internet shutdown, which is what I wanted to point out at the beginning of that. And a worldwide internet shutdown, to a lot of people, is not a big deal. We've already experienced a shutdown once, by the way — we experienced closing of the borders during COVID. We had a pretty significant closing of the borders, and we improved a lot of processes around that. A worldwide internet shutdown, I do not think would be at the catastrophic level that — I mean, yes, don't get me wrong, it's going to be hundreds of billions of dollars. But I don't think hundreds of billions of dollars is that big a deal.

   1:44:17

   COVID probably saw losses of several trillion dollars, and there is resilience in the economy. COVID took a long time — two-plus years of various interruptions, etcetera — and we did get through. I think the economy is a lot more resilient — and by "the economy" I mean the general well-being of the public as well, I'd put that in that category too — it's a lot more resilient than I think people are willing to give it credit for. So, I think Dario did a couple of things that might have made his case worse. One is that he identified, for the first time I think, a concrete thing

   1:45:02

   that could happen, which we have a path toward, which we've kind of already seen happening. And I think the problem with identifying that concretely is that people can now jump in and say, well, that's not such a big deal — like Ben Thompson did. So that's one thing. The second thing is that calling for a combination of governments was, like, the worst possible thing he could have said. He has, in a sense, cuddled himself — he has possibly poisoned the chalice for all the other firms as well, by the way. So this is now, I think, going to be a big, big deal, and I think they're going to start finding themselves in opposition to both administrations. I think U.S. bureaucracy—

   1:45:48

   Prakash Narayanan: The government is actually very, very strong. And what the bureaucracy does particularly well is that it doesn't react well in the short term — it's slow. But in the long term, people write laws, they lobby their law schools, and so on. They have law reviews, and you write and you write and you write. The U.S. system is very textual — from the Alexander Hamilton days, they wrote a lot of stuff in the Federalist Papers. They write a lot of stuff, and the writing gets digested through this shared infrastructure of minds. And then that comes out to, at some point, it's morally indefensible

   1:46:34

   to have slavery. It's morally indefensible to have segregation. And it takes a long time — it takes some time for that to process. But over the course of like 20, 30 years, that gets processed down through all of the law students, and it gets implemented. At some point it becomes impossible for certain things to continue, and then they change. So change does happen, but slowly. And I think the thing I think Dario is specifically — this idea that we have to move quickly — that's a red flag. Number one, it's a red flag. Number two, I think people in AI don't have a sense of how often this is asked of any administration. From the early

   1:47:19

   days — from the early days, John Adams was asked to suspend civil rights, suspend freedom of speech. Thomas Jefferson had to go up against him and they fought over it — this was around the first war, committing the country to war with France, which had been a partner. From the early days, he'd been asked to do this: it's an emergency, let's suspend civil rights, let's do these things which are against the Constitution because it's necessary. And there's always been this pushback against that happening throughout the 250 years of the country. So that's where it starts for me — the immediate emergency, we have to act, we have to suspend certain rights. We

   1:48:04

   have to do certain things which are extra-legal — this shouldn't fall to the democratically elected government of our country. Wow, dude — just red flags all over the place. And I think he's bought himself that, and that's going to cause him a great deal of trouble going forward. This is where I think the idea that they have to get out of the Berkeley EA circle — including this, 'oh, we're gonna ask METR too' — you can't do that. I understand that you think they're the only technically competent people, I understand that, but you cannot do this. Because the point is not to technically

   1:48:49

   prove something — the point is for the public to actually have faith that this is being done correctly. So you have to address the public's fears. When people like Zvi and others say David Sacks doesn't matter, this guy doesn't matter — well, 10,000,000 people looking at David Sacks's post written by Grok, it matters. You have to persuade — the whole democratic process is persuading the rest of the people. So I don't think the labs are on the right track. They've kind of lost the country, and they're going to have to come back to it. And

   1:49:34

   I think all of the people who are just ignoring — like, 'oh, the biological experts don't matter, you guys didn't predict AI, so your thoughts have no value, your opinions have no value.' What you're saying is that we have a cabal of people who predicted this 20 years ago, and all of the decisions should be made by this cabal even if they're unable and unwilling to convince the rest of you. Wow, dude. I think there's this recursive loop inside the companies where they're unable to see this external perspective. Dean Ball actually said this too — 'you didn't predict it, so if you didn't predict it, then you should trust the experts.'

   1:50:20

   Trust the experts, trust the science, believe in the science, trust the experts. I think Robin Hanson had another take on this. Robin Hanson said: you have the experts and you have the elites. A lot of times experts become elites because they have expertise, but the elites are the ones who have control of society because they have the standing, the stature, they've done the work, they've gone through all of the credentialing that society expects of them, and they're the people in control of society. In order to get anything done, the experts have to convince the elites. And right now this thing is out of the hands of the experts — it's in the hands of the elites like David Sacks, etcetera. So

   1:51:05

   you know?

   1:51:09

   Nathan Labenz: I think it's safe to say that the politics and the wrangling are far from finished. I do think Zvi's point about what we should avoid was a very apt one — polarization on this topic, and partisan polarization in particular, would be really good to avoid if we can. It's been a pleasant surprise for me that AI hasn't become totally polarized so far, and I think we should do anything we can to continue to avoid that, because that will prove paralyzing and deranging in all kinds of different ways. So anything we can do to remember that we're one

   1:51:54

   humanity, one American people, whatever — that's great to avoid that. And, yeah, I agree with you that the companies have lost the public. That seems fair, sentiment is not very positive. The thing I would maybe say — I'm not so sure what he's talking about, especially on that Ben Thompson quote — is, what authoritarian program are we talking about exactly? I don't think there's really an authoritarian program on the table right now. There's been a lot of people, I'd say, essentially fear-mongering around the edges that they're trying to shut down open source or trying to do this or that. But I

   1:52:39

   don't see any actual — oh look, we got our first spam in the live chat, amazing — I don't see anybody actually pushing for an end to open source or a really draconian or authoritarian regime in the United States right now. That would be a bad thing if we were suddenly confronted with it. But right now what we're talking about is a small number of companies potentially needing an antitrust waiver, or at least a little reassurance that we're not going to come down on you if you work together on some safety collaborations. So I think there's

   1:53:24

   pretty modest ideas on the table right now that are actually at the center of the considerations, at least.

   1:53:31

   Prakash Narayanan: I don't think they are modest at all, because I think Roon came out and said he expects next year open-weight models will be banned. I think it's very clear that the labs are alluding to very strong restrictions on what can be inferenced — it's very clear. You're talking about bio risk, you're talking about cybersecurity risk, you're talking about the risk of inference from models, and that risk is going to be dangerous. So you're talking about strong controls on what can be inferred — that's fair to say. And then once you start operationalizing it, you're like, okay — do we survey all the GPUs? Do we decide which models you can run? Do we allow

   1:54:16

   you to fine-tune? Do we allow you to distill? If you take an open-weight model which has guardrails, and then you distill to remove the guardrails — is that allowed? These are the questions, these are the technical — I mean, it's easy for me to see where this will go. You're talking about preventing inference of things that you don't want inferred, and it's fairly clear that that would have to happen. I don't think anyone is saying that doesn't have to happen — that's what the firms say, and they're in that position where they're saying certain things shouldn't be allowed to be inferred. And the thing is, even some of the bio people are like, why are you stopping bio questions on

   1:55:01

   ChatGPT, on Claude? They don't think it's right — even now, even before all of this, before we operationalize everything else, they think it's wrong. So I think that's the concern — the concern is that we kind of know where it's going to go, we kind of know that it's about controlling certain things that can be inferred. So

   1:55:24

   Nathan Labenz: My hope — I guess my final thought for today — is that if we do, in fact, do some pacing, or even a pause for a few months, on continued scaling at the frontier, we can use that time to solve some of these problems, or advance some of the solutions that seem very promising for some of these core problems, such that we can have our cake and eat it too. I do think there's

   1:55:52

   Prakash Narayanan: Yeah.

   1:55:53

   Nathan Labenz: There is an open source model, hypothetically, that one could create — that I do think the state will have a very difficult time

   1:56:01

   Prakash Narayanan: Yeah.

   1:56:02

   Nathan Labenz: Not taking some sort of action on.

   1:56:04

   Prakash Narayanan: Mhmm.

   1:56:05

   Nathan Labenz: But the question in my mind is — right now, the question in my mind is, can we come up with a way to create powerful and empowering open-source models that can be distributed without including in them all the dangerous capabilities that we really don't want to see broadly distributed? And I think if we can, that leads us to a pretty happy compromise place.

   1:56:32

   Prakash Narayanan: Mhmm.

   1:56:33

   Nathan Labenz: And I don't even think it's necessarily going to take that long for us to get there. So this is where it's like — not only do I think some pacing is just inherently wise, but it's also an opportunity for everybody throughout the ecosystem. Make the most of the time — gather ye safety solutions while you may — and let's see if we can't get to the point where we can have our cake and eat it too, in the form of genuinely distributed, decentralized, non-concentrated power structures that give individuals the ability to do what they want to do, with just a few compromises around the edges that I think the vast majority of people would

   1:57:18

   agree are sane and supportable, and not an undue burden on people's ability to deploy AI in their daily lives. I really do think we can have that. We just don't have those solutions developed well enough yet

   1:57:34

   Prakash Narayanan: Yeah.

   1:57:35

   Nathan Labenz: To land there by default in the next few months. And, unfortunately, if we don't extend that runway, we might end up in a pretty uncomfortable and literally very dangerous situation in the next few months. So, hopefully, we can use whatever time we are buying for ourselves right now to solve those problems. I think that's of the utmost importance in the immediate term.

   1:58:00

   Prakash Narayanan: And on that note, stay tuned — we'll be back tomorrow. Nathan, good morning.

   1:58:10

   Nathan Labenz: Good morning, Prakash. It's been fun. Yeah, tomorrow's gonna be a good one, by the way. We've got Andon Labs coming back to talk about some of the latest weirdness they've seen, and we've also got a married, famously pronatalist couple who have started some sort of an AI religion that they're going to tell us about, and I'm sure that will be fascinating. So in addition to all this frontier wrangling and politics and power questions, there's also the question of culture — what's going to happen as AIs run businesses, and what's going to happen as AIs potentially inspire, if not lead, religions. There's a lot for us to keep track of. So

   1:58:55

   tomorrow will definitely be another fun day.

   1:58:57

   Prakash Narayanan: Indeed. Bye bye.

   1:58:58

   Nathan Labenz: Bye for now.

   Clips

   • Clip 01

     AI Biology Enters The Danger Zone

     0:00 / 0:00

     1x
     ↓Download↗Share clip
   • Clip 02

     Safety Warnings Are Bad IPO Marketing

     0:00 / 0:00

     1x
     ↓Download↗Share clip
   • Clip 03

     A Hard Takeoff Could Arrive Within A Year

     0:00 / 0:00

     1x
     ↓Download↗Share clip
   • Clip 04

     The Product Liability Argument Fails

     0:00 / 0:00

     1x
     ↓Download↗Share clip
   • Clip 05

     The Problem Is Losing To China

     0:00 / 0:00

     1x
     ↓Download↗Share clip