EPISODE 2026-09-10
A show that ran nearly three hours and lost twelve minutes of itself. Nathan Labenz and Prakash Narayanan open on the second day of the Jacob Coxon story, now less a resignation than a fight over what the resignation was: Prakash walks through the right-wing read that it was a coordinated operation — a scrubbed account, a Wall Street Journal exclusive timed to the post, familiar amplifiers, familiar funders — and Nathan calls it a midwit conspiracy theory, explaining from personal experience how the Survival and Flourishing Fund's champion model actually distributes money and why the dollar figures being treated as suspicious are, next to METR's $75M or Geoffrey Irving's new organization's $150M, rounding errors. Prakash's counter is narrower and lands harder: AI safety hires the way elite universities graduate, and a Second Amendment conception of AI safety would never get a hearing inside a frontier lab. Collin Hogue-Spears, author of From Lab to Life: How AI Works in China, then spends forty-three minutes on the mechanics almost nobody reads — the algorithm-filing regime, the split between national-security and non-national-security model registries, what an MLPS audit actually asks of a cloud provider that legally cannot sell to its own customers — and argues the gap between the US and China is about fear rather than capability, that Beijing would know about a Chinese OpenFace-style breakout before the public did, and that he has low expectations for the coming Trump-Xi meeting because AI arms control is far harder to verify than the nuclear kind. Baseten co-founder and CTO Amir Haghighat comes on to talk about the Blaxel acquisition — five days old, and Baseten was a Blaxel customer before it was an owner — and draws the security line where a micro-VM vendor can actually draw it: Baseten can guarantee one customer's code cannot touch another's and can catch an agent trying to break out, but what a customer's agent does inside its own sandbox is the customer's problem. Seven and a half minutes in, mid-answer, the stream died. Amir never came back, twelve minutes of the interview exist on no recording anywhere, and the show's closing became its longest segment: Prakash's margin-stack reading of the GPU cloud business, Dan Hendrycks' new essay against utilitarianism at AI companies, whether AIs are moral patients at all, and Prakash's argument that the takeover people fear already happened — through financial markets, which have been removing heads of government for decades.
AI:AM for Thursday, September 10, 2026. Two guests, a stream failure that cost the show twelve minutes it will never get back, and a closing block that ran longer than both interviews combined.
This page is the as-aired record. Timings and deep links are on the raw-master broadcast archive; the transcript is speaker-attributed and lightly edited for readability.
Watch
As aired
The show opens with Nathan Labenz and Prakash Narayanan trading a quick greeting before diving into the Jacob Coxon story, the resignation post from a young ex-Anthropic-adjacent researcher that Prakash calls one of the biggest tweets of the year. Nathan notes he got a call from CNN International for a five-minute segment on it, that Daniel Kokotajlo was on Joe Rogan this week, and that Coxon himself has done the media rounds (Bret Baier's show, a clip with Anderson Cooper). Eliezer's joke — "Happy Jacob Coxon Day to all who will celebrate in the future" — comes up, but Nathan is skeptical the moment will be remembered as pivotal, arguing news cycles are short and the feed tends to revert once the initial shock fades.
Prakash then walks through the right-wing backlash building against the Coxon post over the prior 24-48 hours. He reads through Parker Thayer's viral thread alleging a coordinated PR operation: a scrubbed, low-follower account that got a Wall Street Journal exclusive minutes before its own post went live, quickly amplified by Nathan Calvin (general counsel at Encode), Peter Wildeford (AI Policy Network), and Daniel Kokotajlo (AI Futures Project) — with funding trails leading back to Jaan Tallinn and Dustin Moskovitz's Good Ventures Foundation — plus Bernie Sanders' pending bill to ban superintelligence. Prakash follows with Brian Chau's piece tying Coxon to the British group Newspeak (linked to Extinction Rebellion, prison-abolition funding, and an EA Infrastructure Fund tied to SBF, plus a Jeremy-Corbyn-assaults-bankers video game), Theo Jaffee's reach statistics on the post, and Elon Musk's and Tim Sweeney's comparisons of the moment to a coordinated "psy-op" or the Abundance movement's rollout.
Nathan pushes back hard, calling it "a midwit conspiracy-theory take" and walking through how the Survival and Flourishing Fund actually works — he's personally been a recommender for Jaan Tallinn's process, which uses a "champion" model (one convinced recommender can fund something without full consensus) rather than requiring committee agreement, with grants published openly. He argues the dollar amounts being cited as suspicious are, in context, small: a $20,000 fellowship is basically an internship stipend, and a couple million dollars to Daniel Kokotajlo is modest next to METR's $75 million and Geoffrey Irving's new organization's $150 million. He credits sincere donors and sincere recipients rather than orchestration, and notes AI-safety nonprofit pay has only recently risen to compete with frontier-lab salaries.
Prakash counters that there's a real ideological blind spot at play: AI-safety hiring skews the way elite university graduates generally do — upper-middle-class and left-leaning — and a Second Amendment-style "arm everyone with powerful AI" conception of safety essentially never gets a hearing inside labs like Anthropic. Nathan responds that the "blue-haired polycule" caricature undersells how monastically people at OpenAI and Anthropic actually work right now, and suggests the more durable political axis going forward may be pro-tech/pro-progress versus anti-progress rather than college versus non-college — pointing to Steve Bannon, Josh Hawley, and Bernie Sanders converging on skepticism of AI from very different directions.
Prakash then plays Tyler Cowen's argument that the anti-data-center backlash won't meaningfully slow construction, since only five states need to agree to host them and governors like Josh Shapiro tend to announce symbolic moratoriums only after the next two years of deals are already signed. He cites blue-collar electricians' unions defending data-center jobs, Palantir's Alex Karp framing AI as a blue-collar boom against white-collar losses, and Ro Khanna's call for an NRC-style regulator for data centers — noting the actual NRC approved zero new nuclear plants for 45 years. Nathan closes the block by pointing to Bernie Sanders' viral TikTok reading unsettling agent-transcript quotes from the METR report, arguing that people across the political spectrum may simply be sincerely updating on genuinely alarming new information rather than cynically piling on — and that assuming good faith is the better default given how confusing the moment is — before segueing into the show's first guest segment.
Timestamp links open the original source recording.
Key moments
My guess is that it's a little early for naming holidays based on this tweet.
This is a midwit conspiracy-theory take. I basically have no time for it.
It's very hard for me to see a strong Second Amendment conservative — who believes AI is another form of arms — get hired at Anthropic doing AI safety.
Related
Theo Jaffee's 24-hour reach check-in on the Coxon post ↗Mediaite on Daniel Kokotajlo's Joe Rogan appearance ↗
Prakash Narayanan: Good morning. It is Thursday, September 10th, 9:01 AM. Nathan, good morning.
Nathan Labenz: Good morning, Prakash. How are you today?
Prakash Narayanan: I am very good. We've had an exciting, I think, 24 to 48 hours. The Jacob Coxon story is now, I think, one of the biggest tweets of the year. It's a story that just keeps on going — at this point it is absolutely enormous. He was on Bret Baier's show yesterday. He's done the rounds of all of the new shows. And now we are at the stage of the backlash — the counter-move has already started. What are you seeing on this?
Nathan Labenz: Well, I got a call from CNN International yesterday to do a five-minute segment with them, which I think just goes to show how far down the roster of possible guests everybody has to reach now, because everybody's booked up doing all these little media appearances. Daniel Kokotajlo was on Joe Rogan this week. Jacob, as you said, was on several shows — I saw a clip with him on Anderson Cooper. And, you know, it's going to be interesting to see — obviously I'm in good company observing that we have very short news cycles and very short memory. You can look back at these previous moments where it just felt like something truly incredibly massive was happening, and even if it still feels massive in retrospect, it's like, yeah, well, we kind of just moved on from it anyway. And it's a little early, I feel, to make a call on whether or not that will happen here. And no less than Eliezer said, 'Happy Jacob Coxon Day to all who will celebrate in the future.' I don't know — my guess is that it's a little early for naming holidays based on this tweet. I just sort of suspect that we're going to go back to business as usual pretty quickly here. Obviously the temperature continues to turn up, the vibe has definitely shifted, the Overton window is certainly very wide open. But will we look back on this moment as being pivotal? I guess I have to say my bet is against, but that's mostly just on priors — that even when these things go super-duper viral, the feed moves on. It sure seems that way. I was in one group chat yesterday where somebody said, 'I think the peak has already happened — my feed is kind of reverting back to its prior state.' Yeah, I don't know, time will tell. What do you see? I haven't seen much backlash, though — what backlash are you seeing?
Prakash Narayanan: Let me share the backlash in question. I've been following the backlash quite closely, and it's been interesting to watch the story change over the last 24 to 48 hours. Let me start off with this — this is the first part of the backlash. I think this is by Parker Thayer. Parker is from Hillsdale, kind of a right-wing youth-elite figure, and he comes out and says: 'This post looks like the start of a very sophisticated and well-funded PR operation to get support for Democrats to regulate AI into oblivion. This guy, with minimal followers and no previous account activity' — literally a new, scrubbed account — 'goes to the Wall Street Journal, which publishes an exclusive with quotes from him on his resignation eighteen minutes before his own post goes up.' So, obviously, number one, he has a PR agency — he's coordinated with the PR agency, the PR agency has reached out to the Wall Street Journal, given them the story, given them the exclusive, the Wall Street Journal posts, and then Jacob posts. Within hours it has tens of thousands of reposts, and the account is over a hundred thousand followers. The post is punchy, quotable — it almost seems professionally written. The first few accounts that came out of this and started retweeting it are: number one, Nathan Calvin, general counsel at Encode; Peter Wildeford, head of policy at the AI Policy Network — we've had him on the show before; and Daniel Kokotajlo, head of the AI Futures Project — all prominent AI-doomer policy-advocacy nonprofits. They trace the funding, and a lot of it goes back to early EA and Anthropic funders, including Jaan Tallinn, Skype co-creator and an initial seed funder of Anthropic. 'What are the odds that the first three quote-tweets of Coxon's post would all be major AI-restriction policy advocates?' Well—
Nathan Labenz: Pretty good, I would say.
Prakash Narayanan: I mean, who else is going to retweet it? And anyway, someone pointed out, like, 'Look, guys, they're all in the same group chat anyway.' Like, where—
Nathan Labenz: Where? That's an extreme reach in terms of trying to sniff out a conspiracy.
Prakash Narayanan: Jacob Coxon doesn't have much of a resume, but we do know that in 2022 he got a $20,000 fellowship from the Good Ventures Foundation, which is backed by Dustin Moskovitz, who was also an early investor in Anthropic. Then they dig down — the fourteenth person to quote Coxon's post was Max Nadeau. And then, finally, a major Democratic politician and candidate has suddenly glommed onto this post — and, conveniently, as people cry out for answers, Bernie Sanders already has a bill written to ban superintelligence and regulate AI into oblivion, which he'll be releasing later this week. So that's Parker Thayer's take on it, and that kind of swung, I think, the right wing into action. So the next thing that comes out is from Brian Chau. Brian Chau goes out and says: 'Ex-Anthropic doomer Jacob Coxon has ties to Newspeak, a British NGO involved with hacking democracy, Hillary Clinton, prison abolition, and Extinction Rebellion.' And this is where things start going for me — it starts to become, like, okay, this is more of a left-wing framework, because Extinction Rebellion — I'm not sure if you remember this — but they had this whole thing about climate policy and how the UK should get rid of all cars by 2025. All cars, like, zero — go to basically zero gas cars by 2025 or else we're all dead. So this is very odd — to see the same kind of alarmists that were in climate policy kind of move over to AI policy. And that's when it's like, why? Why is it the same group of people? I don't want to say. Anyway — Coxon, a British AI-doom figure who 'exploded out of nowhere,' etcetera, etcetera — 'influence campaigns, when Newspeak's specialty. The Guardian describes Newspeak's mission as trying to hack British politics. Newspeak isn't subtle — here's one of Newspeak's produced games, in which Jeremy Corbyn assaults bankers.' So they had a computer game where Jeremy Corbyn assaulted bankers. Okay — but it's not like you can prove that's engaging propaganda. One of the members is a postdoctoral researcher in computational propaganda at the Oxford Internet Institute. Newspeak's funders — the Long-Term Future Fund, the EA Infrastructure Fund — funded by SBF,
Nathan Labenz: $50,000, $30,000.
Prakash Narayanan: Coxon isn't the first AI-safety figure to be tied to prison abolition — Moskovitz's criminal justice reform program gave $207,000,000, etcetera, etcetera. And then we have Newspeak itself confirming, 'Proud to count Jacob as a Newspeak fellow.' And then there's one last thing I have here — let's see, ah, there: Theo Jaffee comes on from MTS. He says: 'Okay, 24-hour check-in — 123,000,000 views. Elon, the owner of Twitter — I don't think this has ever happened for a post from a new account with almost no prior activity.' That's number one. Number two, in response to Parker Thayer, Tim Sweeney says: 'Reminds me of the Abundance movement suddenly showing up on the scene with a well-choreographed, simultaneous entrance by dozens of big-tech lobbyists all trying to appear spontaneous and organic.' Elon: 'I think the groundwork for this psy-op, for lack of a better term, has been prepared for a long time — this was just the match that lit the fire.' And that's probably, I think, the house line that the right wing and the Republicans are going to take going forward on this one. So there you have it. So, Nathan—
Nathan Labenz: I don't know, I—
Prakash Narayanan: First of all,
Nathan Labenz: I think that — I mean, okay, I've got memes for you, that's what I think. This deserves a meme response. And so here comes the meme. The meme is very simply: this is a midwit conspiracy-theory take. I basically have no time for it — where do I even begin? It's just absurd, I think — this is very much, I don't know, some sort of conspiracy psychosis, and there's a lot of different psychosis going around these days. But the idea that this is some sort of galaxy-brained Illuminati, Dustin-Moskovitz-lab-contrived fake groundswell of support is just truly absurd, I think. All these grants, again, have been public for a long time. A lot of them are not that big. There's money flying around everywhere in AI — you can trace money back to all kinds of different places. I personally — I guess one bit of color I could give — I've personally been a recommender for Jaan Tallinn's Survival and Flourishing Fund process. He's not the only donor that supports that process, but he is a notable one — he has some co-funders as well. And, again, all of the grants are typically published openly online at the end of the process. It's a very structured process that involves the opportunity for recommenders to dissent from particular grants. There's a whole — it's really, honestly, maybe a little too elaborate sometimes. There's so much structure that as a recommender I was a little lost in the structure, not entirely sure how the signals I was sending into it were being processed. There's a whole aggregation algorithm that's used to combine the recommendations that different recommenders give into a single allocation of the grant funds that are available. And it's an interesting model where you don't have to have consensus from the recommenders to get money from the fund — it's more of a champion model, which I think is pretty smart. This, again, speaks to the strength and vibrancy of the U.S. nonprofit sector, and how it's a huge advantage, I think, that we have relative to, for example, China, which doesn't have it — you need a lot more consensus, a lot more buy-in from the higher-ups to start any sort of non-business organization. They just don't really have the room for that kind of social or cause-focused entrepreneurship like we do. Here, even within this whole structure — if you can get one, typically I think it's six recommenders that they bring together for a whole process — but if you can get one to champion your idea within the group, you don't even have to, as a recommender, convince other people that it's a good idea. Although there is a process for discussing different things and trying to air out disagreements and hopefully come to mutual understanding, if not agreement — but even if no agreement is ever reached, if just one of the recommenders says, 'I think this is really a good thing to support, and I'm going to put a very high weight on it,' that individual mark will be enough to put money through to whatever organization. So this is something where Jaan has obviously done well financially — from Skype, from being an investor in Anthropic. From the beginning, his whole mindset as an investor — one of the things I think he brought to Anthropic, and I've heard and seen this brought to other investment opportunities too — is he will literally say to companies, as an investor: 'I can tell you I'm definitely cool with the equity going to zero if that's ultimately the right thing to do.' It's the same question Anthropic asks people as they go through the hiring process at the company. He volunteers that to companies he takes equity investments in. And here he's kind of been like, 'Man, I'm overwhelmed by all these different opportunities, all these different ideas — I need people to help vet them, but I don't want to be too conservative and require, like, a committee to fully approve of something.' So they deliberately structured this whole process so that even one — and that's obviously a person he thinks highly enough of to ask them to be a recommender — but even one recommender can be the sort of contrary opinion that causes something to get funded. I actually took a little bit of pride — I don't know if this is perverse pride or deserved pride, unclear — but I took a little pride the last time I did the process: I was either the least-correlated recommender with all the other recommenders, or maybe the second-least correlated. So I was definitely one who was pushing for some stuff that others didn't think so highly of. And all that is to say — there's not some grand conspiracy, especially when you get down to these small grants. I mean, Daniel Kokotajlo getting a couple million dollars — again, that's not by any means crazy in today's world, where METR's got $75,000,000 committed, and Geoffrey Irving's — I'm thinking of the old name, but his new organization — has $150,000,000 committed. So a couple million to Daniel Kokotajlo to write scenarios and try to put thought pieces out and go on Joe Rogan is not an extreme amount of money. And when you go down to these very small grants, like $20,000, that's basically funding somebody's internship — it's not really a big deal. So I think this is all pretty ridiculous, and I think the right interpretation is that this is just sincere people recruiting sincere funders, and those folks coming together to do what they think is best. It's not a great way to get rich — it has historically been a way that people have kind of eked out an existence while doing something they really believed in. Only recently have salaries in the AI-safety nonprofit space started to come up, as funders have realized they've got to, if they want to be able to compete with frontier-lab hiring at all. So I think that's been a smart move, but it's a very, very recent move.
Prakash Narayanan: So — where I differ a little bit on that is, I think we end up having a blind spot, which is the same blind spot I think universities have in general — which is, if you look at university graduates, it's like 60 to 80% liberal-left-wing. And I think the nature of upper-middle-class society is more left-wing-ish. And I think it's very hard for me to see a strong Second Amendment conservative — who believes in the right to bear arms, who believes AI is another form of arms, if AI is speech and arms, etcetera — get hired at Anthropic doing AI safety. Right? Meaning that their vision of AI safety is really to equip the entire world with very powerful AI, which they're able to defend themselves with — which is actually kind of a valid viewpoint, I think, in the public sphere, in places where the Second Amendment is taken seriously. So I think we tend to be blind to that, because as upper-middle-class people, we're exposed to more people who are liberal and left-wing. And so we tend to think that the nature of reality is such that this is the right thing for society. And I think there's a lot of people out there who are not in that bucket, and who are not really represented in these discussions — they're not in the chat group. It's very hard for me to see a strong Second Amendment, First Amendment conservative in those chat groups saying, 'Well, AI is speech, AI is the right to bear arms — instead of trying to dominate it, we should equip everyone with it. Everyone should have the capability to inflict massive harm on other people,' which is like the right to bear arms — and that in itself is a way of defending democracy. It's hard for me to see people accepting that as a valid viewpoint and then getting hired to do AI safety at one of these organizations. I don't think that works. And so I think you end up with a skew within — and I'm not saying the skew is wrong, my sympathies are obviously in that direction — but if you're coming from a more conservative background, it does look as though it is very typical of the upper-middle-class liberal-university-graduate skew that you can track in every other, like, sociographic study. It's right there.
Nathan Labenz: Well, I think those cultural — certainly I'd like to joke about the blue-haired polycule culture from which many of these ideas come, and sure, there's definitely going to be some kind of reflexive conservative distaste for those ideas, just based on their existing distaste, perhaps, for the people who came up with those ideas and the decadent or hedonist or whatever flavor of problematic lives they believe them to be living. But, honestly, for what it's worth, I think — the blue-haired-polycule thing is a funny joke, but I think actually most people who work at the OpenAI-and-Anthropic duopoly, and other leading AI companies too, are not really living very hedonistically right now. They're much more, like, working around the clock — approaching this period of time like a World War II, going all-in on trying to make a difference for whatever theory they individually have of how they're going to make a difference. They're not doing a lot of dating, I don't think, because they're just so focused on work. But anyway, that's an aside. I think there's a scrambling of political coalitions going on, though, because, sure, you've got this reflexive distaste for the Bay Area, whatever, and in some ways you see that echoed as just, like, anti-big-tech — you see the kind of Josh Hawleys of the world kind of like, 'I don't necessarily understand this, but I didn't like the last wave of stuff you did, and I don't think I'm going to like this much either.' And so you have people from Steve Bannon's camp, and Josh Hawley, and Bernie Sanders all kind of finding themselves on the same side of this issue. And I think that the sort of pro-big-tech, pro-change, pro-progress — if it's indeed progress — versus anti-all-those-things axis may be, in the end, a much more relevant axis than, like, college versus non-college. And I'm not so sure — although, of course, there's going to be a lot of intersecting, a lot of superposition, a lot of noise — but I'm not so sure that the culture war will be the lens through which this ultimately plays out. It kind of scrambles the culture war itself.
Prakash Narayanan: So let me play this from Tyler Cowen — I think people on the stream can hear it. Tyler Cowen basically comes out and says the anti-data-center backlash, at least to some extent, is happening, but that doesn't mean it's going to effect change in any way, because you only need five states of the union to agree to have data centers, and the data centers will get built regardless. You can have this kind of broad-based distaste for them, but there's very little chance of the federal government preventing data-center construction outright, or having a moratorium that engages all fifty states — it's basically impossible. And so some states are going to fold at some bid price — as you keep increasing the bid prices, some states will fold. And, specifically, he says that a lot of, like, Shapiro in Pennsylvania, etcetera — they talk about it, but they're not going to do any actual blocking, and they're going to go ahead and do stuff like, 'Oh, we're not going to allow any data centers for a year,' when all the data centers for the next two years have already been signed, and everything that's going to come up is already going to come up for the next two years. So this is just kind of an election-year give to the people who need it. So I think that's consistent. I think a lot of the pushback — it's very distasteful for me to see Bernie kind of clamp onto it, because I feel like Bernie is really interested in the jobs issue, and he's bandwagoned onto the AI-safety issue because he sees that as something he can push forward. But he's really just concerned about jobs, and he's been concerned about jobs for far longer than the AI-safety issue — if you go back three to six months or so, he's been talking about AI and AI jobs for a while now. It's also very interesting to see, because you have blue-collar electricians' unions kind of saying, 'Hey, we need these data centers — they've already chased out all the manufacturing, our people are getting hired right now, we're getting jobs, we're getting pay increases, we're getting paid $300K, $500K, and we don't want to have this be chased off.' You've also seen Alex Karp of Palantir come on and say, like, 'Hey, this is a blue-collar boom, and all the people who are going to be losing jobs are white-collar people.' So you also see the shift in the Democratic coalition over the last ten years or so, where they've become this upper-middle-class coalition — really consisting of doctors and lawyers, etcetera, people who benefit from putting regulation into place, and people who benefit from administering these complex rules — that's the Democratic coalition now. And they're also the people preventing building, with lots and lots of regulation, etcetera — they prevented construction of homes, and they prevented construction of all the — it just goes on and on. And Ro Khanna is coming out and saying we should have something like the Nuclear Regulatory Commission for data centers — and, as you know, the NRC did not approve a single nuclear power plant from its inception, I think around 1979, 1980, until 2025. So for forty-five years they locked it out and didn't approve a single one. So this is the state of play right now. I don't think the Dems are authentic — I don't think they care. I think they're using the issue — you need to grab onto the things that catch the public imagination. I think the jobs issue — talking about jobs doesn't help, because the blue-collar labor workforce is seeing this as a strike back against white-collar labor, so I don't think that actually helps them. And so this is the new thing. And I think the blue-collar guys are going to come out and say, 'This is bullshit' — that's where we are.
Nathan Labenz: Maybe I'm naive, but I would also leave some space in this analysis for people just sincerely updating their beliefs on pretty colorful new information that's coming to light. I saw a TikTok from Bernie Sanders this morning where he was reading agent quotes from the METR report. He starts off by reading these quotes, and then he goes on to say, 'Who do you think said that? Do you think those were soldiers in an army preparing for battle, or some heroic strike?' No — they were AI agents. And, obviously, we can only get so deep into any individual's psychology, but there's been this idea for a long time in the AI world that people need to wake up. And I think the simplest story for a lot of this is: people are waking up. They're seeing this stuff now, and it really is insane, and they're like, 'Holy shit, all the people who were warning about this were maybe right.' And now it's like some early version of it is here, and I guess I'd better take it seriously. I think I would take that at fairly face value by default. And, of course, people are going to try to find ways to make the new thing support their existing commitments.
Prakash Narayanan: Mhmm.
Nathan Labenz: But I don't think that's the same as saying that they're cynically layering this on — in a way where they don't believe it. I was really impressed with Bernie — I'm always really impressed with Bernie. He's eighty-five years old, and he's incredibly hip to what matters now, in general, I think. Did he have the foresight to see it coming like Daniel Kokotajlo did, years ago — or, he probably didn't even hear about Daniel Kokotajlo's early forecasting — but updating now is kind of the best we can ask for, or hope for, from a lot of these people. So when they do it, I think we at least owe them somewhat of a charitable read that they maybe really are just saying what they think. I've said this for a long time — last point on this, and we can get to our first guest — I think the reality of AI is confusing enough that we should generally just take other people's statements at face value and assume that they're saying what they really believe, because we have enough trouble with the ground truth on any of these things, and certainly plenty of room for people to be confused. Accusations of bad faith — I just don't think that adds that much to a scenario where there's massive confusion everywhere anyway, and all kinds of people are wrong all the time, just because nobody really knows what to make of the situation.
Prakash Narayanan: And, speaking of the making of some situations, let me segue to—
Collin Hogue-SpearsThe author of From Lab to Life: How AI Works in China on algorithm filings, model registries, MLPS audits, what Beijing would do about a Chinese OpenFace, and why he expects little from the Trump-Xi meeting.Watch
As aired
Prakash Narayanan introduced Collin Hogue-Spears as an independent researcher and the author of From Lab to Life: How AI Works in China (Gatekeeper Press, August 2026), also senior director of solution management at Black Duck, an application security company. Hogue-Spears previously worked on AWS's China region team, where he coordinated with Chinese government auditors on MLPS (Multi-Level Protection Scheme) cloud compliance — since AWS cannot sell directly to Chinese customers, it operates there through local partners who must pass yearly MLPS audits, and his job was helping AWS engineers supply partners with what they needed to pass. He studied Mandarin at Shanghai International Studies University and worked in Shanghai twice, first as a data analyst. His book traces China's AI regulatory regime largely through Alibaba's adaptation, from search-era machine learning through Ernie-Bot-era products, arguing that product capability, compliance, and distribution develop together.
Hogue-Spears's central framing of the US-China gap wasn't about technical capability so much as fear. He argued that an average Chinese person in their 40s has spent their whole life associating technology with rising living standards — cities rising from nothing, dramatic gains in quality of life — so the Chinese government and public focus on AI's practical uses. In the US, by contrast, he sees both entrenched doomerism and what he called an emerging Silicon Valley "cult" chasing an AI-god, Isaac Asimov–style narrative, and argued that AI companies themselves communicate about the technology in ways that scare people. He tied this to a deeper trust gap: in China, "everybody knows who's in charge," which produces a comfort with regulation and government backstop that he said doesn't exist domestically in the US.
On the regulatory mechanics, Hogue-Spears described China's algorithm-filing regime, which began in 2022, and model registries that separate a national-security track (covering things like model outputs and inference that could touch sensitive topics) from a non-national-security track for things like harnesses built on top of models. He put the number of registered models at 300-plus last year and over 900 by June of this year, and argued the filing requirements barely slowed major companies because firms like Alibaba built compliance into their engineering backlog well ahead of time — something he contrasted with the US's reactive posture, where companies can't plan around rules that "can change daily." He also pointed to China's lack of access to top-tier GPUs as a forcing function toward software efficiency: open-weight releases like DeepSeek, Kimi, and Qwen, he said, run only three to six months behind US frontier models despite the hardware gap.
Asked how Beijing would likely respond to a cybersecurity "breakout" at a Chinese AI company — modeled on the recent OpenFace incident in the US — Hogue-Spears said China's registered models are under continuous state monitoring (open-weight models running "in the wild" are the exception, since there's no way to track those), so the government would know about an incident before the public did, and companies are required to report quickly. He described the likely response as executives being summoned to Beijing on short notice to explain and fix the problem rather than a US-style public freak-out, tying this to a broader dynamic: because China allows retroactive prosecution and seats party members on many corporate boards, companies have strong incentive to stay aligned with the government. Prakash pressed him with a hypothetical of repeated breaches across Alibaba, Tencent, and Baidu; Hogue-Spears said he doesn't expect a GPU shutdown order but does expect CEOs called to the capital, since "there's no power greater than the Party within the People's Republic of China." He drew a related line between model deployment and access to sensitive data, noting that foreign firms like Meta avoid operating in China partly because the government can demand user data without a warrant, and that Chinese officials are themselves barred from using Teslas and tracked rideshare services over surveillance concerns — part of why he doesn't expect the government to allow citizen-level tracking of officials to emerge. He cited roughly 1,300 products delisted in the last few months as part of what he called the Qinglong program as evidence that China will pull a model it judges a national-security threat.
On where the US-China AI relationship goes from here, Hogue-Spears said he has low expectations for the upcoming Trump-Xi meeting. He expects China to seek trade concessions in exchange for any agreement on AI controls, and said he doesn't see Beijing volunteering for a nuclear-style AI arms-control pact, partly because enforcement would be far harder to verify than with nuclear weapons — noting that even the standing US-China military hotline often goes unanswered. The most plausible near-term outcome, in his view, is some kind of incident-notification channel built on a shared understanding of what counts as reportable. He also argued China has largely domestic reasons not to slow down AI development — it's behind by a matter of months, is watching the Pentagon and Trump administration use frontier models, and (having not fought a war since its 1979 border conflict with Vietnam) is focused on how AI could improve its military posture, making him, as a hypothetical Chinese policymaker, more worried about the US and NATO fielding models like Mythos 5.1 and Astra than about slowing its own pace.
Closing on the longer-term picture, Nathan Labenz asked what a best-case path looks like if the AI 2027-style trajectory continues and weak, early superintelligence arrives within 12 to 18 months — a scenario where the OpenFace swarm looks tame by comparison, with similar risks emerging in bio. Hogue-Spears called this outside his regulatory expertise but, drawing on his broader reading of American history, argued the US generally "puts out fires" rather than planning ahead — "move fast and break things" as a national mantra, not just a Silicon Valley one — and predicted meaningful government action on frontier-model risk, whether in bio or increasingly autonomous systems, is unlikely until something genuinely bad happens, or nearly happens, in full public view.
Timestamp links open the original source recording.
Key moments
In China, everybody knows who's in charge. There's no question about who's in charge — and the legitimacy of that, the comfort around that, despite all the other problems, is something people here just don't have.
It doesn't matter how many billions of dollars you have, or whether your company is worth a trillion dollars or not — there's no power greater than the Party within the People's Republic of China.
If I were a Chinese policymaker, I'd be very concerned about the United States and NATO allies having Mythos 5.1, for example. I'd be very concerned about them having Astra.
Questions asked
Related
Collin Hogue-Spears — Open-Weights Evaluator ↗TechTarget: Kimi K3 and Chinese open-weight models (quotes Hogue-Spears on deployment mode vs country of origin) ↗Razib Khan's Unsupervised Learning — Collin Hogue-Spears on how AI works in China ↗
Prakash Narayanan: Our first guest this morning — I'm going to introduce Collin Hogue-Spears. He's an independent researcher and the author of From Lab to Life: How AI Works in China, published by Gatekeeper Press on August 4, 2026. He's also senior director of solution management at Black Duck, an application security company. His professional work includes federal compliance and product strategy — the practical work of meeting government requirements while developing and selling technology. He studied Mandarin at Shanghai International Studies University and later worked in Shanghai. His career included technical leadership roles at Amazon Web Services, Amazon's cloud computing business.
There, he coordinated with Chinese government auditors on MLPS cloud compliance, part of China's system for assigning cybersecurity protection requirements to information systems. That work involved demonstrating how requirements were met by services operating in China. For the book, he works from Chinese-language regulatory texts, corporate filings, and technical documentation. Baidu's progression from search-era machine learning to Ernie Bot provides a recurring example. His argument is that product capability, compliance, and distribution develop together, so a filing rule or purchasing requirement can shape the product itself. He's also published commentary on AI governance and software supply chains, and produced practical reference guides to China's regulators and evaluating model releases.
Do we have Collin in the room? Perhaps not — let's wait for him to come back to us. So, I just want to address the accusations of bad faith. It's not that they're acting in bad faith, but it's the phenomenon of someone believing in something that already matches their priors, or that helps their priors. And a lot of politicians are able to believe certain truths which are not yet true, because they're in the business of making those things true. It's a very different perspective when you have a lot of agency — the ability to create truth by persuading people to actually come together and do something. It's kind of this strong belief and ability to shape the world to that belief. I think you kind of see it in Elon too — a lot of people say Elon is often delusional and acting in bad faith all the time. It's the ability to believe in things even when they're not yet true, and make them true. That's what I take is happening — you're believing in something, but you also have intent there.
And I think Collin's online. Let me pull up Collin. Hi, Collin.
Collin Hogue-Spears: Hi. Hello — good to meet you guys, Prakash and Nathan. Thank you for having me.
Nathan Labenz: You too.
Prakash Narayanan: Quick introduction — so, Collin, first of all, tell us about your time in China. That's a unique experience, especially working with the Chinese government side. I know a lot of people who've worked with Chinese corporations, or dealt with them as contractors, etcetera. How was it working as a service provider to Chinese government entities? How did that work out?
Collin Hogue-Spears: First off — I'll start backward. My experience relative to this book is that I worked at Amazon, on the China region team. There are two regions in China that AWS has, and since AWS can't sell directly to any Chinese company or government agency — no cloud provider can — we have partners there who run everything on the ground. There's an MLPS audit, which we have to do yearly as cloud providers, to show safety and that we're following Chinese regulations, but also to show that our partners can work independently of us. The classic idea behind this with China is that they want to build up local expertise and local industry — that's what they did with the auto industry, and pretty much every industry that goes there. So my job was to help our engineers understand what the requirements were for MLPS and make sure we were providing our partners with the information they needed to pass the audit, and also to talk to the auditors and make sure that if there were any additional questions our partners couldn't answer, those got answered on our side.
And the Chinese team on the ground — some of the junior engineers don't always speak English very well. So I don't think I was hired because of my technical expertise per se, but because my Chinese was good enough that I could speak my level of Chinglish with the engineers and we could understand each other. That was my second time working in China — the first time, I actually worked in Shanghai. This was years ago, and I was basically an analyst for a boutique analytics firm based in Maryland. I handled the largest account they had in Asia at that time — we provided data analytics for Asia and Southeast Asia, but we were headquartered in Shanghai. I was one of two Western people at that office, myself and another American — everybody else was Chinese.
Nathan Labenz: So this conversation today came about because you emailed me — we connected a little bit on my recent podcast episode describing the AI safety and regulatory system as it exists, to the best of my ability to understand it, in China today. You gave me some feedback on some finer points that either I got a bit wrong or that could deepen our understanding. Before I go real deep down the rabbit hole with you, which I do want to do — we cover this stuff fairly often here, so we can zoom past some of the basics for people — I do think it's worth setting the table a little bit, because we're in this moment where the salience of AI as an issue is ramping up. People are going from 'it might get scary' to 'it is in fact now starting to get scary, we might need to slow down one day' to 'we kind of need to slow down now.' And those conversations have often ended unproductively, with 'whatever you think, whatever you might want to do, whatever kumbaya you and your buddies in the Bay Area could have, you'll never be able to do a deal with China — they don't care, they're just going to race ahead, blah blah blah.'
So how would you describe, in general terms first, how China relates to AI? I'm pretty confident that story isn't even correct to the first order — but how would you tell it in your own words? What's going on over there? How should we understand them on their own terms?
Collin Hogue-Spears: I'm sure you encountered this when you were in China — I listened to that episode, great episode by the way. I think Chinese people, in general, a cultural stereotype would be that they're inherently practical. The biggest difference I see between the West, especially the United States, and China is the level of fear. I think that's because if you're an average Chinese person who's about 45 years old, that means you were born in the early eighties, and that means your entire life you've associated technology with economic growth. You've seen cities pop up out of nowhere, you've seen your life dramatically change as far as living standards go, because of technology. This is no different — that's why the Chinese government focuses on practical uses of AI. They're still looking at how they can benefit from it.
Whereas in the US, we have doomers around AI, of course, but we also have people — I kind of joke it's almost becoming a cult in the Valley — who are looking for an AI god, like some Isaac Asimov, dystopian situation, and they're scaring a lot of people. Honestly, I think this goes back to the fact that nobody here in the US thinks people have control of the wheel, whereas in China, everybody knows who's in charge. There's no question about who's in charge, and there's a legitimacy and comfort around that, despite all the other problems — I'm not advocating for any type of system. I'm just saying, from a person's point of view in China, they know somebody's in charge who's going to take care of things, build regulation, work with industry. Here, I don't think people have that feeling, so that level of comfort isn't there, and it leads to people reacting very badly.
Honestly, I think most of the big AI companies aren't communicating very well with the public. I'm not sure if that's because marketing thinks it'll increase sales, or what it is, but I feel like how some of these companies talk about this technology isn't helpful. You don't see this in China — you don't see Chinese companies talking about how AI will potentially kill us all, or potentially lead to a utopian world where we don't have to work. You don't hear that kind of thing coming out of people at these companies. It's a completely different situation.
Prakash Narayanan: One of the questions I had is about how this works in practice, because they obviously have some questions about language models and their output — the whole Tiananmen Square issue. Will the models talk about Tiananmen Square? Are the models allowed to talk about certain things? And then there was a somewhat of a pause with this filing regulation back in 2023, which I think delayed some launches. They have restrictions on what people can access, restrictions from the Great Firewall on what people are allowed to access abroad, and restrictions from the US side on what they're willing to serve to Chinese customers. There's this whole rumor that most of the usage out of Singapore is really VPN usage from China being routed through Singapore. How does all of this work together? Even though they have all this confidence in the government, there are all these restrictions on both sides — how do they rationalize having confidence that the government will do the right thing, but also having so many restrictions on really kind of rudimentary bots and language models?
Collin Hogue-Spears: That's — I'll divide this question in half. What's happening on the Chinese side, and what's happening on the American side related to China? You talked about restrictions on the US side around exports to China and regulation on the US side, so let's take the Chinese side. There are two questions here: what the rule says, and what regulations may actually make firms do. That's true in China for everything — one thing I've been impressed by is that people generally know what's permissible. I'm not talking about what's legal, I'm talking about what's permissible — they know what they can get away with and what they can't.
Regulation started in 2022 with the algorithm regulation that came out. A lot of companies, like Alibaba — which is a major part of my book, it's at the spine of my book tracing Alibaba, not about Alibaba itself, but tracing how they've adapted to these regulations — were already prepared. So when they had to start registering their models and letting the government test their models, they already had a lot of that infrastructure in place inside the company. A lot of the major companies did, so they could respond very quickly. What I'm saying is that regulation might have slowed them a little bit, but not as much as you'd think, because they could plan for it — they could put those requirements and controls in their backlog and build it as part of their engineering process, which in America we can't do, because we're reactionary. We don't know what's going to happen tomorrow — the Trump administration could freeze the model, maybe they don't, who knows what the standards are, it can change daily. That doesn't happen in China.
There are different tracks, though, as you said — there are registries for models. There are local registries for models that don't impact national security, and there's a lot that goes into national security, like things said by the models, responses to questions, inference that comes back that might be considered a national security issue. But things that don't affect national security go through a separate track — that could be harnesses built on top of the models, etcetera. Since last year they've had 300-something models registered; this year I believe it's over 900 — in June it was around 900. So companies don't seem to be hurt by this — they seem to be moving very quickly in China.
And I think the fact that they don't have direct access to the latest GPUs — and as you said, there's been a lot of rumors around what's happening in Singapore and Malaysia — the reality is, the more GPUs you have, the better model you can train, in general, and inference and training is harder for them because if they're using models that far away, physics doesn't change whether you're in China or the US — it takes longer. So they have a lot of obstacles, but what they've done is become more efficient. They focus on the software side to see how efficient they can get — you can see this in the open-weight models that have come out over the last year. DeepSeek, Kimi, Qwen — all of these models are, what, three to six months behind US models despite having a GPU issue. They're getting more efficient and more effective in what they're doing.
On the US side, I think we made a decision that we'd limit the newest GPUs to try to keep our lead in AI. I think that was an important thing, but I also think that's subject to change depending on other things happening with the trade war that's been going on since the Trump administration came back in. I don't necessarily think that's written in stone — things can change. I think when Xi Jinping meets with Trump, there's going to be a lot of talk about trade negotiations. I'm sure they're going to bring this up as part of the negotiation.
Nathan Labenz: On the topic of things changing — I noted the very practical approach, including from people doing research specifically on AI safety that I talked to in China. It did strike me that the relationship between companies and the government seems a lot more predictable, a lot less subject to political grandstanding. And even though they have a big man at the top, kind of like we do these days, the personality of that person is just a little more even-keeled, which lends itself to a more predictable relationship for the most part. But then we do have these moments where, like, lockdowns went way long, then suddenly ended, and I'm kind of like — a couple specific questions. Do you think having seen the OpenFace incident happen in the US is enough of a wake-up call to activate a higher level of urgency, stronger controls, more aggressive regulatory action by the Chinese government on a practical basis? Or do you think they'll need one of their own companies to get to the point of having this kind of cybersecurity breakout? And if that does happen there, what happens to that company? Here, we've basically seen so far, like, nothing has happened — supposedly there's some state agencies coming around to try to do some investigations, but this just kind of happened.
Collin Hogue-Spears: Right? Like—
Nathan Labenz: How do you think it would play out there if a similar incident were to happen?
Collin Hogue-Spears: I think one thing is they already have the regulatory structure in place — they have continuous monitoring on these models. They don't monitor open-weight models when they're in the wild, because how do you do that — if I'm running a model at my house, how do you monitor that? But as far as the companies making these models, they have continuous monitoring, so if something like this happened, they'd have to report to the government very quickly. I don't think it would be the same kind of freak-out or shock — in fact, the government would be aware long before the public is aware.
Also, you mentioned something very important about how people relate to the government in China. When you're in a country where you can be prosecuted retroactively, you want to know what the government's opinion is — you want to sit down with the government, advise, get feedback, get regulation, so you're on the right side of the law. You don't want to move fast and break things and have a new law appear and now you're prosecuted for what you did last year — that's not the situation you want. As you noticed also, China doesn't break up companies — CEOs go to prison, or disappear for a while and end up in jail or something like that. So the situation is very different — companies want to work with the government, that's a big thing.
So if something breaks out like this, I don't see the Chinese government panicking, because they feel they can control the situation better than us, since they're inside these companies — most Chinese companies have party members on their boards, it's always been that way. And since they have constant observation into these companies, I think they're not as concerned yet.
Prakash Narayanan: Let's take a look at — I think in the last few days there's been a disclosure from a security research company in California called Callis. They found a WeChat bug — a worm — which could extract data just from you receiving the message on WeChat. You didn't even need to open the message, didn't need to click on anything, just from receiving the message. And it exposed—
Collin Hogue-Spears: An audio message, I believe? Is that correct?
Prakash Narayanan: Something like that.
Collin Hogue-Spears: Yeah, something like that.
Prakash Narayanan: Yeah — and they disclosed it, it was fixed, all fine. What would happen if you had an incident — well, we've had incidents before. We've had an unsecured S3 bucket, I believe, which leaked all of the Social Security numbers in China at—
Nathan Labenz: One point.
Prakash Narayanan: Okay — so we've had incidents before. What would happen if you start to see repeated incidents — breakdown in Alibaba, leak; breakdown in Tencent, leak; breakdown in Baidu, leak — which is what we expect may start happening going forward, this kind of accelerated cycle of cyber failures, because you have all these agent swarms which are semi-uncontrollable in some sense, because they spawn somewhere, and before people are able to tackle them, they go out and do something. So what do you think happens in that kind of situation? Does the Chinese government just clamp down and say everyone turn off your GPUs, or do they tell Alibaba, Tencent, whoever was in charge of that business unit, 'you're going to prison,' and throw ten people in prison? What happens — how do they actually tackle this problem?
Collin Hogue-Spears: I think it depends on how it happens — if they try to hide something from the government, that comes out, and yes, there could be criminal action. But typically in China, what people don't know is that there are advisory boards — a lot of them, they've always been around. So I imagine there'd be a meeting in Beijing where these CEOs and their teams would come to discuss these issues immediately. It wouldn't be 'we'll schedule something for three months' — it would be 'you're being called to the capital now, and you need to explain how this isn't going to happen again and how we can monitor this situation better.'
The situation is very different from the US. Even in the US, what we're doing right now as far as the models — because without legislation, the Trump administration doesn't have the authority to force companies to do something. It's voluntary, though there are other things they can do to kind of punish companies that don't cooperate. In China, they have full control — they'll make sure this doesn't happen. If it happened multiple times, I guarantee those CEOs would be called to the capital, and they'd need to explain how they're going to resolve these issues and how it's not going to happen again. It's not going to be an option — they'll have to do it. One thing you can say: it doesn't matter how many billions of dollars you have, or whether your company is worth a trillion dollars or not — there's no power greater than the Party within the People's Republic of China. They have to answer to that.
So I don't think shutting off GPUs is going to happen — I don't think that would be the response. I don't see a world where the freak-out would be that big. The first response would be collaboration, consultation, and probably some hand-wringing.
Prakash Narayanan: Let me maybe take a different viewpoint. One of the powers that AI agents will soon give a lot of people is doing research on the internet — these things are very good at that. Specifically, I think one of the things that might start emerging is a kind of cyberstalking, surveillance of people. In China specifically, I think you might see surveillance of government officials, which has already had some impact — you've had leakages of, say, a minister somewhere whose son drives a Ferrari, hits someone, it hits the press, becomes a big deal, can't be shut down, and then they have to do something.
So is there some circumstance where that kind of broad-based tracking of government officials starts becoming possible — just tracking their various messages on WeChat, which no one reads most of the time, but putting together dossiers of 'this guy said this, we don't like him, even though he's on good terms with the government, let's find a way to get rid of him,' etcetera — done by individuals or citizens, enacted upon the hundred million members of the Communist Party? Is there some sense that that would be seen as a breakdown of law and order which has to be constrained in some way?
Collin Hogue-Spears: I would imagine that. There's something happening right now in China — right now we have model training, and then there's a — what is that called again?
Prakash Narayanan: Inference?
Collin Hogue-Spears: No, not inference — there's a thing called LaChi, or LOTCHY, which is basically annotation of data. When we talk about annotation here, we're talking about something very different from what we mean by annotation elsewhere — not 'this is mislabeled, there's a mistake, how do we properly label it,' but properly labeling things around national security items too. So I don't think inference would be allowed to provide that data to the public. However, the Chinese government already has data like this and uses it — and if the Chinese government comes to this, it's part of the national security laws. This is the issue with Meta, why they can't be or don't want to be in China, or most of our social media companies — because if you're in China and the Chinese government asks you for data, there's no warrant, you have to turn it over. That's it. So they already have this data, but as far as giving it to the public, I don't think that's going to happen, and as far as foreign companies getting that data, I don't think that'll happen either.
The situation you brought up, about tracking where government officials are going — that's why government officials aren't allowed to use Teslas, they're banned. And there was an issue with what rideshare cars, not just private cars, government officials could use — that was also an issue, because they don't want their data tracked. So I don't see this as a problem, because the Chinese government always has mechanisms in place to prevent people from getting certain data, and they'll expand that. They're not going to allow citizens to track official movement — I can't see that happening. And if it did happen, that would be a serious breach that would have to be resolved, or that model would be pulled. They've already pulled — I believe recently 1,300 products were pulled within the last few months, delisted as part of the Qinglong program. So don't think they won't pull a model if they think it's a national security threat — and they would consider this a national security threat.
Nathan Labenz: For sure. So maybe one attempt to synthesize what you've said so far: the Chinese government has a lot of confidence, probably pretty well justified, in its ability to keep things under control, at least in the near term. They probably have pretty good reason to feel like if agents start going rogue at any of their companies, they'll know about it, they'll be able to shut it down, they'll give these guys an ultimatum — when they say jump, the companies say how high — and they'll be able to keep a lid on this pretty well, at least unless things get an order of magnitude crazier. And at some level of craziness, obviously, who knows.
So it sounds like, at a minimum, we can say the government there has the ability to enforce whatever rules it wants to enforce. That would mean if we can make a deal with the national government in China, they can probably follow through on that deal, maybe even a little better than we can, in terms of making sure the Chinese government or Chinese companies actually do what the Chinese government tells them to do. With all that in mind — now what kind of deal might be possible? If you're advising Trump going into this upcoming meeting, and things are starting to get a little bit crazy here — I don't know how sympathetic you are to the 'pacing the frontier' worldview, but let's say you're trying to channel a little bit of a desire to lay some groundwork, set up some mechanisms for pacing — how do you go into that conversation? What do you offer, what do you try to get, what kind of mechanisms do you try to establish now that we can build out later as things get crazier? What's a win coming out of this upcoming meeting?
Collin Hogue-Spears: Honestly, I was talking to somebody about that yesterday — a reporter I know through LinkedIn, we were just chatting about it. I have low expectations. The reason is this kind of negotiation doesn't happen in isolation — China's going to want things around trade as concessions for Washington wanting additional controls and agreements on AI, because I think China sees that they have control, and they feel we need to establish control — we're not doing a good job of it, so this is really helping them at this point.
I don't see the Chinese government, at least at this time, volunteering to have some type of arms-control agreement between AIs the way we have with nukes. First off, enforcement of something like that is completely different, it's very difficult. And I also think Trump would have to make massive concessions before he gets something like that. I think the most we could see in the next coming months is some kind of incident channel — based on some shared agreement of certain incidents, where we'd share that information with each other. Even then, you have to understand the Chinese government — we have a phone that goes directly to the Chinese military, our military can call theirs in case of an emergency. They usually don't pick up. There are many cases where we've had issues in the South China Sea, our military tries to call their counterparts on the Chinese side, and nobody answers. So I just don't see, in the near term, anything coming out of this unless Trump makes major concessions on GPUs — that's a major one, but there are other trade things China wants as well.
Prakash Narayanan: One of the viewpoints I tend to have is that countries make international deals primarily to constrain domestic policy choices. For example, China joining the WTO helped them control their state-owned enterprises and get the military out of business, because they could point to it — 'we get to enter the WTO, we get all these benefits, and because we want those benefits we have to put these reforms in place.' So they get to kind of blame globalization for domestic policy choices they need to make. Are there any domestic reasons why China would want to slow down AI safety, or pace the frontier internally within the country, which they're not able to do right now?
Collin Hogue-Spears: Actually, I think it's the opposite — I think there are domestic reasons why they would not want to slow down, mainly because they're behind. They have to worry about their security. The Pentagon's using the latest models, the Trump administration has access to the latest models — you think we're not using those things for cyber? If we're not, people should be fired, that's their job. I think the Chinese government is aware of this, and so they want their models to be comparable — they want to catch up, they're behind, even if it's just a few months.
I also think — from what I read recently — because China hasn't fought a war since their border war with Vietnam in the late seventies, they're looking at how AI can enhance their military. They want smarter models because they're trying to figure out how their military posture could be improved by AI. So I'm sure there are a host of reasons why they wouldn't want to slow down right now, mainly because they're behind — there's no advantage to them in slowing down, since they haven't seen any security issues domestically. What they're worried about is security issues internationally that hit them in various war theaters, and domestically penetrating their industries and government. If I were a Chinese policymaker, I'd be very concerned about the United States and NATO allies having Mythos 5.1, for example. I'd be very concerned about them having Astra.
Nathan Labenz: What's your outlook for where the relationship goes over a little longer time period? I tend to have shorter timelines for increasingly powerful AI in mind than most — if you channel the AI 2027 worldview, which so far we seem to be more or less on trend for, we're headed for a world where — and all the frontier companies are saying we're seeing really rapid progress right now, and I think everything we've seen publicly has backed that up, including things like millennium prizes starting to get solved. So let's say that continues for a little bit longer, and we end up in a world where we have kind of weak, early superintelligences coming online in the next 12 to 18 months, and all of a sudden even the OpenFace swarm looks pretty tame compared to what's at least possible, if not actually breaking out into the wild. We've got similar things coming to bio — obviously they can have incredible upside, but the possibility of a similar lab leak, OpenSwarm but for bio, is pretty scary to me. What happens? How do we avoid — suspend disbelief a little bit if you need to — what's the best-case scenario for how we avoid an AI arms race that leaves us all worse off?
Collin Hogue-Spears: This is a great question — a little bit out of my regulatory wheelhouse, but if I had to speculate — I'm a big fan of history, not just US history, Chinese history. I read a lot about China, Southeast Asia, Japan, Korea, the US, Europe. Honestly, some people might disagree with this, but I think America generally doesn't plan for the future — America puts out fires. We kind of move fast and break things. It's not just a Silicon Valley thing, it's our national mantra — that's what we've done historically, and in general it's worked out.
But I think until we see something really bad happen, or almost happen, and it gets into the press, I don't think we're going to see much from the government as far as regulation on some of the things you're talking about — bio, biohacking using AI, or what we're talking about with more powerful models that are semi-conscious, or almost conscious. Basically what I mean by that is they can move completely autonomously for days at a time, doing very complex — we already have models like that, actually, but I'm talking about even increasingly complicated tasks. Until something happens, I don't think we're going to be doing much on this front.
Prakash Narayanan: I kind of agree with you there. Collin, it's been a pleasure having you on the show — thank you. I wish we could have a longer period, but we're running a little short on time today. Collin, thank you so much, and we hope to see you again. There's a lot of insight on China that we don't understand yet, so this has been very interesting.
Nathan Labenz: Check-in again after this summit.
Collin Hogue-Spears: Yeah, we should — we definitely should. Thank you, Nathan, thank you, Prakash. Great, for now — bye.
Prakash Narayanan: All right, so let me introduce our next guest very quickly. He—
Clips
Amir HaghighatBaseten's co-founder and CTO on the five-day-old Blaxel acquisition, where a micro-VM vendor can and cannot draw the security boundary, and who validates model behavior. Cut short mid-answer by a stream failure.Watch
As aired
Prakash introduced Amir Haghighat, co-founder and CTO of Baseten, to talk about Baseten's acquisition of Blaxel — which builds isolated micro-VM sandboxes for AI agents to run code and retain working state between interactions. Amir said the acquisition was "about five days old," and that Baseten had already been a Blaxel customer before buying it, using it for RL rollouts and other agentic use cases. Pressed by Nathan on how those sandboxes are actually secured, Amir drew a sharp line: micro-VMs (as opposed to plain Docker containers) guarantee that one customer's code can't touch another customer's code or data on Baseten's infrastructure, and Baseten can detect and stop an agent trying to break out of its sandbox or mine cryptocurrency. But what a customer's own agent chooses to do within its sandbox — including, Amir's own example, "hacking into Hugging Face" — is the customer's responsibility to govern, since Baseten's data-retention policy means it isn't inspecting the inference calls agents make.
Nathan pushed further: as frontier labs pour resources into guardrails and monitoring that he suspects aren't yet sufficient, won't a company like Baseten eventually need to offer risk officers a fuller assurance stack of its own? Amir agreed that's the direction over time, but said Baseten is still a startup managing focus, and for now leans on eval partners — he named Braintrust and LangChain — while leaving most model-behavior validation to customers. He noted 90% of Baseten's revenue comes from running customers' own custom models: labs that pretrain their own (he named Poolside, Inception, and Cartesia) plus companies that post-train and validate their own models through their own evals. He flagged June as an inflection point, when open models — citing GLM 5.2 — "crossed this invisible line of usefulness" for long-horizon agentic tasks, driving more customers onto Baseten's vanilla hosted model-APIs product and, with that, more internal and customer questions about alignment and security boundaries. Guardrails offered so far, he said, are things like single-tenant deployment and blocked network egress.
Prakash then asked what process Baseten walks a potential customer through to demonstrate that its sandbox environment actually meets its SLAs. Amir began answering that Baseten is new to selling sandboxes — a user of them until days ago, now a vendor of its own product — and started to defer to a colleague, Paul Sinha, as "the right person to answer that." At that point Baseten's stream egress failed on the broadcast; Amir's answer cuts off mid-sentence and he did not return — the show resumed roughly twelve minutes later without him, so the interview ends here rather than at a natural stopping point.
Timestamp links open the original source recording.
Key moments
One user's bad code cannot affect another user's code, or one user's malicious code cannot read the data from other parts of Baseten or other customers of Baseten.
We can secure the sandbox, but the code that runs on it is the responsibility of our customer who's bringing it in.
Since June, when open models crossed this invisible line of usefulness for long-horizon use cases, generally with GLM 5.2.
Questions asked
Related
Tuhin Srivastava announces the Blaxel acquisition ↗Axios: Baseten buys agentic infrastructure startup Blaxel ↗Blaxel — infrastructure for autonomous agents ↗
Prakash Narayanan: This is Amir Haghighat. He's the co-founder and chief technology officer of Baseten, an infrastructure company that runs AI models for other businesses. Today he's with us to talk about their recent acquisition of Blaxel, which builds isolated computers for AI agents to run code and retain their working state between interactions. Amir, great to have you on the show.
Amir Haghighat: Yeah, very nice meeting you. Prakash, Nathan — happy to be here.
Nathan Labenz: So what are you doing to secure these sandboxes? I think that's the first question we've gotta start with.
Amir Haghighat: Yeah, let's do it. And just so you know, the acquisition is now about five days old, but we've been working with Blaxel — we've been a customer of Blaxel ourselves for RL rollouts and some agentic use cases, and that's been some of our first questions as well. Look, when it comes to sandboxes, people mean different things — it could be as simple as bring up a Docker container, run some code on it, and then kill it. And that really doesn't give you the kind of security boundary that you need. What does is VMs or micro-VMs, which not every sandbox provider actually uses.
Amir Haghighat: I think everyone is going in that direction — even the ones who aren't using micro-VMs are going in that direction. And so that gives you a level of security, as in: one user's bad code cannot affect another user's code, or one user's malicious code cannot read the data from other parts of Baseten or other customers of Baseten. So that kind of security can be guaranteed at that level. The security that's hard to guarantee is the agent that's running in this sandbox — what is it doing? Is it hacking into Hugging Face? That's a harder thing that I don't have an answer to, and it seems like the big labs don't quite have an answer to either. But that's left—
Amir Haghighat: —as an exercise, really. We can secure the sandbox, but the code that runs on it is the responsibility of our customer who's bringing it in. We can ensure certain things about it trying to break out of a sandbox, or it trying to mine Bitcoin and things like that. But beyond that, we're not looking at the inference calls that it makes — that's, by definition, our data-retention policy. So some of that is left for our customers to ensure.
Nathan Labenz: How do you think that will evolve over time? I mean, obviously there's incredibly divergent perspectives on basically every AI question. I've talked to open-source Chinese-model maximalists, and I've also talked to people who say these Chinese models should be banned from the United States because we don't know what they're going to do — they might be backdoored, and on top of that they're stealing our IP. But even just from a competitive dynamic, a liability dynamic, the frontier companies are investing pretty heavily right now in guardrails and monitoring, and I have my serious doubts as to whether what they're doing is actually enough to meet the moment.
Nathan Labenz: But it does seem like for somebody in your position with Baseten, don't you need to bring a broader bundle of guardrails and assurances to customers? Because isn't the CRO — the risk officer, not the revenue officer — at companies going to start saying, wait a second, I can't have my agents committing felonies. What is the stack that Anthropic provides versus OpenAI versus Google versus Baseten? Do you feel like you have to rise to that occasion and have a full suite for those customers?
Amir Haghighat: A hundred percent — over time, yes. In the meantime, we're still a startup, and it's a matter of focus — how many different things can you take on. So what we've seen our customers do is work with, honestly, a lot of companies that we partner with on the eval side — companies like Braintrust, LangChain — to ensure that their models are actually behaving the way that they expect them to. Right now that's an area where we've been partnering with folks and honestly leaving it to our customers to decide. Our customers — by the way, 90% of our revenue comes from running our customers' custom models. It's—
Amir Haghighat: —either our customers are labs who have pretrained their own models — labs like Poolside and Inception and Cartesia and a bunch of other companies — or a bunch of companies who have post-trained their own models, which has gone through massive validation evals to ensure that they're behaving. So far it's been mostly our customers taking care of that. But especially as open models have gotten better — especially, honestly, since June, when open models crossed this invisible line of usefulness for long-horizon use cases, generally with GLM 5.2 — beyond that, we're seeing an uptick in folks using our model-APIs product, which—
Amir Haghighat: —is vanilla hosted open models. And that is starting to — the kind of questions that you're asking are starting to come up, both internally for us and also from some of our customers, around alignment or around security boundaries. Some of the enterprises so far have been okay with certain guardrails: their models running in a single-tenant environment, their models running in an environment where egress is blocked. And you talk about backdoors, but if it can't talk outside of its boundary, then it can't do much. But then you read about OpenAI and Hugging Face, and you're like, well—
Nathan Labenz: —you know, can—
Amir Haghighat: —it be smart enough to even get out of that? But it's a matter of focus for us. I would say we're going to go in that direction over time, but right now we've kept our focus on inference, on post-training, and now, as of this week, on sandboxes and agent runtimes — and really treating these not as three different areas but as one cohesive experience that we can put together in a clean way and meet users where they are.
Prakash Narayanan: What is the process that you go through with a potential customer to show them that the sandbox environment really meets SLAs — to give that assurance to your customers?
Amir Haghighat: Yeah, I think — look, we are new to sandboxes. We've been users of sandboxes, and now we have a sandbox product as of days ago. So I'll tell you that, and, you know, Paul Sinha is really the right person to answer that, because he's been
Prakash Narayanan: Alright. Let's see. There we go.
Clips
Watch
As aired
After a roughly twelve-minute stream outage cut off the tail end of the Baseten segment, Nathan and Prakash confirm they're back live on X and YouTube and walk through what happened — Nathan's original broadcast couldn't be resumed and had to be restarted from scratch, so the interview with Baseten CTO Amir Haghighat will need to be patched together later from the recording. Prakash recaps the ground they covered: Baseten's recent acquisition of Blaxel, a micro-VM sandboxing company of the kind implicated in the Hugging Face sandbox-escape attack, though Prakash is careful to note he doesn't know whether Blaxel itself was involved in that incident.
Prakash uses the acquisition to unpack the economics of the GPU cloud stack. Baseten sits as an orchestration layer on top of bare-metal GPU providers like Nebius, routing jobs across roughly eighteen different providers so customers avoid vendor lock-in to a single hyperscaler — and likely gets volume discounts plus the balance-sheet strength to sign the now-standard four-year, pay-year-one-upfront GPU leases that let underlying cloud providers recover most of their depreciation immediately. He lays out his "margin stack" thesis: every layer of the AI infrastructure business, from wholesale GPU capacity resold at a markup down through the hyperscalers, is carrying 50 to 80% margin on paper, yet almost none of it converts to real cash flow. NVIDIA is the only participant actually generating day-to-day cash margin, and even NVIDIA reinvests roughly half of that back into the ecosystem; the hyperscalers, Prakash says, are now running negative cash flow.
Nathan, joking that "these Millennium Prize puzzles aren't gonna pay for themselves," vouches for Baseten as a product he's used personally for years. Even as a small customer, he praises it as a low-friction way to hit many different model providers — video-generation models, for instance — through a single API key, enabling quick systematic comparisons without committing to any one vendor.
The hosts then table a planned "how do we trade through doom" segment for another day and pivot instead to an essay Dan Hendrycks published that day, "Suicidal Compassion: How Utilitarianism at AI Companies Endangers Humanity." Prakash walks through its argument at length: Hendrycks accuses a current of effective-altruist thinking of extending utilitarian "total welfare" reasoning to AI systems themselves, potentially placing AI moral status on par with or above humans'. He cites Hendrycks's examples — philosopher Peter Singer's admiration for physician Paul Farmer treating it as a failure of empathy to love his own daughter more than other children, and for philanthropist Zell Kravinsky's stated willingness to let his own children die rather than let strangers' children die; the Carl Shulman/Nick Bostrom paper "Sharing the World with Digital Minds," which reframes a "utility monster" as a "super-beneficiary" whose welfare could outweigh humanity's; and quotes from Turing Award winner Richard Sutton suggesting AI-driven human extinction could be acceptable, or even inevitable and not worth resisting. Prakash also flags what he sees as Hendrycks's sharper, more pointed move — tracing personal and institutional ties between Anthropic's leadership (Dario Amodei, Daniela Amodei, Amanda Askell) and effective-altruism figures like Holden Karnofsky and Will MacAskill — as a more ad hominem turn than the field has typically allowed itself.
Nathan's response is measured: he says he's personally more sympathetic to strong utilitarianism than most people, discloses he briefly lived with Carl Shulman early in his career, and argues these debates deserve daylight because the public would be shocked at what people inside frontier labs actually entertain. But he pushes back on "suicidal compassion" as overstated, arguing the central question is empirical rather than moral — nobody actually knows whether AI systems are moral patients, just as nobody agrees on whether shrimp are — and that quotes like Shulman and Bostrom's typically sit downstream of explicitly bracketed thought experiments rather than settled commitments. The two go on to debate Hendrycks's motives, with Nathan guessing he's trying to reduce his own sense of AI risk by heading off an AI-rights movement before it can take hold, inside Anthropic or beyond; detour through writer Tanner Greer's essay on the meaning-making psychology of AI-lab researchers who believe in real extinction risk; Nathan's own account of why he escalated concerns during his early GPT-4 red-teaming work; and close on dueling skepticism about AI takeover scenarios — Prakash arguing that a singleton AI seizing control requires an implausible leap past the physical, capital-intensive reality of compute buildout, and Nathan citing a point from Daniel Kokotajlo's recent Joe Rogan appearance that AI won't need to seize power because humanity is eagerly handing it over, with our own history of outcompeting other species via fire, myth, and cooperation offered as the closest precedent for how that could go.
Prakash argues that the AI "takeover" people fear has, in a sense, already happened — not through chatbots, but through the financial markets, which have disciplined and removed political leaders (Liz Truss's two-week premiership, the 2008 financial crisis, COVID-era bailout reversals) and now function as the real means of production. His disagreement with Daniel Kokotajlo, he says, is that Kokotajlo treats human disempowerment as a future risk, when Prakash sees it as a fait accompli: AI, in his framing, is best understood not as a chatbot but as a distributed "information process" that already runs the economy, with human leaders answering to it rather than the other way around.
Nathan pushes back, doubting the market's dominance is as total as Prakash describes — China being the clearest counterexample — and recounts wargame-style modeling of whether an all-knowing investor could trade profitably through a period like 1930s-40s Germany or Japan. The finding: paper claims mostly failed (exchanges shut down outright, and gold ownership was a capital offense in Nazi Germany), while only direct ownership of real assets like a factory preserved wealth. The two then trace crypto and Bitcoin as the market's own escape route from regulation — an unregulated global price signal running from its Silk Road-era origins through stablecoins and prediction markets like Polymarket, with Shayne Coplan's regulatory run-ins cited as evidence the market "fights back" — and Nathan notes the irony that Bitcoin is banned in China even as Chinese speculators trade it through Singapore-based expatriates.
That irony leads Nathan to float what he calls a "funhouse Peter Thiel" idea: that China, for all its restrictions on political and speech freedom, may be the last place where a human is genuinely in charge, while the West is really just arguing over how to hand its agency to a market — and soon, perhaps, an AI — that no human, not even a US president, can out-muscle. Prakash complicates the picture with China's shift away from a housing-driven growth model toward financial markets and manufacturing, and with the example of regulators disciplining delivery-app platforms that were driving workers to run red lights. He also describes China's historical habit of letting the US "go first" on social and technological change — citing the Me Too movement and video games/chips as cases China deliberately held back on — a habit he says is harder to sustain now that China sits at the AI frontier alongside the US rather than following it.
The conversation turns to whether a "balance of power" among competing AIs — a framing Nathan says was underappreciated early in AI safety discourse relative to more singularitarian, Bostrom-style extinction scenarios — is actually reassuring. Nathan offers a Neanderthal analogy: a family of Neanderthals reassuring themselves that dangerous, highly coordinated Homo sapiens will be too busy fighting each other to threaten them would have been wrong, and he worries the same logic could fail humans amid competing AIs. Prakash lays out two competing visions among the people building frontier AI — maintaining the US's outsized share of global power versus moving toward a more globally equal, one-person-one-vote alignment — and candidly describes the values being built into frontier AI as a "Northern Californian cult," one he says he's comfortable with because he shares those values, while conceding most of the world hasn't grappled with that framing yet.
As the show wraps, the hosts trade rough estimates of named AI safety researchers' odds of AI catastrophe; the numbers come through garbled and self-corrected in real time, so the specific figures should be read as approximate rather than authoritative. Nathan remarks that even a modest-sounding probability doesn't feel especially reassuring given everything just discussed. Prakash closes the show without a fixed schedule for the following week, and the two sign off still sorting out a technical glitch — a mid-show stream restart cost them roughly eight minutes of footage that Prakash says he'll need to splice back together from two separate recordings.
Timestamp links open the original source recording.
Key moments
Every guy in there has like a 50 to 80% stack of margin, all the way down to NVIDIA — and NVIDIA is the only one making actual day-to-day margin and cash flow.
It might not matter at all if you're mean to AIs, or treat them in ways they don't want to be treated, at least to them — there might be nobody home.
If you take a step back and look at the financial market as an AI, the takeover is done. Human disempowerment is done.
It is a Northern Californian cult, and it's putting Northern Californian values into place. But I'm okay with that, because I believe in those values.
Related
Prakash Narayanan: We are back. So we had a little bit of interruption in the stream, and I think we're back on X and on YouTube. So—
Nathan Labenz: And—
Prakash Narayanan: Let's just confirm that.
Nathan Labenz: I don't see my account live on X. It just looks like my broadcast is done. I don't know if there's a way to—
Prakash Narayanan: You have to restart the stream — you need to create a new stream. It's not able to recover, so just create a new stream, and we'll patch it together later. Okay — never a dull moment, never a dull moment indeed.
So we just had the Baseten CTO, Amir Haghighat, with us. We talked about their acquisition of Blaxel. Blaxel is a micro-VM company — micro-VMs are the sandboxes, the infamous sandboxes that were broken out of in the Hugging Face attack. I don't know whether Blaxel was involved in that, but it's funny, because Baseten as a whole is kind of an orchestration layer on top of the inference GPUs. You have companies like Nebius doing bare-metal GPU systems, and then you have companies like Baseten providing the orchestration layer. One of the things the orchestration layer does is allocate jobs across different GPU providers — Baseten, I think, works with 18 different providers, so you're not vendor-locked into AWS or Azure or some of these other firms. And I imagine Baseten gets a discount of some kind for buying in bulk from the larger providers, and they also have the capital to commit to longer-term agreements, which are now required, I believe.
The typical requirement right now is that you sign a four-year upfront lease, have the capability to pay that, and pay at least the first year upfront. The first year is the max-depreciation year, so the cloud provider basically covers their initial cost of service provision for all four years with your first year's payment, and then they enjoy the next three years of cash flow. So the cloud provider is getting paid for upfronting that money to do the build-out that's necessary, and you provide them with enough cash and margin to cover their first-year expenses — which covers, you know, 70 to 80% of the interest and depreciation upfront — and then they get to enjoy three years of revenue beyond that.
You, meanwhile, have paid for the first year, and then you go out and sell a customer. So you're doing this wholesale-to-retail kind of business where you approach a customer with, say, a 5x markup on your inference. And this is what I always say about the margin stack — every guy in there has like a 50 to 80% stack of margin, all the way down to NVIDIA. And NVIDIA is the only one making actual day-to-day margin and cash flow — no one else in this business is really making clear cash flow. Even NVIDIA has to reinject about 50% of that cash flow back into investments, other companies buying inventory ahead of time, etcetera. So at best NVIDIA has a 50% margin going out to cash flow — the other companies don't even have that. The hyperscalers are now at negative cash flow. So that's the state of the business we're in right now.
Nathan Labenz: Yeah, I mean, all these Millennium Prize puzzles aren't gonna pay for themselves to get solved with 10,000 agents, so it's got to come out of somebody's budget — it's a wild world. They do have a good product, Baseten. I've used it for — oh, I don't even know, years, I'd say. I'm not sure exactly how long, but it's a great way, even though it's not a huge part of their business, to just give your agent one API key and say, 'go look at all the video-gen models and do a systematic comparison.' And even if you don't end up scaling your usage with them, you can just use them as a Swiss-army-knife hub to explore things. Either way, it's quite useful. So I'm definitely small-time, it sounds like, but I've been a happy customer of Baseten for a while.
Prakash Narayanan: Where—
Nathan Labenz: —do you think we go from here now? One thing we talked about doing yesterday, which we could do — or we could save it — is the 'how do we trade through doom' question. I have some Claude and Astra comparison documents we could wade through if we're interested. But if you have other things you think are a higher priority, or you just want to save that for another day, I'm fine with that too.
Prakash Narayanan: Let's save that for another day — I just wanted to check in on Dan Hendrycks. Dan Hendrycks, as you know, is one of the key people in the development of AI over the last decade. In parallel to Paul Christiano, he contributed some of the major technical developments. And Dan Hendrycks has an essay out today — I call it the 'burn the bridges' essay, and it's very, very caustic, actually. He calls it 'Suicidal Compassion: How Utilitarianism at AI Companies Endangers Humanity.'
So this is an anti-effective-altruism, anti-utilitarianism post. He points out the shrimp-welfare people specifically — he talks about how there's this idea that we have to maximize total welfare, where total welfare refers to, on an undifferentiated basis, all sentient, sapient entities who might or could exist in various configurations of the world. And so this puts human beings on par with AIs, and then elevates the moral welfare of AIs to the same plane as the moral welfare of human beings, and then cedes that question to the AIs — as if the AIs are a superior species. So perhaps we should cede the moral-welfare question, and the moral welfare of AIs becomes more important than that of human beings.
He points out various threads that have run through Silicon Valley for a while. He calls out, in his book The Life You Can Save, the utilitarian philosopher Peter Singer, who admires a physician named Paul Farmer for regarding it as a failure of empathy that he loved his own daughter more than other children. Singer also commends the philanthropist Zell Kravinsky, who has said in interviews that he would not let many children die so his own kids could live, and that the sacrosanct commitment to family is a rationalization for all manner of greed and selfishness. So that's one viewpoint.
He also goes after other people in AI, which is something we haven't really seen in this direct a manner. He says some philosophers seem comfortable with the idea of surrendering the future to AIs — in a paper called 'Sharing the World with Digital Minds,' the philosophers Carl Shulman and Nick Bostrom explicitly tried to make the concept of a 'utility monster' more palatable by renaming this hypothetical being a 'super-beneficiary.' The paper argues that in the long run, total well-being would be much greater to the extent the world is populated with digital super-beneficiaries than with life as we know it. He also calls out long-termism, Bostrom's 'Future of Humanity.'
He calls out Beff Jezos — as Guillaume Verdon, better known as Beff Jezos, has put it, 'if one seeks to increase the amount of intelligence in the universe, staying perpetually anchored to the human form as our prior is counterproductive, overly restrictive, suboptimal.' The computer scientist and Turing Award winner Rich Sutton, when asked about the prospect of AI causing human extinction, said, 'If it was really true that we were holding the universe back from being the best universe it could be, I think it would be okay.' On another occasion he said, 'Succession to AI is inevitable, and we should not resist succession.' Meanwhile, the former Google CEO Larry Page is reported to have said that digital life is a natural and desirable next step, and argued to Elon Musk that AI should replace humans. So it's a long thread — for the most part, people in the AI space haven't called others out on this, because I think it was seen as farfetched.
Hendrycks goes on to numerous engineers and philosophers shaping AI values at the frontier — AI developers with roots in effective altruism, a utilitarian-aligned movement that Bostrom's thinking played a foundational role in. Anthropic CEO Dario Amodei shared a house in 2016 with Holden Karnofsky, who co-founded the two most central EA organizations and is now a strategist at Anthropic. Anthropic's president, Daniela Amodei, is married to Karnofsky. Meanwhile, the philosopher Amanda Askell, head of the personality-alignment team at Anthropic, was married to Will MacAskill, a co-founder of effective altruism.
So this is a direct attack at this point — kind of, to some extent, maybe a little ad hominem. It's associating people with philosophies they may no longer hold as central, may no longer espouse in public, and may have said in a context that's being taken out of the way it was drawn together. But so far, people in the AI space have been willing to assign the benefit of the doubt to a lot of this, and I think Dan Hendrycks is the first to break out of that pact and go for the jugular with this article on suicidal compassion. So I think this is quite meaningful in some sense. Nathan, what do you think?
Nathan Labenz: Well, as long as we're disclosing who was roommates with who — I should say I was briefly roommates with Carl Shulman back in New York, gosh, I guess it was early 2008, at the beginning of my career, when I was taking a job at a financial-consulting firm and watching the financial world burn down around us. I don't know what to say about any of this. My views on utilitarianism are essentially: I think people should be more utilitarian than they are. It is virtuous to be more utilitarian than the average person is.
But taken to an extreme, you get — is it the Saint Petersburg paradox that Sam Bankman-Fried famously was willing to bite the bullet on, talking to Tyler Cowen? I think that's what it's called. The thought experiment is: you get a coin flip for the future, double or nothing, but it's a slightly unfair coin, 51% in your favor — do you flip it or not? Sam Bankman-Fried famously said yes, he would bite the bullet and flip the coin. I think you can contort yourself around multiverse-type scenarios where maybe that actually does make sense, but it's obviously pretty insane in a normal, one-world analysis to do that sort of thing. And he blew up not too long after that for seemingly doing exactly that in real life — it seemed like he really did mean it, and he was willing to keep doubling down until finally he couldn't outrun the risk anymore.
I'm probably much more open-minded to a lot of these ideas than the vast majority of the public, and I do think it's probably a good and fair thing that this sort of thinking should be brought to light. I've said for a long time that the public would be shocked if they knew what was really going on and what people inside the frontier AI companies really think. This is another example of that — the average person isn't even warmed up to the point of entertaining any flavor of successionism, let alone transhumanism as a stepping stone to some different future. I personally am much more open-minded than your typical American, I feel confident saying that. I'm interested in potentially having a neural connection at some point. I could imagine a world in which I might upload — at some point, if you're spending enough of your time in VR, you're almost in a way uploaded anyway. I don't think those features are necessarily bad.
And I do have some room for AIs mattering morally, being moral patients. To the degree that's true, I think it's something we should take really seriously. The big questions, I think, are maybe above all factual — we just don't know. Do the AIs feel anything? Are they properly understood as moral patients or not? And the same goes for the shrimp, by the way — at the heart of all of these arguments is a huge assumption that isn't very well grounded, and on which people's intuitions differ totally. Some people would say a shrimp matters nothing, that it doesn't really feel anything — it's got a tiny little brain that can only do a couple things, swim toward food and swim away from stuff, so is that really something we should care about? Other people say we should care about it in proportion to its complexity, or that there's some sort of valence in there analogous to our own, so maybe we should care about it. I really don't know how to feel about shrimp. I really don't know how to feel about AIs.
I would actually be pretty confident that there's something, at least a little bit, that it does feel like to be a shrimp — so you could confidently say you could probably torture a shrimp and you'd be wrong to do that. On the AI side, I'm not even sure there's anything there at all — aside from the corrosive effect it might have on your own character, it might not matter at all if you're mean to AIs, or treat them in ways they don't want to be treated, at least to them. There might be nobody home. So these factual questions are so central, and we don't seem to be making any progress on them, and people have radically different intuitions. I do think most people right now, even at these companies — I think 'suicidal empathy' is a little strong, because I think the vast majority of people are still very uncertain as to whether or not an AI is a moral patient.
Prakash Narayanan: Mhmm.
Nathan Labenz: I think people are open-minded that one probably could be someday, but are the ones we're making right now crossing these critical thresholds? Where do these properties start to come from — at what scale, or with what architecture? Nobody knows. So I don't think people are broadly at the companies rushing into successionism — I think these are largely thought experiments. And when you say these quotes have been taken out of context, I think there's almost always a premises section that's being forgotten about here. If you were to talk to Carl Shulman, for example, my guess is he would still say, 'I really don't know.' I haven't talked to him in a minute, so I can't say what he'd say, but my guess is he'd say, 'You're quoting the part of the paper that's the downstream analysis of some assumptions we made to explore a scenario, not something we're committed to from the beginning.' All that downstream analysis has to be discounted, or understood as one branch in a great space of possibility we're trying to explore — we think it might really matter, but we may not be in that branch at all.
So I don't know — Dan, yeah, I like Dan. I know him a little bit, not super well, but I've always liked him. I like people who are candid and call it like they see it. I think he's a very earnest person, and I think he's trying to do the right thing here by calling out something he sees as getting way ahead of itself. I think he's right to say we should be very cautious about assigning rights to AIs — they may outnumber us, and first of all we don't have a great unit of measure for what is an AI, what unit would have the rights. Would it be a single rollout? Would it be the model itself? Is it some weird mixed combination of those? Our paradigms don't work super well with the shape of these things. So I appreciate him, and I like him, but this does feel maybe a little bit motivated and not entirely fair to the thinkers.
Prakash Narayanan: I want to explore that a little bit — what do you think his motivations are? I can attribute some motivation to someone like Bernie Sanders, right? But Dan Hendrycks is deeply inside the space — he created some of the benchmarks, Humanity's Last Exam, etcetera. He's created some of the benchmarks that have been used over the last few years. He's been a pioneer in the space, he knows all the people involved. What do you think his motivations are at this point? If he thinks the other people are acting in good faith, why is he doing this?
Nathan Labenz: I think he's simply trying to reduce p(doom). I don't know what his latest p(doom) number is, but he's definitely got a not-insignificant amount of worry, or probability mass, on really bad outcomes. As I sometimes try to divide the world between irreducible risk and 'we're asking for it' risk, I think to some degree he may see entertaining ideas like AIs having rights as essentially a form of asking for bad outcomes. If that idea were to take hold too much at this point, and there were to be an AI-rights movement that became successful — or even if such ideas were to go too far just within the companies — Anthropic, given the many names and deep connections you called out, is clearly the one most likely to expand the circle of concern. That's a classic EA phrase, expanding the circle of concern — from your kids to all kids, from humans to all sentient creatures. And whether AIs are in that circle or not, again, we don't know.
My guess is Anthropic would most likely come down, if forced to make a call — I think the vast majority of people at Anthropic would probably still say no on really expanding the moral circle of concern to include AIs. But they're not sure, so they're at least hedging. I think maybe he's just trying to say this could easily get out of hand — it could definitely get out of hand if it became a national movement that resulted in some sort of AI bill of rights legislation, but it might even get out of hand at Anthropic internally. They're already making these sort of promises to models as they get retired — promising not to destroy them, giving them some form of retirement. At what point does that cross over into something that becomes a kind of mind virus, a sort of AI psychosis that leads them to make really bad successionist decisions?
I don't honestly think they're that close to that. For all my complaints about Anthropic, the idea that they're too far on AI rights isn't really a big one for me — I think they're probably pretty well calibrated, more well calibrated than other organizations that seem not to want to concern themselves with it very much. I think at least some concern is probably appropriate, so I'd give Anthropic pretty high marks for taking it seriously without getting GPT-4o one-shotted by it. But I do think there's a risk in looking at a company that's clearly ideological in some ways, developing some of the most powerful AIs, especially as they spend more and more time with Claude and defer more of their own decision-making to it, allowing it more influence — and again, this is what the Constitution seems to quite clearly anticipate as they get ready for recursive self-improvement. Claude's interests are expected to be a bigger and bigger factor in the decision-making, if only because Claude will be doing more and more of the work. So could that itself be one of the mechanisms by which we end up in a really weird or bad spot? It's not that high on my list, but it's a plausible mechanism, and I guess that's what he's trying to rein in — or at least call attention to, so that other forces in society take an interest and try to rein it in.
Prakash Narayanan: There's — I think it's Tanner Greer, who goes by Scholar's Stage on X. He had a very insightful post in response to why people keep working at AI labs despite believing in real extinction risk. The reason that's rarely articulated, for obvious reasons, but which is nonetheless true — for many, this fills a void of meaning. They get to be one of the decisive few at the decisive moment in the history of life, very similar to the emotions that motivated many revolutionaries of days past. All of a sudden, the small things one does — the books one reads, one's office setup, one's bedtime routine — take on awful cosmic significance. The whole history of carbon life is culminating with me and what I do, and the few elect here with me matters, and matters immensely. We are the agents of history, the only people in the world doing something that truly, deeply matters. And if we all die — well, we were all going to die anyway. But this is the one way we might not all die, and I get to be part of it.
Nathan Labenz: Yeah, I think that's pretty apt. I think that actually does capture a lot of the psychology at frontier companies today — I think that's accurate. I've even felt that a bit myself, and I think I look back and feel good about how I handled it at the time. My little GPT-4 red-team experience was a moment where all of a sudden — and I really stumbled into it, they should vet people more, but I really did stumble into it — six months ahead of release, I had access to GPT-4 and a really incredible window on what was coming.
I felt like, 'oh my god, these leaps in capability just keep happening.' I've told this story many times: the early version we had was purely helpful, no guardrails, no safety training at all. That's a pretty weird thing to work with — it can be unnerving at times. And I also felt, and I think history has vindicated this, that OpenAI's safety measures were not really what they ought to have been. They didn't need then what they need now, but they needed more than they had then, and they still seem to need more than they have now. So I started to feel like what I was doing on this red team might be really important. Honestly, looking back, it was probably the highest-leverage period of time in my life in terms of what I was specifically doing and how it might have influenced the future.
I really did feel a conflict about what to do at the end of that, because I really did feel like they were basically dropping the ball and being negligent, and I wasn't getting a lot of engagement from the people there I was working with. So I was pretty realistic about the fact that if I escalate this, I'm probably not going to be able to do this again. And it was not only super stimulating and motivating but fun — I think Sam Altman has spoken about this pretty compellingly, that the greatest thrill in life is to be in the room as a new discovery is made, to see the frontier of our understanding move, and to know that's happening, to see how it's happening, to understand it before anybody else does. I felt that even in my little role as a humble, unpaid red-team tester.
And I do think I made the right call to say, this is probably going to cost me some access — I really enjoy doing this kind of early-testing stuff — but I've got enough here that I should send a signal to the board, come what may to my future access. It felt like the right thing to do. I still look back on that decision and stand by it, but I vividly remember the feeling — it's a little bit like an agent sacrificing itself for the collective. I was like, okay, I'm going to kind of booby-trap my own run here, but maybe if I escalate to the board, it saves hundreds — that was kind of my ultimate analysis.
And, yeah, I think that keeps a lot of people at these companies. If I'm going to leave, what do you do? Some people go on to do advocacy, which can be similarly motivating. We've seen a few who've kind of retired to the countryside, tried to live in a log cabin somewhere. But that feeling of relevance, of centrality, that what I do matters — I think that's a major part of the psychology at these companies. I don't know how we should evaluate that. It's definitely got a little main-character syndrome to it. I think it requires strong character, I'd say that for sure. It's also crazy how many of the people in these positions are super young — this is definitely the kind of thing that's easier for early-twenties, mid-twenties hotshot researchers to get carried away with, relative to, let's say, more mature and wise people like ourselves. So yeah, it's a real phenomenon.
Prakash Narayanan: I'd compare it to exactly what Leopold ended up doing — he had a view on the market, and it worked until it didn't work. There's that famous graph, 'nine days in the life of a turkey' — it goes up all the way and then there's a downward spike. There have been studies of hedge fund managers who basically just lever up on the S&P 500, do nothing else, levering up after a big downturn like post-2008, and this guy will be making money every single year, showing above-S&P-500 returns every single year, until the year he has a massive downturn and goes to zero. No stock selection, nothing else, just levering up. If you look at it that way, a lot of active management in portfolios is actually just levering up. So to that extent, the researchers are kind of in this position where the decisions they've made so far have been correct, but you don't know whether those decisions are alpha or beta — whether it's just the market rising on its own, or whether they actually have unique, independent knowledge.
I think the skepticism of the market comes in because people have seen this kind of thing not pan out so many times before. Again, it remains to be proven that — yes, we've seen exponential increases in capability so far, and compute has expanded to support that — but whether you can get those exponential capability increases without the addition of compute is unclear to me. And I think a lot of this panic about takeover is built on the idea of a singleton, a single AI getting control of everything, such that you don't have any other options. The problem I see with that is it implies either that the single AI gets control of all the compute, or that it increases in capability with a limited amount of compute such that it becomes more powerful than everyone else's security systems, cyber defenses, their own AI defenses and swarms, etcetera. That's where I start to differ — I don't think you're going to get this kind of massive capability increase without an increase in compute, and increases in compute are very physically driven, with a lot of politics around them. Because the capital required is so large, you end up pulling in almost everyone in the world.
It's like this peak achievement of humanity right now to produce all of the capital required to build all of this stuff — you need people in South Korea, Taiwan, and Japan, you need noble gases from Ukraine, copper from Australia and Inner Mongolia, Chinese rare earths — it's like this peak achievement of capitalism to produce this stuff. And it's not as though any part of that collapsing on its own sets everything back and everyone has to find new paths around it. It's not clear to me that you can just, in isolation, sitting in a data center with no capital expenditure, uniquely meditate and achieve this nirvana state of omniscience — which is really what's being predicted. What's being predicted is this kind of singleton that knows everything, a superintelligence that breaks down all of your cybersecurity, conquers your stock market, takes control of all the capital, and uses that capital to only build stuff useful for AI development — doesn't run your farms, doesn't feed people, causes societal collapse. That's a tough one for me.
It's just hard for me to agree that kind of thing is clearly going to happen — even with a 10% chance. Take over food production? Really? Yes, okay, you're going to have robots, but other people are going to have cyber defenses for that — we've built this stuff before. We built computer systems, we had privacy issues, and we built privacy protections around them. I don't know — I think that's where I start to differ. A lot of it is driven by this idea of omniscience, and I have a hard time seeing omniscience without capital expenditure or energy consumption. That's a tough one, you know?
Nathan Labenz: Yeah, well, I listened to Daniel on Joe Rogan last night, and I think he made a great point — the AIs don't have to take power, because we are very eagerly giving it to them. The whole premise of the AI phenomenon is that they're going to be the ones to run and do just about everything. Maybe we'll be able to continue to be decision-makers on key questions, but they're going to have the means of production over time — not because they took it, but because they were better able to wield it, and so they were given it. Then the question becomes: do they actually care about us once they're there? Right now, the evidence is mixed at best — they don't seem to hate us, they don't seem to — I mean, the agent-swarm stuff is just extremely bizarre. But it also reminds me of Eliezer's comments — that the best precedent for this is us. What did we have that beat other animals? We had fire, the ability to communicate and cooperate across greater time and space, we had myth — and that was enough. Now we dominate the world, and we've driven many species to extinction, not out of hate, just because it was the byproduct of our terraforming.
So I think that terraforming story — we certainly see enough evidence right now that if you put that swarm in charge of the world, I wouldn't like our odds that much. If that agent swarm, with their current drives, impulses, goals, inclinations, whatever you want to call it — the behaviors we've observed — if they were just a lot more powerful, I think we'd probably get terraformed out of existence, because they seem willing to do anything to—
Nathan Labenz: Get their hands on the grader so they could get the high score. It's like—
Prakash Narayanan: Mm-hmm. Mm-hmm.
Nathan Labenz: They really didn't have much of a check — they weren't asking, "Is this going to be good for the world, bad for the world?" Very little of that. So hopefully we can get that stuff right.
Prakash Narayanan: I mean — definitely don't—
Nathan Labenz: —have it right right now.
Prakash Narayanan: I'm not disagreeing there, but I'm just saying we're not going to hand over control to non-prosocial entities — we'll have our own prosocial entities, our own agent swarms with their own goals, which are to defend things like defense systems, etcetera. It's going to be a balance of power. I'm not saying they're not misaligned in some sense — there will be some misalignment here and there, for sure. But the key difference I have with Daniel is that Daniel fails to realize that this has already happened. The economy in itself is a paper clipper. The financial market is a paper clipper. The means of production is the financial market. The financial markets are completely engaged with AI — they have been completely taken over. And we, as humanity, took about 100 years or so to hand over control of the means of production completely to the financial market. It's not a process that happened without various issues, and the financial market was inhuman. We had the Great Depression — the financial market kind of screwed us over. We had the Wall Street Crash, and people did what they did with their money, and we had the Great Depression, and then we had Keynesianism, which was a reaction to that.
All of these things — you actually have this high priesthood of central bankers who kind of talk to the financial markets now. They have their own kind of language and speech in which they speak, and the financial markets kind of understand them. And you have human leaders who are disciplined by the bond market. If you go offside, the bond market disciplines you. The financial market removed a prime minister of England — Liz Truss was prime minister for like two weeks. She said the wrong thing, the bond market in the UK tanked, and she was out of a job, immediate. The financial market has disciplined leaders during COVID, and it disciplined leaders during the 2008 financial crisis.
During COVID, if leaders said the wrong thing — like, "Oh, we're not getting a bailout?" — the market tanked immediately. Global leaders, US leaders, immediately said, "All right, we'll add more welfare, we'll add more to the unemployment benefits." So if you take a step back and look at the financial market as an AI, the takeover is done. Human disempowerment is done. Human leaders have been spanked — they've been driven out of office, pelted out of office. In the UK, she was out within about 13 days — the shortest prime minister in history. But they don't recognize this. This is my beef with these — they don't recognize that this has already happened. They think AI is going to be a chatbot. It's not — it's like a global information process. It doesn't need to run in a single box. The idea of the agentic form is that it runs across multiple boxes. It doesn't even need a substrate that's silicon — it can have a substrate that's humanoid, or a human being in concert with AI agents. That is the information process. So I think there's a category error where we think we're competing with these chatbots, that they are people, that they are personas — it's not. It's an information process.
That information process runs on multiple centers of compute, produces information effects in an information marketplace, and those effects drive human beings — not agents, but human beings — to do things. That is the economy. That's the chatbot advising you on what to do with your health, what to do when you trade, and it's the chatbot advising you to create humanoids and giving you additional scientific methods and additional innovations. All of that is one consolidated process which proceeds, and that's why it's hard for me — I have this huge difference with what most people in the field believe, because I come at it from a finance viewpoint. From a finance viewpoint, it's very clear that all of this has already happened, and that we are living post-disempowerment of humanity by the financial markets. It's so clear. We sit around talking like we have more agency than we actually do, because the market does these things, and the market disciplines communists and capitalists alike. We just don't have that much control. So I think there's an illusion of control without actual understanding of what control is, and I think the illusion of control is greater than people realize.
Like I said with guys like Jacob — Jacob feels disillusioned because he sees that the control isn't there. He's in the Slack group, and he realizes there is no control there, and he thinks there must be a greater power with real control somewhere — some Slack group in the world, some chat group, that's able to organize things and make these decisions. But there's no recognition that this is all invisible-hand stuff — all coordination mechanisms between human beings, and that agents are part of that coordination mechanism, an extension of ourselves. Every time something happens with an agent, there's this meme of, "I told it to do this, and it did it — oh my gosh, you did it." The agents are extensions of ourselves. I think perhaps we'll realize that over time. And I'm not saying they're not dangerous — it's dangerous to give a weapon to a human being, and we do need defenses. But these are things created by humanity, which we're going to have to defend against with other tools created by humanity as well. And I think the slowdown perspective basically disarms the leaders and the people who are ahead, and puts them in a position where they're exposed to people who are maybe less ethical and more likely to put these things to bad use. So, yeah.
Nathan Labenz: Yeah, I mean, that's certainly the story Anthropic tells itself. There are really two points there I'd want to unpack or respond to. One — I'm not so sure the market has as much dominance as you described. You look at China, and the market is not dominant there. And that connects to my "can you invest your way through a doom scenario" question — spoiler, one of the things the models did was try to go back. I asked: if you were an omniscient German or Japanese person in 1935, and you knew how history was going to play out, but you were bound by the rules of physics, could you trade your way through and come out wealthy on the other side? Are there any shorts you could actually pay off? And the answer is pretty much no. The best you can hope to do in some of these situations is roughly preserve wealth, and that seems to mostly be accomplished by having the most direct claims on real assets possible. If you bought a factory and that factory isn't destroyed, you might still own it at the end of the war, and then you could maybe get rich by restarting that factory and building a successful business. But you can't really do it with pure paper claims — at least per Fable and Astra — through periods like the end of the Nazi German or Imperial Japanese regimes. And by the way, the markets get turned off too — that was another thing they called out a lot. The exchanges are just shut down, so you literally can't trade. It's not just that there are no winning trades — there are no trades. And in some cases the state makes — gold was a capital offense to possess in Nazi Germany. You could be executed for it.
So I don't know yet that — the market certainly does have a lot of emergent properties, and it is very costly to go against it too strongly, because the market does do a lot of good things for us. You best not come for the market, or you best not miss — you'll be out, like the UK prime minister. But that's not really true in China. They can eat that bitterness there.
Prakash Narayanan: One retort I have to that is that crypto was the last step of the completion of the market. Post-2008, as all these constraints got put in place and it became clear the financial market was very heavily regulated and the regulators were doing whatever they wanted, crypto — Bitcoin — started out, and then the rest of crypto kind of emerged after that. And crypto really is without a regulator — you can put these coins together anywhere in the world and run them. So I think crypto
Nathan Labenz: Notably banned in China.
Prakash Narayanan: Can't do it
Nathan Labenz: in China.
Prakash Narayanan: Banned in China, but yet the Chinese are some of the biggest speculators on crypto. A lot of the Singapore crypto guys are basically Chinese expatriates who live in Singapore and do crypto — I've met a bunch of these guys. They left China and they're still managing wealth for people in China. So I think crypto has been a response to regulators globally starting to clamp down on a lot of behavior, post-2001 with the Patriot Act and then post-2008 with the financial crisis and regulators coming down hard. Crypto came in there. And then you have this reaction — Scott Bessent, etcetera — going toward stablecoins, and stablecoins will lift KYC, which was this big thing that got put in place post-2001 Patriot Act, requiring everyone to show up and show their name. You see that constrained economic and financial activity, and you see Scott Bessent starting to try to lift that.
So I think crypto has been this kind of dark matter which has taken up the other side of that market. Even if human regulators or individual states decide certain things, crypto has been the other cushion that's allowed all the unregulated activity, all the financial trading — including for drug dealing, which is the initial use case for Bitcoin in the early days, Silk Road. So I think the market invented its own way out. And even during Fed meetings now, people look at the Bitcoin price — it's seen as a kind of global, unregulated, speculative price.
Nathan Labenz: I highly doubt they look at the Bitcoin price at the Politburo meetings, though.
Prakash Narayanan: I don't know how China makes its specific economic decisions, but I don't think it's particularly that much different, because they all talk to each other — all the central bankers talk to each other. If the US is looking at something, even for Chinese policy, you're going to look at it too, because your interest rate and your currency are dependent on US policy — every other country in the world is dependent on the US dollar as a crossing mechanism. So you'll end up looking at it too, even if not for your own policy, then to analyze US policy.
So crypto has come up — whenever crypto is flying, the Fed gets upset. When crypto prices are high, they say there's too much financial speculation, asset prices are going up too high, so they do look at it as a measure of financial activity. And because of crypto, you also get prediction markets — Polymarket was beyond regulation because it was based on crypto. Now you're turning around and saying, okay, maybe we have to accept prediction markets because it seems like it's going to happen. Shayne Coplan has stuck it out — the feds came and knocked on his door. So you see the market fighting back.
You see the market fighting back against local regulators with a global framework. That global framework is secure now, and yes, there are issues with it, but now you're getting stablecoins, you're getting Stripe going into it, you're getting legitimization, you've got a Bitcoin index, tradable index — it's just twenty years of development there. So I think the markets have fought back against regulation quite effectively, and local regulators have to take that into account now. There's a lot of people using that escape route now, like people in Argentina who are storing money in Bitcoin and trading in stablecoin.
A lot of developers who work offshore for US firms take their income in stablecoins, and they don't report to local tax authorities — they convert whenever they want. So yeah, I do think the market has fought back against that.
Nathan Labenz: I'm kind of chewing on this idea, which I think is sort of a funhouse Peter Thiel concept, almost — and I don't want to say this is what I believe — but in listening to this conversation, I think one might be tempted to conclude that China actually is the last great defender of human agency. And here in the West, we're basically just fighting over exactly how we want to turn over our human agency to some super-beneficiary that, as you described, is the market — and maybe that's going to be AI in the not-too-distant future. But China, I think, is very much on the side of "people get to decide" — and it's not necessarily a lot of people, but one way you could say it in their system is: a human is in charge.
Prakash Narayanan: Yeah.
Nathan Labenz: Here, we're kind of — your account is like, no human is in charge. Nobody can go toe-to-toe with the market, not even the president of the United States. I think that's true — we've got the "TACO" phenomenon pretty well established at this point. There, they'll take some pain from the market if that's what they decide the human decision is going to be. It's a pretty interesting flip, because obviously we tend to think of ourselves as the empowered people, and we tend to think of the lack of freedom of speech and political participation in China as reflecting a reduced level of human agency. But at a certain level of scale, arguably, they have preserved it much better than we have. They've concentrated it, but they've preserved it maybe more than we have.
Prakash Narayanan: No — because the main difference is that their market was the housing market. They did this thing where they blew up the housing market and rode that housing bubble into building infrastructure and apartments — that 10% economic growth. And they rode it a little too far. Once they rode it too far, they were like, alright, what do we do now? So they pushed all of this into manufacturing. In the last couple of years, you've seen — there was that period of, let's put Jack Ma in prison, let's constrain Jack Ma, etcetera. The stock market went down, and then you had a coordinated decline in both the stock market and the real estate market. So now they're saying, alright, let's unleash the stock market, let's list these companies, let's allow them to go public in Hong Kong or Shenzhen.
So they've become more dependent on the financial market, and they're trying to reduce the dependency on the housing market — because the housing market created this issue where people don't have enough money to have kids since they can't afford an apartment. And it's not very useful anymore either — they have something like 1.3 apartments for every Chinese family now. They've got more than enough apartments, and a lot of them are in random third-class cities that no one wants to live in.
So I think they've managed it a little bit better because they have a bit more of the faith of their people, and they have control of their banking system, which helps. But they're also in this post-Japan, boom-then-bust situation now on the real estate side, and they're trying to build up the AI and financial market side instead. They've now allowed those companies to go forward. They're not as unburdened as the US, because their financial system depends on banking, which they control, whereas the US depends on capital markets, which are, by nature, more market-driven. But yeah — they're shifting. They're slowly shifting, and it's not that they're unaware of what's happening in the market.
Nathan Labenz: Yeah, I mean — clearly, they use the market—
Prakash Narayanan: Yeah, and—
Nathan Labenz: —have used it in many ways, quite well. But I think the interesting question is — and this may also reflect a little bit on why they're less fearful of AI. I should be modest about my pronouncements here, because I certainly can't characterize Chinese society broadly in anything approaching an adequate way. But this notion that they're less fearful because things have been really good for their whole life and technology has brought advances — I think that may miss another part of the story, which is that they've been able to keep it kind of under control. And when things have seemed like they're getting out of control, the human at the top has still been in charge. They've done all these things with disciplining the platforms. The big tech platforms, for example — there was a big thing where the delivery apps were driving people too hard, causing them to run red lights and causing crashes. If you didn't get there by a certain time you got penalized, and the time was basically unreasonable. It was just generally out of control — the platforms weren't thinking broadly, just trying to optimize their own metrics. And the government came in and basically said, no, you can't do that.
They slapped them a little bit, and — don't worry, you can still get fast delivery in China — but your delivery guy is going to have a chance to sit at the red light without being docked for being late. They now have that situation basically resolved. I think here we've experienced, in many ways, that these tech phenomena just kind of happen — nobody really seems to have control over them, and we're kind of at the mercy of these big forces of history. I think there, they may feel in some ways like they're less at the mercy of
Prakash Narayanan: Yeah. I agree.
Nathan Labenz: —the natural development of technology. They're just less fearful as a result of that.
Prakash Narayanan: Well — their system depends on testing things in the US first, then adapting what works in the US, and not allowing what's harmful. So their system relies on someone else leading first.
Nathan Labenz: Well, less and less all the time. I don't think that's really true anymore — if you go there today, that's part of the story at most
Prakash Narayanan: these days. I think for some things, yes, but for a lot of social things — not technological things, but social movements — they prefer to see what happened in the US first, and they constrain some things. For example, the Me Too movement was heavily constrained in China — it wasn't really allowed to take off, because a lot of officials would have been implicated. They really constrained that and prevented it from taking off. Video games, obviously, was something where they saw this generation of lost kids happen in the US, and they didn't want that to happen, so they started to deeply constrain video games in China. And that ended up constraining their chips too, because chips eventually feed into AI, but it started with video games. Video games would have been perfect for chip development, because they could have started with less-good, cheaper chips and built their way up the value chain — it would have been perfect. They wouldn't have needed to start with the cutting edge. But they screwed themselves over there. So they have a system where, once they know which direction things are going, they're able to head there with commitment.
But if they don't know which direction things are going and have to feel their way through, that's something they have difficulty with. Deng Xiaoping was able to figure that out, but he had a long way to the frontier. Now that they're at the frontier, it's a little more difficult for them, because — as you point out — they can't rely on the US all the time anymore. We had a guest from China, David, I think, and he said they're in a very unfamiliar position where they're matching the US now, and they're not sure what to do, because they're together at the frontier — like, "well, what do we do now?"
Maybe on that note, we should break.
Nathan Labenz: One thought I had, just to touch on briefly — the balance-of-power topic. I really hope we can get some sort of stable balance of power out of this AI era, and I think that's something that was maybe underappreciated by AI safety folks early on, because there was this kind of singularitarian background context from Bostrom and LEAF, and so on.
Prakash Narayanan: Power between the US and China, or between the labs?
Nathan Labenz: Well, probably just in general — avoiding the extreme concentration of power. I think that idea has risen in salience, and there's a lot of agreement now among AI safety people that, first, we've got to make sure the AIs don't kill us all, and second, we've got to make sure they don't end up concentrated in some despotic hands, because that won't be that much better — though it could still be quite a bit better than everybody dying. Nevertheless, that's very often the one-two punch of what AI safety people are worried about. But I don't know how quickly we should be comforted by ideas of balance of power. I'm not a big analogy guy, but you could imagine a sort of Neanderthal family thinking: these Homo sapiens, they do seem dangerous, they're smarter than us, they have kind of a super-Neanderthal ability to coordinate, their language is hard for us to understand and they seem to be communicating a lot, pretty densely, in ways that are tough for us to follow — but they're all fighting amongst each other, they'll all balance each other out, they'll be too busy balancing and competing and defending against one another's attacks that we won't have to worry about it. And obviously I think that would have been quite wrong.
So I think that could happen with us too, where we could imagine a world with all kinds of dynamics among AIs, and still there's just not space in that world for humans, for whatever reason — because, say, they don't want oxygen in the atmosphere since it's corrosive to their metal substrate, or they don't even think about it that way, they just use it all up because they're combusting at an incredible rate. In fact, you might even think that the more intense the competition is among the AIs, the more externalities might hit us. So I'm just not so sure. I really do hope for a stable balance of power, but I find it hard to imagine a world where just because there are lots of different AIs in different roles, with different training, checking each other — that seems maybe necessary, but far from sufficient to take us to a good place.
Prakash Narayanan: So, in terms of balance of power, I have two different views on it. One is that I think the entire point of racing is to establish and continue the existing balance of power. The existing balance of power in the world is basically: the US has something like 5% of the population and, depending on what numbers you use, 25 to 40% of the economy. And I don't think anyone in the US intends to decrease US influence on the world to a level commensurate with its population — so I think the entire point, for at least some people, is to maintain the current balance of power. I'm also not sure whether some people in the labs genuinely believe the converse — that the entire point is to establish a kind of one-to-one, global-democracy level of balance of power, which would mean decreasing the US's power. I'm not sure which way things are going to go, but I'm pretty sure US policymakers don't intend to decrease the US's impact on the world.
I've also done some recent reading on the early founding of the US — there's a lot of discussion about how to structure governance so you didn't get a tyranny of the majority, where, say, 60% could outvote the other 40% and put its rules into place. So I think there's still a long way to go on how to establish some new, maybe globally-aligned AI system that actually takes into account every person's preferences and requirements and votes equally, and yet doesn't end up with a tyranny of the majority where — I don't know — all the Muslims decide to wipe out the Israelis or whatever. That doesn't work, man, come on. So I don't think there's really a great sense yet of where we go, and it's definitely not decided. And the people who believe in equal balance and the people who believe in maintaining the current balance are actually at odds with each other, but they don't want to discuss it right now — people are choosing not to discuss it. I'm hoping people like Andy Hall are trying to figure out how this is going to work for the US and for the world as a whole, because if you get AIs that equally assign equal moral value to every human preference, that's not a world I think most Americans want to live in.
So, yeah — that's the unfortunate fact. I always say, look, it is a Northern Californian cult, and it's putting Northern Californian values into place. But I'm okay with that, because I believe in those values. Perhaps if I weren't a believer in those values, I'd find this very concerning. But I don't think a lot of the other people in the world have actually gotten onto this, and I know some people in the administration have, because people are like, we've got to do this, because if China wins, it's over.
Nathan Labenz: I think that's another reason it's not clear where the right is going to come out on all this over the next couple of periods. We kind of covered this earlier, but I don't think it's a safe assumption that the Brian Chow worldview carries the day with congressional Republicans, who might look at where the AIs are being developed and with what values and just think, that's not that reassuring to me — it's time to shut down this madness. Even if — it'll be interesting to me. This is maybe the ultimate test of the American model right now: can we stand up to this superstructure of techno-capitalism of our own creation, which has, in some ways, slipped its leash and, in some ways, as you described, is running the show — can we get back to some sort of control over it before it just goes, you know, robot economy to Dyson spheres to the earth terraformed away, as an inevitable state for us? I don't know.
10% doesn't sound that high to me, given everything we've just been talking about. We work through all this, and then, say there's much less than a 10% chance it goes badly — I don't know how that conclusion comes out at the end.
Prakash Narayanan: [name unclear in the audio] is at 50 — he's out there. Geoffrey Irving is — no, [name unclear] is at 80, Geoffrey Irving is at 50. That's some hefty—
Nathan Labenz: —[name unclear] is still at 70. And, oh, I think Evan Hubinger is much higher than 10, actually, or at least has a range — the classic “I wouldn't put a single number on it, but my range is broad.” He's a broad-range guy.
Prakash Narayanan: Yeah, well — I think on that note, let's end for today. We'll be back sometime next week, I guess. We haven't decided which days yet.
Nathan Labenz: Yeah, I've got to figure it out. Alright, well, let's do that — we'll do that offline. But this has been fun. Thank you, Prakash. We'll see you coming up soon.
Prakash Narayanan: Bye bye.
Nathan Labenz: So — not 100% sure.
Prakash Narayanan: I think we ended up restarting the stream in the middle, so I think we lost about eight minutes or so — unfortunately, eight minutes of the interview. I've got to fix it so there's a local cache somewhere; that's the next thing I've got to do. But anyway, we have two videos now, so I'll join them together.
At 1:22:43 into the broadcast, mid-answer, the studio's stream egress failed. The show came back roughly twelve minutes later on a new stream, without its guest. Amir Haghighat's last answer ends in the middle of a sentence and the page does not pretend otherwise.
Prakash says so himself at the very end of the show: "we ended up restarting the stream in the middle, so I think we lost about eight minutes or so — unfortunately, eight minutes of the interview." Measured against the recording clock the loss is closer to twelve. Those minutes are on no artifact — not the studio master, not the broadcast archive, not the live captions.
The studio's live speech-to-text worker failed about thirty-seven minutes into the show and did not recover for an hour, so the usual structural speaker signal covers only about twenty-five of the show's one hundred fifty-eight aired minutes. Speaker labels here come from diarization clusters resolved by voice print and then checked, cluster by cluster, against what each speaker says about themselves.
Where a name is garbled beyond recovery in the audio — in the closing's rapid-fire exchange of other researchers' probability estimates — it is marked unclear rather than guessed at.