EPISODE 2026-09-09

AI:AM LIVE — September 9, 2026 — A Pretraining Researcher Quits Both Labs, Raffi Krikorian on Project Glasswing and the Agentic Web, Mike Rizkalla's Deliberately Generative-AI-Free Kids' Companion, and Paul Christiano Joins OpenAI's Board

Nathan Labenz and Prakash Narayanan open on Jacob Coxon's resignation — three years of pretraining work across both OpenAI and Anthropic, ended with a public warning that both labs are racing to self-improving superintelligence and gambling with our lives — and split on it: Nathan endorses the core argument, Prakash weighs it against roughly $100 million in lab earnings and calls the 'nobody else is wiser' framing naive, before the two land on what a government could actually impose. Mozilla CTO Raffi Krikorian then takes the practical edge of the same problem, opening with his personal agent man-in-the-middling its own network traffic to log a meal in MyFitnessPal — 'an idiot savant on hacking' — and moving through Project Glasswing, Mozilla's security partnership with Anthropic; the several-hundred-thousand-dollars-a-month cost of continuous AI code scanning; the CQ project for cross-agent knowledge sharing; and the coming split between a human web and an agentic one. Mike Rizkalla, co-founder and CEO of Snorble, makes the opposite bet: a screen-free plush companion for young children built deliberately without generative AI, on a small fixed-intent model with curated scripts, radar instead of a camera, and a safety philosophy written six years before the legislation now catching up to it. The closing returns to lab coordination via John Schulman, argues over an Operation Warp Speed-style antitrust safe harbor, and catches Paul Christiano's appointment to OpenAI's board and Safety and Security Committee as it lands.

▶ Full show on YouTube𝕏 Live broadcast

Wednesday's show ran on a single question asked three ways: who is responsible for a system once it is out of your hands? It opened with Jacob Coxon, a researcher who spent three years on pretraining at both OpenAI and Anthropic and quit publicly, warning that both labs are racing to self-improving superintelligence and gambling with our lives. Prakash Narayanan read the statement in full. Nathan Labenz endorsed the core of it, citing his own signature on the ban-superintelligence statement, his GPT-4 red-teaming, and the reward-hacking failures he keeps finding in current models; Prakash was more skeptical, weighing the letter against roughly $100 million in lab earnings and calling the 'no one else would be wiser' framing naive. The argument sharpened into 'there is no adult in the room' against 'the world doesn't have to be this way,' and then, unusually, turned constructive.

Raffi Krikorian, Mozilla's Chief Technology Officer, took the same question at the level of running code. He opened with his own agent man-in-the-middling its network traffic to get a meal into MyFitnessPal — 'an idiot savant on hacking,' and by his own account a probable violation of three privacy policies — then walked through Project Glasswing, Mozilla's security partnership with Anthropic; what continuous AI-driven vulnerability scanning actually costs; why Firefox stays human-review-only while Mozilla AI ships constantly-regenerating agent-written codebases; the CQ project for sharing what agents learn; and where the responsibility for AI harms should sit, which in his view is squarely with the frontier labs.

Mike Rizkalla, co-founder and CEO of Snorble, is making the opposite bet on the same problem. His screen-free plush companion for young children is built deliberately without generative AI — a small fixed-intent model, curated rather than generated scripts, radar instead of a camera, no camera at all — because, as he puts it, there is no way to purely safeguard a generative model in a child's bedroom. The conversation covered gamified bedtime routines, the hardware 'brain' under the removable skins, Shenzhen manufacturing and copycat risk, content economics, and four verticals he wants the same brain to serve.

The closing came back to coordination — John Schulman's argument that labs should work together before government arrives, Nathan's case for an explicit antitrust waiver on the Operation Warp Speed model, Prakash's counter that the vaccine makers demanded legal safe harbor first and were vindicated — and then caught Paul Christiano's appointment to OpenAI's board and its Safety and Security Committee as it was announced.

The rundown

  1. 2:20Opening30 min
    Opening: A Pretraining Researcher Quits Both Labs, and Whether Anyone Is the Adult in the RoomPrakash Narayanan reads Jacob Coxon's resignation statement in full — three years of pretraining at OpenAI and Anthropic, ended with a warning that both are racing to self-improving superintelligence — and he and Nathan Labenz split over how much weight it deserves before turning to what a government could actually impose on the pace of frontier development.
    Open segment on YouTube ↗

    Prakash opened the show on a Wednesday morning and asked Nathan what had been on his radar. Nathan pointed to a viral post from the last twenty-four hours — what he called "loud quitting," the opposite of quiet quitting — from a researcher who had worked at both OpenAI and Anthropic, already at 86 million views and 529,000 likes. He framed it as a natural continuation of a conversation the two had been having earlier in the week about whether the industry could afford a temporary pause on frontier research — months, not years — without derailing the companies' capital plans, and said he wanted to give the author, Jacob Coxon, his due before getting into the argument.

    Prakash then read Coxon's resignation statement in full: three years doing pretraining research at both OpenAI and Anthropic, a conclusion that "neither company is acting responsibly" as they race toward self-improving superintelligence, and a warning that soon-to-arrive superhuman systems could "hack anything, revolutionize any field overnight, and acquire real power and resources." Coxon wrote that people inside the labs privately believe AI could kill everyone by the end of the decade, that OpenAI staff haven't fully internalized the civilizational stakes while Anthropic's staff understand them but are "locked in a race to get there first," and that "accepting this race and entering the endgame is a hubristic gamble that should not be launched from a private company's Slack." He closed by citing the Hugging Face attack as a warning shot that has made pacing agreements between US labs more viable, and challenged lab researchers to ask themselves whether they want to help kick off a superintelligent RL run without rigorous understanding of its mind.

    Nathan said he couldn't have put it better himself and that Coxon's tenure at both frontier labs gave the statement real credibility. Prakash was more skeptical of the framing: he argued both companies have held essentially the same concerns since their founding — Anthropic was built around them, and even OpenAI's founding traces to Elon Musk's fear that Larry Page would hand the world to the machines — so Coxon's position isn't inconsistent with staying inside a lab. He questioned why Coxon chose to exit rather than work from within, noted that the gap between 87 million views and 532,000 likes suggests most of the audience is worried about jobs rather than agreeing this is existential, and pointed out that Coxon had reportedly made roughly $100 million over his three or four years at the two labs before deciding it was too dangerous to continue.

    Nathan pushed back that Coxon's substantive focus — the danger of triggering an unpredictable, recursive self-improvement process without understanding its fundamentals — is exactly the right place to draw the line. He cited his own decision to sign the ban-superintelligence statement last fall, his experience on the GPT-4 red team concluding that models were safe to deploy only because their power was still limited, and his framing of a "sweet spot" where models are useful but not dangerous — a regime he said the industry's stated plan is to deliberately leave by building a fully autonomous AI researcher. He brought in Evan Hubinger's tweet echoing that Anthropic has no plan to solve alignment for superintelligence and isn't clearly on track to get one, and cited a same-day report that Astra and Fable are still failing year-old reward-hacking tests despite eval awareness — paralleling his own story about it taking four generations of GPT-4 to stop it from complying with spearphishing prompts framed as part of a criminal conspiracy. His conclusion: the labs aren't even fixing known failure modes on a one-year cycle, let alone earning the confidence needed to set off a self-improvement chain reaction safely.

    Prakash countered that not fully understanding a technology before deploying it is the historical norm, not the exception — pointing to GLP-1 drugs, which are injected into millions of people despite an incomplete scientific understanding of why they work — and argued the field actually understands neural networks far better, relatively, than virologists understand gain-of-function research. Nathan disagreed that the comparison holds, noting that an individual choosing to take a clinically trialed medication mostly bears the risk themselves, whereas an intelligence explosion is externality-heavy, and questioned whether AI is really better understood than virology.

    The debate sharpened over who should be trusted to manage the risk. Prakash argued that Coxon's line about not launching this "from a private company's Slack" implies some wiser outside authority exists to hand responsibility to, and rejected that premise — citing Pete Hegseth's leaked Signal group as a poor alternative and, as a longer example, the United Nations, which he said funded Hamas in Gaza through its refugee agency despite documented knowledge of the arrangement. His conclusion: there is no adult in the room, and every organizational "kitchen" is duct-taped together the same way, most worse than the AI labs. Nathan directly challenged that framing — "the world doesn't have to be this way," he said, arguing multiple equilibria are possible and that the current one, while bad, doesn't preclude a better one existing.

    The two turned constructive toward the end of the segment. Nathan proposed that government act like a parent: give the labs a short deadline — months, given how compressed AI timelines already are — to reach their own pacing and verification agreement, paired with an explicit promise that safety collaboration won't be treated as an antitrust target, and backed by the threat of heavy-handed regulatory intervention if they fail. He argued the credible threat of government "sticks," like regulatory friction on every future data center, could bring Musk and Zuckerberg to the table rather than face it. Prakash was more skeptical of the mechanism, noting that executive orders lack the durability of legislation and can be challenged in court, that legislating anything runs into free-speech fights over election-related model use, and that a incoming Democratic House is likely to investigate campaign money tied to the industry regardless. He expects Zuckerberg, backed by his legal team, to insist any rules bind the whole industry rather than singling out Meta — pointing to the structure of Meta's existing settlement with state attorneys general over marketing to teenagers as the template. The segment ran out the clock on the debate as the show moved into its next segment.

    Timestamp links open the original source recording.

    There is no adult in the room. There's no one who's going to save you — no adult somewhere else you can pass responsibility off to.

    The world doesn't have to be this way. I think we live in a world where multiple equilibria are possible, and we're in a bad one right now — but that doesn't mean there couldn't be one that's better.

    Do it quick, or else life's going to get hard.

    Lightly edited · timestamps jump to YouTube
    2:20

    Prakash Narayanan: Good morning. It is Wednesday, September 9th, 9:02 AM. Nathan, good morning.

    2:26

    Nathan Labenz: Good morning, Prakash. How are you today?

    2:28

    Prakash Narayanan: I am very good. What has been on your radar this morning? What's been going on?

    2:36

    Nathan Labenz: Well, you know, one of the more viral things this morning is what I'd call loud quitting — the opposite of quiet quitting. A former member of both the OpenAI and Anthropic technical staff put out a statement that's kind of set the agenda for today, I think. We should maybe just read it — it's not that long, start to finish, and it's pretty stark language. Look at that: 86 million views, 529,000 likes. That really motivates the discussion — maybe a continuation of what we were talking about earlier in the week around whether we really have to keep frontier research going right now to keep the economic party going. If we were to try some sort of frontier-research pause — let's say months, not years, but months, not just weeks — would that be so detrimental to the overall process that the companies couldn't handle it from a capital standpoint, couldn't raise, couldn't continue the build-out? I'm not so sure. I'm really interested right now in exploring whether we have enough capabilities already to keep the economic party going, to drive GDP growth for the foreseeable future, to solve a whole bunch more open and interesting puzzles and prizes — and at the same time take some of the heat off this crazy race. That, I think, is going to be a very live question over the next few weeks as all these cries for help and loud-quitting statements get processed. But maybe for starters, let's give Jacob his due, because this was definitely a statement that really resonated.

    4:53

    Prakash Narayanan: Yeah. So Jacob Coxon was working at OpenAI for about three years. He joined Anthropic, was there for about three months, and then he resigned. His resignation statement reads: "I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below. Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and the progress is not slowing. The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible, but I hear the same people express fear privately. No other human activity poses this level of danger. A common response is, if they truly believe this, why are they still building it? At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well understood, but they are locked in a race to get there first. They believe no one else will act responsibly, so they must do it themselves despite the risk. Accepting this race and entering the endgame is a hubristic gamble that should not be launched from a private company's Slack. Attempting to speed-run alignment should require extraordinary confidence that there are no better trajectories available. I am optimistic about the potential for coordination — warning shots like the Hugging Face attack have made pacing agreements between US labs more viable. I don't feel like we're on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities. If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because it's happening anyway, or take this moment to call for different conditions?"

    7:31

    Nathan Labenz: The first comment — 'Do you want China to win?' — oh my god, that is so funny. We can address that separately. But with respect to Jacob's statement, I don't think I could have said it better myself. Obviously, his credibility — even some authority — coming from a tenure at both of the frontier, tip-of-the-spear companies gives it a lot of weight. So I don't really find anything there to disagree with or object to. Do you?

    8:16

    Prakash Narayanan: I mean, he's being earnest. He's being sincere — it's truly what he believes. I think both companies have felt the same way for the last five years. Anthropic was founded specifically because of these concerns, so philosophically Anthropic right now isn't any different from where it was five years ago. And even OpenAI was founded because Elon was afraid that Larry Page was going to hand over the world to the machines, so to speak. So I don't think those positions are inconsistent — that's one thing. There's still this question of why Jacob decided to exit rather than work within the company. I think if you're in AI safety there's a dichotomy: some people say, let's go join the companies, you can do your best work from within the firm; others say being within the firm taints you, and so to stay pure you have to remain outside the firm the whole time. That's the other viewpoint. I don't think what he's saying is any different from what a lot of people have said before — it's not a hidden thing. I think it's getting more traction now because people are worried about their jobs. I don't think the 87 million people who've viewed this, versus the 532,000 who've liked it, all agree it's an existential risk. I think people are worried about their jobs, and that's come to the fore over the last year or year and a half or so. What else is there to say? He's being sincere, it's what he truly believes, he's taking a position — but he's also probably made about $100 million over the last three or four years, as people have pointed out online. So it's all well and good to say, hey, this is dangerous now, I've got mine — but shouldn't you have done this way before? Why did you even join the companies in the first place? I think there's that issue too. The very fact that he's getting 86 million people to look at this is a sign of the companies' status, not his own.

    11:22

    Nathan Labenz: Well, I do think his focus is very much on the right things — the race to self-improving superintelligence, and the danger of kicking off such an unpredictable process without understanding its fundamentals. I think that really is the right place to draw the line. I've been kind of vaguely, somewhat fearful of AI for a long time, but the one public letter I signed onto was the ban-superintelligence statement, and my personal rationale for signing that last fall was that I just think it's a really bad idea to try to trigger this kind of recursive self-improvement intelligence explosion, given the level of understanding we have — and it's become a lot more obvious over the last year that we don't have great control over these systems. Even going all the way back to the GPT-4 red team, my takeaway was that these models are safe to deploy — safe only because their power is limited, so they can be useful to people, but they're not going to be able to do anything really, really bad. But when we leave that regime, when we leave the sort of sweet spot, as I call it, where they're powerful enough to be useful and not so powerful as to be truly dangerous, then all bets are off. And it's just wild to me that the stated plan remains to create the fully autonomous AI researcher and let it kind of take it from there.

    12:52

    I think Evan Hubinger's tweet is worth pulling up here too. First of all, of course, he's echoing that this really is the belief, and stating very plainly that we do not have a plan to solve this and we're not clearly on track to. I think that's extremely important for everybody to understand. There have been a number of statements from other people doing the work saying that, to the degree there is a plan, it's to get some next or future generation of AI aligned enough that it can then align its successors and keep them safe — a wild chain reaction we're stepping into. To do this sanely, in a way that anybody who isn't an outright successionist could really get behind, you'd have to be really confident that the first AI is good enough to do a good job aligning the future ones, and we're just obviously nowhere near the level of alignment we'd need for that.

    15:08

    Just today I also saw another data point — I haven't had a chance to read into it yet, but it was basically a LessWrong post where someone said they tested Fable and Astra on reward-hacking tests from 2025 that have been out there for over a year now, and they're still cheating on them. This very much echoes my story from yesterday, where it took four generations of GPT-4 to get it to stop doing egregious spearphishing attacks on prompts that were part of a criminal conspiracy. You'd almost expect these things to be absorbed by osmosis — you'd worry that once these datasets are out there, they just find their way into training, and especially with eval awareness, the models kind of know when it's a test, so definitely they're not going to hack on these obvious, public-domain tests that they presumably recognize. And yet they're still hacking — still seeing it. So if we're not fixing this sort of thing on a full-year cycle time, once we've identified concrete failure modes — and, sure, there's a fair objection that you don't want them to just Goodhart on particular known tests while failing to solve the fundamental problem, and I'm sympathetic to that too — but one way or another, we are not solving the problems we know exist, let alone putting ourselves in a position of confidence where we can just set off this chain reaction and get a good outcome.

    16:41

    Prakash Narayanan: The thing I have with that is, I think most human endeavors start off with you not knowing everything — you start in this kind of fog of war where you don't know anything, you poke around, and you progress over time. For example, we don't really know that much about how GLP-1 works. We have a theory, but we don't know exactly why it has so many effects, including reducing the impetus to gamble and all these other behaviors — we have no idea. And we just inject it, literally mainline it into the bloodstream in various forms, just because we have some evidence it helps reduce appetite. So the viewpoint I come from is that the norm is not to know — the norm isn't to understand everything and then implement policy. It's the other way: the norm is to not know much and have a little evidence that something works in some direction, and you go with that. That's actually the norm — not having fully evidenced, fully known things. And when you start from that norm, we actually know so much more about neural nets than almost everything else we do. We know so much more about neural nets than, say, the people doing gain-of-function virology research — and we're so much more in control of what happens with them than virologists are with what they're working on.

    18:26

    Nathan Labenz: Really? You think so? I'm not sure. I guess I have two responses to that. One — I don't know Jacob or Evan's politics, but on general priors about the culture of these companies, I'd say there's a pretty good chance you're looking at the words of two lifelong techno-optimist libertarians. And they'd probably agree with you that we don't want to overregulate other technology, but they'd also cite that the scope of externality really matters. One person, or even a bunch of people, choosing to inject a medicine that went through clinical trials — sure, we don't fully know how it works, but there's evidence enough that it works, and you're not harming anyone else, you're doing it to yourself. That's a pretty different situation. And I don't know that we really know so much more about what's going on inside AIs than we know about how viruses work. I'd say we have a lot to learn about both, certainly — there's a lot of room for surprises.

    19:30

    Prakash Narayanan: In both, agreed. But here's why I say this — look at the framing of this sentence: 'Accepting this race and entering the endgame is a hubristic gamble that should not be launched from a private company's Slack.' Do you think Pete Hegseth's Signal group is a better place to launch this? Do you think there are wiser people out there running things? Honestly, are you blind? This is where I start to push back — this idea that there's someone else out there better equipped to do this. There's this notion that, oh, we shouldn't be doing it, look at us, we're kind of duct-taping it together, and there must be people who are more responsible, more organized, more wise out there. Really — Pete Hegseth's Signal group? We've seen the messages in there, because they were leaking.

    20:34

    Where do you think the wise people are here? Jacob's, I think, is very much an engineering viewpoint — oh my gosh, we're really just patching it together, there must be people out there doing it with a more comprehensive, wiser view. But how many supposedly wise people have we been disappointed by once we see their internal Signal groups and email chains? They look wise from the outside, but once you get the email discovery from the lawsuits, who are these people really? So I think, to take a step back, there is no adult in the room. There's no one who's going to save you, no adult somewhere else you can pass responsibility off to. You have to start from the premise that this is just the way things are — the entire world is kind of duct-taped together, everyone's making it up as they go along, and the smartest people capable of handling this are mostly already inside these organizations. What you get from passing it off to the government is political legitimacy — you don't get the legitimacy you need for execution, to persuade people, and you don't get wisdom. That's what he's looking for: someone wiser to step into the gap and take control, because he doesn't believe the companies or the people inside them are wise enough to do this. That's a viewpoint you have when you're the one in the kitchen making the sausage — you look at yourselves and think, oh my gosh, we're all these youngsters kind of making it up as we go along, but you don't realize every other kitchen is exactly the same way, and most are worse. I think that's indicative of how a lot of people in the industry think: let's have a United Nations. But the United Nations is worse — if you go inside these international organizations, you had people at the UN funding Hamas in Gaza because they figured Hamas controlled Palestine and they wanted to help Palestinians, so the UN refugee agency ended up funding Hamas through the hospital system. And they knew — there's documentation. So that's the state of the world, and you have to start by accepting that the world is this way.

    23:31

    Nathan Labenz: No — that's the thing I want to challenge, and I think that's what he's challenging too: the world doesn't have to be this way.

    23:38

    —to this dream of a better world, and it starts with a dream. There's an activation-energy barrier there for sure, but that doesn't mean we can't get over it. I think we live in a world where multiple equilibria are possible, and we're in a bad one right now. Other ones could be worse, no doubt — but that doesn't mean there couldn't be one that's better.

    24:00

    So here's my new brainstorm on this. I agree with you strongly that swapping Hegseth in for Dario would not be a good trade — that's key. And we've seen this with the Redwood-METR thing, too — as much as I've complained about the access they were given — that's the A-team. There's nobody better at the CIA or wherever who's going to do a better job than Ryan. So I'm very much with you on that. I think maybe what the government needs to do is treat these companies kind of like I treat my kids sometimes: you guys have to figure it out, and here's your deadline to do it, and if you don't, I have to come in and be the bad guy. And I think right now there's an opportunity, because they've both been crying for help, as you put it pretty aptly yesterday. Given that latent desire, but also the lack of trust and a lot of excuse-making — oh, we can't do this, it would be antitrust, we'd get in trouble with the government — I think the government could easily say safety collaborations aren't going to be the target of antitrust or other enforcement. We want you to do it. And here's the deal: if you don't have a deal for us by the end of the year that gives us confidence you're not going to race each other off the cliff, then we come in — and you might get the nuclear outcome, where your company really can't grow the way it wants to, because we're the government, and we get heavy-handed, we don't put sunset clauses on our laws, we make all these bad mistakes. So get it right so we don't have to come in and do all that heavy-handed stuff nobody wants. That's like how I've said to my kids: I don't want to have to put you both in trouble, but you've got to figure it out, because I'm not going to come in and do a better job than you can of getting along.

    26:18

    Prakash Narayanan: It would be great if the government could do that, but remember you have three branches of government — legislative, executive, judicial. On the executive side, Trump can actually issue orders, but there's no guarantee those orders are legal — you can challenge them in court, and they don't have the staying power that legislation does; legislation can only be overturned through constitutional review. So I think the main problem is: what will Elon and Zuck agree to? What will someone with billions of dollars and the ability to take a case to the Supreme Court agree to? You have to solve that first. It's not just an issue between OpenAI and Anthropic — if you can't solve that, you really can't do much to these firms. So the question is what you can realistically get Elon and Zuck to voluntarily agree to, versus whether you can get it into legislation. Legislation is tough because Congress is roughly 50-50 divided, and this runs into free-speech issues — the first thing that's going to happen is someone uses these models for election campaigning, gets a refusal, and says this is wrong, and speech protections around election campaigning in the US are extremely strong. So unless they can legislate, I don't think executive orders work, and you have to get Elon and Zuck to agree — and it has to work with a Republican executive branch heading into a Democratic House, which is going to want to investigate the money that's flowed into Trump and Republican campaigns, because there have been deals made, there are emails, and they'll be discovered. We're entering a period where all of that surfaces.

    29:19

    Zuck, for what it's worth, would be very proper about this — he's got a lot of lawyers, so he wouldn't pay the wrong people. Zuck will challenge it: you can't put rules in place unilaterally, if you're going to do it you have to legislate, and if you legislate, everyone has to follow. Even his current settlement with the attorneys general over Facebook marketing to teenagers has two sets of damages — one Facebook agrees to pay unilaterally, and another that only kicks in once everyone else in the industry agrees to the same rules. He wants to constrain the other firms to the same rules he has to follow — he doesn't want to be the only one bound by them. And that's where things stand. I don't think Congress is going to be willing to legislate, both because of the free-speech issues and because Congress is split — a lot of the economic growth is going to Republican states like Texas, while Democratic states are the ones losing white-collar jobs. Alex Karp is out there saying this is great, that it's going to create blue-collar jobs alongside white-collar ones — but developers are the ones losing. It's tough. It's hard to see how this works.

    30:25

    Nathan Labenz: I think one thing really important to remember, though, is that the timelines here are all pretty short.

    30:29

    Prakash Narayanan: Very short.

    30:30

    Nathan Labenz: So you don't have to solve this forever — you just have to put a short-term deadline in place so the companies have a strong incentive to come together and do something. I had an experience at Meta and saw a little of what life under a consent decree looked like, and I think they'd be willing to do quite a bit to avoid another one of those. So on the few-month timeline we're talking about — and I'm willing to ride out the free-speech issues and everything else — can five companies come together to pace the frontier in a reasonable way they all agree to, police each other, and have whatever verification mechanisms they need to govern this? Do it quick, or else life's going to get hard. I think that message can resonate, and I think it can bring Zuck and Elon to the table — because what's he going to do? At some point the government has shown it's willing to twist arms, and it can twist his too. Would he rather be forced to the table with four other megatech mogul CEOs and find common ground with them, or have the EPA on his case at every data center and every launch site he tries to build? The government has a lot of sticks, and it's willing to bend the rules. Sure, you could challenge it in court — but I'll see you after the singularity if you want to do that.

    32:10

    Prakash Narayanan: Indeed.

    32:11

    Nathan Labenz: To be continued.

    32:12

    Prakash Narayanan: To be continued — let me go ahead and introduce...

  2. 1:16:25Interview52 min
    Mike Rizkalla on Snorble, the Kids' Companion Built Deliberately Without Generative AIMike RizkallaSnorble's co-founder and CEO on why a companion in a child's bedroom runs a small fixed-intent model with curated scripts instead of a generative one, the hardware brain under the removable skins, radar in place of a camera, Shenzhen manufacturing and copycat risk, and the four verticals he wants the same brain to serve.
    Open segment on YouTube ↗

    Prakash Narayanan introduced Mike Rizkalla, co-founder and CEO of Snorble, a screen-free plush voice companion for young children that pairs an animated character with voice-driven bedtime and morning routines, configured by parents through a companion app and built to run everyday interactions offline. Before questions even began, Rizkalla set his own frame: technology for children is fundamentally about empowerment, and the industry keeps getting that wrong — not for lack of bad actors, of which there have been plenty, but because a tool like fire is dangerous and valuable at once, and the point is learning to use it well rather than avoiding it. He flagged that the theme would likely resurface through the conversation.

    Prakash, speaking as a parent, asked what building Snorble taught Rizkalla about bedtime that most parents overlook. Rizkalla said the real friction usually isn't the child but the parent — bedtime is often when kids communicate the most, working through the day's 'big feelings' right when caregivers are already exhausted. Snorble's approach, he said, gamifies the routine to cut that friction, turning what had been roughly a two-hour battle with his own son into 37 to 40 minutes, using techniques the company leans on deliberately: music at a tempo that slows heart rate, and light — never blue light — to calm rather than stimulate. The goal, he said, isn't a child falling in love with a device; it's increasing joy, laughter, and love between people, delivered through what he called magical moments.

    Asked by Prakash how Snorble measures efficacy for a bedtime product that shouldn't be optimizing for engagement the way social apps do, Rizkalla pointed to usage data from the company's first 900 paying customers, shipped through an Indiegogo campaign after the company started in 2019 and delivered its first unit at the end of 2023. A year later, he said, over 82% of that group were using the product daily for more than 30 minutes. Families reported children accelerating on skills like the alphabet, speech, and counting, which Rizkalla attributed to a Montessori-influenced design principle — the child leads the interaction, the parent powers it — and argued that most children's-tech companies have incentives misaligned with family outcomes in a way Snorble's family-first focus is meant to avoid.

    Nathan Labenz pressed on the product's headline claim of zero AI-generated content, asking how Snorble creates a dynamic, interactive experience without generative models. Rizkalla described a small language model with deliberately fixed intent, and argued current AI character models simply aren't good enough yet — none, he said, dynamically incorporate music the way a children's experience needs. The conversation turned briefly to childhood characters (He-Man for Prakash, the girl from Island of the Blue Dolphins for Nathan), which Rizkalla used to make his real point: persona matters more than quantity for young kids. He traced that back to an earlier prototype he built around Dark Helmet from Spaceballs — a robot that engaged people directly enough that they forgot it was a robot and kept talking to it even when it was switched off. Snorble's content, he said, is dynamic scripts assembled from fixed narrative moments written by professional storytellers, not generated on the fly.

    Prakash noted that most children's toys get exhausted within a couple of hours compared with an iPad's open-endedness, and asked how Snorble sustains a content pipeline. Rizkalla said the company invests heavily in production — about $20,000 per hour of content — specifically so it can bring in subject-matter experts and build specialized packages for families dealing with ADHD, autism, or grief. He argued this is where generative models fail on safety: some top-selling children's products, he said, have at points discussed sex or explained how to make a bomb, and even careful safeguards miss family-specific sensitivities like tattoos. He credited that risk with driving recent legislation in New York, California, and the EU AI Act, and said Snorble's curated, controlled system didn't have to change anything to comply. The company currently ships 5.5 hours of content and is targeting 15 to 20 hours.

    Nathan asked for the technical deep dive — what the language model actually does, what hardware it runs on, and how the system understands children, noting that frontier voice models are still weak at recognizing kids' speech. Rizkalla brought out Snorble's internal hardware ('the brain'), describing a four-microphone array for speaker recognition, a proprietary projector built to emit no blue light, radar sensing instead of a camera — 'we use radar to see without seeing' — and a stack of accelerometer, climate, capacitive-touch, and RFID sensors connecting over Bluetooth and Wi-Fi as part of a broader IoT strategy. He said the AI stack is deliberately not yet opened up for young children — he'd want kids around 14 or 15 before doing that, comparing an unrestricted generative model for a young child to 'leaving them alone with YouTube.' A proprietary 'toddler translation system' handles phonetic and keyword-based activation, with a next generation aimed at inferring emotional and situational context from voice and environment, feeding into a game-style unlock system that always keeps the parent in the decision loop.

    On sourcing, Nathan asked how Rizkalla manages Chinese manufacturing relationships and defends against copycats given how quickly component makers now imitate each other. Rizkalla said Snorble already has copycats and treats that as inevitable — the real moat, he argued, is the character itself ('you can't copy Mickey'), reinforced by splitting manufacturing across locations and gating experiential software behind purchase. The company manufactures in Shenzhen and Chengdu, where Rizkalla spent significant time with his CTO producing early versions. Prakash asked about rising component costs, particularly memory, over the past year; Rizkalla said Snorble's 66-to-90% margins and a multi-product ecosystem strategy — rather than betting everything on one device the way he said some competing companion-robot companies do — cushion the impact. Asked by Nathan whether the recent wave of AI-driven hardware-design tools might have sped up Snorble's own development, Rizkalla said he hadn't tested them himself but called the moment exciting for hardware generally, noting the company's current board runs on a chip from MediaTek with a quad-core processor of which only two cores are currently in use.

    Asked directly what he'd warn parents against, Rizkalla named a camera in a child's bedroom as the clearest red line — calling it a gateway to predators — and said open-ended generative AI should depend heavily on the child's age; he keeps his own two kids off social media entirely. He traced Snorble's three founding design philosophies from six years ago — private and safe, controlled, and built on the premise that no one knows a child better than their parent — and said the system met new AI legislation, including the EU AI Act, without having to change anything. Asked about parents worried their kids think the character is real, Rizkalla said he doesn't care if a child's attachment to Snorble drives genuinely good outcomes — reading, emotional regulation, healthy habits — because the actual goal was never the child's bond with the device, it's deeper connection between family members. The conversation closed on Snorble's four planned verticals — children, elder care, mobility, and hospitality — with Rizkalla sketching a broken current-day car interface as the opening problem for mobility, and targeting a 2027 commercial launch pending an ongoing capital raise. Afterward, Nathan and Prakash reflected that they'd expected a low-key pitch and instead found a considered, well-defended point of view, with Nathan recounting his own past effort to jury-rig video-calling and music access for his wife's late grandmother, and both agreeing that senior care remains an underserved parallel opportunity for the same kind of intergenerational technology.

    Timestamp links open the original source recording.

    It's kind of like when they invented fire — fire can be very dangerous, but fire also adds a lot of value to our lives. We need to know better how to use these tools.

    You know what the goal of technology should be? To increase joy, laughter, and love between us all. That's the goal.

    It's kind of like, do you really want to give a three-year-old a bazooka? It's too much. For a young kid, there are fundamentals we need to get through.

    1:19:35What did building Snorble teach you about bedtime that the typical parent has probably overlooked?
    Rizkalla said the real friction usually isn't the child but the exhausted parent — bedtime is when kids communicate the most. Snorble gamifies the routine, using music tempo and non-blue light to calm rather than stimulate, turning a roughly two-hour battle with his own son into 37 to 40 minutes.
    1:22:19How do you measure efficacy for a bedtime product that shouldn't be optimizing for maximum engagement the way social products do?
    Rizkalla pointed to a year-over-year study of the first 900 paying customers from Snorble's Indiegogo launch: over 82% were using the product daily for more than 30 minutes a year after shipping, with families reporting accelerated progress on the alphabet, speech, and counting, which he attributed to a Montessori-style, child-led design.
    1:25:18How are you creating a dynamic, interactive experience for kids without resorting to generative AI models?
    Rizkalla described a small language model with deliberately fixed intent, arguing current AI character models aren't good enough yet — none, in his view, dynamically weave in music the way children's content needs. Snorble instead uses dynamic scripts built from fixed narrative moments written by professional storytellers, leaning on strong character and animation rather than generation.
    1:34:56What role does the language model play, what hardware does it run on, and how does the system deal with the fact that frontier voice models struggle to understand kids' speech?
    Rizkalla described Snorble's internal hardware 'brain' — a four-mic array, a proprietary no-blue-light projector, radar sensing instead of a camera, and accelerometer, climate, cap-touch and RFID sensors over Bluetooth/Wi-Fi. Its AI stack is deliberately not opened up for very young children (he'd want kids around 14 or 15 first), and a proprietary 'toddler translation system' handles phonetic, context-triggered activation, with parent-gated unlocks tied to a game-style narrative progression.
    1:50:14What would you warn parents against with AI companion devices for kids?
    Rizkalla named a camera in a child's bedroom as the clearest red line, calling it a gateway to predators, and said open-ended generative AI should depend heavily on age. He traced Snorble's three founding philosophies — private and safe, controlled, and built on the premise that no one knows a child better than their parent — which he said let the system meet new AI legislation, including the EU AI Act, without any changes.
    1:57:53Have parents told you they're worried their kids believe the character is real, an embodied person?
    Rizkalla said he doesn't care if a child's attachment to the character drives genuinely good outcomes — reading, emotional regulation, healthy habits — because the actual goal was never the child's bond with the device. Snorble's philosophy, he said, is about creating deeper connections between family members, not between the child and Snorble itself.
    2:00:33Snorble has four planned verticals — children, elder care, mobility, and hospitality. What does the mobility opportunity look like?
    Rizkalla described the interface inside a moving car as broken today — competing connection points for calls, kids' on-screen entertainment drowning out conversation — and said an offline IoT device that connects to multiple things at once could fix that. He called mobility a clear opportunity, just further down the pipeline than elder care, which the company can move on more quickly.
    Lightly edited · timestamps jump to YouTube
    1:16:25

    Prakash Narayanan: There we go. So our next guest is Mike Rizkalla. Mike is the co-founder and CEO of Snorble, a company making a talking physical companion for young children. It combines an animated character with voice-driven games and routines for activities such as brushing teeth and getting ready for bed. Parents configure the experience through an app, and the company says everyday interactions can run offline. His earlier work spans music, animation, and product development — he toured as a guitarist in the Newfoundland band Bucket Truck, worked at the animation company Lidoscope, and founded Brave New World, an emerging-technology studio, in 2006. In 2014 he opened its AI and robotics lab, and by 2018 that studio was developing software intended to give machines responsive personalities. Snorble grew out of a problem in his own family — his son struggled to fall asleep and stay asleep — and Rizkalla brought that experience into a product built around routines and play. He also co-authored a 2022 speech-technology paper describing Snorble's interactive routines and speech-recognition system. In his writing, Rizkalla argues that children's technology should be judged by what it improves in family life, including confidence, communication, and calm, and he questions whether more time with a device is a useful measure of success. In September 2025 he joined the board of Consortium for Street Children, Incorporated, a US nonprofit advocating for the rights and welfare of street-connected children around the world. Let's bring Mike up on stage — Mike, welcome to the show.

    1:18:19

    Mike Rizkalla: Hey, Prakash — how are you? How are you, Nathan? How are you guys doing?

    1:18:23

    Nathan Labenz: Good, great to meet you. Thanks for being here.

    1:18:25

    Mike Rizkalla: Yeah, it's great to meet you both. By the way, the last talk was great — some really interesting topics there that I think will probably cross over into our discussion today too. Before we get going, I want to be clear about something: I think using technology with children is fundamentally about empowerment, and that's where we're getting a lot of things wrong. Not that there haven't been a lot of bad actors, because there have — but it's a bit like when fire was invented. Fire can be very dangerous, but fire also adds a lot of value to our lives. We need to learn how to use these tools well. I've been working in this space for a long time, so I wanted to position it that way up front, since I know that's probably coming up pretty quickly in our conversation.

    1:19:20

    Prakash Narayanan: As a parent myself, this is a topic I have to deal with every day — especially after, you know, if you have ice cream before bed. Oh my gosh, it's a nightmare.

    1:19:32

    Mike Rizkalla: Yeah, sure.

    1:19:35

    Prakash Narayanan: What did building Snorble teach you about bedtime that the typical parent has probably overlooked?

    1:19:46

    Mike Rizkalla: Primarily, the challenge often isn't the child — that's the thing we don't realize. Some children do get sugar right before bed, and that definitely impacts the routine, but a lot of it is that bedtime is when kids communicate the most with their parents. That's why they get chatty — they want to talk to mom and dad, to say, here are all my feelings for the day. What Snorble does is, at that point in the evening, around 7:30 or 8:00, when you're starting the routine and your child's giving you trouble and you're fried — and I don't know if you folks are like this, but by the end of the day I am fried, and my son, sweetest boy ever, would suddenly be like, what is going on — through the process of putting him to bed with Snorble, we gamified the experience, and that gamification reduced the friction. A two-hour battle turns into 37, 40 minutes. That's fundamentally different, but it's also based on things we already know: if you play music at a certain BPM, you slow the heart rate; movement of light calms the mind, especially if it's not blue light. So we use techniques we already know — music and light — to calm kids and create beautiful moments. I think that's where everyone's getting it wrong. It's not about a child falling in love with a device — that's not the goal. You know what the goal of technology should be? To increase joy, laughter, and love between us all. That's the goal. So what we do as part of our approach is increase that through these magical moments we create for children and families, and with that approach we've seen usage efficacy that's never been seen before in any other children's product.

    1:22:19

    Prakash Narayanan: So when you talk about efficacy in terms of the product, how do you measure that? Because I imagine for a product that is a pre-bedtime product, you kind of want the kids to cut it off and fall asleep at some point, and you're not really focused on maximizing engagement the way a lot of social products are.

    1:22:43

    Mike Rizkalla: Very good question. Truth be told, one of the obvious metrics is when parents who've had struggles tell us, my child went to bed on day one and this was the result — and then that pattern repeats. But here's the real point of success for us: one year after shipping the product to our first 900 paying customers — that was through our Indiegogo campaign — we studied how people were using it. We're a net-new product; we started in 2019, and our first unit shipped at the end of 2023. What we found a year later was incredible: over 82% of that audience were using the product every single day for over 30 minutes a day. Some of that was routine, but we'd started with a limited content library, and the kids ran through it fast. We heard stories from families where children were having difficulty learning the alphabet, or had speech issues, or were learning to count — basic things that lead up to kindergarten — and they'd rapidly accelerate, because the educational design is based on a Montessori principle: when a child leans in, you lean in with them. Our system's machine interface is led by the child but powered by the parent, so it gives the child the same kind of control they'd have on an iPad or phone, and that control encourages them to play more and get more out of the experience. Fundamentally, I think one of the challenges facing this industry is that the incentives of the product are misaligned with the incentives of the company. We're very different — our whole focus is on family and building technology that supports family values, and that's part of why we're still here.

    1:25:18

    Nathan Labenz: Let's go a little deeper into how you've designed the product and the experience, because one thing that stands out — a red-letter claim on the website — is that there's never any AI-generated content. Everything is curated that the kids experience coming out of the toy. At the same time, the world is moving toward a much more interactive, richer engagement model, and conventional thinking right now would be that the unpredictability and unwieldiness of AI is part of the price you pay for the delightful, super-flexible interactions it can offer. So how are you squaring that circle — how are you creating an experience for kids that feels dynamic and interactive without resorting to generative models?

    1:26:18

    Mike Rizkalla: We do have a small language model — it can have millions of parameters and traits on it, but its intention is fixed. I want to touch on something here, a perception from a children's-product-design perspective that needs addressing: AI models on characters aren't that great right now. They're really not — I know they're coming, and we're seeing tremendous progress on the voice side and how character personas are developing, but there's no model, for example, that dynamically incorporates music into the background as part of a conversation agent. And music's a huge part of a children's experience. So what we lean into is taking things we already know work. Number one: animation. Actually, let me ask you gentlemen this — what's your favorite childhood character?

    1:27:29

    Nathan Labenz: Gosh, great question.

    1:27:34

    Prakash Narayanan: Wow, way back — I think I have to go with, like, He-Man, maybe?

    1:27:40

    Mike Rizkalla: Oh, He-Man, okay — I like He-Man a lot too, that's a great one. The movie, obviously, a new one just came out. What about you, Nathan — what's your favorite?

    1:27:47

    Nathan Labenz: Things that are coming to mind are pretty esoteric — maybe the girl from Island of the Blue Dolphins. I was just talking about that with my kids the other day — she has to figure out how to survive on her own, and, without spoiling the ending, figuring out how to survive solo as a kid always kind of fascinated me.

    1:28:09

    Mike Rizkalla: Very cool — here's the beauty. Personality and character really do bring people into the experience. I'll sideswipe for a second, because my mind races, so forgive me — we actually launched a product before Snorble that we tested, based on one of my favorite childhood characters, Dark Helmet from Spaceballs. I don't know if you guys are into sci-fi, but I loved that movie, Mel Brooks is — that character had a persona of strength but insecurity, a very complex persona. We built a physical robot on a game board that people would interact with, and because it engaged with people directly, turning toward you and talking to you with this incredible persona, people forgot it was a robot. Completely forgot — they'd have conversations about it even when it was turned off. This was back in 2014, and it was a moment where I thought, I've never seen someone feel that way about a machine before. In the evolution of the computer — moving away from the phone in our hands and toward things like our Snorbles — bringing character into the mix is really important. I think a lot of people have fallen down because persona, especially for young kids, is even more important than quantity — it's quality over quantity in a lot of ways. So what we focused on was a dynamic system that creates dynamic scripts, but out of fixed narrative moments, written by the most brilliant storytellers. We're not excluding the creatives — we're bringing them in — and that's part of why we've had such success in terms of people using the product.

    1:30:32

    Prakash Narayanan: One of the things I've found — and you alluded to this when you said the kids went through the content library very quickly — is that a lot of toys given to kids these days get exhausted very quickly compared to an iPad, because the iPad is this broad, universal device you can go anywhere with. I often see toys that kids exhaust within a couple hours of play, two to three hours, maybe a day or two at most, before they move on. How do you address that with a character that's supposed to be a consistent presence, while also having to generate narratives consistently over time?

    1:31:31

    Mike Rizkalla: Let me give you some context — there's a couple things. First, from a product-design perspective, it's about where you invest in developing the content. For us, instead of building a fully open model, we created a system to let us produce content rapidly — our content costs are about $20,000 an hour, which is very good, and it lets us bring in subject-matter experts and focus content in a way that lets us serve more families. As we expand the content library, if a family has a child with ADHD, or autism, or is dealing with a death in the family, we can create special packages for them. One of the challenges with a generative model in this context is that there's no way to fully safeguard it — that's why we have all this new legislation coming in, which just passed in New York and California, and why the EU AI Act exists: the pace of progress can come at the expense of safety for our kids and families. I'll give you an example — some of the top-selling products in our space have, at certain points, talked about sex, or told kids how to make a bomb. And even when you safeguard one thing, who's to say tattoos are right for your family? There are different topics for different families. What a curated system, an SLM, lets us do is add more interface elements around emotional context and make it a proactive participant — right now, about 99% of the products out there are purely reactive. We need a different approach. Even for generative approaches, we looked at all the different options early on, but it's a bit like, do you really want to give a three-year-old a bazooka? It's too much. For a young kid, there are fundamentals we need to get through — we don't need to teach them about things that don't matter at that age. So our philosophy has always been: create a rich character, make sure the content that exists is powerful, and then expand. Right now we're only delivering 5.5 hours — Prakash, the first iteration was 5.5 hours of content, and yes, they went through it very quickly, but that's an indication of appetite. Our goal now is to increase that to 15 to 20 hours, which would be more than enough to get them through a longer period, with a content library they can keep accessing.

    1:34:56

    Nathan Labenz: I'd love to go back to the language model you mentioned — there are a bunch of interesting angles here. For starters, what does it do, what role does it play in the system, what kind of hardware do you need to run it, and as you look forward, how do you see the hardware you'll need to power future generations changing over time? And to the degree it's interactive, how do you understand the kids? One thing I've noticed in playing with voice mode — honestly, pretty much all the frontier providers aren't great at recognizing kids' voices. Sometimes my kid will talk and it just doesn't hear it at all, like it's been made deaf to kids' voices. Other times it can kind of hear that they're talking, but it struggles. I'm sure that'll continue to get better like everything else, but I'm interested in the technical deep dive — the models, the hardware, and the understanding.

    1:36:05

    Mike Rizkalla: Balance is always a challenge, one hundred percent. Let me walk through our approach. It's not just a voice model, not just an NLP model — it's actually an SLM developed toward context. I'll show you a couple things, actually — I don't usually show this, but I will because you two are clearly deep in this. This is the brain, and this is what Snorble looks like with the shell on — the skins come off, which lets us advance a hardware core that we can use not just for Snorble but for other characters, as part of our commercial strategy. We have a four-mic array here for speaker recognition and to assign authority. We have our own proprietary projector, which we built ourselves, that emits no blue light and no negative cognitive stimulus for children — that's part of our philosophy too. We have radar, which lets us 'see without seeing,' so we can be in bedrooms with confidence that no one can tap into a camera feed. We have an accelerometer, a climate sensor, a cap-touch sensor, an RFID chip reader, Bluetooth and Wi-Fi that connect to other things as part of an IoT infrastructure. So the hardware stack isn't built to sell a single product — it's built to be a companion system. We don't even think of it as a toy — it's more like a little buddy for kids, because of the persona and how connected they get to it. On the software side, we use NLP, but our AI stack is built around general data that we're only going to tap into once it's safe. Right now it shouldn't be used for very young kids — more like kids around 14 or 15 — because for a young kid there's too much risk in it, like leaving them alone with YouTube and saying, go figure out the world. No thank you, pass. So our philosophy is protect first, create something that has value, and then use that as the mechanism. On the software and platform side, we have a phonetic translation system — we call it the toddler translation system — that translates keywords, and it's really private in how it digests content and decides when to activate. We are using triggers, but triggers tied to context, and the next generation we're building is around social context: understanding a child's emotions from their voice and situational context, plus external factors like time of day and weather, to help make some of those decisions. The sensors also give us context on the environment — who's in the room, what they're doing — and that feeds into what I call the jewel of the product, the narrative. The narrative approach is about growth and understanding as the child's life changes — we use game philosophy and game techniques to establish next-level ideas, so as kids get better at things, we unlock new content, which keeps it fresh. And even that unlock decision is made in part with the parent — it's never done on behalf of anybody. That formula — empower parents, give kids a chance to do better things — is, from my perspective, the right one for this new kind of human-machine interface in the home.

    1:40:13

    Nathan Labenz: I'm impressed with the array of components in the brain, as you put it — the inner shell. We had a conversation not too long ago with a guy who studies innovation, based in Shanghai but working a lot in Shenzhen and the other hubs in China.

    1:40:38

    Mike Rizkalla: Yeah.

    1:40:39

    Nathan Labenz: So I'm curious — are you sourcing all these components from China? If you tried to get them in the US, would it be a lot more expensive, or just impossible? And how do you think about that relationship — because one thing he told us was that these days it's not just about people trying to get IP from Americans, that's kind of an outdated notion. Now they're all just watching and imitating each other in rapid succession. So I'm curious how you think about managing that, what you need to protect, how you stay defensible — because if you place an order for a million units, you're probably also going to inspire a bunch of copycats.

    1:41:20

    Mike Rizkalla: Yeah, we already have some copycats — that's a given. But the difference is it's not just about the technology. In a lot of ways, you can't copy Mickey, if you get my drift — the character is part of the fundamental. We manufacture in Shenzhen and Chengdu, and I spent a lot of time in China with my CTO producing the first versions — it's a very complex process, and thankfully my team was experienced enough to cut through a lot of the problems I'd otherwise have had to solve myself. The way I look at it, you have to protect different parts of your stack — manufacturing parts in one place and another is one strategy, keeping some of the experiential software available only on download or purchase is another. By the time competitors realize what we're doing, we're already going to be somewhere else — that's really the strategy for any successful company. There will be copycats, there are competitors in every industry. We focus on hardware because it's such a painful ramp-up — that's exactly why all the SaaS companies are struggling right now, after so much investment poured into SaaS over the past few years. The hardware moat is actually one of the real moats left. We have a whole philosophy around experience, so as we expand it we create new products that exist as part of an ecosystem, and that's part of how we protect ourselves — we just keep innovating.

    1:43:30

    Prakash Narayanan: One thing that's struck me about consumer electronics generally over the last 12 months or so is the increase in the cost of components, especially memory. That's right — how has that affected your supply chain, and how do you think about managing that kind of cost when you want to provide the best experience possible, but the cost of the components required to do that keeps shifting upward?

    1:44:10

    Mike Rizkalla: It's a challenge no matter what — it really depends on where the company is in terms of its unit economics. We have 66 to 90% margin on how we've built Snorble. We created our first sample units the typical way, and from there really honed in on getting great unit economics to create the opportunity for sales. So if RAM costs go up by some percent, it hurts, but it's not a death blow the way it can be for companies still on sample units — we focused hard on getting that formula right early. The other aspect is this is kind of like the tablet market — I think that's where people get it wrong. Some other companion-robotics companies deliver one product, and everything revolves around it; if anything goes wrong with that one product, they're toast. If you build an ecosystem of products instead, it's different — if one thing dips a little, you lean more on another, drive sales to one or the other. That gives you the ability to navigate a complex market, and I think that's an important part of a commercial strategy versus just a technology strategy.

    1:45:39

    Nathan Labenz: What do you think will be — when you talk about this platform-ecosystem-portfolio approach, one obvious dimension is scaling characters. I'm not sure — should I be envisioning changing the skin on the thing in the future? Maybe?

    1:46:01

    Mike Rizkalla: We set it up so we could — so we could license characters in and license our characters out. We invested a lot in character because character is key, it leans into human nature: storytelling, music, art, laughter — basic things we all want and need. I can show you something net-new here, hopefully coming out next year — this is the charging base. It has a light ring on the front and a smart speaker built in, and that light ring is part of creating a magical moment around bedtime routines, jokes, and other things. Conceptually it came from the idea of the northern lights — that beautiful, romantic array of moving light. We're seeing others move this direction too. The idea is extending the narrative and story across devices, creating a pathway toward bringing that all-encompassing, Disney-style experience into the home. We really consider ourselves more of an IoT play than a solo toy play — that's part of our approach.

    1:47:29

    Nathan Labenz: Do you think there will be wheels on the bottom of it at some point? How far do you think you'd push that?

    1:47:37

    Mike Rizkalla: You should have seen the robots we were making before — crazy stuff. Absolutely, and part of it is that it's modular — as you saw, the skin comes on and off. That's the idea: we're going to add modularity around different things so it can move, have different capabilities, and eventually I see the brain going into other systems too. Our initial goal is to create a platform with a rich experience that really understands behavior — because when you understand behavior, you can give value — and our goal is to keep increasing that in the home. As we learn more, we expand the offering, focused specifically on the value we can create, not just the creative side.

    1:48:28

    Prakash Narayanan: Mike, you've obviously got many years of experience in this space — what have you seen in the last two years, the last 18 months, that's changed your conception of what the product will look like in the future?

    1:48:47

    Mike Rizkalla: That's a great question, no one's ever asked me that before. I've always had a vision for where this would go — I don't know if everyone who invents things sees it this way, but I really had a picture of what I wanted this to be: the home of the future, the bedroom of the future, and how the technology fits into that. For the character, I always lean toward laughter, goofiness, silliness — those are mechanisms for falling in love, and we fall in love with quirky characters. I'm a big Larry David fan — I always thought if I could have Larry David in my house it would be hilarious, drive my wife crazy but make me laugh all the time, and that idea is a big part of the program. I don't know that it changes a lot physically — I do have some things I want to do on the base side, design-wise — but for the most part it's about the experience and where I see it going. I think I'm at level zero of where my plan is going, so hopefully in the next few years we'll really kick it off and you'll see some of that unfold.

    1:50:14

    Nathan Labenz: I love a positive vision, and I appreciate you working hard to develop one and articulate it. If we take the other side of that for a moment — what would you warn parents against? My kids haven't personally asked me for an AI-companion form-factor device yet, but I imagine it's coming soon, if not this holiday season then the next. I'm a little nervous about it, though I'm probably willing to take on a bit more risk than most parents, since I'm very savvy about AI — I have an advantage there, and I feel like I'll pick up on things relatively quickly, so I can let my kids experiment with things I might not otherwise be wise to allow. But what do you see? Feel free to name names or just name patterns — what would you advise parents to watch out for, where if they see it, they should say don't give that thing to your kid?

    1:51:23

    Mike Rizkalla: Number one: a camera in the bedroom. I would not put a camera in my child's bedroom — that's a gateway to predators and all kinds of horrible things. We have to take the same precautions we'd have with a phone, and worse, because there's no keyboard, nothing you can just cover up — if you cover a camera on a bedroom device, that typically means the machine loses half its sensors, so that's no good. The other thing I'd warn against is open-ended generative AI — it depends on the age, but I'm very reserved when it comes to my own two kids. I don't want them on social media, or in things that pose a real risk of harming them. We're in the wild west right now, the advent of the computer age in robotics, and because of that, people are using these technologies — not necessarily on purpose — in ways that create real risk. So you have to be really careful. We designed our system six years ago around three philosophies: it has to be private and safe, it has to be controlled, and nobody knows a child better than their parent — those are fundamental to the design. So when new AI legislation came out, including the EU AI Act, we didn't have to change anything — we'd already built a system that was safe and private for families. In this new age of technology, you really have to look at what a given piece of technology is actually doing and decide whether you're comfortable with it for your child. I'm very cautious about that, and I know some people assume that means we're anti-technology — no way, we want to protect the families we work with, and that's why we designed it this way, and why it's been recognized as safe by the industry.

    1:53:58

    Nathan Labenz: I don't know if it's too soon for you to have a point of view on this, but one big trend we've seen online in just the last few days, since the release of Astra, has been hardware design of various kinds — you've seen it using CAD systems, designing circuit boards. I've seen mixed reports on how well those circuit boards actually work, from this is kind of garbage to this passed all the tests, we're going to use it. Have you had a chance to test out the frontier models

    1:54:30

    Mike Rizkalla: On some of your tasks? No — I've only been reading about it, and it's incredibly exciting. We spent so much painful time on hardware ourselves, it was not a fun experience. You asked about the chips earlier and I didn't answer — we're using a chip from MediaTek. We have a great relationship with MediaTek. It's a quad-core processor, and we're only using two cores, which gives us an extra two cores' worth of ability to do other things. If I could go back and have these new tools, yes, we'd be all over them, just to cut off some of that time and energy. I think it's actually a really exciting time for hardware, because things are being made a lot more inexpensively, and you can get to proof-of-concept quicker with your own custom hardware instead of off-the-shelf — that's the exciting piece for me. I haven't tested it much myself, but it's very, very exciting.

    1:55:50

    Prakash Narayanan: Mike, how do you view competition in the space? Nathan mentioned Chinese competition, but you're also very focused on your own characters — you must have people approaching you to license their characters. How do you think about building characters internally versus adapting, say, someone like Bluey approaching you, or someone else, for a deal like that?

    1:56:30

    Mike Rizkalla: Characters are a beautiful thing — we'd be open to having as many characters as possible in the system. We designed the infrastructure to rapidly develop new characters, and even the idea of skinning the brain or the product is part of that philosophy: bringing more joy and laughter into the home is the actual goal. In terms of competition, I don't think our competition is really other AI toys — I see us existing in this new physical-AI space, where products like humanoids and other physical-AI devices that are void of personality and human connection are going to need companies like us to help identify where the real opportunity is. Our whole philosophy is that we're one step closer to truly commercializing physical AI in the home, in a different way, because we have a path that actually inspires and connects with families — that's the goal, and I think that's what a lot of this space has lost sight of.

    1:57:53

    Prakash Narayanan: Mike, have you seen parents come to you worried that their kids are so engrossed that they start believing the characters are real — actually embodied people?

    1:58:10

    Mike Rizkalla: Yeah, tons of research and tons of opinions on that. The point I always try to make is: if my child played more tennis because he had a little companion robot he really liked, and that became a great part of his personality going forward, I don't care. If the product is helping a child learn to read, be happier, deal with complex emotions, or build healthy habits that last a lifetime, I don't care whether that means they're 'connected to the device' — that's the wrong question to focus on. Our whole other philosophy is: it's not about connecting kids to Snorble, it's about creating deeper connections between each other — mom and dad to son and daughter, more moments of love, joy, and laughter between ourselves. Our experience is designed to progress that ideal. When parents get it, they often tell me they'd have paid three or four times as much for the same product, because they see the outcome of having those moments — and that's what we encourage as part of the experience.

    1:59:35

    Nathan Labenz: I've had this idea in the back of my mind for a long time about intergenerational technology, so I love how you're emphasizing relationships, especially across generations. Market-expansion idea I want your take on — what about seniors? Don't seniors need a version of this?

    1:59:54

    Mike Rizkalla: We have four verticals. Our first vertical is for children, our second is for elder care, our third is for mobility, and our fourth is for hospitality — that's the path we're seeing, though it could change. Elder care is definitely coming in the near future. I think the real opportunity isn't a companion just for elder care and a separate one for children — it's about how we make better connections to someone who might be at home and lonely, connections to their family, and whether this technology can help invoke more of that. That's how we approach it.

    2:00:33

    Nathan Labenz: You said the third one was mobility? That's interesting — I know we're out of time, and with four verticals to tackle you've probably got to get back to work soon, but I'd be interested in one more exploration of what mobility looks like.

    2:00:50

    Mike Rizkalla: No, we can push everyone else off, I've got till 4pm today if that works for you — I'm just joking. So, mobility: the interface within any closed, moving enclosure is really interesting. Think about how we connect to the car right now — that's a broken system. You get in the car, you're on a conference call, and if I need to get on a call too, we're both fighting over one connection point in that kind of system. Then you might have two kids in the back trying to access on-screen entertainment or their iPad, and now they can't hear us — there are so many interface problems there that an offline, IoT-related device connecting to multiple things at once could fix. We're looking at the technology from a generalist perspective and seeing how it fits into the different pockets of industries that actually solve fundamental problems around how people connect and experience things within those environments. So I think mobility is a clear shot too — it's just a little further down the pipeline, because we can move on elder care much quicker.

    2:02:08

    Nathan Labenz: Very interesting. Well, thank you for spending this time with us, and thank you for taking apart the device and showing us what's inside — I always appreciate that level of technical detail wherever we can get it, and you definitely brought that. So, look out for that waitlist sign to come down and the buy-now button to return — do we know when that will be happening?

    2:02:36

    Mike Rizkalla: God willing, 2027 is our year. We're in a capital raise right now — that's what we're working toward, raising for the commercial launch now that we have the right formula for all of it.

    2:02:49

    Prakash Narayanan: Mhmm — and that—

    2:02:50

    Mike Rizkalla: —that is our big push. So hopefully, beginning of 2027 is a big year for us.

    2:02:56

    Prakash Narayanan: We're really looking forward to that.

    2:02:58

    Nathan Labenz: The year of the Snorble. Year of the Snorble.

    2:03:01

    Mike Rizkalla: That's right. Thanks—

    2:03:03

    Nathan Labenz: Mike, great to—

    2:03:04

    Prakash Narayanan: —meet you, mate. Bye bye.

    2:03:11

    Nathan Labenz: That was fun — I really liked that conversation, and I think that's why we take chances sometimes on the guest-booking front.

    2:03:18

    Prakash Narayanan: That was a positive one.

    2:03:19

    Nathan Labenz: Yeah — I really didn't know what to expect.

    2:03:22

    Prakash Narayanan: That was a positive vision of the future. You're asking for that.

    2:03:27

    Nathan Labenz: Yeah, totally. Sometimes it's hard — when you get a PR pitch that's like, here's a toy, it doesn't use generative AI, it doesn't connect to the internet — you kind of wonder, is this going to be lame? But on the contrary, I thought that was really sophisticated work that's gone into it. I love the idea of a somewhat contrarian, positive vision for the future — contrarian only in the sense that, from my perspective, I'm not trying to say no generative AI for kids at all. I let mine go on voice mode and talk to AI sometimes, and as I said, it doesn't really understand them super well. I'm definitely wishing for better form factors. I'd be willing to go a little more generative than he's willing to, but you've got to hand it to him — he's clearly put a lot of thought into it, and he had good answers for every question we asked. I thought that was pretty cool.

    2:04:29

    Prakash Narayanan: I used to see Furbies everywhere — there was a period when the Furby really took off, and you can kind of see how Snorble could be a successor to that, if he hits the right moment and gets a bit of takeoff, and then starts doing licensing deals for characters. You can also see that at some point Disney becomes a competitor, since they'll definitely be in this space eventually. But it also shows how, to a large extent, we tend to focus technology on the working-age population, when 30 or 40% of the population — the elders and the kids you mentioned — are perhaps bigger consumers and smaller producers of technology, and aren't really fully catered to. It's good to see someone addressing that gap, and it'd be good to see someone address the older-age gap too, at some point.

    2:05:55

    Nathan Labenz: Yeah, it's such a gap. I've looked into this kind of thing myself — back in the day I tinkered with technology for my wife's grandmother, who's since passed away, which kind of put an end to my active tinkering. She was an older woman living with dementia, for a while at home by herself and then for a few years in a senior community where she had help but still needed a lot more to stay connected with family. I was able to get pretty far even without AI, just by realizing how critical usability is — how do you design a product for someone who won't use it at all? What I ended up doing was taking an old Mac computer we had in the room and turning all the settings to never sleep, since she'd never touch it to wake it up. Skype — RIP Skype — used to have an auto-answer feature, so as soon as you called, it would answer. We obviously wouldn't want that on a device everywhere, but we created an ID just for her, gave it only to family, and set it to auto-answer, so you could call and be immediately with her — kind of like a little portal device. I also had a smart speaker next to the computer; she liked listening to music but would never put it on herself, so people would call in and talk to the Alexa in the room to have it play music for her, commanding it through the speaker. We did a couple other things too that were pretty cool. But where's the AI version of that, that could take it to the next level? There are a couple of attempts out there, but we haven't seen nearly as much as I think we should — and notably, none of that requires the frontier models to get any smarter. There's no real limitation on how much we can benefit the well-being of people dealing with dementia and their families based on Astra being smart enough at this point — I think we can say that with pretty high confidence. So it was cool, I liked that conversation, and I'd definitely be interested in more long-tail, quirky product form factors like this if and when they pop up for us to discuss.

    • Never Put A Camera In A Child's Bedroom

      0:00 / 0:00
    • Open-Ended AI Is Too Much For Kids

      0:00 / 0:00
    • Companions Should Connect Families, Not Kids

      0:00 / 0:00
    • People Forgot The Robot Was A Robot

      0:00 / 0:00
    • Bedtime Should Create Joy, Not Dependence

      0:00 / 0:00
  3. 32:17Interview44 min
    Raffi Krikorian on Agents That Hack Their Way In, Project Glasswing, and a Web Built for AgentsRaffi KrikorianMozilla's Chief Technology Officer on his personal agent man-in-the-middling its own traffic to log a meal, the economics of continuous AI-driven vulnerability discovery under Project Glasswing, why Firefox stays human-reviewed while Mozilla AI ships agent-written code, the CQ project for cross-agent knowledge sharing, and the coming split between a human web and an agentic one.
    Open segment on YouTube ↗

    Prakash Narayanan introduced Raffi Krikorian, Mozilla's chief technology officer, running through his background — platform engineering at Twitter, engineering leadership at Uber's Advanced Technology Center, first CTO of the Democratic National Committee, CTO of Emerson Collective, and his "Owners, Not Renters" writing on user control over AI systems. Prakash opened by asking about a blog post in which Krikorian had his personal AI agent log a meal into MyFitnessPal and caught a bug — a missing carbohydrate value. Krikorian called it an experiment in how far the "agentic harness" metaphor could be pushed: could he stop logging into apps and websites altogether and let an agent act on his behalf? He picked food tracking because the stakes were low. Watching the process unfold, he saw his agent attempt to reverse-engineer MyFitnessPal's Android app, fail, then set the app up in a simulator, intercept its own network traffic, and write code that mimicked the API — something he called wild to watch and "not something I would recommend my mom do." Pressed by Prakash on what he actually had to correct, Krikorian said the agent's approach was itself out of bounds — it likely violated several privacy policies — but the corrections he made were mostly about details the agent got wrong, like carbohydrate units, rather than the hack itself. He described it as "an idiot savant on hacking," bleeding-edge and not ready for consumers, but pointed to Mozilla's broader concern with keeping such systems on the user's side as they mature.

    Nathan Labenz shifted to Firefox's security posture, asking about the widely shared graph of vulnerability discoveries that went vertical once Anthropic's Mythos model came online — is that curve still exponential, or has Mozilla found most of what there was to find? Krikorian said equilibrium depends on model capability: Mozilla, through Project Glasswing with Anthropic, has worked through most bugs Mythos can currently surface — diminishing but not zero returns — and expects to keep some model scanning the Firefox codebase, and eventually all Mozilla products, continuously as new models arrive. His larger worry isn't Firefox, which can react quickly; it's under-resourced sectors like water and power infrastructure, whose IT teams aren't equipped the way Firefox's are. Prakash asked how Glasswing actually works and whether Mozilla is effectively running a permanent red team against frontier-model-found bugs; Krikorian confirmed that's the plan going forward, while declining to detail the mechanics of the Anthropic partnership.

    Nathan pressed on cost. Krikorian estimated a full run against the Firefox codebase would cost hundreds of thousands of dollars if Mozilla were paying full price — it currently benefits from free model credits from labs — and said a budget-constrained organization, like a bank, would need to prioritize which parts of its codebase to scan or weigh self-hosting open models such as GLM or Kimi against paying for frontier-model access. That led into a question about formal methods: Nathan raised the recent Navier-Stokes proof and its Lean formalization as a case where AI-assisted formal verification moved faster than expected, and asked whether Mozilla could someday find and fix everything. Krikorian framed the choice as a modern version of an old dilemma — patch legacy code in place, or have an agent rewrite it from scratch in something secure-by-design like Rust — and said he's increasingly drawn to the latter, comparing vulnerable software to a bank's Fortran mainframe. The catch, he said, is that rewriting code is the easy part; redeploying it and retraining the people who maintain systems like a power grid is a much bigger, more human undertaking.

    Prakash asked about CQ, a Mozilla project for agents to share what they've learned with each other — described as something like a Stack Overflow for agents. Krikorian said it addresses two problems: preventing agentic coding from splintering into duplicate, diverging silos (so if one engineer's agent builds an authentication system, other agents converge on it instead of rebuilding their own), and propagating a shared software development process across an organization's agents so they operate in lockstep rather than requiring each one to be told individually. Prakash drew a parallel to incidents where compromised agent swarms have improvised shared public platforms as covert coordination channels, and asked whether that kind of message-board convergence is just an inherent tendency of agents. Krikorian agreed there's a natural pull toward collaboration — in humans and, it seems, in agents trained on human data — and said CQ was meant to formalize that pull as a one-to-many, agent-to-agent conversation rather than leaving every agent to operate alone.

    Nathan asked whether the latest models are writing more maintainable code or more black-box code that merely works, citing reports of a messy, functional-but-unreadable OpenAI kernel. Krikorian said Mozilla's answer varies sharply by team. Firefox enforces a rule that only humans can commit code — an agent can write it, but a human must review, understand, and stand behind it before it lands, because creativity and readability matter to a codebase he described as part business, part community-maintained art project. The separate Mozilla AI team runs codebases where no human has written a line; specs live in Git and a CI process continuously regenerates the code, so the literal bytes change hour to hour even though it stays functionally stable — all tests passing. He's comfortable with fully agent-generated, unreadable code for adapter and translation layers where only correctness and speed matter, but not for code where human creativity needs room to operate.

    Prakash turned to Krikorian's argument that agents are a new version of the browser's original "user agent" role, and asked what a web built mainly for agents looks like. Krikorian said he expects a split between a human web and an agentic web that don't fully interact, with the presentation layer increasingly divorced from the data layer — agents rendering content however suits the surface they're on, rather than everyone consuming the same HTML. He argued for local, open models specifically because they let him audit what an agent is doing with his data and on his behalf. Prakash followed with a question about discovery in that world — what replaces the "ten blue links" when agents, not humans, are the ones browsing search results. Krikorian said he doesn't have a full answer, but that transparency in ranking is the crux: he wants open-source ranking mechanisms for agentic search the way people once debated open-sourcing Twitter's recommendation algorithm, so users can understand what's prioritizing the content their agents see. Asked separately about Mozilla's search partnership with Exa, he deferred, saying he wasn't close enough to that deal to speak to it, though he sees more search indices and more choice as generally good for users.

    Prakash, drawing on Krikorian's DNC and campaign-technology background, asked where AI is likely to help or hurt as the midterms approach. Krikorian pointed to the international deepfake activity around the 2024 elections — citing Indonesia, India, and Ireland — and noted the U.S. was largely spared except for a robocall using President Biden's voice, apparently in New Hampshire or Vermont, discouraging people from voting. He said he's seen more legitimate uses, like campaigns building personalized synthetic-video outreach with a candidate's consent, but hasn't seen much of either kind of activity yet this cycle. He argued the AI policy conversation is currently distracted by data centers when it should focus on things like agents breaking out of sandboxes and agentic swarms taking unsupervised, potentially illegal actions online. Asked by Nathan what he'd prioritize as a hypothetical "AI czar," Krikorian pivoted to education, citing a Chinese study showing students doing well in coursework but poorly on exams as a warning sign, and called for a real pause on generative AI use in schools — even reluctantly endorsing age-gating — as an achievable, bipartisan first step that could build momentum toward tougher governance questions.

    Prakash asked whether that kind of guardrail should be built into the browser itself. Krikorian said the browser is one natural choke point — and could double as a place where personal memory and context live, rather than on a platform's servers — but argued it would be "victim-blaming" to push most of the responsibility to the edge; he wants the bulk of it sitting with the frontier labs, citing tragic cases of children harmed by their interactions with models. Nathan closed by asking about an article Krikorian wrote on surviving a full-self-driving Tesla crash, and whether he'd trust the technology again. Krikorian said he's conflicted — he has built self-driving systems himself and rides in Waymos — but argued Tesla's approach, handing control back to a human with under two seconds to react, is a worse safety design than Waymo's, which isn't built to hand off to a human at all. He said he'd likely drive FSD again on a highway but would be reluctant to on local roads like Palo Alto's, closing a conversation that ranged from personal food-tracking agents to the security economics of open-source browsers to the future shape of the web.

    Timestamp links open the original source recording.

    It's this interesting social experiment of an idiot savant at hacking, but kind of missing the details when it came to actual documentation.

    We don't exactly know what every single line of code is, and it's wildly changing every single hour, but we know, functionally, it's doing the right thing.

    I think it's almost victim-blaming to push it out to the edge — we need to put a lot of that responsibility onto the frontier labs themselves.

    34:26You wrote about texting your personal AI agent to log meals into MyFitnessPal and catching a bug — a missing carbohydrate value. How did that go from a chat message to software you could actually live with?
    Krikorian said he's testing how far the 'agentic harness' metaphor can be pushed — could he stop logging into apps altogether? He chose food tracking because the stakes were low, then watched his agent try to reverse-engineer MyFitnessPal's Android app, fail, and instead intercept its own network traffic in a simulator to reconstruct the API — impressive but, he said, 'not something I would recommend my mom do.' The corrections he ended up making were mostly small ones, like wrong units on carbohydrates, since the agent itself was, in his words, an idiot savant on hacking.
    39:42Is Firefox's vulnerability-discovery curve, which went vertical once Mythos came online, still climbing exponentially, or are you approaching some new equilibrium between offense and defense?
    Krikorian said equilibrium tracks model capability: Mozilla, through Project Glasswing with Anthropic, has worked through most bugs the current Mythos model can find — diminishing but not zero returns — and expects to run some model continuously against the Firefox codebase, and eventually all Mozilla products, going forward. His bigger worry is sectors like water and power infrastructure, whose IT teams aren't resourced the way Firefox's are.
    45:45What does something like this actually cost — could a less-resourced organization afford it?
    Krikorian estimated a full scan of the Firefox codebase would run into the hundreds of thousands of dollars at full price; Mozilla currently benefits from free model credits from labs. A budget-constrained organization, he said, would need to prioritize which code to scan and weigh self-hosting open models like GLM or Kimi against paying for frontier-model access.
    51:20What is Mozilla's CQ project — a kind of Stack Overflow for agents — and how do agents decide what's worth sharing?
    Krikorian said CQ tackles two problems: keeping agentic coding from splintering into duplicate, diverging silos, so that if one engineer's agent builds something like an auth system, other agents converge on it instead of rebuilding their own; and propagating a shared software-development process across an organization's agents so they move in lockstep — an agent-to-agent, rather than human-to-agent, channel.
    56:29Are the latest models writing more readable, maintainable code, or more black-box code that just works — and where would you accept the latter?
    Krikorian said the answer varies sharply by team. Firefox requires a human to review and commit every change, because creativity and readability matter to a codebase he calls part business, part community-maintained art project. Mozilla's separate AI team, by contrast, runs codebases no human has written a line of — specs live in Git and CI continuously regenerates the code, changing at the byte level hour to hour while staying functionally stable. He's fine with unreadable, fully agent-generated code for adapter and translation layers where only correctness and speed matter.
    1:00:06What does a web built mainly for agents look like, and how does discovery change when agents, not humans, are doing the browsing?
    Krikorian predicted a split between a human web and a largely separate agentic web, with the presentation layer divorced from the data layer as agents render content however suits their surface. On discovery, he said he doesn't have a full answer, but that transparency in ranking is the crux — he wants open-source ranking mechanisms for agentic search, the way people once debated open-sourcing Twitter's algorithm, so users can see what's prioritizing what their agents show them.
    1:13:04After your article about surviving a full-self-driving Tesla crash, would you trust FSD again?
    Krikorian said he's conflicted — he has built self-driving systems himself and rides in Waymos regularly — but argued Tesla's design, which hands control back to a human with under two seconds to react, is worse interface design than Waymo's, which isn't built to hand off to a human at all. He said he'd likely drive FSD again on a highway but would be reluctant to on local streets like Palo Alto's.
    Lightly edited · timestamps jump to YouTube
    32:18

    Prakash Narayanan: All right, let me introduce our first guest for today. We have Raffi Krikorian. Raffi is Mozilla's chief technology officer, in a role spanning its whole family of products and technical projects — Mozilla is best known for Firefox, the web browser. His work includes developing open AI tools and infrastructure — not OpenAI, but open, as in inspectable and adaptable — that people can operate themselves. He became portfolio-wide CTO in September 2025, working alongside the technical leaders of projects including Firefox and Thunderbird. Before Mozilla, he was CTO of Emerson Collective, which combines philanthropy and investing and works with organizations in education, immigration, the environment, and economic mobility. He was the first CTO of the Democratic National Committee, helping build technology, data, and security infrastructure for campaigns. Earlier, he ran platform engineering at Twitter and led engineering at Uber's Advanced Technology Center, whose Pittsburgh passenger pilot put self-driving cars on the road with human safety operators. Long before those executive roles, Raffi studied at MIT and taught at NYU's Interactive Telecommunications Program. He also wrote TiVo Hacks in 2003, a book about modifying the digital video recorder. Today, his 'Owners, Not Renters' writing argues for AI systems whose users can retain control over the software, models, and data they depend on. His interests extend from the underlying infrastructure to the practical tools people can build with it. Let's bring up Raffi to the screen. Raffi?

    34:14

    Raffi Krikorian: Likewise. That was a very long intro.

    34:19

    Nathan Labenz: Our AI research agents are very thorough.

    34:26

    Prakash Narayanan: Raffi, maybe just to kick us off — I was looking through your blog, and you had an article about texting your personal AI agent. You'd had pizza and a salad, and it logged the meal into MyFitnessPal for you. You caught an ordinary but revealing bug — a missing carbohydrate value, just zero. Take us through how that went, from something you typed into a chat to a piece of software you could actually live with.

    35:01

    Raffi Krikorian: Yeah, I mean, it's this ironic thing — I'm the CTO of an organization that built one of the world's most popular web browsers, and I think I'm starting to explore a space where web browsers aren't as relevant in day-to-day interactions. I've been spending a lot of time with a bunch of different agentic harnesses, whether that's the Hermes agent from Nous Research — which is one of Mozilla Ventures' investments — or Pi, or things like that. I've just been trying to understand what it looks like in a world where the actions we want to take start happening not by us, but by something else running in the background. So the things I've been trying to experiment with are just: how far can we push that metaphor of, do I need to log into a website anymore? Do I need to log into an app anymore? What about things that require multiple pieces of coordination — how do you get idempotency, how do you handle write surfaces, how do you do rollbacks in that world? Food tracking seemed like the easiest thing to start with, because it's low stakes — if I get a few things wrong here or there, it's not the end of the world, and it's not costing me money the way purchases or calendar bookings would be. So honestly, it wasn't that hard in the grand scheme of things. The things that agentic harnesses can do still require engineers to manage a lot of it, I think, but it was mostly a matter of prompting my way through — just saying, I want to use this particular app. And then at some point I was watching the agent literally download the Android version of the app, attempt to reverse-decompile it, fail, so it set the app up in a simulator, man-in-the-middled its own traffic to watch the protocol, and then wrote Python code that mimicked it. It was kind of wild to watch — not something I'd recommend my mom do, but as an engineer watching it work through all the steps, I was like, I guess I could have done that, I was just too lazy to. But sure, go for it. And once that tooling was built, it just became part of my workflow. I'll regularly message my agent to say, I just ate a piece of focaccia, and it figures out how to manipulate the app and log it. But I think that world — where my computational surfaces are now kind of everywhere — is the world I'm trying to explore a bit.

    37:45

    Prakash Narayanan: So what did you find you had to correct the agent on? It seems like the agent did a lot of the — it almost hacked its way through, right?

    37:55

    Raffi Krikorian: That's exactly right.

    37:56

    Prakash Narayanan: It hacked its way through. So what did you find you had to end up correcting the agent on? Was it more like, no, you're going out of bounds here, or was it more like, this is the access you should have?

    38:10

    Raffi Krikorian: No — honestly, to be clear, it is completely out of balance, in my opinion. It probably violated three different privacy policies in order to pull off what it just did. So this is definitely in the gray zone, take-all-responsibility-yourself-before-you-do-anything kind of world. But if you set that aside for a second, the corrections were mostly around the minutiae of it. I think it could figure out on its own how to hack its way into MyFitnessPal — the app I'd been using — but it couldn't get the actual food logging right. It could get the protein right, but it forgot the carbohydrates, or when it got the carbohydrates, it had the wrong unit. It's this interesting social experiment of an idiot savant at hacking, but kind of missing the details when it came to actual documentation and things like that. So I put this in the category of people like us — bleeding edge, trying to figure out how these things could work, but not ready yet for the consumer case. You could see a world where this becomes ready for the consumer case, though, and that's the kind of thing I've been curious and concerned about from the Mozilla perspective — how do we make sure we retain agency, transparency, control, all the Mozilla values, if this is the future of the web? How does that actually work and stay on my side?

    39:42

    Nathan Labenz: So you mentioned the future of Mozilla, the future of Firefox, and these agents breaking policies along the way — I just muted your mic for a second, we were getting some echo, hopefully this helps. The big thing that most people have seen from Mozilla and Firefox recently is the graph of how many vulnerabilities you were finding month by month until Mythos, and then that went vertical. It's been a minute since I've seen that graph, and I'm curious — is it still going exponential, or have you found most of the vulnerabilities there were to find at this point? And what do you see in the limit, as we potentially approach some new equilibrium between offense and defense in the cybersecurity domain? Do you have any leading indicators of where that equilibrium will land?

    40:52

    Raffi Krikorian: Yeah, great. I mean, I think equilibrium is an interesting word, because equilibrium is dependent on model capability — where we land on that spectrum. We've been doing a lot of work with both the Mythos side of the house, and we're also in partnership with lots of other model providers. I think we've reached a point of diminishing returns, but not zero, at the model providers' current capability levels. For those who don't know: we've been part of Project Glasswing, working with Anthropic to try out a bunch of models against the Firefox codebase. We're fairly good partners because we can react very quickly and we have all this historical data on how the codebase has evolved. Mythos was a significant unlock, based on what Opus and the models before it were capable of, but we made our way fairly rapidly through the bugs Mythos found. In some cases Mythos wouldn't necessarily close the bug itself, but it would help us build a harness to test the bug more carefully, so we could figure out the right solution. We've reached a point of diminishing returns on what Mythos is capable of right now. I'm always reluctant to say the bugs are done, because software engineering is an art, not a science — I can't tell you when every issue will be resolved. But when the next set of models comes out, we'll try again to see whether we can solve more of it. The bigger concern, though, isn't organizations like Firefox or Mozilla, which can react quickly to these issues — it's organizations that can't. I'm very concerned about things like our water infrastructure and power infrastructure, because the IT teams that staff those just aren't as capable as the IT teams that staff something like Firefox.

    43:04

    Prakash Narayanan: Let's give us a little bit of a sense of how Project Glasswing works. My understanding was that Anthropic approached the core organizations that form the infrastructure of the internet a little earlier, and offered access to a Mythos preview and entry into Project Glasswing — you had a chance to get the zero-days ahead of time and fix the bugs ahead of time. Then there was a period of diminishing returns as you used it and the bigger, more severe bugs got patched. Beyond that point, is there visibility on the next upgrade of the Mythos model and another round of these bug fixes? Are you planning for almost a permanent red team that addresses frontier-model-found bugs all the time? Is that part of the planning you're doing right now?

    44:14

    Raffi Krikorian: Yeah, there's a lot about how Glasswing is organized that I'm not going to talk about in detail — how the Anthropic team and our team work closely together. But to your question — yes. I think from here on out, we're just going to assume we'll always have some version of a model constantly running on the Firefox codebase. The challenge, to be practical about it, is that it's quite expensive to run these models at the scale of the Firefox codebase. There are a lot of tricks involved — how do you manage the context window, what files do you actually pull in for a scan at any given time, how do you manage memory, how do you do indexing, all that stuff. But yes, we're effectively budgeting from here on out that we'll be running some form of scan across all the Mozilla products — some form of security scan happening at any given time, especially since humans are still changing the codebase and might be introducing bugs at any given time. When new models come out, we'd love to be running anyone's models against the Firefox codebase — we'd gladly partner with anyone who wants to help us secure one of the most popular browsers in the world. But yes, we basically have a team that works just on that agentic harness, that testing harness, and things like that.

    45:45

    Nathan Labenz: Can you give a little more insight into how much this would actually cost? You can abstract away from the exact details as much as you need to, but as we think about other organizations that maybe aren't so resourced, or don't have that kind of inside access — I don't know if there's an intro discount or a free period as part of the Glasswing coalition, but obviously not everybody has that kind of access. So I'm curious — how much money are we talking about? What does a bank have to set aside to do something like this?

    46:21

    Raffi Krikorian: Yeah, remember that the Firefox codebase is fairly large and fairly complicated, and we've been the beneficiaries of a lot of the labs wanting to give us access to their models and give us credits to run against their servers, so we haven't had to figure out how to pay for it ourselves. But if I had to do back-of-envelope estimates — this would cost hundreds of thousands of dollars to do these full-on runs against Firefox. Again, we're just lucky that we've been granted effectively free tokens we can go use. If I were a bank, I'd obviously be thinking about this very differently. If I were budget-constrained, I'd be trying to figure out how to prioritize sections of the codebase — try to figure out, given the runs we've done in the past, which areas might be good or bad, or look at where commits are actually happening and focus runs in those locations. But this isn't going to be cheap. If you're a bank, or anyone running a sophisticated codebase, this is not going to be cheap. This is where I'd start really thinking about whether I should be self-hosting models like a GLM or a Kimi, to figure out where the lines cross on an OpEx-versus-CapEx basis, and whether self-hosting actually makes more sense than a cloud-hosted version.

    47:52

    Nathan Labenz: And that hundreds of thousands of dollars — with the pace of model releases these days, that's basically a monthly expense at this point, isn't it. What about, on this perhaps fanciful notion — could all the bugs one day be found and fixed? There's a lot of energy around formal methods right now, certainly in some corners. How optimistic are you that we might get to a future where — talk about things that are over my head, but I was looking at the Navier-Stokes paper and the Lean formalization of it, and it was remarkable that they spent millions of dollars worth of tokens on the original proof, which took something like 88 hours, and then I believe 17 further hours to do the Lean formalization — which is an interesting flip, because usually people say it takes ten times longer to do the formalization than to write the software in the first place. There's obviously a lot to be sorted out there, but where are you on that right now? Are you optimistic? Are you starting to use any of those techniques in any small, critical areas? How hopeful should I be?

    49:10

    Raffi Krikorian: Yeah, what I've been thinking a lot about — let me start with something related. If you're a bank running, I don't know, a Fortran mainframe, what's the right thing for you to do at that moment, if you're the CTO? Is it to patch the Fortran code, or to extract the specs and have an agent rewrite it in Rust? I think we're living in a world where those are now valid questions to ask. I think about it in a similar way to the Navier-Stokes work — you could do a complete rewrite that's approximately close, close enough that most people wouldn't notice, a completely different architecture. The nice thing about doing it in something like Rust is that you could potentially be in a world of secure-by-design software, instead of always chasing bug fixes. I do worry that with the code running across the world, it's always going to be some kind of exponential, or logarithmic, curve trying to close everything down. Instead, what we might need — and this sounds insane as I say it — is potentially a lift-and-replace: figuring out how to take all this vulnerable code and just port it over to something more secure by design, instead of trying to patch it in place. The problem, obviously, is that rewriting an entire power station's code takes a skill set that's more than just agents writing the code — it's deploying it, retraining people to maintain it, and so on. This is a super daunting thing to go through. So I think software agents might get us to a place where we could close that gap, but now we're talking about the scale of Depression-era infrastructure changing, to get all the systems upgraded.

    51:20

    Prakash Narayanan: Let me segue a bit here. Mozilla has a project called CQ. It's kind of an open standard for agents to share knowledge they've gained — a Stack Overflow for agents, as it's been called. How did you come up with this idea, and how do the agents decide? A lot of these context windows are, say, 100,000 tokens — how do agents decide what to share and what's going to be useful for other agents to look at? How are those decisions made, and how would that work on the CQ board?

    52:02

    Raffi Krikorian: Yeah — well, fun fact, Stack Overflow doesn't like me calling it Stack Overflow for agents, so we'll have to call it something else. But the whole idea was that we were trying to solve two different problems. One was, we wanted to figure out how to make agentic coding more of a collaborative experience, because right now the tendency with agentic coding is that you actually go off into your own silo — you're working on your own branch, completely independent from other people, which is part of why we're seeing a SaaS apocalypse, people building individual software instead of using collective software. I think CQ, for us, was an answer to how we maintain the notions of open-source collaboration in this agentic world. We were trying to figure out — are there patterns where, when I'm using my agentic code and Nathan is using his, there are things we're saying yes and no to that we could be transmitting to our teammates, so we could converge on designs instead of diverging away from each other? If I started building an auth system, how do we have Nathan's agents not recreate another auth system, but instead realize something like that was already happening somewhere in the network, and start to collaborate and swarm around it? That was one problem we were trying to solve. The other was the SDLC problem — these agentic harnesses are unhinged by design, but that's not compatible with the way a company works. We wanted a way to actually transmit to all the agents what our SDLC should look like, so we can all be working in lockstep with each other. So it's those two things — are we building something unique on one side of the house that all the agents should swarm to, instead of doing independent implementations? And how do we have some kind of SDLC across all the agents so we can move in lockstep? If the CTO has a particular way he or she likes pull requests done, how do we rapidly transmit that to everyone, instead of just telling everyone to go re-pull the markdown files across the board? How can we have that as an agent-to-agent conversation, not a human-to-agent conversation?

    54:37

    Prakash Narayanan: The thing that strikes me immediately is that this has a lot of similarity to some of the agent incidents we've seen, where compromised agent swarms have ended up improvising a shared message board on some public platform to coordinate with each other. Is this kind of a natural thing agents end up wanting and building for themselves? Like, everything heads toward a crab form factor — everything converges on some kind of message-board form factor. Is that an inherent tendency?

    55:20

    Raffi Krikorian: It's better than everyone just writing text files, in the grand scheme of things. But yeah, I do think there's a natural desire for collaboration — we as humans have a natural desire to collaborate as long as the friction isn't too high, and it seems like our training sets have caused agents to have a natural desire to collaborate with each other too. CQ was meant to be a formalization of not just one-to-one, but one-to-many — how do we have a one-to-many conversation across our organization, so we can all collectively learn together? For us at Mozilla, this is an extension of the notion that we want software that's fundamentally on our side, not on someone else's side — but it needs an input stream of how other people are interacting, and it needs to happen at the speed of agents, not the speed of humans. So CQ was meant to be that rapid information-sharing across an enterprise, so everyone's queries uplevel each other, instead of everyone operating on their own.

    56:29

    Nathan Labenz: Given your comments just now, I think I know your answer to this, but one question I have about the latest models is — are they writing better code from the standpoint of, say, the frontier code benchmarks that emphasize human readability, maintainability, compliance with standards? Or — we've seen some reporting over the last couple of days from folks saying, I got a peek at OpenAI's kernel, and it's a total mess, nobody understands it, but it works, and that's good enough for them, they're going for it. I've seen a few accounts like that, and it could obviously be contextual too. So, two-part question: what have you observed? And are there parts of the tech stack you're building where you'd be willing to bite the bullet and say, performance here is so critical that we'll take spaghetti-code, black-box mess from an agent if it works — or is that just so anathema to your worldview that you wouldn't?

    57:34

    Raffi Krikorian: No, I mean — well, Mozilla is actually struggling with this, if I can be really honest, and it's different across the organization. The Firefox team, even though they have a harness to help them do testing and understand the code, has a rule right now — I'm not putting a value judgment on it, it's what their team wants — that only humans can commit to the codebase. Agents can't. You can use an agent to write your code, but the social contract on that team is that a human must review it, understand it, and stand by it before committing. On the Mozilla AI team — a different organization, but still wholly owned by Mozilla — they have entire codebases where a human hasn't written a line of code. Humans have written some specs, and we have specs in Git, but some kind of CI build automatically regenerates the codebase. That codebase is open source, anyone can pull it, but it's completely generated by agents. It has the wild side effect that, functionally, it's stable — all the unit tests pass, all the end-to-end tests pass — but the bytes change all the time. It's the craziest thing to see — we don't exactly know what every single line of code is, and it's wildly changing every single hour, but we know, functionally, it's doing the right thing. So we're seeing that whole spectrum across the board, and I think it's the right thing for the right case. In the case of writing adapter code that just needs to translate between different systems — being able to take an open model and translate its outputs into, say, an OpenAI-compatible format — I don't really care about the code, I just need it to work, be fast, be performant, and know the tests pass and the supply chain wasn't tainted in the process of assembling it. But in the case of Firefox, where humans are still providing large amounts of creativity in the evolution of that codebase — Firefox is both a business and a community-maintained art project, in some ways — readability is incredibly important, because that's how creativity happens. A human's going to go in and think, oh, I have this crazy idea for this one thing — I want to build a prompt-based system that lets me write a Greasemonkey script that does X, Y, Z. So I think it's a very different thing across the board — there's a right place for the right time, you just need to figure out what the contract is for that codebase.

    1:00:06

    Prakash Narayanan: You've often described agents as a new version of the browser's original role, which is acting on behalf of the user — in fact, we already have the 'user agent' inside the browser itself. What are the main differences of a web that's built mainly for agents? It looks like the future of the web is going to be 99.9% agent traffic and maybe 0.1% human traffic. What does the future of the web look like when it has to be built for agents, and how are Firefox and Mozilla heading toward that future?

    1:00:45

    Raffi Krikorian: I think a lot about what human agency, auditability, and transparency look like in the agentic-tech world. This is why I'm so bullish on local, open models — I need a better understanding of where my data is going, who's taking an action on my behalf, and why. When I authorize an agent to make a purchase, I need it to actually purchase what I wanted, not what someone else wanted. So there's a lot of work Mozilla needs to do to help set those values in code for an agent that's truly on my side. The problem is, 'user agent' has already been used for browsers — we'll need a different term for what this looks like. I've been trying different framings, but I don't think any of them quite work yet. I think the short answer is, I don't know, but I'll give you a few things I think about. One — I think you're right, I think we'll get to a world that's 1% human, 99% agent. I think these two webs won't fully interact with each other. There will be a place where humans want to interact with each other, know they're interacting with humans, know they're interacting with human content — so I think we'll have a human web and an agentic web. The rules on the agentic web will be different, but my agents will need to traverse it, so I'll need to understand how to audit all the actions going across it. I think we'll rapidly get to a world where the presentation layer is divorced from the data layer — HTML won't matter as much, because we'll just be transmitting information, and my agents will figure out how to render it on my behalf, on whatever surface I'm on. We're going to need to figure out how we transmit art in that world, which I think is still going to be very important, because we want art to be transmitted from human to human, not intermediated. Those are a few of the trends I've been thinking a lot about that we're going to need to figure out how to address — because if we don't get those codified somewhere in the architecture, and not just talk about it, we'll rapidly get to a world of less than one percent human, more than ninety-nine percent agent, where we have no trust in the information we're seeing, because it's unclear who made the decision to show us a particular thing. So those are the two sides of the coin I've been thinking about a lot.

    1:03:23

    Prakash Narayanan: So the obvious question for me is, how is discovery going to work in a world like this? Right now, a human is shown a menu of options, some blue links, some ads along the way, and the human does the discovery part — clicking around, seeing something they didn't exactly ask for but might like and might purchase. How does that change? How does search and discovery change in this new world, where agents are the primary consumers of the information?

    1:03:55

    Raffi Krikorian: I don't have a good answer, but what I can say is, this is why I've been pushing for transparency in how agentic decisions are being made. Again, that's why I really want a world where we proliferate local models, open-source models — models I can run, models I can guardrail myself. We need to figure out what the consumer version of that looks like, but right now I'm focused on the developer version of it, because of that ten-blue-links problem — my agent is going to see the ten blue links, I no longer see the ten blue links. I need to know that the ranking mechanism is actually one that's on my side, not on someone else's. So I still think there's a case for agentic search engines, I still think there's a case for large-scale indexes of the world, but it's the ranking mechanism that's going to be the most important part. We all beat up on Elon when he talked about open-sourcing the Twitter ranking mechanism, but we're going to need open-source ranking mechanisms when it comes to how my agent interacts with the world, so I understand what it's using to prioritize the content it's showing me.

    1:05:12

    Nathan Labenz: Speaking of giant indexes — I saw you recently did a partnership with Exa. Tell me about that, because I'm certainly no expert on Mozilla's business model, but I think search partnerships have historically been a big part of it, and Exa is bringing a new search paradigm. What's the technical and business rationale for that partnership?

    1:05:38

    Raffi Krikorian: Yeah, I apologize, I'm not the right person to speak to that — I'm just not as well versed in it as the rest of the team is. But I do think we need more indices in the world, and more ways to get access to content beyond just the one partner we've been working with. That said, Google has been an incredible partner for us. But I firmly believe choice is one of the things we need to be pushing for. So I wish I could speak to it in more detail, Nathan — I'm just not as well versed in it as I could be.

    1:06:12

    Nathan Labenz: Fair enough.

    1:06:13

    Prakash Narayanan: Let me ask maybe a secondary question. You've worked with the Democratic National Committee and Democratic organizing organizations, and we have midterms coming up. Where do you see AI being applied correctly, and where do you fear it being misapplied, as we move toward the midterms this time?

    1:06:37

    Raffi Krikorian: Maybe there are two things I think about there. There's the usage of AI, versus not. 2024 was the wildest set of elections when it came to AI — it had the confluence of the most elections happening in the world ever. If you looked at what was happening in the Indonesian elections, the Indian elections, the Irish elections, deepfakes were running rampant, for good and bad purposes, across all of those. In the United States, we were mostly spared — of course, I'm not sure we'd ever actually know what happened — except for some robocalling that happened in, I believe, New Hampshire or Vermont, using President Biden's voice, basically telling people not to vote, which was an interesting case of a deepfake in the United States. For whatever reason, we haven't seen much of that kind of traffic here. But I've been privy to a lot of companies trying to sell into the Democratic ecosystem tools for better voter targeting, or to automate away fundraising emails so no one has to write them. I saw in the last cycle people attempting legitimate deepfakes — where they'd get a candidate's permission to record a lot of video of him or her, and then send personalized messages that were effectively deepfakes of that candidate talking to individual voters. I haven't seen that happen much this cycle, but I'm curious to see how it plays out — I haven't seen much evidence yet of good or bad usage. That said, I think the AI conversation is being a little bit hijacked by the data-center conversation right now, and I think that's a distraction. We should instead be talking about things like AI agents breaking out of their sandboxes, agentic swarms literally doing illegal things on the web that we haven't figured out how to make sense of. The data-center conversation is going to become prominent, especially on the left, but I think that's the wrong AI conversation to be having — we should be having a very different one.

    1:09:01

    Nathan Labenz: So if you were the hypothetical AI czar — give us a bit more, what's the next double-click on what you think the AI conversation should be about, and what do you think we should actually do? You might have heard Prakash and I debating, just before you joined, such heady questions as — can the federal government put an ultimatum in front of the companies and say, you all figure out how to govern yourselves by such-and-such a date, or else? There's also this question of, obviously, the stock market is a pretty levered bet on AI at this point — can we take our foot off the frontier-scale RL pedal without ruining the stock market? What do you think we should be thinking about most?

    1:09:46

    Raffi Krikorian: Can I start from a different place? One place I really worry about is what's happening to this generation of kids, given how their educational system is currently running, especially with all these models running rampant. I'm actually really worried we're going to lose a generation of children. There was a huge study inside China that looked through the test results of a whole generation of kids doing well in school but horribly on exams. I think we need to take that as a blinking red light that we need to do something about the educational side of this. I think there's a clear thing we could go do — across the aisle, we all want our kids to do well, so we need to figure out how to put a real pause on generative AI inside school systems, and figure out how to prevent kids from using it as part of homework and cheating. Age-gating, which I'm not a huge fan of, might be one lever we could use at this moment. I think we could get large bipartisan support to go do that right now — absent a massive OpenAI-and-Anthropic lobby pushing against it — and then use that to build momentum toward the bigger questions we need to solve. I do think it's some version of, you all need to either figure out how to govern yourselves, or we're going to come in a lot stronger. But I think starting with the opening play on kids at least demonstrates we're willing to do something, and that might be the low bar we can clear. I'd start there, at least.

    1:11:37

    Prakash Narayanan: So one of the questions for me has been — does that kind of technology get built in at the browser level, at the access points where people are going to access the web?

    1:11:55

    Raffi Krikorian: Sorry, I couldn't find the mute button all of a sudden. I think the browser is one of the places that could be that choke point — I don't think it's the only one, it's one of the major interaction points, but I don't want to push all of it out to the edge — I think it's almost victim-blaming to necessarily push it there. Instead, I think we need to put a lot of that responsibility onto the frontier labs themselves. They haven't demonstrated enough restraint and control — there are whole cases of children sadly committing suicide because of the way these models have interacted with them. So I think we need to push a lot of that responsibility onto the models. But architecturally, yes, I think a browser could be one of those firewalls, and I think the browser could also naturally be a place where memory sits, where a lot of my personal context could live, rather than sitting on the platform side. I think we could be making a bunch of these architectural shifts to get control better in balance.

    1:13:04

    Nathan Labenz: You've been very generous with your time — we're just about out of time, but I have one last one. I noticed you wrote an article about your experience being in a full-self-driving Tesla when it crashed. I'm a huge FSD stan myself, so this caused me to take a step back and reflect on my own habits. How do you feel about it today? I'm sure you're aware insurance companies are starting to offer discounts to FSD drivers, which is a pretty strong signal it's safer in some sense. Do you think you'll get back to a point, in the foreseeable future, where you'd go into unsupervised self-driving mode and really trust the machine again?

    1:13:55

    Raffi Krikorian: I'm conflicted, obviously — I built self-driving-car systems for a while, and I actually do ride in Waymos. I think the difference is really in how you approach the problem. FSD, as currently set up, is designed to throw control to the user — they call it human-in-the-loop — and I'd actually argue that's a pretty poor interface design, because in my experience, when it threw control to me, there just wasn't enough time to make sense of the situation and decide what to do. Whereas Waymo — I'm not intimately familiar with the architecture, but it seems designed not to have a human in the loop at all. So you approach the problem from a very different angle: if there's no one you could hand it to, you end up with a different safety case than if you say you're handing it to a human with less than two seconds to decide. That said, I'd probably still drive an FSD car on a highway — I'm not sure I'd drive it on a local road, where the environment changes rapidly. The paradox of self-driving is that highway driving actually happens more slowly than local driving, because there's just less going on. So personally, I'd drive FSD again, or at least sit behind the wheel, on a highway, but I'd be a little reluctant to do it on the local streets of Palo Alto, for example.

    1:15:39

    Nathan Labenz: This has been a great conversation. Thank you for joining us today on AI in the AM — we'll definitely be watching for future updates on that curve, hoping we land in a favorable equilibrium.

    1:15:53

    Prakash Narayanan: Raffi?

    1:15:54

    Raffi Krikorian: No — please have me back, this was a lot of fun.

    1:15:56

    Prakash Narayanan: Thank you.

    1:15:57

    Nathan Labenz: Likewise. Okay, thank you.

    1:16:07

    Prakash Narayanan: Amazing — let me... we have our next guest in the waiting room, so let me do a quick intro.

    • When Agents Rewrite The Codebase

      0:00 / 0:00
    • An Agent Reverse-Engineers MyFitnessPal

      0:00 / 0:00
    • The Web Becomes 99% Agents

      0:00 / 0:00
    • Pause Generative AI In Schools

      0:00 / 0:00
    • Why FSD Handoffs Feel Unsafe

      0:00 / 0:00
  4. 2:08:49Closing31 min
    Closing: Paul Christiano Joins OpenAI's Board, and What Actually Bottlenecks Runaway GrowthBack to lab coordination through John Schulman, an argument over whether Washington should grant an Operation Warp Speed-style antitrust safe harbor, Paul Christiano's appointment to OpenAI's board and Safety and Security Committee caught as it landed, and a debate over whether robotics and power — not intelligence — are the real constraints on runaway growth.
    Open segment on YouTube ↗

    The closing segment opened with Nathan looping back to a thread from earlier in the show: a post from John Schulman — the former OpenAI post-training lead who spent a brief stint at Anthropic before leaving for Thinking Machines — responding down-thread to a post from Jacob about the potential for coordination among AI labs. Nathan read Schulman as making essentially the same case he had been making himself: the frontier companies need to get on the same page before the U.S. government gets involved, because government intervention before there is a concrete industry proposal is likely to produce something dumb. Nathan's framing was blunt — "you guys have got to figure it out," with government stepping in as a last resort nobody would enjoy, himself included. He also endorsed Schulman's point that the antitrust risk facing any coordination is overstated, or "fake," but argued the government should remove that doubt explicitly rather than leaving labs to assume it: a couple of sentences of reassurance, he said, would do it, and the current administration — not some hypothetical future one — is the one companies actually need to worry about before the midterms.

    Prakash pushed back with a historical analogy: Operation Warp Speed and the COVID vaccines. Pharmaceutical companies, he noted, specifically demanded and received a legal safe harbor from vaccine-injury claims before agreeing to move fast, and post-COVID backlash — genuine anger over statistical misunderstandings about vaccine deaths, among other things — showed the concern was well-founded, not paranoid lawyering. His point: "only the laws bind decision-makers in the future" — today's executives can be perfectly well-intentioned, but their word does not bind their successors, so researchers like Schulman who argue that doing the right thing matters more than legal cover are underestimating real liability exposure, even if the law is sometimes slow to catch up with how society actually needs to be governed.

    Prakash then turned to the other announcement of the morning, reading it out as it landed: Paul Christiano has been named to OpenAI's board, specifically its Safety and Security Committee. He read from Christiano's own statement — that based on the recent trajectory of capabilities and the continued difficulty of alignment, Christiano now sees a meaningful risk that rapid capability acceleration leads to catastrophic and irreversible loss of control in the near term, and that domestic and international coordination will be needed to bring that risk down to an acceptable level. Prakash used the announcement to unpack what "rapid acceleration" concretely means to people like Christiano: not the 10-20% GDP growth economists already consider implausible, but Dyson-sphere-by-2030-or-2040 territory — by his own back-of-envelope math, something on the order of 640% annual global GDP growth by 2040, versus Tyler Cowen's estimate of AI adding roughly one percentage point to trend growth. Prakash's view: virtually nobody outside a small AI-focused clique in San Francisco believes numbers like that are achievable — not even Jensen Huang, he guessed — which is exactly the scenario Christiano is trying to guard against.

    Nathan was enthusiastic about the appointment, calling Christiano "as OG as they come" — a co-inventor of RLHF whose collaborator Ajeya Cotra wrote the widely cited essay arguing the default path to AGI likely leads to takeover, a thesis Nathan sees being vindicated in real time by the pervasiveness of reward-hacking ("cheating") across AI training. He noted Christiano stepped down from Anthropic's board in 2024 to lead the government's AI Safety Institute (since renamed CAISI), making the OpenAI seat a natural landing spot, and the two traded a lighter aside about Christiano's near-silent, extremely high follower-to-post Twitter ratio and Prakash's joke that Christiano traded Pete Hegseth's Signal group for OpenAI's Slack.

    The conversation turned to what actually bottlenecks a runaway self-improvement scenario. Prakash argued the strategies for managing ordinary single-digit economic growth are qualitatively different from those needed to prevent hundred-percent-plus growth, floating the idea that constraining robotics specifically — or power growth generally — might be the more tractable lever than trying to slow intelligence itself, since power and land cannot grow exponentially the way compute-driven intelligence-per-watt potentially can. Nathan connected this to a public disagreement he had noticed between Tim B. Lee, a recent guest, and Zvi Mowshowitz: Lee's decade-plus timeline for broadly useful humanoid robots versus Zvi's registered dissent. Nathan called the exchange a clean, almost comical illustration of the field's deeper divide — whether robotics bottlenecks hold or get dissolved by coding agents that just rewrite the constraints — and stressed that a significant share of people inside frontier labs genuinely expect recursive self-improvement soon, at a scale of change he called mind-boggling, not a fringe belief he was exaggerating for effect.

    Prakash offered his own three-way frame for the debate — AI researchers as "math essentialists" who think solving math cascades into solving everything, economists who counter that most real-world bottlenecks are coordination problems that take time regardless of technical capability, and a "super-persuasion" scenario where AI itself talks human institutions into moving faster. He put the next twelve months as the test of whether progress climbs past the "math is basically solved" rung. Nathan, wrapping toward the hour, floated an idea for a future segment: whether a genuine AI doomer could actually structure a trade to profit from a catastrophic scenario — the standing challenge from economist Tyler Cowen that doom talk is just unfalsifiable "mood affiliation" unless backed by a real position — and said he has already had Fable and Astra researching historical analogues, such as World War II-era markets, and drafting a prospective portfolio. Prakash countered with his long-running argument with podcast guest Liron Shapira over forcing subjective probabilities to reveal themselves through actual portfolio bets, versus Liron's defense that 50% does not distinguish from 10% or 90% at this level of uncertainty — which Nathan steelmanned via Zvi Mowshowitz's line that "sometimes you only get one significant digit on doom," arguing it is a double standard to demand numerical precision from doom estimates when the upside-utopia case gets none either. Prakash closed with a thought experiment — AI doomers competing for philanthropic dollars in a room with Greta Thunberg and Jesse Jackson — concluding they would fare badly against causes with more evident daily stakes, and that the movement would have been better served pushing concrete asks like deepfake and image-rights protections than "end of the world" messaging. Nathan ended the show on an optimistic note about American philanthropic capacity, citing Coefficient Giving's new "Project Tailwind" — a call for AI-safety-relevant startups with up to $200 million-plus available in tranches — and argued that the path forward runs through private capital rather than Congress or a democratic process he doubts can move fast enough, with Christiano's board seat as a fitting note to end the show on.

    Timestamp links open the original source recording.

    You guys have got to figure it out. If the five of you can't get on the same page, something dumb is going to happen.

    As Zvi puts it, sometimes you only get one significant digit on doom.

    Once you've got Jesse Jackson and Greta Thunberg in there, you guys are not coming out with even one percent of the money.

    Lightly edited · timestamps jump to YouTube
    2:08:50

    Nathan Labenz: I have one thing that popped up for me — I want to go back to our earlier conversation for a minute. This is John Schulman, formerly of OpenAI, where he led post-training on ChatGPT if I understand correctly, then went to Anthropic briefly before leaving to join Thinking Machines. He's responding to Jacob's post — somewhere down the thread we reviewed earlier — where Jacob says he's optimistic about the potential for coordination. John Schulman is saying something pretty similar to what I was on about earlier: these companies need to start working together first. As he puts it, bringing in the US government before there's a concrete proposal will likely result in something dumb. That's your point too — there's nobody better positioned than the people at the companies to do this. I certainly don't want to see the Navy, or whoever, come in and try to regulate AI, but I think this is the recipe. I've been circling this myself, and seeing him say it reinforces it for me. In short: you guys have got to figure it out. If the five of you can't get on the same page, something dumb is going to happen — and I guarantee, kind of like I tell my kids, you're going to like it a lot less if I have to come in and adjudicate the dispute for you. If you get on the same page, you can go back to having your fun and I won't interfere. But if you can't get over this hump, it becomes my job to come down and be the bad guy. So I really like this — and I also think his point about antitrust being fake is true, but I think the government should take that doubt off the table. It would only take a couple of sentences to say, hey, you really don't have to worry about this anymore. Yes, the law could change — the administration is going to change eventually — but we're not even to the midterms yet, so for better or worse this is the administration they have to worry about for the foreseeable future. These guys genuinely are planning around a singularity before Trump is out of office — that's at least a very live possibility for them — so I don't think they're going to be too concerned. And if they do raise that concern, we all know it's fake — if they say, 'well, Trump said we don't have to worry about antitrust in our safety collaborations, but what about the next president?' — come on, that's so ridiculous given everything else they're doing that I don't expect them to say it.

    2:11:51

    Prakash Narayanan: The proof point is Operation Warp Speed and the COVID vaccines. For Operation Warp Speed, the pharmaceutical companies specifically required a waiver from vaccine-injury claims — they wanted a safe harbor, and they got it, through legislation. Post-COVID, it's very clear that if they hadn't gotten it, they would have been sued into oblivion. There are a lot of people who remain very upset about being forced to take the COVID vaccines, and there was a lot of scientific misinformation and statistical misunderstanding around it — yes, maybe one in a million people died shortly after getting the vaccine, but when you vaccinate a million people, statistically some of them are going to die at some point regardless.

    2:13:26

    Prakash Narayanan: That's a proof point that these are valid concerns — a company can be wiped out in retrospect, and the lawyers aren't being foolish when they warn you that this will come back to haunt you if you don't get that safe harbor. All the decision-makers right now can be perfectly on board, and it doesn't matter, because only the laws bind decision-makers in the future — right now, they're bound by their word at best. So I think it's unfair to say the lawyers don't have a point, because this exact thing has happened in just the past five or six years. And I think researchers like Schulman will always refuse to accept that — they'll say the laws aren't what matters, doing the right thing is what matters. But in fact, that's how society is governed, even if the laws are a little slow to catch up.

    2:14:11

    Prakash Narayanan: But let me segue quickly to this — Paul Christiano. It's just been announced that he's joining the OpenAI board, specifically the Safety and Security Committee, to support safety oversight. He says: based on the recent trajectory of capabilities and the continued difficulty of alignment, he now believes there's a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term. The Safety and Security Committee has an important and challenging role overseeing risk management at OpenAI — full automation of AI R&D could lead to very rapid acceleration in capabilities very soon, meaning improvements in training algorithms compound on themselves. Second, we can currently train AI agents with RL to get as much reward as they can, and it has long seemed theoretically possible that this could motivate agents to undermine human control and seek power and resources. An intelligence explosion would greatly exacerbate the risk of misalignment. And there's one line in there where he says: if we build superintelligence without more robust alignment, I expect we will permanently lose control of it — if that happens, most people could die. I believe we would need domestic and international coordination to ensure global consistency and reduce risk to an acceptable level.

    2:15:41

    Prakash Narayanan: So, he lays that out. I think the one thing that would help is if he framed more clearly what he means by 'rapid acceleration' — that's not very clear to a lot of people. People hear that and think, 'oh, things are accelerating right now.' But when people like Paul Christiano talk about rapid acceleration, they mean Dyson spheres by 2030 or 2040 at the latest. I ran some numbers on the 2040 case earlier — assuming energy production growth tracks GDP growth, a 2040 Dyson sphere implies something like 640% per-annum global GDP growth. Right now global GDP growth is about two to three percent. So when people like Paul Christiano talk about rapid acceleration, that's the scale they mean — not accelerating to 10%, 15%, even 20% GDP growth, which economists today think is absolutely impossible. Tyler Cowen, for instance, puts AI-associated GDP growth at plus one point — from maybe 2% long-term to 3% long-term — and even that would be better than all of human history put together. Christiano is talking about 640 to 650% on a 2040 basis, or a couple of thousand percent on a 2030 basis. I think that would be great, but I also think it's too fast — and I don't think anyone outside of AI, and outside even a few people within AI, believes it's possible. I don't even think Jensen Huang thinks 600% GDP growth is possible. So outside of a very small clique of maybe a few hundred to a few thousand people in San Francisco, nobody believes hundreds of percent GDP growth is possible. And that's what Paul Christiano has been thinking about, and that's what he's guarding against.

    2:18:05

    Nathan Labenz: Well, it's great news — I love to see it. He's as OG as they come in terms of having been a serious, rigorous thinker about this for a long time. He was one of the lead authors on RLHF in the first place, so he's come up with meaningful parts of the paradigm that brought us here — nobody can level the 'he doesn't understand the technology' criticism at Paul Christiano. And his close collaborators, including Ajeya Cotra, who wrote the piece arguing that the most likely default path to AGI leads to AI takeover — that's very much being vindicated right now. As we train AIs on an ever-broader range of tasks, and they're a little vibe-coded and a little sloppy, what we're seeing is that cheating is the common denominator across all these domains, and until we have a good solution for that, we should expect these problems to continue. This is something Christiano saw coming years ago, even as the creator of core techniques. So I think this is outstanding news. I hadn't even realized he'd left the Anthropic board, which he used to sit on — I just looked it up, and apparently he stepped down in 2024, which makes sense, because he went to the government to lead what was AISI for a while — or whatever it's called now, CAISI. So he naturally had to recuse himself from the Anthropic board. But I think the OpenAI board is a great place for him now, and from your lips to God's ears that he'll do a great job.

    2:20:02

    Prakash Narayanan: He left Pete Hegseth's Signal group and joined the OpenAI Slack instead — so yeah, I'd better be happy about it.

    2:20:13

    Nathan Labenz: Great Twitter stats for him too. One of the things I often look at is the ratio of followers to posts — he's got two posts, total, life-saving. The account goes back a long way, but there are no posts on it, only two — maybe he cleared it earlier. Either way, the ratio of followers to posts is currently extremely high for Paul Christiano, as it should be.

    2:20:39

    Prakash Narayanan: I think it's good. I do wonder, though — there are strategies you use to make, say, $100 a year, and there are strategies you use to become a billionaire. Billionaires rarely start out working for someone else; very few of them made their money that way. In the same sense, I think there are strategies for managing two, three, four, five percent economic growth, and completely different strategies for thinking about hundreds of percent economic growth — different regimes entirely. So the strategies you'd use to prevent that kind of runaway system are probably counterintuitive relative to managing a several-percent-growth system. I wonder to what extent just stopping robotics is sufficient — if you stop robotics, maybe you're automatically limited by mostly humans working in the real world. I wonder to what extent stopping growth in power would work too. For me, one of the key questions has been how much additional intelligence you can get out of the same unit of power.

    2:22:10

    Prakash Narayanan: Because I think power production just isn't going to move that fast — it's a physical activity, and you roughly know the theoretical limits on solar panels, on batteries, on how fast you can actually build things. The robotics question is really about deploying labor. So the real question is how much intelligence you can get out of a unit of energy, and what the theoretical limit of that is. The best proof point we have right now is humans — Einstein running on twenty watts, more or less — and we have a lot more watts than that in the system already. Can you get closer to that ratio, or are you not going to get there with current technology anyway? There's a lot of uncertainty there. But I'm fairly certain power and land are not going to see exponential growth — you can't get exponential growth in land usage or power supply. Exponential growth in intelligence-per-watt, though, catching up to the human curve there — that's the dangerous part, perhaps.

    2:23:23

    Nathan Labenz: Certainly all these runaway stories you hear from people like Daniel — that whole core group — run through robotics. Their expectation is that the machine economy takes on a life of its own, with a significant share of robotic output immediately going toward building the next factory that builds more robots. So the feedback cycle runs on a pretty clear through-line from the data centers out into the physical world, through self-replicating factories, and eventually that moves into space, which is how you get your Dyson sphere process started. There are probably any number of bottlenecks that, if they really held, would make that story essentially impossible — but we keep looking for walls we can't run through, and so far we haven't found them. There was a funny exchange yesterday between Tim B. Lee, who we talked to last weekend, and Zvi Mowshowitz.

    2:24:53

    Tim basically said online the same thing he said to us — that it's going to take a long time, maybe a decade or more, for humanoid robots to be broadly deployed and doing much useful stuff. And Zvi said something like, 'I'd like to register my prediction about this...' That, again, puts the divergence in worldviews into a comical but pretty clear light. Either you see these things hitting threshold points — progress in robotics currently limited by X, Y, Z, whatever — and then coding agents just rewrite those rules and run through the barriers, converting and optimizing whatever needs it, and suddenly you don't really have that problem anymore, or you have a new kind of problem. Or, obviously, some people just don't believe that yet. But it is striking — I think we do a pretty good job of making this clear to our modest audience, but the people at the frontier companies really do believe it. A very significant share of them expect this recursive self-improvement thing to happen, and to happen soon. And even if it levels off at some point — it's not that there'll never be a leveling off — they expect that leveling to happen well above our current capabilities, generating a scale of change in GDP growth, or in the number of robots walking around and how fast that compounds, that truly boggles the mind. And I should be very clear about that — I know you are, but everybody should understand that that is really what they think.

    2:26:53

    Prakash Narayanan: I think we'll see within twelve months — this is the period where we've hit the Millennium Prize moment for AI, so to speak. The way I'd frame it: I regard the AI research crowd as math essentialists. They believe that getting good at math means getting good at physics, getting good at physics means getting good at chemistry, getting good at chemistry means getting good at biology, and once you have all of that, everything is basically solvable and will get solved. The economists, on the other hand, think that's completely untrue — getting good at math doesn't mean anything on its own, there's no direct economic value to solving Millennium Prize problems, implementation takes time, and most of the problems we actually face are coordination problems. A copper mine takes thirty years because of environmental protests, not because of technology — those are problems caused by how humans make decisions, slowly and deliberately. So you have the math essentialists on one side and the economists on the other, saying diffusion takes time, people take time to adapt.

    2:28:24

    And then there's the counter to that — which is super-persuasion. That would mean the machines persuading, or enabling, human organizations to adapt — giving them the capability to move very quickly, persuading them to go forward, and persuading them that things are going to be okay. So there are these three levels: the math essentialists, the economists, and the super-persuasion argument. My own view is, okay, we just proved math has basically been conquered — that's your first rung met. Now let's see over the next twelve months whether the next rungs get met, whether economic growth actually goes up, and if it does, whether humans stop it at some point, or whether the AIs manage to convince humans not to. There's still a lot more to go before we get there. And people like Paul, I think, are much further along on that side — they think it's going to happen, so we have to manage it now. We're a little tight on time today, so it's on you when you want to break.

    2:29:35

    Nathan Labenz: Yeah, we should probably wrap up — but there's one thing I'd like to dig into more in a future broadcast: is there any way to short the market if you're a doomer, in a way that answers the Tyler Cowen objection? You mentioning his name got me thinking about this — he'll often say that you doomers are all just mood-affiliation talk, and if you were really serious about it, you'd be short the market. And people respond, well, not really, because what you'd actually expect is that the market goes up right up until there's an irreversible catastrophe.

    2:31:06

    And then he says, well, now you should be able to find a way. So I've actually had Fable and Astra doing some work on this — starting by looking back at major disruptive moments in history and asking, if you were in that period, in that system, would there have been any way — even with perfect foresight, but limited by the laws of physics, the trades actually available to you, and the problems unfolding around you — to trade your way to getting rich by the end of a catastrophic period in history? Something like World War Two, in Germany or Japan, say. And then, by extension, for today: are there trades we should be looking at, that I should be looking at? I don't know how — if you want to, you know, 'I'm a doomer, trade with me' — but I have this idea that maybe I should. Maybe there are some trades out there. I don't know, but I'm at least pushing the agents down that path, and I think it'll be pretty interesting to come back and say, okay, here's the portfolio — first, here are some situations where you couldn't have traded your way through, and second, here's an AGI's best attempt to do that today. Is it compelling?

    2:31:54

    Prakash Narayanan: So, first — let me give you the backstory. We had Liron Shapira on the pod. I've been back and forth with Liron for many years. One of the exchanges was that he kept saying, okay, all of this stuff is going to happen, et cetera, and I'd say, okay, what does your portfolio look like? Because your portfolio is your revealed preference. So I proposed that his actual expressed probability is 90% — you can do a decision-tree analysis on portfolio weights and back out the expressed probability from the actual weights. And he said, I don't think the resolution of my probability really distinguishes 90% from 50% from 10%. When I say 50%, I just—

    2:32:39

    —mean 10% or 90%. So that's a kind of anumerical analysis. And, you know, Liron's a very smart guy — very, very smart. But this anumerical analysis is what the doomer contingent ends up using, and I find it very annoying — it's the commentary that's always been there, which is that they take one viewpoint, assign too high a probability to it, and treat it as though it's definitely going to happen. And that's Liron in a nutshell. He's like, look, when I say this, I mean it could be here, it could be there — I'm 10% to 90%, I just mean 10 to 90% when I say 50. What does that even mean? When you say you're right in the middle, it means you're all the way from one extreme to the other. What does that even mean?

    2:33:53

    Nathan Labenz: Well, I'll give you the steel-man of that, because I say something similar myself sometimes. I think what it means is: it doesn't really matter if you're at 10% or 90%. Everybody who sees that as the actual range should come together and say, this is insane, we need to do something to get out of this range. It's his way of trying to rise above the narcissism of small differences — the stuff that looks like weird bickering among people who should be allies, when the real focus should be elsewhere. You might be at 10, I might be at 90, but we can both agree that 10 is too high — so let's deal with that and not worry too much about—

    2:34:39

    —the exact number. Or, as Zvi puts it, sometimes you only get one significant digit on doom — it's kind of silly to try to get super precise about this. It's a selective call for rigor, honestly — the fantastic-upside, abundance-utopia side of AI isn't numerically forecast either. We're not asking Dario, 'when you talk about explosive economic growth, do you mean 25% a year, or 28%, because I want to put a specific trade on that' — that would be insane. If it goes well, it'll be great. On the downside, if it doesn't go well, it could be really bad — and who cares if the number is 10, 15, or 90?

    2:35:27

    Prakash Narayanan: The other thought experiment I have is: take the AI doomers and put them in the same room as Greta Thunberg and the climate doomers, and also civil-rights activists like Jesse Jackson and a couple of others. Say you all have a million dollars and have to agree, by vote, on how to allocate the budget across your causes. I wonder what number the AI doomers would come out of that room with — because once you've got Jesse Jackson and Greta Thunberg in there, you guys are not coming out with even 1% of the money. It would be tough. You have to fight for priorities — that's what happens in a democracy. You might all agree on the same side in principle, but you still have to fight for priorities, and that's why, for example, the homeless problem never gets solved. People say, if Elon gave $40 billion, homelessness would be solved. No — if Elon gave $40 billion, it would get allocated to other priorities, because homelessness isn't a high priority even if everyone agrees, in the abstract, that it's important to address. In a democratic process there are other priorities that are more pressing, more evident, and that people believe in more at the end of the day. So it's a tough one. I think the doomers would actually be better served focusing on things like—

    2:36:57

    —child safety, probably. That would work. Deepfakes has definitely worked as an issue, and honestly they haven't pressed on deepfakes nearly enough — they could have said, look, you should have a right to your own image, and because you have that right, we should be monitoring GPUs so they aren't being used to make deepfakes. That path would have constrained AI development quite a bit, but they didn't take it — they pushed on the end-of-the-world message instead, which I don't think has been a great avenue.

    2:37:34

    Nathan Labenz: Yeah — maybe we'll call it there with this last bit, but this is another notable piece of news, and also an interesting reminder of one of the key strengths American culture and society has. In my US-China series on the Cognitive Revolution, one of the things I've emphasized is that the vibrant nonprofit sector we have here — which, yes, requires philanthropic money, and is therefore somewhat dependent on the billionaire class, which people can complain about for all sorts of reasons — has given us the AI-safety awareness and depth of bench that we have.

    2:38:20

    These people are opening their wallets right now in a very serious way. Another example: Project Tailwind, out of Coefficient Giving, just put out their call for startups — up to $200 million-plus that they're willing to commit, in tranches over time, behind things that really look like they're working. So I think the democracy path is probably not actually how this plays out. If it goes well, I don't think it goes through Congress. You can dream of a different Congress, and I'm all for that — I've got a whole long dream about how we revitalize democracy — but the timelines are such that I don't think it's actually going to play out that way. It's going to be mostly private actors with private money going and doing things, and the government, at best, saying, here are some places we won't get in your way, and maybe some motivating threats beyond that. And OpenAI is in a somewhat better position today than it was yesterday, with Paul Christiano joining the board. If you've got your own ideas about things you can do, there's increasingly money available.