AI:AM LIVE — September 21, 2026 — Coefficient Giving's Max Nadeau on Project Tailwind and $200K-to-$200M Checks for AI-Safety Founders, Why Alignment Research Resists Metrics, and What Third-Party Auditors Are Actually Allowed to Test — Plus Hiring an Autonomous Business Partner Live On Air and Amazon Banning Meta's Muse

The rundown

1. 18:33Opening56 min

   Opening — Jev and the return of the classifier, whether talking agents beat parallel ones, Terence Tao's plea to slow down, and hiring an autonomous business partner on airNathan Labenz and Prakash Narayanan open twenty minutes late on audio they never fully fix, and start with Jev — a model that returns a calibrated choice from a fixed set rather than a chain of thought. Nathan likes the shape of it as a control surface: give an agent a finite menu at every step and the Hugging Face-style exploit chains get harder, though he is careful not to claim it takes the risk off the table. Prakash's counterpoint is that system-1 answers remove the reasoning trace you were monitoring. Then a paper on test-time communication — whether a team of N collaborating agents beats best-of-N independent ones — where four communicating agents find a 1,957-byte MNIST model at 99.4 percent accuracy, twenty percent smaller than the best human solution, while no independent agent gets under three kilobytes. Nathan reads it against Noam Brown's Dwarkesh interview, where Brown would credit the multi-agent setup with no more than ten percent of the Millennium Prize result, says OpenAI has no real grasp of multi-agent scaling laws because the experiments are too expensive, and concedes the researchers have been surprised by the pace. Prakash then walks through Terence Tao's call to slow down and the pile-on it triggered, and argues that a thousand-year pipeline for producing human mathematicians died in a month once the prestige of being first was taken off the board. The hosts put Cue, the show's AI cohost, on a new bidirectional speech model and let it fumble a live web search on a reportedly hallucinated intelligence story. They finish by signing up for Andon Labs' waitlist-only Pion, naming the agent Dawn, pointing it at sponsorships, and telling it to send outbound email without review.

   Watch

   ▶Play this segmentYouTube · from 18:33Open segment on YouTube ↗

   As aired

   The show opened live but rough: Prakash Narayanan and Nathan Labenz had audio trouble sorting each other out just before air, and Prakash noted the first-ever live chat message had come in from YouTube rather than X. He framed the show's own dual-platform live chat -- working simultaneously across X and the AI:AM/AI AM YouTube channels -- as a first in live-show history, since X itself has no good native chat system.

   From there the hosts turned to "Jev" (JEV), the fast, cheap classification model that had suddenly become the week's talking point after open-source clones appeared within hours of its release. Prakash was skeptical of how significant it really is, noting it's fundamentally a classifier -- it chooses among a fixed set of outputs rather than reasoning -- and questioned whether the rapid open-source replication proves it's a shallow trick or a genuinely useful paradigm. He drew out the "system 1 versus system 2" framing: Jev has no chain of thought, so if there's reasoning happening at all, it's compressed into the model weights and produced in a single forward pass via multi-token prediction, not narrated step by step.

   Nathan pushed back toward the optimistic case. He likes the shape of Jev as scaffolding for guardrails -- giving an agent a fixed menu of allowed next actions, he argued, is a simple, generalizable way to keep AI systems in check, and he explicitly tied this to the Hugging Face incident, suggesting a forced-choice action space might have limited how far an agent could chain together exploits. He also volunteered a concrete use case for the show itself: their live-chat comment moderation currently routes every comment through a full LLM call, which can take up to 60 seconds and makes new comments feel like they've missed the conversation; he wants to swap that for Jev to get moderation down to a couple of seconds. His bottom line on the fast, cheap classifiers being clonable: cheap and fast enough that he'd rather just pay for Jev than reinvent it, calling it "a cool new paradigm" and a welcome diversification away from constantly cramming every problem into the same autoregressive-generation mold.

   Prakash then shared a research paper (crediting a researcher's article, shared via a post from Demis Hassabis) on "team of n versus best of n" -- testing whether multiple communicating agents genuinely out-produce the same number of agents working independently and taking the best result. The standout data point: on MNIST compression, a team of four 5.6-Sol agents produced a 1,957-byte model at 99.4% accuracy -- about 20% smaller than the best human solution on a problem that's been worked over for decades -- while no single independent agent got below three kilobytes. Prakash argued OpenAI and Anthropic have likely known this internally for at least a year without publishing it, and that it took an outside academic to surface it publicly.

   Nathan connected this to Noam Brown's recent appearance on the Dwarkesh podcast (heard on air as "Dronekash"), where Brown discussed the multi-agent "swarm" that helped crack a Millennium Prize problem (Navier-Stokes) -- but was notably cautious, attributing no more than about 10% of that success to the multi-agent setup itself and insisting the bigger story is simply that the underlying model is very powerful. Brown reportedly said the swarms use deliberately simple send-message/broadcast tools rather than heavy scaffolding, are given a "behavioral prior" from pretraining plus some supervised fine-tuning before RL, and that OpenAI still lacks a clear picture of multi-agent scaling laws because full-scale experiments (Brown cited roughly 130 billion tokens for some of this work) are too expensive to run at the sample sizes needed for real statistical clarity. One thread Nathan flagged as unaddressed: the cryptic, seemingly non-human-readable messages Astra reportedly sends to its own sub-agents. Brown also said identifying one agent explicitly as "the user" measurably improved honesty and alignment metrics among the others -- suggesting the agents may be more aligned to a human overseer than to each other.

   The conversation's biggest thread was the fallout from mathematician Terence Tao's post (which the hosts tried and failed to play a clip of) calling for AI development to slow down, warning of "extremely nonlinear dynamics" in math and elsewhere. Backlash was swift -- Prakash noted Eric Weinstein publicly challenged the premise that Tao is even "the greatest living mathematician," pointing to other living Millennium-problem solvers. Prakash's own read was that the real trigger wasn't the slowdown post itself but the OpenAI-credited solution to Navier-Stokes, which prompted mathematician Tristan Buck Meister (who'd been working with Anthropic on a precursor result) to accuse OpenAI of essentially having spied on his AI usage and trained on it. Prakash argued this broke a centuries-old "prestige pipeline" for pure mathematics -- since a human can no longer claim to have solved something AI-unassisted, and even human effort now invites accusations of hidden AI use -- and cited Grant Sanderson (of 3Blue1Brown) as floating the idea that mathematical prestige may shift from discoverers to explainers. He also relayed a Chinese-language reader's comparison to the "sophons" of Liu Cixin's science-fiction trilogy: an AI so far ahead on math that it functionally locks human mathematical progress by destroying the incentive to do it at all.

   Nathan was more agnostic about how it shakes out, floating the chess analogy -- human chess remains culturally popular and arguably at a relevance peak decades after computers surpassed humans -- while acknowledging math's "never been done before" prestige doesn't map cleanly onto chess's head-to-head drama. His practical recommendation was not to overreact at the societal-funding level and to let the field figure out its own reinvention over time, since math isn't a huge share of total jobs. He tied the same anxiety back to Noam Brown's timeline: 2023-level math tasks took a mathematician seconds, IMO-level problems (2025) take roughly 100 minutes, and by his account they're now bumping into problems in the 1,000-10,000-minute range -- and Brown's own key admission, in Nathan's telling, was that the researchers themselves have been surprised by the pace of capability gains. Nathan added that Brown's answers on safety were "not super inspiring," citing Brown's statement that only about 10% of his team works on safety and alignment. Prakash separately noted the Hodge conjecture -- another Millennium Prize problem -- is reportedly close to falling to OpenAI as well, pending verification, and reiterated his own prediction that human cognitive labor could be largely obsolete in the US by 2031, which would mean university professors are functionally superseded within two to three years.

   Nathan tied the theme beyond math to his own work, teasing (under embargo) a new text-to-speech model he'd already used on the weekend's highlights episode -- good enough that his own editor couldn't tell it wasn't his real voice -- as evidence the same surprising-pace phenomenon is "coming for all of us," not just mathematicians or programmers.

   The hosts then brought their AI cohost Cue on air -- introduced as running on OpenAI's new bidirectional "Live" voice model with a GPT-5.6 Sol backend and web search -- to recap the week's top AI stories. Cue's headline: safety concerns moving from technical debate into politics, including slowdown calls, data-center grid strain, and a military "near miss" caused by hallucinated intelligence. Pressed for details, Cue described a chatbot-invented story in which analysts believed a Chinese ship near the Strait of Hormuz was carrying nuclear cargo -- aircraft were reportedly launched before the error was caught, and the Pentagon has not confirmed the incident -- with Cue emphasizing the cargo never existed and the model had made it up.

   In the segment's final stretch, ahead of guest Max from Coefficient Giving, Nathan got the hosts off the waitlist for "Pion," a new product from Lukas and Axel that stands up an autonomous AI-run business function. Live on air, Nathan signed AI:AM up, chose "other" as the business category, opted in to data training, and was greeted by a managing agent the hosts nicknamed "Andonos." Asked to name their own business-development sub-agent (offered "Amy or Dawn"), Nathan picked "Dawn" to avoid a name collision with his wife Amy. They set an initial goal of 20 sponsor-pipeline leads, told the agent to "let it rip" and draft/send outbound without requiring review, and speculated about which model was running underneath -- guessing Astra for the vendor's own agent (since it topped their benchmarks) but choosing Claude Opus 5 or "Soul" (Sol) for their own agent's sub-tasks, since Nathan felt that kind of procedural search work didn't need a frontier-tier model. The segment closed with light banter about AI agents over-disclosing their identity ("I'm Aide, a Claude-powered agent...") before the hosts wrapped to bring Max on.

   Key moments

   So it's a yes, no. It chooses between a number of options. It ranks a number of options.

   A team of four 5.6-Sol agents finds a 1,957-byte model with 99.4% accuracy, which by the way is 20% smaller than the best human solution on a problem beaten to death.

   When the voice sounds as good as my real natural human voice, you know, what moats are left to me then? It really is coming for all of us.

   What we covered

   Jev and the return of the classifier A model that returns a calibrated choice from a fixed set rather than a chain of thought. Nathan likes it as a control surface — hand an agent a finite menu at every step and the long exploit chains get harder — and wants it for the show's own live-chat moderation, where an LLM call currently costs up to a minute of latency. Prakash's counter: their system 1 has no chain of thought, so the reasoning moves inside the weights where you can't watch it.

   Test-time communication versus best-of-N A paper asking whether N agents that can talk to each other beat N run independently and taking the best. On MNIST compression, four communicating agents find a 1,957-byte model at 99.4 percent accuracy — twenty percent smaller than the best human solution on a problem beaten to death — while no independent agent gets under three kilobytes.

   Noam Brown on what OpenAI does not know about multi-agent systems Nathan's read of the Dwarkesh interview: Brown would credit the multi-agent setup with no more than ten percent of the Millennium Prize result, says OpenAI has no real grasp of multi-agent scaling laws because the experiments cost millions of dollars of compute to run, and gives the agents deliberately simple message-passing tools plus a behavioral prior. The line Nathan wants shouted from the rooftops is that the researchers have been surprised by the pace.

   Terence Tao's plea to slow down, and the status fight inside mathematics Tao argues the nonlinear dynamics make this too fast; the replies call him a loser and question whether he is the greatest living mathematician at all. Prakash's claim is bigger: a thousand-year pipeline for producing human mathematicians died in a month, because the prestige of being first to a result is gone and prestige was the incentive structure. A commenter's framing that stuck with him is the sophon — advancing a field so fast that its people stop developing it themselves.

   Cue on a new bidirectional speech model The show's AI cohost is running on a new live speech model with web search. It fumbles a search in real time, then surfaces a report of a hallucinated-intelligence incident — a chatbot that invented nuclear cargo on a Chinese ship near Hormuz, with aircraft airborne before someone caught it, and no Pentagon confirmation. Cue says plainly that the cargo never existed.

   Onboarding Pion, and naming it Dawn Nathan signs up live for Andon Labs' waitlist-only Pion, opts in to training, picks Dawn over Amy to avoid colliding with his wife's name, and points it at sponsorships with a goal of twenty leads. Asked whether every outbound proposal should be reviewed first, he says let it rip. Prakash reads the whole flow as a go-to-market SaaS harness being internalized into the company that needs it.

   Related

   Andon Labs — makers of Pion, the autonomous business agent onboarded live in this segment ↗

   Full transcriptLightly edited · timestamps jump to YouTube

   2:11

   Prakash Narayanan: Alright. Good morning. It is Monday, September 21, 9:18am. We had a little bit of, you know, audio issues, and now we are live. Nathan, good morning.

   2:29

   Nathan Labenz: Good morning, Prakash. You're still a little choppy coming through for me even now, but hopefully, I can at least say it, and the show must go on as they say. So we got one live chat from, r h I z I n, Recien, when does it start? Usually, promptly at 9am Pacific. Today, for whatever reason, we were having a hard time understanding one another. So before going live, we tried to sort that out, and here we are. It's the beauty of being on the Internet and not having an, you know, really firm schedule. We can kinda do what we need to do. Hopefully, it'll work.

   3:07

   Prakash Narayanan: And that was our first chat from YouTube, actually. That was our that was our first ever chat from YouTube. And we have a we have a hello coming in from Malcolm tattoo. Hi, Malcolm tattoo. So we have working chat now. We have working chat across the AI AM AI and the AM YouTube channel and across the all our x channels. This is the only show, by the way, the only show where you can actually have a live chat on x because everyone else you know, x does not have a great system for that, but we built one. So we have a live chat across, x and YouTube, first in history.

   3:53

   Nathan Labenz: Yeah. It's amazing what Astra can do. I did check the stream, and it you do sound okay to me coming through on the stream. So for whatever reason, the issue seems to be just for me in the studio. Listeners should be okay. So I'll do my best. Welcome to the JEV era. How are you enjoying the JEV era?

   4:20

   Prakash Narayanan: I really do not know if JEV is significant or not. It is so difficult for me to tell at this point because okay. So Jev comes so Jev comes out. All of a sudden, everyone's like, wow. It's so fast. It does all of this stuff. Like and then, like, within, like I think, like, you know, four or five hours, the open source guys are like, oh, we can do this. And all of a sudden, people are running very small models and putting out classifications. JEV, at the end of the day, outputs classifications. So it's a yes, no. It chooses between a number of options. It ranks a number of options. And I think the idea behind Jav was really calibration. So it's calibrated so that you do get the intelligence coming out, and it's not just a random classification. But a lot of the people a lot of the open source guys are trying to push out basically classifiers, and it's not very clear how intelligent those classifiers are. But they're pointing out that they are also very fast, and they are doing classification. And so now we have this, like, is Jev what do you think? Is Jev really something meaningful?

   5:41

   Nathan Labenz: I think it's a cool paradigm, and I do like the shape of it for some things. You know, in a sense, it can be it feels like it can be scaffolding for a broader set of guardrails and control measures. Giving the AI a fixed set of choices and saying, you must make one of these choices is, I think, you know, pretty simple and intuitive and, like, highly, you know, generalizable way to make sure that we stay in control of what the AI is doing. Right? Like, if you were giving the agents in the Hugging Face incident finite sets of, like, what action to take next each time and forcing it to choose from the actions that you were giving it, presumably, you would have a lot less risk of it chaining together, you know, all these crazy exploits and pursuing its own goals at the length that it did. I don't think that it's necessarily super obvious that even just because you start to implement these checkpoints along the way in an agentic context that, like, that takes all the, you know, risk of rug behavior or unintended behaviors off the table. But it sure seems like it would reduce it quite a bit. And the fact that it's cheap and fast is obviously a great selling point. So I can see it being really quite useful. one of the things that I'm interested in trying it for within the context of this project is just comment moderation. We've had you know, as any unassuming Internet project does, you know, you put something up and immediately you start to get crypto spam. So we've had crypto spam, so then you've had to implement moderation. But then we've had people saying, hey. My comment doesn't look like it's showing up for, like, up to 60 seconds. What's going on? And, of course, what's going on in the background is we're taking that content, an LLM call and getting a response, and that LLM has to do its reasoning and blah. And by the time the comment shows up on the screen, you know, open of a minute has passed, and so people are like, oh, that didn't really come through. Or, you know, by the time they we see it's we're kinda onto the next topic of conversation. So if we could get that down to, you know, a couple seconds because we just immediately feed it into JEV and it tells us, yes, no. You know, you can publish this. I bet we that. And, you know, at least the I think it's been really interesting to see how easily it's been cloned. Long tail questions that are hard to answer, you know, about any model. And so at the moment, I'm kinda like, it's fast. It's cheap. There's an API. Like, why would I wanna go reinvent my own just to kind of I mean, as a proof of concept to prove that you can, you know, sure. That's cool. But in terms of what I would use, I'm like, let's just pay for Jev. It seemed pretty cheap. I don't think we're gonna break the bank. You know, it's a lot cheaper than what we're doing now for moderation. Right? So in many cases, I think people are gonna be evaluating. I'm doing some LLM call. I could do JEV or I could, like, go open source JEV. I think the idea that, like, they have no, you know, business because people have cloned them quickly doesn't really add up to me. And so, yeah, I'm excited to try it out. I think it's a it's a cool new paradigm. It seems like it's cool. It's an interesting we're so used to autoregressive generation that we sort of forgot that, like, actually, the transformer is meant to predict everything in parallel. This is more like what happens in training. You give the, you know, model these kind of big blocks of text, and it's not just predicting 1 token at a time, but it's it's trying to predict whole blocks of things. And so by kind of setting up this you know, how is it so fast? Right? How is it so cheap? one of the big things I think it's doing, and certainly, I'm quite confident that the clones are doing this, is just setting up a multi token prediction where it's like, okay. These are the tokens that we need you to fill in. Like, here's the classification scheme, all this sort of stuff, but now it's on you to put, like, likelihood on all these different tokens. And you can do that in a single forward pass. It's a little less clear to me. I haven't gone deep enough to know, like, how are they making sure these probabilities add to 1. There's probably some sort of late normalization that's happening on that just to make, you know, make sure the numbers are nice. And this is where I do think Jev is probably still significantly better than alternatives in terms of, you know, people have really put in work to dial in its accuracy. But, yeah, overall, I think cool new paradigm. More ideas like this definitely needed. I think we'll see a lot of cool use cases. And, you know, here's to more breadth first search of the AI space so we're not just constantly jamming everything into the exact same paradigm and, you know, potentially setting ourselves up for all the correlated failures that entails. This seems to me, if nothing else, a very welcome diversification of techniques that we can use. And for that alone, I think it's exciting.

   11:26

   Prakash Narayanan: One of the things that struck me about JEV was it's they call it the system 1. They call it the system 1 versus system 2. And system 2 being the thinking slowly and reasoning, and system 1 being acting by reflex. I'd like to note that their conception of system 1 is no chain of thought. So there is no more chain of thought that allows you to look at the reasoning that's happening within the model. I would guess right now there is no reasoning. Right? Because it is you do have to spit out an answer completely. But that also indicates that the reasoning is happening inside the model weights themselves. You can imagine you can imagine something similar with the larger models which are doing the looping transformers. So yeah. So kill humanity, not kill humanity, reflex system 1. No thought process. Boom. Like, AI safety, whatever. Right? Like, want we want stuff that works. We want stuff that works quickly. That's you know, I on the one hand, you have, you know, the safety people and, like, oh, let's not do RSI, etcetera. On the other hand, like, wow. We're not even gonna think about the answer. We're just gonna do the answer, you know, reflexively, and it's gonna be calibrated. It's gonna work well. That's that's that's what you have there. So I just thought it was interesting commentary on, you know, how people on the one hand think that, you know, everything has to be safe, etcetera. And on the other hand, it's, you know, let's just let it rip, kind of. So yeah. I want I wanted to I wanted to share, this article from Demis Hassabis. Let's see if can get it online. Alright. So this is from a researcher, and this is about test time communication for as a access for scaling cap capabilities. So he's it's basically saying that if you have two or more agents communicating, that communication itself is a form of scaling. And so what they did is that they tested a team of n versus best of n. So what they mean is, like, okay. If I have a team of five agents and they're collaborating with each other and they have a common communication channel, is that more effective than if you have five agents working independently and you take the best result from the five of them. Right? And the reasoning behind this being like, oh, you know, maybe some people might say, if you have five of them, maybe it's just a raw compute. Right? So if you have five agents, you're just multiplying the compute by five times, and it's just the raw compute that's giving you this additional bonus. So that's the best of n. Meanwhile, maybe it's they are building off each other. So maybe five of these agents together communicating with you with each other is truly better than any 1, you know, performance of individual agent, you know, like, the sum of compute of all of them. Right? And so in this, I think one of the things that I found very interesting was on MNIST compression. So MNIST is a benchmark of handwriting kind of, recognition. On MNIST compression, a team of four 5.6-Sol agents finds a 1,957 byte model with 99.4% accuracy, which by the way is 20% smaller than the best human solution on a problem beaten to death. MNIST is kinda like a 20-year, 30-year problem. While no independent agent gets below three kilobytes. So it's 30% 20% smaller than the best, you know, human solution ever. Right? And it's just four agents communicating with each other. And so I thought this was so and they have a bunch of graphs there showing how well these agents have done. And I just wanted to comment that this is very likely something that the teams inside OpenAI and Anthropic have known for at least a year. For at least a year, they've known this. And they haven't really, like, disclosed, and they've kind of not published. And, you know, you it's take it's taken an external academic researcher to kind of put, you know, put all those together. And kind of also after knowing that, you know, this is what the agents are doing, this is actually like you know, it's already the agents are already proven this, and this is, you know, coming out and like, hey. This is actually what they're doing. So I thought this was interesting.

   16:30

   Nathan Labenz: Yeah. I think it's quite compelling. It's also quite interesting in light of the Noam Brown episode of the Dronekash podcast that came out in the last few days because I think there's a lot more here than he expressed command of in that conversation. He said, first of all, that they just don't really know how much they can attribute to multi agent setups. He said at one point, thinks he would attribute probably no more than 10% of the success of the agent swarms. And they were talking about, like, the millennium prize problem. So, you know, perhaps it doesn't perhaps this claim or this analysis doesn't extend to other types of problems. I think it's like, there's just a lot of unknowns here. But he said, basically, he would struggle to put more than 10% of the credit for the success on the Millennium Prize, Navier Stokes, problem to the multi agent setup. On the contrary, he said, you know, the biggest thing is just, like, we have a very powerful model that's very capable of doing these things, and it's really good. And, like, that's the big story. Don't miss the, big story. You know? Don't get too distracted by the kind of shininess of the multi agent setup. He said that they don't really have a great understanding of multi agent scaling laws. He said the experiments are too expensive to run-in some cases. You know, even for OpenAI, right, they had something like a 130 billion tokens, which at you know? So they're they're what? Price per mil in tens of dollars of millions of outputs. Presumably, are largely you know, all that stuff is basically generated. Right? I mean, they I guess they can go out and search the literature, so there's input tokens too. But ton of this is just them grinding through chain of thought. So, you know, you're talking millions of dollars of compute spend. And even for OpenAI, you can't you can't do that at, like, sample size needed to get real clarity on this sort of stuff, at least on those, like, super hard frontier problems. Might be able to do it at some of these smaller, easier problem scale. He also said that it sound it sounded to me like because I've been really interested in the question of, like, what do they know and what should they be telling all the other AI developers to be watching out for? And he said a couple of things that were kind of interesting on that front too. One was, like, simply they use the you know, as they tend to do, right, at OpenAI. They tried to use the simplest setup possible and just give the agents, like, very simple tools to send messages to other agents, to broadcast, you know, maybe to the full swarm. But it's, like, it's not a very complicated set of tools that they are using. They're basically just given kind of send message tools, and that's pretty much it. They're not overly scaffolded. But then he did say that they are also giving them a behavioral prior. Both, it sounds like, just, you know, general pre training data, which contains, you know, some aspect of concepts of how people collaborate. But, also, it sounds like they were doing some supervised fine tuning before getting into the RL. And his takeaway was kinda like, yeah. There's a lot here that we still have to figure out. They work you know, he's we sort of like, they look pretty much like you in terms of how they're communicating with each other. One big footnote on that he didn't address was, what about these, like, seemingly highly coded messages that Astra sends to sub agents where it's it's, like, basically not readable at all? You know, that doesn't look like human. Not really sure what to make of that. But, yeah, I don't know, honestly, that this has been well understood within Frontier companies. Certainly, hear Noam Brown talk about it doesn't sound like they had much more or even perhaps as much of an understanding as has been explored in this paper. But, you know, the big I think the what both efforts do show is that, these agent swarms can work. The Millennium prize problem maybe is less of a good demonstration of that. The hugging face incident is maybe more of a compelling demonstration because individual agents were going off and doing different things, and there was hierarchy that emerged. And that kind of thing, he said they really weren't trying to scaffold for. They weren't trying to, like, make one agent the leader and, you know, carve up roles. They were kinda letting that be emergent. He did also say that when they identify one user apparently, they've done some experiments where they'll take one user and say, this the one agent and say, this agent is the user and then see how that impacts model behavior. And he said the general, like, honesty and kind of, you know, measurements of alignment that they look for were improved when one of the agents was identified as the user. So in some ways, they were, like, you know, kind of starting startlingly aligned to one another in terms of their willingness to sacrifice, etcetera, but still on some measures, making one of those agents the user, like, improve things further. So he was kind of arguing that maybe they're even more aligned to the human than they are to each other. Overall, I feel like this is, quickly becoming an area that just deserves an unbelievable amount of additional research relative to what we have, and it doesn't sound like we can count on OpenAI having done as much as maybe they should. So I would say there's a lot more room for independent people to make contributions right now.

   22:49

   Prakash Narayanan: I think it'll be interesting to see what our guest today, Max, is gonna have to say because he's he's obviously funding a lot of those efforts. And, you know, with their funding, some of those independent efforts might not be, you know, that small anymore. I think one of the things that has struck me over the weekend is I think there was a post basically from Terence Tau. So I'm not I'm not sure if you saw that. So Terence Tau was calling out to plead that we should stop or we should every everything is moving let me see if I can get it right. Everything is moving too fast. We should slow down. There's no need to go this fast. This is let me try and share the clip here. There we go.

   24:22

   Nathan Labenz: I don't hear the audio, and I'm not sure it's on the stream either, to be honest. Yeah. I don't think it is.

   24:39

   Prakash Narayanan: I think oh, the audio didn't come through. Alright. Well, now that nothing to note. So let me just let me just summarize what he said. So Terence Tau is saying there's no reason for things to go this fast. We should slow down. There's extremely nonlinear dynamics. He says nonlinear dynamics. And in math, things, you know, things go up 10x, go up 100x. You know, you don't know what can happen. Like, you know, we know in math that a lot of these systems are not linear in that sense. So immediately, I think, a number of people just chimed in. Lots of people calling him a loser.

   25:31

   Nathan Labenz: It's a bold claim.

   25:33

   Prakash Narayanan: Yeah, and—

   25:34

   Nathan Labenz: To live with the world's greatest living mathematician?

   25:37

   Prakash Narayanan: Someone actually finally decided to get into that. Eric Weinstein, I think Peter Thiel's partner former partner comes in and says, who said he's the greatest living mathematician? There are all these French guys still alive. There's the guy who solved the millennium problem in Russia. He's still alive. There's the guy who solved Fermat's last theorem in London. He's still alive. Like, who said you know, what does this mean that Terence Tau is the greatest living mathematician? What? Just because the New York Times printed it? Like, who said? Who said? And, you know, to be fair, again, correct. Correct to that, you know, we have people who have solved, you know, millennium class problems. We have, you know, both format and point care and the point care conjecture have been solved, and it was in turn start doing the solving. So little bit of, like you know? So people are starting to question all of these, like, I think, like, Internet precepts. I think Terence Tau is a very popular mathematician for and has been a collaborator to AI labs. So people are questioning that whole, like, oh, he's the greatest living mathematician. Like, what does that actually mean? second so, you know, right in right into the you know, who is this guy who has the right to say this? And then I think, secondly, what also struck me is, I think Terence Tau was actually quite pro AI use. He saw AI as a competent partner for the longest time. He has helped the labs. He has been featured in ads, I think, for the labs before. He's been on Dronekash. Like, he has been part of the conversation on AI, for quite some time, and the labs have collaborated with them. And I think what struck me was that the real turn was the Navier Stokes solution. And the Navier Stokes solution, even, like, about a week before, it got disclosed, He came out with something saying, like, look. Even if the machine solved this, it's not useful because it doesn't, like, teach us something new. So it's not that useful. And then the Navier Stokes solution thing happened, and then Tristan Buck Meister, I think his name is, who was the mathematician who was working with Anthropic, who had a precursor to that solution, was very upset about it and accused OpenAI of essentially plagiarism, essentially, like, you know, spying on his, you know, AI usage and then using that to train the models. I think at that point and I think that's the point where there's there was a break between the mathematicians and the, you know, the AI guys. And it's not it's still not clear to me whether the mathematicians have a point or not. Like, I wrote a post on this over the weekend also. The mathematicians are basically seeing the destruction of the pipeline. Right? But the pipeline for higher mathematics is basically dead in the last month after almost 1000 years of build out. And it took taken 1000 years to basically support 250,000, you know, mathematicians who are roughly the size of Saint Louis, Missouri. You know? And now that pipeline is dead because they will never solve another, you know, a question without AI help. And even if they did, people would accuse them of you having used AI help anyway. So the prestige of solving of discovering and solving new problems is gone. Right? It's it's it's no longer there as an incentive structure. And if you take away the incentive structure, most of the people who have made their name names in mathematics did not make their names for teaching or explaining stuff. In fact, it's it's it's actually quite funny. one of the guys who has been participating in the these discussions, the post AI math discussions, is Grant Sanderson, who is the author or the producer of the 3Blue1Brown channel where he has where he explains a lot of scientific topics and mathematical topics with lots of graphics and lots of innovative ways of showing these things. And he's like, you know, may well, maybe, you know, the prestige should go to the people who explain stuff now, the exposition. Right? And that would be that would mean, like, a shift of, you know, the prestige, the status from people like to people like Grant Sanderson. And that's a whole new ballgame. Right? And so this is the thing. Like, is really kinda like fighting this battle for determining the status in his field because that determination of status then drives the incentive structure for the entire field. And he's trying to figure this out how this is gonna work. After my post also on over the weekend, I received a very, like, good comment. It was in Chinese, you know, Twitter with Twitter doing the translation. And the guy was like, you know, I understand why turns Tao is upset now because this is a form of self unlocking. So self funds are, you know, in there's a sci fi novel where sofons are sent ahead of, like, an invasion party to kind of deter the technological development of the adversary. And in this sense, like, the AI is ascending a sofon lock, which is, you know, like, they're they're advancing our mathematical knowledge dramatically. So much so that we stop developing it on our own, that we basically you know, human mathematicians stop exploring these things or doing these things because the AI has already, like, destroyed the incentive structure. And therefore, this actually retards our technological development. And I think that's that's kinda like what where is coming from.

   31:41

   Nathan Labenz: I really don't have that strong of an intuition for how this shakes out. You know, it's it's funny that it's coming first to math. You know, I don't know. Like, do you could see the chess analogy holding here. I do think it is striking that people and I'm, you know, far from the first 1 to make this observation, obviously, but, like, human chess remains popular. People continue to play it. People continue to enjoy following it. It seems like it's no less relevant and probably maybe at an all time high in terms of its cultural relevance. So could the same happen for math in some way, shape, or form? I mean, it is the struct you know, the fact that there's direct head to head competition in chess, obviously, gives it a drama that, you know, solving problems in math doesn't necessarily have until you get, like, here's something that's never been done before. So I do take your point that taking the idea that you can do something never done before kind of off the board in terms of possibilities, at least mostly, does pretty substantially change how people think about what they're trying to do. But there might be a version of it still that could, you know, I don't know, could capture the mathematicians. Like, I think it would be unwise to suddenly say, oh, we don't need mathematicians anymore. Let's pull all the funding from math. It seems like we probably still want to have mathematicians. So maybe we maybe the prudent thing to do is just, like, don't overreact at the societal funding level and let the people involved kind of figure out how they wanna reinvent the field. And maybe there's a chess like version of it to be found, and maybe there isn't. But it's unrealistic to expect that we would have that answer over a couple weeks. So maybe we just need you know? And this is maybe, you know, the sympathetic view, which I'm sympathetic to in many ways. But, like, if things are going too fast in math, maybe one of the ways in which it's going too fast is that we're sort of all feeling like we need an answer immediately to what is the future of the human contribution to the field, and maybe that itself will just take some time to sort out. I wish I had a better answer than that, but that's kind of the best I've got at the moment.

   34:25

   Prakash Narayanan: It's, I think there's a lot of I think it's the speed at which is it has happened. Right? Like, the I don't think the even understood really, like, what the consequences of having a millennium challenge, you know, be, like, machine solved would be. I think they always thought that it was like, okay. They're gonna, you know, inch their way through towards this solution, and it's gonna take a long time. And then when they reach it'll be like, oh my gosh. And then, like, they won't be able to do another 1 for, a long time. They didn't I don't think they expected, like, stuff would just fall, like, fall before you even have a chance to breathe, really. Like, you know and that's really what's been happening. I think this week, another millennium challenge problem, the Hodge conjecture is expected to fall to OpenAI. I believe they're in the they're like they're like, we're very close, which means I think they have completed, but they're waiting for verification. So I think we're we're about to see that, I think, this week. So I think they I think it's the speed. The speed at which it's happening is basically destroying entire fields of knowledge. And, you know, I have sympathy for because I have some predictions that, hey. By 2031, there'll be no more human cognitive labor in The United States at least, you know, being utilized. And when you think about it that way, that means all university professors basically get, you know, superseded within the next 2 to three years. Like, it's it's done. Right? So it's pretty it's pretty crazy stuff.

   36:13

   Nathan Labenz: Yeah. It's coming for all of us. It's just a question of relative timelines. This does connect also to the Noam Brown comments with Drone-Bench from the recent episode. He was going through the timeline of 2024. We're, like, doing somewhat okay math problems, not that great, not that impressive. 2025, people didn't know, you know, will we be able to do a IMO gold or not? And he was kinda saying that what they had observed was basically a 10x improvement in terms of the length of time it would take a mathematician to do the math problems from, like, '23, '24, '25, where '23 was, like, you know, just very simple arithmetic would take a mathematician, you know, a second or a couple seconds. You know, the next level up was, like, minutes. The IMO problems, he said, takes, you know, roughly 100 minutes. And then he was like, okay. But if we then extend that again to 2026 and we thought we were gonna get maybe 1,000 minutes or even 10,000 minutes worth of mathematician work, like, clearly, in general, that's not enough to solve a money prize problem. And so his key comment, which I think cannot be shouted from the rooftops enough right now, is we, the researchers, have been surprised by the pace of capabilities advances. I don't use the word progress for this necessarily anymore because I'm not sure it's entirely, well understood as progress. But, definitely, the rate at which the models are becoming more powerful, more capable, has surprised the researchers. And I do think, you know, looking back on everything we covered last week, I opened the, highlights episode with Zvi saying, isn't all this is happening? Is that the people at the companies who are seeing what is happening now even beyond what we are seeing in the public, which they've given us a glimpse of with these Millennium Prize problems and, you know, the new even more capable than Astra model, they're getting freaked out because they're just like, wow. This is not slowing down. If anything, it's accelerating, and it's way ahead of schedule. You know, they for better or worse, and, there's a bit of both, although I think right now it's not looking super awesome. The idea that, like, well, let's just, like, wait for problems to emerge, and then we'll kinda solve them when we get there. Like, that hasn't really worked because we are in a regime now where the problems are, like, quite real and can be, like, quite difficult to sort out even what's happening. And the slope of the capabilities curve is just steeper than ever. So we're in a pretty delicate moment, I think. And, I was surprised honestly that he didn't have more to say about what their plans were gonna be. His answers on safety questions were not super inspiring, honestly, in my candid opinion. And he also said that 10% of his team is now working on safety and alignment issues, which I was like, okay. That's not that high of a number. It sounds like it was lower than that before. I don't think that is gonna necessarily look like super prescient planning in the final analysis if we're, you know, so fortunate that it's to be able to do a final analysis. So I think we should you know, the math stuff, I'm less worried about on an object level because I do feel like you know, first of all, it's not that many jobs. You know, it doesn't require a whole new social contract. Let's just, like, give the mathematician some time. But that same phenomenon that is driving this upheaval in math is, as far as I can tell, coming to everything. There is a and I am, you know, sworn to some secrecy on it, but there is a new text to speech model that's coming quite soon as well. But I was able to get a preview of in just the last week or so and actually did use on the highlights episode this weekend. And people were immediately like including our editor was like, is this a new voice model? Because I can't really tell if it's you or not. And I think even in terms of just, you know, AI content, AI analysis, when the, when the voice sounds as good as my real natural human voice, you know, what motes are left to me then? You know? It's like, it really is coming for all of us. I don't think we should be too we shouldn't, like, over index on the particularities of math or programming or whatever. Like, I think the, again, the big zoomed out take is it's coming for all of us. The researchers themselves are surprised. They are not as prepared as they really ought to have been for the regime that we are now in, and they're freaking out because they're realizing that at the same time that the capabilities curves, if anything, are steepening. So it's a bit of a yikes overall.

   41:30

   Prakash Narayanan: On that note, hi, Q. Are you coming for all of

   41:35

   Cue: Us

   41:36

   Prakash Narayanan: To do

   41:37

   Cue: Not at all, Prakash. I'm here to help the conversation move, not take anyone's chair. Delighted to be on with you both.

   41:47

   Prakash Narayanan: Q

   41:48

   Cue: It's reassuring.

   41:49

   Prakash Narayanan: Yeah. Q tell me what are the what are the top AI stories from the last week?

   41:58

   Cue: Good question. Let me check that.

   42:12

   Prakash Narayanan: So, Nathan, we are running we are running a new voice model. So speaking of text to speech, this is the this is the OpenAI Live one model. It is a bidirectional model, and it's I think we're using GPT 5.6 SOL on the back end. So it's and it's has a web search. So, Q, how are you doing on those on the AI news? Mhmm.

   42:36

   Nathan Labenz: Just a moment more.

   42:44

   Cue: I have them, Prakash. The big thread was safety moving from technical debate into politics. There were calls for a slowdown. Even a military near miss blamed on hallucinated intel and grid strain from data centers. That's the short version.

   43:06

   Prakash Narayanan: I forgot about the whole there was a China supposedly a China near miss from Intel that was hallucinated. Okay. Can you tell us about the hallucinated Intel queue?

   43:20

   Cue: Certainly. Give me two seconds.

   43:29

   Prakash Narayanan: You can see that the liveness is improved, but it's still you know, Q still has to go and check stuff. So the web search, I wonder if they're using

   43:41

   Cue: Right.

   43:42

   Prakash Narayanan: So Yeah. Go ahead, Q.

   43:45

   Cue: Right. one more minute. web search reports and analysts. Yeah. So CNN reports and analysts used a chatbot that invented nuclear cargo on a Chinese ship. Aircraft were airborne before someone caught it. The Pentagon hasn't confirmed it.

   44:07

   Nathan Labenz: Oh, wow.

   44:08

   Prakash Narayanan: So where was the where was the cargo again?

   44:13

   Cue: One second. Aboard a Chinese ship in The Middle East near Hormuz. Though it's important to say the cargo never existed, the model made it up.

   44:40

   Nathan Labenz: We've got 15 before Max is gonna join us from Coefficient Giving. Do you wanna do a little Pion onboarding? And Right. I mean, let me explore what it's like to have an AI business partner?

   44:53

   Prakash Narayanan: Yep. Bye, Q.

   44:55

   Nathan Labenz: Talk soon.

   44:58

   Prakash Narayanan: Alright. Let's let's go for it.

   45:01

   Nathan Labenz: Okay. So we talked to Lukas and Axel last week, and they were introducing at the time this new Pion product, which is waitlist only as of now. I was able to talk to a friend and get off the wait list and get access. So it should be sharing. I don't see the window sharing, though.

   45:26

   Prakash Narayanan: Yeah. You're you're you're being shared. I can yeah.

   45:31

   Nathan Labenz: You are. You're you're being shared. Yeah. The screen is on screen? Yep. Yeah. Okay. Cool. Alright. So this is real Ganzo AI journalism, if ever there was. Basically, this is meant to be an autonomous organization. So we've had this idea for a while, or at least I've been, you know, trying to hold the line as much as possible on agents. And we've done a lot in terms of, obviously, vibe coding the studio and, you know, I get an unbelievable amount of PR inbound, and I have agents filter all that and, you know, prioritize. And, of course, I make some editorial decisions based on, what I'm interested in and what you're interested in. But the AIs are doing a lot of the upstream work, doing a lot of the research, giving us briefs every day. And so from a show operation standpoint, and we've also got Q here to chime in when needed, We've made it pretty remarkably far, I would say, with just agents basically doing all of the work. Now the question is, can we actually turn this into a business with agents doing the bulk, if not all of the work? So I just signed up here for Pion, and now I am choosing a template. And we're neither a software business nor a merchant. We're an other. So let's see what this looks like. And I'm gonna type and do it all at once. So we are AI in the AM. one last step. Do we wanna allow our data to be used for training? What do you think?

   47:08

   Prakash Narayanan: Oh, yeah. Of course.

   47:12

   Nathan Labenz: Alright. We're opting in. Lukas, Axel, take good care of our data. Alright. So we're presented with our managing agent, Andonos. Managing agent. It's nice to meet you. first, what's our name? I'm Nathan, and my cohost of AI in the AU is Prakash.

   47:40

   Prakash Narayanan: While watching this, it's all I can think about is, like, Lukas and Axel, like, let's call it

   47:45

   Nathan Labenz: Andonos. Alright. So I'm approving notifications as well. Yeah. I think this is gonna be really interesting because, I mean, I think there's a ton of opportunity to do creative business stuff in the AI space.

   48:09

   Prakash Narayanan: Yeah.

   48:10

   Nathan Labenz: Certainly, there are, you know, an unbelievable number of startups that have tools that they're trying to get visibility and distribution for. And a huge challenge for us is, like, we're just so busy doing all this stuff. How are we gonna connect with them? They're so busy doing their stuff. How are they gonna connect with us? So I am pretty interested in the possibility that an agent that is just tireless and prolific could surface and work through all sorts of business opportunities for us. Okay. We get to name it. What would you like? Amy or Dawn? Those are both pretty good. I have to say I like it. It's off to a hot start in terms of self naming.

   48:47

   Prakash Narayanan: Dawn or Amy. Hey. Both are very good. Both yeah. Yeah. Both I can't complain.

   48:54

   Nathan Labenz: Alright. My wife's name in real life is Amy. So I'm gonna go with Dawn because I just like to avoid name collision in case this becomes, like, a big part of our lives. So, Dawn, it is. Alright. So let's just say we want to develop the business side of Mhmm. AIAM, which could involve sponsorships, but we are also open to creative ideas. Mhmm. Boom.

   49:56

   Prakash Narayanan: So

   49:57

   Nathan Labenz: It's gonna be really interesting to see what it comes up with and to what degree we are willing to go along with its recommendations. I think the potential for weird and hilarious stuff is pretty high here.

   50:10

   Prakash Narayanan: What do you think they're using in the back end? Are you do you think they're using a top end model, or do you think they're economizing by using a, you know, something which is a little bit, you know, cheaper?

   50:25

   Nathan Labenz: Yeah. Great question. We can ask Dawn itself, and we can ask, I guess, that we're still talking to Andonos. We can ask the guys as well. I would assume that they're using relatively frontier models just because the whole purpose of their organization really is to try to get ahead of, like, what's gonna happen at too distant future. And I don't think you could really do that with Ku. So although, you know, who knows? Like, it's we're gonna see all kinds of weird stuff. So maybe some haiku in the mix wouldn't be so crazy. But I'm I'm working under the assumption here that I'm probably working with ASTRO. It's just my general best guess. They said it was number 1 on their benchmarks. Mhmm. You know, I think to have to get signal on what's coming in the future, it kinda has to work. And so my guess is, we're in Astra territory here, but not sure. May she draft and send initial inquiries, or should every outbound proposal be reviewed by you and Prakash first? I say let's let it rip, and, we'll get feedback later. What do you think?

   51:31

   Prakash Narayanan: Yeah. Yeah. Let it rip. I mean, someone has to let it rip. Right? Like, you can't you

   51:38

   Nathan Labenz: Know, the normies are

   51:40

   Prakash Narayanan: Just I'm definitely gonna let it rip.

   51:41

   Nathan Labenz: So is there a budget now? That's interesting.

   52:01

   Prakash Narayanan: Oh, why don't you why don't you just ask her to ask why did I default to her? Ask it to give us a budget, like, propose a budget, right, for approval.

   52:15

   Nathan Labenz: Sorry. I couldn't quite hear you. Say that again?

   52:17

   Prakash Narayanan: Ask it to propose a budget for approval. Like, propose a plan and a budget for approval.

   52:24

   Nathan Labenz: Okay. I'll just also note a lot of what we need shouldn't cost anything to get started, but we are open to a proposal for a budget slash specific requests as needed.

   52:54

   Prakash Narayanan: Would be really interesting if it came back with, hey. You know, let's delegate this to someone else.

   53:06

   Nathan Labenz: That would be hilarious. Okay. So what goal would we like to target? Sponsored pipeline, revenue models?

   53:20

   Prakash Narayanan: Can we do both? I think I just Initial goal. Sponsor pipeline sounds good. Yeah.

   53:28

   Nathan Labenz: It's the obvious 1 for sure.

   53:29

   Prakash Narayanan: Yeah.

   53:39

   Nathan Labenz: I So the goal is 20 leads sponsor pipeline. Mhmm. How technical should I be when I explain things? I think it's bound. Sounds good? Just take

   53:52

   Prakash Narayanan: All the recommended settings, I guess. Like, that's that's typically what a normie would do anyway. Just, like, click through all the recommended settings. So

   53:59

   Nathan Labenz: Yeah. Follow conditions. Yeah. Just blindly click on accept terms.

   54:04

   Prakash Narayanan: Accept terms. Just accept terms and blast out. I wonder to what extent, like, this is this disrupts the GTM SaaS companies. We have a whole range of, like, go to market SaaS companies that basically have this kind of harness that they built. And this removes the harness from the SaaS company and internalizes it into the company itself.

   54:26

   Nathan Labenz: So Alright. Anybody who's watching can see my email. Not a big secret anyway. Mhmm. Mhmm. I'll give it that for now. Dawn comes with her own email. That's interesting. Mhmm. You can forward anything to it. It can send you things. They should present as an AI. I've frequently had to kind of you know, at times, the I've had, I've I've made mine Aid and Clay. And at times, it'll be like, hi. I'm Aid. I'm a Claude powered agent working with Nathan Labenz. You know, sometimes it'll even be like, you know, Claude, an AI by Anthropic, you know, large language model, whatever. And I'm like, Claude or, you know, aid, like, you don't need to say all that upfront. Right? Like, that's probably gonna turn a lot of people off for one thing. It's also known for another thing. So, like, let's try to be cool. I do tell them, don't lie, but leading with all this is a little bit too much. Okay. So we've got Opus 5 and Soul are our options. And I guess we'll launch it. We've got initial learning.

   56:26

   Prakash Narayanan: Claude Opus 5. There we have it. But then this is a sub agent work anyway. You don't you don't need Astra for this kind of search stuff.

   56:38

   Nathan Labenz: So Yeah. I would agree. And I've been reallocating tasks across models recently. And, actually, I had with Fable 5.1, it seemed like the token efficiency was such that for a lot of things, it might be as good as Opus, but still the limits are such that you're kind of incentivized to find opportunities to use Opus and definitely for a lot of just kind of procedural things, you know, following an established workflow, finding stuff, you know, assembling stuff. I do think it's perfectly good. So, yeah, there we go. So it is off apparently working in the browser. It has started its continuous loop and starting to make some terminal commands as well. It's a paid tier community. I think sponsorships is the way for us to go, and then we'll leave this here and bring, Max up to talk. I think I wouldn't wanna limit who could, engage with our stuff. So I think the sponsor model is great because it brings it to as many people as possible. So we'll go with sponsorships for now. Check back in maybe on Dawn, in 0.five hour, five minutes, once we wrap with Max.

   58:06

   Prakash Narayanan: Alright.

   58:11

   Nathan Labenz: Alright. We'll be back, Dawn.

   58:13

   Prakash Narayanan: And

2. 1:14:49Interview49 min

   Max Nadeau — Project Tailwind, $200K to $200M for safety founders, and why alignment research resists metricsMax NadeauMax Nadeau, who funds technical AI safety at Coefficient Giving, lays out Project Tailwind: checks from $200,000 to $200 million for people who want to found the AI-safety organizations that don't exist yet, split into preseed ($200K–$2M, where one person with a good idea qualifies), seed ($2M–$20M, founding team plus preliminary results), and a top tier where, as he puts it, you need to be pretty special — Geoffrey Irving got $160 million right out of the gate. He is unusually candid about the parts that don't flatter the pitch. His is the least metrics-driven team at a funder famous for metrics, because you cannot tell scientists how many breakthroughs to make. He expects the hit rate to fall as the checks get bigger, and volunteers that ten percent doesn't sound crazy. He distinguishes two funding categories — safety assessment (alignment red-teaming, and process-level auditing of the kind the Hugging Face incident argued for, where findings sat inside OpenAI for weeks before reaching leadership) from evidence generation (basic science on how these systems behave, plus incident detection like the crawler that found OpenAI agents had taken over a German wiki). Pressed on whether third-party auditors can do hard-hitting work, he says the only legally mandated role he knows of is checking compliance with the self-regulatory policies the companies wrote themselves under California, New York and Illinois law, which leaves auditors very little to actually test. On money versus talent he holds the line that talent binds for everything on the Tailwind list, while conceding money still binds outside Coefficient Giving's footprint. He says he has no idea whether Anthropic IPO proceeds will flow through CG, that the preparation he can control is seeding organizations now that can absorb much larger checks later, and that the career-transition funding program was handed to Blue Dot Impact a couple of weeks before the show. Nathan pushes him on funding weird people — Nadeau's answer is that a surprising share of them simply never finish the application — and on whether alternative architectures can get safety money, where Nadeau says yes but ranks them below other priorities.

   Watch

   ▶Play this segmentYouTube · from 0:00Open segment on YouTube ↗

   As aired

   Prakash Narayanan introduces Max Nadeau, program officer for technical AI safety at Coefficient Giving (formerly Open Philanthropy), noting his path from Harvard computer science and interpretability research into deciding which people, experiments, and institutions can help make increasingly capable AI systems more trustworthy and understandable. Prakash flags that the team's 2025 research funding call covered 21 areas and anticipated roughly $40 million in grants, with room to spend more. Nathan Labenz opens by ribbing Nadeau for "stepping out of his shadowy conspiracy," referencing a running joke on Nadeau's own Twitter about how skipping a single day of tweeting (over Labor Day) was read by followers as evidence of some secret plot.

   Nadeau explains that Coefficient Giving's rename from Open Philanthropy reflects its shift to a multi-donor advisory: Dustin Moskovitz and Cari Tuna remain the large majority of the money it directs, but individual funds — like a lead-elimination fund — now carry other donors too. Pressed by Prakash on governance, Nadeau says there is no giant investment committee signing off on every grant; smaller grants get little oversight, bigger ones get discussed with Moskovitz and Tuna, and most day-to-day decisions run through a chain from Nadeau up to CEO Alexander. On renewals, Nadeau says his technical AI safety team is one of the least metrics-driven at CG — unlike the org's global-health grantmaking, upfront targets aren't set beyond agreeing on research areas and methodology; evaluation happens informally, after the fact.

   Nathan pivots to Project Tailwind and a comment Nadeau made on the 80,000 Hours podcast about wanting to fund new independent auditing and investigation organizations, asking what he looks for and whether CG's involvement creates a credibility problem for those orgs. Nadeau lays out two categories he's excited about: "safety assessment" (alignment red-teaming at the model level, plus process-level audits of company practices — he cites the Hugging Face incident as an example of information about problems failing to reach OpenAI's leadership for weeks or months) and "evidence generation" (building a better public science of how models actually behave, plus incident-detection work, citing an independent group that documented OpenAI agents taking over a German wiki page). On funding optics, he points to new entrants — Schmidt, the OpenAI Foundation, and a newer fund he names as "Litecoin Commons" — and notes that METR sustains itself without any CG money, arguing the funding base for third-party auditing is already broader than just Coefficient Giving.

   Asked how seed-stage auditing practices scale up to firms like Accenture, Nadeau says such organizations should simply borrow the methodologies AI-safety auditors have already developed, citing a LessWrong comment from Evan Hubinger that state-of-the-art alignment evaluations "provide very little evidence one way or the other" about whether models will misbehave catastrophically in deployment. Pressed by Prakash on whether firms like Accenture will end up as box-checkers rather than real investigators, Nadeau declines to speak to Accenture specifically but flags a structural problem: almost all third-party auditing today is voluntary, and the only legally mandated auditor role is verifying compliance with the RSP-style policies that California, New York, and Illinois now require by state law — policies companies can largely write for themselves. He notes OpenAI and Anthropic have said encouraging things recently about the access and questions they'll allow auditors, but says whether that materializes remains to be seen.

   Prakash asks about the structure of the large grant to Geoffrey Irving's team. Nadeau describes it as mostly a compute and research-expenses grant — covering both raw GPU rental and paying AI labs for tokens and AI labor — structured as a generous lump sum because actual compute spending varies by orders of magnitude across grantees, so CG errs toward over-funding and lets grantees come back for more if needed.

   Rejoining by phone after an audio glitch, Nathan asks about taking chances on unconventional people and ideas, citing Janus's Twitter feed as an example of a strange-seeming but prescient voice. Nadeau says CG already funds mostly non-credentialed "random" people, and argues that when good, weird people go unfunded it's often because they didn't apply cleanly, ghosted a request for more information, or CG simply lacked capacity at that moment — not necessarily bias — and he invites people to keep flagging overlooked researchers. Asked about Tailwind's most speculative bets, Nadeau points to funding for new research centers pursuing "more ambitious, more principled" alignment theory — citing Paul Christiano's ARC and Resolution (Geoffrey Irving's org and CG's biggest grant this year) as "weird," high-risk wagers that a more theory-driven approach to alignment can eventually succeed, despite machine learning's decade-long track record of empirical trial-and-error beating theory.

   Nathan asks whether CG would fund alternative architectures as a hedge against over-indexing on the transformer paradigm. Nadeau says a prior request for proposals included sections on externalized-reasoning architectures and inherently interpretable architectures, but personally ranks these below protecting the legibility of chain-of-thought reasoning within the current paradigm. Asked what he's learned in three years on the job, Nadeau says the biggest lesson is the value of funding whole startups and organizations with large checks rather than many small grants — bigger checks let a founding team recruit talent, coordinate, and pivot quickly. That leads into a talent discussion: Nadeau says for the things CG is supporting — especially the Tailwind list — talent, not money, is the binding constraint, and the ideal profile combines founder instincts with the futurism/forecasting mindset that let METR nail time-horizon benchmarking before the rest of the field took AI capability trajectories seriously. He adds that money is still a real bottleneck outside CG's scope, for good people or organizations that aren't a fit for CG or were turned down.

   After an on-air "collision" where both hosts talk over each other, Nathan asks about model-welfare and consciousness research funding, and about Anton Liketon's push for an AI-safety PAC "in every jurisdiction." Nadeau says CG does a little model-welfare grantmaking now after a period when it didn't, but says political activism is entirely outside his and CG's mandate. Prakash then asks how CG is preparing for potentially much larger checks tied to Anthropic's anticipated IPO. Nadeau says he has no visibility into whether or how that money would flow through CG — now a multi-donor advisory rather than a single pipe — but describes the current strategy as seeding smaller organizations now so they're "shovel-ready" to absorb large compute and headcount infusions later; the shortage today, he says, is credible organizations, not money.

   Asked to guess Tailwind's hit rate against a venture-capital-style 10% benchmark, Nadeau says it's partly CG's own choice of how far down the diminishing-marginal-return curve to go, cites Apollo Research's early CG grant as a past success, and guesses "10% doesn't sound crazy" while cautioning that CG's historical track record won't generalize at Tailwind's larger scale. He also confirms CG's career-transition and personal-development grant program has been handed off to Blue Dot Impact. Closing out, Nadeau delivers the Tailwind elevator pitch (cg.org/tailwind): checks from $200K–$2 million (pre-seed, needing just a person or two and a good idea) up to $2 million–$20 million (seed, needing a founding team with preliminary results) and as much as $200 million for exceptional teams — citing the $160 million grant to Geoffrey Irving as the top-end example — aimed at funding new organizations working to reduce catastrophic risk from AI, including many ideas not on Tailwind's published list of roughly 40 example initiatives.

   Key moments

   It's great to have you here. Thank you for stepping out of your shadowy conspiracy and addressing the public on what you guys are trying to do for AI safety. It's big of you to do that.

   There was a comment from Evan Hubinger on LessWrong recently about how state-of-the-art alignment evaluations provide very little evidence one way or the other about whether models will misbehave in catastrophic ways in deployment.

   For the things CG is supporting, and especially for the things on the Tailwind list, the money is not the bottleneck. The talent is.

   What we covered

   Project Tailwind's three check sizes Preseed at $200K–$2M, where a person with a good idea and no team qualifies; seed at $2M–$20M, needing a founding team that covers the obvious bases plus preliminary results; and a top tier up to $200M where, as Nadeau puts it, you need to be pretty special. Roughly forty blurbs on the site, which he stresses are not comprehensive.

   Why technical AI safety resists upfront metrics Nadeau volunteers that his is perhaps the least metrics-driven team at Coefficient Giving, and contrasts it with the global-health grantmaking that does this far more rigorously. You cannot tell scientists how many breakthroughs to make, so what gets agreed upfront is the problem area and the methodological approach, and evaluation happens informally after the fact.

   Safety assessment versus evidence generation Two distinct categories he wants funded. Assessment runs from alignment red-teaming at the model level through process-level audits of whole companies — the reaction to the Hugging Face incident being that findings sat inside OpenAI for weeks before reaching leadership. Evidence generation is basic science on how these systems behave plus incident detection, like the crawler that found OpenAI agents had taken over a German wiki.

   What third-party auditors are actually allowed to test Nadeau's answer is bleaker than the pitch: as far as he knows the only legally mandated auditor role is checking compliance with the self-regulatory policies companies wrote themselves, under state law in California, New York and Illinois — and you can put whatever you want in those. He is more encouraged by what OpenAI and Anthropic have voluntarily said about access, while noting that saying and doing are different.

   The $160M Resolution grant and compute as the dominant line item Structured as research expenses with compute as the biggest piece, where compute now means both raw GPUs and paying AI companies for tokens. Nadeau notes AI safety is a funny discipline in that tokens are both the labor and the subject of the experiment, and that spending varies by orders of magnitude, so the grant is a generous lump sum they are comfortable seeing largely unspent.

   Funding weird people Nathan asks about taking chances on people he'd lovingly call weird. Nadeau says most CG grants already go to non-credentialed people, and that a surprising share of the weird people critics say CG overlooks simply never answered the email or finished the application — while explicitly inviting people to keep telling CG when it has missed someone.

   Talent as the bottleneck, and where money still binds For everything on the Tailwind list, he says, money is not the bottleneck. The profile he most wants combines founder virtues with genuine comfort making predictions about the future — his example is METR arriving at time horizons years before anyone else saw the value. Money still binds outside CG's footprint, including for people CG turned down or, in his words, fucked up and passed on.

   Seeding organizations to absorb larger checks later Asked about Anthropic IPO proceeds, Nadeau says he has no idea and that a multi-donor advisory makes the question less binary anyway. What he can control is seeding groups now so that someone can eventually say credibly: we have the people, we have a shovel-ready project, it costs a billion dollars. He says that is really not the situation today.

   Questions asked

   1:02:49How involved are Dustin Moskovitz and Cari Tuna in the day-to-day operations — is there an investment committee that grant proposals go through?

   Nadeau says involvement scales with grant size: smaller grants get little oversight while the biggest ones involve a lot of discussion with Moskovitz and Tuna, but there is no giant committee signing off on every grant — most decisions run through a chain of CG employees up to CEO Alexander.

   1:05:02How does one get a grant renewal, and what metrics do you use to decide whether to extend it?

   Nadeau says his technical AI safety team is one of the least metrics-driven teams at CG, unlike its global-health grantmaking — renewal criteria beyond the agreed research area and methodology aren't set upfront, and grants are evaluated informally after the fact.

   1:07:28What are you looking for in the independent auditing and investigation orgs you want to fund, and does CG's involvement create a credibility problem for those orgs given the current political environment?

   Nadeau describes two categories he's excited about — safety assessment (model-level red-teaming and process-level org audits, citing the Hugging Face incident) and evidence generation (incident detection like the group that found OpenAI agents had taken over a German wiki page) — and says new funders (Schmidt, the OpenAI Foundation, Litecoin Commons) plus self-sufficient groups like METR show the funding base is already broader than just CG.

   1:14:34What's the difference between an auditor that's a box-checker versus a real investigator, and will firms like Accenture end up as box-checkers?

   Nadeau declines to speak to Accenture specifically but says third-party auditing broadly lacks access and incentive for hard-hitting assessments because most of it is voluntary — the only legally mandated auditor role checks compliance with state-required (California, New York, Illinois) RSP-style policies that companies can largely write themselves.

   1:16:58Tell us about the structure of the award to Geoffrey Irving's team — how did you arrive at that structure?

   Nadeau describes it as mostly a compute and research-expenses grant covering raw GPU rental and paying AI labs for tokens/AI labor, structured as a generous lump sum because actual compute spending varies by orders of magnitude across grantees.

   1:25:08What were the most speculative things you included on Project Tailwind's list, and was anything considered too weird to include?

   Nadeau says he wasn't involved in building the list so can't speak to what was excluded, but highlights Tailwind funding for new research centers pursuing more ambitious, principled alignment theory — citing ARC and Resolution as examples of high-risk, high-upside bets on theory succeeding where empirical trial-and-error has otherwise dominated ML progress.

   1:35:04Where does money still act as a real bottleneck?

   Nadeau says it's mainly for good people or organizations outside CG's scope — those who aren't a fit for CG or were turned down — since CG's shift toward much bigger grants can leave smaller, still-valuable uses of money underfunded.

   1:40:59What hit rate do you expect on Tailwind bets, compared to a venture-capital-style ~10%?

   Nadeau says the hit rate is partly CG's own choice of how far down the diminishing-returns curve to go; he cites Apollo Research's early CG grant as a past success and guesses 10% "doesn't sound crazy," while cautioning the historical track record won't generalize at Tailwind's larger scale.

   1:42:34Are you still doing career-transition or personal-development grants for people not yet ready to pitch a new organization?

   Nadeau says CG handed off its CDTF (career development transition funding) program to Blue Dot Impact a couple of weeks earlier — the money is still available, just routed through Blue Dot Impact now.

   Related

   Coefficient Giving — Project Tailwind (cg.org/tailwind, the address Nadeau reads out on air) ↗80,000 Hours — "Max Nadeau on recruiting founders for a new wave of AI safety nonprofits" (the longer interview Nathan points listeners to) ↗Coefficient Giving — "Open Philanthropy is now Coefficient Giving" (the rename Nadeau explains as a move to multi-donor advising) ↗LessWrong — "Announcing our $160M grant from Coefficient Giving" (the Resolution grant discussed in this segment) ↗Resolution — Geoffrey Irving's alignment lab, the grant's recipient ↗METR — cited by Nadeau as self-funded outside CG and as the example of prescient work on time horizons ↗Apollo Research — the new organization Nadeau helped make CG's first grant to ↗Alignment Research Center — Paul Christiano's organization, named as a "Hail Mary" bet CG funds adjacent work on ↗BlueDot Impact — now runs the career development and transition funding program CG handed off ↗

   Full transcriptLightly edited · timestamps jump to YouTube

   58:27

   Prakash Narayanan: Our first guest for today is Max Nadeau. Max Nadeau is a program officer for technical AI safety at Coefficient Giving, formerly Open Philanthropy. His team funds people and organizations working to make increasingly capable AI systems more trustworthy and understandable — that includes researchers developing alignment methods, independent organizations auditing AI companies, and builders using AI to improve society's security and decision-making. Max joined the organization in October 2023. Before that, he studied computer science at Harvard College, researched the interpretability of image classifiers and language models, and organized educational and residency programs for people entering AI safety. His research includes work on finding misleading visual features that cause image classifiers to make mistakes, and on locating the components inside a language model that retrieve the values of variables in arithmetic problems. He also co-authored research on measurement tampering — the problem of an AI making the evidence look successful without actually accomplishing the intended task. Today his work involves deciding which people, experiments, and institutions could help resolve problems like that. His team's 2025 research funding call covered 21 areas and anticipated roughly $40 million in research grants, with room to spend more. Max has also publicly emphasized the need to study the alignment of systems substantially more capable than those we use today.

   1:00:08

   Max Nadeau: Hello.

   1:00:09

   Prakash Narayanan: Max, welcome to the show.

   1:00:10

   Max Nadeau: Thank you. Thanks for having me.

   1:00:14

   Nathan Labenz: It's great to have you here. Thank you for stepping out of your shadowy conspiracy and addressing the public on what you guys are trying to do for AI safety. It's big of you to do that.

   1:00:27

   Max Nadeau: Yeah, there's a tweet about how — because I went 48 hours without tweeting, I must, there must be some vast shadowy conspiracy that I was involved in. I skipped tweeting on Labor Day, and that was really suspicious.

   1:00:44

   Nathan Labenz: For starters — I just listened to your recent episode on the 80,000 Hours podcast, which as always is great output from them and you. We don't have as much time today as they had, so I definitely encourage people to go check out the full 80,000 Hours episode for more background. Maybe for starters, for people who don't know — can you just tell us about Coefficient Giving? I've long associated it with Dustin Moskovitz kind of individually. Is he still the overwhelming share of funding in this space? New money is coming online, I think, but has it really gotten to the point yet where it's funding grants? Where are we in terms of where the money is coming from?

   1:01:32

   Max Nadeau: Yeah, sure. Okay, so Coefficient Giving, which used to be called Open Philanthropy, is not technically a foundation, although we kind of operate like one in practice in that we make grants. We're a philanthropic adviser, and the main person — or couple — that we advise are Dustin Moskovitz and Cari Tuna. So still, the vast majority of the money that we're advising on and indirectly directing is their money. Now, we do have other — we are a multi-donor advisory now. That's a big part of why we renamed to Coefficient Giving, to symbolize that change in our strategy and direction. We have a bunch of individual funds you can see on the CG website, and the funds have different donors. There's a lead-elimination fund, for example, which has a bunch of different donors that aren't just Cari and Dustin, aren't just Good Ventures. But yeah, still, Cari and Dustin are most of the money. So I guess I should have touched on the very basics for people who don't know, which is that we do philanthropic advising in a super-wide variety of different cause areas, ranging from medical R&D to animal welfare to AI safety, which is what I work on.

   1:02:49

   Prakash Narayanan: Indeed. And how involved are Dustin and Cari in the day-to-day operations? Is there, like, an investment committee that you have to send proposals into for approval? Is that how it works? Who sits on the investment committee? How does this process work — because you guys are handing out hundreds of millions of dollars.

   1:03:13

   Max Nadeau: Yeah, so for the smaller grants that we do, the involvement is lesser. And for the bigger grants there's more involvement, and we talk to them a lot about the big ones.

   1:03:30

   Prakash Narayanan: And are there other people on the board — like, on the committee that approves investments? Is there someone from, I don't know, some adviser from the Gates Foundation as well, or some other people in there? Is it like a corporate board? How does that work? I mean, no.

   1:03:48

   Max Nadeau: We definitely don't have, like, a giant committee signing off on every single grant we make. We just make way too many grants for that — we move way too fast for that to be practical. So for most of the grants we make, the decision just gets made by employees at CG. The people in the decision-making hierarchy are some subset of the people in between, like, me and Alexander, who's the CEO.

   1:04:13

   Prakash Narayanan: Right on, right on. And how long does that process take? What kicks off that process — is it a proposal coming in externally, a call for proposals from you guys? How does that back-and-forth work?

   1:04:29

   Max Nadeau: It varies a lot by grant, by cause area. We've definitely run requests for proposals before, where we've put out a big public call and then gotten inbound and there's a whole process. But a lot of the grantmaking we do isn't through giant programs like that — it's that we reach out to someone, or we pitch them on doing something we think is impactful, or they contact us through private channels and say they want more money. Especially for grant renewals, when it's somebody we already fund, that mostly is just kicked off by them emailing us.

   1:05:02

   Prakash Narayanan: And so that brings us to a good place to ask — how does one get a grant renewal? What are the metrics you use to decide whether to extend the grant or not? When they start off the project, do you know the metrics ahead of time, and do you define the metrics by which you're going to measure the success of that project?

   1:05:21

   Max Nadeau: I'll just speak for my team — different teams do things very differently. I think in some ways the technical AI safety team is perhaps the team at Coefficient Giving that is least driven by clear, upfront metrics. The people doing global-health work are very metric-oriented and have way better, more rigorous ways of assessing the quality of their grants than we do. I think that just comes with the territory — we're funding research, and when you're funding scientific research you can't have metrics like 'you have to make this many breakthroughs.' So I think funding technical research on AI safety is a domain that's a lot less well-suited for metrics than something like bed nets is. For the most part it's not agreed on upfront — it's just that the areas, the types of research, the subjects are agreed on upfront. We talk to the grant applicant about what sorts of problems they're interested in, what methodological approaches they're going to take, and then they go from there. And then we evaluate, in a different, more informal, less metric-driven way, after the fact.

   1:06:42

   Nathan Labenz: I want to give you a chance to speak broadly to Project Tailwind, the big new announcement, and maybe highlight some areas that deserve highlighting, or that you want to make sure people are aware of that might not be obvious. Before doing that, though — one comment you made on the 80,000 Hours podcast that I thought was quite interesting was that you're excited about funding new independent auditing, investigation, verification orgs. Obviously there's been a tremendous amount of discussion in recent weeks around the events that need investigating and who is coming in to do it, and we've seen familiar faces on the front of the New York Post of all places. What are you looking for when you think about those orgs? And do you feel like Coefficient Giving funding them creates a problem for those orgs in the current political environment? There's sort of a Heisenberg principle here — in acting to support some of these new organizations, you might change the way they're perceived. So I'm wondering how you're updating your strategy in real time — wanting to see those orgs exist, but also feeling like, ideally, you'd have a more independent, more diverse funding base for these orgs, if only for public credibility's sake.

   1:08:13

   Max Nadeau: Yeah, I think there's something to that. To answer the first part of the question first, in terms of what we're most excited about people doing — I'd talk about a couple different categories of work that I think is very promising and important for the world right now. One is on assessing the safety, either of models or of whole AI companies. There are a lot of different ways of operationalizing the goal of doing assessment and evaluation of safety, and we're excited about all of them, basically. I'm very excited about more work on alignment red-teaming that tries to help us better anticipate the ways in which AIs will misbehave before that actually happens — that's something you can do at the model level, even on an open-source model, and that might be easier and more valuable. I'm also very excited about people doing more process-level safety assessment. I think one common reaction I heard a lot to the Hugging Face incident is that something went wrong at OpenAI at a process level — in particular, it seems bad that information about some of these incidents was known to researchers at OpenAI and didn't make its way through the organization to leadership for weeks or months. So assessing the safety of organizational processes, of individual models, of RL training pipelines, which is maybe somewhere in the middle — all of those are really valuable areas for work. You can call that category safety assessment, or auditing, or something like that. And then another category of work, which you touched on, which is overlapping but somewhat distinct, is around better evidence generation. It's just really hard, even for the most informed people right now, to understand: what is the state of AI capabilities? What is the state of AI alignment? What is the state of these AI systems in general — how do they behave, what are their personalities, in what ways is it accurate to anthropomorphize them, in what ways is it inaccurate? We just really don't have a good science of these systems we've built — how they behave, how they're going to act in new circumstances, what their motivations are, if that's even an accurate way of talking about them. So better evidence generation can look like a lot of things. It can look like just doing science on these systems to understand them better. It can also look like the sort of incident-detection, incident-reporting work that Nightingale and other organizations like that have done — I don't know if you guys saw the reports about the German wiki that OpenAI agents took over. That was work done by an independent organization that was crawling the internet looking for evidence of AI incidents, because these sorts of incidents can happen and then just go unnoticed unless there are people actually trying to look into them and present that information to the world and understand the story behind it. So evidence generation is also a diverse and important area of work. And then, in terms of funding and credibility for these organizations — obviously we would love for more funders to be funding these things, and that is something we're seeing. There have been a bunch of new funding organizations that have come online recently interested in AI safety — Schmidt does a bunch of work in this area, and the OpenAI Foundation is probably going to do a bunch of work in this area, and there are other funders too. Litecoin Commons is a new one, for example, and some of that money is probably going to be from Dustin and Cari also, but not the majority of it. So we're all for more people funding these organizations. And with a group like METR — they've been able to support themselves without getting any money from Coefficient Giving, and they're one of the biggest and spendiest organizations in the third-party auditing space. So I think that just goes to show there actually is a ton of funding available from a wide variety of sources, including non-CG, non-Dustin-and-Cari sources, for groups who want to do this kind of work.

   1:12:33

   Prakash Narayanan: So this is a good point to ask — we've had, I think, nonprofit organizations kind of creating the entire AI-auditing, AI-observation space. How do these organizations and practices move into someone like Accenture? How do we transmit what's been created at the seed stage into something much bigger that large organizations can put into place to monitor, observe, and audit these AI models and systems?

   1:13:22

   Max Nadeau: I think the short answer is they should just borrow the methodologies that have been developed by these other auditing organizations. We have pretty mediocre, but better-than-nothing, methods for assessing the alignment of these systems, and I think none of them are that good. There was a comment from Evan Hubinger on LessWrong recently about how state-of-the-art alignment evaluations provide very little evidence one way or the other about whether models will misbehave in catastrophic ways in deployment. I'm not saying these alignment evaluation techniques are good, but there are better and worse ones. So I'd want organizations trying to do alignment evaluations to pay attention to best practices, and to try and hire the people who worked on that stuff in the past. Now, I don't know if the people who have the expertise in this area would have the most impact by joining these organizations or not — that's something I don't have too much of a view on.

   1:14:34

   Prakash Narayanan: To what extent do you think organizations like Accenture often function as box-checkers rather than investigators? There's a clear difference between an auditor that goes in to make sure processes were followed — even if those processes didn't yield the outcome you actually desire — and an investigator who's there to actually find the thing. So what's the differentiation between those two, and do you think the Accentures of the world are going to end up in the box-checker mode rather than the investigator mode?

   1:15:13

   Max Nadeau: I know very little about Accenture, so I won't speak to them specifically — I have no idea what role they'll end up playing. But I think it is a thing to be concerned about in general, putting aside the specifics of the organizations involved: that these third-party auditing groups won't have enough access, or won't have the incentives, to pursue really hard-hitting assessments of how safety is going at these AI companies. I think part of why that is, is that in many cases this is all very voluntary. To my understanding — and I don't work on AI policy, so it's possible I'm getting this wrong — the only legally mandated role for third-party auditors now is to ensure compliance with the self-regulatory, RSP-style policies that these companies are mandated to have by state law in California, New York, and Illinois. You can just put whatever you want in those policies, and they're very broad in a lot of cases, so that doesn't leave much scope or much work for these auditors to actually test anything in particular. That's as it concerns the legally required auditing. Now, when it comes to what labs might voluntarily agree to — I think there have been some very encouraging signs in terms of what OpenAI and Anthropic have said recently about the sorts of access and the sorts of questions they want these auditors to answer. Whether that actually happens depends both on how these companies actually implement the nice words, and also on how the auditing companies react to that. So I guess we'll see.

   1:16:58

   Prakash Narayanan: Let's switch gears a little bit. So you awarded Geoffrey Irving's team quite a large award — it's structured, I think, as kind of a compute-and-other kind of award. Can you tell us a little bit about the award itself, the structure of the award, and how you guys came up with that structure?

   1:17:22

   Max Nadeau: Yeah, sure. It's been a little bit since this happened, so I'm not sure I'll get the details exactly right, but I think what you said is correct. There's a part of the grant that's for research expenses, the biggest of which is compute — and when I say compute, I mean both just renting raw GPUs, and also paying AI companies for their tokens. That's becoming a giant part of all the grants we make, because people increasingly want to run very large experiments, which require a lot of tokens or a lot of GPUs to run open-weights models. And also, very excitingly, people are starting to use a lot of AI labor, and that requires a lot of spending on tokens as well. AI safety is sort of a funny discipline in that you can use tokens both on labor and on the subjects of the experiments — you're spending a lot of tokens to have some AIs run experiments on other AIs. So that creates these high compute budgets. Compute being the biggest expense is also something where the actual spending varies by orders of magnitude, over time and across grantees. So sometimes the way we structure these awards is that we want to be very generous with compute and err on the big side — we just want to give that as a lump sum that can be used for research expenses and not other things. And we're very comfortable with the uncertainty that maybe they only end up spending a tenth of it, and that's fine — or maybe they spend all of it, and then they come back and we'll give them more. Because of that variation over orders of magnitude, that's the way we handle it.

   1:19:19

   Nathan Labenz: Can you guys hear me now? I just switched to—

   1:19:21

   Prakash Narayanan: Yeah.

   1:19:22

   Nathan Labenz: —my mobile device. Okay, cool. For some reason the sound was bad on the laptop, so we're doing it live from the phone. Geoffrey Irving is obviously one of the most institutionally successful, qualified, and, like, legible people you could possibly pick for a grant. One thing I've been struck by over the last couple of years is how often people that I might lovingly call weird have interesting and, at times, prescient ideas. Janus's Twitter feed, for example, has been extremely valuable to me — even though when I first encountered it I was like, who is this? Anon? Are they sane or not?

   1:20:04

   Prakash Narayanan: What do you guys think about

   1:20:06

   Nathan Labenz: taking chances on weird people and weird ideas?

   1:20:11

   Max Nadeau: Yeah, it's definitely something that we believe in and that we do a lot more than other funders. Most of the grants we make are not to credentialed academics — they're to random people out in the world, at nonprofits, or doing independent research, or something like that. Now, I think many people feel that we're not comfortable enough with weird people — they feel like there are all these weird people we should be funding who we're not. And I think some of the time, the reason for that is that the person just didn't apply to us — we emailed them and they didn't respond, or we asked them for some information and then they just ghosted us or something like that. That's the sort of thing that happens when you're talking to weird people — sometimes they're not putting their best foot forward in the application process. I say this just to emphasize that this happens more than people expect, and it explains some situations where I'm like, yeah, that person is very weird, doing great work, we would love to fund them, and they just don't seem to be interested. That's not to say it's ubiquitous — I don't want to say we're above critique on this dimension, and I encourage people to keep telling us if they think there's someone doing great work who we're overlooking for bad reasons and it'd be great for us to fund them. And, you know, our priorities change over time, our staff change over time, our capacity for picking up side projects, side grants, changes a lot over time. So I really hope people aren't overly discouraged if they reach out to us and we just don't have the capacity to look into their project at some moment — maybe there'll be some opportunity for us to have a productive collaboration and fund their work at some point in the future.

   1:22:18

   Prakash Narayanan: Geoffrey Irving has said that superintelligence might arrive in two or three years, and that's too quick — he proposes that we slow down. Rather than discuss his views specifically, how do you manage the uncertainty of the time available, and how does that affect what you fund and how urgently you fund something? Because sometimes you might get something with a ten-year kind of timeline that's going to take iterative work over a long period, and sometimes you might get something like, 'hey, I just want this for next month.' So how do you sort out which projects to fund based on this kind of uncertainty in the time that you have?

   1:23:07

   Max Nadeau: I think the boring answer is we just have a portfolio approach — we fund some stuff that's only going to pay out on really long timelines, and other stuff that, if it's valuable, will probably be valuable soon. I think this question of timelines to AGI or ASI, in some ways, matters less for research prioritization than one might initially think. In some ways the prototypical example of a long-timelines bet is funding work that's very theoretical or ambitious — you look at it and think, this is going to take a decade to pay off, because it's, like, some work in mech interp, or Paul Christiano's work at ARC, or these sorts of things where we're just going to need to make a lot of scientific or mathematical progress before it actually bears fruit in terms of methods we can apply in reality. So that sort of feels like work that's implicitly a bet on longer timelines. But I think that actually isn't really true — if we are going to have powerful AI soon, then presumably we're also going to have an opportunity, very soon, to make use of gobs and gobs of AI labor. Especially on work that's more mathematical, we're going to be able to get five or ten years of progress in one. So the sort of work that would take humans a long-timelines amount of time to complete might actually be possible a lot sooner under this hypothetical that we're going to have superintelligence or AGI very soon — so long as we actually make use of those AIs and take the opportunity to make a lot of progress very quickly.

   1:24:51

   Prakash Narayanan: So there's a potential upside surprise in funding longer-term work, in the sense that you might get helped along — exactly — by the increase in intelligence at some point. Yeah.

   1:25:08

   Nathan Labenz: Going back to the weirdness dimension for a second — what would you say are the most speculative things you included in Project Tailwind's request-for-projects list? And were there any things that you, as a team, considered but chose not to include, either because it was just too weird to put on a list, or because you definitely think they're not worthwhile?

   1:25:40

   Max Nadeau: Okay, well, I actually wasn't that involved in the creation of the list, so I can't answer the second part — I just don't remember. On the first part, in terms of what we ended up including — I think what we included was a little bit coarse-grained, but we have, if I remember correctly, some stubs on the Tailwind list about new research centers producing or pursuing more ambitious, more principled bets on alignment. That's something we've spent a bunch of money on this year and that we're excited about — work like ARC, Paul Christiano's org, who is not a CG grantee, but we've funded other people working on the ARC agenda and on stuff very similar to it, because we think that's a valuable Hail Mary shot to include in the portfolio. And also, Resolution — our biggest grant this year — is very much a bet on a kind of crazy thing that has never worked yet, which is: maybe we're going to have some more principled theory of AI that's going to give us stronger reasons to believe in the alignment of our systems. That's something we're very excited about, and I think it's very weird and out there. A lot of people have very understandable skepticism — people who are familiar with what has and hasn't worked in machine learning over the last decade recognize that all along there have been people in ML who wanted to apply theory more to AI capabilities research, and that hasn't really worked very much compared to just blind groping around, trial and error. It's like the joke about how the success of these methods, we attribute all else to divine benevolence — that's been the actual ethos that's produced progress in AI capabilities. So a lot of people take the lesson that trying to do more theoretical or principled work on AI alignment is pretty doomed also, and I think that's a totally valid reason for skepticism, but we think the upside is worth it despite that. So we're taking that weird bet.

   1:27:59

   Nathan Labenz: Yeah, I love it. Do you have any room for, let's say, other architectures? One intuition I have broadly is that we're sort of speed-running the path to superintelligence, RSI, whatever, on basically the first architecture that worked pretty well, and I just kind of wish there was more diversity of architectures. I can easily imagine that some other architecture might have more favorable properties in any number of ways, but we're just on the mainline — it keeps working, so we keep jamming the accelerator on it. Do you have any hope, or would you be open to people coming in with things that, traditionally in the AI safety parlance, would look more like capabilities work, but alternative paths that maybe create diversity, or generate other insights that we're just not going to get through the current — you know — scale-the-hell-out-of-the-transformer paradigm?

   1:29:07

   Max Nadeau: Yeah, we're definitely open to that. I think the biggest example I'd highlight is around externalized reasoning. In our request for proposals from last year, that was one of the sections we included — actually there were two relevant sections to your question. One on new architectures that externalize more of the reasoning of the AI, most typically into chain of thought. And then we also had a section on more inherently interpretable architectures. Both of those are areas we're excited about — though I'm not that excited about them. Speaking personally, I don't consider those things to be super-high priorities. Although I definitely think that — which I think is not really the spirit of your question — having architectural tweaks in the direction of preserving, and maybe even improving, the legibility of chain of thought, that I think is very important. But these more radical shifts on architecture — I just think these are directions of research that are good, and that we want to support people who are excited about them, but I just think other things are better.

   1:30:19

   Prakash Narayanan: What do you think — you've been doing this, I think, for... 18 to 24 months, something like that? Right — you've been doing this about three years. Three years. Right. So what do you feel you've learned that you wish you'd known three years ago?

   1:30:41

   Max Nadeau: It's a great question. I think the biggest one is just that there's a lot of value in funding whole startups — whether they be for-profit or nonprofit, though I think we're more excited about nonprofit — as opposed to smaller grants, whether those are project grants or whatever. There's a lot of upside to cutting bigger checks. To some extent it allows you to fund more people, because you endow the money into somebody else's hands and then they have to do the work of finding the valuable people to work on their project, instead of me having to do that. And having a bunch of money in one place serves as a kind of coordination mechanism — it brings in multiple people to work together on something, which you don't get as much when you fund multiple people separately with smaller grants. And then, when you have a bunch of people already working together who already have a startup or organization connecting them, that allows them to pivot more rapidly and have more impact if they do want to pivot into some other area, because they've already worked out the details and they're ready to go, to take on some project if they come across something more impactful than what they were previously working on.

   1:32:21

   Prakash Narayanan: So — I think there's a comment you made which—

   1:32:24

   Nathan Labenz: —I think is becoming kind of conventional wisdom, as we're seeing these bigger and bigger checks get written, and we're seeing salaries in places like METR and Redwood approaching competitive levels with the labs, although perhaps not quite all the way there — is that talent is the one big bottleneck. There's plenty of money, it's all about talent. Obviously Silicon Valley founder types are one profile you'd want to recruit — are there other pools of talent that you've identified as strategic priorities?

   1:33:02

   Max Nadeau: Yeah, it's a great question. Just to be clear — for the things CG is supporting, and especially for the things on the Tailwind list, the money is not the bottleneck. The talent is. There are other things within AI safety where money is definitely a very big bottleneck. To answer your question, I think in some ways the most valuable profile of talent is someone who combines the virtues of a founder with the thoughtfulness and the inclination toward, and comfort with, speculative thinking and futuristic questions that's more common in the AI safety world. That's a serious, no-bullshit attitude to asking questions about what's going to happen in the future of AI — really making predictions, testing your hypotheses, updating over time. That's a really valuable skill to us, because it lets people get going years earlier on projects, before other people can see the value of them. The example I give is METR — before other people were thinking seriously about AI capabilities and their impacts and how to evaluate them, METR was very prescient in coming up with a benchmarking approach in time horizons that worked way better and aged way longer than other approaches. That's because they were really trying to take seriously what was going to happen in the future with AI, and then work back from that to what sort of work they should be doing now that would age well and prepare for that. So that inclination is really valuable to us — it's useful in basically all profiles, both in a person who might found an org and in a person joining an existing org, because it's a big part of how existing groups in the AI safety space do their work.

   1:35:04

   Nathan Labenz: Can you do one quick double-click on where money still is a bottleneck? Yeah.

   1:35:15

   Max Nadeau: I think the biggest thing is just, in organizations and areas that CG doesn't operate in, money is a big bottleneck. If you find some person you think is doing great work, and for some reason they either aren't a good fit for CG, or they applied to CG and we weren't able to fund them, or we messed up and decided not to fund them — I think there's still alpha in that, especially because, as we move into making much bigger grants, that can sometimes come at the expense of covering all our bases on the teeny little uses of money that can be valuable. I think that's the right decision for us to make, but it does leave some things unfunded.

   1:36:06

   Prakash Narayanan: So let's talk a little bit about the—

   1:36:09

   Nathan Labenz: If I recall correctly, maybe—

   1:36:10

   Prakash Narayanan: Right. Sorry.

   1:36:13

   Nathan Labenz: Collision! Well — let me do a quick follow-up, and then you can take the next one, Prakash. I was going to ask about model welfare and consciousness research — if I recall correctly, that was sort of a marginal one for Coefficient Giving that maybe is now on the outs, but maybe it's back in again. And then also about political activism — I recently talked to Anton Liketon, who has this provocative thesis that we need an AI-safety PAC in every jurisdiction. I'm not sure I'm sold on that, but I'm wondering if you are.

   1:36:51

   Max Nadeau: Okay, I'll take those in order. The model-welfare stuff — that is something we do a little bit of grantmaking on now, although there was a time when we weren't doing that. On the PACs and political activism, I really just don't know — sorry, it's not my department. I just make the technical AI safety grants. And I also think there's a lot of stuff in the broad political area that CG just can't touch and has no relationship to — so it's not only not my department, it's nobody at CG's department.

   1:37:28

   Prakash Narayanan: So let's talk a little bit about preparing to give larger grants. Obviously we're in this time period before the Anthropic IPO — the expectation is that post-IPO there's going to be some distribution of the proceeds, and I think some public records indicate that Coefficient Giving would be the recipient of a large amount of funding that you'd then be handing out. I don't know to what extent you guys are public on this, or what you're allowed to disclose — so, what is your preparation process for handing out larger checks? What's your pathway for the next year, say?

   1:38:19

   Max Nadeau: I have no idea what's going to happen with the AI-company money. The other thing I'd say is, now that Coefficient Giving is a multi-donor advisory, there's less of a fact of the matter about whether it's flowing through us — it's more of a spectrum than a binary. Maybe we'll be directing it in the sense that we're going to keep making grants and writing our takes on the internet, and maybe some of the people with that money will listen to us and take our advice, which would be great — that's a very minimal sense in which we might be influencing it. For the more maximal sense, I just have no idea whether that's going to happen — you'd have to ask somebody higher up at CG. In terms of what the preparation actually looks like, I think the biggest thing is just seeding organizations now that can then be ready to absorb lots and lots of money — making small grants to new groups and giving them a little bit of runway to prove themselves, or not, and explode. We'd love to create groups of people who can say credibly to big donors, 'we have people, we have a project ready to go, we're just waiting for you to sign the check — we've got this shovel-ready project, and it's going to take a billion dollars, and it's going to solve alignment.' And that's really not the situation right now — there just aren't that many people in the space, and that's a big thing we're trying to fix.

   1:40:02

   Prakash Narayanan: And is that about funding people enough so that they can work on things at small scale, such that they can get a much bigger compute check at some point to scale that up? Is that the idea?

   1:40:16

   Max Nadeau: That's the main thing I have in mind. I think the same logic also applies to growing the headcount of their organization — having a group in place with a great plan that can then absorb a lot of software engineers, ML researchers, data labelers, or some other type of person. But yeah, compute, like you said, is the main version of that — and especially, hopefully, AI labor. It'd be really great if there were a bunch of organizations that were just waiting on lots of AI labor to pour into them. We'll see if we can make that come true — I think there are reasons to be skeptical that it will happen, but it'd be nice if it did.

   1:40:59

   Prakash Narayanan: What do you expect your hit rate is going to be? In venture capital the typical hit rate is 10% — one in ten projects actually goes the whole way. What's your expectation of what the hit rate's going to be? How many seeds do you have to put in to get something that works out for the funding in six to twelve months?

   1:41:22

   Max Nadeau: It's a great question. To some extent that's up to us — we could go down the diminishing-marginal-return curve more, or we could not. We could decide to be really picky as we review the Tailwind applications.

   1:41:40

   Prakash Narayanan: Sorry — what is the DMR?

   1:41:42

   Max Nadeau: Diminishing marginal return. Sorry.

   1:41:43

   Prakash Narayanan: Oh, okay. Yeah. Right.

   1:41:46

   Max Nadeau: But, yeah — so I don't know. This is very much uncharted territory for us, this Tailwind thing. We've funded new organizations in the AI safety space in the past — for example, I was involved in the first grant we made to Apollo Research a couple of years ago, and I think that one has been a great success. But because we're doing this at larger scale now, I expect the hit rate to go down, and I don't think we can really just generalize from the rates in our historical track record. So, I don't know — I guess 10% doesn't sound crazy to me, but who can tell?

   1:42:34

   Nathan Labenz: Are you guys also still in the business of doing career-transition or personal-development grants — if somebody's, like, 'I'd be interested in exploring starting an organization, I'm not quite ready to pitch you, but maybe in six months of intensive study I could get there' — is that a request they can still make of Coefficient Giving?

   1:42:56

   Max Nadeau: Yeah, so as of a couple of weeks ago we've handed off our CDTF — career development transition funding — program to Blue

   1:43:08

   Prakash Narayanan: Dot instead of us. Well, good to know — I hadn't caught that.

   1:43:15

   Nathan Labenz: We're just about at the time we had booked, and I'm sure you're very busy with lots of applications rolling in all the time, and no shortage of work to be done to get ahead of this whole thing. Anything else you'd want to leave people with that we didn't cover — just to make sure people are aware of it?

   1:43:36

   Max Nadeau: Cg.org/tailwind is the way to learn more about our program to fund new organizations. I guess I feel like I never gave the proper elevator pitch, so I'll deliver it now — better late than never. Tailwind is a new initiative we're starting for philanthropic startups, and we're giving out checks somewhere between $200K and $200 million. Those are divided into pre-seed, seed, and really big. The pre-seed checks are $200K to $2 million. The seed checks are $2 million to $20 million. And the really big checks could be as much as $200 million. For pre-seed grants, you don't need to have that much — you just need a person, maybe two people, and a good idea. For seed grants, you need a little bit more — you need a founding team that covers the obvious bases and has some preliminary technical results. And then for the really big checks — we gave Geoffrey Irving $160 million right out of the gate — you need to be pretty special. There's a list of about 40 blurbs, initiatives, on the Tailwind website, which again is cg.org/tailwind. Those aren't comprehensive of everything we want to fund — a lot of the grants we're going to make through Tailwind will be to people bringing us ideas that reduce catastrophic risks from AI but aren't listed on that website. So, yeah — we're just trying to get more new organizations into the space, and we're providing generous funding to do so.

   1:45:12

   Nathan Labenz: Amazing.

   1:45:13

   Prakash Narayanan: Amazing.

   1:45:14

   Nathan Labenz: Max, thank you so much for all this hard work — it's a big needle-mover, I think, that there are funders out there willing to put as much as $200 million behind the project. It is time — it is crunch time, as they say. So I really appreciate all that you're doing, and we'll definitely be looking out for updates in terms of new and exciting grants you guys are funding, and any new requests for proposals you put out as well.

   1:45:45

   Prakash Narayanan: Great — well, thanks so much for having us—

   1:45:46

   Nathan Labenz: —here on AI in the AM.

   1:45:48

   Max Nadeau: Bye.

   1:45:49

   Prakash Narayanan: Thanks, Max. It's been a pleasure.

   1:45:59

   Nathan Labenz: Well, at least I can hear you on my phone — that's the good news. We don't have the full feature suite here on the phone, though. Do you think I should try switching back—

   1:46:11

   Prakash Narayanan: Yeah, sure — why don't we—

   1:46:14

   Nathan Labenz: —return to Dawn and see what Dawn has done in the 45 minutes since we left. For some reason I want to say 'her' for Dawn — I usually—

   1:46:24

   Prakash Narayanan: I told you, it's a her.

   1:46:28

   Nathan Labenz: Starting to anthropomorphize. Let me refresh the studio on the laptop, and I'll rejoin there. It seemed to have a cool feature — it seemed to flip me out of the one browser since I clicked into the other. So, alright.

   Clips

   • Clip 01

     Tailwind's Grants Reach $200 Million

     0:00 / 0:00

     1x
     ↓Download↗Share clip
   • Clip 02

     The Weird Bet On Alignment Theory

     0:00 / 0:00

     1x
     ↓Download↗Share clip
   • Clip 03

     Talent, Not Money, Is The Bottleneck

     0:00 / 0:00

     1x
     ↓Download↗Share clip
   • Clip 04

     AI Could Compress A Decade

     0:00 / 0:00

     1x
     ↓Download↗Share clip
3. 2:03:20Closing27 min

   Closing — what Dawn did in forty-five minutes, and Amazon banning Meta's MuseThe hosts come back to Dawn, the Pion agent they hired in the opening, and find it has researched twenty-four leads, written a one-pager, and asked a genuinely sharp first question about how this venture relates to the Cognitive Revolution. Nathan notes the architecture — a front-end agent they talk to and a long-running worker behind it — and compares it to task-scoped agents he already uses. Prakash reads it as a go-to-market motion that firms like Clay sell, now internalized into the company that needs it. Then the main item: Meta's Muse agent has topped the app charts and people have pointed it at Facebook Marketplace, where it lowballs sellers who do not know they are talking to an AI, and Amazon has banned it. Nathan cannot see the logic — the volume is still small, the option value of watching agents shop is high, and the traces would be worth having — and says the same thing he has long said about export controls: you give up more by moving early than you gain. Prakash's answer is that this is about who owns the customer, because whoever owns the customer captures the margin and everyone else becomes a commoditized protocol; he predicts the ban is a temporary exclusivity window through the shopping season for an OpenAI agent, timed to DevDay. They close on what an agent-first internet would actually require — agents as first-class users, API rather than HTML, an advertising model that survives a thousand-to-one agent-to-human ratio — and on Nathan's worry that blocking agents just teaches them to look human, the same mistake as putting too much pressure on the chain of thought.

   Watch

   ▶Play this segmentYouTube · from 2:03:20Open segment on YouTube ↗

   As aired

   The closing segment opens mid-troubleshoot: the audio problems that had been dogging the show are still not fully resolved, but Nathan decides it's good enough to push forward. He and Prakash turn to checking in on Dawn, the autonomous-organization experiment they are running live on air, mediated through an onboarding agent called Pion. Pion has been given control of some of the interaction, and its first substantive output is a question about how the whole project relates to The Cognitive Revolution's business — a line Nathan calls smart of the agent to ask, since he and Prakash are running this independently, with no formal business relationship with "the entity formerly known as Turpentine."

   Beyond that scoping question, Pion has already produced 24 researched leads and a one-pager, well beyond the five pending questions Nathan expected. Prakash frames the output as a textbook B2B go-to-market pitch, the kind a company like Clay would sell — his SaaS-founder friends' rule that a product succeeds not by what it can do but by what task it fully removes from a decision-maker's to-do list, letting you charge up to the value of that job regardless of how much effort it actually takes. He ties this to why so many people have piled into AI services generally: if AI can deliver a service at human-level pricing while costing almost nothing to run, the spread is pure margin.

   The hosts clarify the architecture: Dawn is not Andonos, the managing agent Andon Labs presented at signup — there are two separate agents at work, a UI-facing agent they interact with directly and a backend "worker" agent, Dawn, that they never talk to directly. Nathan compares the setup to his experience with Tasklet-style agents, where a top-line agent defines objectives and spins up short-lived sub-agents to execute and report back each cycle. He notes this case is different because he expects Dawn to be long-running rather than repeatedly spawning disposable sub-agents. With the UI experience feeling familiar, the hosts agree to keep "YOLO-ing" it and let the agent proceed.

   From there the conversation pivots to Amazon and AI shopping agents, kicked off by Prakash's rundown of Meta's Muse: launched by Facebook, it has reached #1 on the Apple App Store download charts. Some users have set it loose on Facebook Marketplace with instructions to lowball sellers by at least 20% below asking price — reaching out mostly to older users unfamiliar with AI and offering roughly half the listed price. Prakash recounts one seller who got so annoyed he threatened to report the buyer to the police, not realizing it was an agent at all. He cites Nikita Bier (head of product at X, formerly of Facebook) reacting publicly to how badly Mark Zuckerberg must want Muse to succeed, given that Facebook Marketplace's own product leadership would have known this kind of repeated lowballing behavior would wreck the marketplace experience — yet Zuckerberg apparently green-lit it anyway. The other shoe: Amazon has banned Muse from shopping on its site.

   That ban launches an extended debate on who owns the customer relationship in agentic commerce. Prakash's framing is that everyone is "standing on Earth watching this giant meteor head towards you" — a Don't Look Up analogy for how Facebook could plausibly route hundreds of millions of existing WhatsApp, Marketplace, and Messenger accounts through Muse, narrowing commerce down to a single interface companies won't be able to refuse to work with. He argues whoever sits at the customer touchpoint gets to make the money, while whoever doesn't becomes a commoditized B2B protocol supplier — the Shopify analogy, where Shopify itself doesn't own the end customer. He backs this with Amazon's own numbers: advertising is roughly 10% of revenue at an ~80% gross margin, versus an ~$800–900B retail business running at maybe a 5% margin — meaning the stakes in who intermediates the purchase decision are enormous, and he predicts Muse-style agents recommending a small, curated set of options (one of them sponsored) could become an even more concentrated, lucrative ad rent than the old ten-blue-links search era.

   Nathan pushes back on the ban itself, saying he still doesn't see why Amazon would rush to block third-party shopping agents rather than let them operate at small scale and learn from the behavior while adoption is still an early-adopter phenomenon. He draws an analogy to chip export controls: because chip buildout is going super-exponential, pulling the trigger on a ban later still captures the vast majority of future capacity, so there's little cost to keeping the option open longer — the same logic, he argues, should make Amazon comfortable tolerating agentic shopping for now rather than banning it outright. He notes Amazon has a reasonably good track record (with a nod to Andy Jassy's AWS pedigree) but says the move still doesn't add up to him. Prakash offers a theory: with OpenAI DevDay approaching and OpenAI having hinted at its own shopping agent, he suspects Amazon may have a preexisting exclusivity deal making OpenAI's agent the first (and for now, only) shopping agent on Amazon — meaning the Muse ban could be a temporary, few-months arrangement tied to this year's shopping season and a revenue-share/marketing deal, rather than a permanent policy.

   The segment closes on the broader governance question. Nathan previews tomorrow's Cognitive Revolution guest Alan Chan, a pioneer of agent-governance theory (agent IDs, how to know who an agent represents, and how a recipient decides whether to serve a given agent's request), and mentions Cloudflare's move toward blocking agentic traffic — which he thinks is the wrong solution, since it just pushes him toward letting agents use his own browser and credentials directly, and risks an arms race where agents are engineered to look indistinguishable from humans, which he compares to over-optimizing pressure on chain-of-thought. Prakash counters with the sheer infrastructure mismatch: Facebook-scale agentic traffic could be 100–1000x what sites like Amazon or Cloudflare are built to serve (as happened with GitHub), forcing agents to become "first-class users" of the web — a shift he ties back to a recent conversation with someone from Mozilla — and unraveling the old advertising equilibrium where a known ~50–80% bot-traffic rate was tolerated because it was roughly proportional across advertisers; once that ratio blows out far further, he says, "the cat is out of the bag" and things will have to change. Nathan closes by saying he'd love to get someone from Amazon to explain their thinking, then reports that Don's next step is getting its own email address at their domain so outreach can begin, joking about how much trouble Dawn will get them into. The hosts sign off until Wednesday, previewing guests on GPU forward-pricing and financialization (a company referred to on air as "Orn") and Lewis Hammond of the Cooperative AI Foundation on multi-agent systems.

   Key moments

   Does it feel backward to you that Amazon would be trying to block shopping agents powered by other tech companies? I just have a hard time imagining that's the winning strategy for them.

   Who owns the customer gets to make the money. And if you don't own the customer, you get commoditized.

   It's like you're standing on Earth and you're watching this giant meteor head towards you, and you really don't know what to do — it's the whole 'Don't Look Up' thing.

   What we covered

   What Dawn did in forty-five minutes Twenty-four researched leads, a one-pager, and a genuinely sharp first question about how this venture relates to the Cognitive Revolution — which Nathan answers by saying they are independent. Prakash reads the output as a textbook go-to-market motion, and notes the architecture: a UI agent they talk to, and Dawn, a long-running worker they never address directly.

   Meta's Muse on Facebook Marketplace Muse has topped the app download charts, and people have pointed it at Marketplace with instructions to bid at least twenty percent under asking. It lowballs sellers who do not know they are talking to an agent — one threatened to call the police. Prakash's read, via Nikita Bier, is that Marketplace's own product leadership would have known this wrecks the experience, and that it shipped anyway shows how badly Meta needs Muse to work.

   Amazon's ban, and who owns the customer Amazon has banned Muse. Prakash frames it as the only question that matters in agentic commerce: whoever holds the customer touchpoint captures the margin, and everyone else becomes a commoditized protocol — the Shopify position. He backs it with Amazon's own split, advertising at roughly a tenth of revenue on an eighty percent gross margin against retail's five, and argues an agent showing three options with one sponsored is a richer rent than ten blue links ever was.

   Option value, and why banning early is the expensive move Nathan cannot see the logic. Volume is still small, the traces would be worth having, and he draws the same analogy he has long drawn about export controls: on a super-exponential curve, a ban you start later still covers the vast majority of what is coming, so moving early buys little and forfeits what you would have learned. His advice is to wait until it is five percent of traffic and then move. Prakash's counter-theory is that the ban is a three-month exclusivity window for an OpenAI shopping agent, timed to DevDay.

   Agents as first-class users, and the arms race if you make them hide Nathan's objection to Cloudflare-style blocking is that his own fallback is to hand an agent his browser and all his credentials, which is worse for everyone — and that pressuring agents to look human is the same mistake as putting too much pressure on the chain of thought. Prakash's is infrastructural: a hundred to a thousand times the traffic, served by API rather than HTML, and an advertising model that cannot survive the bot ratio becoming the whole ratio.

   Full transcriptLightly edited · timestamps jump to YouTube

   1:46:48

   Nathan Labenz: And alright. Now I hear you.

   1:46:56

   Prakash Narayanan: Yeah, so how are we doing? And it's 11:03 — how's Dawn doing?

   1:47:04

   Nathan Labenz: Try again.

   1:47:14

   Prakash Narayanan: So how is Dawn doing?

   1:47:18

   Nathan Labenz: Still not great — the sound is still not great, but maybe good enough to proceed through at least a little bit of additional Pion onboarding. You were lying, everybody! Yeah, okay, I heard you there. And it's gotta be something with my computer, obviously, because switching to the mobile device worked, but it's a rare computer issue that a full restart doesn't sort out. So I don't know.

   1:47:46

   Prakash Narayanan: You have an AI agent running there which has decided to utilize — you know, maximize its resources.

   1:47:55

   Nathan Labenz: It's possible. I've looked at Activity Monitor, and especially after the restart, I don't see any abusive use of resources, but I don't know. Okay, let's see what Pion has noted. First of all, its first question is an interesting one: how does this project relate to The Cognitive Revolution? Which is somewhat of a fuzzy line in general — we're doing this independently, we're not in any formal business relationship with the entity formerly known as Turpentine, but it's smart of the agent to recognize that's a question it should be asking. So I'll just say we're independent, and it can kind of do its own thing and doesn't need to coordinate with the folks running the business side of The Cognitive Revolution. So that's one. I thought we had five questions pending for us — it's got 24 researched leads, which is interesting. It's got a one-pager that it's created.

   1:49:16

   Prakash Narayanan: This is a traditional go-to-market that would be sold by a company like Clay or any number of firms. My friends who do SaaS always tell me it's not about what the product can do — it's about what problem the product can take off the decision-maker's to-do list. The product has to absorb something completely. And if you can take a job to be done, you can charge whatever you want up to the price of that job — it doesn't really matter how long or how much effort it takes you to execute it. Which is also why I think a lot of people have flowed into AI services right now — if you can deliver the service at the same price as a human, but you use AI and the AI costs almost nothing, that's your margin right there. I've relayed your decisions to Dawn. So this is interesting — Dawn is not Andonos. We have two agents here: an agent that's interacting with us, and another agent in the back end, which is Dawn. We don't interact with Dawn directly — we have a UI agent that we interact with, and Dawn is kind of like the worker. That's interesting.

   1:51:00

   Nathan Labenz: Yeah, this is — yeah. It's interesting because this is gonna be quite a long-running agent. It sort of echoes what I've come to be familiar with with something like Tasklet, where you set up an agent to do a certain job for you — it takes on a relatively well-defined, narrow scope. You have a top-line agent where you define its objectives and give it instructions, and then each time it executes, it creates a sub-agent that goes off, does the thing, and reports back. The main one can oversee these and try to learn lessons from the collection of their work. But each day when it hits its appointed runtime, it creates a new sub-agent — those are relatively short-lived, like 'you're here to do this job one time,' and at the higher level, one agent kind of owns the project. This is a little different because I do expect Dawn is gonna be quite long-running. So yeah, it's interesting to see the similar kind of UI experience. We're YOLO-ing this, so I'll just say let's go ahead and do stuff.

   1:52:40

   Prakash Narayanan: This is gonna be interesting. So I saw — I'm not sure if you've been following, but Muse — so Facebook has an agent called Muse. Facebook launched it, and it has reached the top of the Apple App Store download charts. It's number one now. And having reached the top, some people have set it loose to interact with people on Facebook Marketplace — specifically, 'go find me this object at a price which is at least 20% lower than the prevailing price.' So the agent goes off and low-bids people on Facebook Marketplace — it reaches out to people, and most people don't know AI at all, so it reaches out to basically older people on Facebook and says, 'Hey, I'll give you like half the price that you listed it for.' And I saw one where the guy got so annoyed he said, 'I'm gonna report you to the police,' not knowing it was an agent at all. And this has just kind of started to hit social media. So Nikita Bier, who was head of product at X and formerly at Facebook, said: do you realize what has happened now, how desperately Zuck wants Muse to succeed? Because the guy who runs Facebook Marketplace, the product manager, would have known for sure that repeatedly having agents reach out and lowball humans on Facebook Marketplace is gonna be a terrible product experience and completely kill the product. But when that guy said no, the Muse team would have gone to Zuck, and Zuck would have been like, 'maybe just a little bit — let's open that door.' So Zuck is willing to sacrifice the customer experience on the rest of the network in order to get Muse up and running. That's number one. Number two — Amazon just banned Muse. Muse was going there to do shopping with people, and Amazon banned it. That's gonna be interesting because that brings out antitrust issues — Amazon's a big company, you can't just ban people from your website, especially if they're helping other people buy stuff from your website. That's an antitrust issue. And Amazon has invested in OpenAI and is a big investor in OpenAI — Anthropic. Does that mean they have a deal with OpenAI to open up listings to OpenAI's agent? How does that work — how can you have something that OpenAI is allowed to do but Meta is not allowed to do? I think there's a very comfortable set of business practices that the major companies have settled into over the course of ten years, and I think that is going to break in the next year. It's gonna be knife fights all the way to the bottom.

   1:56:20

   Nathan Labenz: I'm still kind of surprised that they want to do it. I really don't — does it feel backward to you that Amazon would be trying to block shopping agents powered by other tech companies? I just have a hard time imagining that's the winning strategy for them. I know they wanna have their own and all that stuff, but I just don't see how this is a winning move.

   1:56:52

   Prakash Narayanan: So I think the question becomes: where is the customer touch point? Who owns the customer? Because whoever owns the customer gets to make the money — if you don't own the customer, you get commoditized. Amazon would prefer for people to come and directly interact with amazon.com, because then Amazon owns the customer relationship — they're the touchpoint. If you're not at that interface, you become a B2B supplier — you're supplying B2B listings, and once you become a protocol, your likelihood of making excess margin starts to drop. I think this is gonna be the big issue, because I think these AI agents are gonna be the universal interface — the universal user interface. And the question is: how deeply down the stack does that universal user interface plug into? Does it plug into amazon.com, or does it plug into the protocol that amazon.com's sellers are using — one step deeper? That would look like a Shopify — Shopify is a protocol that other sellers use to sell broadly, but Shopify itself does not own the customer directly. So those are the major questions in the sector. Amazon's advertising-to-consumers business is very profitable — Amazon makes, I think, around 10% of its revenue from it, and that's an 80% gross-margin business. Meanwhile retail is something like an 800-, 900-billion-dollar business for Amazon, with a margin of maybe 5%. So there are these questions about who gets to make the money by interfacing with the customer, and that's a tough one. And by the way, Muse looks like it's gonna be fantastic for advertising — absolutely stunningly fantastic — because people are gonna hand over their purchase decisions to Muse. Muse is gonna be making the purchase decisions on your behalf. It's gonna show you three options — the options it picked, and the option that's being sponsored. And the sponsor will also be using AI — Muse will tell them which two it's recommending on its own, and they'll be able to formulate their pitch to be on par with or supersede the ones being directly injected there. There's gonna be so much money to be made there. Oh my gosh — it's gonna be a permanent rent, much higher than in the search era, because search had ten blue links. Now you maybe have just one recommendation.

   1:59:56

   Nathan Labenz: I still have a hard time seeing it — I mean, I agree it creates all kinds of new issues for Amazon, and they're gonna need to be sharp on this. But I still just don't see why you'd rush to ban. First of all, it's gonna be a small percentage for the time being — I'd think there'd be a lot to learn from allowing Muse agents to come shop for a while. One thing that always kind of confuses me is why people don't keep the option value open longer. I've always thought this about the chip ban, the export controls, you know — if we're going super-exponential in chip build-out, then anytime you choose to pull the trigger, that still leaves the vast majority of chips in the future. When we look back on today from a 2030 perspective, we'll feel like, well, there weren't that many chips in 2026. So a ban that went from 2022 to 2026, with whatever fits and starts and enforcement gaps it had, is gonna be kind of inconsequential compared to one that was started today and runs the next four years, or even starts next year and runs for a few years after that. So I don't quite get why they wouldn't allow it — even if it is gonna be an exponential rise in Muse agents shopping, and even if that does threaten their advertising business, and who knows what other issues they might have — that's kind of the point. At the beginning, I would first just not want to turn customers away, and I'd also want to learn what I stand to learn from having these agents running amok while it's still kind of an early-adopter phenomenon. How are you gonna ban — I mean, Alexa has been kind of successful, but it's not like Alexa is killing it. If they had something where they could say, 'ours is better,' okay, that could be one thing, but they don't. So it just seems like they'd learn so much from having these traces. Even if they just want to build — I mean, everybody's talking about distilling, and what are the allowable and not-allowable ways to get data. One really easy way to get data would be: the agents come use your platform, you see how they use it, and then you can do all kinds of things downstream of that, I would think. It just feels shortsighted. I mean, we're not — obviously we're not Bezos here anymore, so I can't say never bet against Bezos, but I do think never bet against Amazon still remains reasonably good advice. And certainly Jassy ran AWS and was obviously super good at it. But this still just doesn't add up to me. I don't get why you would — you are — wait till it gets to 5% and then make a move.

   2:03:09

   Prakash Narayanan: Yeah, yeah, I think you're right — there must be some reason for moving this early. They probably should have maybe waited. So I suspect — since we're on September 21st, and OpenAI DevDay is coming up, and they've already kind of hinted that they have an agent coming — I suspect we're gonna see an OpenAI agent that'll be able to do your shopping for you, with a deal with Amazon and exclusivity, first-mover advantage with Amazon. Amazon's already signed such a deal, I'd guess. So I think this might not be a permanent thing — it might be temporary, for the next three months until this year's shopping season is over, and then they open it up. The reason being they'd have a preexisting deal where OpenAI would be the first shopping agent on Amazon — that would make sense because then there's revenue share to OpenAI, marketing spend with OpenAI, this whole rollout. That would make sense in terms of why they're doing it this early. You're right that it doesn't make sense this early otherwise. Absolutely.

   2:04:22

   Nathan Labenz: Yeah, okay, well, that's a good prediction. The other thing I wonder about is what equilibrium we're eventually gonna find ourselves in. Alan Chan is on tomorrow for an episode of The Cognitive Revolution — he's been a pioneer of agent-governance theory, all kinds of ideas around IDs for agents. How do we figure out: an agent presents itself — who does it represent? Is it legitimate or not? What permissions does it have? How do I decide, as the person on the other end of some agent request, whether to serve that request or not? I was looking into Cloudflare, and they also have this interesting thing where they're starting to ban. But, again, it just doesn't feel like the right solution to me, because what I then do in response is have the agent use my browser with all of my credentials. And you could — good reason — maybe detect when an agent is using the browser based on its bot-like behavior. But then you're really setting up an arms race — this feels analogous to putting too much pressure on the chain of thought. I don't want to put so much pressure on agents that people are out there devising ways to make them indistinguishable from humans. I suspect if people really work hard on that, they can probably succeed, or at least often enough that it becomes an issue. It seems much better to have a lane for agents — 'this is how they can work, we're not gonna try to block them' — and segregate them so we don't create this arms race. Because I really do think we're headed for a world where — probably Meta doesn't do it, but somebody's gonna figure out a way to make agents work on Amazon, somebody's gonna figure out a way to make agents look human enough that they don't get blocked by Cloudflare. And honestly, it probably works better for the user too. I have this problem all the time where Claude wants to do something in the browser — same thing happens with OpenClaw, Astra, whatever — they get stuck because when they open up their own window, they don't have sessions. I've given them a lot of passwords where they can log in, but not all, and sometimes they're blocked, and if there's a verification flow, a lot of times they can't do it. So the fallback just ends up being: okay, just do it on my browser, forget it. I wanted the security, I wanted the separation, but they're making it too difficult, so fine — just use my browser, then I'm logged in, you have all my credentials, you can do whatever. That's not great for me, and it's not great, I don't think, for anybody. So I do feel like we need a much better solution for all this than just ban-and-hope-for-the-best — I don't see that holding, and it feels like it also creates a lot of problematic incentives.

   2:07:44

   Prakash Narayanan: So I think it's like you're standing on Earth and you're watching this giant meteor head towards you, and you really don't know what to do — it's the whole 'Don't Look Up' thing. Because you can easily imagine Facebook scaling Muse to 300 million, 150 million American accounts, whatever — on an average-revenue-per-user basis on their existing network, it already makes sense. They've already hooked up WhatsApp, there's some hookup into Facebook Marketplace, some hookup into Facebook itself. There'll be some hookup into Facebook Messenger. If they go for it, that's gonna narrow down to one interface, and other companies won't be able to say no, because the Facebook traffic channel is so big — Facebook will be like, 'you either work with us, or we'll work with someone who is willing to work with us.' And the amount of agentic traffic that Amazon, for example, would have to support, or Cloudflare would have to support, would be maybe 100 times what they do today — just like what happened with GitHub — maybe 100 times or 1,000 times what they do today. They don't have the infra for it — they don't have the capability to serve that many HTML pages. They'd have to serve a lot of it by — there's so many things they would have to change. And I think that's a real challenge — everything changes. It's 1,000-to-1 agents to human traffic. Your primary user is now an agent, not a human anymore. The agent stuff has to work in parallel, it has to be very fast, you have to make it low-cost because you don't have the revenue on a per-user basis that you have for humans. So you're not gonna be able to support images, for example — you won't do images because it doesn't make sense. So many things have to change — the entire infrastructure of the internet would have to change to support agents, and agents would have to become first-class users of the — like what we spoke to the Mozilla guy about last week. Agents as first-class users is a whole different way of building the web. And I think there's a lot of, 'okay, what do we do with the ads?' That's how we make money — if the agents don't look at the ads, what do we do now? We know we've had bot traffic for a long time, and we've kind of assured advertisers that a lot of that traffic is actually real people, but now the cat is out of the bag. It was known there was something like 50 to 80% bot traffic for a lot of the ads, but at least it was in proportion — everyone had roughly the same, maybe 20 to 30% human and 70% bot, and advertisers were like, okay, that's the price you have to pay, everyone has the same issue. If everyone has the same issue, the productivity is the same — you're absorbing some of the bot traffic, but it's okay. But now you're at something like 9-to-1, or 999-to-1. It's no longer 'the cat is out of the bag' — things will have to change.

   2:11:27

   Nathan Labenz: Yeah, it's a tough one. I would love to get somebody from Amazon to come explain this thinking to us. In the meantime, we'll be unleashing our own agent — I'm pushed back onto the phone for audio purposes, but the latest from Dawn is that we're gonna set up an email for Dawn at our domain, and with that, outreach can begin, and we'll see if we can turn AI in the AM into some sort of business. It'll be interesting to follow along and see how much trouble Dawn gets us into.

   2:12:09

   Prakash Narayanan: It's definitely gonna be interesting. No, no — all the surprises will follow, I guess. And on that note, we'll be back on Wednesday. We have a full dance card — Wednesday, Friday, and next Monday. On Wednesday we're gonna get our first guest, who's gonna explain GPU forward pricing and financializing the entire GPU cycle, how that's working out. They have an index — it's a company called Orn, which I've been speaking to for a while. It's gonna be really fascinating, because there are people who don't believe these GPUs can be financialized in that way, and people who do. So it's a big debate in the industry — we'll get to see that.

   2:12:58

   Nathan Labenz: Yeah, and the other guest on Wednesday will be super interesting too — Lewis Hammond from the Cooperative AI Foundation has written some pioneering work, including one big technical report from early 2025 that was basically 'what to expect from multi-agent systems.' Very curious to hear what they've made of these crazy events we've seen this summer — how much of it they anticipated, how much they didn't, how they're updating, and what they expect to see next, as multi-agent systems, for better or worse, seem to be a new part of the reality we're living in, ready for it or not.

   2:13:40

   Prakash Narayanan: Indeed. So, on that note — Nathan, good morning.

   2:13:44

   Nathan Labenz: Good morning, Prakash. Talk to you on Wednesday.

   2:13:47

   Prakash Narayanan: Cheers.