Podcast: Kimi K3.1 Pro, the new cyber reality, and Muse musings

We — Nathan and Florian — wanted to discuss all the messy trade-offs of a busy week in AI and ask all the pressing questions like: Why did Kimi K3.1 Pro have a mixed reaction with such incredible benchmarks? Will it be adopted? Were we wrong about the risks of open-weight models and cybersecurity? Why do cyber experts discuss recent events so differently than AI experts? What comes next in the few months? What will change our minds? We will write more about this subject, but wanted to bring you behind the curtain as we aspire to quickly learn more about the cybersecurity industry and trends across the last few years that have been leading us today. We conclude the podcast with some lighter topics, from Muse-maxing to excitement in still hiring junior technical talent. Please send us questions and topics, as we plan to record more of these podcasts going forward. Share Listen on Apple Podcasts , Spotify , and wherever you get your podcasts . For other Interconnects podcasts, go here . YouTube ↓ Chapters 00:00 Kimi K3.1 Reaction 08:15 Kimi’s open-weight cyber principles 11:23 You can hack everything era 15:54 PHP2Shell, LiteLLM, and the new internet 24:00 Abliterated APIs, safeguards, and trusted access 31:41 Anthropic not hiring juniors employees 36:16 Florian’s Muse addiction 45:30 Wrap-up Links we discuss My thread with Dean Ball about the “boy who cried wolf” story of the frontier labs Kimi K3.1 Pro announcement blog Kimi K3.1 Pro on Artificial Analysis Intelligence Index Kimi K3.1 on Artificial Analysis defensive cyber benchmarks Program Bench Open-weight Mythos would’ve been fine take PHP “hasn’t caused a disaster yet” update Abliteration.ai serving abliterated models API Morph serving abliterated DeepSeek via API Joshua Saxe’s Substack on AI and cybersecurity Azeem Azhar, “The internet’s Wuhan moment” Anthropic, “How employment is changing at Anthropic” LiteLLM security incident report Transcript 00:00:00 Nathan Lambert: We are back with another Interconnects podcast. We are with Florian, eval guy, recently moved to SF. Florian, is Kimi K3.1 Pro , name is a mouthful, going to live up to the benchmark scores, for example on Artificial Analysis , or where is it going to land? 00:00:17 Florian Brand: In the past, the Kimi models always lived up to the expectations, but this release is kind of weird compared to the others. I think it is also partially because they are becoming more professional, and the previous model drops were kind of out of nowhere. This time, which also got some criticism, they launched a bigger social media campaign, which allegedly also involved paying people to post about the model without disclosing. 00:00:56 Nathan Lambert: I think a lot of companies do this and tell you not to mention it, but it’s a mess. It’s bad form. No company should do it. Essentially, one person tweeted a screenshot of the press instructions for Kimi that said, “Don’t say this is a paid promotion.” And to my knowledge, no one has corroborated it after that. 00:01:15 Florian Brand: Yeah, and there are right now a few of these model launches, including by American companies, that are just very obviously astroturfed. We have seen launches where it’s thousands of retweets and a few hundred likes, because it works so well to get the attention. Some of them disclose that it is a paid ad while others don’t. But in general, we have seen the professionalization of model launches over time, and pretty quickly. A year ago, everyone just dropped the model, then patched vLLM afterwards, so no one was actually able to run any of the open models for a while. Now you get all the inference frameworks on day one, you get all the inference providers on day one. Or, which is also what we see more and more, the model creator has a few weeks of exclusive access before they release the weights. And that’s also what’s happening with K3.1 Pro. I think it’s two weeks of exclusive access before they release the model weights. 00:02:38 Nathan Lambert: They said the weights would come out by November 4th. And the headline things... I don’t know if I’m going to be able to share screen. I don’t have this set up on this computer. But the headline is this Artificial Analysis number where it’s ahead of GPT-6 Astra. And then if you look at the Kimi K3.1 blog, they have their headline numbers, which are really strong, and then they also have some numbers later on which seem a bit more jagged. I think it was Terminal-Bench Science , which is 20 points behind GPT-6 Astra. I think there were a few others, like ProgramBench . Do you know Terminal-Bench Science and ProgramBench? Are any of these things that people think are important benchmarks for frontier models? Or how do you explain the fact that on Artificial Analysis the model is top, what, three, four? It’s the two latest Claude 5.5 models, Muse Wonder Full, and then Kimi K3.1, which looks crazy. It just looks like totally crazy scores, and I think everybody is still surprised by it. But then there’s this long tail of details that you use to evaluate a model now. I had posted a thread on Twitter of, “What are people’s impressions?” And there’s a lot of people that are like, “This seems not great,” and then some people that are like, “Yeah, it’s fine.” I don’t know, there’s a mixed bag. Relative to K3, which I think was pretty unanimously, this feels like a big model and the post-training seemed a little bit undercooked for that. Maybe this is just a tiny bit RL fried. We’ll see where we land. 00:04:26 Florian Brand: For the benchmarks, one of the things is that the benchmarks that are right now used by Artificial Analysis, they will post their new version probably very soon after we release this podcast. 00:04:43 Nathan Lambert: We can’t tell you the exact date, but both of us know. 00:04:48 Florian Brand: In general, the Artificial Analysis benchmarks are rather short in terms of time horizon, whereas something like ProgramBench is really long and also very explorative. Other benchmarks are telling the model, “All right, this is the goal, and I want you to reach it,” whereas ProgramBench is more like, “All right, here’s what I want, but you need to explore a lot.” And I think that might be the reason that we see some differences: their RL training is still more towards the exploitative side of model training versus the explorative side, where you need to find the solution somehow before you actually know what you’re doing. 00:05:41 Nathan Lambert: But if you’re using a three trillion parameter model, I feel like you kind of want it to be pretty robust at exploration. Because if you’re going to want to use a model for pretty known coding tasks, like using DeepSeek-V4.1-Flash-1017 , which is not that far behind on these scores, it’s just going to be way faster. It might not be wildly different on cost per task with the actual token numbers. That’s why I think it’s pretty reasonable criticism of this Kimi release: a big model has got to be... Because it’s going to be slower, it’s got to be like Fable, which is just... I still think Fable is really nice if you’re doing very blurry things. I revert to it when checking my slides over sometimes, where I just feel like it kind of gets me a bit better, where Opus is really, really grindy. I think the open-weight models are better in this kind of grindy perspective, because you can just get them to be faster and cheaper. 00:06:44 Nathan Lambert: It seems like Kimi is gearing up for this Flash-style release with their naming change. Previous Kimi models were just K2, K2.6, K3, and now this is K3.1 Pro. So a lot of the company forecasting, investment-y side of interest on Twitter was, a lot hinges on this for Kimi, because it’s kind of true. We’re not the inference experts, but serving the model seems like you’ve got to commit way more resources to serve this three trillion versus this 700 billion parameter model. You’re going to need NVL72 racks, I’m guessing, to do it well. 00:07:21 Florian