The Neuron Basis: Goodfire's Curt Tigges on Agentic Interp, Monosemanticity & Reverse Engineering

This transcript is automatically generated; we strive for accuracy, but errors in wording or speaker identification may occur. Please verify key details when needed.

Introduction

[00:00] Hello, and welcome back to The Cognitive Revolution. Today, my guest is Curt Tigges, member of technical staff at Goodfire, and with Nick Cammarata, co-author of the new paper, "Neurons as an Interpretable Basis," the more academic and more LLM-focused companion to Nick's recent viral essay, "Neural Networks Are Already Clean." This work directly challenges what had been the received wisdom, that neural network internals are such a tangled mess and that individual neurons are so frequently reused across disparate concepts that we can't realistically hope to decode them directly and instead need helper models, such as sparse autoencoders, to make sense of their internal states. Curt's position, as you'll hear, is that individual neurons are in fact much more directly interpretable than previously believed, that most neurons, in fact, have one main function, that these functions are often better thought of as verbs and control flow gates than as entity recognizers, and critically, that LLM agents are now capable of grinding through a network and describing each neuron's role with remarkable accuracy. We begin with a detailed overview of the LLM they study, Qwen 3.6-35B-A3B, a 35-billion parameter model containing 5 million MLP neurons, which are spread across 40 layers, each of which has 256 experts containing just 512 neurons each. At every layer, 8 experts are active in any single forward pass. We linger on the highly expressive architecture of modern SwiGLU MLPs, which provide 2 distinct sets of signals to each neuron, one linear and one gated by the SiLU activation function. This specific MLP architecture was proposed as a transformer improvement by Noam Shazeer in 2020, in a five-page paper where he famously wrote that, quote, "We offer no explanation as to why these architectures seem to work. We attribute their success, as all else, to divine benevolence." Six years later, I'm struck by how similar this structure is to some of the gated architectures we've seen in Mamba and other highly expressive state space models. And my own intuition is that these parallel paths allow the model to simultaneously gather semantically relevant information and effectively filter it for relevance. Interestingly, because the activation function does allow for negative values to be transmitted, each neuron can have different meanings when activated in its positive and negative directions. From there, we move on to the techniques that Curt used to support the idea that a model's large activations are all you need for solid performance. The key concept is sparsity. In short, Curt shows that you can ablate roughly 90% of the individual neurons, keeping only those with the strongest activations with surprisingly minimal performance degradation. In some cases, such as on the ARC-AGI puzzles, such intensive ablation actually improved performance. And while it did cause the model to struggle on reasoning tasks, the authors do show that a very small amount of additional post-training on the sparsely activating model is enough to recover a large majority of the performance loss. Having established that the model's weaker activations matter less than most had believed, we move on to discuss the Neuron Scientist, the LLM agent that they use to scale the work of characterizing individual neurons. Given a corpus of text and relevant interpretability tools, a frontier model generates and then iteratively refines paragraph-length hypotheses that attempt to describe what each neuron does. These hypotheses in turn are validated by asking another model to indicate based on the description where in a given bit of text that specific neuron should be expected to fire, and then showing that the original model's behavior is substantially restored relative to ablating the neuron entirely when the other LLM's guesses are patched in to the model's activations. Along the way, we also touch on monitoring as the first and most obvious application of this work, the small circuits called neuron communities they identify and begin to explore, the fact that individual experts within the mixture of experts architecture are also more interpretable than previously discussed, how this work relates to the current consensus understanding of how models store and recall facts, as well as the roles of the residual stream and attention mechanisms, the possibility that small activations may matter much more in long context scenarios, the path from here to ambitious reverse engineering, how we should be thinking about scaling laws for interpretability, and what, if anything, this work suggests we might ought to do differently in the training phase in order to produce more readily interpretable models in the future. The bottom line for me is that this work does constitute a significant step forward in interpretability. Where much of the field previously assumed that helper models were required to disentangle model internals, there is now good reason to believe that we can study neurons directly in a highly scalable way. At the same time, whether such low level understanding can be effectively laddered up to higher order understanding of circuits and behaviors remains to be seen. There is much more work to be done, and while I totally understand how hungry people are for hope after a summer of so much bad news about our prospects for alignment through behavioral training alone, we should be very careful, as Curt is throughout this conversation, not to jump to unwarranted conclusions. Interpretability remains unsolved. With that, I hope you enjoyed this deep dive into the surprising legibility of individual neurons with Curt Tigges, member of technical staff at Goodfire.

Main Episode

[05:55] Nathan Labenz: Kurt Tigas, member of technical staff at Goodfire. Welcome to the cognitive revolution.

[06:00] Curt Tigges: Thanks, Nathan. It's great to be on the podcast.

[06:03] Nathan Labenz: I'm excited for this. This is the latest in a long line of Goodfire mechanistic interpretability episodes, and in some ways, might be the biggest one yet. Occasion here is that you have worked closely with Nick Camerata, who recently made some big waves in the AI space with his essay, neural networks are already clean. And, now we've got the follow-up paper, which goes deeper especially into the language model side of this and quantifies it in a variety of different ways and and brings some new techniques to bear on the problem as well. And so I'm really excited to go into it and and really get into the the nitty gritty details. But maybe before we do that, just to kind of get us started, I would be really interested to hear... Because I think one thing that's been interesting about this particular essay is that more so than most papers, there's like a sense that there's... That you wanna change people's minds, you know, that you're kind of trying to induce a a paradigm shift with this work. So maybe just lay out, like, what is the core argument that you're making or the core takeaway that you want people to take on board and and use to inform their work, and then we'll proceed to unpack.

[07:18] Curt Tigges: So the core takeaway of what we're... Our experiments and what our writing tries to convey is that, basically, the individual neurons in MLPs in neural networks in general are more interpretable than the field has thought to this point. They are very difficult to interpret. They're complicated objects. They were very confusing in the initial, like, days of interp. And because of that, I think it became not very common to study them in detail. There always has been a group of people, like, trying to do some interp on individual neurons. But, yeah, I think it just was seen as a very difficult thing and that this net representations would be distributed against across many neurons rather than localized in individual ones. So our message is that we can understand them, that you often don't need very many active at once, and that even when a neuron has more than one meaning, which can be very confusing, it's usually not, like, an unbounded number of hundreds and hundreds of meanings that are just, like, very difficult to understand. Rather, it's like a compact set of of meanings. And, yeah, that's the main message.

[08:45] Nathan Labenz: One detail from the essay that I would like to understand better was the idea... Basically, I I would summarize it as they actually used to be, like, a year... A few years ago when a lot of this work was originally done and hypotheses were formed, that maybe the neurons at that time were in fact less interpretable because of somewhat less good training techniques. And so there's been kind of a improvement in the underlying substrate. Like, the problem isn't maybe gotten easier in some ways, and we weren't, like, looking because we'd kind of thought that this was hard and given up on this line of inquiry. What has changed? What... How would you describe the... It's... You know, I feel like most things have drifted toward being like less interpretable. You know, we're worried about chain of thought not being faithful or not being monitorable. Why has this gotten easier?

[09:41] Curt Tigges: Well, there's a confluence of two different things that has made this easier, I think. One of them is that we have better tools available. So we have agents that can, like, comb through all of the situations where a neuron activates and just, like, look for patterns and, kind of extract these and give us, like, a compact set of descriptions of what the neuron is doing. So we have, like, one such agent that we've scaffolded the neuron scientist who, will go through these and provide an analysis. We also have better ways of, like, looking at the individual circuits that go into and out of a neuron to trace and, like, quantify these behaviors. But another thing I think is that you're right that, like, a lot of the earlier networks were kind of difficult to interpret. They were messier, certainly smaller, and this might be going a bit into the weeds there. But, also, I think that there is an effect with, the activation functions. So ReLU is the most well known one. But in the networks that we've looked at so far, the activation functions, ReLU and GELU are harder to interpret, and then SWIGLU becomes easier to interpret probably because it allows for, like, a, sort of, like... It's like a little circuit that's done... Able to do some logic. And so that opens possibilities for greater interpretability.

[11:12] Nathan Labenz: I'm certainly no stranger to trying to get agents to do all my work, so I I empathize very much with that approach. I... Let's take it one minute also and just orient ourselves on the model that is kind of the core object of study in the paper. In the in the essay, there was a lot more stuff on vision models of various kinds. Here we're talking more language model. So the model that you studied the most is Quinn 3.635 b a three b, which folks will know is a mixture of experts with 3,000,000,000 active parameters at any given time. Can we just kind of describe that thing in a little bit more and potentially tedious detail? Like how many layers does it have? I I read that there's 5,000,000 neurons, but I'd I'd just kinda like to get a little bit of a sense of the layout of spread across how many layers, how many experts in a layer within an expert, you know, what is the sort of width and depth of a... Of an individual expert. And then I I do wanna spend a little bit of time too on the two signals that feed into each neuron and and how we should understand those. I honestly think that is something that I have not understood as deeply as I probably should have before this conversation.

[12:26] Curt Tigges: The structure of a gated MLP is probably not something that a whole lot of people are familiar with. So, Quinn, as you said, it's a mixture of experts model, and it has 40 layers. Each layer has about 130,000 total neurons, if I'm remembering my numbers correctly, and those are divided into 256 experts for each layer. So at each layer, the router will decide to send the input to eight out of those 256 experts. And so it'll just, like, go through the experts one at a time as a little subnetwork of the entire model. So this means that, you know, only a few of the model's neurons are active at any one time just because, you know, only eight out of 256. So each expert has 512 neurons, and those are always on to some degree, usually just a tiny bit, but some of them will be... Will have strong activations. And those are the ones that we look at in particular. So some other interesting things about this model, it's worth knowing that even though we have 5,000,000 neurons, there are effectively 10,000,000 things that the neurons can express because each neuron has a positive and a negative activation, and those can have different meanings. And they're read geometrically in a different way by the model. So it... Sometimes if you have one neuron that's about a topic or one pole will be about a topic, and the negative poll will be about a different topic. Sometimes they're related, but there's a lot of variety there. So other reasons why we chose this model, it performs quite well on benchmarks, and it also has a form of attention called a gated delta net, which, it has regular attention, but then that's only one fourth of the attention layers. The three fourths are these gated delta nets. And this model just has a lot of features that are also present in, like, the the biggest open source models, the frontier open source models. So that made it a good subject for our investigation. Now when it comes to the SWIGLU activations or the the individual neurons and what you're alluding to before with the two paths into the the neuron. Basically, you can think of each neuron as having an up projection and then a gate projection. So instead of... Input will come into the neuron layer, and then it will go through two... Those two sets of parameters instead of just one. And those are two, like, parameter matrices that are kept completely separate so they can recognize different things. So what happens in practice is that the gate matrix will detect some aspect of the input, and then the up matrix, the up projection, will detect different thing about the input. And then once they do their computations, the result is multiplied together. So that allows for you to have, like, a conjunction where it's basically... If you see... You know, maybe the gate covers, like, context, and then the up projection covers a specific topic topic about the context. Like, maybe the context is cooking, and then the up projection recognizes an ingredient like garlic. And then when they're multiplied by each other, it activates extra strongly.

[16:20] Nathan Labenz: So maybe... I don't know if you know, like, the intellectual history of this very well, but, like, the sort of traditional MLP that I think most people imagine when they think of an MLP is basically just rows of neurons where each one has a parameter that says how strong its activation will be multiplied by as it connects to each of the different positions in the next layer. Right? That is vanilla. That's kind of what I've become accustomed to thinking about. When did this more complicated mechanism from layer to layer start to become the norm? Do we know why it became the norm? And is there anything more that I should be just kind of... Like sometimes with with the attention mechanism, you know, we've got like our keys, queries, value, matrices, and we have kind of a story as to like what each one of those is doing. You know, this is saying this token has this information. It's looking for this information. Now having found that, it's gonna load this other kind of information. Do we have a story around that here that is like principled that led to its design, or are we kind of posthocking that story based on the fact that this just kinda seems to work?

[17:37] Curt Tigges: To be honest, I'm not sure of the exact history of the development of this. But the larger family, the gated MLP family that includes, SwigLU, but also includes, Red Glue... Well, it's a little bit different, but it uses ReLU instead of in the middle. These have been around at least since the Lama 3.1 family. So that's been a few years now.

[18:02] Nathan Labenz: More than two years.

[18:04] Curt Tigges: Yeah. Yeah. And probably predating that. But right now, the vast majority of modern language models use gated MLPs, and among those, most of them are SWGLU. They just perform better in practice, and I think the people who developed it weren't thinking about interpretability at all. Just it worked better. Probably because it's more expressive. You know, you get twice as many input parameters to the MLP. We do have a post hoc story about how it allows the network to function, and we go into this in the paper a little bit as well, which is that if you have this combination of gate and up, it means that you can switch contexts more easily. So for example, this is... If you have a neuron that has two meanings, then you'll probably want it to be read by two sets of downstream readers and... So that each group of downstream readers will interpret the neuron differently. That way you won't get your signals mixed up. For example, there's a neuron where the one meaning is oral health and the other meaning is marine wildlife. And these are very different concepts. Right? In practice, when the neuron has one meaning, the set of readers is completely different. When it has the other meaning, the set of readers is, again, separate. And what we see is that the context of the text that's sent to the model will activate these downstream readers in advance so that they'll be prepared to, like, receive the target neuron signal. So it's like you have a switching function, basically. So this just allows more combinations of things to happen in the network.

[20:03] Nathan Labenz: If we zoom in on one particular neuron, it's getting now basically two signals from every neuron position in the layer before it. One is the sort of simple signal of just like the strength of that firing times a parameter that says, you know, in a linear way, here's how strong you should fire based on my contribution. And then there's this additional parameter, correct me if I get anything wrong, I probably will. Again, comes from every neuron in the layer before, but now it's the strength of the activation is put... Is multiplied by a parameter and put through this gate function, which is the SWIGLU or the SILU or could be a variety of different ones, but you said SILU. I see SILU in the paper, but you said SILU is like the most commonly used, so maybe you can deconfuse me a little bit on that. But it's not necessarily the case where we we shouldn't think of it necessarily, I guess, as the... In fact, I'll I'll state state it the other way. We probably should think of it as the gates coming from different places than the signals because if the gates and and the signals came from the same place, they'd be they'd be inseparable. And the whole point of this is that we have a neuron being told when to be active and then also kind of how strong to be active, but coming from different places so that this can... These two signals can reach it independently, and that independence is where we get this greater expressivity. How am I doing?

[21:45] Curt Tigges: Yeah. So that's basically right. It is... It does get a little messy because the strength of the activation can come from both the up projection and from the gate. So to... Also to answer one of your earlier questions, so the the gated MLP as a whole is called SwigLU, this particular design. That includes the up projection, the gate projection, and the decoder direction. Then inside of that, the gate is multiplied by the activation function, which is silo. If you had regLU instead of swigLU, then the gate would be multiplied by a ReLU activation. And what's... The silo does is allow the gate to be, like, a little bit negative sometimes, but mostly positive, whereas ReLU in that place would only be, like, only positive. But, yes, we do indeed see that they often have somewhat disjoint sets of things that activate them. Sometimes there's some overlap, but usually it's quite separate, which allows for that expressivity.

[22:57] Nathan Labenz: Gotcha. Okay. So one big maneuver that plays out a few different ways in this paper that's really core to all the findings is sparsity. In a few different ways, you sort of look at how sparse can we make this thing and still get, you know, pretty similar results. One thing that I would love to understand is kind of when we move things to sparse, I'm not entirely clear on what that always means. Like, the simple way to think about it would be like turn them all to zero, but I I take away that it's actually a little more complicated than that. Like, there's some sort of baseline average activation that's non zero that is considered to be like this... That sort of default or neutral state when a neuron is ablated or, you know, essentially turned off. But turned off doesn't mean going all the way to zero. Help me understand that a little bit better.

Sponsor

[23:03] Claude : Claude is the AI collaborator built for problem solvers, extending your thinking across coding, modeling, and strategic workflows. Get started and discover Claude Pro at https://claude.ai/tcr

[24:21] OutSystems : OutSystems is the leading agentic systems platform that enables enterprises to engineer, orchestrate, and govern AI applications on a single unified platform. Learn more and see how it works at https://outsystems.com/tcr

Main Episode

[27:14] Curt Tigges: Yeah. So it's a bit tricky in practice because when an expert is activated, all of its neurons are on a little bit. You know? There is no true zero for SWIGLU or SILU. When we do the sparse experiments, there's a variety of things we do for... Where we... In some cases, we ablate everything except for the extreme activations that we've measured precisely for a given neuron. And then when we do that, we set the other values of all the other neurons to their average. So it's a mean ablation. Now this mean ablation in practice is actually quite close to zero because the neuron is bipolar. That's both positive and negative. The average just ends up being, like, very, very close to zero or small. Another kind of ablation we do where we just select the top k neurons from each expert, for example, top 64 neurons from an expert, in this case, those are the only signals we allow to pass through. The other items, we do not, they don't have any signal, so they're effectively zero.

[28:24] Nathan Labenz: So is is much at stake there, like, in the scenario where you use the average if you just literally set them all to zero, meaning, like, there was no contribution from that neuron to the next layer, would that be... Would that, like, yield dramatically different results, or is it

[28:41] Curt Tigges: not a big difference in practice? So in practice, when we do

[28:45] Nathan Labenz: this,

[28:46] Curt Tigges: the model performs quite well still on a number of benchmarks, just basically equivalent to the original model. So when we initially do this, then reasoning does suffer. So the models will start going into loops sometimes. It'll, like, be unconfident about its answer and just, like, self check too many times. But with a little bit of fine tuning, we can correct that, and the model, like, can perform as well or even better in the benchmarks that we tested in this ablated state.

[29:20] Nathan Labenz: Yeah. So that's really weird. I definitely wanted to try to get some intuition around that. So just to... One one more fine point of clarification. So there's these different ablation techniques. Top 64 is I think where the, like, eighty eight percent number comes from basically. Right? And then you also have... And then you looked at top 32, and it seems like there's significantly more degradation of performance when you go from top 64 to top 32. But then there's also this like percentile thing where you ablate like... Because it's bipolar, I understand you're taking like the middle. Right? Saying that the strongest positive, strongest negative will keep, but will ablate like the 10 to 90 percentile or the five to 95. Why is it why is it meaningfully different to have top 64 versus just keeping the bottom and and top five percentile? Is that like... It seems like it's almost two ways of saying the same thing, but there must be something more there that's causing those to be, like, substantially different to the point where you're even bothering to report them separately in the paper.

[30:28] Curt Tigges: Each form of ablation has kind of a different meaning to the model. One thing we noticed is that, if you rank on single neurons activations or, the places where it activates by these percentiles, like p 75, p 90, p 95, and so forth, different neurons start to descend into noise at different levels. So one neuron might be meaningful all the way down to p 75, and another neuron might be meaningful just down to p 99. And this can also differ by layer. So sometimes just a strict ablation based on percentiles doesn't always make sense or might not be principled. But another way that you can do it is if you do the top top n, top k, top 64, for example, then what this says is given that you've activated this expert, what are the most important things in the expert? And this kind of lets the model decide in a way what's important rather than just, like, the pure percentiles. And the percentiles can be kind of dependent on the dataset as well. But by letting the model do that, we get a little better performance. Practice, what this turns into is the vast majority of things that activate are indeed already above p 95 on the positive side or below p five on the negative side, but it'll include, like, a handful of neurons that are, like, you know, in t 90 or something like that. They're a little bit weaker.

[32:12] Nathan Labenz: So is it... Is the core difference that the percentile based thing is happening model wide versus the top k is happening at the per... On a per expert basis, or is the percentile also per expert? The percentile is

[32:27] Curt Tigges: actually per neuron. So it's like, given this neuron's historical activation range, you ablate based on that individual neuron's value for p 90 or whatever the case may be.

[32:41] Nathan Labenz: But that's,

[32:42] Curt Tigges: like,

[32:42] Nathan Labenz: global in

[32:42] Curt Tigges: the in the context of all 5,000,000

[32:45] Nathan Labenz: neurons?

[32:45] Curt Tigges: Yeah. All at once.

[32:47] Nathan Labenz: Gotcha. So the the top k is sort of like a stratified approach where you're gonna ensure that ex... On an expert by expert level, you keep like a a similar amount of activity present.

[33:03] Curt Tigges: Yeah. Yeah. Exactly. Yeah. Because maybe you have an expert where most of the neurons have... Still have some kind of meaning below, you know, p 95. Maybe they still... Maybe all of the neurons in that expert have a meaning at, like, p 90, and it's, like, very clear and still useful. And then you have another expert where nothing is usable below p 99. And so in such cases, you'd want to let each expert decide what the threshold should be.

[33:37] Nathan Labenz: Can you maybe give a little more of the intuition behind the language you're using there around, like, meaning down to, you know, could be p 99, could be down to p 90? Because I was thinking of it maybe wrongly, but kind of in a reverse way where I was thinking like the most important neurons would be the ones that would make the more stringent filters. Right? Like the the 25 to 75, you're cutting half of the neurons out. The 10 to 90, you're cutting eighty percent of the neurons out. As you push this middle band wider, you're left with presumably like the most important neurons. But I think you're saying something different, which is like something could be in a high percentile and not that important, whereas something else could be in a lower percentile and still important. And that's why it's worthwhile to go down to the per expert level. Do I have that right?

[34:29] Curt Tigges: Yeah. Essentially. So to be clear, the variance is quite small. It's almost always, like, the ex... Extreme activations that are the most meaningful and most the most causally relevant to the model's computation. So to give some intuition about how we see this, we collect a whole bunch of examples where the neuron activates at every percentile, like, the top 1,000 and then p 99 dot 99 and so on, just, like, down these ranks all across the activation range. And then we study where the, like, where the coherence stops. So we just, like, read the examples and study them and see, like, sometimes it dissolves into noise at a certain point, but it's always in the nineties pretty much. It's just like, you know, sometimes it's going to be p 95. Sometimes it's gonna be p 99 or something. But it's also worth noting that the quantiles themselves do depend on the dataset. So we use a pretty large mixed dataset, but it's not as large as the model's training distribution, which was, I believe, something like several trillion tokens. We use 10,000,000,000 tokens to extract these examples.

[35:54] Nathan Labenz: So as I'm looking at the performance table, what you're saying makes sense as I look at p 25 to 90 or p 25 to 75. So the middle half of neurons that are least extreme in their activations being ablated, you scroll down that column and look at all the different performance results on all these different benchmarks and there's like not that much change. So from that, you could say at a very high level, like you can cut out the half of neurons that are least extremely activated and basically not much goes wrong. And even interestingly on the reasoning stuff, like you don't see much damage there either. Then you go more extreme, cut the middle 80%, and then cut the middle 90%, and the impact of performance becomes more significant. So we're kind of pushing like where does this curve get steep in activation extremity space and trying to find like how how far we can go and still have things like mostly work. I'm really interested to get your intuition for how is it that some of these things do better. It's notably across the top. You've got four different arc AGI sections, and the middle, you know, 50% doesn't move it much, but then moving... Removing 80, removing 90% of neurons, you actually get a little bit of a bump in performance. Do you have any account of why that is happening?

[37:32] Curt Tigges: So some of these benchmarks will care about how quickly the model reaches an answer. So, like, if it reasons for too long before it puts out an answer, then then it counts as, a failure. Basically, if you cut out a lot of the stuff inside, the model becomes more confident in those benchmarks where it performs better. It reaches the answer more quickly, and I suppose some of the benchmarks that you mentioned don't involve reasoning. But the ones that don't involve reasoning is just because the model is more confident. It doesn't have as much internally to distract itself from the right answer, and the most confident thing rises to the top. In the case of reasoning, then with our fine tuning, we see the same thing or just reaches the answer that the original model knew just, like, faster. Interesting. Less to distract itself. That's fascinating. So so far, just to kind of put a signpost in the ground here, we have talked through in somewhat excruciating detail the structure of the network, what it means to ablate neurons, and we've established that

[38:43] Nathan Labenz: you can ablate a significant majority of neurons up to 90% or 88 if you take the top k 64 and still get surprisingly non degraded performance, at least on these like non reasoning tasks. Okay. Now the next question though of course would be like, well, it's one thing to say only the extreme ones matter. It's another one to say, can we interpret what's going on? So maybe walk me through the process that the neuron scientist goes through. You already kind of described like you have a big corpus, you collect a lot of activations, you can rank them, you can, you know, put different thresholds in place. Now how do we turn that massive blob of activation data into 5,000,000 stories about what each individual neuron in a whole big 35 b three... A three b network is actually doing?

Sponsor

[39:07] Tasklet : Tasklet empowers your business with AI agents that connect to your tools and automate recurring workflows with no code required. Visit https://tasklet.ai and use code cog rev for $50 in free credits

Main Episode

[41:31] Curt Tigges: Great question. So I like talking about the neuron scientist because I think it's a pretty cool agent, basically, and we put a lot of work into, like, trying to improve it. But... So as you said, we basically collect these, thousands of examples of prompts where this, neuron becomes active, and then we group them. We cluster them, by similarity. We have a few ways of doing this. One of them is you can look at the input connections and group them by... Or cluster them by that. You can also look at the, just the text itself and what other neurons activate and cluster them by what other act... What other things activate it. And in practice, that gives you, like, different contexts where the neuron becomes active. But the central thing is that we just get an initial grouping, and then we hand these examples and the groups to the agent, the neuron scientist. So the neuron scientist's job is to come up with a hypothesis that unifies these different examples where the neuron activates and then predicts where it'll activate in the future. So sometimes that will be one meeting. Like, that's more common. Sometimes it'll be multiple meanings, and there's different patterns, like different words that neurons prefer to activate on, different contexts, different, concepts. We give the neuron scientists a lot of, latitude in which kind of hypothesis it comes up with, but we give it a bunch of tools as well. So it can, create example sentences or modify example sentences and send them to the model and then get the result of how much the neuron activated on them. It can run other experiments, like, two very similar sentences with, like, one minor difference and then see, like, what the difference in... Is an activation there. It can do steering. It can do, basically, all... This... All the same things that an interpretability researcher can do. So it'll come up with an initial hypothesis and then propose this, and then we will test

[43:48] Nathan Labenz: it

[43:49] Curt Tigges: objectively by... With another system that is separate from the agent so that the other system will just basically see, okay. Based on this description of the neuron, can we

[44:01] Nathan Labenz: predict

[44:02] Curt Tigges: where, in this example text, the neuron is going to fire and by how much? And this will happen for, like, a few texts. And then if it fails, if the description just isn't good, then it'll come back to the neuroscientist with feedback. It will see how it needs to adjust its hypothesis, improve it a little bit, send it back, and so forth. And then at the end, a helper agent will create a page that basically shows us the results of what the hypothesis is. So the beginning of that sounds pretty similar to auto labeling of SAE features. I think people have seen that in the past, right, where there's like, okay, here's a bunch of passages that cause this particular SAE feature to light up. Like, hey, model, what do you think that appears to be? And then where we've taken this farther now is we're not just asking that question, taking the answer and calling it a day, but rather saying

[45:09] Nathan Labenz: burn some tokens to really like dial in your explanation, test it, and refine it and, you know, let's really try to make this good. Because obviously a lot of SAE features are deemed uninterpretable and certainly, like, honestly, I have looked at, like, lists of passages that make SAE features fired, some of which were considered uninterpretable, others which were considered interpretable. And at times I've been like, I don't know, dude. Like, you're saying that's what this means? I mean, it's not like really jumping out at me that that's like really what this means. So, like, what do you see the agent doing as it goes down this path? Could you tell a story of one where it was like not obvious at first, but through use of tools, it was able to kind of get to insight as to what this particular neuron meant?

[45:59] Curt Tigges: Yeah. For sure. So, yeah, a lot of the auto labeling stuff, the auto interpret stuff that's used for SAEs and that we first used for neurons, when the function of the thing is obvious, it will tell us what's going on. But when it's not obvious, it'll usually be some vacuous label like this fires on words in the context of medical discussions, and that's just not very useful. That doesn't teach us much at all. So one thing that we have noticed is that as we apply increasing levels of intelligence, then more and more of what the neuron is doing becomes legible and predictable. And we have a graph of that in our paper where, like, the auto interpret stuff is the lowest. And then as we get better agents and they're allowed to do more work, then we get better predictions. And the neuron scientists usually will use a frontier model, one of... Like, whichever the latest one that seems to perform well is. So one example of a neuron that seemed uninterpretable at first, but then as we applied more intelligence, became more interpretable. There was a a neuron where if you just look at the examples and just, like, the 50 token window where it fires, it seemed, like, completely uninterpretable. It was always... I mean, it would just be some phrase that could be across the field of cooking or instructions or, like, legal documents or anything, really. And you couldn't really tell what it was. There was just nothing unifying the patterns. But with a lot of investigation using agents, we were able to identify that this actually fires. The neuron fires when it's the second item in a list

[48:05] Nathan Labenz: where

[48:06] Curt Tigges: contents of the list had been introduced in a different format a long way before in the text, maybe a thousand tokens before in the text, but this was being recapitulated here in this place in a different format. So, like, maybe before it was a commented or comma based list, and now it's, like, one, two, three, four or something. So this is, a structural feature that you

[48:29] Nathan Labenz: wouldn't

[48:30] Curt Tigges: necessarily think to look for but was present. And there's a lot of features like that where the idea isn't obvious at first, but that's just just one example.

[48:46] Nathan Labenz: So the... I thought this was pretty creative, and I I... Correct me if I'm wrong, but I think all this technique would be pretty equally applicable to SAEs. Like, there's nothing that would prevent you from giving an agent a similar set of tools around SAEs and allowing it to tinker and, you know, change text and rerun and all that that same kind of thing. Right? So that's an interesting innovation regardless of, you know, which lens we're using to look at things. So let me see if I can get this right. So the the neuron scientist kind of iterates for a while. It does it does its experiments. It does counterfactuals. It maybe, you know, expands scope to look at longer bits of text to try to see these like bigger picture patterns that weren't apparent in the very, you know, excerpted text that we're accustomed to seeing. And it eventually settles on basically what sounds like a paragraph length description of what the neuron is. Okay. Now how do we validate that that in fact is what it is? The answer is give it to another model from I guess like a different model maker or, you know, different family of models and say, okay. Your task is to take this description of a neuron, look at this text, go token by token, and tell us for each token how strongly that neuron will fire. And then based on the strength of those neurons, you actually like patch its answers into the model. Now we're back to the original model, the model being interpreted, and say, how well does the model perform if instead of its actual natural activations, we force the predicted activations from this other model that was based on the combination of the description and the text at hand. And the answer there is we recover 74% of the KL divergence loss, which I'd like to understand a little bit better. Like, I'm not... I don't have great intuition. I don't see in, like, KL divergence color, I don't think. Maybe you do. So tell me, did I get anything wrong about the process? Is there anything else we should understand about the process? And then how should we understand that 74% recovery bottom line out of that?

[51:09] Curt Tigges: So that's essentially correct. Yeah. So the, the neuron scientist comes up with a paragraph length description, of the neuron, highlights, you know, what sorts of tokens it fires on as well as the, context and the concept overall. So this is one of our evaluation methods. It's currently our primary one. And, as you said, we first ablate the neuron completely from the passage, and then this other model will say, okay. Based on the description, out of a score... Out of zero to nine, how strongly will this neuron fire? And then it labels each token in the sequence with that. Most of them are zeros, but a lot of them are gonna be high. And then as you say, we patch those into the model. And then we see how much of the damage from deleting the neuron entirely was recovered by patching it in to these places at these values. So this works pretty well. And the... A model that we use right now is JEV, so it's, like, very different from the model that generated the prediction. And this gives us an idea of how much of the damage is recovered. And we use, as you mentioned, KL divergence. So this basically says, at each step for each token in the sequence, the model is predicting... It has a prediction for every token in its vocabulary, each of the logits in its vocabulary. And so that's a distribution of predictions over all of those tokens. And the KL divergence is best understood as, like, the difference between the distribution when it's damaged versus the distribution when it's patched in or repaired. And so that distribution of predictions over tokens basically shows how much damage we've caused to the model, the model's ability to predict the next token. And we measure that for each of... Like, we combine it over the entire sequence. So at each token position, we have that value, and then we just, like, measure it over the whole sequence that the model's predicting. Now something that's worth noting is that we do weight it by damage. So when we say 74% damage recovery, that means that cases where the neuron basically didn't matter or it mattered very, very, very little get a lower weight than cases where the neuron mattered a whole lot to the model's prediction. So that's the the basic setting. And this is, our primary evaluation, but we have a couple of others that we use too that are worth mentioning. So a very intuitive one is we say... We give another model the description, and we say, based on this description of the neuron, make a text, write a text that's going to activate the neuron more than anything in the dataset. Like, we've given... We've run a 10,000,000,000 token dataset through the model and collected these examples. Let me say this other model, exceed that. We don't give it any examples, so it has nothing to work from except for the description. And then it just, like, iterates until it's produced a sentence that will activate the neuron more than anything in the dataset. And then other metrics that we might use are, we say, given this sequence, can you tell me whether this neuron should activate in this sequence at all? Is it relevant at all? And at this one, you know, we often see rates in the the nineties because it's an easier task than token by token prediction. So this all adds up to we should have some, you know, reasonably high level of confidence that, like, the process by which the paragraph length

[55:11] Nathan Labenz: description of the neuron is created is, like, working. Because if it wasn't working, we wouldn't expect to be able to recover this damage. We wouldn't expect to be able to get these predictions right. How do I... How now do we, you know, now we we start to get into the interpretation phase of this. Right? So now I'm looking at a paragraph length description. It's still a little bit in the eye of the beholder. Looks like a paragraph can obviously contain two different concepts or related concepts. We have the question of, like, how do we decide to draw boundaries between concepts? So, like, how do you think about modest romanticity coming out of this? Like, what counts as one concept? I definitely wanna hear your thoughts around the nouns verse verbs thing. You know, maybe give me a little more of, a qualitative, intuition for, like, what are these what are these paragraph length... I'm sure you've read, you know, thousands of them. What what is the nature of these paragraph length descriptions?

[56:07] Curt Tigges: They can vary widely between neurons. We see neurons doing so many different things. As you said, sometimes they'll contain multiple ideas. Like, this neuron will activate, you know, on oral health and in... Environments usually on the point at which someone decides to get oral surgery at... Or at the point at which a specific animal in the marine environment was described, the description will read something like this. Most neurons do have something of a central concept that unifies them. But as I've looked at, you know, thousands of neuron descriptions and what they actually tend to represent, I think I've, like, come to a somewhat different philosophical perception of things than simple monosemanticity or polysemanticity. So, like, monosemanticity or just, like, semanticity in general is this idea that a a neuron or something else like a a feature represents a single thing, a single concept. It's just detecting whether that's there.

[57:21] Nathan Labenz: But

[57:22] Curt Tigges: in practice, I think it ends up much more complex. So each neuron can connect to thousands of other neurons upstream, and they... That provides a lot of information to this neuron. One thing that you might see is a neuron that responds to, like, discussions of insects, for example. It'll fire specifically on three types of things, the food that insects prefer to eat, the habitats they prefer to live in, and the places where they prefer to lay their eggs. Now is this three concepts, or is it one concept? Kind of hard to say because it's not firing on everything about insects. It's not firing on every preference of insects. It's just firing on, like, those three subtopics within the insect sphere. Now a neuron that fires on, you know, musical instruments like banjos or legal documents about alimony, that's probably going to be... That's very easy to see. Like, okay. This is two different things. Right? Another problem that we see also with the concept of human recognition of monosemanticity or polysemanticity is that we don't always know why the model's putting things together. There's a... An example, like, let's say you have a neuron that fires on clothing. It fires on exercise equipment, and it fires on dishes and on tablecloths. Now seems very polysemantic. Right? Like, several different meanings. But what if the concept that the model has is things you might find at a garage sale? Then suddenly those things are unified,

[59:18] Nathan Labenz: but

[59:18] Curt Tigges: you you might never see it from the outside. So for me, the important question about mono and polysemanticity is less about is a thing strictly monosemantic? Is it strictly polysemantic? It's more, can you elucidate all of the things that the neuron represents? And sometimes you might want to zoom in and find... Like, looks like a single concept. You zoom in. You find a few concepts inside, but then you just list them. Right? You just... You know what they are. You elaborate over that set of things. And when those things are disjoint, when they're, like, totally different from each other, then you just list that as well. And, the interesting thing about this also from a mechanistic perspective is that we can see the signatures of these concepts in the weights themselves, which means, like, you can basically find among the thousands of neurons that are connected to a neuron, a subset that goes to the neuron when it's discussing one topic and a subset for another topic, and you can just kind of analyze thousands of examples and pull out these subcircuits in this way. And then that gives you, you know, this perspective of, like, how the model actually has these primitives inside of its own mind.

[1:00:44] Nathan Labenz: So as you've looked at these circuits, do you have kind of a simple story around like, I mean, this sounds silly to to say, but, like, kind of how it works? Like, I mean, what what sort of motifs or, you know, moves stand out as sort of the, you know, notable primitives that could give us some intuition. I mean, I guess this gets to, you know, one of the big questions, of course, with all of this is like, at what point do we get to the ambitious reverse engineering phase of all this? And I think that's been one of the things that has been, like, most discussed about this essay and and now the paper. I'm sure we'll generate a lot of similar discussion. To what degree do we now have a better story of... It's one thing to say, okay, we can kind of say, actually, look, these neurons do have a pretty interpretable thing that they do individually, but how does that fit into context? How does that start to turn into behavior? Like, how do we aggregate that into, you know, real understanding? How how far are we on that? Do you have any kind of story that you can share that gives us a little bit better intuition around, again, like, how it works as opposed to just like what an individual neuron means?

[1:02:08] Curt Tigges: Well, some great questions in there. I have.

[1:02:10] Nathan Labenz: Only the only the big ones Yeah. For

[1:02:16] Curt Tigges: So first of all, I I think the shift from noun to verb can be seen in a couple of different ways. But one way that I like to think of it is this is less of a detection of a single feature or object and more like the neuron the neuron is doing something. Right? It's not just detecting a thing. It's like sending a signal that is going to induce the model to do something else. So in that sense, it is a verb, but also it detects, basically, shifts of what's going on in the text or what things are happening and shifting towards a different kind of generation of text. I guess that... That's how I think of the noun versus verb distinction, although Nick might think of it slightly differently, but I think it's quite similar between our thoughts. So let me just... Let me echo that back to you real quick. So the the verb is like...

[1:03:23] Nathan Labenz: With respect to the model, the verb is like this neuron sends a signal downstream that activates a certain set of circuits. And presumably, some are still very, like, noun ish, but these sort of... These are, like, control flow nodes that we can describe as, like, if this, then that kind of operations. But but we should understand their effect as not, like, representing something, but sort of directing traffic or routing energy or activation through the model to the right places downstream to be used.

[1:04:03] Curt Tigges: Exactly. Yeah. They're not just, like, static. Like, each layer just detects a certain set of concepts, and now these concepts are there in the model. They are, like you said, control flow. They direct the model towards specific things or away from specific things, and we can see this with, some interesting neurons that respond to, reasoning in particular. So there's this whole class of neurons that, they fire on and direct the model to be more or less confident in its reasoning or to do things like wait or to, like, recheck or to, like, continue along a specific direction or pivot to a different approach. There's a lot of reasoning specific ones that are just really interesting in this way, and these are very much like verb and steerable type neurons. There are others that are like, oh, this number is wrong or this one's, like, off by one or, like... Yeah. Just like these sorts of, operations. Relating to a topic of ambitious reverse engineering, I think... So, certainly, there's more than just neurons to reverse engineering. We need to understand attention and other things in the model. The residual stream in particular is, like, a really complicated thing to unpack. But I think understanding neurons and the specific circuits and subcircuits they instantiate is pretty critical to that. Now you can understand every single neuron in the model and just have a big list of what they all represent and what they all do and not have an ambitiously reverse engineered model. You need to see how they operate together. And turning individual neurons into understanding requires, like, basically, understanding circuits. And you can get circuits through a variety of ways. Like, you can run millions of prompts through the model and gather attribution graphs for them using one of the existing methods and then see, like, what the average...

[1:06:50] Curt Tigges: Just, like, pick out these communities of neurons from there. That's one thing that we mentioned in our paper. You can also see what this circuit is for a specific prompt. To turn this into understanding, then you need to find the right ones and then see how they impact behavior. So from this lowest layer, we need to go, like, slightly up this tree of abstractions and find, like, a sub circuit for, like, all these little pieces of knowledge that the model has and for all of its, like, little decisions, proclivities. Maybe you find, like, how it decides to use a particular Python package or how it decides to decide which painters are impressionist versus post impressionist or figure out, like, which capital belongs to which state. So we actually have done some work in there too, and we see how this sort of knowledge is represented. The... Like, for example, for the impressionist example, there's a community of, I think, just few dozen neurons that represent this idea of which painters are impressionist, neo impressionist, or post impressionist. And then the neurons representing individual artists are linked to the tokens for those movements, and it's all, like, just a simple little mechanism that will decide this topic. So I think we have to work our way up the tree of abstraction and just kind of compile this huge list of things at the neuron level, at the sub neuron level where we see the individual circuits for the different meanings that a neuron could have, and then up towards neuron communities and super circuits and so on that just do these given tasks. Now I think there's two things you have to do. One of them is you want to do this for the model in general at least a few times. Right? You want to understand the full model, all of the little circuits, all of the pieces of understanding that you can. If you understand the neurons and you have the right primitives to, like, put into these slots, you can say, okay. Now that we have a neuron community and we know what the neurons are, we can say what it's doing. So going really wide and getting this huge collection of things is important, but you also need to do things that are, like, really specific to your alignment and safety objectives too. So that means doing a lot of experiments with datasets that elicit specific safety relevant properties, like reward hacking behavior or scheming, planning, or whatever the case may be. You need to see how the communities affect this behavior and, like, how you can steer it in one way or another. And you need to see, yeah, basically what fires when those things are happening.

[1:09:38] Nathan Labenz: I have a bunch more specific questions too, but, like, where do you think we are on this story? I mean, when I hear a little circuit that, you know, determines what wave of impressionism, you know, pre or post or whatever a particular artist is in, I'm like, okay. That's something. You know, it's a result. There would be, you know, a zillion of those. Right? So in a say... In a sense, we could, you know, map out a bazillion of those things and then kind of still be sort of in the same place that we're at with respect to the neurons themselves where it's like, okay. Well, now we've got, you know, an equally long list or maybe even a longer list of, like, micro circuits that do things. How does this ladder up into, like, a holistic understanding? Do you feel like you see that path from here? Because I think this is one of the things that people sort of initially were like, oh my god, you know, they've reduced interpretability to a normal science now. We can just grind the agents away forever and we'll like have a pretty good understanding of how the models work. And then other people were kinda like, I don't know. You know, we can, like, have a... As you said, like, long list of neuron descriptions or maybe, you know, an even longer list of these micro circuits that doesn't really tell us, like, macro scopically, what's it gonna do next? Or, you know, what's gonna do if I put it in this situation it's never seen before. Right? Like, do... Yeah. I mean, I guess out of out of distribution generalization is always, a critical question. Do we feel like there's any traction on all this labeling, all this stuff? Like, can we now make any better predictions about what will happen when you put a a model into a novel situation? Well, a lot

[1:11:26] Curt Tigges: of the neuron stuff is new enough that we have just begun to do this. There's a lot of interesting work online inspired by our first essay on this where one user found a single neuron that fires on reward hacking text. So, you know, that neuron could be used for monitoring potentially, at least for that subtype of reward hacking, maybe not all of reward hacking.

[1:11:54] Nathan Labenz: There's

[1:11:54] Curt Tigges: also a lot of interesting work on persona creation and steering with neurons, and that's surprisingly practical. It seems like a lot of neurons just affect how the model responds and its personality. But getting back to your question about having this huge collection of microcircus, huge collection of neurons, where does this get us? Like, how does it, like, actually turn into larger level primitives? I think to some extent, we haven't yet figured out what those higher level primitives are. Very much like how... You know, we have physics, we have chemistry, and we have, biology. And, theoretically, you could describe everything in physics or in, biology in terms of physics, but you would never want to because it's too too granular, too detailed, and you don't need to because there are higher level loss that have been discovered at the chemical level and then at the bio... Biological level. So I think for neural networks, we still need to find these laws and the science of the higher levels of abstraction. But I think by understanding the lower level stuff better, we can more quickly come to these these next levels of abstraction and also validate them and, like, see cases where they might not apply. So you could have a persona steering vector, right, which I think, you know, we've seen working this before from Anthropic and others, where it's just a vector that can steer the model's persona in different directions. We can understand that in a neuron basis by just seeing, like, what neuron circuits lead to this persona or which of them are important to it? Which of them, if you ablate them, will cause problems with it? Are there any neurons that would, like, sabotage it if it got a particular piece of input where it, like, breaks the persona and cause it to do something different? Like, how... So many so many models will, like, sometimes just snap and do something really crazy, like, one out of 10,000 times or something like that. If you want to debug why the model does things like this, then I think neurons are a good place to look. Not necessarily guaranteed to have the answer there, but it'll be useful. But... Okay. So I see our place being somewhere, like, we're discovering these primitives, like molecules that are important to the model's behavior. But we need to understand alloys. We need to understand fluid dynamics. We need to understand these higher levels of science. So if you have the neurons, you can model all of these things better. I see it also as a process that's happening from two ends. One of them is the lower level, get all the little pieces, get all the little components, see how they work. From the top down, we also see just, like, finding these vectors, like, difference of means vectors or steering vectors that do these, like, big things. We don't necessarily understand how they work mechanistically. Or you do some training, and that also is very top down. But you start with a lower level. You start with a higher level, and you see... You keep working until they converge. So build higher level abstractions, more and more of these until you reach the level where you see what's actually happening. When you train a model to do x, like, what changes at these levels that makes it want to do a reward hacking, that makes it eval aware, that makes it, you know, better at solving problems, that makes it more cooperative, and so on. Yeah. So, basically, we have a lot of work left to do, but I think neurons are a big piece of that, and they will enable all the work that we wanna do.