What do LLMs Really Learn About Model Weights? A Stratification Approach to Examine Weight Space Programming

TLDR I attempted to replicate “Sparks of Weight-Space Programming” by Takehi et al (2026) with stratification on two reasoning models, GPT 6 Astra and Deepseek-V4.1-Flash. The task for the LLMs is to identify the game a ML model is trained for giving the weight of the model (using exact same prompt and weight defined by Takehi). Although both of these models do well on the surface (4/4 correct for Astra and 3/4 correct for Flash), the stratification approach exposes the difference in mechanism between these two models in approaching the problem. The experiment result shows that Flash depends a lot on channel label for the answer (while Astra's performance does not degrade when channel label does not present). Additionally, Astra is able to identify the game with good accuracy using only the first layer of weight or as few as 1000 weight values selected at random out of 17253 weight values (Flash's performance degrade substantially on these). So Astra survives this first round of stratification tests just for now. Comparing the results between Astra and Flash, it is obvious that Flash is relying heavily on prompt metadata instead of reading the weights. Background On November 2 an a