Gods of straight lines on graphs strike Starburst. I am shocked and puzzled.

LLM Intelligence, measured by Starburst, has been basically linear over time since that start of the reasoning era. By Chapin Lenthall-Cleary and Cole Gaboriault In summer 2024, my friend Cole and I started a hobby research program into perceptions of (human) intelligence. We started by creating a bespoke 1-5 intelligence scale [1] and rating people we knew. Our plan was to then make a bunch of tests, give them to the people we rated, and see which tests, if any, correlated with our ratings. One of those tests was Starburst. We’ve talked about it before ; in short, it’s a puzzle game where one tries to figure out the laws of physics of a fictional universe. We never did get very many people to play it. But using the results of two early playthroughs (Cole’s and our friend Ryan’s), some further test playthroughs where we pretended to be people of certain intelligences, some a priori reasoning about which parts of the game would be difficult, and a healthy dose of raw intuition, we pieced together a correspondence between Starburst scores and our intelligence scale. Then Ryan suggested we give Starburst to LLMs. Here's a plot of SOTA [2] model intelligence over time, measured using t