How We Lose

“It’s unbelievable,” the CEO said. “As in, I literally don’t believe it. Model X can do all of that, but still can’t automate tasks that take our researchers a day?” The engineer nodded carefully. He’d been dreading this meeting ever since the numbers came in. “The topline results look bad. But they’re inherently noisy. And if you look at the later part of the report, you’ll see it’s by far our best model on almost every software task.” The CEO didn’t have to look. He’d had his model read the report and summarize it for him. “I know, I know. Best at web design, best at cybersecurity, best at systems. So why is it that on the one thing that matters, accelerating its own development, Model X is weaker than what we had last year?” “You really need to pay attention to the error bars,” the engineer said. “Error bars? Look at the math scores! Look at bioengineering! Writing! Conceptual reasoning! And you’re telling me the AI R&D results are swallowed by those teensy little error bars?” The CEO paced around the room, circling the engineer as a shark might circle a bleeding sailor on a raft made of dolphin carcasses. Some of the engineers felt squeamish about automating their own jobs. The