Scaling Discovery with Shangyin Tan - Weaviate Podcast #149!

Shangyin Tan, a Ph.D. student at U.C. Berkeley joins the Weaviate Podcast to discuss Scaling Discovery Requires Forgetting. This paper explores why long-running AI optimizers stall and how to get them climbing again. The conversation opens by defining discovery through the lens of GEPA, the prompt optimizer that uses an LLM as the proposer, and its successor, Optimize Anything, which extends the idea to code, GPU kernels, and agent harnesses. Shangyin separates three kinds of problems: generalization (prompts and harnesses that must work on unseen inputs), single-problem optimization (one kernel where the score you see is the score you get), and multi-problem optimization in between.From there, the discussion dives into forgetting. Running GEPA, Karpathy-style autoresearch, and Meta Harness behind one front end, every backend hit the same plateau after a couple of hours, and handing the best candidate to a fresh optimizer with no other context unblocked it every time. The evidence is striking: as sessions grow, agents use fewer thinking tokens per turn, edit fewer lines, and drift toward tweaking numbers instead of trying new algorithms, with one model repeatedly declaring it had "reached the practical limit" right before finding another improvement. And the best candidate isn't a loss of history, since it already encodes the ideas from every iteration that built it.The conversation then moves into multi-agent teams, where agents communicate only by reading each other's Git diffs rather than full files, so a weaker agent borrows an idea without collapsing into a copy. Teams climb faster and forgetting prevents plateaus, together making a strong autoresearch harness. Shangyin and Connor also cover GEPA's Pareto frontier as a tool for diversity, routing inputs to specialized harnesses, the Fork and Flush paper's idea basins, just-in-time databases, and why expensive or undefinable evaluations are the biggest gap between autoresearch and production.The episode closes with the mismanaged genius hypothesis: frontier models are already incredibly smart, and the real work is building harnesses that stop getting in their way!