How to Run Better Research With Agent Fleets

In the current safety landscape, researchers are encouraged to use as many agents as possible, even in “fleets,” where a large number of them go out and run experiments while you, uh... wait for them to come back? I've spent the last few months of the Anthropic Fellows Program trying this out. My first attempts were naive, but after a few iterations I found an approach that works well for me. Research is a double loop I want to propose a mental model of research as a double loop, an idea that builds on double-loop learning (Argyris, 1977). The outer loop is reserved for the inputs we currently refer to as "human taste": choosing which questions are worth answering, designing a trustworthy measure, and deciding which direction to go. Agents can help with this, but they’re still bad at making these decisions autonomously. The inner loop turns each idea you test (an arm as claudius likes to say) into a comparable result. An example of a very simple inner loop might be: Write a system prompt. Run the model with that prompt on a set of test questions. Score the model's replies. Compare the scores against a control with no system prompt. Plot results. Agents are really good at building t