The Manifund team scores AI safety milestones

Scott Alexander posted a survey of utility weights for a bunch of different AI safety accomplishments, discussed more in Towards the Autograntor . The survey asked people to assign point values to AI safety milestones. The values were based on the anchor of 1 million points = a pause with China à la AI Futures Project’s Plan A, so a milestone assigned 10,000 points is 1% as good as that. Here’s a link to our survey responses. To be clear, we (especially Austin) didn’t spend that much time on this and have low confidence in our answers! But we think being transparent about even half-baked opinions is good. Headline numbers As examples, here are a few milestones where the two of us (and the autograntor reflecting Scott’s worldview) gave relatively similar values: Milestone Austin Caroline Scott Founding of effective altruism 100,000 75,000 100,000 Small-grant ecosystem (Long-Term Future Fund, SFF and similar) 20,000 20,000 30,000 Lighthaven 10,000 5,000 9,000 OpenAI or Anthropic becomes 1 point more alignment-pilled (on a scale where 0 = NVIDIA, 100 = Anthropic) 2,000 500 3,000 Austin’s and Caroline’s average ratings for alignment research and community-building milestones were simil