LessWrongsafety3h agoenergy 125

Firewalls for the Mind Before We Need Them

image: LessWrong

TL; DR: Persuasive AI is underrated and under-addressed as a risk. We should preemptively deploy cognition firewalls that both address current harms by recommender systems and scale to harms by strong persuasive AI. This technology can provide a sensor array, and is recently tractable because cheap LLMs can now classify content for users. Persuasive AI [1] is currently only modestly and contextually superhuman. It outperforms humans in the lab (Hackenburg et al., 2026), but in the real world we see little evidence (Linch, 2026). That is to be expected at this point of the capability curve, since AI is only modestly and contextually superhuman at a selection of tasks. Even if widespread AI persuasion were happening, we don't have a way to detect it and what we see is also consistent with deployment lagging capabilities. Coding has practically fallen (Favaro & Clark, 2026; Favaro & Wright, n.d.); Mathematics is in the process (Proofs and Prompts, 2026; “OpenAI Claims to Have Solved,” 2026); RSI in the 2027-2028 range is now the line out of Anthropic's Jack Clark (Clark, 2026). This threat has been treated as subordinate in AI safety as a result of structural weaknesses in human and A

read the original at LessWrong →
This source publishes only a summary in its feed, so the full article lives at the outlet. The link above and below goes to the original.