Some “coherence arguments” are two different things in a trenchcoat

tl;dr I go through three “coherence arguments”, related to (1) market competition, (2) AI self-reflection and self-modification, and (3) AI training algorithms. I argue that each of these implies a pressure for (something like) “being self-consistent”, and also a pressure for (something like) “only caring about long-term eventual outcomes, with ruthless indifference to anything else”. But the latter pressure tends to be weaker and/or more contingent than the former. So it's bad to lump them together and call them by the same name. 1. An example “coherence argument” that I’ll focus on for most of the post I’ll talk about two other “coherence arguments” at the end (§6), but let’s start with the one based on market competition: In a stock market, people with circular preferences will systematically transfer money to those without circular preferences. For example, if I think I’m winning every time I trade A for B, and B for C, and C for A, then someone can just siphon off as much money as they like from me by repeatedly offering me deals that each look good to me in isolation, but in aggregate comprise a Dutch book against me. Now, that’s the case of circular preferences, but the core