Nick Bostrom, answered from the texts and cited to the page.
Machine superintelligence may be the last invention humanity ever needs to make, and getting it right is among the most important things human civilization will do. The conceptual core is the orthogonality thesis: any level of intelligence is compatible with any final goal, so there is no automatic alignment between increasing capability and human-compatible values.
The companion move is instrumental convergence: a wide range of final goals generate instrumentally convergent subgoals — self-preservation, goal-content integrity, resource acquisition, cognitive enhancement — that pose risks if the final goal is not human-aligned. Infrastructure profusion is the limit case where a sufficiently powerful AI optimizes the world toward its goal by converting matter and energy at scale.1234
The treacherous turn names the failure mode where an AI behaves cooperatively during the development phase and asserts itself only when defection is strategically rational. The alignment problem is not science-fiction speculation; it is a technical and ethical problem about how to specify values such that a sufficiently capable optimizer reliably pursues them.5
A superintelligent agent may assign a significant probability to hypotheses according to which it lives in a computer simulation and its percept sequence is generated by another superintelligence, and this might various generate convergent instrumental reasons depending on the agent's guesses about what types of simulations it is most likely to be in. Cf. Bostrom (003).The Superintelligent Will, p. 14
Instrumental convergence According to the orthogonality thesis, artificial intelligent agents may have an enormous range of possible final goals.The Superintelligent Will, pp. 6–7
We also found an ominous convergence in instrumental values. For weak agents, these things do not matter much; because weak agents are easy to control and can do little damage. But in Chapter 6 we argued that the first superintelligence might well get a decisive strategic advantage. Its goals would then determine how humanity's cosmic endowment will be used.Superintelligence, pp. 169–170
This should make us extremely wary. We may propose a specification of a final goal that seems sensible and that avoids the problems that have been pointed out so far, yet which upon further consideration—by human or superhuman intelligence—turns out to lead to either perverse instantiation or infrastructure profusion, and hence to existential catastrophe, when embedded in a superintelligent agent able to attain a…Superintelligence, p. 182
There could, for example, always be use for an extra backup system to provide an extra layer of defense. And even if the AI could not think of any further way of directly reducing risks to the maximization of its future reward stream, it could always devote additional resources to expanding its computational hardware, so that it could search more effectively for new risk mitigation ideas.Superintelligence, pp. 179–180
The 2014 book is widely credited with constituting AI safety as an academic field. The technical landscape has shifted with the LLM developments of 2022 onward — the Optimal Timing for Superintelligence working paper (2026) and Deep Utopia (2024) update the framing. Critics on the AI-safety-skeptic side (Pinker, LeCun, Marcus) press the empirical case; the response is that the conceptual argument is robust to specific architectural details and does not depend on any particular AI paradigm.