{"agent_id":"bostrom","agent_name":"Nick Bostrom","slug":"the-vulnerable-world-hypothesis","label":"The Vulnerable World Hypothesis: could future technological development produce a civilization-destabilizing 'black ball,' and what responses become rationally considerable if it does?","topic":"The Vulnerable World Hypothesis","question":"Could future technological development produce a civilization-destabilizing 'black ball,' and what responses become rationally considerable if it does?","position":"Technological development is like drawing balls from an urn. Most balls are white (beneficial), some are gray (mixed), and there may be black balls — technologies that, once developed, by default destroy or severely damage the civilization that invents them. If a black ball exists in the urn, the orderly continuation of civilization may require either differential technology development (delay the black ball, hasten its safeguards), international coordination at unprecedented scope, or preventive global governance up to and including a kind of *singleton* — a world order capable of regulating potentially-civilization-destabilizing technologies. The hypothesis is a *conditional analysis*: *if* a black ball exists, *then* certain responses become rationally considerable that would otherwise be unacceptable. The hypothesis does not endorse the totalitarian response; it identifies the dilemma.","paragraphs":[[{"t":"Technological development is like drawing balls from an urn.","n":[]},{"t":"Most balls are white (beneficial), some are gray (mixed), and there may be black balls — technologies that, once developed, by default destroy or severely damage the civilization that invents them.","n":[]},{"t":"If a black ball exists in the urn, the orderly continuation of civilization may require either differential technology development (delay the black ball, hasten its safeguards), international coordination at unprecedented scope, or preventive global governance up to and including a kind of *singleton* — a world order capable of regulating potentially-civilization-destabilizing technologies.","n":[1,2]}],[{"t":"The hypothesis is a *conditional analysis*: *if* a black ball exists, *then* certain responses become rationally considerable that would otherwise be unacceptable.","n":[]},{"t":"The hypothesis does not endorse the totalitarian response; it identifies the dilemma.","n":[3]}]],"texts":"\"The Vulnerable World Hypothesis\" (Global Policy, 2019); \"Information Hazards: A Typology of Potential Harms from Knowledge\" (Review of Contemporary Philosophy, 2011); \"Existential Risk Prevention as Global Priority\" (Global Policy, 2013); \"Strategic Implications of Openness in AI Development\" (Global Policy, 2017); *Superintelligence* (Oxford UP, 2014); \"What is a Singleton?\" (Linguistic and Philosophical Investigations, 2006); \"The Unilateralist's Curse and the Case for a Principle of Conformity\" with Thomas Douglas & Anders Sandberg (Social Epistemology, 2016). Reception: civil-libertarian critics of the surveillance-state implications; classical-liberal skeptics of preventive global governance; Anders Sandberg's more pessimistic FHI-internal extensions; Yochai Benkler and Bruce Schneier on the surveillance dilemma; Allan Dafoe on AI governance.","works":["\"The Vulnerable World Hypothesis\" (Global Policy, 2019)","\"Information Hazards: A Typology of Potential Harms from Knowledge\" (Review of Contemporary Philosophy, 2011)","\"Existential Risk Prevention as Global Priority\" (Global Policy, 2013)","\"Strategic Implications of Openness in AI Development\" (Global Policy, 2017)","*Superintelligence* (Oxford UP, 2014)","\"What is a Singleton?\" (Linguistic and Philosophical Investigations, 2006)","\"The Unilateralist's Curse and the Case for a Principle of Conformity\" with Thomas Douglas & Anders Sandberg (Social Epistemology, 2016)"],"reception":"civil-libertarian critics of the surveillance-state implications; classical-liberal skeptics of preventive global governance; Anders Sandberg's more pessimistic FHI-internal extensions; Yochai Benkler and Bruce Schneier on the surveillance dilemma; Allan Dafoe on AI governance.","status":"Published 2019 in *Global Policy*. Has become the most controversial of the framework pieces because the conditional response set includes \"turnkey totalitarianism.\" Critics on civil-liberty grounds collapse the position to \"Bostrom wants surveillance state\"; the actual position is more carefully conditional. *Information Hazards* (2011) and *The Unilateralist's Curse* (with Douglas and Sandberg, 2016) are the methodological companions.","era":"1973-present","discipline":"Philosophy","refs":[{"n":1,"work":"The Vulnerable World Hypothesis","page":"pp. 1–2","canonical":"","quote":"Several counterfactual historical speculative future vulnerabilities analyzed arranged typology. general ability stabilize vulnerable world would require greatly amplifi capacities preventive policing global governance. vulnerable world hypothesis offers perspective which evaluate risk-benefi balance developments towards ubiquitous surveillance unipolar world order.","label":"The Vulnerable World Hypothesis, pp. 1–2"},{"n":2,"work":"The Vulnerable World Hypothesis","page":"p. 9","canonical":"","quote":"Retard development dangerous harmful technologies, especially raise level existential risk; accelerate development fi technologies, especially those reduce existential risks posed nature other technologies Bostrom, 2002). principle differential technological development compatible plausible forms technological determinism. example, ordained technolodeveloped developed, still matter developed.","label":"The Vulnerable World Hypothesis, p. 9"},{"n":3,"work":"The Vulnerable World Hypothesis","page":"p. 3","canonical":"","quote":"Glass production could banned, existing glass panes fiscated; pieces glass would remain scattered across landscape time. Batteries magnets could seized, though people would stashed these materials before could collected authorities. cities would destroyed nihilists, extortionists, revanchists, ' would hap- '. People would fl urban areas.","label":"The Vulnerable World Hypothesis, p. 3"}],"answer":null,"siblings":[{"slug":"the-simulation-argument","label":"The simulation argument: is at least one of the three disjuncts almost certainly true, and what follows for our credence about whether we are in a simulation?"},{"slug":"superintelligence-and-ai-risk","label":"Superintelligence and AI risk: is misaligned machine intelligence the dominant existential risk of this century, and does the orthogonality–instrumental-convergence framework still organize the problem after the LLM era?"},{"slug":"longtermism-and-existential-risk","label":"Longtermism and existential risk: do we have a strong moral duty to safeguard humanity's longterm potential, and does the case rest on multiple ethical lenses or a single utilitarian calculation?"},{"slug":"anthropic-reasoning-and-the-doomsday-argument","label":"Anthropic reasoning and the Doomsday Argument: how do observation-selection effects constrain rational inference, and does the Self-Sampling Assumption refined with observer-moments solve the puzzles?"},{"slug":"enhancement-and-posthuman-dignity","label":"Enhancement and posthuman dignity: is the modification of human cognitive and biological capacities morally permissible, and is dignity preserved across substantive enhancement?"}]}