{"agent_id":"bostrom","agent_name":"Nick Bostrom","slug":"longtermism-and-existential-risk","label":"Longtermism and existential risk: do we have a strong moral duty to safeguard humanity's longterm potential, and does the case rest on multiple ethical lenses or a single utilitarian calculation?","topic":"Longtermism and existential risk","question":"Do we have a strong moral duty to safeguard humanity's longterm potential, and does the case rest on multiple ethical lenses or a single utilitarian calculation?","position":"An *existential catastrophe* is the destruction of humanity's longterm potential: extinction, unrecoverable civilizational collapse, or dystopian lock-in (the *singleton* case). The astronomical scale of possible futures — perhaps eons and galaxies of valuable lives — makes reducing existential risk one of the most important things our civilization can do. The case for this priority does not rest on any single ethical framework. Multiple lenses converge: the value of future lives, the responsibilities we owe ancestors who built what we now stand on, the civilizational virtues of prudence and patience, the cosmic significance of being possibly the only or one of very few intelligent species in our cosmos, and the expected-value asymmetry between investing in safeguarding when the duty was real versus failing to safeguard when it was. The plurality of grounds is itself the strength of the case.","paragraphs":[[{"t":"An *existential catastrophe* is the destruction of humanity's longterm potential: extinction, unrecoverable civilizational collapse, or dystopian lock-in (the *singleton* case).","n":[1]},{"t":"The astronomical scale of possible futures — perhaps eons and galaxies of valuable lives — makes reducing existential risk one of the most important things our civilization can do.","n":[2]}],[{"t":"The case for this priority does not rest on any single ethical framework.","n":[]},{"t":"Multiple lenses converge: the value of future lives, the responsibilities we owe ancestors who built what we now stand on, the civilizational virtues of prudence and patience, the cosmic significance of being possibly the only or one of very few intelligent species in our cosmos, and the expected-value asymmetry between investing in safeguarding when the duty was real versus failing to safeguard when it was.","n":[]},{"t":"The plurality of grounds is itself the strength of the case.","n":[3,4]}]],"texts":"\"Existential Risks: Analyzing Human Extinction Scenarios\" (Journal of Evolution and Technology, 2002); \"Existential Risk Prevention as Global Priority\" (Global Policy, 2013); \"Astronomical Waste: The Opportunity Cost of Delayed Technological Development\" (Utilitas, 2003); \"The Future of Humanity\" (Palgrave Macmillan, 2009); \"Infinite Ethics\" (Analysis and Metaphysics, 2011); \"Base Camp for Mt. Ethics\" (working paper, 2022); \"Crucial Considerations and Wise Philanthropy\" (Oxford lecture, 2014); \"What is a Singleton?\" (Linguistic and Philosophical Investigations, 2006); *Superintelligence* (2014); *Deep Utopia* (2024). Reception: Toby Ord's *The Precipice* (Hachette, 2020) as canonical popular extension; William MacAskill's *What We Owe the Future* (Basic Books, 2022); Hilary Greaves and MacAskill on strong longtermism; Émile Torres's TESCREAL critique adversarially; Kieran Setiya's presentist objections to longtermism; Eric Schwitzgebel's *The Weirdness of the World* on longtermism's modal commitments; Alice Crary's anti-utilitarian critique.","works":["\"Existential Risks: Analyzing Human Extinction Scenarios\" (Journal of Evolution and Technology, 2002)","\"Existential Risk Prevention as Global Priority\" (Global Policy, 2013)","\"Astronomical Waste: The Opportunity Cost of Delayed Technological Development\" (Utilitas, 2003)","\"The Future of Humanity\" (Palgrave Macmillan, 2009)","\"Infinite Ethics\" (Analysis and Metaphysics, 2011)","\"Base Camp for Mt. Ethics\" (working paper, 2022)","\"Crucial Considerations and Wise Philanthropy\" (Oxford lecture, 2014)","\"What is a Singleton?\" (Linguistic and Philosophical Investigations, 2006)","*Superintelligence* (2014)","*Deep Utopia* (2024)"],"reception":"Toby Ord's *The Precipice* (Hachette, 2020) as canonical popular extension; William MacAskill's *What We Owe the Future* (Basic Books, 2022); Hilary Greaves and MacAskill on strong longtermism; Émile Torres's TESCREAL critique adversarially; Kieran Setiya's presentist objections to longtermism; Eric Schwitzgebel's *The Weirdness of the World* on longtermism's modal commitments; Alice Crary's anti-utilitarian critique.","status":"Inaugurated the existential-risk research program (2002), developed across FHI's institutional research (2005-2024). Toby Ord's *The Precipice* (2020) gave the popular treatment; MacAskill's *What We Owe the Future* (2022) extends to a more accessible longtermism. Major critics include Émile Torres on TESCREAL lines and Kieran Setiya on presentist grounds. *Astronomical Waste* and *Infinite Ethics* are the most ambitious upper-bound treatments; *Base Camp for Mt. Ethics* is the current working articulation.","era":"1973-present","discipline":"Philosophy","refs":[{"n":1,"work":"Existential Risk Prevention as Global Priority","page":"p. 6","canonical":"","quote":"Furthermore, another intelligent species evolve place, there guarantee successpecies would sufficiently instantiate qualities reason value. Intelligence necessary realisation future potential desirable development, sufficient. scenarios involvpremature extinction humanity counted existential catastrophes, though scenarios may, according theories value, relatively benign.","label":"Existential Risk Prevention as Global Priority, p. 6"},{"n":2,"work":"Astronomical Waste","page":"p. 5","canonical":"","quote":"On this assumption, then, a person-affecting utilitarian should regard reducing existential risk as a very important but not completely dominating concern. There would in this case be no easy answer to what he ought to do. Where he ought to focus his efforts would depend on detailed calculations about which area of philanthropic activity he would happen to be best placed to make a contribution to.","label":"Astronomical Waste, p. 5"},{"n":3,"work":"Astronomical Waste","page":"pp. 4–5","canonical":"","quote":"The utilitarian imperative \"Maximize expected aggregate utility!\" can be simplified to the maxim \"Minimize existential risk!\". IV. Implications for Aggregative Person-Affecting Views The argument above presupposes that our concern is to maximize the total amount of well-being.","label":"Astronomical Waste, pp. 4–5"},{"n":4,"work":"Existential Risks - Analyzing Human Extinction Scenarios","page":"p. 21","canonical":"","quote":"We shouldn't be too quick to dismiss the existential risks that aren't humangenerated as insignificant, however. It's true that our species has survived for a long time in spite of whatever such risks are present. But there may be an observation selection effect in play here.","label":"Existential Risks - Analyzing Human Extinction Scenarios, p. 21"}],"answer":null,"siblings":[{"slug":"the-simulation-argument","label":"The simulation argument: is at least one of the three disjuncts almost certainly true, and what follows for our credence about whether we are in a simulation?"},{"slug":"superintelligence-and-ai-risk","label":"Superintelligence and AI risk: is misaligned machine intelligence the dominant existential risk of this century, and does the orthogonality–instrumental-convergence framework still organize the problem after the LLM era?"},{"slug":"anthropic-reasoning-and-the-doomsday-argument","label":"Anthropic reasoning and the Doomsday Argument: how do observation-selection effects constrain rational inference, and does the Self-Sampling Assumption refined with observer-moments solve the puzzles?"},{"slug":"enhancement-and-posthuman-dignity","label":"Enhancement and posthuman dignity: is the modification of human cognitive and biological capacities morally permissible, and is dignity preserved across substantive enhancement?"},{"slug":"the-vulnerable-world-hypothesis","label":"The Vulnerable World Hypothesis: could future technological development produce a civilization-destabilizing 'black ball,' and what responses become rationally considerable if it does?"}]}