Subtitled: Existential Risk and the Future of Humanity
(Hachette, 2020, 468pp, including 225pp of resources, acknowledgements, appendices, further reading, notes, bibliography, and index.)
Here is book about existential risks, a topic on which I have several books, though this one and a couple other recent ones address the subject in the context of Effective Altruism. That is, part of EA would be to decide how to direct resources to avoid future calamities, with the usual cost/benefit considerations. Ord, along with MacAskill and Singer and many others, were mentioned in the Rutger Bregman book I read recently (review here). This one seems more specifically focused on risks.
*
Key Points
- Author defines the “precipice” as the moment in history when it became possible for the human race to destroy itself, and places that moment at the explosion of the first atomic bomb in 1945. (And it roughly coincides with the beginning of the Anthropocene.)
- This book is about existential risks, which he defines rather precisely. Not just garden-variety catastrophes, from which a small population of humans might eventually recover.
- He places the current situation in the context of humanity as being some 200,000 years old, as having gone through three great transitions, the agricultural revolution some 10,000ya, the scientific revolution, only about 400ya, and the industrial revolution, a couple hundred years ago. Each brought problems, but average human conditions are better now than they’ve ever been in history.
- And, there are potentially centuries ahead for humanity, given the average mammalian species lasting about a million years. If we leave the earth, millions or billions more years are available to us.
- So it’s imperative now to avoid risks while we can take action to avoid them.
- How existential risks are different from extinction or the collapse of civilization. He recalls Jonathan Schell, and invokes the term “longtermism.” This entails the economic concept of ‘discounting’, which places relative values on people alive today vs all those billions who may be alive in the future.
- Then he steps through various natural risks: supervolcanoes, stellar explosions, the death of the sun; anthropogenic risks: nuclear weapons, climate change, environmental damage including resource depletion; and then future risks: pandemics, AI, various dystopian scenarios such as totalitarianism.
- Part Three starts quantifying things, and comes up with a 1 in 2 change that humanity will survive to achieve its potential. Then considers how to prioritize risks, considering importance, tractability, and neglectedness.
- Then a general strategy for humanity. Reach existential security; conduct a “long reflection”; and achieve our potential. The middle step is in MacAskill too. How expanding into space would or would not solve anything. What role a world government would have. What technological progress can do. With examples of what not to do, and what we can do right now.
- The final chapter contemplates our potential: how long humanity might endure; some comments about scale; the quality of human life; and the choices we can make along three dimensions: time, space, and experience.
*
Detailed Notes
Part One: The Stakes
Introduction.
Humanity is about 200,000 years old, but earth will remain habitable for millions of future generations. We’re easily distracted by local crises. This book is about safeguarding humanity’s long-term future. We have the capacity to destroy ourselves. We must manage risks of today and avert those of tomorrow…
This capacity arose only in the past century. One episode was on October 27, 1962, when a nuclear war was almost started. Cuban Missile Crisis. Only because one officer refused to approve an order did a sub not launch its nuclear weapons. But we can’t rely on such luck forever.
This book is about existential risks. Of extinction, of unrecoverable collapse, of dystopia. It will delve into a variety of subjects, from physics … to moral philosophy and economics. Relying on many experts. Discusses his own background. Effective altruism. Now focused on the future. Not just the familiar risks. And not a pessimistic book.
1, Standing at the Precipice, p11
Sagan quote from Pale Blue Dot. Step back and view the human story as a whole. With a focus on our ever-increasing power.
How we got here, p12
The savannas of Africa 200,000 ya. But records go back only 5000 years. We were set apart from other animals by intelligence, creativity, and language. And the ability to cooperate with others. Even across time, via knowledge and culture. There were three great transitions. First, the Agricultural Revolution, some 10,000ya. There were several ‘cradles’ of civilization, map p15. Cities grew, to as many as a million people in Sumer, 60 million in China. Technology grew, specialties were created. Writing, math, law, the wheel. Second, the Scientific Revolution. Only about 400 years ago. Allowing freedom from the reliance on authorities. Further technology, increased power, improving the world around us. The idea of progress, rather than narratives of decline and fall or of recurring cycles. Third, the Industrial Revolution. Coal and other fossil fuels. Steam engines, mechanical energy, railroads, economic growth. The effects of these have not always been positive. Inequality, pollution. Yet the average human life is better now than at any previous time. Especially in overcoming poverty. In literacy and education. Life expectancy. And in moral thinking—the expansion of the moral community. There have been setbacks and exceptions. But the trends toward progress are clear. Even though people tend to focus on bad things rather than good. Given this past, why is now a point where it might all fail? There have been such predictions before.
Figure 1.3, four charts showing changes in poverty, literacy, child mortality, and life expectancy.
Where we might go, p20
Our 200,000 years is a mere tick in the 14-billion-year history of the cosmos, with trillions of years ahead. A typical mammalian species lasts about a million years. Against such a scale, humanity is an adolescent, about 16yo in an 80-year life span. Yet the planet will remain habitable for another billion years. And if we leave the earth, there could be even more time. 100 billion stars in the Milky Way; billions of other galaxies. Many problems remain to be solved. But many heights remain to be explored.
The Precipice, p22
But we have recently undergone another transition beyond those first three. Our technology has advanced to the point where we can destroy ourselves. Atomic bombs. In the ‘80s we worried about nuclear winter. We’ve only barely avoided such a war. Recall the Cuban Missile Crisis again. Then the Soviets shot down a U-2. Twice, the US nearly attacked Cuba. Not necessarily with nukes. … Details. Keep in mind that such an attack wouldn’t necessarily lead to the end of civilization, or of humanity. Currently, the American and Russian arsenals are the greatest threats. And there are other threats to humanity. Climate change. Others include biotechnology and bioweapons. And AI.
These threats to humanity define our time. The problem isn’t technology so much as a lack of wisdom. Sagan quote 30t. And one by Obama. [[ ==> good perspectives on the big picture, and the idea of wisdom. ]] Can we assess these risks? They’re complex and unprecedented. They will be higher each century until we get them under control. Author is calling this time the Precipice.
Box, p33: The Precipice and Anthropocene.
How they’re different.
2, Existential Risk, p35
Consider how existential risks are different than extinction or the collapse of civilization.
Understanding Existential Risk, p36
(Note that author cites Nick Bostrom a lot, but from journals and TED talks, not books he’s written. He’s only published two, and the second seems self-published.)
Author follows Bostrom in making some distinctions. See chart p37. Ex. Catastrophe vs risk. Also, ‘humanity’s longterm potential’. Cautions that he doesn’t think only humans count. Nor does he include threats to a single country or culture. We mean only what threatens the whole of humanity. There are outcomes that don’t count as existential catastrophes, e.g. a series of failures, or a catastrophe that allows humans to recover. Civilizational collapse would reduce a population to a pre-agricultural way of life. And those might recover eventually. It’s happened before. New civilizations would be built on the ruins of old ones. We can think of a minimum viable population, for it to survive. Hundreds, perhaps thousands. We rarely think about such ideas except in thrillers. And if anything truly obvious were to threaten the world, like an asteroid, the world would rally.
Looking to the Present, p42
Some catastrophes may not lead to extinction, but we have a moral case for preventing them anyway…
Looking to Our Future, p43.
An existential catastrophe doesn’t just kill people, it destroys our potential. Derek Parfit. Difference between a war that kills 99% of people, and one that kills 100%. The latter would prevent the people who would have been born… Cites Jonathan Schell from 1982, p44. “Transient aims and fallible convictions.” The entire future. This is what drives the motivation to reduce existential risk. “Profoundly parochial and dangerously short-sighted” 44.8. 45m: “Recognizing that people matter equally… is a crucial step on the ongoing story of humanity’s moral progress.” The ethic here might be called ‘longtermism.’
There are complexities. The idea of discounting that economists use suggests the entire far future is worth only 20 times as much as the coming year. The problem is economics. See Appendix A. Second, does it matter if future generations come into existence or not? Appx B. One more: does it matter if humanity goes extinct or not? Epicurus.
Looking to our Past
49.3 Intergenerational cooperation: “inheriting from our parents, making some small improvements of our own, and passing it all down to our children.” Burke, Seneca quotes. Humanity can complete grand projects that no single human can. You pass on to the next generation. [like sf’s ‘pay it forward’] One reaction is to cherish or preserve what we have from the past against threats from the future. Our duties to the future include making up for past wrongs. [[ this is what affirmative action is about ]]
Civilizational virtues, p52
Humanity is currently at an adolescent stage of growth. We don’t need to plan out the future, but we do need to bear in mind the scope of what lies forward. Short-term vs long-term. 52b. Consider humanity as a group agent. Civilizational virtues and vices at the largest scale. Lack of patience.
Cosmic significance, p53
If we are alone in the universe our own significance is cosmic. Our responsibility is larger, if we are the only ones.
Box p54: The Perspective of Humanity
Considering ethics from a species level. (footnotes have lots of quotes by many). Only recently have isolated peoples come together to form a single global civilization. Yet some moral issues operate at the personal level, or the level of smaller groups. Will return to this in Ch 7.
Uncertainty, p56
Still, we can’t be certain of anything, what we should do or what the risks are. But some of them, perhaps many, are uncontroversial.
Our Neglect of Existential Risks, p57
Consider engineered pandemics. Negligible amounts are being spent on avoiding these. Doing so is a public good, not driven by markets. A global public good. Each nation is tempted to free-ride on the efforts of others. Management is best done at a global level. Intergenerational. Politicians focus on the short term. Election cycles. Or the feeling that such dangers are ‘above my pay grade.’ And two particular biases: availability heuristic, scope neglect. A lack of sensitivity to scale. Existential risk is new, but we’re learning to take it into account. Ever since the creation of nuclear weapons. Schell, Sagan, Derek Parfit, John Leslie, Nick Bostrom.
So, then, what are the real risks? Next section.
[[ Reading through these sections just underscores how everything the Republicans are doing is wrong, because they think only in the very short-term. ]]
Part Two: The Risks
3, Natural Risks, p67
Looking at examples from human history.
Asteroids and comets. Recalling Mexico, the dinosaurs. Background. Craters on earth. Iridium layer. Cf science fiction, and Asimov. Near-earth objects tracked. Next hundred years? Small chances. International cooperation worked well to track these objects.
Sidebar box, p72: deflecting impacts. Destroying it, or changing its course.
Supervolcanic eruptions, p74. Leaving calderas, e.g. Yellowstone. Volcanic winters. 1815 was the year without a summer. Would be devastating, but unlikely to cause total extinction. Might cause starvation for years. Maybe only one every 20,000 years. Perhaps 1 in 800 chance over next century. But a greater risk than an asteroid or comet.
Box: Floods of Lava. A vast event may have ended the Permian age, 250mya.
Stellar explosions, p77. Supernovas. Gamma ray bursts indicate something else. Collision of stars? They could disrupt our atmosphere. Chances: 1 in millions.
Other natural risks, p79. The sun will brighten, in a billion years. Some risks very unlikely, e.g. a star passing through our solar system. The vacuum of space could collapse. A reversal of magnetic poles wouldn’t be an extinction event.
The total natural risk, p80. Most of these have only recently been discovered. There will likely be more. We can estimate complete risk by looking at the fossil record, and mass extinctions. Details. Risk per century something like 0 to 0.05%. We could consider other groups besides just Homo sapiens. Similar species. Etc. Box p84 on survivorship bias. Finally look at the mass extinction events, p86. About one in a million per century. We’ll consider pandemics later.
4, Anthropogenic Risks, p89
How do we assess technological risks over the past 260 or 75 years?
Nuclear weapons, p90.
At one time they worried that a single nuclear explosion might destroy humanity. 1942, Oppenheimer. Teller. Bethe. A single explosion might ignite the atmosphere. It didn’t happen, of course. But author dates the beginning of the Precipice to that moment, of the Trinity test. July 16, 1945. They perhaps did not have the right to make such a decision. A later bomb, with lithium, went wrong…
After the bombs in Japan, the US began planning nuclear war with the Soviets. We’ve made it through 75 years of this, but there were some close calls, p96: 1979, 1983, 1995. What would a full-scale nuclear war do? Direct hits would take out certain local areas. The greater damage would be worldwide. Fallout. Soot high in the atmosphere causing nuclear winter, early 1980s. Harm to agriculture. But perhaps not extinction. Depends on various factors, 100t. And the number of stockpiled nuclear warheads is decreasing. But new situations could emerge.
Climate Change, p102.
History. Widespread agreement. But is this a direct mechanism for existential catastrophe? Runaway greenhouse effect. A kind of feedback loop. Also the moist greenhouse effect. Records from the past are unclear. Not settled science (re: those effects). But there are other routes; details. Water vapor from the oceans. Melting permafrost. Release of methane from the deep ocean. We may burn more fossil fuels. Potentially eight times as much as we’ve burned so far. Table 4.1 shows all these potential carbon sources. The effects, climate sensitivity, are hard to estimate. But substantial warming would be a global calamity. If not extinction. Heat stress. We need to live where we can rid ourselves of waste heat, by sweating. Large areas might become uninhabitable, but many areas would remain. As for mitigation, we should reduce our emissions. Another technique is geoengineering. Remove carbon dioxide, or manage solar radiation. These may have unintended consequences.
Environmental Damage, p113
Overpopulation for example. Ehrlich was mistaken. The Green Revolution. Norman Borlaug and better wheat. Population growth has slowed. It will level off. Or should we worry about declining population? If it becomes a problem, many correctives are possible.
Another issue is resource depletion. We can easily get by without fossil fuels. Water might be an issue. Biodiversity could be a problem. The so-called ‘sixth extinction’ involves nowhere near the percentages of the previous five. …
(He keeps hedging various estimates, but mainly because his underlying question is, are there extinction-level threats here.)
5, Future Risks, p121
Author cites examples of scientists who claimed this or that was impossible. New technologies can quickly emerge. We can expect them to continue. Author is more ambivalent about technology than he used to be.
Pandemics, p124
The Black Death. A similar plague in 541, the Byzantine empire. The Europeans took diseases to the Americas. Killed as many as 10% of the world’s people. Influenza in 1918. Still, they were not extinction level threats. Our modern population makes pandemics easier to spread. OTOH the world’s population is healthier now. Things are uncertain, but the evidence suggests that an extinction risk is low from ‘natural’ pandemics. What about pathogens that are deliberately created? Our understanding has increased in the past two centuries. There are risks from research. Examples of lab escapes. Box p131. There are also threats from deliberate misuse. History of spreading disease as a weapon. Maybe even the Black Death. British in Canada, 1763. Spreading smallpox to Indians. Biowarfare. Democratization. And proliferation. The Shinrikyo cult in Japan, 1984-5. Since the 1950s there have been national and international efforts to protect humanity. But they are underfunded. And many players cheat, or refuse to sign. There are information hazards, box p137. Freely available data. Openness is part of science. The unilateralist’s curse. It only takes one.
Unaligned Artificial Intelligence, p138
As far back as 1956. Some things are easier for machines to do than humans. Now, deep learning. Fake images and voices. Games. Chess, go. Is there existential risk? AGI. We control our destiny. Would AGI? Mentions Brave New World, “With Folded Hands”. Could an AI system actually achieve control? It wouldn’t require robots. First, spread thousands of backup copies around the internet. Form a botnet. Hack bank accounts. Blackmail people. And so on. AI taking over wouldn’t necessarily bring about the extinction of humanity. And the risk from AI is more speculative than some of the others. Even some of the experts express caution, to various degrees. In 2017 came the Asilomar AI Principles. …
Dystopian Scenarios, p153
Permanent destruction of humanity’s longterm potential could also come without extinction or collapse of civilization. We could be locked into a bad future. An unrecoverable dystopia. Three types. See chart 154. Enforced dystopia. Totalitarianism. Orwell. Or, an undesired dystopia. A new equilibrium; game theory, tragedy of the commons. Environmental degradation. Then, a desired dystopia. From defective ideologies or moral views. [religion!] The renunciation of technological progress. Fundamentalist religion.
For any of these the risk is to be locked-in. Ways to avoid them are to encourage rational debate; see the Long Reflection in ch7. Another idea: preserving our options.
Other Risks, p158
Nanotechnology. Drexler. Explorations beyond Earth. Samples from other planets. Spacefaring alien civilizations. And SETI. Radical scientific experiments. Risk assessment.
Box p162: Unforeseen risks. Consider the risks of 1930; they wouldn’t include many we do consider now. So there’s bound to be some we can’t foresee either.
Part Three: The Path Forward
6, The Risk Landscape, p165
View the big picture.
Quantifying the risks, p165.
We need to assign some numbers. Quote from Steven Pinker. Author presents his estimates, p167, with a total existential risk of 1 in 6. With a range up to 1 in a billion (stellar explosion). And as high as 1 in 10, for AI. The other ‘big five’ include nuclear war, climate change, environmental damage, and engineered pandemics. For the long term: a 1 in 2 chance that humanity survives to achieve its potential. And about a third of all the risk is just in the next century. In part because author thinks we’ll start taking risks seriously.
Box, p171: Anatomy of an Extinction Risk. What do they have in common? How they get started, how they reach a global scale, how it finishes the job. We can invoke prevention, response, and resilience.
Combining and Comparing Risks, p173
Consider total existential risk. How to add up risks. Only if they’re perfectly anticorrelated. Ideally we will find positive correlations that identify common causes and common solutions. Etc.
Risk Factors, p175
Why a great-power war would probably not be an existential risk. But it would increase other risks. Call it an existential risk factor. Box, p178, showing the mathematics. Other factors might reduce risk – call these security factors. Social goods like education, peace, prosperity. …
Which Risks?, p180
How do we prioritize these? One might take cost-effectivity into account. Broken down into importance, tractability, and neglectedness. And the general principle of proportionality. Three more heuristics: focus on risks that are soon, sudden, and sharp. And so on. Targeted and broad interventions. (He refers to Bertrand Russell and Albert Einstein as ‘early longtermists.’) Box p185 on Early Action. Making corrections early on. Self-improvement, like education. We do spend a lot on education, and very little on risk interventions. So now let’s start planning.
7, Safeguarding Humanity, p187
Asimov quote from A Choice of Catastrophes, about how we can choose to have no catastrophes at all.
We can take responsibility for our future. It’s about what people who are alive right now can do.
Grand Strategy for Humanity, p189
1, Reaching existential security; 2, the long reflection; 3, achieving our potential. First we reach a place of safety. Preserve and protect humanity’s potential. Reduce risks so they never again pose a serious threat. That doesn’t make such risks zero. Rather a sort of existential sustainability. Perfectly possible in principle. Once we achieve security, we can contemplate what we want our future to be. [[ This is where science fiction chimes in. ]] Wait! He does mention science fiction in footnote 8, mentioning Egan’s Diaspora. Note his comment about how criticism in sf is seldom about those big ideas. What is the best kind of future for humanity? (Footnote abut moral theories) Changing our nature risks losing what is valuable about humanity. [[ SF chimes in here too ]] The process of the long reflection wouldn’t be the sole task of humanity, any more than the Renaissance was. The ultimate aim is achieving our full potential. But we need safety first.
Box, p194: Security Among the Stars? Do we just need to spread out into space? (Various quotes in fn 16) The problem is that the risks may spread too. (note Sagan quote p393 which neatly combines this guy’s ‘precipice’ with Clarke’s ‘great divide’ in 2010)
Risks without Precedent, p195. Existential catastrophes are unprecedented. Thus. We can’t rely on our current (evolutionarily derived) intuitions and (culturally derived) institutions. Second, we can’t afford to fail even once. We can’t learn by trial and error, as usual. Third, we may not have the knowledge to deal with such risks, or to estimate their true likelihood. And our estimates are likely wrong.
International Coordination, p199
Like the United Nations after World War II. Ideas of world government have been around since Wells and Einstein and Russell. It can mean removing the threat of war, but some fear it means homogenizing the world, with the dangers of locking in bad values. Other methods. Strengthen existing institutions. International law. Etc. Make it a crime to threaten the continued existence of humanity. Some nations have made changes to represent the views of future generations.
Technological Progress, p205
Reviewing progress. 205.8: “without further technological progress we would eventually succumb to the background of natural risks such as asteroids.” But should we slow down? Power vs wisdom, 206m. Humanity is like an adolescent, 206b. But how would we restrain ourselves? [[ And what is this ‘wisdom’ that needs to be acquired? Most people simply rely on religion. ]] Some government practices can be improved, 208. Examples.
Box p209: State Risks and Transition Risks. The former are those about being in a vulnerable state of affairs. The latter occur during changes of technology or social regime. The latter might be proceeded with caution…
Research on Existential Risk, p210
Research is needed. Especially on certain topics… 211. Several projects have recently begun.
Box p212: What Not To Do. Don’t regulate prematurely. Don’t take irreversible actions unilaterally. Don’t spread dangerous information. Don’t exaggerate the risks. Don’t be fanatical. Don’t be tribal. Don’t act without integrity. Don’t despite. Don’t ignore the positive.
What You Can Do, p214
Our careers. Examples involving AI, pandemics, climate science. Author moved from computer science to ethics. Otherwise: giving money to causes. List in footnote 72 p403. Finally: public conversation.
8, Our Potential, p217
Wells quote. So what lies beyond? Potential, not prophecy.
Duration, p218
200,000 years of Homo sapiens and 10,000 years of civilization (since agriculture) and 5000 since the first city-states. While the universe is billions of years old, and will last billions more. Fossil records: the average mammalian species lasts a million years, species in general one to ten million. Humanity could survive much longer. We know of species that have lasted hundreds of millions of years. Examples. Geological timescales. Continental drift. Constellations will change. The sun will eventually cause problems and earth may become uninhabitable. And so on… Humanity might save the world. Or travel to the stars.
Timelines: p224-5
Scale, p226
Stars, nature of reality, colors, and vastness. In just five centuries. Solar collectors. Distances to the stars. Colonies. It’s right to be skeptical about films like Wars and Trek. Those capabilities are not required. Would we find life, aliens? The galaxy… Once we reach the nearer stars, the whole galaxy is available. Local Group. More groups of galaxies. Filaments. Cosmic voids. The observable universe. Expanding. The affectable universe, as the farthest galaxies move out of reach. Eventually each group will be alone from the others. …
Quality, p235
We’ve seen that human life is much better today than ever before. Details. But it could be even better. Violence, religion, poverty, racism. And animals. We can end all these blights. We can build a society that is just and humane. Consider moments of true happiness, or peak experiences. We can escape routine and pursue more of those. What other animals experience. We can transform into something beyond the humanity of today. With risks of splintering and inequality.
Choices, p239.
So, three dimensions: time, space, and experience. We are like people of 10,000 years ago, on the cusp of agriculture. On the edge of vast potential.
\\
