Syllabus for MAS.S67 Safeguarding the Future 2022

At our current rate of economic growth, we’d be using every atom in the galaxy in less than 10,000 years… which is a problem because the Milky Way is more than 50,000 light-years across. In less that 150 years, we’ve gone from fountain pens to tweets, cavalry to ICBMs, and snake oil to CRISPR-mediated T cell therapy … yet somehow we imagine that the future will consist of people like us who are essentially still making all the decisions. If the present world cannot endure without catastrophe or radical transformation, then the future will be quite different from what most people expect. It might offer delight and fulfillment literally beyond our current capacity to imagine, a hellish dystopia, or an empty universe. If we prefer the first option, we’d better figure out how to live with what we will have.

Feb 3 - Storytelling and history

We’re creatures of the story. There is no human history without somebody capable of presenting a compelling narrative. And we can shape the future by deciding how to tell our story. How, then, can we describe the arc of human history in ways that will improve the future? What can we learn from it? How many people lived when, under what conditions? When and what were the turning points that changed history? Were they inventions, ideas, or wars? Which stories have changed the world? …for the better? And which have changed it for the worse?

Due:

“This can’t go on” (H. Karnofsky)
“All possible views of humanity’s future are wild” (H. Karnofsky)
“1960: The year the singularity was cancelled” (S. Alexander)

Feb 8 - Nuclear game theory Guest: Andrew C. Weber

Leo Szilard, inventor of the chain reaction, warned that dropping the bomb on Japan in 1945 would accelerate an arms race that would make American cities vulnerable to foreign militaries for the first time in half a century. He was right. Nuclear weapons cannot be blocked, and can only be deterred through mutually assured destruction. Whether they kept the Cold War cold is debatable, but it was always touch and go. We came close to losing around half of humanity in 1983. What can we learn from our first experience with a technology capable of threatening civilization?

Due:

Recommended:

The Dead Hand (D. Hoffman)
NUKEMAP (A. Wellerstein)

Feb 10 - Stories, cooperation, and Great Powers

Why do some narratives resonate in one place but not in others? How do different peoples view their history, and how does their perspective change how they interact with one another, and with technology? How can gatherings be structured to best create the most impactful outcomes?

Due:

The Pugwash Conferences (J. Rotblat)

Feb 15 - Biological catastrophe

COVID-19 was caused by a single, comparatively mild pandemic virus that wasn’t even engineered to cause harm. Around thirty thousand people today could single-handedly assemble most viruses from synthetic DNA using cheap equipment. That number gets significantly larger every year as the price of computing power drops and the tools of genetic engineering grow. For years, scientists have tried to find viruses in nature that can harm us. Should they be doing that? What would happen if they were to post a list of credible pandemic viruses found in nature? Or learn how to engineer more destructive ones? How can we best immunize civilization against catastrophic biology… and delay disaster until we have succeeded?

Due:

“The Deadliest Virus: Did a scientist put millions of lives at risk, and was he right to do it?” (M.Specter)
“Inoculating science against information hazards and potential pandemic pathogens” (K. Esvelt)

Feb 17 - Timelines to transformative AI Guest: Ajeya Cotra

We have already created systems with superhuman capacities. Soon AIs will be able to reliably accelerate the pace of science and technology development. That will likely return the world to a cycle of exponential growth and innovation that halted when we stopped having as many babies as we could feed, interrupting the sequential conversion of inventions into calories into inventors. The resumption of that cycle will mark the end of our current world. When will it happen, and how can we begin to prepare?

Due:

“Timelines to transformative AI” (A. Cotra)
“Provably beneficial artificial intelligence” (S. Russell)

Recommended:

Superintelligence (N. Bostrom)

Feb 24 - Learning from history

Who had the most positive impact on the world? Which inventions and ideas were critical? Conversely, which ideas and inventions were most harmful? Why? Was there ever a year in which discoveries caused more harm than good? Can we infer anything from that?

Due:

“Smallpox and vaccines” (J. Crawford)
“Forgotten benefactor of humanity” (G. Easterbrook)
"Leaded Gas Was a Known Poison the Day It Was Invented" (K. Eschner)
“Information hazards: a typology of potential harms from knowledge” (N. Bostrom)

Recommended:

The Wizard and the Prophet (C. Mann)
The Alchemy of Air (T. Hager).

Mar 1 - Mistakes and defection

Do most catastrophic risks arise from accidents or deliberate actions? History suggests that our record on accidents is mixed, and could certainly be improved. But the real uncertainty concerns the number of people who would push a doomsday button, if given the opportunity. To date, only great powers have had that chance, and their actions are largely tempered by game theory. Should many more individuals gain that power, we may expect a different outcome.

Due:

Parsons, Keith M.; Zaballa, Robert A. (2017). Bombing the Marshall Islands: A Cold War Tragedy. Cambridge University Press. Chapter 3: Runaway Bomb. ISBN 978-1108508742. (accessible through MIT libraries as PDF download)
The fear of setting the planet on fire (Y. Yiu)
“The Unilateralist's Curse” (N. Bostrom et al)
“Who would destroy the world? Omnicidal agents and related phenomena” (P. Torres)

Mar 3 - Resilient logistics: food, water, and power

There is an inherent tradeoff between efficiency and robustness. By definition, a maximally efficient system has no redundancy. Just-in-time manufacturing minimizes warehouse storage time and wasted materials, but is fragile to supply chain disruptions. How can we render our supply chains resilient when market forces work to render them fragile? Which are particularly at risk, and what can we do?

Due:

"The fragility of interdependency" (A. Vespignani)
“Supply chains and the coronavirus” (L. O’Leary)
“Feeding everyone if the sun is obscured and industry is disabled” (D. Denkenberger)

Recommended reading:

1177 BC: The Year Civilization Collapsed (E. Cline)
The Collapse of Complex Societies (J. Tainter)

Mar 8 - The planetary thermostat Guest: David Keith

Few people continue to doubt that climate change is real and that human activity is the major cause. Many assume that warming of 2oC - a figure that only twenty years ago was considered catastrophic - is now unavoidable. That will dramatically increase water and food shortages in the poorest parts of the world and gradually submerge land occupied by a significant fraction of humanity. The long-term impacts, let alone the tail risks, will almost certainly make everything else more difficult. What can we do?

Due:

“The Climate Fixers: Is there a technological solution to Global Warming” (M. Specter)
“Economic impacts of tipping points in the climate system” (Dietz et al)
“Evaluating climate geoengineering proposals in the context of the Paris Climate Agreement temperature goals” (Lawrence et al)
“Marine cloud brightening” (Latham et al)
“Halving warming with idealized solar geoengineering moderates key climate hazards” (Irvine et al)
“How green sand could capture billions of tons of carbon dioxide” (Temple)

Mar 10 - Right idea, wrong PR

Some view technology as deterministic. But even if people will eventually adopt whatever is useful, multiple technologies may fill a need, in which case outside factors can decide which will be developed. To what extent does the success of a technology depend on its inventor? On a popularizer? A funder? What happens when a true and important idea is devised by the wrong person, or the wrong place? Or the opposite?

Due:

“The doctor who championed hand-washing and briefly saved lives” (R. Davis)
“The vulnerable world hypothesis” (N. Bostrom)
"Fritz Haber and Carl Bosch" (C. Flavell-While)

Mar 15 - Coordinating preparedness Guest: Rick Bright

COVID-19 gave us a reasonable snapshot of how well human civilization can be expected to respond in a global crisis. What can we learn from the last two years? Which institutions can we rely on? In which countries? Are there any patterns? How can we compensate for weaknesses and uncertainties?

Due:

“10 lessons I’ve learned from COVID-19” (H. Branswell)
“The American Pandemic Preparedness Plan” (White House)

Mar 17 - choice of topic to be determined by class interest

Mar 29 - Scientific norms & biorisk Guest: Megan Palmer

Many scientists, especially in biology, strongly believe in open research and data sharing. When the genome of the 1918 influenza virus and the report of its resurrection was published, the editor-in-chief of Science wrote that the journal would have published the genome sequence regardless of what the government advisory panel decided, and would only have refrained if threatened with jail. Can we reconcile the culture of openness with information hazards?

Due:

“Better never than late” (D. Kennedy)
“Learning to deal with dual-use” (M. Palmer)
“Protocols and Risks - when less is more” (J. Pannu et al)

Recommended:

“The Elephant in the Brain” (Simler and Hanson)

Mar 31 - choice of topic to be determined by class interest

Apr 5 - Aligning transformative AI Guest: Paul Christiano

What happens when we create agents more capable than we are? After all, chimpanzees are nearly extinct and mostly reside in zoos. Ideally, we want anything with power to be not only as aligned with our interests as the average human, but much more so. Even given perfect alignment with human interests, machines with significantly superhuman competence will still upend the strategic gameboard, potentially creating incentives for people to act dangerously. To give just one example, superhuman competence in biology might create catastrophic threats that could reliably overcome our human-designed defenses. How can we both align transformative AI systems and prevent their superior capabilities from causing harm?

Due:

“What failure looks like” (P. Christiano)
“Overview of the AI alignment landscape” (N. Nanda)
"Yudkowsky Contra Christiano on AI Takeoff Speeds" (S. Alexander)

Recommended:

Human Compatible (S. Russell)

Apr 7 - Building Refuges Guest: Ajay Karpur

If most of civilization falls, where might humanity survive and thrive? Can we deliberately build refuges that would help? Where should they be located, what will they need, and who should go there?

Due:

How much could refuges help us recover from a global catastrophe? (N. Beckstead)
Optimizing island refuges (M. Boyd and N. Wilson)
Isolated refuges for surviving global catastrophes (S. Baum et al)

Relevant fiction:

Seveneves (N. Stephenson)

Apr 12 - Rebuilding civilization

Given the non-trivial probability of civilization collapsing, it’s remarkable how little has been done to help the survivors rebuild. What do we most want to pass on to our successors? To their descendants? Can we learn anything from past civilizational collapses and recoveries? To what extent do the answers depend on the nature of the threat that ends the rest of us?

Due:

“The Library of Utility” (K. Kelly)
“Manual for civilization” (A. Rose)