Current and Near-Term AI as a Potential Existential Risk Factor

Benjamin S. Bucknall and Shiri Dori-Hacohen ask how AI could increase existential risk without itself becoming an uncontrollable general intelligence. Their position paper connects effects on institutions, power and information to society’s ability to prevent or respond to other hazards. It offers a qualitative causal map and research agenda, rather than estimating the probability of catastrophe.^[1]

A risk factor is not the catastrophe

The authors distinguish an existential risk factor from an existential catastrophe: a situation can make a catastrophe more likely without constituting one itself. Their definition of catastrophe includes permanent destruction of humanity’s long-term potential, not only extinction. A stable, permanently repressive regime is therefore one of the outcomes considered (§2).^[1]

“Near-term” refers to systems already existing or under active development with a clear route to deployment at the time of the 2022 paper. The unit of analysis includes the political, corporate and technical arrangements around AI. A recommender system’s objective, the company operating it and the institutions affected by its outputs belong to the same explanation; model capability alone is not the proposed cause (§2.1).^[1]

Power relationships and information

The argument separates three relationships. Between states, perceived competition over AI could encourage arms-race behavior, divert resources from safety or intensify military tensions. The authors do not require believing that one country’s technological advance is inherently harmful: believing that it is a threatening race can itself change behavior. Between states and corporations, concentrated technical capabilities and lobbying could weaken public oversight. Between states and citizens, surveillance could help a regime suppress opposition and become more durable (§3).^[1]

The information pathway concerns collective decisions. The authors propose that recommendation incentives, polarization and misinformation can undermine shared understanding and trust. That could make institutions less able to coordinate on problems such as pandemics or climate change. These are the paper’s links between a synthesis of nearer-term effects and much larger hazards, not findings from a new experiment demonstrating the complete chain (§§4–5).^[1]

How the causal map works

Figure 1 connects AI’s effects to intermediate factors and then to risk sources. Diverted resources and compromised political decision-making can affect several hazards at once. Other links are more specific: great-power conflict connects to nuclear and biological risks, surveillance connects to lasting repression, and the paper considers direct environmental costs and possible malicious uses. It treats cybersecurity mainly as a contributor to changing state relations in this diagram, rather than independently drawing an edge to every catastrophe (§§5–6).^[1]

The arrows distinguish mechanisms, not measured causal strengths. The authors tentatively prioritize state relations because they connect to several risks, then acknowledge that such comparisons depend on the magnitude of each link and the underlying hazards. Counting connections cannot establish either a probability or the best intervention. A quantitative extension is left as difficult, speculative future work (§6).^[1]

Historical context

The reviewed arXiv v1 identifies AIES 2022 proceedings, an August 1–3 conference and September 21 arXiv submission. Chronology uses the proceedings year; conference dates do not establish the initial publication day.^[1]

The paper acknowledges beneficial AI applications and leaves its cited empirical claims and proposed pathways open to further investigation. Is Power-Seeking AI an Existential Risk? instead centers on advanced agents’ objectives, deployment and failed correction. This account adds a different question for AI Alignment and Society-in-the-Loop: how the institutions using AI change the conditions under which people can collectively address danger.^[1]