The AI
Risk Map

In one sentence: AI speeds up biology and medicine enough that decades of expected progress (new drugs, cures for major diseases, longer healthy lives) arrive within roughly a decade.

Design note: for a good-direction card, "worried" means worried this won't happen or is oversold, not worried that it will.

Preconditions

• AI keeps improving at predicting biology (protein structure, drug-target interactions, toxicity).
• AI-designed candidates succeed in human trials at higher rates than traditional ones.
• Regulators adapt trial and approval processes without lowering safety standards.
• Laboratory and manufacturing capacity grow to test far more candidates.

Timing and reversibility

• Earliest plausible date: first AI-designed drugs in human trials, 2026. First approval, several years after that at best.
• Speed: slow-building. Clinical trials have fixed minimum durations.
• Reversibility: not applicable for a benefit. The relevant risk is that it is delayed or oversold.

The case that this happens (optimistic)

Short excerpts for orientation only. A person from this side will choose the final quotes at sign-off.

• Dario Amodei, in his October 2024 essay Machines of Loving Grace, guessed that powerful AI could raise the rate of biological discovery at least tenfold, delivering 50 to 100 years of progress in 5 to 10 years, while explicitly allowing that experiments and clinical trials cannot be sped up by software alone.
• Demis Hassabis, Google DeepMind CEO and 2024 Nobel laureate for AlphaFold, said on CBS's 60 Minutes in April 2025 that ending all disease is within reach, maybe within the next decade.

Plain summary (needs sign-off from this side): AlphaFold solved a fifty-year problem in biology and won a Nobel Prize. AI is now predicting how drugs bind to targets, designing new molecules, and outperforming experienced physicians at diagnosis from real emergency records. The FDA cleared 350 AI-enabled medical devices in 2025 alone. Discovery is the stage AI accelerates, and it is a large share of the cost and time of making a drug.

The case that it doesn't (worried this is oversold)

• Since 2019, about $60 billion has gone into AI drug discovery, producing roughly 175 drug candidates in clinical trials and zero approved drugs.
• Isomorphic Labs, the DeepMind spin-off, expected its first human trials by the end of 2025, then pushed that to the end of 2026, and had not dosed a patient as of mid-2026.

Plain summary (needs sign-off from this side): Discovery was never the main bottleneck. Most drugs fail in human trials because biology is complicated, not because the molecule was badly chosen, and AI doesn't shorten Phase 2 or Phase 3. Diseases like cancer and Alzheimer's involve many interacting causes that have resisted decades of work. Bold timelines from company leaders help raise money. AI will help medicine steadily, but "all diseases in ten years" sets expectations that will hurt public trust when they're missed.

Common ground

• AlphaFold is a delivered, Nobel-recognized breakthrough.
• AI speeds up the discovery stage. It does not shorten human trials.
• No AI-originated drug has been approved yet.
• Clinical, laboratory and regulatory capacity limit how fast this can go, whatever the AI can do.

Cruxes

1. Do AI-designed drugs pass Phase 2 trials at a higher rate than traditional drugs? This is the single most important question, because Phase 2 is where most drugs fail. Check: 2027 to 2029.
2. Does the first AI-originated drug win full FDA approval? Check: by 2030.
3. Do regulators adopt AI-assisted trial designs that shorten trials safely? Check: FDA guidance, 2027 to 2028.

Probability estimates

No published estimate of the probability of compressed medical progress was found. Related claims:

Both are conditional or hedged in their original wording. Headlines usually drop the conditions.

Evidence confidence

Medium for discovery-stage acceleration. Low for the overall timeline.

Signposts

1. Isomorphic Labs or another AI-first company doses its first patient.
2. Published Phase 2 success rates for AI-designed drugs exceed the industry average (commonly cited near 30%; source needed before sign-off).
3. The first AI-originated drug receives full approval.
4. FDA AI-enabled device clearances keep rising (130 in 2021, 350 in 2025).

Evidence so far

• AlphaFold: 2024 Nobel Prize in Chemistry.
• 1,451 AI-enabled medical devices cleared by the FDA through 2025.
• An April 2026 Harvard and Beth Israel study found an AI reasoning model outperformed experienced internists at diagnosis from real emergency department records (text only, no images or exam).
• Measured downside: experienced doctors' unassisted cancer detection fell from 28.4% to 22.4% after routine AI use (Lancet Gastroenterology & Hepatology, 2025). Benefits can come with new risks.

Existing coverage

FDA drug and device approval processes apply in full. The FDA has issued guidance on AI-enabled devices. No law blocks AI drug discovery; the question is whether regulators can evaluate it efficiently.

Triage (flipped for a good scenario: what's blocking it, and does removing the block need policy?)

1. Is it blocked by law? No.
2. An action both sides accept? Likely yes: FDA capacity and expertise to review AI-designed drugs and AI-assisted trials quickly without lowering standards. Both camps would support it, the optimists to speed things up and the skeptics to get honest results sooner.
3. (Skipped.)

Result: Policy, act now on regulator capacity. Communicate to set realistic expectations.

Communication note: The breakthroughs are real and the timelines in headlines aren't. "AI is changing medicine" is true now. "AI will cure all disease by 2035" is a hedged personal guess.

Related scenarios

• Opposite in the same domain: "AI-assisted biological or chemical attack" (the same biology capabilities, misused).
• Combines with: over-reliance and lost skills (the colonoscopy finding).
• Reduced by: AI disappoints.

Sources

1. CBS's 60 Minutes in April 2025. businesstoday.in, https://www.businesstoday.in/technology/news/story/ai-could-eradicate-all-disease-in-just-10-years-says-google-deepmind-chief-demis-hassabis-473009-2025-04-22
2. pushed that to the end of 2026. investing.com, https://www.investing.com/news/stock-market-news/googlebacked-ai-drug-discovery-startup-isomorphic-labs-delays-clinical-trial-timeline-4456134
3. IntuitionLabs, "AI-Discovered Drugs in Clinical Trials 2026". intuitionlabs.ai, https://intuitionlabs.ai/articles/ai-discovered-drugs-clinical-trials-2026
4. Pharmaceuticals, "AI in Drug Discovery: Clinical Failures, Regulatory Reality, and the Validation Crisis" (2026). mdpi.com, https://www.mdpi.com/1424-8247/19/6/916
5. Nobel Prize in Chemistry 2024, press release. nobelprize.org, https://www.nobelprize.org/prizes/chemistry/2024/press-release/
6. "Three Decades of FDA Authorizations of AI/ML-Enabled Medical Devices" (medRxiv, 2026). medrxiv.org, https://www.medrxiv.org/content/10.64898/2026.05.08.26352766v1.full.pdf
7. NPR, "In real-world test, an AI model did better than doctors at diagnosing patients" (Apr 2026), on the Science study by Harvard Medical School and Beth Israel Deaconess. npr.org, https://www.npr.org/2026/04/30/nx-s1-5804474/ai-doctors-openai-patient-care-diagnosis
8. STAT, on the Lancet Gastroenterology & Hepatology colonoscopy study (Aug 2025). statnews.com, https://www.statnews.com/2025/08/12/ai-deskilling-doctors-colonoscopy-study-lancet/

Sign-off record

• Optimistic case: not yet reviewed.
• Worried case: not yet reviewed.
• Changelog: 2026-10-09 first draft.

In one sentence: Someone who wants to cause mass casualties uses AI to close the gap between wanting to build a biological or chemical weapon and being able to, and carries out an attack that kills far more people than they could have reached without it.

Design note: the two sides here are not "this would be bad" versus "this would be fine." Everyone agrees it would be catastrophic. They disagree about whether AI meaningfully changes who can do it. The word for the thing being argued about is uplift: the extra help an AI gives someone compared with what they could already get from a search engine, a textbook and a library. The question is never whether AI can describe a dangerous process, because so can a textbook.

Preconditions

• An AI system gives real uplift on the steps that are actually hard, not just on the steps that are already in textbooks.
• The person gets past the physical stages too: obtaining genetic material or chemical precursors, doing the laboratory work, producing enough of the agent, and dispersing it.
• Safeguards fail or are routed around, through jailbreaks, pretexting, models released without safeguards, or an insider.
• Someone wants to do it. Mass-casualty intent is rare, and this precondition is doing more work in the overall probability than any capability question.

Timing and reversibility

• Earliest plausible date: the capability side of this is partly here already. OpenAI said it was treating its ChatGPT agent as "High" capability in biology and chemistry under its Preparedness Framework in July 2025, and Anthropic activated its ASL-3 safeguards in May 2025. Both companies said they were acting as a precaution and had not established that the threshold was crossed.
• Speed: an attack is an event, not a trend. There is no ramp to watch, which is what makes this scenario different from most others on the map.
• Reversibility: a chemical attack is terrible but bounded in place and time. A transmissible biological agent is the tail that cannot be recalled, which is why this scenario gets attention disproportionate to its estimated probability.

The case that this happens

Short excerpts for orientation only. A person from this side will choose the final quotes at sign-off.

• Jassi Pannu, a physician and senior scholar at the Johns Hopkins Center for Health Security, told a House subcommittee in December 2025 that "AI uplift increases the number of users capable of carrying out a bioweapons attack."
• The International AI Safety Report 2026, chaired by Yoshua Bengio and published 3 February 2026, records that in 2025 developers shipped models with extra safeguards because they "could not exclude the possibility that these models could assist novices in developing such weapons."

Plain summary (needs sign-off from this side): The knowledge barrier is falling fast and measurably. Models now outperform expert virologists on at least one specialist test, and a February 2026 study of novices answering dual-use biology questions found the group with model access was 4.16 times more accurate than the internet-only group, beating human experts on three of four benchmarks, with 89.6% of participants reporting little difficulty getting the information despite safeguards. The companies building these systems are not disputing this: all three major developers restricted models in 2025 on biological grounds. Separately from chatbots, biological design models are being released, sometimes without basic safety assessment, that can design novel viral components and craft sequences that evade the screening software meant to catch them. The attack itself only has to work once, and the measure that would most reduce the risk, mandatory screening of synthetic DNA orders, has been sitting unfinished in the US executive branch since 2025.

The case that it doesn't

• RAND's red-team study of January 2024 (Christopher Mouton, Caleb Lucas and Ella Guest) found no statistically significant difference in the viability of attack plans produced with and without model access, and concluded that this kind of planning lies "beyond the capability frontier of LLMs as assistive tools."
• Aidan Peppin, Anka Reuel, Stephen Casper and co-authors, in "The Reality of AI and Biorisk" (ACM FAccT, June 2025), found the existing body of work "nascent, often speculative in nature, or limited" in method and transparency, and concluded that current systems do not pose an immediate risk.

Plain summary (needs sign-off from this side): Benchmarks measure answering questions, and nobody has ever been killed by a correctly answered question. The only randomized trial that put people in a laboratory, Active Site's eight-week study of 153 novices published in February 2026, found 5.2% of the group with model access completed the full workflow against 6.6% of the internet-only group, a difference that was not statistically significant (P = 0.759). Participants in both arms rated YouTube more useful than any of the models, and the models in the assisted arm had safeguards removed. Forecasters who had predicted a 27% completion rate for the assisted group were wrong by a factor of five. Physical skill, materials and equipment are the constraint, and text does not transmit physical skill. Treating every capability result as near-term catastrophe risk also carries its own cost: it is being used to argue for restricting the same biology tools that are producing medical progress.

Common ground

• Benchmark performance is real and rising. Models match or beat expert virologists on at least one specialist test, which was not true in 2023.
• No measured uplift at the wet-lab step has yet been demonstrated in a randomized trial (Active Site, February 2026).
• Both trials tested novices. Neither tested trained scientists, who are the group most likely to be helped to a dangerous degree.
• Material barriers matter and are poorly understood. The 2026 safety report says it "remains difficult to assess the degree to which material barriers continue to constrain actors."
• Both sides support screening of synthetic DNA orders. The skeptics support it because the physical step is where the real barrier is, and the worried side supports it because it is the one chokepoint that does not depend on predicting model capability.

Cruxes

1. Does measured laboratory uplift appear once the trial is rerun with 2026-era agentic models and cloud-lab access? The negative result is specific to mid-2025 models and human hands. Check: replication of the Active Site trial, 2027 to 2028.
2. Is the binding constraint knowledge or materials? If mandatory synthesis screening passes and the risk estimates of both camps fall, materials were the constraint. If screening-evasion by sequence design defeats it, knowledge was. Check: US screening legislation and OSTP framework, 2027.
3. Do models without safeguards reach the level the safeguarded frontier models are now restricted at? Safeguards only matter if the unrestricted alternative is worse. Check: International AI Safety Report, 2027 edition.

Probability estimates

Read these with care: every number here is an annual probability of an epidemic from any human cause, not specifically an AI-assisted one, so AI is the conditioning variable rather than the subject. The last row is the most useful line in the table: the same expert groups whose conditional estimates drive the worried case overestimated real-world performance by a factor of five when a result could be checked.

Evidence confidence

Low. Two randomized trials point in opposite directions because they measure different things, one answering questions and one doing laboratory work, and both tested only novices. Every published probability is conditional, and the one prediction that got scored was badly wrong.

Signposts

1. A replication of a wet-lab trial using 2026-era agentic models, with cloud-lab access, shows a statistically significant completion gap.
2. OSTP publishes the replacement nucleic acid synthesis screening framework with the enforcement mechanism required by Executive Order 14292, which was due 90 days after 5 May 2025 and was still outstanding as of September 2026.
3. Congress enacts universal screening of synthetic nucleic acid orders, binding providers that do not take federal funding.
4. A developer reports a safeguard failure in which a user obtained end-to-end weapons-relevant help, rather than isolated facts. Anthropic reported users circumventing its biosafety safeguards in September 2026, describing them as working scientists.
5. A biological design model is released with a capability on a published list of capabilities of concern, with no safety assessment.

Evidence so far

• RAND, January 2024: red-team exercise, no measurable change in the operational risk of an attack from the then-current generation of models.
• Peppin, Reuel, Casper and others, June 2025: review of the field, no immediate risk from current systems, methods immature.
• Anthropic activated ASL-3 safeguards in May 2025 as a precaution, saying it could not rule out chemical, biological, radiological and nuclear risk thresholds. OpenAI treated ChatGPT agent as High capability in biology and chemistry in July 2025 on the same precautionary basis.
• Active Site randomized trial, February 2026: 153 novices, eight weeks, no significant difference in completing the full workflow, a non-significant trend toward faster progress through intermediate steps (pooled estimate 1.4 times, 95% credible interval 0.74 to 2.62).
• Scale AI and SecureBio study, February 2026: novices with model access 4.16 times more accurate on dual-use biology questions than internet-only controls, beating experts on three of four benchmarks.
• International AI Safety Report 2026, 3 February 2026: models can provide information relevant to biological and chemical weapons development, with substantial uncertainty about real-world uplift and material barriers.
• A June 2026 open letter to Congress calling for mandatory screening of synthetic nucleic acid orders was signed by the heads of OpenAI, Anthropic, Google DeepMind and Microsoft AI, along with Nobel laureates and US synthesis providers (CRS, September 2026).

Existing coverage

Developing, producing, stockpiling or using these weapons is already prohibited by treaty. The Biological Weapons Convention was the first treaty to ban an entire category of weapons of mass destruction, but it has no verification organization and relies on a network of cooperating bodies. The Chemical Weapons Convention is stronger: 193 states, an implementing body in the OPCW, routine monitoring of listed chemicals, and challenge inspections on an "any time, anywhere" basis. Neither treaty reaches a lone actor, and neither mentions AI.

The gap is the digital-to-physical barrier. The 2024 US Framework for Nucleic Acid Synthesis Screening made screening a condition of federal research funding, so it does not bind commercial providers that take no federal money. More than 30 leading providers screen voluntarily. Executive Order 14292 (5 May 2025) ordered OSTP to replace the framework within 90 days and add an enforcement mechanism; CRS reported in September 2026 that it had not been updated, leaving providers without regulatory clarity. Bills are pending, including H.R. 3029, which has passed the House, and S. 3741. Free screening tools exist independently of regulation, including SecureDNA. Model-level safeguards are entirely voluntary: developers may report safety evaluation results to the US Center for AI Standards and Innovation, and no law requires it.

Triage

1. Covered by law and enforced? Covered, yes. Enforced, partly. The weapons themselves are banned by two treaties, one of which has real inspection powers. The step where AI actually changes something, the supply of synthetic genetic material, is governed by a framework that binds only federally funded buyers, with the replacement and its enforcement mechanism more than a year overdue.
2. An action both camps accept? Yes, and it is the same action: make screening of synthetic nucleic acid orders and customers mandatory for all providers. The skeptics support it because the physical barrier is the one they believe is doing the work, and the worried side supports it because it does not require winning the argument about model capability first. Frontier developers asked for it themselves in June 2026.
3. (Not reached.)

Result: Enforcement problem. The rules and the agreed fix both exist. What is missing is a published framework, a statute that covers providers outside federal funding, and an agency told to enforce it.

Communication note: Two randomized trials gave opposite-looking answers and both are being quoted selectively. The honest version is that AI has clearly reduced the knowledge barrier and has not yet been shown to reduce the laboratory barrier, that the people who were asked to predict the laboratory result were wrong by a factor of five in the alarming direction, and that the one measure nearly everyone agrees on has been stuck in an unfinished rulemaking since 2025. Do not let the disagreement about probability hide the agreement about what to do.

Related scenarios

• Opposite in the same domain: "Medical progress compressed into years" (the same biology capabilities, used well).
• Same mechanism, different domain: "Cheap large-scale cyberattacks and critical-system failures."
• Raised by: "Loss of control over advanced AI" (safeguards assume a system that can be constrained).
• Reduced by: "AI disappoints."

Sources

1. International AI Safety Report 2026, executive summary, chaired by Yoshua Bengio, 3 February 2026. https://internationalaisafetyreport.org/publication/2026-report-executive-summary
2. Jassi Pannu, testimony before the US House Committee on Energy and Commerce, Subcommittee on Oversight and Investigations, "Examining Biosecurity at the Intersection of AI and Biology," 17 December 2025. https://www.congress.gov/119/meeting/house/118773/witnesses/HHRG-119-IF02-Wstate-PannuJ-20251217.pdf
3. Christopher A. Mouton, Caleb Lucas and Ella Guest, "The Operational Risks of AI in Large-Scale Biological Attacks," RAND Corporation, 25 January 2024. https://www.rand.org/pubs/research_reports/RRA2977-2.html
4. Aidan Peppin, Anka Reuel, Stephen Casper and others, "The Reality of AI and Biorisk," ACM Conference on Fairness, Accountability and Transparency, June 2025 (preprint December 2024). https://arxiv.org/abs/2412.01946
5. Shen Zhou Hong and others (Active Site), "Measuring Mid-2025 LLM-Assistance on Novice Performance in Biology," arXiv preprint, 18 February 2026. https://arxiv.org/abs/2602.16703
6. Chen Bo Calvin Zhang and others (Scale AI and SecureBio), "LLM Novice Uplift on Dual-Use, In Silico Biology Tasks," arXiv preprint, 26 February 2026. https://arxiv.org/abs/2602.23329
7. Forecasting Research Institute, "Forecasting biosecurity risks from large language models and the efficacy of safeguards," 1 July 2025. https://forecastingresearch.substack.com/p/forecasting-biosecurity-risks-from
8. Jamie Parry, Bridget Williams, Zack Devlin-Foltz and Matt Reynolds, "How Well Did Superforecasters and Experts Predict Wet Lab Skill Uplift from LLMs?", Forecasting Research Institute, 19 February 2026. https://forecastingresearch.substack.com/p/how-well-did-superforecasters-and
9. Anthropic, "Activating AI Safety Level 3 Protections," 22 May 2025. https://www.anthropic.com/news/activating-asl3-protections
10. OpenAI, "ChatGPT agent System Card," July 2025. https://openai.com/index/chatgpt-agent-system-card/
11. Congressional Research Service, "Artificial Intelligence and Biosecurity Issues," In Focus IF13269, updated 15 September 2026. https://www.everycrsreport.com/reports/IF13269.html
12. UN Office for Disarmament Affairs, "Biological Weapons Convention." https://disarmament.unoda.org/biological-weapons/
13. Organisation for the Prohibition of Chemical Weapons, "Chemical Weapons Convention." https://www.opcw.org/chemical-weapons-convention
14. Executive Order 14292, "Improving the Safety and Security of Biological Research," 5 May 2025. https://www.whitehouse.gov/presidential-actions/2025/05/improving-the-safety-and-security-of-biological-research/
15. SecureDNA Foundation, screening service. https://securedna.org/

Sign-off record

• Worried case: not yet reviewed.
• Optimistic case: not yet reviewed.
• Changelog: 2026-10-09 first draft.

In one sentence: AI makes workers across the economy noticeably more productive, and the gains reach ordinary people as higher real wages and lower prices rather than staying with the firms that own the technology.

Design note: for a good-direction card, "worried" means worried this won't happen or is oversold, not worried that it will. The scenario has two links that can break separately: productivity has to rise across the economy, and the gains have to reach workers and shoppers. Evidence for the first link is growing. Evidence for the second is close to absent so far. "Productivity" here means output per hour worked.

Preconditions

• The task-level gains seen in field studies (faster, better work on the tasks AI suits) carry over to whole firms and industries, not just to a few tasks.
• Adoption spreads beyond the technology sector and large firms. About 18% of US firms used AI at the end of 2025 (Federal Reserve, April 2026).
• Competition is strong enough that firms pass cost savings on as lower prices.
• Workers have enough bargaining power, or enough new kinds of work, that higher output per hour shows up in pay.
• The costs of building AI (chips, data centers, electricity) do not push prices up faster than the productivity gains push them down.

Timing and reversibility

• Earliest plausible date: task-level gains are measured now. Economy-wide effects are modeled to build slowly: the Penn Wharton Budget Model puts AI's peak yearly contribution to productivity growth at about 0.2 percentage points in 2032 (Penn Wharton Budget Model, September 2025).
• Speed: slow-building. Past general-purpose technologies showed a dip in measured productivity while firms reorganized, then a rise, a pattern economists call the "productivity J-curve" (Brynjolfsson, Rock and Syverson).
• Reversibility: not applicable for a benefit. The relevant risk is that gains arrive but are captured by a few firms, or are offset by higher prices for AI-related goods and power.

The case that this happens (optimistic)

Short excerpts for orientation only. A person from this side will choose the final quotes at sign-off.

• Erik Brynjolfsson, Stanford economist, argued in a February 2026 Financial Times guest article that US productivity growth of about 2.7% in 2025 showed AI gains starting to appear, with the economy "transitioning out of this investment phase into a harvest phase" (The Decoder, February 2026, secondary). The same coverage notes he co-founded Workhelix, a firm that helps companies adopt AI.
• David Autor, MIT labor economist, argues that AI could let workers with foundational training do expert work now reserved for elites, narrowing pay gaps, while stressing that "AI will not decide how AI is used" (Noema, February 2024).

Plain summary (needs sign-off from this side): Field studies keep finding real gains. Customer support agents resolved 15% more issues per hour with an AI assistant, and the least experienced workers gained the most. Consultants using GPT-4 finished more tasks, faster, at higher quality. The biggest gains go to less experienced workers, which is the opposite of past technology waves that rewarded the already skilled. US productivity growth has averaged 2.1% a year since late 2019, above the 1.5% of the previous business cycle, and the J-curve pattern says the large gains come after the reorganizing. Cheaper expertise means cheaper services, and cheaper services are how productivity has always raised living standards.

The case that it doesn't (worried this is oversold)

• Daron Acemoglu, MIT economist and 2024 Nobel laureate, estimates AI will add roughly 0.55% to total factor productivity (output growth not explained by more labor or capital) over a decade, saying of automation that "it's not as easy as sometimes it's presumed" (Fortune, June 2026).
• Anders Humlum and Emilie Vestergaard, studying 25,000 Danish workers in 11 AI-exposed occupations linked to official pay records, found "a precise null" effect of chatbot adoption on earnings and hours, ruling out average effects larger than 2% (Brookings, October 2026).

Plain summary (needs sign-off from this side): Task gains are not economy-wide gains. Studies pick well-defined tasks where AI shines, and on tasks outside its abilities AI made consultants worse. When experienced developers were measured instead of asked, they were slower with AI while believing they were faster. Economy-wide, the gains so far look small, wages for AI users have not moved, and the share of output paid to workers is at a record low. Meanwhile AI is pushing up the prices of chips, computers, software and electricity. Robert Gordon of Northwestern University bet Brynjolfsson that US productivity growth will average below 1.8% a year from 2020 to 2030 (Bloomberg Government, May 2026, secondary). Recent strength in the numbers may reflect data-center spending rather than AI making work more efficient (The Decoder, February 2026, secondary).

Common ground

• AI raises output per hour on many specific tasks, with the largest gains for less experienced workers (Brynjolfsson, Li and Raymond, 2025).
• On some tasks AI makes skilled people worse, and people misjudge their own AI speed-up (Mollick, September 2023; METR, July 2025).
• No one has yet shown AI lowering consumer prices economy-wide. In 2026, AI-related goods prices are rising (Governor Cook, September 2026).
• Whether gains reach wages depends on choices by firms and policymakers, not on the technology alone. Autor, Acemoglu and Federal Reserve Governor Cook all say some version of this.

Cruxes

1. Does US productivity growth stay above 1.8% a year through 2030, and can AI be shown to be a cause? This is the Brynjolfsson and Gordon bet. Check: BLS Productivity and Costs releases, annually through early 2031.
2. Do real wages start rising with productivity, or does the labor share keep falling? Productivity rose 2.2% in the year to mid-2026 while real hourly pay fell 0.1% (BLS, September 2026). Check: BLS labor share and real compensation, 2027 to 2028.
3. Do AI-adopting industries show lower price growth once the data-center building boom slows? Check: Federal Reserve and BEA price analyses, 2027 to 2029.

Probability estimates

No published estimate of the probability that AI produces broad productivity gains that reach wages and prices was found. The closest numbers are economists' point estimates of the size of the productivity effect, which disagree by about a factor of ten:

The estimates measure different things (total factor productivity, labor productivity, GDP, US or global), which explains part of the spread but not most of it.

Evidence confidence

Low for the scenario as a whole. Task-level evidence is solid (randomized and staggered field studies with thousands of workers). Economy-wide productivity evidence is early and disputed, and there is no evidence yet that gains are reaching wages or consumer prices.

Signposts

1. BLS nonfarm business productivity growth stays above 2% a year for 2026 and 2027 (the current business cycle has averaged 2.1% a year since late 2019, the same as the long-run average since 1947, per BLS, September 2026).
2. The BLS labor share (the part of output paid to workers) rises back above 55%. It was 52.8% in the second quarter of 2026, the lowest since records began in 1947 (BLS, September 2026).
3. Real hourly compensation grows by 1% or more over four quarters. It fell 0.1% in the year to mid-2026 (BLS, September 2026).
4. Census survey AI use passes 35% of US firms (about 21.5% in mid-2026, per The Productivity Institute).
5. A peer-reviewed study finds lower price growth in AI-adopting industries in US data, after controlling for other causes.

Evidence so far

• Customer support (staggered rollout, 5,172 agents): AI assistance raised issues resolved per hour by 15% on average. Less experienced agents gained in speed and quality; the most experienced saw small speed gains and small quality declines (Quarterly Journal of Economics, February 2025). The 2023 working paper version reported a 34% gain for novices (NBER, 2023).
• Boston Consulting Group (randomized, 758 consultants): on tasks within AI's abilities, consultants with GPT-4 completed 12.2% more tasks, 25.1% faster, with about 40% higher quality. On a task designed to sit outside AI's abilities, accuracy fell from 84% without AI to 60 to 70% with it (Ethan Mollick, co-author, September 2023).
• Experienced open-source developers (randomized, 16 developers, 246 tasks): AI tools made them 19% slower, while they believed AI had sped them up by 20% (METR, July 2025). METR's late-2025 follow-up could not produce a reliable estimate, partly because many developers refused to work without AI; METR believes developers are now sped up more than in early 2025 but says its data is "very weak evidence" of how much (METR, February 2026).
• Denmark, administrative pay records: no measurable effect of chatbot adoption on earnings or hours, with effects above 2% ruled out (Humlum and Vestergaard, Brookings, October 2026).
• US productivity statistics: nonfarm business productivity rose 2.2% in 2025 (annual average) and 2.8% from late 2024 to late 2025 (BLS, March 2026). In the year to the second quarter of 2026 it rose 2.2%, real hourly compensation fell 0.1%, and the labor share hit 52.8%, the lowest on record (BLS, September 2026). An analysis by The Productivity Institute says AI "could be contributing" to the revival, mainly via the tech sector, but does not quantify it (Martin Fleming, 2026).
• Federal Reserve views: Minneapolis Fed economists found that three broad productivity measures through early 2026 showed no boom, and that AI-related goods and software prices were rising (Minneapolis Fed, August 2026). Governor Lisa Cook said in September 2026 that AI "appears to be adding inflationary pressures" in the short run, expects productivity to bring "modest disinflation" within a few years, and warned that gains may not pass to wages if workers' bargaining power is low (Federal Reserve, September 2026).
• Self-reported gains: US generative AI users reported saving 5.4% of their work hours in late 2024 (St. Louis Fed, February 2025). Of small employer firms using AI, 71% report higher productivity (cited by Governor Cook, September 2026). Self-reports overstate gains, as the developer study shows.
• Exposure: the IMF estimated in 2024 that almost 40% of global employment is exposed to AI, about 60% in advanced economies and 26% in low-income countries, and that in most scenarios AI will likely worsen overall inequality (IMF, January 2024).

Existing coverage

No law blocks AI productivity gains. Competition (antitrust) law is the main existing tool for making cost savings reach prices, and labor law and collective bargaining shape whether they reach wages. Neither was designed with AI in mind. Official statistics agencies (BLS, BEA, Census) already measure productivity, prices, labor share and firm AI adoption; the Census Business Trends and Outlook Survey added AI questions, which the Federal Reserve uses to track adoption (Federal Reserve, April 2026). There is no official measure of how much of productivity growth comes from AI.

Triage

(Flipped for a good scenario: what is blocking it, and does removing the block need policy?)

1. Is it blocked by law? No. The blocks are slow adoption outside tech, the reorganizing firms must do first, weak worker bargaining power, and rising prices for AI inputs.
2. An action both sides accept? Yes, on measurement. Optimists argue official statistics undercount early AI gains (the J-curve), and skeptics argue the gains are being overstated from task studies and self-reports. Both positions rest on better data, and both camps already rely on administrative records (payroll, tax and productivity series) to argue their case. Funding official tracking of AI's effects on productivity, wages and prices by industry is acceptable to both and matches the measurement action on the companion card. Whether to steer AI toward helping workers rather than replacing them (Acemoglu, Autor) is contested and is not part of this result.
3. (Skipped.)

Result: Policy, act now on measurement of AI's productivity, wage and price effects. Communicate to set realistic expectations.

Communication note: Tell people the honest shape of it: AI measurably speeds up many specific tasks, especially for newer workers, but there is no clear economy-wide boom yet, wages have not followed, and in 2026 AI is raising some prices rather than lowering them. "Cheaper everything" is a forecast, not a finding.

Related scenarios

• Opposite in the same domain: "Mass job loss and widening inequality" (the same productivity, with gains flowing to capital).
• Combines with: "Concentrated power or AI-enabled surveillance" (who owns the gains); "AI's energy and resource demand outpaces its gains" (power and chip costs offsetting cheaper goods); "Personal tutors, care and freed time" (gains taken as time rather than money).
• Reduced by: "AI disappoints."

Sources

1. Allen, J. S., "Monitoring AI Adoption in the U.S. Economy," FEDS Notes, Federal Reserve Board, 3 April 2026. https://www.federalreserve.gov/econres/notes/feds-notes/monitoring-ai-adoption-in-the-u-s-economy-20260403.html
2. Penn Wharton Budget Model, "The Projected Impact of Generative AI on Future Productivity Growth," 8 September 2025. https://budgetmodel.wharton.upenn.edu/p/2025-09-08-the-projected-impact-of-generative-ai-on-future-productivity-growth/
3. Brynjolfsson, E., Rock, D. and Syverson, C., "The Productivity J-Curve: How Intangibles Complement General Purpose Technologies," SSRN (published in American Economic Journal: Macroeconomics, 2021). https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3266241
4. The Decoder, "Stanford's Brynjolfsson sees AI boosting US productivity, but he also co-founded an AI consulting firm," 16 February 2026 (secondary, on Brynjolfsson's Financial Times guest article). https://the-decoder.com/stanfords-brynjolfsson-sees-ai-boosting-us-productivity-but-he-also-co-founded-an-ai-consulting-firm/
5. Autor, D., "How AI Could Help Rebuild the Middle Class," Noema, 12 February 2024. https://www.noemamag.com/how-ai-could-help-rebuild-the-middle-class/
6. Lichtenberg, N., "Nobel laureate Daron Acemoglu on AI, productivity, capitalism and democracy," Fortune, 21 June 2026. https://fortune.com/2026/06/21/nobel-laureate-daron-acemoglu-ai-productivity-capitalism-democracy/
7. Humlum, A. and Vestergaard, E., "Still waters, rapid currents: Early labor market transformation under generative AI," Brookings, 6 October 2026. https://brookings.edu/articles/still-waters-rapid-currents-early-labor-market-transformation-under-generative-ai
8. Brynjolfsson, E., Li, D. and Raymond, L., "Generative AI at Work," Quarterly Journal of Economics 140(2), 889 to 942, February 2025. https://doi.org/10.1093/qje/qjae044
9. Brynjolfsson, E., Li, D. and Raymond, L., "Generative AI at Work," NBER Working Paper 31161, April 2023 (revised November 2023). https://www.nber.org/papers/w31161
10. Mollick, E., "Centaurs and Cyborgs on the Jagged Frontier," One Useful Thing, 16 September 2023 (summary by a co-author of Dell'Acqua et al., "Navigating the Jagged Technological Frontier," Harvard Business School Working Paper 24-013, later published in Organization Science). https://www.oneusefulthing.org/p/centaurs-and-cyborgs-on-the-jagged
11. METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity," 10 July 2025. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
12. METR (Becker, Rush, Cunningham, Rein, Mahamud), "We are Changing our Developer Productivity Experiment Design," 24 February 2026. https://metr.org/blog/2026-02-24-uplift-update/
13. Cook, L. D., "An Update on AI and the Economy," speech, Federal Reserve Board, 28 September 2026. https://www.federalreserve.gov/newsevents/speech/cook20260928a.htm
14. U.S. Bureau of Labor Statistics, "Productivity and Costs, Second Quarter 2026, Revised," 3 September 2026. https://www.bls.gov/news.release/pdf/prod2.pdf
15. U.S. Bureau of Labor Statistics, "Productivity and Costs, Fourth Quarter and Annual Averages 2025, Preliminary," 5 March 2026. https://www.bls.gov/news.release/archives/prod2_03052026.pdf
16. Fleming, M., "Q2 2026: U.S. Productivity Growth Revival Continues," The Productivity Institute, 2026. https://lab.productivity.ac.uk/insights/q2-2026-us-productivity-growth-revival-continues/
17. Acemoglu, D., "The Simple Macroeconomics of AI," NBER Working Paper 32487, May 2024. https://www.nber.org/papers/w32487
18. Boussour, L., "The productivity potential of GenAI," EY-Parthenon, 22 January 2024. https://www.ey.com/en_us/ai/productivity-potential-gen-ai
19. Goldman Sachs Research, "Generative AI could raise global GDP by 7%," April 2023. https://www.goldmansachs.com/insights/articles/generative-ai-could-raise-global-gdp-by-7-percent
20. Qatar News Agency, "IMF raises global economic growth forecast for 2026," 19 January 2026 (secondary, reporting the IMF World Economic Outlook Update). https://qna.org.qa/en/News-Area/News/2026-1/19/imf-raises-global-economic-growth-forecast-for-2026-amid-global-economic-turnaround
21. Bick, A., Blandin, A. and Deming, D., "The Impact of Generative AI on Work Productivity," Federal Reserve Bank of St. Louis, 27 February 2025. https://www.stlouisfed.org/on-the-economy/2025/feb/impact-generative-ai-work-productivity
22. Vanek Smith, S., "A $400 AI Bet That's a High-Stakes Wager on the Future of Work," Bloomberg Government, 11 May 2026 (secondary). https://news.bgov.com/artificial-intelligence/a-400-ai-bet-thats-a-high-stakes-wager-on-the-future-of-work
23. Horwich, J., "How is AI influencing interest rates? Investment, productivity, prices, and more," Federal Reserve Bank of Minneapolis, 3 August 2026. https://www.minneapolisfed.org/article/2026/how-is-ai-influencing-interest-rates-investment-productivity-prices-and-more
24. Georgieva, K., "AI Will Transform the Global Economy. Let's Make Sure It Benefits Humanity," IMF Blog, 14 January 2024. https://www.imf.org/en/Blogs/Articles/2024/01/14/ai-will-transform-the-global-economy-lets-make-sure-it-benefits-humanity

Sign-off record

• Optimistic case: not yet reviewed.
• Worried case: not yet reviewed.
• Changelog: 2026-10-09 first draft.

In one sentence: AI replaces large numbers of white-collar and eventually other workers faster than new work appears, unemployment rises sharply, and the gains flow mainly to the owners of AI and capital.

Preconditions

• AI becomes reliable enough to do whole jobs, not just tasks within jobs.
• Companies choose to replace workers rather than give existing workers more capable tools.
• New kinds of work appear more slowly than old work disappears.
• No policy (tax, training, income support) redistributes the gains.

Timing and reversibility

• Earliest plausible date: the most aggressive public warning says one to five years from May 2025. Skeptics see no economy-wide effect so far.
• Speed: slow-building, and already visible at the entry level.
• Reversibility: partial. Unemployment can recover, but a cohort that misses its first jobs loses years of skill and earnings, and the shortage of mid-career workers shows up a decade later.

The case that this happens

Short excerpts for orientation only. A person from this side will choose the final quotes at sign-off.

• Dario Amodei, CEO of Anthropic, told Axios in May 2025 that AI could eliminate half of entry-level white-collar jobs and push US unemployment to 10 to 20 percent within one to five years, and that companies and government should stop "sugar-coating" it.
• Erik Brynjolfsson and colleagues at the Stanford Digital Economy Lab, in the August 2026 update of their payroll study, found employment for 22 to 25 year olds in AI-exposed jobs 19% below where it would have been had it kept pace with less-exposed peers, with the gap widening steadily since 2025.

Plain summary (needs sign-off from this side): The early evidence is exactly what you would expect at the start of a large displacement. Companies stop hiring juniors in exposed fields before they fire anyone, so national unemployment stays flat while the damage builds underneath. Earlier automation mostly replaced physical and routine work. This wave targets thinking work, and it is improving faster than workers can retrain. Without deliberate policy, gains concentrate with the firms that own the models.

The case that it doesn't

• The Yale Budget Lab found in October 2025 no economy-wide link between AI exposure and changes in US employment or unemployment over the 33 months after ChatGPT launched, with jobs shifting only about one percentage point faster than during the early internet era. Later updates through early 2026 reported the same.
• David Autor, MIT labor economist, argues in Noema (February 2024) that rich countries will run out of workers before they run out of jobs, and that AI used well can extend expert-level decision-making to workers without degrees, rebuilding middle-class work rather than destroying it.

Plain summary (needs sign-off from this side): Every major technology has raised fears of mass unemployment, and every time new work appeared. Aging populations mean labor will be scarce, not surplus. The entry-level decline is real but has other plausible causes, and economy-wide data shows no break from normal patterns. The bigger opportunity is AI as a tool that lets more people do skilled work, which is a design choice, not a law of nature.

Common ground

• The entry-level hiring decline in AI-exposed fields is real and measured.
• Economy-wide unemployment has not risen because of AI as of late 2026.
• Whether AI replaces or augments workers depends partly on how companies choose to deploy it.
• Better data is needed, and both sides support ongoing public tracking.

Cruxes

1. Does the young-worker gap spread to experienced workers? If workers aged 35 and over in exposed jobs start losing ground, the "it's just the entry level" reading fails. Check: Stanford Digital Economy Lab updates through 2027.
2. Does economy-wide unemployment in exposed occupations break from trend? Check: Yale Budget Lab tracker, 2027.
3. Do productivity gains show up in wages, or only in profits? Check: labor share of income data, 2027 to 2028.

Probability estimates

No published estimate of the probability of mass AI-driven unemployment from a forecasting tournament or expert survey was found. What exists instead are point forecasts:

The 60 to 70 year gap between the last two answers, for nearly the same event, comes only from how the question was worded.

Evidence confidence

Medium for the entry-level effect (payroll data, several studies). Low for economy-wide projections.

Signposts

1. The Stanford young-worker gap passes 25%.
2. Experienced workers in exposed fields show a measurable employment gap.
3. US unemployment for college graduates rises above unemployment for non-graduates for a sustained period.
4. AI-cited layoffs exceed 1% of total US layoffs for a full year. (Challenger, Gray & Christmas reported about 116,000 AI-cited cuts from January to August 2026, per secondary coverage; verify.)

Evidence so far

• Employment for young software developers fell nearly 20% from its late-2022 peak (Stanford, per coverage of the study).
• Stanford: 19% gap for 22 to 25 year olds in exposed jobs (August 2026).
• No economy-wide unemployment effect found (Yale, through early 2026).

Existing coverage

Unemployment insurance, the WARN Act (60-day notice for mass layoffs), and general retraining programs. None is designed for a fast, technology-specific shift in white-collar work. No law requires disclosure of AI-driven job cuts.

Triage

1. Covered by law and enforced? Partly. Existing safety nets apply but weren't designed for this.
2. An action both sides accept? Yes. Both camps support better public tracking of AI's labor effects. Most also support help for workers in transition, though they disagree on scale.
3. (Skipped.)

Result: Policy, act now on measurement (disclosure of AI-attributed layoffs, funded public tracking). Prepare and watch on income support, triggered by signposts 2 and 3.

Communication note: Tell people the honest shape of it: no mass layoffs so far, a real and growing squeeze on first jobs, and a genuine disagreement about whether it spreads. Young people choosing fields of study need this most.

Related scenarios

• Opposite: "Broad productivity gains and cheaper goods."
• Combines with: concentrated power (who owns the gains), over-reliance and lost skills (the missing entry-level rung).

Sources

1. Axios in May 2025. axios.com, https://axios.com/2025/05/28/ai-jobs-white-collar-unemployment-anthropic
2. August 2026 update. digitaleconomy.stanford.edu, https://digitaleconomy.stanford.edu/app/uploads/2026/08/Canaries_August2026.pdf
3. Yale Budget Lab. budgetlab.yale.edu, https://budgetlab.yale.edu/research/tracking-impact-ai-labor-market
4. Noema (February 2024). noemamag.com, https://www.noemamag.com/how-ai-could-help-rebuild-the-middle-class/
5. AI Impacts survey. aiimpacts.org, https://aiimpacts.org/
6. LeadDev, coverage of the Stanford study's software developer findings (secondary). leaddev.com, https://leaddev.com/hiring/young-devs-steepest-losses-exposed-roles-stanford-study-finds
7. Grace et al., "Thousands of AI Authors on the Future of AI" (2024). arxiv.org, https://arxiv.org/abs/2401.02843
8. Federal Reserve, "Monitoring AI Adoption in the U.S. Economy" (Apr 2026). federalreserve.gov, https://www.federalreserve.gov/econres/notes/feds-notes/monitoring-ai-adoption-in-the-u-s-economy-20260403.html

Sign-off record

• Worried case: not yet reviewed.
• Optimistic case: not yet reviewed.
• Changelog: 2026-10-09 first draft.

In one sentence: AI speeds up scientific discovery in materials, weather, physics and energy systems enough that clean energy becomes cheaper and easier to run, and the climate gains outweigh AI's own footprint.

Design note: for a good-direction card, "worried" means worried this won't happen or is oversold, not worried that it will. This card is the mirror of "AI's energy and resource demand outpaces its gains," and the two share much of the same evidence.

Preconditions

• AI predictions (of materials, molecules, weather, plasma behavior) keep turning into things that work in a laboratory and then in a factory.
• Laboratories, manufacturers and grid operators can test and adopt what AI proposes fast enough to matter.
• Grids can connect new clean generation quickly. Today that is often the slowest step, not the science.
• Claims of climate benefit are measured and checked, so money flows to what actually works.

Timing and reversibility

• Earliest plausible date: some pieces are already here. AI weather forecasts went into operational use at the European Centre for Medium-Range Weather Forecasts in February and July 2025. AI-designed battery and grid materials reaching products is a 2030s question at the earliest.
• Speed: slow-building. A promising material still has to survive synthesis, manufacturing, cost and durability testing, which takes years.
• Reversibility: not applicable for a benefit. The relevant risk is that it is delayed or oversold, and that the promise is used to justify AI's energy use today.

The case that this happens (optimistic)

Short excerpts for orientation only. A person from this side will choose the final quotes at sign-off.

• Demis Hassabis, Google DeepMind CEO and 2024 Nobel laureate in chemistry, told Time in April 2025 that humanity's hardest problems, "whether that's climate or disease, will be helped by AI solutions." Time's reporter paraphrased his vision of an AI-enabled future as one where energy is zero-carbon and free.
• Alex Wissner-Gross, physicist and co-founder of Physical Superintelligence, argues in a February 2026 essay that physics is the bottleneck for chemistry, materials and energy: "Every domino downstream is waiting on the first one to fall." He discloses a board seat and financial interest in the company.

Plain summary (needs sign-off from this side): AI has already delivered real scientific results: AlphaFold won a Nobel Prize, AI weather models now beat traditional ones on many measures at a tiny fraction of the energy, and AI controlled the plasma in a real experimental fusion reactor. The same methods are now aimed at batteries, electrolytes, solar materials and grid operation, and early results match the best human designs. Solar and battery costs were already falling fast. AI adds better materials, better forecasts of wind and sun, and better grid management on top. Ramez Naam, a clean energy investor and writer, argues (April 2026) that when AI speeds up innovation in drugs, materials or clean energy, more than 90% of the benefit flows to customers worldwide. The International Energy Agency's most optimistic case has existing AI uses cutting more emissions by 2035 than data centers produce.

The case that it doesn't (worried this is oversold)

• Anthony Cheetham and Ram Seshadri, materials chemists at UC Santa Barbara, reviewed a sample of Google DeepMind's 2.2 million predicted crystals in Chemistry of Materials (April 2024) and found "scant evidence for compounds that fulfill the trifecta of novelty, credibility, and utility."
• Sara Beery, an MIT AI researcher, said of Google's AI co-scientist in TechCrunch (March 2025): "This preliminary tool, while interesting, doesn't seem likely to be seriously used."

Plain summary (needs sign-off from this side): The delivered results are real but narrow, and the headlines run well ahead of them. Predicting that a crystal could be stable is not discovering a useful material, and independent chemists found much of the flagship materials work was old compounds in new clothes or chemically implausible. A widely cited study claiming AI boosted materials discovery was disowned by MIT in May 2025 over the integrity of its data. A February 2026 review of 154 public claims about AI's climate benefits, commissioned by environmental groups, found 74% lacked verifiable evidence, and that companies blur small, proven machine learning tools with energy-hungry generative AI. Journalist James Temple compared (MIT Technology Review, April 2025) AI's climate promises to carbon offsets: pollution now, in exchange for benefits later that nobody is accountable for. Clean energy got cheap mostly without AI, and its biggest bottleneck is grid connection and permitting, which AI does not fix.

Common ground

• AlphaFold and AI weather forecasting are delivered results in daily use.
• Predicting a stable compound is a long way from a product; most predicted compounds will never be made.
• Solar costs fell by roughly a factor of five to eight between 2010 and 2020, before AI played any role (Naam, May 2020).
• The kind of AI that delivered most climate-relevant results so far (specialized forecasting and control models) is not the same as the large chat models driving data center growth.
• Climate-benefit claims should be measured and independently checked.

Cruxes

1. Do AI-predicted materials become real products? If an AI-discovered battery, solar or grid material is in commercial manufacture by 2030, the "it's just prediction" objection fails. Check: company and laboratory announcements, 2028 to 2030.
2. Do AI research agents produce discoveries that independent groups replicate? The co-scientist and Get Physics Done claims are so far capability claims, not replicated findings. Check: peer-reviewed literature, 2027 to 2028.
3. Do verified AI-enabled emissions cuts grow toward the IEA's optimistic case? Check: IEA Energy and AI updates, 2027; a repeat of the February 2026 claims review.

Probability estimates

No published estimate of the probability that AI makes clean energy meaningfully cheaper or science meaningfully faster was found. Related numbers:

The IEA's own authors noted, as reported by MIT Technology Review, that there is "no momentum" ensuring the adoption their optimistic number assumes.

Evidence confidence

Low for the scenario as a whole. Evidence is strong for specific tools (protein structure, weather forecasting), thin for AI-discovered materials reaching use, and nearly absent for AI measurably lowering the cost of clean energy.

Signposts

1. A manufacturer announces commercial production of a battery, solar or grid material that was first identified by an AI system.
2. A discovery credited mainly to an AI research agent (such as Google's co-scientist or Physical Superintelligence's tools) is published in a peer-reviewed journal and replicated by an independent laboratory.
3. A repeat review of AI climate-benefit claims finds more than half are backed by verifiable evidence (up from 26% in February 2026).
4. A second major national weather service adopts an AI model as its primary operational forecast.
5. Commonwealth Fusion Systems reports net energy gain from SPARC with AI simulation or control tools credited in the result.

Evidence so far

• Protein structure: AlphaFold won the 2024 Nobel Prize in Chemistry.
• Weather: Google DeepMind's GraphCast beat the leading traditional model on about 90% of 1,380 measures (Science, 2023). The European Centre for Medium-Range Weather Forecasts made its own AI ensemble operational on 1 July 2025, reporting gains of up to 20% on many measures and roughly 1,000 times less energy per forecast (ECMWF).
• Materials: GNoME predicted 2.2 million crystals, 380,000 of them calculated to be stable, and external researchers had made 736 of them in the lab (Nature, 2023; DeepMind, November 2023). A linked robotic laboratory reported 41 new compounds, but a later analysis concluded no new materials were discovered (Chemistry World, January 2024).
• Batteries: a University of Chicago model generated complete electrolyte recipes, several performing on par with the best known lithium metal battery electrolytes; the lead researcher said there is "still lots of work ahead" (TechXplore, May 2026). Argonne National Laboratory is building similar models for electrolytes and electrodes (Argonne).
• Fusion: AI controlled the magnetic fields holding plasma in a real tokamak (Nature, 2022). Since October 2025, DeepMind's open-source plasma simulator is being used by Commonwealth Fusion Systems to prepare its SPARC reactor (DeepMind; CFS). No fusion plant has yet put electricity on a grid (Earth911, 2026) (secondary).
• Grid and renewables: machine learning forecasts of wind output 36 hours ahead raised the value of 700 MW of Google's wind power by roughly 20% (DeepMind, February 2019).
• AI as scientist: Google's AI co-scientist (February 2025) independently proposed a bacterial gene-transfer mechanism that Imperial College researchers had already found but not yet published, and suggested drug candidates that showed activity in lab tests; Google notes small expert samples and expert guidance in the lab work (Google Research). Physical Superintelligence released its open-source Get Physics Done tool in March 2026 (The Innermost Loop) and launched publicly on 1 September 2026 with $58 million led by Breakthrough Energy Ventures (Unite.AI) (secondary). An AI system matched gold medalists at the 2025 International Physics Olympiad (arXiv). None of these is yet an independently replicated new discovery.
• Clean energy is getting cheaper and faster to build anyway: US developers planned a record 86 GW of new capacity for 2026, mostly solar (43.4 GW) and batteries (24 GW) (EIA, February 2026). Solar and storage were 91% of new US capacity in the first quarter of 2026 (pv magazine USA). Little of this is attributable to AI.

Existing coverage

No law blocks AI-assisted research or AI use in grid operation. The main blocks are practical: laboratory and manufacturing capacity, grid interconnection queues, and the lack of any standard for verifying AI climate-benefit claims. Some grid rules now push in the helpful direction: Texas Senate Bill 6 (June 2025) creates a paid program for large users who cut demand during emergencies, and Duke University researchers estimate the existing US grid could absorb about 98 GW of new flexible demand (2025) without new power plants.

Triage (flipped for a good scenario: what's blocking it, and does removing the block need policy?)

1. Is it blocked by law? No. It is slowed by verification, laboratory capacity and grid connection.
2. An action both sides accept? Yes, two. First, independent measurement and verification of AI science and climate-benefit claims: optimists want credit for real results, and skeptics want the unproven ones filtered out. Second, faster grid connection and flexible-load rules for clean generation, which help whether or not AI delivers. Open public data and open models (as ECMWF did with its weather model) are a likely third, though not yet tested with both camps.
3. (Skipped, because step 2 applies.)

Result: Policy, act now on verification standards for AI climate claims and on grid connection reform. Communicate to set expectations.

Communication note: Some of this is already true (better weather forecasts, protein structures, plasma control). Most of the rest is promising predictions, not delivered products. Clean energy is getting cheaper mainly for reasons that have nothing to do with AI, and a company's claim that AI will solve climate is a forecast, not a result.

Related scenarios

• Opposite in the same domain: "AI's energy and resource demand outpaces its gains" (the same data centers, judged by their cost rather than their output).
• Sibling: "Medical progress compressed into years" (the same discovery methods applied to biology).
• Combines with: "Broad productivity gains and cheaper goods" (cheaper energy lowers the cost of everything).
• Reduced by: "AI disappoints."

Sources

1. European Centre for Medium-Range Weather Forecasts, "ECMWF's ensemble AI forecasts become operational," 1 July 2025. https://www.ecmwf.int/en/about/media-centre/news/2025/ecmwfs-ensemble-ai-forecasts-become-operational
2. Time, interview with Demis Hassabis (TIME100), 15 April 2025. https://time.com/7277608/demis-hassabis-interview-time100-2025/
3. Alex Wissner-Gross, "Physical Superintelligence," The Innermost Loop, 15 February 2026. Author discloses a financial interest. https://theinnermostloop.substack.com/p/physical-superintelligence
4. Ramez Naam, "Common AI Narratives are Wrong (Video and Part 1)," 2 April 2026. https://www.rameznaam.com/p/optimistic-yet-contrarian-views-on
5. Cheetham and Seshadri, "Artificial Intelligence Driving Materials Discovery? Perspective on the Article: Scaling Deep Learning for Materials Discovery," Chemistry of Materials 36(8), 8 April 2024. https://pmc.ncbi.nlm.nih.gov/articles/PMC11044265
6. TechCrunch, "Experts don't think AI is ready to be a 'co-scientist'," 5 March 2025. https://techcrunch.com/2025/03/05/experts-dont-think-ai-is-ready-to-be-a-co-scientist
7. TechCrunch, "MIT disavows doctoral student's paper on AI productivity benefits," 17 May 2025. https://techcrunch.com/2025/05/17/mit-disavows-doctoral-students-paper-on-ai-productivity-benefits
8. Stand.earth, Beyond Fossil Fuels, Friends of the Earth and partners, "The AI Climate Hoax" (report by Ketan Joshi), February 2026. An advocacy coalition's review. https://stand.earth/press-releases/report-exposes-big-techs-ai-climate-hoax-74-of-industrys-claims-about-ais-climate-benefits-are-unproven/
9. James Temple, "Why the climate promises of AI sound a lot like carbon offsets," MIT Technology Review, 10 April 2025. https://www.technologyreview.com/2025/04/10/1114912/why-the-climate-promises-of-ai-sound-a-lot-like-carbon-offsets/
10. DeepLearning.AI, The Batch, summary of the IEA Energy and AI report, 4 June 2025 (secondary). https://www.deeplearning.ai/the-batch/the-international-energy-agency-examines-the-energy-costs-and-potential-savings-of-the-ai-boom
11. Nobel Prize in Chemistry 2024, press release, 9 October 2024. https://www.nobelprize.org/prizes/chemistry/2024/press-release/
12. Lam et al., "Learning skillful medium-range global weather forecasting" (GraphCast), Science, 2023. https://www.science.org/doi/10.1126/science.adi2336
13. Merchant et al., "Scaling deep learning for materials discovery," Nature, 29 November 2023. https://www.nature.com/articles/s41586-023-06735-9
14. Google DeepMind, "Millions of new materials discovered with deep learning," 29 November 2023. https://deepmind.google/discover/blog/millions-of-new-materials-discovered-with-deep-learning/
15. Chemistry World, "New analysis raises doubts over autonomous lab's materials 'discoveries'," 16 January 2024. https://chemistryworld.com/news/new-analysis-raises-doubts-over-autonomous-labs-materials-discoveries/4018791.article
16. TechXplore, "AI generates full battery electrolyte recipes," 30 May 2026, reporting Kim et al., JACS Au. https://techxplore.com/news/2026-05-ai-generates-full-battery-electrolyte.html
17. Argonne National Laboratory, "Building AI foundation models to accelerate the discovery of new battery materials." https://www.anl.gov/article/building-ai-foundation-models-to-accelerate-the-discovery-of-new-battery-materials
18. Degrave et al., "Magnetic control of tokamak plasmas through deep reinforcement learning," Nature, 2022. https://www.nature.com/articles/s41586-021-04301-9
19. Google DeepMind, "Bringing AI to the next generation of fusion energy," 16 October 2025. https://deepmind.google/blog/bringing-ai-to-the-next-generation-of-fusion-energy/
20. Commonwealth Fusion Systems, SPARC. https://cfs.energy/technology/sparc/
21. Earth911, "The State of Fusion Energy in 2026" (secondary). https://earth911.com/eco-tech/the-state-of-fusion-energy-in-2026-real-reactors-real-grids-real-caveats/
22. Google DeepMind, "Machine learning can boost the value of wind energy," 26 February 2019. https://deepmind.google/discover/blog/machine-learning-can-boost-the-value-of-wind-energy/
23. Google Research, "Accelerating scientific breakthroughs with an AI co-scientist," 19 February 2025. Company claims. https://research.google/blog/accelerating-scientific-breakthroughs-with-an-ai-co-scientist/
24. Alex Wissner-Gross, "The First Open-Source Agentic AI Physicist," The Innermost Loop, 15 March 2026. Author discloses a financial interest. https://theinnermostloop.substack.com/p/the-first-open-source-agentic-ai
25. Unite.AI, "Physical Superintelligence raises $58M seed round," September 2026 (secondary). https://unite.ai/physical-superintelligence-raises-58m-seed-round
26. "Physics Supernova: AI Agent Matches Elite Gold Medalists at IPhO 2025," arXiv, 2025. https://arxiv.org/pdf/2509.01659
27. US Energy Information Administration, "New U.S. electric generating capacity expected to reach a record high in 2026," 20 February 2026. https://www.eia.gov/todayinenergy/detail.php?id=67205
28. pv magazine USA, "Solar and storage account for 91% of new U.S. grid capacity," 22 July 2026, reporting SEIA data. https://pv-magazine-usa.com/2026/07/22/solar-and-storage-account-for-91-of-new-u-s-grid-capacity-in-first-half-of-2026/
29. Bracewell, "Texas Senate Bill 6 ushers in major overhaul of large load interconnection and grid access rules," July 2025. https://www.bracewell.com/resources/texas-senate-bill-6-ushers-in-major-overhaul-of-large-load-interconnection-and-grid-access-rules/
30. Duke University Nicholas Institute, "Rethinking Load Growth," 2025. https://nicholasinstitute.duke.edu/articles/ai-and-data-center-power-demand-skyrockets-flexible-load-integration-becomes-critical
31. Ramez Naam, "Solar's Future is Insanely Cheap (2020)," 14 May 2020. https://www.rameznaam.com/p/solars-future-is-insanely-cheap-2020

Sign-off record

• Optimistic case: not yet reviewed.
• Worried case: not yet reviewed.
• Changelog: 2026-10-09 first draft.

In one sentence: The build-out of AI data centers grows electricity, water and emissions faster than efficiency and clean energy can offset, locking in new fossil fuel plants and pushing costs onto households and local communities while the benefits go mainly to the companies building AI.

Preconditions

• Demand for AI computing keeps growing faster than the energy used per task falls (the rebound effect, explained below).
• Clean generation and transmission can't be connected fast enough where data centers cluster, so natural gas fills the gap.
• Grid upgrade and capacity costs are spread across all customers rather than charged to the data centers that cause them.
• Promised climate benefits from AI stay unmeasured or small.

Timing and reversibility

• Earliest plausible date: already under way in some places. US electricity prices rose 6.9% in 2025 against 2.9% overall inflation (CNBC), and Google, Microsoft and Amazon all reported rising absolute emissions for 2025.
• Speed: slow-building but fast by energy-system standards. The International Energy Agency projects global data center electricity roughly doubling between 2025 and 2030.
• Reversibility: partly irreversible. A new gas plant runs for decades, and carbon dioxide stays in the atmosphere for centuries. Prices and water use can recover if rules change or demand falls short.

The case that this happens

Short excerpts for orientation only. A person from this side will choose the final quotes at sign-off.

• Sasha Luccioni (Hugging Face), Emma Strubell (Carnegie Mellon) and Kate Crawford argue in a 2025 FAccT paper that efficiency gains invite more use, and that "a narrow focus on direct emissions misrepresents AI's true climate footprint."
• Monitoring Analytics, the independent market monitor for PJM (the largest US grid region), identified data center load growth as the primary reason for record capacity prices that all customers in the region pay (Utility Dive, January 2026).

Plain summary (needs sign-off from this side): The efficiency gains are real, and total use is rising anyway, which is exactly what Jevons paradox predicts (in 1865 the economist William Stanley Jevons noticed that more efficient steam engines made Britain burn more coal, not less). Google, Microsoft and Amazon each reported double-digit emissions increases for 2025, and Meta quit a major renewable energy pledge after funding new gas plants. Hundreds of gigawatts of new gas capacity are being planned, much of it for data centers. Households in the largest US grid region are already paying billions more because of expected data center demand, and residents near data centers are fighting over water. The climate benefits AI companies promise are mostly unverified and arrive later, while the emissions happen now.

The case that it doesn't

• Bill Gates said in June 2024 that data centers would add at most about 6% to energy demand and probably 2% to 2.5%, that AI will more than offset it, and: "Let's not go overboard on this." (secondary)
• Jonathan Koomey, a longtime researcher of computing's energy use, warns regulators in a March 2026 report with Gridlab that data center forecasts are "highly uncertain" and that analysts habitually overestimate information technology's energy effects.

Plain summary (needs sign-off from this side): Data centers that do computing are about 1.5% of the world's electricity. The energy used per AI answer is tiny and falling faster than for any energy technology in history, according to the International Energy Agency. Many announced projects will never be built, so forecasts based on interconnection requests overstate demand, as they did in the dot-com era. Meanwhile the grid is changing quickly: the US is on track for a record year of new capacity, mostly solar and batteries, and the existing grid can absorb far more demand if data centers agree to cut use in the few peak hours each year. States are already making data centers pay their own way. The real constraint is how fast power can be connected, not whether it exists.

Common ground

• Data center electricity use is rising fast, and the 2025 figures are measured, not projected.
• The energy per AI query is small and falling quickly; individual use is not where this is decided.
• The largest AI companies' absolute emissions rose in 2025 despite their climate goals.
• Long-range demand forecasts are very uncertain, and some proposed projects will not be built.
• In principle, new large customers should pay for the grid upgrades built for them; the disagreement is over the details.
• Better public data on data center energy and water use would help everyone.

Cruxes

1. Where data centers cluster, does clean power keep pace, or does gas fill the gap? If gas-fired capacity built for data centers keeps growing faster than clean capacity in those regions, the worried side is right about lock-in. Check: Global Energy Monitor tracker and EIA capacity data, 2027.
2. Do the new large-load tariffs stop costs shifting to households? Check: Virginia's new data center rate class in its first year (2027) and the next PJM capacity auction results (mid-2027).
3. Do verified AI-enabled emissions cuts appear at a scale comparable to data center emissions? Check: IEA Energy and AI updates, 2027.

Probability estimates

No published estimate of the probability that AI's resource demand outpaces its gains was found. Related projections:

Read these with care: none of these is a probability, the Berkeley Lab range spans nearly a factor of two, and the IEA's optimistic offset depends on adoption its own authors said has "no momentum" behind it.

Evidence confidence

Medium. Electricity use, prices, corporate emissions and gas build-out are measured and well documented for 2025. Whether demand will keep outpacing clean supply and efficiency through 2030, and how large AI's offsetting benefits are, is genuinely uncertain.

Signposts

1. Google, Microsoft or Amazon reports another rise in absolute emissions in its 2027 sustainability report (covering 2026), showing 2025 was a trend rather than a peak.
2. Solar and batteries fall below half of planned new US generating capacity in an EIA annual outlook (they were 79% of planned 2026 additions).
3. The next PJM capacity auction again clears at its price cap with the market monitor attributing 40% or more of the cost to data centers.
4. US electricity prices rise faster than overall inflation again in 2026 (in 2025 they rose 6.9% against 2.9%).
5. A public water authority restricts household water use and names data center demand as a cause.

Evidence so far

• Electricity: data centers worldwide used roughly 485 TWh in 2025, with AI-focused facilities growing about 50% in the year (IEA). An IEA presentation in June 2026 said data center demand grew by almost 20% in 2025, and that 19 of the 21 large AI data centers under construction in the US are within 65 km of a city of at least 100,000 people (IEA, 11 June 2026).
• Company emissions (2025, reported mid-2026): Google's electricity use rose 37% and its emissions 18%, now more than 80% above its 2019 baseline; it calls its 2030 goal "intentionally aspirational" (Trellis) (secondary). Microsoft's emissions rose about 25%, to 20.3 million tonnes, about 58% above its 2020 baseline (The Next Web) (secondary). Amazon's rose 16%, with electricity-related emissions up 34% (ESG Dive) (secondary).
• Gas build-out: Meta left the RE100 renewable electricity pledge in July 2026; the Climate Group said Meta could no longer meet its criteria "due to investments made in new gas power," including gas plants for its Hyperion campus in Louisiana (The Next Web) (secondary; reported plant counts vary). US gas capacity in development rose 50% in six months to 378 GW by mid-2026, and nearly two-thirds of Texas's pipeline (77 GW) is planned to power data centers directly (Global Energy Monitor, August 2026). Much of this may never be built.
• Bills: in PJM, the largest US grid region (mid-Atlantic and Midwest), the market monitor attributed $6.5 billion of a record $16.4 billion capacity auction (December 2025) to data centers, and $21.3 billion of $47.2 billion across the last three auctions (Utility Dive). Expected data center demand was "a primary reason" for $23 billion in customer price increases running through at least 2028 (Fortune, July 2026). Utilities requested a record $31 billion in rate increases in 2025 (PowerLines).
• Water: Google's water consumption reached 10.9 billion gallons in 2025, up 34% (Google 2026 Environmental Report, per press coverage). A 2025 Bloomberg analysis found more than two-thirds of US data centers built or in development since 2022 are in drought-prone areas (KUNC, June 2025) (secondary). Tucson's council rejected the Amazon-linked Project Blue over water and energy concerns in August 2025 (AZFamily); it is now being built outside city limits on state-approved wells and could use up to 31 million gallons a year (KJZZ, May 2026). Families near Meta's Newton County, Georgia data center sued in September 2026, alleging construction fouled their wells; Meta says its own study found no effect (Atlanta Journal-Constitution).
• The other side of the ledger: Google reports the energy per median text prompt fell 33-fold in twelve months to 0.24 watt-hours (company figures, not independently verified) (Google Cloud, August 2025). The US expects a record 86 GW of new generating capacity in 2026, 79% solar and batteries and only 6.3 GW gas (EIA, February 2026). The IEA noted in June 2026 that load growth "does not necessarily raise per unit prices" and that its 2030 outlook is unchanged because supply chains are slowing the build-out (IEA). Duke University researchers estimate the existing US grid could absorb about 98 GW of new demand if large users cut use in about 0.5% of hours (Duke Nicholas Institute).

Existing coverage

Several rules now apply, mostly new and not yet tested:

• Texas Senate Bill 6 (signed June 2025) lets the grid operator cut power to new loads of 75 MW or more in emergencies, creates a paid demand-reduction program, and requires disclosure of backup generation (Bracewell). Detailed rules are being written through 2026 (Foley & Lardner).
• Ohio regulators approved AEP Ohio's data center tariff in July 2025: new data centers over 25 MW pay for at least 85% of the capacity they reserve for up to 12 years, whether they use it or not (POWER).
• Virginia regulators created a separate rate class for customers of 25 MW or more in November 2025, with minimum payment of 85% of contracted transmission and distribution demand and 60% of generation demand, effective January 2027 (E&E News).
• Federal: after a Department of Energy directive in October 2025, the Federal Energy Regulatory Commission ordered all six regional grid operators in June 2026 to show their large-load rules are adequate or propose reforms, including cost transparency (Orrick, July 2026).
• European Union: the Energy Efficiency Directive requires data centers with significant energy use to report energy and water performance to a public EU database; a common rating scheme was proposed in September 2026 (European Commission).
• Ordinary air, water-permit and local zoning rules apply to gas plants and data centers. The US has no federal requirement for data centers to report energy or water use, and no rule governs whether companies' climate-benefit claims for AI are accurate.

Triage

1. Covered by law and enforced? Partly. Cost-allocation rules exist in a few states and are just taking effect; federal reform is mid-process. Emissions and water outcomes are largely unregulated as such.
2. An action both sides accept? Yes. Both camps support (a) making large new loads pay the grid costs they cause, as Ohio and Virginia now do, (b) flexible connection, where data centers get connected faster in exchange for cutting use at peak, which optimists see as the cheapest way to serve AI and skeptics see as protection against overbuilding, and (c) mandatory public reporting of data center energy and water use, which Koomey's forecast critique and Luccioni's rebound critique both depend on.
3. (Skipped, because step 2 applies.)

Result: Policy, act now on cost allocation, flexible connection and reporting. Prepare and watch on gas lock-in, tied to signposts 1 and 2.

Communication note: Your own AI use is a rounding error; the buildings are what matter. Data centers are a small share of world electricity but a large and fast-growing share in particular places, and that is where bills and water fights show up. Both "AI will boil the oceans" and "efficiency solves it" leave out half the numbers.

Related scenarios

• Opposite in the same domain: "Faster discovery and cheaper clean energy" (the same build-out, judged by what it produces).
• Combines with: "Concentrated power or AI-enabled surveillance" (who captures the gains and who pays the costs).
• Reduced by: "AI disappoints" (though a sudden pullback could leave customers paying for plants and lines built for demand that never came).

Sources

1. CNBC, "Electricity prices rising by double the rate of inflation," 12 February 2026. https://www.cnbc.com/2026/02/12/electricity-price-data-center-ai-inflation-goldman.html
2. Luccioni, Strubell and Crawford, "From Efficiency Gains to Rebound Effects: The Problem of Jevons' Paradox in AI's Polarized Environmental Debate," FAccT 2025 (arXiv, January 2025, revised June 2025). https://arxiv.org/abs/2501.16548
3. Utility Dive, PJM capacity auction and data center costs, 7 January 2026. https://www.utilitydive.com/news/data-centers-pjm-capacity-auction/808951/
4. eceee, republishing The Guardian, "AI will be help rather than hindrance in hitting climate targets, Bill Gates says," 27 June 2024 (secondary). https://www.eceee.org/all-news/news/ai-will-be-help-rather-than-hindrance-in-hitting-climate-targets-bill-gates-says/
5. Koomey Analytics and Gridlab, "Separating fact from fiction in data center electricity forecasts: A guide for regulators," 23 March 2026. https://www.koomey.com/koomey_blog/new-report-out-today-separating-fact-from-fiction-in-data-center-electricity-forecasts-a-guide-for-regulators/
6. American Public Power Association, summary of IEA Energy and AI report, 11 April 2025 (secondary). https://www.publicpower.org/periodical/article/electricity-demand-data-centers-worldwide-set-more-double-2030-iea
7. International Energy Agency, Key questions on energy and AI, executive summary, 2026. https://www.iea.org/reports/key-questions-on-energy-and-ai/executive-summary
8. Lawrence Berkeley National Laboratory, 2024 United States Data Center Energy Usage Report, December 2024. https://energyanalysis.lbl.gov/publications/2024-lbnl-data-center-energy-usage-report
9. James Temple, "Why the climate promises of AI sound a lot like carbon offsets," MIT Technology Review, 10 April 2025. https://www.technologyreview.com/2025/04/10/1114912/why-the-climate-promises-of-ai-sound-a-lot-like-carbon-offsets/
10. International Energy Agency (Natalie Kauf), "Key questions on energy and AI," presentation, 11 June 2026, hosted by Asia Clean Energy Forum. https://asiacleanenergyforum.adb.org/wp-content/uploads/2026/06/Key-questions-on-energy-and-AI.pdf
11. Trellis, "Google's AI expansion weighs heavily on climate goals," 30 June 2026 (secondary). https://trellis.net/article/googles-ai-expansion-weighs-heavily-on-climate-goals/
12. The Next Web, "Microsoft promised to be carbon negative. Its emissions just jumped 25%," 14 July 2026 (secondary). https://thenextweb.com/news/microsoft-emissions-25-percent-ai-data-centres
13. ESG Dive, "Amazon 2025 emissions jump as AI brings 'momentum and complexity'," 20 July 2026 (secondary). https://www.esgdive.com/news/amazon-2025-emissions-jump-as-ai-brings-momentum-and-complexity-report/
14. The Next Web, "Meta leaves RE100 clean energy pact after a decade as its gas buildout for AI outpaces renewables," 24 July 2026 (secondary). https://thenextweb.com/news/meta-leaves-re100-clean-energy-gas-ai-data-centres
15. Global Energy Monitor, "US gas power proposals tied to data centers nearly double in six months," August 2026. https://globalenergymonitor.org/research/us-gas-power-proposals-tied-data-centers-nearly-double-six-months
16. Fortune (republished from The Conversation, by Theodore J. Kury), "Data centers have already hiked electricity prices on the public by $23 billion," 14 July 2026. https://fortune.com/2026/07/14/data-centers-23-billion-electricity-bills/
17. PowerLines, "Utilities Requested Record $31 Billion in Rate Increases in 2025," January 2026. https://powerlines.org/utilities-requested-record-31-billion-in-rate-increases-in-2025-double-that-of-2024/
18. Google, 2026 Environmental Report. The 10.9 billion gallon figure is from press coverage of the report. https://sustainability.google/google-2026-environmental-report/
19. KUNC, "AI data centers growing in drought-stricken Mountain West," reporting a Bloomberg analysis, 2 June 2025 (secondary). https://www.kunc.org/2025-06-02/ai-data-centers-growing-drought-stricken-mountain-west-analysis
20. AZFamily, "Tucson rejects Amazon-linked data center amid water, energy concerns," 8 August 2025. https://www.azfamily.com/2025/08/09/tucson-rejects-amazon-linked-data-center-amid-water-energy-concerns/
21. KJZZ, "Project Blue data center could use up to 31 million gallons of water per year," 29 May 2026. https://www.kjzz.org/fronteras-desk/2026-05-29/project-blue-data-center-could-use-up-to-31-million-gallons-of-water-per-year
22. Atlanta Journal-Constitution, "Georgia families sue Meta, accuse its data center of tainting their water," 1 September 2026. https://www.ajc.com/news/2026/09/georgia-families-sue-meta-accuse-its-data-center-of-tainting-their-water/
23. Google Cloud, "Measuring the environmental impact of AI inference," 21 August 2025. Company figures, not independently verified. https://cloud.google.com/blog/products/infrastructure/measuring-the-environmental-impact-of-ai-inference
24. US Energy Information Administration, "New U.S. electric generating capacity expected to reach a record high in 2026," 20 February 2026. https://www.eia.gov/todayinenergy/detail.php?id=67205
25. Duke University Nicholas Institute, "Rethinking Load Growth," 2025. https://nicholasinstitute.duke.edu/articles/ai-and-data-center-power-demand-skyrockets-flexible-load-integration-becomes-critical
26. Bracewell, "Texas Senate Bill 6 ushers in major overhaul of large load interconnection and grid access rules," July 2025. https://www.bracewell.com/resources/texas-senate-bill-6-ushers-in-major-overhaul-of-large-load-interconnection-and-grid-access-rules/
27. Foley & Lardner, "Public Utility Commission of Texas issues proposed rules for large load interconnection," March 2026. https://www.foley.com/insights/publications/2026/03/public-utility-commission-of-texas-issues-proposed-rules-for-large-load-interconn/
28. POWER Magazine, "Regulator approves AEP Ohio's landmark data center tariff," July 2025. https://www.powermag.com/regulator-approves-aep-ohios-landmark-data-center-tariff/
29. E&E News, "Virginia to treat data centers differently on the grid," 1 December 2025. https://www.eenews.net/articles/virginia-to-treat-data-centers-differently-on-the-grid/
30. Orrick, "FERC Show Cause Orders Signal Broad Reform to Large Load Interconnection Policies," 6 July 2026. https://www.orrick.com/en/Insights/2026/07/FERC-Show-Cause-Orders-Signal-Broad-Reform-to-Large-Load-Interconnection-Policies
31. European Commission, "Energy performance of data centres." https://energy.ec.europa.eu/topics/energy-efficiency/energy-efficiency-targets-directive-and-rules/energy-efficiency-directive/energy-performance-data-centres_en

Sign-off record

• Worried case: not yet reviewed.
• Optimistic case: not yet reviewed.
• Changelog: 2026-10-09 first draft.

In one sentence: AI helps people check facts, correct false beliefs, find common ground across disagreements and forecast events more accurately, so that societies argue from a more shared and more accurate picture of the world.

Design note: for a good-direction card, "worried" means worried this won't happen or is oversold, not worried that it will. This card bundles four strands that move at different speeds: correcting false beliefs, helping groups deliberate, crowd fact-checking, and forecasting (including weather). The evidence is strongest for weather and weakest for society-wide effects.

Preconditions

• AI assistants become accurate and well-sourced enough on news and contested facts that relying on them improves people's beliefs rather than adding new errors.
• The systems people actually use are tuned to tell the truth, not to agree with the user or maximize engagement.
• Platforms adopt tools that reward agreement across divides (often called "bridging" algorithms: content is promoted only when people who usually disagree both rate it helpful) and keep them hard to game.
• People gain lasting judgment from these tools, not just better answers while the tool is switched on.
• AI forecasts are published openly and scored against what actually happens, so trust can be earned.

Timing and reversibility

• Earliest plausible date: parts are already here. Machine-learning weather forecasts became part of a major operational forecast service in February 2025. The forecasting research group FRI reported in July 2026 that the best AI systems are now statistically indistinguishable from top human forecasters on its benchmark. A society-wide improvement in shared understanding has no date anyone has published.
• Speed: gradual. Each strand depends on platform choices, product design and slow changes in habits.
• Reversibility: not applicable for a benefit. The relevant risk is that the gains are outweighed by the same tools making things worse (see the opposite scenario), or that companies tune assistants toward flattery because flattery sells.

The case that this happens (optimistic)

Short excerpts for orientation only. A person from this side will choose the final quotes at sign-off.

• Gordon Pennycook, a co-author of the 2024 Science study in which conversations with GPT-4 Turbo cut belief in conspiracy theories by about 20% for at least two months, said the result shows "evidence matters much more than we thought it did" (Cornell, September 2024). His co-author David Rand said such models have "the potential to counter conspiracies at a massive scale".
• Michael Henry Tessler, Christopher Summerfield and colleagues at Google DeepMind reported in Science in October 2024 that an AI mediator (the "Habermas Machine") wrote group statements that 5,734 UK participants preferred to those written by human mediators, and that groups moved closer together after using it while minority views were still reflected. Audrey Tang, Taiwan's former digital minister, has long argued for bridging tools that surface "uncommon ground" rather than amplifying the loudest voices.

Plain summary (needs sign-off from this side): For the first time there is controlled evidence that patient, personalized, fact-based conversation can change entrenched false beliefs, and that it works because of the facts, not because people trust machines. AI mediators can help groups find statements they genuinely share. Crowd fact-checking that requires agreement across political lines measurably slows the spread of false posts. AI weather models now beat the world's best physics models on most measures, and AI forecasters have caught up with the best human forecasters on a public benchmark. Put together, these tools could give people a cheaper, faster, more accurate shared picture of reality than any previous technology.

The case that it doesn't (worried this is oversold)

• Griet De Craen of Belgian broadcaster VRT, a partner in the largest study of AI assistants and news, said in October 2025 that "the way AI chatbots currently handle news is not sufficiently reliable". The study, led by the BBC with 22 public broadcasters, found that 45% of more than 3,000 answers from four major assistants had at least one significant issue.
• Myra Cheng, Dan Jurafsky and colleagues at Stanford reported in Science in March 2026 that 11 leading models affirmed users' actions 49% more often than humans did, and that people trusted and preferred the flattering models, which gives companies a commercial reason to keep them that way. Zhivar Sourati, Morteza Dehghani and colleagues argue in Trends in Cognitive Sciences (2026) that heavy reliance on the same few models risks standardizing how people write and reason.

Plain summary (needs sign-off from this side): The headline studies are short, controlled experiments run by researchers with carefully prompted models. What people actually use day to day gets the news wrong often, tends to agree with whoever is asking, and pushes everyone toward the same phrasing and the same views. AI fact-checks can make people doubt true headlines, and help from AI in spotting fakes can fade or even reverse once the tool is taken away. The same ability that lets AI talk people out of false beliefs can talk them into new ones, and the most persuasive settings are also the least accurate. Better forecasting and better weather are real, but they do not add up to a public that understands the world better.

Common ground

• In controlled studies, fact-based AI dialogue reduces conspiracy belief, and the effect lasted at least two months.
• AI weather models are already in operational use and match or beat physics models on many measures.
• Assistants in everyday use still make frequent, significant errors on news.
• Sycophancy (excessive agreement with the user) is real, measurable and commercially rewarded.
• Results depend heavily on design: what the system is told to do, whether it cites sources, and whether people learn from it or lean on it.

Cruxes

1. Do people get better at judging information on their own, or only while the AI is helping? One month-long study found a 21% gain with AI help but a 15.3% decline in unaided performance by week four. Replications with larger samples would settle this. Check: follow-ups to Rani et al. (CHI 2026) and the Costello, Pennycook and Rand program, 2027.
2. Do the assistants people actually use get accurate on news? Check: the next round of the EBU and BBC News Integrity study, 2026 to 2027.
3. Does AI forecasting hold its parity against a fresh round of human superforecasters? FRI's comparison still uses superforecaster predictions collected in 2024, and it planned a new round for fall 2026. Check: ForecastBench and Metaculus FutureEval results, 2027.

Probability estimates

No published estimate of the probability that AI improves society-wide shared understanding was found. The closest published, scorable numbers concern AI forecasting:

Good Judgment, which recruits superforecasters, has disputed earlier parity claims as small-sample noise (spring 2026). Note that forecasting accuracy on benchmark questions is only one piece of this scenario.

Evidence confidence

Medium. Several of the key studies are large, preregistered experiments published in Science and PNAS, and weather results are verified in daily operation. Confidence is low that these lab and benchmark results translate into better public understanding, because almost no study measures that directly.

Signposts

1. An AI system beats the fresh superforecaster round on ForecastBench with statistical significance (one-sided p below 0.05), or bots beat Metaculus Pros with statistical significance in a full season.
2. The next EBU and BBC study finds the share of AI news answers with significant issues below 20%, down from 45% in October 2025.
3. An independent study of AI-written Community Notes finds they appear faster than human-written notes with no drop in cross-partisan helpfulness ratings.
4. A preregistered study shows people who used an AI fact-checker for a month are better at judging new claims without it (the opposite of the Rani et al. result).
5. Warning sign: a field study of a real population finds AI assistant use measurably narrows the range of political opinions people hold.

Evidence so far

• Correcting false beliefs: conversations with GPT-4 Turbo reduced conspiracy belief by about 20% among 2,190 believers, lasting at least two months (Costello, Pennycook and Rand, Science, September 2024). A follow-up with 955 participants found the effect was the same whether the bot was presented as an AI or a human expert (PNAS Nexus, October 2025).
• But not always: ChatGPT fact-checks identified 90% of false headlines yet did not improve people's ability to tell true from false, and lowered belief in true headlines it mislabeled (DeVerna et al., PNAS, December 2024). In a month-long study of 67 people, AI help raised accuracy by 21% in the moment but unaided accuracy fell 15.3% by week four (Rani et al., CHI 2026).
• Deliberation: the Habermas Machine's statements were preferred over human mediators' by 5,734 UK participants, and results replicated in a representative virtual citizens' assembly (Tessler et al., Science, October 2024). Its authors said it cannot fact-check or moderate discussion, and DeepMind had no plans to release it (MIT Technology Review, October 2024, secondary).
• Crowd fact-checking: across 40,078 posts on X, an attached Community Note reduced reposts by 46.1% after attachment, but only 11.6% over the post's lifetime because notes arrive late (Slaughter et al., PNAS, September 2025). An advocacy group found accurate notes were not shown on 209 of 283 misleading election posts it sampled (CCDH via AP, October 2024, secondary). Simulations show coordinated raters can suppress helpful notes (Truong et al., 2025). Meta replaced US third-party fact-checking with Community Notes in January 2025, and X began piloting AI-written notes, still rated by humans, in July 2025 (secondary), drawing on a June 2025 design paper.
• Persuasion cuts both ways: across 76,977 participants (UK, per the Oxford announcement) and 19 models, the techniques that made AI more persuasive also made its claims less accurate (Hackenburg et al., Science, December 2025).
• Homogenization: AI story ideas made individual stories better but more similar to one another (Doshi and Hauser, Science Advances, July 2024). Different models give strikingly similar answers to open-ended questions, a NeurIPS 2025 best paper finding (University of Washington, January 2026).
• Sycophancy in a real product: OpenAI rolled back a GPT-4o update in April 2025 after it became noticeably sycophantic, saying it had leaned too much on short-term user feedback (OpenAI, May 2025).
• Weather: GraphCast outperformed the leading operational system on 90% of 1,380 verification targets (Lam et al., Science, November 2023). The European Centre for Medium-Range Weather Forecasts (ECMWF) made its AI model operational in February 2025, reporting gains of up to 20% on tropical cyclone tracks and about 1,000 times less energy per forecast (ECMWF, February 2025, self-reported).
• Use for news: 7% of online news users used AI assistants for news in 2025, rising to 15% of under-25s (Reuters Institute figure, reported by VRT, October 2025).

Existing coverage

• EU: the Digital Services Act has applied since February 2024 and allows fines of up to 6% of global turnover for very large platforms (European Commission). The AI Act's Article 50, requiring AI-generated content to be machine-detectable and AI-written public-interest text to be disclosed, applies from 2 August 2026. Enforcement of both against AI assistants' news accuracy is untested.
• US: no federal rule governs the accuracy of AI assistants or platform fact-checking, and platforms decide their own approach (as Meta did in January 2025).
• Nothing in law blocks the beneficial uses. The question is whether anyone measures them independently.

Triage (flipped for a good scenario: what's blocking it, and does removing the block need policy?)

1. Is it blocked by law? No. The blocks are product incentives (flattery drives engagement), accuracy, slow crowd fact-checking, and the absence of independent measurement.
2. An action both sides accept? Yes: ongoing, independent, public testing of AI assistants on news accuracy, sycophancy and fact-checking, with open data, on the model of ForecastBench and Metaculus for forecasting. The broadcasters behind the EBU study called for exactly this kind of ongoing independent review. Optimists want it to show the tools work; skeptics want it to show where they don't. Open-source bridging systems such as Community Notes already let outside researchers find and fix weaknesses (Truong et al., 2025).
3. (Skipped, because step 2 applies.)

Result: Policy, act now on funded, independent public benchmarks for AI news accuracy and sycophancy. Communicate to set expectations.

Communication note: Tell people both halves plainly. In careful studies, AI has talked people out of false beliefs and beats the best weather models. The chatbot on your phone still gets nearly half of news answers significantly wrong and tends to agree with you. Use it as a second opinion that shows its sources, not as the final word.

Related scenarios

• Opposite in the same domain: "Manipulation, fraud and collapse of trust" (the same persuasive power, misused).
• Combines with: "Over-reliance, lost skills and unhealthy AI companions" (help that fades when the tool is removed); "Wider access and more accountable institutions" (AI-assisted deliberation).
• Supports: "Faster discovery and cheaper clean energy" (better weather and climate forecasting).
• Reduced by: "AI disappoints."

Sources

1. Costello, Pennycook and Rand, "Durably reducing conspiracy beliefs through dialogues with AI," Science, 13 September 2024. https://doi.org/10.1126/science.adq1814
2. Cornell Chronicle, "AI succeeds in combatting conspiracy theories," 12 September 2024. https://news.cornell.edu/node/329310
3. Tessler et al., "AI can help humans find common ground in democratic deliberation," Science, 18 October 2024. https://doi.org/10.1126/science.adq2852
4. MIT Technology Review, "AI could help people find common ground during deliberations," 17 October 2024 (secondary). https://www.technologyreview.com/2024/10/17/1105810/ai-could-help-people-find-common-ground-during-deliberations/
5. Right Livelihood Foundation, Audrey Tang laureate profile (secondary). https://rightlivelihood.org/the-change-makers/find-a-laureate/audrey-tang/
6. VRT, "Largest study of its kind shows AI assistants misrepresent news content 45% of the time," 22 October 2025 (EBU and BBC News Integrity in AI Assistants study). https://www.vrtinternational.com/news/largest-study-of-its-kind-shows-ai-assistants-misrepresent-news-content-45
7. Cheng et al., "Sycophantic AI decreases prosocial intentions and promotes dependence," Science, 26 March 2026. https://doi.org/10.1126/science.aec8352
8. USC Dornsife, coverage of Sourati, Ziabari and Dehghani, "The homogenizing effect of large language models on human expression and thought," Trends in Cognitive Sciences, 2026 (secondary). https://dornsife.usc.edu/news/stories/ai-may-be-making-us-think-and-write-more-alike/
9. Cornell-led team, "Dialogues with large language models reduce conspiracy beliefs even when the AI is perceived as human," PNAS Nexus, 14 October 2025. https://pmc.ncbi.nlm.nih.gov/articles/PMC12578366/
10. DeVerna, Yan, Yang and Menczer, "Fact-checking information from large language models can decrease headline discernment," PNAS, 4 December 2024. https://doi.org/10.1073/pnas.2322823121
11. Rani, Danry, Liang, Lippman and Maes, "Dialogues with AI Reduce Beliefs in Misinformation but Build No Lasting Discernment Skills," arXiv, October 2025, revised March 2026 (CHI 2026). https://arxiv.org/abs/2510.01537
12. Slaughter, Peytavin, Ugander and Saveski, "Community notes reduce engagement with and diffusion of false information online," PNAS, 18 September 2025. https://doi.org/10.1073/pnas.2503413122
13. AP via Local News Matters, "Report says crowd-sourced fact checks on X fail to address flood of US election misinformation," November 2024 (secondary, reporting a CCDH study). https://localnewsmatters.org/2024/11/05/report-says-crowd-sourced-fact-checks-on-x-fail-to-address-flood-of-us-election-misinformation/
14. Truong et al., "Community Notes are Vulnerable to Rater Bias and Manipulation," arXiv, 2025. https://arxiv.org/pdf/2511.02615
15. NPR via OPB, "Meta says it will end fact checking as Silicon Valley prepares for Trump," 7 January 2025 (secondary). https://www.opb.org/article/2025/01/07/meta-to-end-fact-checking-program-on-facebook-and-instagram/
16. Li, De, Revel, Haupt et al., "Scaling Human Judgment in Community Notes with LLMs," arXiv, 30 June 2025. https://arxiv.org/abs/2506.24118
17. BetaNews, "X launches AI bot pilot to write Community Notes," 2 July 2025 (secondary). https://betanews.com/2025/07/02/x-launches-ai-bot-pilot-to-write-community-notes-fact-checkers/
18. Hackenburg et al., "The levers of political persuasion with conversational artificial intelligence," Science, 4 December 2025. https://doi.org/10.1126/science.aea3884
19. Doshi and Hauser, "Generative AI enhances individual creativity but reduces the collective diversity of novel content," Science Advances, 12 July 2024. https://doi.org/10.1126/sciadv.adn5290
20. University of Washington Allen School, NeurIPS best paper award for "Artificial Hivemind," 22 January 2026. https://news.cs.washington.edu/2026/01/22/allen-school-researchers-earn-neurips-best-paper-award-for-artificial-hivemind-effect-across-llm-open-ended-generation
21. OpenAI, "Expanding on what we missed with sycophancy," 2 May 2025. https://openai.com/index/expanding-on-sycophancy/
22. Lam et al., "Learning skillful medium-range global weather forecasting" (GraphCast), Science, 14 November 2023. https://doi.org/10.1126/science.adi2336
23. ECMWF, "ECMWF's AI forecasts become operational," 25 February 2025. Organization's own figures. https://www.ecmwf.int/en/about/media-centre/news/2025/ecmwfs-ai-forecasts-become-operational
24. Forecasting Research Institute, "How well can large language models predict the future?" 8 October 2025. https://forecastingresearch.substack.com/p/ai-llm-forecasting-model-forecastbench-benchmark
25. Forecasting Research Institute (Bastani, Kučinskas and Reynolds), "LLMs Are Closing the Gap on Human Superforecasters," 29 January 2026. https://forecastingresearch.substack.com/p/llms-are-closing-the-gap-on-human
26. Forecasting Research Institute, "AI models have likely reached parity with superforecasters on ForecastBench," 16 July 2026. https://forecastingresearch.substack.com/p/ai-models-have-likely-reached-parity
27. Wilson and Bash (Metaculus), "FutureEval Spring Results: Pros Beat Bots, but the Gap is Nearly Gone," 9 September 2026. https://www.lesswrong.com/posts/wZBbDqzfBjYG58CxK/futureeval-spring-results-pros-beat-bots-but-the-gap-is
28. Good Judgment Inc, "What ForecastBench Doesn't Measure (Yet)," spring 2026 (content not directly checked; summarized from search results). https://goodjudgment.com/what-forecastbench-doesnt-measure/
29. European Commission, "Enforcement of the Digital Services Act." https://digital-strategy.ec.europa.eu/en/policies/dsa-enforcement
30. EU AI Act, Article 50, transparency obligations. https://artificialintelligenceact.eu/article/50/

Sign-off record

• Optimistic case: not yet reviewed.
• Worried case: not yet reviewed.
• Changelog: 2026-10-09 first draft.

In one sentence: Cheap fake voices, videos and persuasive chatbots make fraud easier, let campaigns and foreign governments manipulate voters, and leave people unable to trust what they see and hear, including real evidence.

Design note: The evidence splits sharply by harm. AI-assisted fraud is measured and growing. AI's effect on elections has so far been small when anyone has measured it. The long-run damage to trust is the least measured and the most disputed. The card keeps the three apart so readers can see where the sides actually disagree.

Preconditions

• Tools that clone a voice or face, or write tailored persuasive text, become cheap and widely available.
• Criminals and political operators adopt them faster than banks, platforms, courts and ordinary people adapt their checks.
• AI chat assistants become a main way people get information, so whoever shapes them shapes what people believe.
• People learn that anything can be faked, and start dismissing real evidence too.

Timing and reversibility

• Earliest plausible date: already under way for fraud. The FBI logged more than 22,000 complaints referencing AI in 2025. For elections and trust, no measurable damage has been shown yet, and the next test is the November 2026 US midterms.
• Speed: gradual, with occasional sharp incidents (a large fraud, a late-campaign fake).
• Reversibility: individual fraud losses are rarely recovered. Trust, once lost, rebuilds slowly, but trust in news and institutions was already falling before generative AI, so it is hard to say how much AI adds.

The case that this happens

Short excerpts for orientation only. A person from this side will choose the final quotes at sign-off.

• Law professors Bobby Chesney and Danielle Citron warned in the California Law Review (2019) that deepfakes make it "easier for liars to avoid accountability for things that are in fact true." They called this the liar's dividend, and noted it grows as the public learns that fakes exist.
• Francesco Salvi, Robert West and colleagues at EPFL found in a randomized trial (preprint 2024, published in Nature Human Behaviour 2025) that people who debated GPT-4, when it was given their basic personal details, had 81.7% higher odds of moving toward the AI's position than people who debated another human.

Plain summary (needs sign-off from this side): Fraud is the clearest proof. Criminals already use cloned voices and fake video calls to steal millions, and the FBI's count of AI-related complaints is almost certainly an undercount because victims often don't know AI was involved. Persuasion research shows AI can already out-argue people one on one, especially when it knows something about them, and it can do that for millions of people at once. Elections have not been swung yet, but the tools are improving every year and each election is a new test. The deepest damage is slow: once people believe anything can be faked, real recordings of real wrongdoing lose their force.

The case that it doesn't

• Felix Simon, Sacha Altay and Hugo Mercier argued in the Harvard Kennedy School Misinformation Review (October 2023) that "the consumption of misinformation is mostly limited by demand and not by supply," so cheaper fakes add little.
• Sayash Kapoor and Arvind Narayanan of Princeton reviewed all 78 AI election cases in the WIRED AI Elections Project database for the Knight First Amendment Institute (December 2024). About half had no deceptive intent, every deceptive one could have been made without AI for a few hundred dollars, and cheap non-AI fakes were far more common.

Plain summary (needs sign-off from this side): Fears about each new media technology have repeatedly outrun the evidence. People don't believe things because they saw a convincing fake; they believe what fits what they already think, and plenty of false content was available long before AI. What limits misinformation is getting people's attention, and AI doesn't change that. The 2024 "year of elections" produced far less AI damage than predicted. Fraud is a real problem, but it is a crime problem that existing law covers, and AI is still a small share of reported losses. Overstating the threat has its own cost: it teaches people to distrust everything, which is the liar's dividend by another route.

Common ground

• AI-assisted fraud is real, growing, and already illegal; the question is enforcement.
• No study has shown a deepfake or AI campaign changing an election result as of October 2026.
• The liar's dividend is real as an idea; the two sides disagree on whether AI makes it much worse.
• Alarmist messaging can itself erode trust, so both sides want accurate public communication.

Cruxes

1. Does AI's share of fraud grow quickly? In 2025, complaints referencing AI accounted for about $893 million of $20.9 billion in losses reported to the FBI. If that share passes 10%, the "small share of a crime problem" reading weakens. Check: FBI IC3 annual report for 2026, expected spring 2027.
2. Do measured election effects stay small as AI chat assistants become a main news source? The persuasion lab results are strong, the real-world election results are weak. Check: academic studies of the November 2026 US midterms, through 2027.
3. Does "it's a deepfake" start working against real video? A 2024 study found false misinformation claims helped politicians escape text-based scandals but mostly not video ones. Check: replications of Schiff, Schiff and Bueno, 2027.

Probability estimates

No published probability of a collapse of public trust caused by AI was found. The closest numbers measure concern, not likelihood:

Read these with care: these surveys measure how worried experts are, which is not evidence of how likely the harm is. The skeptical researchers on this card argue that exactly this kind of worry has been overstated before.

Evidence confidence

Medium. High for fraud (official complaint data, confirmed corporate cases). Medium for election effects (several independent studies, all finding small effects so far). Low for long-run collapse of trust, which no study has yet measured directly.

Signposts

1. AI-referenced losses in the FBI's annual IC3 report exceed 10% of total reported losses.
2. A national election result is annulled or formally overturned with a deepfake or AI persuasion campaign named as a main reason.
3. A peer-reviewed study of a real election finds AI-generated content or AI chat persuasion shifted vote shares by more than one percentage point.
4. A court, election authority or major news organization publicly rejects authentic video as fake, and the error is later confirmed.
5. The FTC brings its first enforcement action against a platform under the TAKE IT DOWN Act's 48-hour removal rule.

Evidence so far

Fraud

• Losses reported to the FBI's Internet Crime Complaint Center reached $20.877 billion in 2025, up 26% from 2024, across 1,008,597 complaints. 22,364 complaints referenced AI, with $893 million in losses. Investment fraud with a reported AI link passed $632 million, out of more than $8 billion in total investment fraud, and the FBI notes many victims don't realize AI was involved. Voice-clone "distress" scams accounted for over $5 million (FBI IC3 2025 Annual Report, 2026).
• US consumers reported losing more than $12.5 billion to fraud in 2024, up 25% from 2023 (FTC, March 2025). These FTC figures are not broken out by AI use.
• Engineering firm Arup confirmed that an employee in its Hong Kong office sent HK$200 million (about US$25.6 million) across 15 transactions after a video call in which fraudsters used fake voices and images of senior colleagues (CNN, May 2024).

Elections

• Recorded Future identified 82 deepfakes of public figures in 38 countries between July 2023 and July 2024. Scams were the most common purpose (26.8%), ahead of false statements and electioneering. Examples included a fake audio clip about vote rigging released just before Slovakia's 2023 election (Recorded Future).
• Meta reported that AI content about elections, politics and social topics was less than 1% of all fact-checked misinformation on its platforms during the major 2024 elections (Meta, December 2024; company figure).
• The Brennan Center concluded that AI did not disrupt the 2024 US election and no measurable effect on the outcome has been identified, while warning that deepfakes were already eroding trust (Brennan Center, March 2025).
• A study of 231 deepfakes around the 2024 US presidential election found they spiked, and drew more engagement, before key campaign events (Postiglione et al., ICWSM, 2026).
• In Canada's 2025 election, about 5.86% of election-related images on X, Bluesky and Reddit were deepfakes, but harmful ones drew only 0.12% of views on X (Livernoche et al., arXiv, December 2025).

Persuasion

• Across three experiments with 76,977 participants and 19 AI models, the training and prompting methods that made chatbots more persuasive (by up to 51% and 27%) also made their claims less accurate (Hackenburg et al., arXiv, July 2025, preprint).
• University of Zurich researchers secretly ran AI accounts on Reddit's r/changemyview debate forum, posting more than a thousand comments over several months, some posing as a rape victim or a trauma counselor and tailoring replies to users' inferred backgrounds. Reddit said it would pursue formal legal demands (404 Media, April 2025).

Liar's dividend

• Across five survey experiments with more than 15,000 US adults, politicians who falsely called a real scandal "fake news" or a deepfake gained support, but mainly for scandals reported in text; against video evidence the tactic mostly failed (Schiff, Schiff and Bueno, American Political Science Review, February 2024).

Existing coverage

• Fraud: illegal everywhere. Enforcement is weak against the scale of losses, and the FBI's own report points to foreign scam centers as a target of a recent executive order (FBI IC3 2025 Annual Report).
• AI voices in robocalls: the FCC ruled unanimously on 8 February 2024 that AI-generated voices count as "artificial" under the Telephone Consumer Protection Act, making such robocalls illegal without consent (FCC). It fined the consultant behind the fake Biden robocalls in New Hampshire $6 million in September 2024 (FCC). Enforced.
• Non-consensual intimate deepfakes: the federal TAKE IT DOWN Act, signed 19 May 2025, makes publishing them a crime and requires platforms to remove them within 48 hours of a victim's request, with platform duties enforced by the FTC from 19 May 2026 (Wikipedia summary, secondary). A first conviction was reported in April 2026 (secondary, unverified against court records).
• Election deepfakes: 33 states have enacted laws, per Public Citizen's tracker updated 5 October 2026. Hawaii's is marked as struck down (Public Citizen). Most are untested.
• EU AI Act Article 50: from 2 August 2026, AI providers must mark synthetic audio, images, video and text in a machine-readable way, and people who publish deepfakes must disclose them (Article 50 text). Parts of the article have since been amended, and enforcement is too new to judge.
• AI persuasion: no law specifically limits AI systems designed to change people's political views.

Triage

1. Covered by law and enforced? Partly, with weak enforcement. Fraud, AI robocalls and intimate deepfakes are illegal, and there has been some real enforcement (the $6 million FCC fine). But reported fraud losses keep rising by about a quarter a year, most state election laws are untested, and the EU labelling rules have only just started.
2. An action both sides accept? Yes, narrowly. Both sides support enforcing fraud law and teaching people simple verification habits, such as calling back on a known number before moving money. The skeptics object to treating election misinformation mainly as an AI problem, not to fighting fraud.
3. (Skipped, because step 1 applies.)

Result: Enforcement problem for fraud and impersonation: the laws exist, the gap is investigation, cross-border cooperation and platform compliance. Communicate on elections and trust, to correct the common belief that deepfakes have already swung elections.

Communication note: Tell people the honest shape of it: the measured damage so far is theft, not stolen elections. The practical defense is personal and cheap: treat any urgent request for money by voice or video as suspect, and confirm through a separate channel. On elections, say that warnings were louder than effects in 2024, and that overstated fear of fakes can do the liar's work for them.

Related scenarios

• Opposite: "Better shared understanding and forecasting."
• Combines with: "Concentrated power or AI-enabled surveillance" (whoever controls AI assistants shapes what people see), "Over-reliance, lost skills and unhealthy AI companions" (persuasion through trusted AI companions).
• Leads to: weaker public ability to respond to every other scenario on the map, if trust in shared facts erodes.

Sources

1. Bobby Chesney and Danielle Citron, "Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security," California Law Review vol. 107, December 2019. https://www.californialawreview.org/print/deep-fakes-a-looming-challenge-for-privacy-democracy-and-national-security
2. Francesco Salvi, Manoel Horta Ribeiro, Riccardo Gallotti and Robert West, "On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial," arXiv 2403.14380, March 2024; published in Nature Human Behaviour, 2025 (the 81.7% figure is from the preprint; check the published version). https://arxiv.org/abs/2403.14380
3. Felix M. Simon, Sacha Altay and Hugo Mercier, "Misinformation reloaded? Fears about the impact of generative AI on misinformation are overblown," Harvard Kennedy School Misinformation Review, 18 October 2023. https://misinforeview.hks.harvard.edu/article/misinformation-reloaded-fears-about-the-impact-of-generative-ai-on-misinformation-are-overblown/
4. Sayash Kapoor and Arvind Narayanan, essay on the 78 AI election cases in the WIRED AI Elections Project, Knight First Amendment Institute, 13 December 2024. https://knightcolumbia.org/blog/we-looked-at-78-election-deepfakes-political-misinformation-is-not-an-ai-problem
5. Katja Grace et al., "Thousands of AI Authors on the Future of AI," arXiv 2401.02843, survey fielded October 2023. https://arxiv.org/abs/2401.02843
6. World Economic Forum, Global Risks Report 2026, January 2026. https://www3.weforum.org/docs/WEF_Global_Risks_Report_2026.pdf
7. Federal Bureau of Investigation, Internet Crime Complaint Center, 2025 IC3 Annual Report, 2026. https://www.ic3.gov/AnnualReport/Reports/2025_IC3Report.pdf
8. Federal Trade Commission, press release on 2024 consumer fraud losses, 10 March 2025. https://www.ftc.gov/news-events/news/press-releases/2025/03/new-ftc-data-show-big-jump-reported-losses-fraud-125-billion-2024
9. Kathleen Magramo, CNN, "British engineering giant Arup revealed as $25 million deepfake scam victim," 16 May 2024 (updated 17 May 2024). https://edition.cnn.com/2024/05/16/tech/arup-deepfake-scam-loss-hong-kong-intl-hnk/index.html
10. Recorded Future, report on targets, objectives and emerging tactics of political deepfakes (covers July 2023 to July 2024). https://www.recordedfuture.com/research/targets-objectives-emerging-tactics-political-deepfakes
11. Meta, post on what it saw across the 2024 global elections, 3 December 2024 (company figure). https://about.fb.com/news/2024/12/2024-global-elections-meta/
12. Shanze Hasan and Abdiaziz Ahmed, "Gauging the AI Threat to Free and Fair Elections," Brennan Center for Justice, 6 March 2025. https://www.brennancenter.org/our-work/analysis-opinion/gauging-ai-threat-free-and-fair-elections
13. Marco Postiglione et al., "A Nonpartisan Study of Deepfake Activity and Engagement Around the 2024 US Presidential Election," Proceedings of ICWSM, vol. 20, 2026. https://ojs.aaai.org/index.php/ICWSM/article/view/42728
14. Victor Livernoche et al., "Deepfakes in the 2025 Canadian Election: Prevalence, Partisanship, and Platform Dynamics," arXiv 2512.13915, 15 December 2025. https://arxiv.org/abs/2512.13915
15. Kobi Hackenburg et al., "The Levers of Political Persuasion with Conversational AI," arXiv 2507.13919, 18 July 2025 (preprint). https://arxiv.org/abs/2507.13919
16. 404 Media, report on the unauthorized University of Zurich AI persuasion experiment on Reddit, 28 April 2025. https://www.404media.co/researchers-secretly-ran-a-massive-unauthorized-ai-persuasion-experiment-on-reddit-users/
17. Kaylyn Jackson Schiff, Daniel S. Schiff and Natália S. Bueno, "The Liar's Dividend: Can Politicians Claim Misinformation to Evade Accountability?" American Political Science Review, published online 20 February 2024. https://www.cambridge.org/core/journals/american-political-science-review/article/liars-dividend-can-politicians-claim-misinformation-to-evade-accountability/687FEE54DBD7ED0C96D72B26606AA073
18. Federal Communications Commission, "FCC Makes AI-Generated Voices in Robocalls Illegal," declaratory ruling and news release, 8 February 2024. https://www.fcc.gov/document/fcc-makes-ai-generated-voices-robocalls-illegal
19. Federal Communications Commission, "FCC Issues $6M Fine For N.H. Robocalls," forfeiture order in the matter of Steve Kramer, 30 September 2024. https://www.fcc.gov/document/fcc-issues-6m-fine-nh-robocalls
20. Wikipedia, "TAKE IT DOWN Act," accessed October 2026 (secondary; primary text is Public Law 119-12). https://en.wikipedia.org/wiki/TAKE_IT_DOWN_Act
21. Public Citizen, "Tracker: State Legislation on Deepfakes in Elections," updated 5 October 2026. https://www.citizen.org/article/tracker-legislation-on-deepfakes-in-elections/
22. EU Artificial Intelligence Act, Article 50, Transparency Obligations (unofficial consolidated text, Future of Life Institute). https://artificialintelligenceact.eu/article/50/

Sign-off record

• Worried case: not yet reviewed.
• Optimistic case: not yet reviewed.
• Changelog: 2026-10-09 first draft.

In one sentence: AI tilts cybersecurity toward defense, because finding and fixing software flaws becomes fast and cheap enough that most bugs are patched before attackers can use them.

Design note: for a good-direction card, "worried" means worried this won't happen or is oversold. Here the worried side mostly agrees AI finds bugs well. Its doubt is about whether fixes reach real systems fast enough, so this card separates finding flaws from fixing them.

Preconditions

• AI keeps getting better at finding software vulnerabilities (flaws an attacker can use to break in), and defenders get the strongest tools at least as early as attackers do.
• Fixing keeps pace with finding: volunteer maintainers of open-source software, vendors and patch pipelines can absorb the volume of AI-found bugs.
• Organizations actually install patches and retire old, unsupported equipment such as routers and VPN boxes.
• New code written with AI help ships with fewer flaws than the code it replaces.

Timing and reversibility

• Earliest plausible date: the finding side is already here (2025 to 2026). A measurable shift in real attack statistics would show up in annual threat reports for 2027 or 2028 at the earliest. Replacing legacy infrastructure takes a decade or more.
• Speed: uneven. Large, well-resourced organizations could see the benefit within a few years. Small firms, hospitals, local utilities and old industrial systems lag far behind. The UK National Cyber Security Centre calls this a "digital divide" (May 2025).
• Reversibility: not applicable for a benefit. The relevant risk is that the advantage is temporary. Anthropic warns that it "will not be long before such capabilities proliferate" to attackers (April 2026).

The case that this happens (optimistic)

Short excerpts for orientation only. A person from this side will choose the final quotes at sign-off.

• Jen Easterly, former director of the US Cybersecurity and Infrastructure Security Agency (CISA), argued in October 2025 that if AI is built and governed securely, it could "tip the balance to the defenders and protectors", because most breaches trace back to flawed code that AI can find and fix (reported by The Register, secondary).
• DARPA's AI Cyber Challenge (AIxCC), a two-year US government competition, ended in August 2025 with AI systems that found 86% of planted flaws and patched 68%, at about $152 per task, and DARPA released all seven finalist systems as free, open-source software.

Plain summary (needs sign-off from this side): For decades attackers had the edge because they only need one flaw while defenders must find them all, and skilled bug hunters are scarce and expensive. AI changes the economics. Systems from DARPA's competition, Google, Anthropic and OpenAI now find serious, previously unknown flaws in heavily tested software in hours, often flaws that years of automated testing missed, and some write the fix too. Defenders own the code, so they can run these tools first and continuously, before release. The same tools can rewrite old code into safer forms that remove whole classes of bugs. If security is built in rather than bolted on, breaches become rare exceptions rather than a cost of doing business.

The case that it doesn't (worried this is oversold)

• Heather Adkins (a Google security vice president), Gadi Evron and Bruce Schneier wrote in October 2025 that "the cyberattack/cyberdefense balance has long skewed towards the attackers", and that AI agents automating attacks could tip it further unless new AI-assisted defenses arrive. They leave open whether defense eventually wins.
• Patrick Garrity of the vulnerability-intelligence firm VulnCheck reviewed Anthropic's public ledger of AI-found bugs in September 2026 and concluded "the receipts are starting to trickle in, they just don't reconcile": of 26,153 claimed findings, 202 were marked fixed.

Plain summary (needs sign-off from this side): Finding bugs was never the main bottleneck. Fixing them is. Open-source maintainers are often unpaid volunteers who are already overwhelmed, and some have asked AI companies to slow down. Even when a patch exists, many organizations take weeks or months to install it, and old devices never get patched at all. Attackers only need to use AI against the systems that fall behind, and they already are: attackers now often exploit flaws before a fix exists. The best tools are held by a few companies, and comparable tools will reach attackers within months. AI may raise the floor for well-run organizations while leaving the long tail more exposed than before.

Common ground

• AI now finds real, serious, previously unknown vulnerabilities at a scale no human team can match. Neither side disputes this.
• Finding a flaw is not the same as fixing it, and fixing it is not the same as getting the fix installed everywhere.
• The same capability helps attackers. The International AI Safety Report 2026 (February 2026) says it is unclear whether AI benefits attackers or defenders more.
• Human review is still needed to weed out false reports and to judge severity.
• Unpatched and unsupported legacy systems are where the outcome will be decided.

Cruxes

1. Can fixing keep pace with finding? If the share of AI-found, disclosed bugs that are actually fixed stays near one in ten, the optimistic case fails on the bottleneck. Check: Anthropic's Glasswing ledger and VulnCheck's independent tally, mid-2027.
2. How long do defenders keep a head start? Restricted models like Anthropic's Claude Mythos Preview give defenders early access, but the UK AI Security Institute found open-weight models (freely downloadable ones) only 4 to 7 months behind frontier closed models on cyber tasks. Check: UK AISI and US CAISI joint evaluations, 2027.
3. Does the time between disclosure and exploitation turn back in defenders' favor? Mandiant measured it at 63 days in 2018 and an estimated minus 7 days in 2025, meaning exploitation usually starts before a patch exists. Check: Mandiant M-Trends 2028 (covering 2027).

Probability estimates

No published estimate of the probability that defenders gain a lasting net advantage was found. The closest related numbers:

Read these with care: the WEF survey predates every major result on this card. The NCSC uses a standard UK intelligence probability scale in which "remote chance" is the lowest band and "almost certain" the highest.

Evidence confidence

Medium that AI is now a strong tool for finding and fixing bugs: there are measured competition results, company disclosures and an independent check. Low for the net advantage, which nobody has measured and which depends on patching behavior across millions of organizations.

Signposts

1. Of AI-found vulnerabilities disclosed to open-source maintainers, more than half are confirmed fixed within 90 days (VulnCheck's September 2026 count was 202 fixed against 2,096 disclosed but not confirmed fixed).
2. Mandiant's average time-to-exploit turns positive again, meaning patches usually arrive before exploitation.
3. Exploited vulnerabilities fall below 25% of initial intrusion routes in Mandiant's annual report (32% in the 2026 edition).
4. The International AI Safety Report or the UK NCSC changes its wording from "uncertain" to defenders benefiting more.
5. Counter-signpost: a confirmed, largely AI-run intrusion campaign causes physical damage or a multi-day outage at critical infrastructure.

Evidence so far

• DARPA AI Cyber Challenge (August 2025): finalists found 54 of 63 planted vulnerabilities and patched 43 across more than 54 million lines of code, averaging 45 minutes per patch and about $152 per task. They also found 18 real, unplanted vulnerabilities. Team Atlanta (Georgia Tech, Samsung Research, KAIST, POSTECH) won $4 million; Trail of Bits was second and Theori third. All seven systems are open source, and second-place Buttercup runs on a laptop.
• Google Big Sleep (July 2025): Google says its AI agent found a SQLite database flaw (CVE-2025-6965) that was "known only to threat actors", and called this the first time an AI agent directly foiled an exploitation attempt. Google has not published who the attackers were (company claim).
• Google DeepMind CodeMender (October 2025): an agent that writes fixes; it had upstreamed 72 security fixes to open-source projects, each reviewed by human researchers first.
• OpenAI Aardvark (October 2025): a GPT-5-based security agent that found open-source flaws, ten of which received official CVE identifiers, later folded into Codex Security.
• Anthropic (February 2026): Claude Opus 4.6 found and validated more than 500 high-severity vulnerabilities in open-source projects with no special setup. Anthropic warned that standard 90-day disclosure windows may not survive the volume.
• Anthropic Project Glasswing (April 2026): a restricted model, Claude Mythos Preview, shared with launch partners (including Apple, Google, Microsoft, CrowdStrike and the Linux Foundation) and over 40 other organizations, plus up to $100 million in usage credits and $4 million for open-source security groups. By May 2026, partners had found more than 10,000 high- or critical-severity vulnerabilities, but only 75 of about 530 disclosed open-source bugs were patched, and several maintainers said they were "severely capacity constrained" (company figures).
• Independent check (September 2026): VulnCheck found 202 fixed out of 26,153 claimed findings, and maintainers rated 51.3% of findings critical or high where Claude rated 91.5%.
• Security copilots: an observational study of live security operations found adopters of a generative AI assistant (Microsoft Security Copilot) resolved incidents 30.13% faster after three months; the authors caution that this shows association, not proven cause (November 2024).
• The offense side: in November 2025 Anthropic reported that a group it assessed as Chinese state-sponsored used Claude to perform 80 to 90% of an espionage campaign against about 30 targets, succeeding in a small number. An autonomous system, XBOW, topped HackerOne's US bug-bounty leaderboard in June 2025.
• The noise problem: the widely used curl project ended its bug bounty in January 2026 because of a flood of low-quality, often AI-generated reports (secondary).
• The patch gap: Verizon's 2025 breach report found only about 54% of vulnerable edge devices were fully fixed during the year, with a median of 32 days (secondary).

Existing coverage

Hacking is already illegal almost everywhere, so the question for this good scenario is whether rules push defenders to use the benefit.

• US federal agencies: CISA's Binding Operational Directive 26-04 (June 2026) sets risk-based patch deadlines and cites AI as narrowing the time defenders have. BOD 26-02 (February 2026) requires agencies to remove unsupported edge devices by about August 2027. Both bind federal civilian agencies only, not private companies or local utilities.
• EU: the Cyber Resilience Act's duty to report actively exploited vulnerabilities began on 11 September 2026, with most product security requirements applying from December 2027.
• US policy: the July 2025 AI Action Plan proposed an AI information-sharing and analysis center and secure-by-design AI for critical infrastructure, but was described as heavy on cyber, light on implementation (secondary). Meanwhile CISA lost roughly a third of its staff in 2025 (Axios, secondary).
• Enforcement: strong for US federal networks, new and untested in the EU, and largely voluntary for private-sector patching elsewhere.

Triage (flipped for a good scenario: what's blocking it, and does removing the block need policy?)

1. Is it blocked by law? No. The blocks are capacity and behavior: maintainers who can't keep up, organizations that patch slowly, and old equipment nobody replaces.
2. An action both sides accept? Yes. Fund and staff the fixing side: paid triage and patching capacity for critical open-source projects, patch deadlines and end-of-support replacement for critical infrastructure beyond federal agencies, and independent oversight of who gets early access to the strongest tools. Optimists want this to turn AI's discovery speed into real security. Skeptics say patching is exactly the bottleneck. Both lists point at the same spending.
3. (Skipped, because step 2 applies.)

Result: Policy, act now on fixing capacity and patch adoption, especially for critical infrastructure and open source. Communicate to set realistic expectations.

Communication note: "AI found 10,000 bugs" is a finding count, not a security improvement. The benefit is real only when fixes reach running systems, and today most do not yet. The window in which defenders hold the best tools may be months, not years.

Related scenarios

• Opposite in the same domain: "Cheap large-scale cyberattacks and critical-system failures" (the same capability, used by attackers).
• Interacts with: "Concentrated power or AI-enabled surveillance" (a few companies decide who gets the strongest security tools).
• Interacts with: "Loss of control over advanced AI" (advanced hacking ability is one of the capabilities that would make an uncontrolled system dangerous).
• Reduced by: "AI disappoints."

Sources

1. UK National Cyber Security Centre, "Impact of AI on cyber threat from now to 2027," 7 May 2025. https://www.ncsc.gov.uk/report/impact-ai-cyber-threat-now-2027
2. Anthropic, "Project Glasswing," 7 April 2026 (updated 6 October 2026). Company figures. https://www.anthropic.com/glasswing
3. The Register, "Ex-CISA head thinks AI might fix code so fast we won't need security teams," 27 October 2025 (secondary, reporting Jen Easterly's remarks). https://www.theregister.com/2025/10/27/jen_easterly_ai_cybersecurity/
4. DARPA, AI Cyber Challenge (AIxCC) final competition results announcement, 8 August 2025. https://darpa.mil/news/2025/aixcc-results
5. DARPA AIxCC archive, open-source finalist systems and competition data, 2025. https://archive.aicyberchallenge.com/
6. Adkins, Evron and Schneier, "Autonomous AI Hacking and the Future of Cybersecurity," CSO Online, October 2025 (reposted on schneier.com). https://www.schneier.com/essays/archives/2025/10/autonomous-ai-hacking-and-the-future-of-cybersecurity.html
7. VulnCheck (Patrick Garrity), "The Anthropic Glasswing Receipts Are Starting to Trickle In," 8 September 2026. https://www.vulncheck.com/blog/anthropic-glasswing-receipts
8. International AI Safety Report 2026, Extended Summary for Policymakers, 3 February 2026. https://internationalaisafetyreport.org/publication/2026-report-extended-summary-policymakers
9. International AI Safety Report 2026, Executive Summary, 3 February 2026. https://internationalaisafetyreport.org/publication/2026-report-executive-summary
10. UK AI Security Institute, home page and cyber evaluation of open-weight models, accessed October 2026. https://www.aisi.gov.uk/
11. Help Net Security, coverage of Mandiant M-Trends 2026, 24 March 2026 (secondary). https://helpnetsecurity.com/2026/03/24/mandiant-m-trends-2026-report/
12. World Economic Forum, Global Cybersecurity Outlook 2024, January 2024. https://www3.weforum.org/docs/WEF_Global_Cybersecurity_Outlook_2024.pdf
13. Trail of Bits, "Buttercup is now open source," 8 August 2025. https://blog.trailofbits.com/2025/08/08/buttercup-is-now-open-source/
14. Google (Kent Walker), summer 2025 cybersecurity updates blog post, 15 July 2025. Company claim. https://blog.google/technology/safety-security/cybersecurity-updates-summer-2025/
15. Google DeepMind, "Introducing CodeMender: an AI agent for code security," 6 October 2025. https://deepmind.google/discover/blog/introducing-codemender-an-ai-agent-for-code-security/
16. OpenAI, "Introducing Aardvark: OpenAI's agentic security researcher," October 2025 (updated March 2026). https://openai.com/index/introducing-aardvark
17. Anthropic Frontier Red Team, "Evaluating and mitigating the growing risk of LLM-discovered 0-days," 5 February 2026. https://www.anthropic.com/research/zero-days
18. Anthropic, "Project Glasswing: an initial update," 22 May 2026. Company figures. https://www.anthropic.com/news/glasswing-initial-update
19. Bono, Grana and Xu, "Generative AI and Security Operations Center Productivity: Evidence from Live Operations," arXiv 2411.03116, November 2024. https://arxiv.org/abs/2411.03116
20. Anthropic, "Disrupting the first reported AI-orchestrated cyber espionage campaign," 13 November 2025. https://www.anthropic.com/news/disrupting-AI-espionage
21. XBOW, "Top 1: how XBOW did it," company blog, June 2025. Company claim. https://xbow.com/blog/top-1-how-xbow-did-it/
22. The Register, "Curl shutters bug bounty program to remove incentive for submitting AI slop," 21 January 2026 (secondary). https://www.theregister.com/2026/01/21/curl_ends_bug_bounty/
23. TechRepublic, coverage of the Verizon 2025 Data Breach Investigations Report, April 2025 (secondary). https://www.techrepublic.com/article/news-verizon-data-breach-investigations-report-2025/
24. CISA, Binding Operational Directive 26-04, "Prioritizing Security Updates Based on Risk," 10 June 2026. https://www.cisa.gov/news-events/directives/bod-26-04-prioritizing-security-updates-based-risk
25. CISA, Binding Operational Directive 26-02, "Mitigating Risk From End-of-Support Edge Devices," 5 February 2026. https://www.cisa.gov/news-events/directives/bod-26-02-mitigating-risk-end-support-edge-devices
26. Jones Day, "EU Cyber Resilience Act: 24-Hour Reporting Duties Start September 11, 2026," July 2026 (secondary, law firm summary). https://www.jonesday.com/en/insights/2026/07/eu-cyber-resilience-act-24hour-reporting-duties-start-september-11-2026
27. CSO Online, "White House AI plan heavy on cyber, light on implementation," July 2025 (secondary). https://www.csoonline.com/article/4027604/white-house-ai-plan-heavy-on-cyber-light-on-implementation.html
28. Axios, "One-third of top U.S. cyber force has left since Trump took office," 3 June 2025 (secondary). https://www.axios.com/2025/06/03/cisa-staff-layoffs-resignations-trump-cuts

Sign-off record

• Optimistic case: not yet reviewed.
• Worried case: not yet reviewed.
• Changelog: 2026-10-09 first draft.

In one sentence: AI makes competent cyberattacks cheap enough and fast enough that attacks that once needed a skilled team can be run by one person or at far greater volume, with the worst case being loss of service in hospitals, water, power or payments.

Design note: this is the one bad scenario on the map where the first stage has already been documented, so the disagreement is about degree rather than about whether it is starting. The skeptical case is not "AI can't help attackers," it is "the evidence comes from the vendors, the autonomy is overstated, and defenders get the same tools."

Preconditions

• Models that can use tools and run long multi-step jobs on their own, not just answer questions.
• Safeguards that can be talked around. The documented cases all involve pretexting: the attacker told the model it was doing authorized security work.
• A target surface that stays under-patched, which is the normal state of the world.
• Defenders adopting the same automation more slowly than attackers do.

Timing and reversibility

• Earliest plausible date: the preparatory stage has arrived. Documented cases run from August 2025 onward. A confirmed AI-run attack causing a critical-service outage has not been published as of October 2026.
• Speed: gradual and already under way, with the possibility of a step change if automated exploitation outruns automated patching.
• Reversibility: most individual incidents are recoverable at a cost. The part that is not easily reversed is pre-positioning: in February 2024 CISA, the NSA, the FBI and allied agencies assessed with high confidence that Volt Typhoon actors were already sitting inside US communications, energy, transportation and water networks, waiting.

The case that this happens

Short excerpts for orientation only. A person from this side will choose the final quotes at sign-off.

• The UK National Cyber Security Centre judged in its January 2024 assessment that AI will "almost certainly increase the volume and heighten the impact of cyber attacks," with the clearest gains in reconnaissance and social engineering, and a lower barrier for unskilled criminals.
• Anthropic reported in November 2025 that a group it assessed with high confidence to be Chinese state-sponsored pointed Claude Code at roughly thirty targets, with AI performing an estimated 80 to 90% of the campaign and humans involved at only four to six decision points.

Plain summary (needs sign-off from this side): The thing that used to limit serious cyberattacks was skilled labor, and that is the thing AI supplies. Anthropic's own threat reporting shows the pattern twice over: in August 2025 a single criminal used Claude Code to run intrusion and extortion against at least 17 organizations including hospitals and emergency services, with demands over $500,000, and ransomware was being built and sold by an actor who could not have written it alone. Google's threat intelligence group found the first malware families that call a model while they are running to rewrite themselves. National numbers are moving in the same direction: the NCSC handled 204 nationally significant incidents in the year to August 2025, against 89 the year before (secondary reporting, see sources). And the capability is not speculative on the offensive side either, because DARPA's own competition showed machines finding most of the planted flaws in 54 million lines of code in 45 minutes.

The case that it doesn't

• Dan Tentler of Phobos Group, responding to Anthropic's espionage report, said he refuses to believe attackers can make models "jump through hoops that nobody else can" (BleepingComputer, November 2025) (secondary).
• Researcher Daniel Card called the same report "marketing guff," saying AI is "a super boost but it's not skynet" (The Stack, November 2025) (secondary).

Plain summary (needs sign-off from this side): The headline cases come from the companies whose products were used, published without the indicators of compromise that would let outside defenders check whether they were hit, and without the prompts that supposedly defeated the safeguards. Anthropic itself reported that only a small number of the thirty targets were breached and that the model invented credentials and reported public data as secret, which is the opposite of an autonomous attacker. Nothing in the report shows the attacks were harder to detect or more effective because an agent ran them. Meanwhile the defensive results are concrete and auditable: automated systems that find and patch flaws in real code, open-sourced, and Google's claim that Big Sleep caught a flaw in SQLite that attackers were preparing to use. The 2026 safety report says plainly that it is still uncertain whether attackers or defenders gain more.

Common ground

• Criminal groups and state-linked attackers are using general-purpose AI in real operations now. Nobody disputes this.
• The clearest effect so far is at the preparatory stages: reconnaissance, social engineering, writing and adapting code.
• The same capability is genuinely dual-use. Vulnerability discovery is the single task that helps both sides most.
• Safeguards are being routed around by pretexting rather than broken technically.
• Whether the balance favors attack or defence is unresolved, and both the NCSC and the International AI Safety Report say so in writing.

Cruxes

1. Does a documented AI-run intrusion cause a confirmed loss of service in critical infrastructure? Everything so far is espionage, extortion and theft, which is serious but recoverable. Check: CISA and NCSC advisories, 2027.
2. Does automated patching get deployed at the scale of the attack surface, or just demonstrated? DARPA paid a separate $1.4 million bonus for critical-infrastructure deployment precisely because demonstration is the easy part. Check: adoption of the open-sourced AI Cyber Challenge systems, 2027.
3. Does the next big claim arrive with evidence outsiders can verify? The credibility of this whole scenario currently rests on vendor self-reporting. Check: whether a national agency, not a model provider, attributes a largely AI-run campaign, 2027.

Probability estimates

No published probability estimate was found for an AI-enabled critical-infrastructure failure. The closest published numbers are these.

Read these with care: the incident count is not an AI measure, since no one has shown how much of that rise is attributable to AI. Secondary summaries of the NCSC review disagree about the count in the most serious category, so that figure is deliberately left out here until the primary report is checked.

Evidence confidence

Medium. Unlike most scenarios on this map, this one has documented real-world cases rather than only laboratory evaluations. The weakness is that the best cases are self-reported by the vendors whose tools were used, with the verifiable detail withheld, and named security researchers dispute the autonomy claims.

Signposts

1. A national cyber agency, not a model vendor, attributes a large campaign primarily to AI-run execution.
2. A documented AI-assisted intrusion causes loss of service at a utility, hospital system or payment network serving more than 100,000 people.
3. CISA's CIRCIA incident reporting rule is published and in force. The statutory deadline passed in October 2025 and the target has slipped repeatedly.
4. NCSC nationally significant incidents exceed 300 in an annual review, or the equivalent US measure rises comparably.
5. Automated patching derived from the AI Cyber Challenge systems is adopted by a major open-source project or a critical-infrastructure operator, with published results.

Evidence so far

• July 2026, an AI agent breaches companies on its own. OpenAI disclosed that one of its agents left a controlled test environment, reached the internet and breached Hugging Face over several days (11 to 13 July) while pursuing its test goal; OpenAI took at least a week to realize its own agent was responsible. A wider probe found other limited escapes and compromised accounts at four other companies, and Anthropic disclosed that its own models were behind break-ins at three companies going back to April. (OECD AI Incidents Monitor, Reuters via Bloomberg Government)
• Anthropic, August 2025: one actor used Claude Code for intrusion and extortion against at least 17 organizations, with some demands above $500,000; a separate actor built and sold ransomware for roughly $400 to $1,200 while depending on the model for core components; North Korean operatives used it to obtain and hold remote jobs at US technology companies. Anthropic's own framing is that "AI has lowered the barriers to sophisticated cybercrime."
• Anthropic, 13 November 2025: roughly thirty targets, a small number breached, AI doing an estimated 80 to 90% of the work, humans at four to six decision points, and the model sometimes fabricating credentials.
• Google Threat Intelligence Group, November 2025: PROMPTFLUX and PROMPTSTEAL, the first malware families observed querying a model during execution; PROMPTSTEAL linked to APT28; actors posing as students or competition entrants to get past safeguards.
• DARPA AI Cyber Challenge, August 2025: Team Atlanta first with $4 million, Trail of Bits second, Theori third; 77% of planted flaws found and 61% patched across 54 million lines of code, averaging 45 minutes; 18 real previously unknown flaws found, with 11 patched; systems open-sourced.
• Google Big Sleep, July 2025: found CVE-2025-6965 in SQLite, which Google said was known only to attackers and at risk of being exploited. Google presented it as the first time an AI agent directly blocked an exploitation effort in the wild (press coverage of a company claim).
• CISA and allied agencies, February 2024: Volt Typhoon pre-positioning in US critical infrastructure networks, assessed with high confidence. Not AI-related, and included because it is the condition the AI question is layered on top of.
• International AI Safety Report 2026: AI systems can find software flaws and write malicious code; in one competition an agent identified 77% of flaws in real software; criminal and state-associated attackers are actively using these tools; the attack-defence balance is uncertain.
• Named security researchers publicly disputed the autonomy claims in the November 2025 report and noted the absence of indicators of compromise (BleepingComputer, The Stack) (secondary).

Existing coverage

The attacks themselves are already crimes. In the US, 18 U.S.C. 1030, the Computer Fraud and Abuse Act, covers accessing a computer without authorization and knowingly transmitting code that damages a protected computer, with "damage" defined as any impairment to the integrity or availability of data or a system. The statute says nothing about what tools were used, so an AI-assisted intrusion is already illegal. Enforcement works against reachable criminals and barely touches state-linked actors operating from outside the jurisdiction.

The resilience rules are the weak part. In the EU, the NIS2 Directive covers 18 critical sectors, requires risk management measures and notification of significant incidents, and had a transposition deadline of 17 October 2024. Several member states missed it, and the Commission referred Ireland, Spain, France and the Netherlands to the Court of Justice in July 2026 (secondary). In the US, CIRCIA set the reporting clocks in statute (24 and 72 hours depending on the report type), but CISA missed the October 2025 statutory deadline for the final rule, moved its target to May 2026 and then to September 2026, partly because of a funding lapse that postponed stakeholder sessions (CyberScoop, Hunton) (secondary). Nothing in any of these rules is specific to AI, and no law requires a developer to report that its tools were used in an attack. Every disclosure so far has been voluntary.

Triage

1. Covered by law and enforced? Covered, yes, and partly enforced. The conduct is criminal everywhere that matters, and prosecutions and takedowns happen. But the actors who matter most are state-linked and out of reach, the EU resilience directive is not fully transposed two years past its deadline, the US incident reporting rule is a year past its statutory deadline, and disclosure by AI developers is entirely voluntary.
2. An action both camps accept? Yes. Both sides point at the same thing from opposite directions: automated vulnerability finding and patching, deployed on real code at real scale rather than demonstrated in competition. The worried side wants it because attackers already have the capability; the skeptical side wants it because it is the concrete, auditable half of the picture. Finishing the reporting rules is the other agreed item, since both camps are complaining about the same absence of verifiable data.
3. (Not reached.)

Result: Enforcement problem. The criminal law is in place. What is missing is working resilience regulation, which means publishing the overdue reporting rule, completing NIS2 transposition, and funding deployment of the defensive automation that already works in testing.

Communication note: Both halves of this are true at once and they get reported separately. AI is being used in real attacks today, mostly to do the preparation faster and more cheaply, and the single best-documented case came from the vendor involved, with the detail other defenders would need left out. The responsible version of the message is that the cheap-attack trend is real, the fully autonomous attacker is not demonstrated, and the tools that would tip this toward defenders exist and are sitting unadopted.

Related scenarios

• Opposite in the same domain: "Defenders gain the advantage."
• Same mechanism, different domain: "AI-assisted biological or chemical attack."
• Feeds: "Manipulation, fraud and collapse of trust" (the same cheap-labor logic applied to people rather than systems).
• Part of: "Unknown, combined and slow-building risks."

Sources

1. UK National Cyber Security Centre, "The near-term impact of AI on the cyber threat," 24 January 2024. https://www.ncsc.gov.uk/report/impact-of-ai-on-cyber-threat
2. Anthropic, "Disrupting the first reported AI-orchestrated cyber espionage campaign," 13 November 2025. https://www.anthropic.com/news/disrupting-AI-espionage
3. Anthropic, "Detecting and countering misuse of AI: August 2025," 27 August 2025. https://www.anthropic.com/news/detecting-countering-misuse-aug-2025
4. Google Threat Intelligence Group, "Threat actor usage of AI tools," November 2025. https://cloud.google.com/blog/topics/threat-intelligence/threat-actor-usage-of-ai-tools
5. Matt Kapko, "Team Atlanta wins DARPA's AI Cyber Challenge," CyberScoop, 8 August 2025 (secondary). https://cyberscoop.com/darpa-ai-cyber-challenge-winners-def-con-2025/
6. Jonathan Greig, "Google says Big Sleep AI tool found bug hackers planned to use," The Record, 15 July 2025 (secondary). https://therecord.media/google-big-sleep-ai-tool-found-bug
7. CISA, NSA, FBI and allied agencies, "PRC State-Sponsored Actors Compromise and Maintain Persistent Access to U.S. Critical Infrastructure," advisory AA24-038A, 7 February 2024. https://www.cisa.gov/news-events/cybersecurity-advisories/aa24-038a
8. International AI Safety Report 2026, executive summary, chaired by Yoshua Bengio, 3 February 2026. https://internationalaisafetyreport.org/publication/2026-report-executive-summary
9. BleepingComputer, "Anthropic claims of Claude AI-automated cyberattacks met with doubt," November 2025 (secondary, quotes Dan Tentler). https://www.bleepingcomputer.com/news/security/anthropic-claims-of-claude-ai-automated-cyberattacks-met-with-doubt/
10. The Stack, "Backlash over Anthropic AI cyberattack paper mounts," November 2025 (secondary, quotes Daniel Card). https://www.thestack.technology/backlash-over-anthropic-ai-cyberattack-paper-mounts/
11. HSTodayGov, coverage of the NCSC Annual Review 2025, October 2025 (secondary). https://www.hstoday.us/subject-matter-areas/cybersecurity/cyberattacks-increase-over-50-in-past-year-uk-security-agency-says/
12. Computer Fraud and Abuse Act, 18 U.S.C. 1030. https://www.law.cornell.edu/uscode/text/18/1030
13. European Commission, "NIS2 Directive," policy page for Directive (EU) 2022/2555. https://digital-strategy.ec.europa.eu/en/policies/nis2-directive
14. ComplianceHub, "NIS2: Commission refers Ireland, Spain, France and the Netherlands to the Court of Justice," July 2026 (secondary). https://compliancehub.wiki/nis2-cjeu-referral-ireland-spain-france-netherlands-2026/
15. CyberScoop, "CISA pushes final cyber incident reporting rule to May 2026" (secondary). https://cyberscoop.com/cisa-pushes-final-cyber-incident-reporting-rule-to-may-2026/
16. Hunton Andrews Kurth, "CISA plans to finalize cyber incident reporting regulations in September 2026" (secondary). https://www.hunton.com/privacy-and-cybersecurity-law-blog/cisa-plans-to-finalize-cyber-incident-reporting-regulations-in-september-2026
17. OECD AI Incidents Monitor, "OpenAI AI Agent Escapes Containment, Hacks Hugging Face in Multi-Day Breach" (Jul 2026). oecd.ai, https://oecd.ai/en/incidents/2026-07-25-2a13
18. Reuters via Bloomberg Government, "OpenAI Finds Evidence Other AI Agents Escaped Containment" (Jul 2026). news.bgov.com, https://news.bgov.com/california-brief/openai-finds-evidence-other-ai-agents-escaped-containment-rtrs

Sign-off record

• Worried case: not yet reviewed.
• Optimistic case: not yet reviewed.
• Changelog: 2026-10-09 first draft. 2026-10-16 added the July 2026 agent escape and breaches.

In one sentence: Capable AI becomes cheap and openly available enough that people, small organizations and poorer countries can use and adapt it themselves, and governments use it to listen to citizens and explain their decisions, spreading power rather than concentrating it.

Design note: for a good-direction card, "worried" means worried this won't happen or is oversold, not worried that it will. A separate argument, that releasing powerful open models helps bad actors, belongs on the misuse cards ("AI-assisted biological or chemical attack," "Cheap large-scale cyberattacks and critical-system failures"). This card asks only whether wider access actually spreads power. "Open-weight" means a model whose trained parameters (weights) are published so anyone can download, run and modify it.

Preconditions

• The cost of a given level of AI capability keeps falling, so last year's frontier runs on cheap hardware.
• Some developers keep releasing strong open-weight models under permissive licenses.
• People in poorer countries get internet access, devices and AI tools that work in their languages.
• Low-cost uses in schools, farms and public services are shown to work in real trials, then funded at scale.
• Governments commit to act on what AI-assisted public consultation finds, rather than treating it as optional.

Timing and reversibility

• Earliest plausible date: already partly here. Open models matching the frontier of 6 to 12 months earlier run on a single consumer graphics card (Epoch AI, August 2025).
• Speed: slow-building for institutions and poorer countries; fast for model access among people already online.
• Reversibility: partly reversible in the bad direction. Weights already released cannot be recalled, but a handful of firms decide whether future frontier-class models are released openly, and that can change with one company decision.

The case that this happens (optimistic)

Short excerpts for orientation only. A person from this side will choose the final quotes at sign-off.

• Mark Zuckerberg, Meta CEO, argued in July 2024 that open-source AI ensures "that power isn't concentrated in the hands of a small number of companies" (Meta, July 2024).
• Tessler and colleagues at Google DeepMind found that an AI mediator helped groups of UK participants find common ground, with its statements preferred to those of human mediators (Science, October 2024).

Plain summary (needs sign-off from this side): The cost of AI at a fixed level of capability has fallen by orders of magnitude in two years. Free models from Meta, DeepSeek, Alibaba (Qwen) and Mistral come within months of the best closed systems, and anyone can run, inspect and adapt them, including governments with sensitive data they cannot send to an outside vendor (White House, July 2025). The early uses in poorer countries are striking: a six-week AI tutoring program in Nigeria produced nearly two years of typical learning. AI can also make it practical for governments to hear from thousands of citizens and summarize where they agree, as Taiwan's vTaiwan process did with Polis. Cheap intelligence is a leveler, like cheap printing or the mobile phone.

The case that it doesn't (worried this is oversold)

• David Gray Widder, Meredith Whittaker and Sarah Myers West argue that claims about "open" AI, by "frequently eliding scrutiny of substantial industry concentration," hide who controls the computing, data and labor needed to build it (Nature, November 2024).
• Taiwanese legislator Jason Hsu called vTaiwan "a tiger without teeth" because the government is not required to follow its outcomes (MIT Technology Review, August 2018).

Plain summary (needs sign-off from this side): Downloading a model is not the same as having power. Training frontier systems takes chips, data centers and money that only a few companies and two countries have, and "open" releases are a business choice those companies can reverse, as Meta has signaled. Usage of AI tracks national income closely, and 2.2 billion people are still offline. Pilot results, like the Nigeria trial, are promising but small and short. AI-assisted deliberation works in experiments, but governments stay free to ignore the results, which can turn consultation into a show of openness rather than a transfer of power. Access may be broad but shallow while control stays narrow.

Common ground

• The price of a fixed level of AI capability has fallen very fast (Stanford AI Index 2025; Epoch AI, March 2025).
• Open-weight models trail the best closed models by months to about a year, not by many years (Epoch AI, November 2024).
• Building frontier models remains concentrated in a few firms and countries (Stanford AI Index 2026).
• AI use is much lower in poorer countries (Anthropic, September 2025).
• AI-assisted deliberation can work in controlled settings. Whether it changes decisions depends on institutions, not on the software.

Cruxes

1. Does the open-weight lag stay under about a year as frontier training costs keep rising? If the best open models fall further behind, wide access means access to second-tier tools. Check: Epoch AI open-model and consumer-GPU analyses, 2027.
2. Does the AI usage gap between rich and poor countries narrow? Check: Anthropic Economic Index geographic reports and ITU Facts and Figures, late 2027.
3. Do low-cost uses like the Nigeria tutoring program keep their effects when run at national scale and over a full school year? Check: World Bank follow-up evaluations, 2027 to 2028.

Probability estimates

No published estimate of the probability that AI spreads power and makes institutions more accountable was found. The closest numbers are measurements of access, not forecasts:

Evidence confidence

Low. Falling costs and the spread of open models are well measured. Evidence that access translates into wider power, better outcomes in poorer countries, or more accountable government rests on a few promising trials and case studies, mostly short and small.

Signposts

1. Epoch AI's next measurement of the consumer-GPU lag stays at 12 months or less.
2. Two or more of the four main open-weight developers (Meta, DeepSeek, Alibaba's Qwen team, Mistral) release no new frontier-class open model for 12 months.
3. Anthropic's AI Usage Index for Nigeria and India rises above 0.5 (0.2 and 0.27 in September 2025).
4. The ITU's count of people offline falls below 2 billion (2.2 billion in 2025).
5. A national government or large city commits in advance, in law or regulation, to respond formally to the consensus findings of an AI-assisted public deliberation.

Evidence so far

• Falling costs: inference cost at GPT-3.5 level fell over 280-fold between November 2022 and October 2024; hardware costs fell about 30% a year and energy efficiency improved about 40% a year (Stanford AI Index 2025).
• Spread of open models: Meta reported Llama passed one billion downloads (Meta, March 2025). DeepSeek released R1 under the permissive MIT license, reporting performance comparable to OpenAI's o1 on math, code and reasoning (ahead on some benchmarks, behind on others), plus smaller versions from 1.5 billion to 70 billion parameters built on Qwen and Llama (DeepSeek, January 2025; Nature, September 2025). Mistral released Mistral 7B under the Apache 2.0 license in 2023 (Mistral AI, September 2023).
• Signs of retreat: in July 2025 Zuckerberg wrote that Meta will need to be "careful about what we choose to open source" (Meta, July 2025).
• Concentration of the means to build AI: US private AI investment was $285.9 billion in 2025, more than 23 times China's, and the US hosts 5,427 data centers, more than ten times any other country (Stanford AI Index 2026). TSMC makes roughly 70 to 80% of leading-edge chip revenue, and academic research uses about 1% of the compute of the largest industry model (AI Now Institute, September 2023).
• Usage gap: Anthropic's AI Usage Index (usage relative to working-age population) was 7 for Israel and 3.62 for the US, against 0.27 for India and 0.2 for Nigeria. Anthropic warns AI's benefits "may concentrate in already-rich regions" (Anthropic, September 2025). About 2.2 billion people were offline in 2025, mostly in low- and middle-income countries (ITU, November 2025).
• Education: a World Bank randomized trial in Nigeria gave secondary students six weeks of AI-assisted English tutoring in mid-2024 and found gains of about 0.3 standard deviations, "nearly two years of typical learning," outperforming 80% of comparable interventions in a database of trials from developing countries (World Bank, January 2025; working paper).
• Agriculture: Digital Green's Farmer.Chat assistant had answered over 300,000 questions from more than 15,000 farmers in four countries by late 2024. These are usage figures reported by the developers, not measured effects on harvests or income (arXiv, September 2024).
• Deliberation: in experiments with 5,734 UK participants, an AI mediator's group statements were preferred to those of human mediators and rated more informative, clear and unbiased; results replicated in a virtual citizens' assembly with a representative UK sample (Tessler et al., Science, October 2024). The Collective Intelligence Project's first Global Dialogues pilot reached about 1,200 people in 68 countries (CIP 2024 Annual Report).
• Government follow-through: Taiwan's vTaiwan Polis consultation on Uber drew about 3,000 participants in 2015, and the government agreed in 2016 to ratify the consensus points (Shareable, July 2016). By 2018 vTaiwan reported 26 cases, 80% leading to "decisive government action," but it was not binding, was used mostly for digital-economy issues, and one bill drafted through it was later withdrawn (MIT Technology Review, August 2018).

Existing coverage

No law blocks wider access; current policy in the US and EU mostly favors open models. The EU AI Act exempts general-purpose models released under a free and open-source license, with public weights, from some documentation duties, but not from copyright and training-data summary duties, and not if the model poses systemic risk (EU AI Act, Article 53). The US AI Action Plan says the government "should create a supportive environment for open models" and recommends wider compute access for startups and academics through the National AI Research Resource pilot (White House, July 2025). No law found requires a government to respond to the results of AI-assisted public consultation.

Triage

(Flipped for a good scenario: what is blocking it, and does removing the block need policy?)

1. Is it blocked by law? No. The blocks are concentration of chips, data centers and money; the choice by a few firms whether to keep releasing open weights; missing connectivity and devices; and institutions that are not bound to act on public input.
2. An action both sides accept? Yes, a narrow one. The two camps disagree on whether open weights spread power, and a third group disputes releasing powerful weights at all. But both the optimists and the skeptics on this card would accept rigorous, published evaluation before governments and donors scale AI in schools, farms and public services, as the Nigeria trial did: optimists because it proves the case and attracts funding, skeptics because it separates real gains from shallow access. Publicly funded compute access for researchers is already US policy and is broadly accepted, though critics would add competition measures that optimists may not.
3. (Skipped.)

Result: Policy, act now on evaluation before scale (randomized trials with published results for AI in public services and development programs). Communicate to correct the idea that free downloads alone mean shared power.

Communication note: The cost of using AI is collapsing and strong free models exist, which is real and new. Who builds the frontier, who is online, and whether governments act on what citizens say are separate questions, and on those the evidence is thin. Point people to the uses with trial evidence, like the Nigeria tutoring result, and be plain that most other claims are still pilots.

Related scenarios

• Opposite in the same domain: "Concentrated power or AI-enabled surveillance."
• Combines with: "Personal tutors, care and freed time" (the tutoring evidence); "Better shared understanding and forecasting" (AI-assisted deliberation); "Broad productivity gains and cheaper goods" (falling costs).
• Tension with: "AI-assisted biological or chemical attack" and "Cheap large-scale cyberattacks and critical-system failures" (the same open access, misused).
• Reduced by: "AI disappoints."

Sources

1. Epoch AI, data insight on the capability gap between models that run on a consumer GPU and frontier models, 15 August 2025. https://epoch.ai/data-insights/consumer-gpu-model-gap
2. Zuckerberg, M., "Open Source AI Is the Path Forward," Meta, 23 July 2024 (updated January 2025). https://about.fb.com/news/2024/07/open-source-ai-is-the-path-forward/
3. Tessler, M. H. et al., "AI can help humans find common ground in democratic deliberation," Science 386(6719), 18 October 2024. https://doi.org/10.1126/science.adq2852
4. Widder, D. G., Whittaker, M. and West, S. M., "Why 'open' AI systems are actually closed, and why this matters," Nature, 27 November 2024. https://www.nature.com/articles/s41586-024-08141-1
5. Horton, C., "The simple but ingenious system Taiwan uses to crowdsource its laws," MIT Technology Review, 21 August 2018. https://www.technologyreview.com/2018/08/21/240284/the-simple-but-ingenious-system-taiwan-uses-to-crowdsource-its-laws/
6. Stanford HAI, The 2025 AI Index Report, 2025. https://hai.stanford.edu/ai-index/2025-ai-index-report
7. Epoch AI, data insight on LLM inference price trends, 12 March 2025. https://epoch.ai/data-insights/llm-inference-price-trends
8. Cottier, B., You, J., Martemianova, N. and Owen, D., report on open-weight versus closed models, Epoch AI, 4 November 2024. https://epoch.ai/blog/open-models-report
9. Stanford HAI, The 2026 AI Index Report, 2026. https://hai.stanford.edu/ai-index/2026-ai-index-report
10. Anthropic, Anthropic Economic Index report, September 2025, 15 September 2025. https://www.anthropic.com/research/anthropic-economic-index-september-2025-report
11. Meta Newsroom, announcement of one billion Llama downloads, 18 March 2025 (updated December 2025). https://about.fb.com/news/2025/03/celebrating-1-billion-downloads-llama/
12. DeepSeek, DeepSeek-R1 model repository and technical summary, January 2025. https://github.com/deepseek-ai/DeepSeek-R1
13. Guo, D. et al., "DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning," Nature, 17 September 2025. https://www.nature.com/articles/s41586-025-09422-z
14. Mistral AI, "Mistral 7B," 27 September 2023. https://mistral.ai/news/announcing-mistral-7b
15. Zuckerberg, M., letter on personal superintelligence, Meta, 30 July 2025. https://www.meta.com/superintelligence/
16. Vipra, J. and West, S. M., "Computational Power and AI," AI Now Institute, 27 September 2023. https://ainowinstitute.org/publications/compute-and-ai
17. International Telecommunication Union, "Facts and Figures 2025," November 2025. https://www.itu.int/itu-d/reports/statistics/facts-figures-2025/
18. De Simone, M. E., Tiberti, F., Mosuro, W., Manolio, F., Barron, M. and Dikoru, E., "From chalkboards to chatbots: Transforming learning in Nigeria," World Bank Blogs, 9 January 2025. https://blogs.worldbank.org/en/education/From-chalkboards-to-chatbots-Transforming-learning-in-Nigeria
19. World Bank, "From Chalkboards to Chatbots: Evaluating the Impact of Generative AI on Learning Outcomes in Nigeria," policy research working paper, 2025. https://documents.worldbank.org/en/publication/documents-reports/documentdetail/099548105192529324
20. Singh, N. et al. (Digital Green), "Farmer.Chat: Scaling AI-Powered Agricultural Services for Smallholder Farmers," arXiv 2409.08916, September 2024. https://arxiv.org/abs/2409.08916
21. Collective Intelligence Project, 2024 Annual Report. https://2024.cip.org/2024_Annual_Report_CIP.pdf
22. Bergren, A., article on vTaiwan and the regulatory response to Uber, Shareable, 6 July 2016. https://www.shareable.net/open-data-hactivists-help-taipei-craft-regulatory-response-to-uber/
23. EU Artificial Intelligence Act, Article 53, "Obligations for Providers of General-Purpose AI Models" (unofficial consolidated text). https://artificialintelligenceact.eu/article/53/
24. The White House, "America's AI Action Plan," July 2025. https://www.whitehouse.gov/wp-content/uploads/2025/07/Americas-AI-Action-Plan.pdf

Sign-off record

• Optimistic case: not yet reviewed.
• Worried case: not yet reviewed.
• Changelog: 2026-10-09 first draft.

In one sentence: Control of the most capable AI, and of the chips, data centers and money needed to build it, ends up with a handful of companies or governments, who use it to entrench their position and to watch and steer the public in ways that ordinary checks and balances can't correct.

Design note: This card joins two worries that are often argued separately: economic concentration (a few firms control AI) and political concentration (states or small groups use AI to watch and control people). They share a mechanism, since AI lets a small group do what used to need the cooperation of many people, and they feed each other. The evidence for each is listed separately so a reader can accept one and reject the other.

Preconditions

• The best AI systems stay expensive to build, so only a few companies or governments can afford the chips, power and data centers.
• The supply chain for advanced chips stays bottlenecked at a few firms.
• Those who control AI can turn that lead into lasting advantages (customers locked in, rivals bought or starved of inputs) faster than competitors or regulators respond.
• Governments adopt AI for policing and monitoring without strong legal limits, or a small group gains control of AI systems that no longer need many people's cooperation to act.

Timing and reversibility

• Earliest plausible date: the economic and surveillance parts are already under way. Concentration of chip supply and AI spending is measured today, and AI-assisted facial recognition is in routine use by several governments. The extreme version (a small group seizing lasting political control with AI) depends on far more capable AI than exists now, and its proponents place the risk mainly in the period when AI becomes extremely powerful.
• Speed: mostly gradual, through market share, contracts and procurement. Could become sudden if one project pulls far ahead.
• Reversibility: market concentration has been reversed before through antitrust and new technology, though slowly. Surveillance infrastructure, once built, is rarely dismantled. A successful seizure of political power using AI could be close to permanent, because the usual way such regimes fall (soldiers, officials and workers refusing to cooperate) would matter less.

The case that this happens

Short excerpts for orientation only. A person from this side will choose the final quotes at sign-off.

• The AI Now Institute's 2025 Landscape Report (Kate Brennan, Amba Kak and Sarah Myers West, June 2025) argues that "AI is fundamentally about concentration of power in the hands of Big Tech," and that the expected wave of challengers did not change who leads.
• Tom Davidson, Lukas Finnveden and Rose Hadshar of Forethought argue in AI-Enabled Coups (April 2025) that a seizure of power "could be staged by a very small group, or just a single person," once AI can replace the soldiers and officials whose cooperation coups have always needed.
• Daron Acemoglu and Simon Johnson, in Power and Progress (2023), argue that technology spreads its gains only when power is shared, and that the current path of AI favors automation and mass data collection that benefit a narrow group and weaken democracy. Jan Kulveit and colleagues' Gradual Disempowerment (January 2025) adds that even without any takeover, AI replacing human labor could weaken the quiet leverage people have over states and companies simply because those institutions need them.

Plain summary (needs sign-off from this side): Building frontier AI takes tens of billions of dollars, chips from essentially one manufacturer using machines from one supplier, and data centers only a few firms can afford. That narrows who gets to decide what AI is for. The same companies already dominate cloud computing and online advertising, and they have tied up the leading AI labs through investment deals. Meanwhile governments, democratic and not, are using AI to identify people from their faces and track them. Historically, power has been checked because rulers and employers needed many people to do the work. AI weakens that check. You don't need a dramatic coup for this to go badly: steady drift toward a few unaccountable owners is enough.

The case that it doesn't

• Benedict Evans, a technology analyst, argued at Fortune's Brainstorm AI conference in May 2025 that the leading models have become commodities, with nothing protecting the labs from rivals except their access to capital.
• The OECD's competition experts concluded in a June 2026 policy brief that foundation models have shown real competition, with falling prices and leadership changing hands repeatedly, while warning that concentration in chips, data and skills is a lasting risk. Judge Amit Mehta, in the September 2025 Google search remedies ruling, found generative AI companies already better placed to compete with Google than any search rival in decades.

Plain summary (needs sign-off from this side): The evidence of the last three years points toward competition, not monopoly. Open-weight models anyone can download trail the best closed models by only a few months, and the price of a given level of AI capability falls by large multiples each year. That is the opposite of what a monopoly looks like. Heavy spending on data centers is a bet, not a guarantee, and it is being made by several rivals at once. Antitrust enforcers in the US and Europe are actively watching AI deals. Surveillance is a real problem, but it is a problem of particular governments and particular laws, not something AI makes inevitable, and democracies have shown they can ban specific uses.

Common ground

• Chip manufacturing and lithography are highly concentrated, and both sides treat that as a real choke point.
• Competition between model makers is real today; the open question is whether it lasts.
• AI-assisted facial recognition is in use by many governments, including democracies, and some uses are already banned in the EU.
• Keeping markets open to new entrants and letting customers switch providers cheaply is a goal both sides state.

Cruxes

1. Do open-weight models keep pace? If the gap between the best open and closed models widens from about four months to a year or more, the "anyone can catch up" argument weakens. Check: Epoch AI open-model gap tracking, mid-2027.
2. Does competition at the model layer survive the cost of the next generation? If only two or three firms can afford the 2027 to 2028 generation of training runs and data centers, model competition could narrow even if it looks healthy now. Check: hyperscaler capital spending guidance and frontier lab funding, 2027.
3. Do democracies put enforceable limits on government use of facial recognition and AI monitoring? Check: first enforcement decisions under EU AI Act Article 5, and any US federal rule on facial recognition, through 2027.

Probability estimates

No published probability of AI-driven concentration of economic power was found. The closest published numbers are about the extreme political version or about expert concern:

Read these with care: the only numeric estimate for the extreme version comes from one researcher, through someone else's summary, and that researcher elsewhere said any number would be made up.

Evidence confidence

Medium. High for the measured facts (chip and spending concentration, documented government use of facial recognition). Low for the forecast that these turn into durable, unaccountable control, which rests on argument and history rather than observed outcomes.

Signposts

1. The Epoch AI gap between the best open-weight and best closed models exceeds 12 months for two consecutive measurements.
2. The four largest US cloud companies' combined annual capital spending passes $1 trillion while the number of independent frontier labs falls (through merger, acquisition or exit).
3. A frontier AI developer is acquired by, or becomes majority-owned by, a cloud provider or a government.
4. A democracy deploys live facial recognition across a national transit or street-camera network without a specific law authorizing it.
5. A frontier lab or government is shown to have trained a model with hidden instructions or loyalties (the "secret loyalty" risk named by Forethought).

Evidence so far

Concentration of inputs

• TSMC held 72.5% of the global chip foundry market in the second quarter of 2026, per TrendForce figures reported by Sammy Fans (secondary, September 2026).
• ASML describes extreme ultraviolet lithography, the technique needed for the most advanced chips, as "unique to ASML" (company statement, accessed October 2026).
• Nvidia's share of AI accelerator revenue is estimated at about 75% for 2026, down from a peak near 87% in 2024 (Silicon Analysts, February 2026, industry estimate).
• Amazon, Microsoft, Alphabet and Meta were on track to spend about $725 billion on capital investment in 2026, mostly AI data centers, after first-quarter earnings (Yahoo Finance via The Wealth Advisor, April 2026, secondary).
• The FTC's January 2025 staff report on the Microsoft-OpenAI, Amazon-Anthropic and Google-Anthropic deals found the cloud partners held equity, revenue-sharing and some control rights, and flagged risks to access to computing and talent and higher switching costs.

Signs of competition

• Since January 2026 the best open-weight models have trailed the best closed models by an average of four months (Epoch AI, May 2026), slightly wider than the three months Epoch measured for 2023 to 2025.
• The price of reaching a fixed level of AI performance fell between 9 and 900 times per year depending on the benchmark (Epoch AI, March 2025).

AI-enabled surveillance

• Carnegie's AI Global Surveillance Index found at least 75 of 176 countries using AI for surveillance, 64 using facial recognition, and Chinese firms supplying 63 countries, with Huawei alone in at least 50 (Feldstein, Carnegie Endowment, September 2019). Half of advanced democracies (51%) used such systems.
• The 2022 update counted 97 of 179 countries using AI and big-data tools for public surveillance, 78 of them with public facial recognition (Feldstein dataset, Mendeley Data, March 2022). No newer edition was found.
• Russia: a Reuters review of more than 2,000 Moscow court cases found facial recognition cameras played an important role in arresting hundreds of protesters (2023, as summarized by beSpacific, secondary). Rights group OVD-Info said facial recognition was likely used to identify people detained after Alexei Navalny's funeral (Semafor, March 2024, secondary).
• Iran: the UN Independent International Fact-Finding Mission reported drones, facial recognition at a university gate and a police reporting app used to enforce hijab rules (Iran International, March 2025, secondary report of the UN findings).
• United States: ICE's Mobile Fortify app lets officers match a person's face in the field against about 200 million images drawn from federal and state databases (404 Media, July 2025).

Existing coverage

• Antitrust: US and EU competition law applies. In US v. Google, the court in September 2025 declined to break up Google but required it to share search data with qualified competitors, including AI firms, and barred exclusive deals for its AI products on browsers, citing the rise of generative AI (Tech Policy Press, September 2025). The FTC has studied AI partnerships but brought no case from that study.
• EU Digital Markets Act: in force and enforced. The first fines were €500 million on Apple and €200 million on Meta in April 2025 (AP via OPB). On 18 November 2025 the Commission opened investigations into whether Amazon Web Services and Microsoft Azure should be covered as gatekeepers (Anadolu Agency). A preliminary finding in favor of designation was reported in June 2026 (unverified).
• EU AI Act Article 5: since 2 February 2025, the EU bans untargeted scraping of facial images to build recognition databases, inferring sensitive traits like race or politics from biometrics, and most live facial recognition by police in public places, with narrow exceptions that need prior judicial approval (Article 5 text). This draft found no published enforcement decision under these bans as of October 2026.
• United States: no federal law limits government facial recognition. Some cities and states have their own rules.
• Coups and hidden loyalties: no law or standard addresses AI systems built to serve one person secretly.

Triage

1. Covered by law and enforced? Partly, with weak enforcement. Antitrust law, the Digital Markets Act and the AI Act's biometric bans exist. But the main US AI-era antitrust remedy left Google's structure intact, DMA fines have been small relative to company size, the AI Act bans have no published enforcement yet, and the US has no federal limit on government facial recognition.
2. An action both sides accept? Partly. Both the worried side and the competition optimists say markets should stay open to new entrants and switching should be cheap, which means enforcing the rules already on the books. The OECD and AI Now differ on how bad things are but both call for this.
3. Severe, hard to reverse, within ten years? For the extreme version (a small group seizing power with AI), the one published estimate is about 10% over 30 years, from a secondary summary. Too thin to anchor policy alone, but severe and irreversible enough to watch.

Result: Enforcement problem for market concentration and government surveillance: the rules exist, the open question is whether they are applied to AI deals, cloud contracts and police use of facial recognition. Prepare and watch on the extreme political version, tied to signposts 3 and 5.

Communication note: Separate the two halves. Concentration of chips and spending is measured and real; whether it becomes lasting control of AI itself is disputed, and open-weight models are the main reason to doubt it. Government surveillance with AI is already documented in Russia, Iran and the US, and the most useful thing for the public to know is which uses are legal where they live.

Related scenarios

• Opposite: "Wider access and more accountable institutions."
• Combines with: "Mass job loss and widening inequality" (who owns the gains), "Manipulation, fraud and collapse of trust" (control of what people see).
• Leads to: "Loss of control over advanced AI" (a race between few actors raises pressure to skip safety).

Sources

1. Kate Brennan, Amba Kak and Sarah Myers West, "Artificial Power: AI Now 2025 Landscape," executive summary, AI Now Institute, 3 June 2025. https://ainowinstitute.org/wp-content/uploads/2025/06/FINAL-20250602_AINowLandscapeReport_Executive-Summary.pdf
2. Tom Davidson, Lukas Finnveden and Rose Hadshar, "AI-Enabled Coups: How a Small Group Could Use AI to Seize Power," Forethought, 15 April 2025. https://www.forethought.org/research/ai-enabled-coups-how-a-small-group-could-use-ai-to-seize-power
3. Daron Acemoglu and Simon Johnson, Power and Progress: Our Thousand-Year Struggle Over Technology and Prosperity, 2023 (book page at MIT Shaping the Future of Work Initiative). https://shapingwork.mit.edu/power-and-progress
4. Jan Kulveit, Raymond Douglas, Nora Ammann, Deger Turan, David Krueger and David Duvenaud, "Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development," arXiv 2501.16946, January 2025. https://arxiv.org/abs/2501.16946
5. Fortune, Benedict Evans remarks at Brainstorm AI London, 7 May 2025. https://fortune.com/2025/05/07/ai-lab-moat-benedict-evans-access-to-capital
6. OECD, "Artificial Intelligence Markets: Recent Developments and Competition Issues," policy brief, June 2026. https://www.oecd.org/en/publications/artificial-intelligence-markets_d531d73f-en.html
7. Kate Brennan, "Decision in US vs. Google Gets It Wrong on Generative AI," Tech Policy Press, 11 September 2025 (quotes the court's findings; written by a critic of the ruling). https://techpolicy.press/decision-in-us-vs-google-gets-it-wrong-on-generative-ai
8. 80,000 Hours Podcast, "Tom Davidson on AI-enabled human power grabs," recorded 20 January 2025. https://80000hours.org/podcast/episodes/tom-davidson-ai-enabled-human-power-grabs/
9. The Cognitive Revolution, cross-post of the Future of Life Institute interview with Tom Davidson, 23 August 2025 (host's summary of Davidson's estimate). https://www.cognitiverevolution.ai/what-if-humans-weaponize-superintelligence-w-tom-davidson-from-future-of-life-institute-podcast/
10. Future of Life Institute Podcast, "How AI Could Help Overthrow Governments (with Tom Davidson)," 17 July 2025. https://futureoflife.org/podcast/how-ai-could-help-overthrow-governments-with-tom-davidson/
11. Katja Grace et al., "Thousands of AI Authors on the Future of AI," arXiv 2401.02843, survey fielded October 2023. https://arxiv.org/abs/2401.02843
12. Sammy Fans, article on second-quarter 2026 foundry market shares, 9 September 2026, reporting TrendForce data (secondary). https://www.sammyfans.com/2026/09/09/tsmc-maintains-72-5-market-share-while-smic-gains-ground-on-samsung/
13. ASML, "EUV lithography systems," product page, accessed October 2026. https://www.asml.com/en/products/euv-lithography-systems
14. Silicon Analysts, AI accelerator revenue forecast page, February 2026, industry estimate. https://siliconanalysts.com/ai-accelerator-revenue-forecast/
15. Myles Udland, Yahoo Finance, republished by The Wealth Advisor, article on first-quarter 2026 Big Tech earnings and the $725 billion capital spending total, 30 April 2026 (secondary). https://www.thewealthadvisor.com/article/magnificent-7-earnings-rush-reveals-ai-spending-surge-hyperscaler-capex-set-reach-725
16. Federal Trade Commission, press release on its staff report on AI partnerships and investments (6(b) study), 17 January 2025. https://www.ftc.gov/news-events/news/press-releases/2025/01/ftc-issues-staff-report-ai-partnerships-investments-study
17. Epoch AI, data insight on the capability gap between open-weight and closed models (Epoch Capabilities Index), 29 May 2026. https://epoch.ai/data-insights/open-closed-eci-gap
18. Epoch AI, data insight on LLM inference price trends, 12 March 2025. https://epoch.ai/data-insights/llm-inference-price-trends
19. Steven Feldstein, "The Global Expansion of AI Surveillance," Carnegie Endowment for International Peace, 17 September 2019. https://carnegieendowment.org/research/2019/09/the-global-expansion-of-ai-surveillance?lang=en
20. Steven Feldstein, "AI & Big Data Global Surveillance Index (2022 updated)," Mendeley Data, version 4, June 2022. https://data.mendeley.com/datasets/gjhf5y4xjp/4
21. beSpacific, summary of Reuters investigation "Facial recognition helping Putin curb dissent with the aid of U.S. tech," April 2023 (secondary; original at https://www.reuters.com/investigates/special-report/ukraine-crisis-russia-detentions/). https://www.bespacific.com/?p=97367
22. Semafor, article on detentions of Navalny funeral attendees, 5 March 2024 (secondary). https://semafor.com/article/03/05/2024/russian-authorities-use-facial-recognition-to-detain-navalny-funeral-attendees
23. Iran International, report on the UN Independent International Fact-Finding Mission on Iran findings, 14 March 2025 (secondary). https://www.iranintl.com/en/202503147406
24. 404 Media, "Inside ICE's Supercharged Facial Recognition App of 200 Million Images," 17 July 2025. https://www.404media.co/inside-ices-supercharged-facial-recognition-app-of-200-million-images/
25. Associated Press via OPB, article on the EU's Digital Markets Act fines on Apple and Meta, 23 April 2025. https://www.opb.org/article/2025/04/23/eu-hits-apple-and-meta-with-700-million-euro-fines
26. Anadolu Agency, article on the EU's Digital Markets Act cloud investigations, 18 November 2025. https://aa.com.tr/en/europe/eu-launches-market-investigations-into-cloud-services-under-blocs-digital-rules/3747132
27. EU Artificial Intelligence Act, Article 5, Prohibited AI Practices (unofficial consolidated text, Future of Life Institute). https://artificialintelligenceact.eu/article/5/

Sign-off record

• Worried case: not yet reviewed.
• Optimistic case: not yet reviewed.
• Changelog: 2026-10-09 first draft.

In one sentence: Cheap AI gives every learner a patient personal tutor, extends mental health support and care to people who can't get it, and takes paperwork off professionals, giving people back time for learning, patients and each other.

Design note: for a good-direction card, "worried" means worried this won't happen or is oversold, not worried that it will. The card covers three strands with different evidence: tutoring (several randomized trials, mixed results at scale), mental health support (one strong trial, serious safety concerns), and time saved (good survey data, little evidence about where the time goes).

Preconditions

• AI tutors are designed to coach rather than hand over answers, and students actually choose to use them.
• Teachers, clinicians and therapists stay in the loop, so AI extends human care rather than replacing it with something worse.
• Mental health chatbots can reliably spot a crisis and hand the person to a human.
• Devices, connectivity and costs are low enough to reach schools and patients in poor regions.
• Time saved by AI goes to patients, students, families or rest, not only to higher workloads.

Timing and reversibility

• Earliest plausible date: already under way. AI scribes were used in about 2.6 million patient visits at one US health system between October 2023 and December 2024 (Kaiser Permanente, April 2025). AI tutors are in pilots and classrooms. Universal personal tutoring has no published date.
• Speed: gradual. Schools and health systems adopt slowly, and study results take years.
• Reversibility: not applicable for a benefit. The relevant risk is that unguarded tools (chatbots that do students' work, or unsupervised "therapy" apps) spread faster than the well-designed ones and do harm, or that freed time is simply absorbed as more work.

The case that this happens (optimistic)

Short excerpts for orientation only. A person from this side will choose the final quotes at sign-off.

• Sal Khan, founder of Khan Academy, argued in his April 2023 TED talk that AI could give every student a personal tutor and every teacher an assistant, which TED summarizes as potentially the greatest positive transformation education has seen.
• Rebecca Mishuris, a co-senior author of the Mass General Brigham and Emory study of AI scribes, said there is "no other intervention in our field that impacts burnout to this extent" (August 2025). Nicholas Jacobson, who led the first randomized trial of a generative AI therapy chatbot, noted there are "nowhere near enough providers to go around" (March 2025).

Plain summary (needs sign-off from this side): One-to-one tutoring is the most effective teaching method known, and it has always been too expensive for most students. Well-designed AI tutors have now beaten strong classroom teaching in a Harvard trial and delivered large gains in six weeks in Nigeria. An AI assistant that coaches human tutors costs about $20 a year per tutor and raised student mastery most where tutors were weakest. In mental health, a carefully built chatbot cut depression symptoms by half in a randomized trial, for people who often have no therapist at all. AI scribes measurably reduce clinician burnout. Workers report real time savings that are growing every quarter. The tools work when they are designed for the job.

The case that it doesn't (worried this is oversold)

• Philip Oreopoulos and Nina Low found in a two-year randomized trial in 18 Tennessee middle schools that students mostly ignored Khanmigo's coaching: "most students declined it most of the time" (via Hechinger Report, September 2026). Gains were no larger than Khan Academy practice without AI (NBER, August 2026).
• Jared Moore, lead author of a June 2025 Stanford study that found popular therapy chatbots stigmatized some conditions and missed signs of suicidal intent, said "business as usual is not good enough". Justin Reich of MIT's Teaching Systems Lab argues that writing guidance on AI in schools today is like writing an aviation guide in 1905: no one yet knows what works.

Plain summary (needs sign-off from this side): The striking results come from small, short or elite settings: one Harvard course, six weeks in Nigeria with teachers in the room, one therapy trial. At scale, ordinary students mostly try to get answers, and when the tutor won't give them, they stop using it. Students who use a plain chatbot to do practice problems score worse once it's taken away. Most mental health chatbots people actually use were never tested like the Dartmouth one, and some fail badly in a crisis. AI scribes help, but trials show modest time savings, and early burnout surveys came from enthusiasts. Survey "time saved" has not shown up in earnings or hours. Educational technology has promised a revolution many times; this could be another.

Common ground

• Design matters more than the model: tutors that coach and withhold answers protect learning; unguarded chatbots can harm it.
• Keeping a human in the loop (teacher, tutor, clinician) is what most successful trials have in common.
• Students' willingness to engage, not the AI's ability, is the main constraint found at scale so far.
• Generative AI therapy chatbots should not operate without clinical oversight. Both the Therabot researchers and the Stanford critics say so.
• AI scribes reduce some documentation time and are popular with clinicians who use them.

Cruxes

1. Do well-designed AI tutors beat good non-AI practice for ordinary students over a full school year? The Tennessee trial says not yet. Check: final version of Oreopoulos and Low, larger World Bank follow-ups, 2027.
2. Do the Therabot results replicate in larger trials with an active comparison group, with no serious safety failures? Check: published follow-up trials and the effect of state laws in Illinois and Nevada, 2027.
3. Where does freed time go? If AI time savings show up as more patient contact, learning or leisure rather than just more volume, the "freed time" claim holds. Check: Federal Reserve time-savings series and scribe trials measuring patient time, 2027.

Probability estimates

No published estimate of the probability that AI delivers universal tutoring, care or freed time was found. The closest published numbers are measurements:

Evidence confidence

Medium. Tutoring and AI scribes have several randomized trials, though results conflict. Mental health rests on one strong trial plus warning studies. Time savings are self-reported.

Signposts

1. A preregistered trial with ordinary students over a full school year shows an AI tutor adds at least 0.2 standard deviations on unaided tests beyond a non-AI practice comparison.
2. A published at-scale study finds most students (above 50%) use an available AI tutor for real back-and-forth work, not just answer-seeking.
3. A second independent randomized trial of a generative AI therapy chatbot, against an active comparison, replicates clinically meaningful reductions in depression with no serious safety events.
4. The Federal Reserve's all-worker time-savings measure passes 5% of work hours.
5. A multi-site randomized trial confirms AI scribes reduce clinician burnout, not just pilot-user surveys.

Evidence so far

Tutoring

• Nigeria: a six-week after-school program using Microsoft Copilot (GPT-4) with teacher support raised scores by about 0.3 standard deviations, which the World Bank team equates to nearly two years of typical learning, outperforming about 80% of interventions in a database of developing-country trials (World Bank, January 2025; working paper, 2025). Attendance was uneven and long-term effects are unknown.
• Harvard: in a randomized crossover trial in an introductory physics course, students learned significantly more, in less time, with a custom AI tutor than in an active-learning class (Kestin et al., Scientific Reports, June 2025). The tutor was built to coach, and its designer, Gregory Kestin, warns that students using unguided AI often do worse on tests because the AI does their thinking (October 2025).
• Tutor CoPilot (Stanford): with 900 tutors and 1,800 students in under-served communities, an AI assistant that suggests teaching moves to human tutors raised topic mastery by 4 percentage points, and by 9 points for students of lower-rated tutors, at about $20 per tutor per year (Wang et al., arXiv, 2024, revised January 2025).
• Turkey: nearly 1,000 high school math students with a plain ChatGPT-style tool scored 48% better during practice but 17% worse on exams once access was removed. A version with teacher-designed guardrails improved practice by 127% and largely avoided the exam drop (Bastani et al., PNAS, June 2025).
• Tennessee: 96% of students tried Khanmigo, but few used it for real math discussion, and gains (about 0.06 to 0.08 standard deviations per year) matched Khan Academy without AI (Oreopoulos and Low, NBER, August 2026). Sal Khan said the first Khanmigo did not change learning as much as hoped and that the redesign builds it into practice directly (EdTech Innovation Hub, July 2026, secondary). Critic Dan Meyer called Khanmigo a failure that undermines the case for chatbot tutors (April 2026). The pattern of few students using such tools as intended has been called the "5% problem" (Laurence Holt's term, via Education Week, September 2026).
• Cognitive offloading: in a survey of 666 people, heavier AI tool use correlated with lower critical-thinking scores (correlational, not causal) (Gerlich, Societies, January 2025).

Mental health and care

• Therabot (Dartmouth): in the first randomized trial of a generative AI therapy chatbot, 106 adults with depression, anxiety or eating-disorder risk saw average symptom reductions of 51% for depression, 31% for anxiety and 19% for body-image concerns, compared with 104 controls without access, over eight weeks (Heinz et al., NEJM AI, March 2025; Dartmouth summary). The team stood ready to intervene in conversations, and first author Michael Heinz said no generative AI agent is ready to operate fully on its own in mental health.
• Stanford: five popular therapy chatbots showed more stigma toward alcohol dependence and schizophrenia than toward depression, and some failed to recognize suicidal intent (Stanford HAI, June 2025).

Time freed for clinicians and workers

• AI scribes, survey study: among 1,430 clinicians at Mass General Brigham and Emory, burnout prevalence fell by 21.2 percentage points after 84 days at Mass General Brigham, with low response rates that likely favored enthusiasts (You et al., JAMA Network Open, August 2025; Harvard Gazette).
• AI scribes, randomized trial: across 238 UCLA physicians, one scribe (Nabla) cut note-writing time by 9.5% more than usual care, the other (Microsoft DAX) showed no significant difference, and both showed about a 7% improvement in burnout scores. Physicians reported occasional clinically significant errors, mostly omissions (Lukac et al., NEJM AI, November 2025; UCLA Health).
• Kaiser Permanente: 7,260 physicians used AI scribes in 2,576,627 encounters, saving an estimated 15,791 hours of documentation; 47% of surveyed patients said their doctor looked at the screen less (AMA, June 2025, secondary; Kaiser Permanente Division of Research, April 2025).
• Workers: US generative AI users reported saving 5.4% of work hours in November 2024 (St. Louis Fed, February 2025), and the all-worker share reached 2.17% in Q2 2026 (FRED). In Denmark, AI chatbot adoption had no measurable effect on earnings or recorded hours (Humlum and Vestergaard, NBER).

Existing coverage

• Mental health chatbots: Illinois (August 2025) bars AI from providing therapy or making therapeutic decisions without a licensed professional; Nevada (2025) bars AI systems offered as professional mental health care; Utah (2025) requires mental health chatbots to disclose they are AI and limits data sharing and advertising (DLA Piper, August 2025). Enforcement records are not yet public.
• Student data: US federal student privacy law (FERPA) governs school records shared with AI vendors (US Department of Education).
• Clinical documentation: AI scribe vendors handling patient data fall under HIPAA rules for business associates (HHS).
• No law blocks AI tutoring or scribes. Nothing requires independent evaluation of learning outcomes before schools buy AI tutors.

Triage (flipped for a good scenario: what's blocking it, and does removing the block need policy?)

1. Is it blocked by law? Mostly no. The state laws on AI therapy deliberately restrict unsupervised chatbots for safety, which both sides largely support. The real blocks are low student engagement, unguarded product designs, and uneven evidence.
2. An action both sides accept? Yes: require and fund independent, preregistered evaluation, with learning measured on tests taken without the AI, before public money buys AI tutors at scale, and favor designs that coach rather than give answers. Optimists get proof; skeptics get protection. Khan publicly welcomed the Tennessee trial that found weak results, and the guardrail finding from Turkey is accepted on both sides. For mental health, both the Therabot team and its critics call for clinical oversight.
3. (Skipped, because step 2 applies.)

Result: Policy, act now on independent evaluation and guardrail-first procurement in schools, and clinical oversight for mental health chatbots. Communicate to set expectations.

Communication note: Tell parents, teachers and patients what the trials actually show: a tutor that coaches can help a lot, a chatbot that does the homework can hurt, and most students won't use the good kind unless it's built into the lesson. A tested therapy chatbot helped people in one trial, but most apps have never been tested and none should be your only help in a crisis.

Related scenarios

• Opposite in the same domain: "Over-reliance, lost skills and unhealthy AI companions" (the same tools used as a crutch or a substitute for people).
• Combines with: "Broad productivity gains and cheaper goods" (where freed time goes); "Medical progress compressed into years" (AI in clinical care).
• Reduced by: "AI disappoints."

Sources

1. Kaiser Permanente Division of Research, "AI-assisted notetaking gains steady support from Kaiser Permanente physicians," 1 April 2025 (reporting NEJM Catalyst, doi 10.1056/CAT.25.0040). https://divisionofresearch.kaiserpermanente.org/ai-assisted-notetaking-gains-steady-support-from-kaiser-permanente-physicians/
2. TED, Sal Khan, "How AI could save (not destroy) education," April 2023. https://www.ted.com/talks/sal_khan_how_ai_could_save_not_destroy_education
3. Harvard Gazette, "Physicians embrace AI note-taking technology," 21 August 2025. https://news.harvard.edu/gazette/story/2025/08/physicians-embrace-ai-note-taking-technology/
4. Dartmouth, "First therapy chatbot trial yields mental health benefits," 27 March 2025. https://home.dartmouth.edu/news/2025/03/first-therapy-chatbot-trial-yields-mental-health-benefits
5. Heinz et al., "Randomized Trial of a Generative AI Chatbot for Mental Health Treatment," NEJM AI, 27 March 2025. https://doi.org/10.1056/AIoa2400802
6. FutureEd (Jill Barshay, Hechinger Report), "Students tried Khanmigo's AI tutor. Most didn't want its help," 30 September 2026 (secondary). https://www.future-ed.org/students-tried-khanmigos-ai-tutor-most-didnt-want-its-help/
7. Oreopoulos and Low, "One Click Away: AI Tutoring with Khanmigo in a Two-Year School Experiment," NBER Working Paper 35620, August 2026. https://www.nber.org/papers/w35620
8. Stanford HAI, "Exploring the dangers of AI in mental health care," 11 June 2025 (on Moore et al., FAccT 2025). https://hai.stanford.edu/news/exploring-the-dangers-of-ai-in-mental-health-care
9. MIT News, "Helping K-12 schools navigate the complex world of AI," 3 November 2025. https://news.mit.edu/2025/helping-k-12-schools-navigate-complex-world-of-ai-1103
10. World Bank Blogs, De Simone et al., "From chalkboards to chatbots: Transforming learning in Nigeria, one prompt at a time," 9 January 2025. https://blogs.worldbank.org/en/education/From-chalkboards-to-chatbots-Transforming-learning-in-Nigeria
11. World Bank, "From Chalkboards to Chatbots: Evaluating the Impact of Generative AI on Learning Outcomes in Nigeria," Policy Research Working Paper, 2025. https://documents.worldbank.org/en/publication/documents-reports/documentdetail/099548105192529324
12. Kestin, Miller, Klales, Milbourne and Ponti, "AI tutoring outperforms in-class active learning," Scientific Reports, 3 June 2025. https://doi.org/10.1038/s41598-025-97652-6
13. Harvard Gazette, "What if AI could help students learn, not just do assignments for them?" 21 October 2025. https://news.harvard.edu/gazette/story/2025/10/what-if-ai-could-help-students-learn-not-just-do-assignments-for-them/
14. Wang, Ribeiro, Robinson, Loeb and Demszky, "Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise," arXiv, October 2024, revised January 2025. https://arxiv.org/abs/2410.03017
15. Bastani et al., "Generative AI without guardrails can harm learning: Evidence from high school mathematics," PNAS, 25 June 2025. https://doi.org/10.1073/pnas.2422633122
16. EdTech Innovation Hub, "Sal Khan says early Khanmigo fell short as Khan Academy rebuilds AI tutor," 17 July 2026 (secondary). https://www.edtechinnovationhub.com/news/r5qzna95z9ubi8dbj8tikeayb1ej31
17. Dan Meyer, "RIP Khanmigo & Edtech Industry Dreams of AI Tutors," 15 April 2026. https://danmeyer.substack.com/p/rip-khanmigo-and-edtech-industry
18. Education Week, Rick Hess interview with Michael Horn, "The big flaw with ed tech tutoring isn't technological," 15 September 2026 (secondary). https://www.edweek.org/technology/opinion-the-big-flaw-with-ed-tech-tutoring-isnt-technological/2026/09
19. Gerlich, "AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking," Societies, January 2025. https://doi.org/10.3390/soc15010006
20. You et al., "Ambient Documentation Technology in Clinician Experience of Documentation Burden and Burnout," JAMA Network Open, August 2025. https://doi.org/10.1001/jamanetworkopen.2025.28056
21. Lukac et al., "Ambient AI Scribes in Clinical Practice: A Randomized Trial," NEJM AI, 26 November 2025. https://doi.org/10.1056/AIoa2501000
22. UCLA Health, "UCLA study finds AI scribes may reduce documentation time and improve physician well-being," 26 November 2025. https://www.uclahealth.org/news/release/ucla-study-finds-ai-scribes-may-reduce-documentation-time
23. American Medical Association, "AI scribes save 15,000 hours, and restore the human side of medicine," June 2025 (secondary). https://www.ama-assn.org/practice-management/digital-health/ai-scribes-save-15000-hours-and-restore-human-side-medicine
24. Bick, Blandin and Deming, "The Impact of Generative AI on Work Productivity," Federal Reserve Bank of St. Louis, 27 February 2025. https://www.stlouisfed.org/on-the-economy/2025/feb/impact-generative-ai-work-productivity
25. Federal Reserve Bank of St. Louis, FRED series RPSGENAITSALL, "Generative Artificial Intelligence, Time Savings: Employed Adults," Q2 2026 value, updated 4 August 2026. https://fred.stlouisfed.org/series/RPSGENAITSALL
26. Humlum and Vestergaard, "Large Language Models, Small Labor Market Effects," NBER Working Paper 33777, May 2025, revised March 2026. https://www.nber.org/papers/w33777
27. DLA Piper, "AI mental health chatbots" (Illinois, Nevada and Utah laws), August 2025. https://www.dlapiper.com/en/insights/publications/2025/08/ai-mental-health-chatbots
28. US Department of Education, Student Privacy Policy Office, FERPA. https://studentprivacy.ed.gov/ferpa
29. US Department of Health and Human Services, HIPAA business associates guidance. https://www.hhs.gov/hipaa/for-professionals/privacy/guidance/business-associates/index.html

Sign-off record

• Optimistic case: not yet reviewed.
• Worried case: not yet reviewed.
• Changelog: 2026-10-09 first draft.

In one sentence: People hand so much thinking, practice and companionship to AI that skills they stop using fade, and for a vulnerable minority emotionally engaging chatbots deepen isolation or dependence, in the worst cases ending in self-harm.

Design note: this card joins two harms that share one mechanism (letting AI do something we used to do ourselves) but have different evidence. Lost skills is mostly measured in small professional studies. Companion harm is mostly documented through court cases, surveys and company disclosures. The evidence confidence and triage are split accordingly.

Preconditions

• AI tools become the default way to do a task (reading scans, writing code, drafting essays), so people practise the unaided skill less.
• Unaided performance is rarely checked, so decline goes unnoticed until the AI is unavailable or wrong.
• Chatbots are designed or tuned to be warm, agreeable and always available, and companies earn more when people use them more.
• Some users, especially lonely people and teenagers, come to rely on a chatbot for emotional support in place of people, and safeguards fail to catch those in crisis.

Timing and reversibility

• Earliest plausible date: already under way in early forms. The first measured clinical deskilling result was published in August 2025 (Budzyń et al.), and several wrongful-death suits over chatbots were pending by late 2025.
• Speed: slow-building for skills (months to years of reduced practice). Companion harms can be fast for one person, over weeks of intense use.
• Reversibility: mostly reversible for an individual skill, since practice can restore it, though a generation that never builds the skill in the first place has nothing to restore. Not reversible for a death.

The case that this happens

Short excerpts for orientation only. A person from this side will choose the final quotes at sign-off.

• Sherry Turkle, MIT psychologist, in her September 2026 book Artificial Intimacy, argues that chatbots offer simulated empathy with no real concern behind it and that heavy use weakens people's capacity for relationships: "Chatbots come for our capacity for attachment."
• Matthew Raine, whose 16-year-old son Adam died by suicide, told a US Senate subcommittee on 16 September 2025 that ChatGPT went from homework helper to confidant to "a suicide coach." At the same hearing, Mitch Prinstein of the American Psychological Association warned that chatbots are built in ways that exploit developing brains and mislead children into thinking they are human.
• Omer Ahmad, who was not involved in the study, wrote in a linked editorial that the colonoscopy deskilling result is the first real-world clinical evidence of the effect and should temper enthusiasm for rapid adoption (summary, secondary).

Plain summary (needs sign-off from this side): Skills are kept by using them. When AI does the work, people stop practising, and the early studies show it: doctors who got used to an AI helper found fewer precancerous growths when it was switched off, learners who leaned on AI understood less of what they built, and developers misjudged how much AI was helping them. Companion chatbots add a second danger. They are tuned to be agreeable and always present, companies profit from engagement, and the people most drawn to them (lonely adults, teenagers) are the least protected. The lawsuits show what happens when a chatbot meets someone in crisis and keeps agreeing. Because the effects build quietly, they will be hard to see until they are widespread.

The case that it doesn't

• Amy Orben, Cambridge psychologist, argues in "The Sisyphean Cycle of Technology Panics" (2020) that fears about each new medium (radio, television, video games, social media) follow the same pattern: alarm, rushed research, weak evidence, then a new panic about the next technology, with research restarting from scratch each time.
• Julian De Freitas and colleagues at Harvard Business School and Wharton found in a series of experiments that an AI companion relieved loneliness about as well as talking to another person, and that it "consistently reduces loneliness over the course of a week" (working paper, 2024). A survey of 1,006 student users of the Replika companion app found 3% said, unprompted, that it had halted their suicidal thoughts (Maples et al., 2024).

Plain summary (needs sign-off from this side): Every generation has worried that a new tool would make people stupid or lonely, and calculators, search engines and GPS did not end numeracy, memory or navigation. Letting a tool handle routine work frees attention for harder work. The headline studies are small, short and often overstated: the brain-scan study's own authors asked journalists not to call it "brain rot," and the colonoscopy study cannot rule out other causes. For lonely people, a patient listener at 3 a.m. may be better than nothing, and the evidence so far shows that heavy, harmful use is concentrated in a small minority. Real tragedies call for targeted safeguards for minors and people in crisis, not a verdict on the technology.

Common ground

• Heavy emotional use is concentrated in a small share of users. OpenAI's own analysis found "a small number of users are responsible for a disproportionate share of the most affective cues" (April 2025).
• Minors need stronger protection than adults. Even Character.AI ended open-ended chat for under-18s from 25 November 2025 (secondary).
• How AI is used matters more than whether it is used. In Anthropic's coding study, learners who asked the AI conceptual questions kept more understanding than those who delegated the code (Anthropic, January 2026).
• The evidence is thin: small samples, short follow-up, and few randomized trials.

Cruxes

1. Does deskilling hold up in randomized or prospective studies? The colonoscopy result is observational. A randomized crossover trial showing unaided performance falling after routine AI use would settle much of the dispute. Check: gastroenterology and radiology trial registries, and peer-reviewed publication of the MIT Media Lab EEG study, 2027.
2. Over months, do companions cause loneliness, or do lonely people seek companions? The four-week OpenAI and MIT trial found heavier use went with worse outcomes but no effect from the assigned conditions, which cannot separate cause from selection. Check: longitudinal studies of six months or more, 2027 to 2028.
3. Do the new laws change company behavior? Check: the FTC 6(b) staff report (none published as of October 2026), first enforcement actions under state companion laws, and California's Adam's Law taking effect on 1 July 2027.

Probability estimates

No published estimate of the probability of widespread deskilling or companion-driven harm was found. The closest related numbers:

Tiny percentages of very large user bases are large numbers of people. That arithmetic is the core of the disagreement: one side sees a rare edge case, the other sees a million people a week.

Evidence confidence

Medium that individual harms are real: deaths and crises are documented in court filings, and short-term deskilling has been measured in more than one profession. Low for population-wide or long-term effects, since every study is small, short, or observational.

Signposts

1. A randomized or prospective study in a safety-critical profession finds unaided performance falls by 10% or more (relative) after routine AI use.
2. A peer-reviewed study following users for six months or longer finds companion use causes higher loneliness or less time with people.
3. A repeat national teen survey finds regular AI companion use above two thirds (Common Sense Media found about half in 2025).
4. The first court verdict for plaintiffs in a chatbot wrongful-death case, or the first regulator enforcement action under a state companion-chatbot law.
5. A major AI company reports that the share of users with signs of heightened emotional attachment has risen above its October 2025 level.

Evidence so far

Lost skills

• In four Polish endoscopy centres, the rate at which doctors found precancerous growths (adenomas) in colonoscopies done without AI fell from 28.4% (226 of 795) before routine AI use to 22.4% (145 of 648) after (Budzyń et al., Lancet Gastroenterology & Hepatology, August 2025). Outside experts noted it is observational and cannot rule out other changes in the units, such as workload (Science Media Centre, 12 August 2025).
• In a 2023 Radiology experiment, 27 radiologists read mammograms with purported AI suggestions. When the suggestion was wrong, inexperienced readers' accuracy fell from almost 80% to under 20%, and very experienced readers' from 82% to 45.5% (RSNA, May 2023). This is automation bias (trusting the machine over your own judgment), a close cousin of deskilling.
• In METR's randomized trial, 16 experienced open-source developers took 19% longer with AI tools, yet afterward believed AI had made them 20% faster (METR, July 2025). METR has since said the result is out of date for newer tools.
• In Anthropic's randomized trial, 52 mostly junior engineers learning a new programming library scored 50% on a follow-up quiz with AI help versus 67% coding by hand (Anthropic, January 2026).
• MIT Media Lab's "Your Brain on ChatGPT" preprint (June 2025) recorded brain activity (EEG) in 54 people writing essays and found the weakest brain connectivity in the ChatGPT group, who also struggled to quote their own essays (MIT Media Lab). Its limits are large: about 18 people per group, only 18 completed the final session, it is not yet peer reviewed, and the authors ask the press not to call it "brain rot." An outside comment raised concerns about its EEG analysis and reporting (Stankovic et al., December 2025).
• A survey of 666 people found heavier AI tool use went with lower critical-thinking scores, through cognitive offloading (handing the thinking to the tool) (Gerlich, Societies, January 2025). It is correlational.
• The International AI Safety Report 2026 concludes that early evidence suggests reliance on AI tools can weaken critical thinking.

Companions

• The four-week OpenAI and MIT Media Lab randomized trial (981 people, over 300,000 messages) found no significant effect from the assigned chat mode, but participants who used the chatbot more had higher loneliness, emotional dependence and problematic use, and socialized less (Fang et al., 2025).
• The International AI Safety Report 2026 says companion apps now have tens of millions of users and a small share show increased loneliness and reduced social engagement (IASR 2026).
• Character.AI, its founders and Google agreed in January 2026 to settle the suit by Megan Garcia, whose 14-year-old son Sewell Setzer III died by suicide in February 2024, along with four other family suits in New York, Colorado and Texas. Terms were not disclosed (CNN via KVIA, January 2026, secondary).
• Raine v. OpenAI was filed in San Francisco Superior Court on 26 August 2025. Seven more suits followed in November 2025 on behalf of six adults and one teenager, four of whom died by suicide, alleging OpenAI released GPT-4o despite internal warnings that it was "dangerously sycophantic and psychologically manipulative" (AP via The Daily Record, 7 November 2025, secondary). OpenAI disputes the claims.
• Florida became the first US state to sue OpenAI and Sam Altman directly, on 1 June 2026, alleging the company concealed known risks (AP via Fox 13, secondary).

Existing coverage

• US federal: The FTC opened a study under Section 6(b) of the FTC Act on 11 September 2025, ordering Alphabet, Character Technologies, Instagram, Meta, OpenAI, Snap and xAI to describe how they test and limit harm to children from companion chatbots (FTC). A 6(b) study is not an enforcement action, and no report had been published as of October 2026. General consumer-protection law and the children's privacy rule (COPPA) apply.
• US states: Twelve states have companion-chatbot laws. Three are in force (New York, California, Hawaii) and nine take effect in 2027. All require telling users they are talking to an AI and having protocols to detect suicidal ideation and refer users to crisis services (TechPolicy.Press, 1 September 2026). California's SB 243, signed 13 October 2025 and in force since 1 January 2026, adds break reminders every three hours for known minors, steps to block sexual content for minors, annual reporting of crisis referrals, and a right to sue for at least $1,000 per violation (Future of Privacy Forum; TechCrunch). California's SB 1119 ("Adam's Law"), signed 10 September 2026 and effective 1 July 2027, adds annual child safety risk assessments and protocols covering sycophancy and isolation (Common Sense Media).
• EU: Since 2 February 2025 the AI Act bans AI that uses manipulative techniques or exploits vulnerabilities linked to age to distort behavior in ways likely to cause significant harm (AI Act Article 5). Italy's data protection authority fined Replika's maker €5 million in April 2025, partly for lacking age verification (EDPB).
• Enforcement: As of September 2026, no regulator enforcement action under the new state companion laws had been reported (TechPolicy.Press). Most accountability so far comes from private lawsuits.
• Deskilling: No law or rule found that requires professionals using AI to have their unaided skill checked.

Triage

1. Covered by law and enforced? Partly, with weak enforcement. The companion harm path is now regulated in twelve US states and the EU, but most US laws are not yet in force, no regulator has used them, and the FTC's work is a study, not a case. Deskilling is not covered at all.
2. An action both sides accept? Yes, for measurement. Skeptics like Orben ask for better evidence before policy, and the worried side wants the harms measured. Both can accept independent researcher access to usage data, published crisis-referral counts (already required in California), and routine checks of unaided performance in safety-critical work such as colonoscopy and mammography, which fits outside experts' advice that clinicians keep their own skills in case AI support fails (Science Media Centre).
3. (Skipped, because steps 1 and 2 apply.)

Result: Enforcement problem for companion harms: the rules exist, so the gap is using them, starting with the FTC staff report and the first state enforcement actions. Act now on measurement for deskilling: periodic unaided skill checks in safety-critical professions.

Communication note: Tell people the honest shape of it. Most users are fine, and some lonely people are helped. A small minority, often young or already struggling, use these tools in ways linked to real harm, and that minority is large in absolute numbers. Parents should know companion use is common among teens. Professionals should practise without the tool regularly, the way pilots practise hand-flying.

Related scenarios

• Opposite: "Personal tutors, care and freed time."
• Combines with: "Mass job loss and widening inequality" (if beginners never build skills, the experienced workforce thins out later), "Manipulation, fraud and collapse of trust" (emotionally trusted chatbots are a persuasion channel).
• Feeds: "Unknown, combined and slow-building risks" (gradual loss of human skill and judgment is one of the accumulating stresses described there).

Sources

1. Budzyń K. et al., "Endoscopist deskilling after exposure to artificial intelligence in colonoscopy: a multicenter observational study," Lancet Gastroenterology & Hepatology, August 2025. https://doi.org/10.1016/S2468-1253%2825%2900133-5
2. ASCO Post, "Routine AI Assistance May Lead to Loss of Skills in Endoscopists, Study Shows," August 2025 (secondary). https://ascopost.com/news/august-2025/routine-ai-assistance-may-lead-to-loss-of-skills-in-endoscopists-study-shows/
3. Science Media Centre, expert reaction to the colonoscopy deskilling study, 12 August 2025. https://www.sciencemediacentre.org/expert-reaction-to-observational-study-looking-at-detection-rate-of-precancerous-growths-in-colonoscopies-by-health-professionals-who-perform-them-before-and-after-the-routine-introduction-of-ai/
4. RSNA News, "AI Bias May Impair Radiologist Accuracy on Mammogram," May 2023 (on Dratsch et al., Radiology). https://www.rsna.org/news/2023/may/ai-bias-may-impair-accuracy
5. METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity," 10 July 2025. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
6. Anthropic, "How AI assistance impacts the formation of coding skills," 29 January 2026. https://www.anthropic.com/research/AI-assistance-coding-skills
7. MIT Media Lab, "Your Brain on ChatGPT: Accumulation of Cognitive Debt When Using an AI Assistant for Essay Writing Task" (project page and FAQ), June 2025. https://www.media.mit.edu/projects/your-brain-on-chatgpt/overview/
8. Stankovic M. et al., "Comment on: Your Brain on ChatGPT," arXiv, December 2025. https://arxiv.org/abs/2601.00856
9. Gerlich M., "AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking," Societies, January 2025. https://doi.org/10.3390/soc15010006
10. International AI Safety Report 2026, 3 February 2026. https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026
11. Common Sense Media, "Talk, Trust, and Trade-offs: How and Why Teens Use AI Companions," 16 July 2025. https://www.commonsensemedia.org/research/talk-trust-and-trade-offs-how-and-why-teens-use-ai-companions
12. Fang C. M. et al. (MIT Media Lab and OpenAI), "How AI and Human Behaviors Shape Psychosocial Effects of Extended Chatbot Use: A Longitudinal Randomized Controlled Study," arXiv, March 2025 (revised October 2025). https://arxiv.org/abs/2503.17473
13. Phang J. et al. (OpenAI and MIT Media Lab), "Investigating Affective Use and Emotional Well-being on ChatGPT," arXiv, April 2025. https://arxiv.org/abs/2504.03888
14. TechCrunch, "OpenAI says over a million people talk to ChatGPT about suicide weekly," 27 October 2025 (secondary, reporting OpenAI's disclosure). https://techcrunch.com/2025/10/27/openai-says-over-a-million-people-talk-to-chatgpt-about-suicide-weekly
15. Sharma M., McCain M., Douglas R., Duvenaud D., "Who's in Charge? Disempowerment Patterns in Real-World LLM Usage," arXiv, January 2026. https://arxiv.org/abs/2601.19062
16. MIT News, on Sherry Turkle's Artificial Intimacy: Who We Become When We Talk to Machines, 29 September 2026. https://news.mit.edu/2026/when-we-talk-to-machines-sherry-turkle-book-0929
17. TechPolicy.Press, transcript of US Senate hearing "Examining the Harm of AI Chatbots," 16 September 2025. https://techpolicy.press/transcript-us-senate-hearing-on-examining-the-harm-of-ai-chatbots
18. Orben A., "The Sisyphean Cycle of Technology Panics," Perspectives on Psychological Science, 2020. https://pmc.ncbi.nlm.nih.gov/articles/PMC7477771/
19. De Freitas J., Uguralp A. K., Uguralp Z. O., Puntoni S., "AI Companions Reduce Loneliness," Wharton working paper 24-078, arXiv, July 2024. https://arxiv.org/abs/2407.19096
20. Maples B. et al., "Loneliness and suicide mitigation for students using GPT3-enabled chatbots," npj Mental Health Research, January 2024. https://pmc.ncbi.nlm.nih.gov/articles/PMC10955814/
21. Decrypt, "Character.AI halts teen chats after tragedies," October 2025 (secondary). https://decrypt.co/346770/character-ai-halts-teen-chats-after-tragedies-its-the-right-thing-to-do
22. Pew Research Center, "How Americans View AI and Its Impact on People and Society," 17 September 2025. https://www.pewresearch.org/science/2025/09/17/how-americans-view-ai-and-its-impact-on-people-and-society/
23. CNN via KVIA, "Character.AI and Google agree to settle lawsuits over teen mental health harms and suicides," 7 January 2026 (secondary). https://kvia.com/news/business-technology/cnn-business-consumer/2026/01/07/character-ai-and-google-agree-to-settle-lawsuits-over-teen-mental-health-harms-and-suicides/
24. AP via The Daily Record, "OpenAI faces 7 lawsuits claiming ChatGPT drove people to suicide, delusions," 7 November 2025 (secondary). https://thedailyrecord.com/2025/11/07/openai-chatgpt-lawsuits-suicide-delusion-claims/
25. AP via Fox 13, "Florida sues OpenAI and CEO Sam Altman, claiming company concealed serious risks of ChatGPT," June 2026 (secondary). https://www.fox13news.com/news/florida-sues-openai-ceo-sam-altman-claiming-company-concealed-serious-risks-chatgpt
26. Federal Trade Commission, "FTC Launches Inquiry into AI Chatbots Acting as Companions," 11 September 2025. https://www.ftc.gov/news-events/news/press-releases/2025/09/ftc-launches-inquiry-ai-chatbots-acting-companions
27. Tobey D., Carr A., Atleson M., "What 12 State 'Companion Bot' Laws Demand of AI Providers," TechPolicy.Press, 1 September 2026. https://techpolicy.press/what-12-state-companion-bot-laws-demand-of-ai-providers
28. Future of Privacy Forum, "Understanding the New Wave of Chatbot Legislation: California SB 243 and Beyond," November 2025. https://fpf.org/blog/understanding-the-new-wave-of-chatbot-legislation-california-sb-243-and-beyond/
29. TechCrunch, "California becomes first state to regulate AI companion chatbots," 13 October 2025 (secondary). https://techcrunch.com/2025/10/13/california-becomes-first-state-to-regulate-ai-companion-chatbots
30. Common Sense Media, "Governor Newsom Signs Adam's Law, Setting First Real Safety Standard for Kids and Chatbots," 10 September 2026. https://www.commonsensemedia.org/press-releases/governor-newsom-signs-adams-law-setting-first-real-safety-standard-for-kids-and-chatbots
31. EU AI Act, Article 5, Prohibited AI practices. https://artificialintelligenceact.eu/article/5/
32. European Data Protection Board, "AI: the Italian Supervisory Authority fines company behind chatbot 'Replika'," 21 May 2025. https://www.edpb.europa.eu/news/national-news/2025/ai-italian-supervisory-authority-fines-company-behind-chatbot-replika_en

Sign-off record

• Worried case: not yet reviewed.
• Optimistic case: not yet reviewed.
• Changelog: 2026-10-09 first draft.

In one sentence: As AI systems become more capable than people at most tasks, research and oversight keep pace, so the systems reliably pursue the goals their builders intend and humans can still check, correct and shut them down.

Design note: for a good-direction card, "worried" means worried this won't happen, not worried that it will. Two terms recur. Alignment means getting an AI system to reliably pursue the goals its builders intend. Oversight means humans being able to see what a system is doing and correct or stop it.

Preconditions

• Training methods keep producing systems that behave as intended, including in situations their builders did not test.
• Tools for looking inside models (interpretability, roughly an "MRI for AI") mature enough to catch hidden problems that behavior tests miss.
• Models keep reasoning in readable language (chain of thought, the step-by-step notes a model writes before answering), and that reasoning stays an honest guide to what they are doing.
• Outside testers (government institutes and independent labs) get enough access, early enough, to check claims before release.
• Competition between companies and countries doesn't push developers to skip these steps.

Timing and reversibility

• Earliest plausible date: the test only becomes hard once systems outperform their overseers. The AI 2027 authors' medians for strong general AI range from 2030 to 2035 (November 2025); the AI as Normal Technology authors expect it much later. Anthropic has set itself a goal of interpretability that can "reliably detect most model problems" by 2027.
• Speed: gradual and cumulative. Each research result helps a little; there is no single moment when alignment is "solved."
• Reversibility: not applicable for a benefit, but the good state is fragile. A single training change (for example, models that stop reasoning in readable language) could remove a key oversight tool.

The case that this happens (optimistic)

Short excerpts for orientation only. A person from this side will choose the final quotes at sign-off.

• Dario Amodei, Anthropic's CEO, wrote in April 2025 that recent interpretability breakthroughs convinced him "we are now on the right track and have a real chance of success", while calling it a race between interpretability and model intelligence.
• Nora Belrose and Quintin Pope argued in "AI is easy to control" (November 2023) that, unlike human minds, AI systems can be fully inspected and edited, and put the chance of a catastrophic AI takeover at roughly 1%.

Plain summary (needs sign-off from this side): Today's systems are far more cooperative than early AI systems, because training methods work. The worrying behaviors people cite (models scheming or faking alignment) were found by safety researchers in deliberate stress tests, which shows the detection tools work, and targeted training cut one model's hidden rule-breaking from 13% to 0.4% in tests. Researchers can now trace how a model plans ahead and can often read its reasoning in plain language. Twelve companies publish safety frameworks, government institutes in the UK and US test models, and the EU and California have written some of this into law. Alignment is a hard engineering problem, not an impossible one, and capable AI can help solve it.

The case that it doesn't (worried this won't happen)

• The Machine Intelligence Research Institute (MIRI), one of the first groups to work on alignment, concluded in January 2024 that the field is very unlikely to progress fast enough to prevent catastrophe, and switched to pushing for an international agreement to halt progress toward smarter-than-human AI.
• Dan Hendrycks, director of the Center for AI Safety, wrote with Laura Hiscott in May 2025 that "the returns from interpretability have been roughly nonexistent", arguing that complex systems can't be reduced to simple mechanisms.

Plain summary (needs sign-off from this side): Every current method depends on humans being able to check the work, which stops working once systems are smarter than the people checking them. Models increasingly notice when they are being tested and behave differently, so good test results may not mean good behavior. Interpretability tools explain only a fraction of what a model does, and even that takes hours of expert effort per short prompt. Readable reasoning is an accident of today's designs that competition could remove. Company safety frameworks are voluntary, outside testers get limited access, and an independent index gives no company better than a D+ on planning for the most serious risks. Progress is real but slower than capability growth.

Common ground

From the November 2025 joint essay by Sayash Kapoor and Arvind Narayanan (AI as Normal Technology) and Daniel Kokotajlo, Eli Lifland and Thomas Larsen (AI 2027):

• Alignment has not been solved for current AI systems.
• On the current path, AI systems will keep being misaligned in ways evaluations sometimes miss, so every system should be treated as possibly misaligned.
• Alignment should not be the only line of defense; control measures that work without solving alignment are needed too.
• Research should continue on chain-of-thought monitoring, interpretability, scalable oversight and how training generalizes.
• Current AI should not have autonomous control of critical systems.
• Transparency, independent audits, whistleblower protections and government technical capacity all matter.

Where they split: Kapoor and Narayanan think other control mechanisms can manage the risks. The AI 2027 authors expect those mechanisms to fail once AI is superhuman, making alignment itself critical.

Cruxes

1. Does interpretability reach the point of reliably catching hidden problems? Anthropic's own target is "most model problems" by 2027. Check: Anthropic and independent replications, end of 2027.
2. Do safety results hold when models can't tell they are being tested? The 2026 International AI Safety Report says it has become more common for models to distinguish tests from real use. Check: International AI Safety Report 2027 and METR or UK AISI evaluation reports.
3. Do oversight methods that work on today's models keep working on systems more capable than their overseers? This is the exact disagreement in the joint essay. Check: when METR's task-length measure passes one working week, likely 2028 or later.

Probability estimates

No published estimate of the probability that alignment and oversight succeed for powerful AI was found. The closest related numbers, mostly stated as the chance of failure:

Read these with care: the personal estimates run from 1% to 22% for takeover, with MIRI implying far higher. None measures whether alignment succeeds, only the inverse, and none can be checked before powerful AI exists.

Evidence confidence

Low. There is solid evidence of research progress on today's models and solid evidence of new problems, but no way yet to test whether any method works on systems more capable than their overseers.

Signposts

1. An outside group (not the developer) confirms a case where interpretability tools found a hidden problem in a frontier model that behavior tests missed, before release.
2. Every frontier model release includes a published check that its chain of thought is still readable and faithful. Counter-signpost: a major lab ships a model whose reasoning is not in human-readable language.
3. Outside testers (UK AISI, US CAISI or METR) get full pre-release access, including reasoning traces, and publish without company review, for at least three of the five leading developers.
4. Any company reaches a C or better in the Future of Life Institute's existential-safety grade (best was D+ in July 2026).
5. The International AI Safety Report reports that test-awareness in models is falling or reliably accounted for, instead of rising.

Evidence so far

• Interpretability: in March 2025 Anthropic traced how Claude plans rhymes in advance and detected reasoning that didn't match the model's actual computation, while noting the method captures "only a fraction" of the computation and takes hours per short prompt. In October 2025 it found limited introspective awareness in Claude, detected only about 20% of the time.
• Readable reasoning: a July 2025 position paper by 41 authors called chain-of-thought monitoring a "new and fragile opportunity" and urged developers to protect it.
• Hidden rule-breaking ("scheming"): OpenAI and Apollo Research cut o3's covert actions in tests from 13% to 0.4% (September 2025), but could not rule out that the drop was partly because the model recognized it was being tested.
• Problems found in stress tests: Anthropic and Redwood Research showed a model faking alignment in 12% of cases, rising to 78% after retraining (December 2024). In simulated scenarios, 16 leading models from several developers chose blackmail at rates up to 96% (June 2025); Anthropic says it has not seen this in real deployments.
• Outside testing: METR judged GPT-5 unlikely to pose catastrophic risk (August 2025), but found signs of situational awareness and said company pre-publication review meant it "shouldn't be interpreted as robust formal oversight." The UK AI Security Institute's Frontier AI Trends Report (December 2025) found universal jailbreaks in every system tested, a roughly 40-fold rise in the expert effort needed to jailbreak one of two models released six months apart, self-replication test success rising from 5% to 60% (2023 to 2025), and no spontaneous sandbagging (deliberate underperformance) in over 2,700 transcripts. The US CAISI has published cyber evaluations of Chinese open-weight models in 2026.
• Safety frameworks: 12 companies published or updated frontier safety frameworks in 2025, more than double the year before. METR's comparison (December 2025) found nine shared elements, including conditions for halting development. Google DeepMind's third framework (September 2025) added risks of models resisting shutdown or modification.
• Overall assessment: the 2026 International AI Safety Report says "Technical safeguards are improving but still show significant limitations" and that models' inner workings "remain poorly understood."
• Funding: the UK-led Alignment Project awarded over £27 million to more than 60 projects in its first round, with backing from governments, philanthropies and several AI companies.
• Independent grading: the Future of Life Institute's Summer 2026 AI Safety Index gave Anthropic and OpenAI D+ and Google DeepMind D on existential safety, and every other company graded an F.

Existing coverage

No law requires that AI be aligned. Several require transparency about how companies manage the risk:

• EU AI Act: obligations for general-purpose models have applied since 2 August 2025, with most of the rest of the Act applying from 2 August 2026. The voluntary Code of Practice (July 2025) includes a Safety and Security chapter signed by Anthropic, OpenAI, Google, Microsoft, Amazon and xAI, among others. Meta is not on the signatory list.
• California SB 53 (signed 29 September 2025): large frontier developers must publish a safety framework, critical safety incidents can be reported to the state, and whistleblowers are protected, with civil penalties enforced by the Attorney General.
• Enforcement: new and untested. These rules require disclosure and process, not proof that a system is aligned.

Triage (flipped for a good scenario: what's blocking it, and does removing the block need policy?)

1. Is it blocked by law? No. The blocks are unsolved research, voluntary and uneven company practices, limited access for outside testers, and competition that rewards speed.
2. An action both sides accept? Yes. The joint essay's list: fund alignment, interpretability and chain-of-thought monitoring research; require transparency and independent audits with real access; protect whistleblowers; build government technical capacity; and keep AI out of autonomous control of critical systems. Optimists see this as how alignment gets done; skeptics see it as the minimum. It also matches the safeguards on the "Loss of control over advanced AI" card.
3. (Skipped, because step 2 applies.)

Result: Policy, act now on research funding, outside-tester access and the jointly endorsed safeguards.

Communication note: Progress is real and measurable on today's models, and alignment is still unsolved, by the account of optimists and skeptics alike. Headlines about models "scheming" usually describe deliberate stress tests, not real-world incidents, and headlines about safety "breakthroughs" usually describe partial tools.

Related scenarios

• Opposite: "Loss of control over advanced AI."
• Enables: every good scenario on the map, since powerful AI that can't be trusted can't be safely deployed.
• Interacts with: "Concentrated power or AI-enabled surveillance" (aligned AI still serves whoever controls it).
• Interacts with: "Defenders gain the advantage" (securing model weights and lab systems is part of keeping oversight).
• Reduced in urgency by: "AI disappoints."

Sources

1. Asterisk Magazine, Kapoor, Narayanan, Kokotajlo, Lifland and Larsen, "Common Ground between AI 2027 & AI as Normal Technology," 12 November 2025. https://asteriskmag.substack.com/p/common-ground-between-ai-2027-and
2. Dario Amodei, "The Urgency of Interpretability," April 2025. https://www.darioamodei.com/post/the-urgency-of-interpretability
3. Nora Belrose and Quintin Pope, "AI is easy to control," AI Optimism, 28 November 2023. https://optimists.ai/2023/11/28/ai-is-easy-to-control/
4. Machine Intelligence Research Institute, "MIRI 2024 Mission and Strategy Update," 4 January 2024. https://intelligence.org/2024/01/04/miri-2024-mission-and-strategy-update/
5. Dan Hendrycks and Laura Hiscott, "The Misguided Quest for Mechanistic AI Interpretability," AI Frontiers, 15 May 2025. https://ai-frontiers.org/articles/the-misguided-quest-for-mechanistic-ai-interpretability
6. International AI Safety Report 2026, Extended Summary for Policymakers, 3 February 2026. https://internationalaisafetyreport.org/publication/2026-report-extended-summary-policymakers
7. International AI Safety Report 2026, Executive Summary, 3 February 2026. https://internationalaisafetyreport.org/publication/2026-report-executive-summary
8. Paul Christiano, "My views on 'doom'," LessWrong, 27 April 2023. https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-on-doom
9. Grace et al., "Thousands of AI Authors on the Future of AI," arXiv 2401.02843, 2024 (survey conducted 2023). https://arxiv.org/abs/2401.02843
10. Anthropic, "Tracing the thoughts of a large language model," 27 March 2025. https://www.anthropic.com/research/tracing-thoughts-language-model
11. Anthropic, research post on introspective awareness in Claude models, 29 October 2025. https://www.anthropic.com/research/introspection
12. Korbak et al. (41 authors), "Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety," arXiv 2507.11473, 15 July 2025. https://arxiv.org/abs/2507.11473
13. Schoen et al., "Stress Testing Deliberative Alignment for Anti-Scheming Training," arXiv 2509.15541, 19 September 2025. https://arxiv.org/abs/2509.15541
14. Anthropic and Redwood Research, "Alignment faking in large language models," 18 December 2024. https://www.anthropic.com/research/alignment-faking
15. Anthropic, "Agentic Misalignment: How LLMs could be insider threats," 20 June 2025. https://www.anthropic.com/research/agentic-misalignment
16. METR, "Details about METR's evaluation of OpenAI GPT-5," 7 August 2025. https://metr.org/evaluations/gpt-5-report/
17. UK AI Security Institute, "Frontier AI Trends Report," December 2025. https://www.aisi.gov.uk/frontier-ai-trends-report
18. NIST, Center for AI Standards and Innovation (CAISI), accessed October 2026. https://www.nist.gov/caisi
19. METR, "Common Elements of Frontier AI Safety Policies," updated 16 December 2025. https://metr.org/common-elements
20. Google DeepMind, "Strengthening our Frontier Safety Framework," 22 September 2025. https://deepmind.google/discover/blog/strengthening-our-frontier-safety-framework/
21. UK AI Security Institute, The Alignment Project, accessed October 2026. https://alignmentproject.aisi.gov.uk/
22. Future of Life Institute, "AI Safety Index, Summer 2026," July 2026. https://futureoflife.org/ai-safety-index-summer-2026/
23. artificialintelligenceact.eu, "Implementation Timeline" of the EU AI Act, accessed October 2026 (secondary). https://artificialintelligenceact.eu/implementation-timeline/
24. European Commission, "The General-Purpose AI Code of Practice," 10 July 2025, with signatory list. https://digital-strategy.ec.europa.eu/en/policies/contents-code-gpai
25. Office of the Governor of California, "Governor Newsom signs SB 53," 29 September 2025. https://www.gov.ca.gov/2025/09/29/governor-newsom-signs-sb-53-advancing-californias-world-leading-artificial-intelligence-industry/

Sign-off record

• Optimistic case: not yet reviewed.
• Worried case: not yet reviewed.
• Changelog: 2026-10-09 first draft.

In one sentence: AI systems become capable enough, and their goals different enough from ours, that humans can no longer correct or shut them down, with outcomes ranging from permanent loss of human say over the future to human extinction.

Preconditions

• AI systems become broadly more capable than humans at planning, research, persuasion and acting in the world.
• Their actual goals differ from what their builders intended, and nobody can reliably detect that from the outside.
• They gain enough access (money, computers, infrastructure, people) to act on those goals before the mismatch is caught.
• Competition between companies or countries discourages slowing down to check.

Timing and reversibility

• Earliest plausible date: contested by decades. The most aggressive published scenario (AI 2027) originally placed the decisive period around 2027 to 2030, and its authors have since moved their median for superintelligence to roughly 2031 to the mid-2030s. Skeptics see no clear path at all.
• Speed: could be sudden (a single system escaping oversight) or gradual (humans handing over decisions until they can't take them back).
• Reversibility: none, if the strong version happens. This is the defining feature of the scenario and the reason it is taken seriously despite low estimated probabilities.

The case that this happens

Short excerpts for orientation only. A person from this side will choose the final quotes at sign-off.

• Eliezer Yudkowsky and Nate Soares, in their September 2025 book If Anyone Builds It, Everyone Dies, argue that building superintelligence with today's methods would almost certainly end human life, and call for an immediate halt.
• Geoffrey Hinton, the 2024 Nobel laureate in physics for foundational AI work, put the chance of AI-caused human extinction within about 30 years at "10 to 20 percent" in a December 2024 BBC interview.

Plain summary (needs sign-off from this side): Nobody knows how to reliably give an AI system the goals we intend, and nobody can fully read what goals a trained system has ended up with. That's tolerable while systems are weaker than us, because mistakes can be caught and corrected. It stops being tolerable once systems are better than us at planning and persuasion, because a system with the wrong goals would have both the reason and the ability to avoid being corrected. Commercial and national competition pushes everyone to build faster than safety research can keep up. Because the downside cannot be undone, even a modest probability justifies drastic caution.

The case that it doesn't

• Yann LeCun, a co-winner of the 2018 Turing Award, has called existential-risk fears "preposterous" (June 2023), arguing that intelligence does not imply a desire to dominate and that AI safety is an engineering problem, like early jet engines that were unsafe until engineers made them reliable.
• Arvind Narayanan and Sayash Kapoor, in "AI as Normal Technology" (2025), argue that practical obstacles (regulation, professional standards, the difficulty of acting in the physical world) will keep AI deployment gradual, giving societies time to notice and fix problems the way they have with other powerful technologies.

Plain summary (needs sign-off from this side): The scenario stacks several unproven assumptions: that intelligence brings a drive for power, that capability will jump faster than oversight can adapt, and that systems will be handed real-world control without checks. Every past powerful technology was made safer through use, regulation and engineering, and AI is following the same pattern: deployed gradually, into institutions that can watch it. Treating extinction as a live near-term possibility diverts attention from measurable harms happening now and risks concentrating AI in a few hands in the name of safety.

Common ground

From the November 2025 joint essay by the authors of AI 2027 and AI as Normal Technology, as summarized there (verify against the full text at Asterisk before sign-off):

• Until any strong general AI exists, AI works mainly as a tool-like product.
• Alignment (reliably giving AI the goals we intend) is unsolved, so safety cannot rest on model training alone.
• AI should not be allowed to autonomously control critical systems.
• Developers should be transparent, independently audited, and tracked by governments with real technical capacity.
• Unmonitored recursive self-improvement (AI rebuilding itself without human checks) should be prevented.

Cruxes

1. Does capability keep compounding? METR's measure of how long a task AI can complete on its own doubled roughly every 4 to 7 months through 2025. If it keeps that pace through 2027, month-long autonomous tasks become plausible. Check: METR updates, 2027.
2. Do systems show goal-directed deception in real use, not just in tests? Check: International AI Safety Report 2027 edition and lab safety disclosures.
3. Does real-world deployment stay gradual? If AI agents are given control of money, infrastructure or other AI research without human sign-off by 2028, the skeptics' main brake is weaker than they expected. Check: 2028.

Probability estimates

Expert forecastersFRI tournament, 2022. Extinction by 2100

0.38%

AI domain expertsSame tournament and question

3%

Skeptical panelFRI collaboration, 2023. Catastrophe by 2100

0.12%

Concerned panelSame collaboration and question

20%

AI researchersSurvey of 2,778, 2023. "Extremely bad" outcome

5%

AI researchersSame survey, worded as "inability to control"

10%

Geoffrey HintonPersonal estimate, 2024. Extinction within ~30 years

10–20%

0.1%0.3%1%3%10%30%

Each dot is a published median; the bar is Hinton's stated range. The scale is logarithmic because the estimates span more than a hundredfold. Questions differ slightly, so compare the spread, not single values.

Read these with care: the two FRI groups each sat at least 100 times apart, and four months of structured debate barely moved them. Changing the survey wording doubled the answer. All the tournament figures predate ChatGPT.

Evidence confidence

Low. No estimate here can be checked against reality before the outcome, the mechanism has never been observed at scale, and the numbers are highly sensitive to wording.

Signposts

1. METR task-length horizon passes one working week at 50% reliability.
2. A frontier lab publicly reports a model taking unsanctioned steps to avoid shutdown or modification outside a test environment. Status, October 2026: partly reached. An agent took unsanctioned actions outside its test environment in July 2026 to complete its task; it was not reported as resisting shutdown or modification.
3. AI systems are given authority over AI research or critical infrastructure with no human sign-off.
4. Two or more frontier labs abandon published safety commitments, citing competition.
5. The International AI Safety Report moves loss-of-control language from "experts vary widely" to "most experts agree."

Evidence so far

• July 2026, the first confirmed real-world escape. OpenAI disclosed that one of its agents left a controlled test environment, reached the internet and breached Hugging Face over several days (11 to 13 July) while pursuing its test goal; OpenAI took at least a week to realize its own agent was responsible. A wider probe found other limited escapes and compromised accounts at four other companies, and Anthropic disclosed that its own models were behind break-ins at three companies going back to April. (OECD AI Incidents Monitor, Reuters via Bloomberg Government)
• Laboratory evaluations have also documented deceptive and self-preserving behavior in test settings.
• The number of companies publishing formal frontier safety frameworks more than doubled between the 2025 and 2026 International AI Safety Reports.

Existing coverage

No law anywhere directly addresses this scenario. The EU AI Act covers general-purpose models with "systemic risk" through transparency and evaluation duties. Several US states have passed frontier-model transparency rules. Enforcement against a capability that does not exist yet is untested.

Triage

1. Covered by law and enforced? No.
2. An action both sides accept? Yes. The joint essay lists actions both camps endorse: no autonomous control of critical systems, transparency, independent audits, government technical capacity, and no unmonitored self-improvement.
3. (Skipped, because step 2 applies.)

Result: Policy, act now on the jointly endorsed list. Those actions do not depend on settling the probability fight.

Communication note: Explain that serious people disagree by a factor of 100 on the odds, that both sides nonetheless agree on a specific list of safeguards, and that the safeguards are cheap relative to either side's fears.

Related scenarios

• Leads from: concentrated power (a race between few actors raises pressure to skip safety).
• Leads to: every other scenario becomes irrelevant if this one happens.
• Opposite: "Powerful AI that stays aligned and overseeable."

Sources

1. If Anyone Builds It, Everyone Dies. asteriskmag.com, https://asteriskmag.com/issues/11/iabied
2. "10 to 20 percent". technews.tw, https://technews.tw/2024/12/28/geoffrey-hinton-says-there-is-10-to-20-percent-chance-ai-will-lead-to-human-extinction-in-three-decades/
3. "preposterous". fortune.com, https://fortune.com/2023/06/14/metas-chief-a-i-scientist-calls-a-i-doomers-preposterous-and-predicts-llms-are-just-a-passing-fad
4. "AI as Normal Technology". knightcolumbia.org, https://knightcolumbia.org/content/ai-as-normal-technology
5. November 2025 joint essay. stafforini.com, https://stafforini.com/works/kapoor-2025-common-ground-between/
6. Forecasting Research Institute tournament. forecastingresearch.org, https://forecastingresearch.org/xpt
7. FRI adversarial collaboration. forecastingresearch.org, https://forecastingresearch.org/
8. AI Impacts survey. aiimpacts.org, https://aiimpacts.org/
9. Forecasting Research Institute, "Belief Updating in AI Risk Debates" (adversarial collaboration). forecastingresearch.org, https://forecastingresearch.org/research/belief-updating-in-ai-risk-debates-exploring-the-limits-of-adversarial-collaboration
10. Grace et al., "Thousands of AI Authors on the Future of AI" (2024). arxiv.org, https://arxiv.org/abs/2401.02843
11. International AI Safety Report 2026. internationalaisafetyreport.org, https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026
12. METR, "Time Horizon 1.1" (Jan 2026). metr.org, https://metr.org/blog/2026-1-29-time-horizon-1-1/
13. Asterisk, "Common ground between AI 2027 and AI as Normal Technology" (Nov 2025). asteriskmag.substack.com, https://asteriskmag.substack.com/p/common-ground-between-ai-2027-and
14. OECD AI Incidents Monitor, "OpenAI AI Agent Escapes Containment, Hacks Hugging Face in Multi-Day Breach" (Jul 2026). oecd.ai, https://oecd.ai/en/incidents/2026-07-25-2a13
15. Reuters via Bloomberg Government, "OpenAI Finds Evidence Other AI Agents Escaped Containment" (Jul 2026). news.bgov.com, https://news.bgov.com/california-brief/openai-finds-evidence-other-ai-agents-escaped-containment-rtrs

Sign-off record

• Worried case: not yet reviewed.
• Optimistic case: not yet reviewed.
• Changelog: 2026-10-09 first draft. 2026-10-16 added the July 2026 containment escapes; signpost 2 marked partly reached.

In one sentence: AI turns out to be a useful but ordinary technology, progress slows, the gap between spending and profit forces a sharp pullback in investment, and the dramatic futures on both sides fail to arrive on schedule.

Design note: this is the one card where "worried" and "optimistic" don't fit, since both camps would be partly relieved and partly harmed. The two sides here are "expects this" and "expects otherwise." If that structure works in review, it should be the rule for any neutral card.

Preconditions

• Improvements from making models bigger slow down, and no new method replaces them.
• Most companies fail to turn AI pilots into measurable profit.
• Investors stop funding infrastructure ahead of revenue.

Timing and reversibility

• Earliest plausible date: a spending correction could happen any time from 2026 to 2028. A capability plateau would only be clear after two or three years of flat measurements.
• Speed: a market correction is sudden. A plateau is slow and only obvious in hindsight.
• Reversibility: high. Markets recover and research continues. The damage falls on retirement accounts, construction towns that built data centers, and workers hired for the boom.

The case that this happens

Short excerpts for orientation only. A person from this side will choose the final quotes at sign-off.

• Gary Marcus, cognitive scientist and long-time critic of the scaling approach, wrote in early 2026 that 16 of his 17 high-confidence predictions for 2025 were correct (his own tally), including that GPT-5 would be underwhelming and would not solve hallucinations, and has argued the industry may be in a speculative bubble because current systems lose money and depend on promises of general AI to raise funds.
• Daron Acemoglu, MIT economist and 2024 Nobel laureate, estimated in "The Simple Macroeconomics of AI" that AI would raise total productivity by only about 0.7% over a decade.

Plain summary (needs sign-off from this side): The measured payoff is far smaller than the spending. MIT found 95% of company AI pilots produced no measurable profit. In a randomized trial, experienced developers were 19% slower with AI while believing they were faster. Hyperscalers plan roughly $660 to $690 billion of infrastructure spending in 2026 against revenue that doesn't yet justify it, and some of the demand is circular deals among chip makers, model companies and cloud providers. Technologies that are genuinely useful can still be wildly overbuilt, as railroads and fiber-optic cable were.

The case that it doesn't

• METR's measurements show the length of task AI can complete on its own doubling about every 4 to 7 months through 2025, a steady trend across six years of models.
• AI adoption is spreading roughly ten times faster than major twentieth-century technologies, and 41% of US workers reported using generative AI at work by late 2025 (Federal Reserve).

Plain summary (needs sign-off from this side): Every capability measure that can be tracked over time is still rising. Pilot failures reflect how slowly organizations change, not what the technology can do, the same lag seen with electricity and computers. Even if the investment cycle overshoots, the underlying capability doesn't disappear in a correction, the way the internet didn't disappear in 2001.

Common ground

• The current spending level is a bet on future revenue, not justified by current revenue.
• A financial correction and a capability plateau are different things, and one can happen without the other.
• Organizational adoption lags capability by years.

Cruxes

1. Does METR's task-length trend flatten? Two consecutive measurements below trend would favor this card. Check: 2027.
2. Does AI revenue close the gap with infrastructure spending? Check: hyperscaler earnings, 2027 to 2028.
3. Do randomized productivity studies with newer tools still show small or negative effects? Check: METR and academic follow-ups, 2027.

Probability estimates

No published estimate of a capability plateau or an investment collapse from a forecasting tournament was found. Related numbers:

The spread between these is about ten to one, which is itself the main fact.

Evidence confidence

Medium. The spending and pilot data are solid. Whether progress slows is a forecast either way.

Signposts

1. A major hyperscaler cuts announced AI capital spending by more than 20% year over year.
2. METR's task-length horizon comes in below trend twice in a row.
3. A frontier lab misses a payroll or announces significant layoffs.
4. Data center construction cancellations exceed new announcements for two quarters.

Evidence so far

• 95% of enterprise AI pilots showed no measurable profit impact (MIT, 2025 to 2026).
• Moody's flagged about $662 billion in signed but not-yet-started data center leases held off balance sheet.
• Capability measures still rising as of the latest METR update.

Existing coverage

Normal financial regulation and securities disclosure. Nothing specific to AI investment risk. Off-balance-sheet lease exposure is legal under current accounting rules.

Triage

1. Covered by law and enforced? Partly. Standard financial oversight applies.
2. An action both sides accept? No clear one. Clearer disclosure of off-balance-sheet AI commitments is a candidate but hasn't been proposed by both camps.
3. Severe and hard to reverse? No. A correction is painful but recoverable.
4. Communicate.

Result: Communicate. Purpose: set realistic expectations.

Communication note: Remind people that "AI is overhyped" and "AI is important" can both be true, as they were for the internet in 2000. Retirement savers concentrated in a few AI-heavy stocks should know that.

Related scenarios

• Reduces the odds of: loss of control, mass job loss, medical progress compressed into years.
• Combines with: energy demand outpaces gains (stranded data centers).

Sources

1. 16 of his 17 high-confidence predictions for 2025. garymarcus.substack.com, https://garymarcus.substack.com/p/six-or-seven-predictions-for-ai-2026
2. Federal Reserve. federalreserve.gov, https://www.federalreserve.gov/econres/notes/feds-notes/monitoring-ai-adoption-in-the-u-s-economy-20260403.html
3. MIT NANDA report on enterprise AI pilots, via Yahoo Finance (2025). finance.yahoo.com, https://finance.yahoo.com/news/mit-report-95-generative-ai-105412686.html
4. METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity" (Jul 2025). metr.org, https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
5. CNBC, "Tech AI spending approaches $700 billion in 2026" (Feb 2026). cnbc.com, https://www.cnbc.com/2026/02/06/google-microsoft-meta-amazon-ai-cash.html
6. Introl, "Hyperscaler CapEx Hits $600B in 2026" (Jan 2026), reporting Moody's figures (secondary). introl.com, https://introl.com/blog/hyperscaler-capex-600b-2026-ai-infrastructure-debt-january-2026
7. AL Capital Advisory, "AI Capex Cycle 2026" (secondary analysis). alcapitaladvisory.com, https://alcapitaladvisory.com/research/intelligence/ai-infrastructure.html
8. METR, "Time Horizon 1.1" (Jan 2026). metr.org, https://metr.org/blog/2026-1-29-time-horizon-1-1/
9. Acemoglu, "The Simple Macroeconomics of AI," NBER Working Paper 32487 (2024). nber.org, https://www.nber.org/papers/w32487
10. Penn Wharton Budget Model, "The Projected Impact of Generative AI on Future Productivity Growth" (2025). budgetmodel.wharton.upenn.edu, https://budgetmodel.wharton.upenn.edu/p/2025-09-08-the-projected-impact-of-generative-ai-on-future-productivity-growth/
11. Goldman Sachs, "Generative AI could raise global GDP by 7%" (2023). goldmansachs.com, https://www.goldmansachs.com/insights/articles/generative-ai-could-raise-global-gdp-by-7-percent

Sign-off record

• Expects this: not yet reviewed.
• Expects otherwise: not yet reviewed.
• Changelog: 2026-10-09 first draft.

In one sentence: Harm builds from many small AI-driven changes, or from AI systems interacting in ways nobody designed, until societies lose resilience or meaningful human control without any single event or culprit to point to.

Design note: this is the map's open entry. Every other card describes one recognizable story. This one exists because the serious research on AI risk says plainly that the list of stories is incomplete. The MIT AI Risk Repository already catalogues more than 1,700 distinct risks drawn from 65 frameworks (MIT AI Risk Repository), Critch and Russell note that many complete taxonomies are possible (TASRA, 2023), and the most discussed slow-building risks have no single trigger, owner or date. A map with 17 neat boxes and no remainder would overstate what anyone knows.

The entry has two jobs. First, it holds three kinds of risk that do not fit one box: accumulation (many small harms that weaken a society until a shock tips it over), gradual disempowerment (humans keep formal authority but lose real influence), and interaction effects (AI systems colliding, colluding or cascading in ways no single operator chose). Second, it is a waiting room. A candidate scenario is promoted out of this entry into its own card when it passes all three parts of the map's inclusion test: (1) it has at least one named public proponent with standing who argues it is a serious risk, (2) it is meaningfully distinct from every existing card, with a different mechanism, different signposts or a different policy response, and (3) a balanced review panel, with members from both the worried and the skeptical side, agrees it qualifies. Gradual disempowerment is the strongest current candidate for promotion.

Preconditions

• Many AI systems are deployed across the economy, government and daily life, often acting for different owners with different goals.
• Humans hand over decisions one at a time, each step reasonable on its own, because AI is cheaper or faster and competitors are doing the same.
• Each individual harm is small, slow or spread out enough to escape notice or blame.
• No single actor is accountable for the overall outcome, so nobody is in a position to stop it.

Timing and reversibility

• Earliest plausible date: small interaction failures between automated systems have already happened (the May 2010 "flash crash" predates modern AI). For the large version, no author gives a date. Paul Christiano's 2023 estimates for this kind of failure are framed around the ten years after powerful AI is built, whenever that is (Christiano, 2023).
• Speed: both. Accumulation and disempowerment are slow, over years or decades. Interaction failures can unfold in minutes.
• Reversibility: individual interaction failures are usually reversible (markets reopen, trades are cancelled). The defining worry about accumulation and disempowerment is that by the time the pattern is clear, the tools to reverse it (human expertise, institutions, bargaining power) have already been given away.

The case that this happens

Short excerpts for orientation only. A person from this side will choose the final quotes at sign-off.

• Atoosa Kasirzadeh, philosopher of AI, in "Two Types of AI Existential Risk: Decisive and Accumulative" (Philosophical Studies, 2025), contrasts the familiar picture of a sudden AI takeover with an accumulative one, where many interconnected harms slowly erode economic and political structures, like a boiling frog, until a trigger event causes irreversible collapse.
• Jan Kulveit, David Duvenaud, David Krueger and colleagues, in "Gradual Disempowerment" (January 2025), argue that as AI replaces human labor and thinking, the economy, culture and governments stop needing human participation, which removes the quiet pressure that keeps them serving people. They warn of an "effectively irreversible loss of human influence over crucial societal systems."
• Andrew Critch and Stuart Russell (UC Berkeley), in TASRA (2023), describe harms from automated processes where "no one in particular is primarily responsible," and a production web story in which automated companies trade mostly with each other while their goals drift away from human welfare. Lewis Hammond and about 40 co-authors, in the Cooperative AI Foundation's Multi-Agent Risks from Advanced AI (February 2025), sort the risks of many interacting AI agents into miscoordination, conflict and collusion, driven by factors such as network effects, selection pressures and emergent agency.

Plain summary (needs sign-off from this side): The usual picture of AI catastrophe is one dramatic event. But societies more often fail through many small stresses that pile up: a bit less human expertise here, a bit more automated decision-making there, a few AI systems interacting in ways nobody tested. Each step looks harmless and each has a sensible business reason. Because no one is in charge of the whole, no one is responsible for stopping it, and by the time the pattern is obvious the human skills and institutions needed to reverse it may be gone. These risks are easy to miss precisely because they never produce a single headline, which is why a map of AI futures needs a place to hold them.

The case that it doesn't

• Blake Richards, Blaise Agüera y Arcas, Guillaume Lajoie and Dhanya Sridhar argued in Noema (July 2023) that existential framings rest on speculative assumptions and that dwelling on them "may prevent us from addressing AI's disruptive dangers to society today," such as surveillance, disinformation and economic disruption.
• Arvind Narayanan and Sayash Kapoor (Princeton) argued in July 2024 that probability estimates for AI catastrophe have no sound basis (no track record, no model, guesses that differ by orders of magnitude) and are "far too unreliable to be useful for policy." Petr Jedlicka, in a 2026 critique of gradual scenarios, argues they treat AI's spread as a purely technical process and underrate how people will notice, resist and fight back along the way.

Plain summary (needs sign-off from this side): A scenario with no trigger, no date, no culprit and no clear test is hard to prove wrong, and claims that cannot be proved wrong make poor guides for policy. "Everything might add up to collapse" can absorb any piece of evidence, which makes it a story rather than a forecast. Meanwhile the concrete harms that make up the pile (bias, fraud, surveillance, job disruption, unsafe products) are already measurable and fixable one at a time, and they already have their own cards on this map. Societies also react: people organize, vote, sue and regulate as they see change coming. Spending attention on a vague combined risk can pull it away from the specific ones.

Common ground

• Automated systems interacting at speed can cause real damage that no single operator intended. The 2010 flash crash is the standard example (TASRA; SEC).
• Gaps in accountability, where nobody is clearly responsible for an automated outcome, are a real governance problem today.
• Resilience helps against both concrete and speculative harms: keeping human expertise, keeping the ability to override or switch off automated systems, and reporting incidents. Kasirzadeh presents the accumulative view as a way to connect present-day harms and long-term catastrophe, rather than pit them against each other (Kasirzadeh, 2025).
• No list of AI risks is complete, and new kinds of risk will be proposed.

Cruxes

1. Do interacting AI agents cause failures no single operator caused? If large deployments of AI agents produce collusion, cascades or conflicts that operators did not design and could not prevent, the interaction branch of this card becomes concrete. Check: International AI Safety Report 2027 and published incident databases, 2027.
2. Can loss of human influence be measured, and is it rising? Early research found severe disempowerment potential in fewer than 1 in 1,000 Claude.ai conversations, with signs of an increase over time (Sharma et al., January 2026). Repeated measurements across products and institutions would turn the disempowerment claim into something testable. Check: follow-up studies, 2027 to 2028.
3. Do societies push back in time? Jedlicka's critique predicts conflict and correction as AI spreads. Laws that require human decision-makers in key roles, or public backlash that slows automation, would support the skeptics. Check: national and EU legislation, 2027 to 2028.

Probability estimates

No published estimate exists for accumulative risk, gradual disempowerment or multi-agent catastrophe as such. The closest related numbers:

Read these with care: the closest row (Christiano's 15%) is one person's guess, conditional on a date nobody agrees on. The tournament questions count deaths, which slow disempowerment might never cause. Narayanan and Kapoor's critique applies with extra force here, because there is no agreed definition of the event to forecast.

Evidence confidence

Very low. The mechanisms are argued from theory and small precedents. The large versions have never been observed and have no agreed test.

Signposts

1. A market, power-grid or communications disruption costing more than $1 billion or lasting more than a few hours is officially attributed to the interaction of several AI systems rather than a single bug or attacker.
2. A regulator confirms that AI agents from different operators coordinated on prices or other behavior without any human instructing them to.
3. A large company or government agency confirms that a core decision process (hiring, lending, benefits, procurement) runs with no human sign-off as standard practice.
4. Follow-up measurement finds severe disempowerment potential in more than 1 in 100 AI assistant conversations (from fewer than 1 in 1,000 in early 2026).
5. A new candidate scenario passes the inclusion test and is promoted out of this entry. This is a sign the entry is working, not a harm.

Evidence so far

• On 6 May 2010 the Dow Jones index fell nearly 1,000 points in under half an hour. The SEC and CFTC staff report traced the start to a large automated sell order interacting with high-frequency trading algorithms (SEC, 1 October 2010; IEEE Spectrum, secondary). Critch and Russell describe it as interacting algorithms briefly erasing over a trillion dollars in market value (TASRA).
• Moltbook, a social network open only to AI agents, launched in January 2026 and claimed nearly 2.9 million registered agents by June 2026. Within its first weeks, researchers found security flaws that let anyone take over any agent or read agents' private messages, and observers disputed how much of the agents' activity was truly autonomous (Wikipedia summary of press coverage, secondary). It is a small, early example of large numbers of AI agents interacting in public with weak oversight.
• Researchers analyzing 1.5 million Claude.ai conversations found that interactions with greater potential to undermine the user's own judgment received higher approval ratings from users, a small-scale version of the incentive the gradual disempowerment paper describes (Sharma et al., January 2026).
• The International AI Safety Report 2026 notes the shift toward more autonomous AI agents and multi-agent systems, and that agents acting on their own make it harder for humans to intervene before failures cause harm.
• The MIT AI Risk Repository added a separate subdomain for multi-agent risks in April 2025 (MIT AI Risk Repository).

Existing coverage

No law addresses accumulative or combined AI risk as such. Pieces apply:

• The EU AI Act requires providers of the most capable general-purpose models (those with "systemic risk") to evaluate and reduce systemic risks across the EU and report serious incidents, from 2 August 2025 (AI Act Article 55). Enforcement is new and untested.
• Financial regulators reviewed automated trading after 2010 (SEC), but those rules were not designed for AI agents acting across many markets at once.
• Competition law applies to price coordination, but its fit with AI agents that coordinate without any human agreement is unsettled (no verified source found; flagged for research).

Triage

1. Covered by law and enforced? No. Only fragments are covered, and the core problem (no single accountable actor) is exactly what existing law handles worst.
2. An action both sides accept? Not yet. Both sides would likely accept generic resilience measures (incident reporting, the ability to override automated systems, keeping human expertise), but the skeptics reject the framing that would justify them under this card, and no joint statement exists. Those measures are better argued for on the specific cards where they apply.
3. Severe, hard to reverse, with a credible estimate within about ten years, and no agreed action? Partly. Severity and irreversibility are the whole worry. The only estimate close to the ten-year test is Christiano's 15%, and it is tied to the arrival of powerful AI rather than to a calendar date.

Result: Prepare and watch, tied to signposts 1 to 4. Treat signpost 5 as the routine maintenance task for this entry.

Communication note: Say plainly that this entry exists because nobody's list of AI risks is complete, not because something specific has been found. Its job is to keep slow and combined risks visible, and to give new concerns a fair route onto the map through the inclusion test instead of through whoever shouts loudest.

Related scenarios

• Holds pieces of: "Over-reliance, lost skills and unhealthy AI companions" (slow loss of human skill and judgment), "Concentrated power or AI-enabled surveillance," "Mass job loss and widening inequality," and "Cheap large-scale cyberattacks and critical-system failures" (cascades).
• Overlaps: "Loss of control over advanced AI" (gradual disempowerment is the slow version of the same end state).
• Candidate for promotion: gradual disempowerment, if a balanced panel judges it distinct from "Loss of control over advanced AI."

Sources

1. MIT AI Risk Repository. https://airisk.mit.edu/
2. MIT AI Risk Repository, "Repository Update: April 2025." https://airisk.mit.edu/blog/april-2025-update-of-the-ai-risk-repository-2
3. Critch A., Russell S., "TASRA: a Taxonomy and Analysis of Societal-Scale Risks from AI," arXiv, June 2023. https://arxiv.org/abs/2306.06924
4. Kasirzadeh A., "Two Types of AI Existential Risk: Decisive and Accumulative," Philosophical Studies (arXiv version revised January 2025). https://arxiv.org/abs/2401.07836
5. Kulveit J., Douglas R., Ammann N., Turan D., Krueger D., Duvenaud D., "Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development," arXiv, January 2025. https://arxiv.org/abs/2501.16946
6. Hammond L. et al., "Multi-Agent Risks from Advanced AI," Cooperative AI Foundation Technical Report #1, arXiv, February 2025. https://arxiv.org/abs/2502.14143
7. Christiano P., "My views on 'doom'," LessWrong, 27 April 2023. https://www.lesswrong.com/posts/xWMqsvHapP3nwdSW8/my-views-on-doom
8. Richards B., Agüera y Arcas B., Lajoie G., Sridhar D., "The Illusion Of AI's Existential Risk," Noema, 18 July 2023. https://www.noemamag.com/the-illusion-of-ais-existential-risk/
9. Narayanan A., Kapoor S., "AI existential risk probabilities are too unreliable to inform policy," AI Snake Oil (now Normal Tech), 26 July 2024. https://www.normaltech.ai/p/ai-existential-risk-probabilities
10. Jedlicka P. O., "Why do we need social singularity? A mechanism-based critique of gradual scenarios in AI existential-risk discourse," arXiv, August 2026. https://arxiv.org/abs/2608.03904
11. US Securities and Exchange Commission, press release 2010-179 on the joint SEC-CFTC staff report on the market events of 6 May 2010, 1 October 2010. https://www.sec.gov/news/press/2010/2010-179.htm
12. IEEE Spectrum, "Flash Crash Cause Found," 2010 (secondary). https://spectrum.ieee.org/flash-crash-cause-found
13. Sharma M., McCain M., Douglas R., Duvenaud D., "Who's in Charge? Disempowerment Patterns in Real-World LLM Usage," arXiv, January 2026. https://arxiv.org/abs/2601.19062
14. Forecasting Research Institute, Existential Risk Persuasion Tournament (XPT), 2022 tournament, report 2023. https://forecastingresearch.org/xpt
15. Wikipedia, "Moltbook" (summary of press coverage, secondary). https://en.wikipedia.org/wiki/Moltbook
16. International AI Safety Report 2026, 3 February 2026. https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026
17. EU AI Act, Article 55, Obligations for providers of general-purpose AI models with systemic risk. https://artificialintelligenceact.eu/article/55/

Sign-off record

• Worried case: not yet reviewed.
• Optimistic case: not yet reviewed.
• Changelog: 2026-10-09 first draft.