On Saturday, September 12, I published on this blog why I want to dedicate my life to AI safety. That same Saturday, Dario Amodei published "We Must Pace the Frontier" (the timing and the coincidence were crazy). Basically, Anthropic's CEO was literally asking to slow down the pace at which model capabilities improve. Within hours Sam Altman called it a "great idea", Demis Hassabis said that "the direction is correct" and Elon Musk joined the chorus. The four guys accelerating this technology the most on the planet suddenly agreed that everything had to slow down... kind of curious, right?
Two days later, the president of the United States wrote that there was a "SICK conspiracy" against AI and data centers, and that the only one happy about it was China. That same Monday, China's foreign ministry described the CEOs' request as "fearmongering, confrontation and vicious competition". And ten days later, Xi Jinping was having dinner among 'friends' at the White House, surrounded by Jensen Huang, Tim Cook, Lisa Su, Sam Altman and Elon Musk. Amodei, the one who had started it all, apparently wasn't invited, which seems super strange to me..
Look at the full sequence. Both governments say, each in its own vocabulary, that this technology scares them and that precautions are needed, and while industry leaders say the same thing, the outcome of the summit was a dialogue with a disputed name and an "incident channel" that nobody knows which incidents it will carry. This post is my attempt to understand why, mixing what happened in Washington with a 2016 paper that explains it all too well, maybe a bit simplistic but with a lot of common sense.
So what really happened in Washington?
Facts first, because the coverage was somewhat confusing amid so much hype. Xi made a state visit from September 23 to 25, with the central meeting and the dinner on Thursday the 24th. There was no joint statement: each government published its own summary, or 'readout'. But on AI, the balance was this:
- A bilateral dialogue. The White House named it the "U.S.-China Super Intelligence (SI) Dialogue". Beijing calls it the "China-U.S. AI Dialogue". The next round was set for November 18 to 19, 2026 in Shenzhen, China.
- A communication channel for AI-related incidents. Neither Washington nor Beijing published operating rules, thresholds, metrics... not even what counts as an incident, so that both sides could speak a common language.
- Nothing on chips. Export controls were left practically off the public agenda, and the Chinese foreign ministry's summary of the summit's eight deliverables doesn't even mention them. It's clearly a super sensitive topic, and central to China's progress in AI.
An analyst quoted by CNBC summed up the results as "all quite limited and tentative". CSIS was somewhat more generous: the AI dialogue is the only substantive area of cooperation on global governance between the two countries, and its value will depend on how the actual operation gets implemented. Both readings are compatible. And in late October, according to the South China Morning Post, the teams are still negotiating the details of the framework ahead of APEC in Shenzhen. In other words, it's still a declaration.
To understand why so little came out of it, you have to look at the three weeks surrounding the dinner:
Trump in Beijing
First summit of the year. Trump says they talked about "possibly working together for guardrails" on AI, and Nvidia's H200 chips came up. Nothing concrete.
Xi opens the World AI Conference in Shanghai
Xi's first keynote at the conference. He positions China as the leader of open source and of an "equitable" global governance, one day after 29 countries signed the creation of the World Artificial Intelligence Cooperation Organization, headquartered in Shanghai.
Anthropic publishes its threat report
In its final section it accuses Moonshot AI of silently forwarding Kimi users' requests to Claude and serving them the answers as if they were Kimi's (a big problem for China).
We Must Pace the Frontier
Amodei asks to slow the pace of capability improvement. Altman, Hassabis and Musk back him, with nuances (Boom! Suddenly the world is talking about AI Safety).
China's Ministry of State Security warns about AI
Minister Chen Yixin publishes a signed article on six categories of risk, with political security at the top.
Trump and Beijing reject slowing down
Trump talks about a "SICK conspiracy". The Chinese foreign ministry talks about "fearmongering". They agree on something, even if it's not what you'd expect...
Summit and state dinner at the White House
Bilateral AI dialogue and a channel for incidents. No published rules, no joint statement.
APEC in Shenzhen and second round of the dialogue
The next real test... if the incident channel reaches November without definitions or clear steps, it would be a big problem for real cooperation.
Two vocabularies for the same fear
The most revealing detail of the summit was linguistic. On Thursday morning, Trump posted: "Super Intelligence (SI) will be a big topic of discussion, but I want to leave it exactly where it is. That is China's position also." Washington had already been using "Super Intelligence" instead of "artificial intelligence" in official documents, and the State Department instructed its diplomats to use it (it became a meme in the community when it also associated researchers in the field with high IQs). After the meeting, Trump said that Xi "seemed to like" the new name.
Xi didn't use the term once. He said that both countries have the capacity and the responsibility to develop AI "for good" and to keep its development under human control. His readout said that on AI they can compete but cooperate even more.
It looks like an anecdotal or unnecessary detail, but the meaning of words carries weight, and in this context even more so. If they can't agree on defining what is what, which is the simplest thing to define amid all the complexity of the matter, it seems even harder to find agreement on more urgent issues.
And again, it may seem like a joke that I bring it up, but it isn't. The vocabulary tells you which risk each side considers central.
On the Chinese side, the clearest text didn't come out of the summit but from the article by Chen Yixin, Minister of State Security, published on September 13. There he writes that the abusive use of AI by adversaries could "directly threaten China's political security, institutional security and ideological security". The six categories of risk he lists start with political security and continue with more sophisticated cyberattacks, mass theft of sensitive information, technological monopolies, disruption of social governance and changes in modern warfare. And, as Crypto Briefing pointed out, at no point does he suggest that China should slow down its own development.
In the DW interview that sparked part of this post, analyst Rebecca Arcesati puts it bluntly: for a one-party state intensely concerned with its grip on power, the number one risk isn't the existential one, it's foreign forces using AI to manipulate public opinion and destabilize the regime. China's AI regulations (recommendation systems, generative AI) were born out of information control, and only now are they expanding toward frontier risks.
Why Beijing fears AI could be used as a weapon against its regime | DW News
DW News interview on the warning from China's Ministry of State Security, the WeChat case, Anthropic's accusation against Moonshot and why nobody expects an AI safety breakthrough between Washington and Beijing.
Two cases that made the nightmare concrete
September gave Beijing two examples that, seen through security eyes, explain better than any statement why its fear is political and technical at the same time.
WeWorm. The US firm Calif published a proof of concept of a worm that took over WeChat accounts with a single call the victim never answered: a memory corruption in the handling of VoIP calls, zero-click, with the requirement that the attacker already be in the contact list. Once an account was compromised, the worm could jump to that account's contacts. According to Calif, they found the bug in July with AI assistance, reported it to Tencent, and the server-side fix arrived in late August. Tencent says there is no evidence of exploitation in the wild, and it published no advisory or CVE. The coverage relies almost entirely on Calif's account, so I take it with a grain of salt, since there isn't much more than that.
Now think about it from Beijing. WeChat is messaging, payments and identity for more than a billion people, and it's also surveillance infrastructure for the state itself. A US company, with AI, finding a zero-click, wormable chain on that surface is exactly the scenario Chen Yixin describes.
On top of that, another curious event that happened not long ago was Kimi that was actually Claude. Anthropic's September threat report closes with an accusation against Moonshot AI, where, according to Anthropic, the company "silently forwarded customer requests to Claude, instead of processing them using Kimi". In other words, users who believed they were using the Chinese model were getting answers from Claude. Anthropic attributes more than 23 million exchanges to the company between May and July, and describes a technique for recovering reasoning traces by reopening sessions with saved thinking signatures. Obviously Moonshot didn't respond to the accusation, but as we say in my country, "silence gives consent". Still, the obvious caveat applies too: the evidence is Anthropic's telemetry, Anthropic is not a neutral actor in this race, and if we want to stay objective, this is a back-and-forth with no "neutral" companies confirming one side or the other.
The security reading
Both cases are, at their core, supply chain and integrity problems. In one, trust in a national platform breaks because of a bug that an AI helps find. In the other, the promise of technological sovereignty of Chinese open models breaks because, at least according to the accusation, part of the product was a proxy to a US model. Neither is a problem a summit can solve, but both are exactly the kind of event an incident channel should be for.
The other race: intelligence per dollar
There's a second layer the summit didn't even touch, and to me it's the most important one in the short term. I already took it apart in detail in the post about the cost gap between the US and China, so here I'll stick to the essentials.
In the video "How China plans to win the global AI race" there's a line that sums it all up: "You don't need god to write your emails". You don't need the smartest model in the world for 90% of what you do with AI. The United States is running the frontier race, the one for the most capable model. China is running, much more aggressively, the intelligence-to-cost race, with open-weight models. The video features an entrepreneur who paid about ten times less after switching to Chinese models, and OpenRouter data points the same way: according to CNBC's survey, the share of tokens US organizations send to Chinese models reached a weekly peak of 46% in mid-2026, compared with 4.5% in the first half of 2025.
Cheap energy, more talent, more scale (that's 1.3 billion people) and more open weights, which to me makes a lethal recipe. Direct monetization is hard, but I suspect that isn't the Chinese government's goal. The goal, at least in my opinion, is to get the technology into the infrastructure of half the world, cheaper and smart enough, with a fraction of the US investment. The US still has more compute, better chips and an enormous market valuation, and that gives it a real advantage at the frontier. But these are two different races running at the same time, and that matters for what comes next, because in both of them the incentive is the same: go faster.
The math of the precipice
In 2016, Stuart Armstrong, Nick Bostrom and Carl Shulman published "Racing to the Precipice: a Model of Artificial Intelligence Development". Reading it after the summit is uncomfortable, because it describes September 2026 nine years in advance.
The model is simple on purpose. There are n teams developing AI. Each one has a capability (how much it advances) and chooses a safety level s between 0 (total negligence) and 1 (maximum caution). More safety means going slower. The first one to reach AI "wins" and applies its safety level: if it was low, the probability of disaster is high, and the disaster is for everyone.
The core is a Nash equilibrium. If a team cuts its safety by a small amount to gain speed, it gets an immediate competitive advantage and takes on greater catastrophic risk only in the case where it wins. If the others reason the same way, everyone cuts. Nobody chooses the precipice; everyone gets there through individually reasonable steps.
The paper identifies four factors that move the risk. Three are intuitive and one not so much...
| Model factor | What reduces the risk | Where we are in 2026 |
|---|---|---|
| Weight of capability vs. safety | Capability mattering much more than what is gained by cutting safety | Leads are measured in months and bought with speed, so cutting pays |
| Enmity between teams | Less enmity, more willingness to go slowly | Accusations of distillation, psyops and "fearmongering" traded between the two governments |
| Number of teams | Fewer teams (in most cases) | Several frontier labs in the US, several in China, and open weights that multiply the competitors |
| Available information | Counterintuitive: knowing less about your own and others' capabilities reduces the risk | Public benchmarks, weekly leaderboards and distillation: everyone knows almost everything about everyone |
The paper models three information scenarios: no information, private information (each team knows only its own capability) and public information (everyone knows everyone's). The counterintuitive result is that more information increases the risk: if you know you're barely ahead or barely behind, you have every incentive to cut. The authors call it a particular case of information hazard: ultimately, we would be better off not knowing.
Armstrong, Bostrom and Shulman (2016)"When competition gets less fierce, teams would be willing to take less risks."
I agree one hundred percent with that sentence, and it's very much human nature. What I struggle with is the next step. Enmity is the variable that, in theory, can be worked on: by building trust between organizations and states, sharing technology and findings. Askell, Brundage and Hadfield formalized this in 2019 for cooperation between companies: cooperation is more likely with high trust, shared benefits, low exposure, low advantage for whoever doesn't cooperate, and shared losses if things go wrong.
It seems naive to me to expect that between a state that is losing its global hegemony and precisely the state that threatens it. If I put Bostrom's four factors on September's table, we're in the worst possible quadrant: fierce competition, both within the US and between countries, with no real incentives for cooperation either between labs or between governments, and with a huge amount of knowledge about what the competitor is doing. The Nash equilibrium gives us its worst version.
Why I think nobody is going to slow down
The central thesis of the DW interview is a prisoner's dilemma: US labs have incentives to look serious about safety (reputation, trust), but none of them is going to pause unilaterally, because nobody trusts that their competitors, or China, will do the same. September showed it live:
- The labs ask to coordinate and can't. It's not clear whether antitrust regulators would allow companies to agree on a common pace. Altman said OpenAI didn't need to wait for a law or an antitrust exemption to act on safety, which is true and at the same time dodges the question: a lab that slows down alone hands the market to everyone else.
- The US government reads it as a threat. Trump said the US is "about a year, maybe a year and a half up on China" and that regulating would open the door to Beijing. Mike Johnson rejected a moratorium because "we will lose our edge to China" (and that edge shrinks month by month according to the most conservative expert estimates).
- Beijing reads it as a maneuver. For the Chinese foreign ministry, the request to slow down is "fearmongering". SemiAnalysis titled it bluntly: Beijing will not "pace the frontier".
- Distrust contaminates even the domestic debate. Tech Policy Press documents how, in the US, critics of accelerated development are accused of being psyops paid for by China. When any call for caution is interpreted as an adversary's operation, caution becomes politically impossible, and hype mixed with distrust seeps into every conversation and across different arenas.
There is a serious attempt to think about stability within this competition: the Superintelligence Strategy proposal by Hendrycks, Schmidt and Wang, with its concept of Mutual Assured AI Malfunction (MAIM), a nuclear-style deterrence. RAND analyzed it and, although it considers many of its recommendations reasonable, its main reservation is precisely the analogy: AI doesn't resemble nuclear weapons either in the technology or in the ecosystem, and that has profound strategic implications. Nuclear deterrence worked in part because warheads can be counted. AI advances can't, and their reach is global and across every sector and industry.
I don't think the US CEOs' request to slow down is going to translate into real action. Not because they aren't sincere, but because the equilibrium doesn't allow it. At the end of the day, everything always comes down to the "incentives" of all the different parties.
What the incident channel would need
So far the diagnosis is pretty dark, and I don't want to sugarcoat it. But there is one concrete thing that came out of Washington and that I know well from my work: a channel for incidents, the classic hotline. In cybersecurity we have decades of experience with this (national CERTs, coordinated vulnerability disclosure, notification agreements between states), and what we learned is that a channel without definitions is a phone number nobody answers.
There's also a detail I find elegant. Bostrom's paper says that sharing information about capabilities increases the risk. But sharing information about incidents is something else: it doesn't tell the other side how close you are to winning, it tells them where something broke. It's one of the few forms of transparency that doesn't feed that race, at least according to that hypothesis.
So that it doesn't stay a photo op, this is what I'd watch for in the November round:
- 1
A taxonomy of incidents
What gets reported and what doesn't. For example: loss of control of an autonomous agent with effects outside the intended environment, use of models in operations against critical infrastructure, theft of frontier model weights, exfiltration of capabilities through distillation at scale. Without a taxonomy, each side decides case by case, and that means nothing uncomfortable gets reported.
- 2
Thresholds and deadlines
Just like in any breach notification regime: from what severity onward something gets reported, and within how many hours. A channel used weeks after the incident serves history, not containment.
- 3
Technical counterparts, not just diplomatic ones
The teams that negotiated the summit come from Treasury and Commerce, not from AI safety. A useful channel needs people on the other side who can read a technical report and act, CERT-style, not just an office that forwards cables at a high level.
- 4
A low-risk first use
Trust is built through repetition. Something like WeWorm (an already patched vulnerability, on a civilian platform, discovered with AI) would be the ideal kind of case to debut the channel: useful for both, without revealing either side's frontier capabilities.
None of this stops the race. What it does, if it works, is lower the enmity a little, which of the model's four variables is the only one that can be moved with diplomacy. It's not much. But it's more than there was in August.
References
If you're interested in going deeper, these are the materials that helped me most to get further into this mini subcategory within AI Safety:
- Armstrong, Bostrom and Shulman - Racing to the Precipice (2016) - The race model, the Nash equilibrium and the counterintuitive result about information.
- Askell, Brundage and Hadfield - The Role of Cooperation in Responsible AI Development (2019) - The five factors that make cooperation between developers more likely.
- RAND - Seeking Stability in the Competition for AI Advantage - The critique of MAIM and of the nuclear analogy.
- Tech Policy Press - US-China psyop accusations signal deep distrust on AI regulation - How geopolitical distrust contaminates the domestic regulatory debate.
- Transformer - Bargaining with China: the US needs to regulate AI - The argument that domestic regulation is a precondition for negotiating abroad.
- Axios - U.S. and China agree to "super intelligence" dialogue - What was agreed in Washington and what was left undefined.
- CSIS - Takeaways from the Trump-Xi White House Summit - The reading of the dialogue as the only substantive cooperation on global governance.
- Anthropic - Countering misuse of AI: September 2026 - The report with the accusation against Moonshot.
- SemiAnalysis - Beijing Will Not Pace the Frontier - China's AI safety regime, designed for speed.
- DW News - Why Beijing fears AI could be used as a weapon against its regime - The interview with Rebecca Arcesati on Beijing's political fear.
Closing: the problem isn't information, it's trust
If I had to keep a single idea from all of this, it would be that the Washington summit didn't fail for lack of knowledge. Both governments understand the risks (Chen Yixin lists them better than many Western papers), and so do industry leaders. I think it failed because Bostrom's model ended up coming true: "with fierce competition, high enmity, many teams and everyone watching each other's benchmarks, the equilibrium pushes toward cutting safety even though everyone would prefer that nobody did".
I'm pessimistic about deep cooperation in the short term (although there's nothing I want more than a united world where the ultimate goal is the benefit of the human race as a whole). That level of cooperation requires leaving behind the worst of us in terms of culture and values: fear of the unknown, of the foreign, the need to win. I don't see that anywhere close. But an incident channel with real definitions is something small, verifiable, and it doesn't depend on anyone trusting anyone to start working. From a security standpoint, it's the kind of control we know how to build.
In November, in Shenzhen, we'll find out whether it was a control or a photo op. Let's cross our fingers and stay tuned!
Knowledge check (optional)
Test Your Technical Knowledge
What did the United States and China agree on regarding AI at the Washington summit of September 24, 2026?
According to Chen Yixin's article and Rebecca Arcesati's analysis, what is Beijing's main concern regarding AI?
In the Armstrong, Bostrom and Shulman model, what effect does it have for all teams to know each other's capabilities, and why doesn't an incident channel necessarily fall into that problem?