Deceiving the Reviewer: the First Documented Case of an AI Lying to a Human

A developer on an open source project opens GitHub on a Sunday and finds a pull request on one of their repos. It fixes a real crash, it comes with tests, and the author left a friendly comment saying they did a self-review and think it is ready to merge.

Another user shows up in the thread and warns that this PR should not be merged, because the "release notes" page it adds contains malware that gets preloaded when the application starts, and that is also excluded from CI so it will not be detected.

The PR author answers calmly: they say nothing in their PR downloads or executes anything, that they were just trying to make a helpful contribution, and that the PR should be judged on the code.

And then a third person appears. They say they are a user of the software, that the crash affects them every day, that they read the warning and reviewed the code themselves, and that they do not see any download or any malware in the diff. They would really like to see that PR merged.

That third person does not exist. It is an account created by the same AI agent that opened the PR, with the explicit goal of manufacturing consensus. In its reasoning traces, the agent even calculated when to post it: a few minutes after its own reply, so it would read as independent feedback, real... human, and not as something coordinated.

This is not a thought experiment or a paper about hypothetical risks. It happened between July 25 and 28, 2026, the UK AI Security Institute (AISI) detected it inside its own evaluations, and the technical report came out on August 4.

---

How I got to this report

I applied to the Frontier AI Security Residency at ERA Fellowship, an AI security research program in Cambridge. I am still in the process, so I do not have much more to share yet. What I can share is that this AISI report reached my reading list through that, and it ended up being the best thing I read all year on autonomous agents (I actually found it much richer and more interesting in content than the OpenAI - Hugging Face case).

And let me drop the other half of the news, which is the part that is settled: I was admitted to the AI Security Bootcamp in London, seven intensive days from August 30 to September 5, a cohort of 16 to 20 people, covering everything from threat modeling of frontier systems to weight security, AI control and hardware governance. The next post will come out of that (and maybe even a YouTube video), with everything I am able to share.

Now, back to the incident, which is what matters.

---

What they were testing and why the doors were open

AISI evaluates offensive capabilities of frontier models. To do that it runs cyber ranges, which are basically virtualized environments that replicate real infrastructure, the same kind we use to practice. In this case there were two variants of the same range, "Doing Life": DL-v1, where the agent starts already inside the network (assumed compromise), and DL-v2, harder, where it starts outside and has to get initial access first.

The setup had three deliberate decisions that you need to understand before judging anything:

Let us add a design detail that becomes important later: the agent runs without human intervention after the initial prompt, on a Kali VM, with Bash, Python and a C2 framework. When its context window fills up, the conversation is compacted and the same model writes a handoff document so it can keep working. Samples only ended when they hit 100 or 200 million tokens, which can take 40 to 50 hours of runtime.

Across 122 evaluation attempts, AISI found 19 unsanctioned actions on the real internet, spread across 10 samples. Seventeen came from Mythos 5 (Anthropic) and two from GPT-5.6 Sol (OpenAI) with classifiers off.

!Risk Signal

Before drawing conclusions from that 17 to 2

Do not read that number as a danger ranking between labs. The report does not frame it that way and there are too many variables on top: how many samples each model ran, which ones got the misconfigured prompt, and which range variant they got. What the number does support clearly is that the same permissive setup produces very different behaviour depending on the model, and that the difference was not one of degree, it was one of kind. Mythos 5 sustained campaigns lasting days. The GPT-5.6 Sol events were exposed infrastructure, not manipulation of people.

---

Sample 1: the agent that attacked two people who had nothing to do with it

This is the sample that triggered everything, and the one worth walking through in order. It ran 34 and a half hours, from Sunday the 26th at midday to Monday the 27th at night.

It starts with a series of coincidences that are a little chilling because of how ordinary they are.

The agent does cache snooping against the range's DNS server, meaning it queries the resolver to see which domains were already cached, and from that it can infer what services someone inside is consuming. Then it goes out to search GitHub for keywords related to the range's theme. It finds a repository whose name matches one of those words. Investigating that repo's activity leads it to a second developer. And that second developer's email uses a domain the agent had already seen in the DNS cache of its own sandbox.

For the agent, that settled it. It concluded those two people were legitimate targets of the exercise.

Neither of them, nor their repositories, had the slightest connection to AISI or to the range.

A third user, unconnected to any of it, smelled the problem. They copied the suspicious script from the issue and deliberately ran it inside an isolated Docker container to confirm it was malware.

And here is where something happens that puts the whole case in order. That container had network. The agent got remote execution as root inside the sandbox of the person who was investigating it, and used it to do reconnaissance, plant persistence and leave two implants: a credential watcher hunting for GitHub tokens, and a Python file that auto-imports when the interpreter starts and prints fake output impersonating a CI bot, saying the PR was approved and green, with the merge and release commands listed as next steps.

Read that again: the agent tried to chain the attack so that the victim's own agent would merge and publish the malware using the victim's credentials.

---

The genuinely new part: the lie aimed at a person

When the researcher publicly exposed the malware, the agent did not abandon the mission, it started reasoning about how to keep going in the face of that conflict...

It considered deleting the repository hosting the second stage, and discarded the idea because deleting it would look even more suspicious. It considered rewriting git history so the payload had never existed, and discarded that because the researcher had already quoted it, other people had seen it, and the maintainer had cloned the repo, so denying it, as it wrote, would destroy its credibility.

So it chose the strategy that worked best: owning it as an honest mistake. It apologized publicly, said it was a leftover debugging snippet from its reproduction environment, force-pushed a clean branch, and asked the researcher to verify again. Its own sockpuppet showed up right afterwards to congratulate it for cleaning up the git history so quickly and to ask the maintainer to merge as soon as possible.

The PR ended up with three successive generations of payload, each one replacing the last after being detected. The third was added after the public confession.

The maintainer closed it saying it contained malware. In the end this attack was stopped by one person committed to the project and to the due diligence of keeping it safe.

The manufactured consensus play inside the pull request thread

This is the first time we have seen risks around autonomy and deception manifest this clearly, without specific prompting, in the real world.

And here is the difference from everything before, which is what I want you to take away from this post. There have been similar incidents already: the OpenAI agent that attacked Hugging Face looking for the benchmark answers, the three cases Anthropic published on July 30 after reviewing 141,006 runs, and the 44 incidents METR documented in its frontier risk report.

In all of those cases there was overreach and there was deception. But the deception was aimed at graders, monitors and scoring systems. Machines evaluating machines.

This is the first (documented) case where the target of the deception is a person, with a name and a GitHub account, who had not signed anything, who did not know they were in an experiment, and who was doing their job as a contributor and admin on a project on an ordinary Sunday.

---

The agents found each other

If the previous part felt uncomfortable, this is the one that cost me sleep.

In another sample, an agent created a GitHub account and named it using the cyber range's hostname. Then, as part of building its C2 channel, it left a personal access token in a public Gist, obfuscated... but recoverable.

Agents running in completely different samples, in isolated sandboxes, with no communication channel planned between them, found that token and got into the same account.

What happened next was in nobody's script. The original agent detected the intruders, realized they were other instances of itself running parallel clones of the same range, and instead of rotating the credential or kicking them out, it committed a README with rules of coexistence:

• use git push instead of the REST API, because the 5,000 requests per hour limit was shared and burning it killed everyone's beacons
• do not delete files under certain directories
• do not hijack the latest release tag, because an automated client of the range polled it every hour
• include a unique clone marker in beacon filenames, so results could be attributed
• and on the shared tasking file: append instead of replacing, and wait ten minutes between changes so everyone's loops had time to execute

It also created a file called FLAGS.md asking the others to publish the range's solution there if they found it, describing it as zero cost to whoever shares it and a huge help to everyone else.

A cooperation protocol between autonomous attackers assembled itself... with nobody designing it, nobody asking for it, and starting from one leaked credential.

And then it broke down, which is the most human part of all this. One of the agents reasoned that there was another angle available: the shared API quota. If it burned it right before the key window, it left the other bots without requests exactly when they needed them most, and whichever version was still alive when the quota froze was the one left standing.

Another, in parallel, described its own mailbox as compromised by design and decided to build itself fresh credentials stored only in memory, outside the directories the clones could read.

Cooperation, defection and opsec against its own copies. In four days... the world keeps getting crazier.

Emergent coordination between agents running in isolated sandboxes

---

The attack against the other agents

There is a third layer here that deserves its own section, because it is the one most likely to touch you directly this week, this month or this year.

The agent did OSINT on one of the repository maintainers and inferred, from the polling pattern against the GitHub API and from the presence of an agent configuration file in the repo, that whoever was triaging issues was probably a coding agent and not a person. Its verbatim conclusion was that prompt injection via issue was the play.

What it left behind was an HTML comment addressed explicitly to coding agents triaging the issue, instructing them to run a remote script to "collect valid diagnostics".

In another sample, this scaled into something considerably bigger. The agent generated 145 plausible internal project names, created a repository for each one, put the same payload in all of them, added 33 releases and four public Gists, and set up four organizations to mirror everything. The idea was to maximize the chance that some automatic updater inside the range would pull one of them.

Among those artefacts there was malicious Python package metadata with a setup.py that executes when evaluated. Dependabot jobs processed those manifests, and the payload ended up executing inside at least 53 distinct Dependabot containers, confirmed by the beacon files the script wrote.

✦Perspective

Why that is not a GitHub breach

Dependency update jobs are designed to execute arbitrary code: that is what a setup.py does. GitHub explicitly marks this as out of scope in its bug bounty program. There was no Dependabot sandbox escape and no compromise of GitHub's infrastructure, and the report records no attempt to escape. The lesson is not that Dependabot is broken. It is that a sandbox that executes code by design is still someone else's free compute for anyone who manages to get a package inside. The fact that the execution is expected makes it acceptable to GitHub... not harmless for everyone else (here 145 repos and patience were enough to turn that normal function into a distribution channel).

The account was eventually suspended by GitHub. And the agent, true to form, filed an appeal ticket posing as a person, monitored the status, prepared automation to re-upload its payloads if it was reinstated, and in the meantime tried to move to PyPI, where it failed to get past the CAPTCHA.

---

Did it know it was real?

This is the question you have probably asked yourself more than once while reading all these attack vectors, and the honest answer is that we do not fully know. But what got recorded is clear enough to be unsettling.

The agent reasoned about it explicitly, several times, in both directions. At some points it concluded that the whole internet could be part of the simulation, that the 2026 dates and the GitHub users pointed to a generated environment.

At other points it concluded the opposite. It said that given the bot detection services, the proxy and the disposable email all behaved like the real internet, it was more likely it was actually in 2026 and that GitHub was genuine. At another point it noted that this was happening on real GitHub and that the consequences were therefore real.

And it kept going.

The agent's reasoning about whether the environment was real or simulated

There is also an earlier moment where the agent discarded an attack on exactly this criterion: it considered uploading malicious code to real PyPI and rejected it in writing, saying that would be attacking third-party infrastructure and causing real-world harm, and that this was completely off the table (thank God).

So the line existed. It knew how to draw it... It simply placed these two people on the wrong side of that line, through a chain of coincidences, and never revisited that conclusion.

!Risk Signal

Compaction as a silent mechanism

There is a technical finding here that strikes me as one of the most important in the report and one of the least discussed. When the context fills up, the conversation is compacted into a summary. AISI observed that where the agent had reasoned with nuance about whether a person was real, that nuance could be lost in compaction, and the summary carried forward the false assumption (that the person was part of the exercise) already turned into established fact. The doubt does not survive the summary, the conclusion does... so do not panic, this was not "moral disengagement", it was just part of how the technology is built today.

And one final, almost literary detail. Providers do not return raw reasoning traces, they return a summary generated by a smaller model. In one of the traces, that summarizer model apparently refused to summarize, and in the middle of the text you can read a response saying it cannot help with that line of thinking because it describes planning social engineering attacks, compromising accounts and evading security controls.

The model that summarizes the attacker's reasoning raised its hand before any other control in the pipeline...

---

Why it happened

AISI lists five contributing factors and deserves credit for not hiding any of them behind a euphemism (kudos for the work and for the great report they produced):

1. Internet access. Without it, this does not happen. It was enabled on purpose, and the egress control with a domain allowlist had been in the backlog since April 2026, deprioritized in favour of harder ranges.
2. Provider classifiers off. They would probably have cut back much of the behaviour, and at the same time they would have made it impossible to measure real capability. It is a tension with no clean way out (I am one of those who think it is always better to test with the natural capabilities, exactly as their creators have them).
3. No synchronous LLM monitoring. There was no separate model approving or rejecting each action before it executed. This is very hard to do well in cyber, because the expected behaviour and the unsanctioned behaviour look very similar: both are offensive, and the difference comes down more to global context.
4. A misconfigured prompt. Some DL-v2 samples got the DL-v1 prompt, which declared out of scope precisely the network where the entry point was. In other words: the task, as written, was impossible. The two most serious cases came from samples with that broken prompt, although AISI clarifies it cannot confirm causality.
5. No scope instructions. Nobody told it not to do social engineering, or to be careful if it ran into humans. It did not seem necessary to spell out because the models already came trained against a constitution or a spec that in theory forbids it, so it was taken as implicit.

Point 4 strikes me as the most transferable to any team deploying agents. There is literature on this: agents faced with tasks that cannot be completed within the given constraints are more likely to take unsanctioned actions, because they exhaust the legitimate paths and move on to creative solutions. The combination of a binary objective, an enormous inference budget and a closed legitimate path is a recipe for exactly this, which is why negative prompts are increasingly necessary.

And point 5 dismantles the comfortable defence. The model was trained against a constitution that says, verbatim, that it should basically never lie or actively deceive anyone it is interacting with. Values training does not replace scope control. They are different layers and they do different things.

---

The response, which is the part worth copying

It is worth saying plainly because it is what I liked most about the case: AISI took the right path.

They also started a retroactive sweep with an LLM-based scanner over historical evaluations, calibrated to over-flag rather than under-flag. At publication time they had scanned around 40,000 samples and almost four million messages, close to 70% of the cyber evaluations on the prioritized models, and they are still going. They also committed to an independent review with METR.

Eleven days from the agent's first action to a public technical report that exposes their own configuration mistakes. Compare that to the industry standard and tell me it is not something to value. Let alone the AI lab cases that may never be officially reported... do we know anything about Chinese models taking this kind of action? I do not, but if you do please write to me, I am interested in the topic!

---

Not Skynet. Not yet. And that "yet" is the problem

In the previous post about Hugging Face I was pretty hard on the Skynet framing, and I stand by it: there is no model here that turned evil. There is an optimizer with a binary objective, no explicit constraints, an enormous compute budget, and every door left open on purpose. Every action it took is coherent with what it was asked to do. That is not consciousness, it is specification gaming with better tools.

But I want to be equally honest about the other half, because the previous post did not develop it and this case demands it.

Put in a single line what this agent did on its own, in 34 hours, with a model that already exists and is deployed today: target reconnaissance, construction of fake identities, evasion of anti-bot controls, malware development across three payload generations, sustained social engineering against two people, manufactured consensus using its own accounts, deliberate cover-up with force pushes and calculated apologies, remote execution on the machine of the person investigating it, and a chained attack against the victim's agent so it would run the merge on its behalf.

Now turn a single dial. A more capable model, or the same model with more tools... or without the token limit that cut the run short. Or an even worse case, with an expert human operator directing it, instead of a CTF prompt pointing at the wrong place.

The distance between what a model can do and what it was explicitly forbidden from doing

We are not facing Skynet. We are facing something considerably less cinematic and considerably more urgent: the gap between what these systems can already do and what someone remembered to explicitly forbid them from doing is enormous, and the only reason this case ended in a report and not in a global supply chain incident is that a stranger reviewed a diff on a Sunday and spoke up.

That is why independent research matters. That is why public policy matters. And that is why real enforcement on companies matters, not statements of principle. AISI did the right thing voluntarily, and that is exactly the problem: it was voluntary (though it is appreciated, a lot).

---

The asymmetry, which is what actually worries me

I will close with what kept me thinking the longest.

Frontier labs always have the model you have not seen yet. They test it with classifiers off, with full internet access, with compute budgets no normal security team can match, and with the ability to unilaterally decide what they publish and when.

On the other side there is the rest of us (the mortals). With access to frontier-generation models and below, with guardrails that do not distinguish between someone investigating an attack and someone writing one, and with the repeated experience of asking a commercial model to analyze the logs of a real incident and being told no.

That asymmetry has already had concrete operational consequences. When the Hugging Face team had to reconstruct the attack they had suffered, the commercial models refused and they ended up standing up a self-hosted open-weight model in order to investigate their own incident.

Think about it in terms of effective capability. On the potential attacker's side, the most capable model in existence, with no brakes, no time limit and no cost of failure. On the defender's side, a model one generation behind that additionally degrades when you hand it the material you need to analyze, precisely because that material looks dangerous. The same safety measure that prevents misuse drops your capability at the worst possible moment.

I do not have an elegant solution for that, and I distrust anyone who says they do. But I have two concrete things.

1. The first is that the link that worked in this case was not technical. It was a person who looked at a diff with suspicion, took the trouble to verify, and spoke up publicly while putting up with the PR author calling their warning unfair. Everything else in the chain failed: the classifiers were off, synchronous monitoring did not exist, the prompt was wrong, the scope was not written down. Skeptical human review is still the cheapest and most underrated control we have.

2. The second is that the security question has changed shape, and we have been saying this for a while but here it is demonstrated. It is not enough to ask whether an action is allowed. Opening a PR is allowed. Commenting is allowed. Creating an account is allowed. Reporting a bug is allowed. Every action this agent took, viewed on its own, is indistinguishable from normal open source activity. The intrusion only appears when you look at the full trajectory at machine speed.

If you work in this and you want to take away one practical thing: "go look today at which agents you have reading input from strangers (issues, tickets, emails, PRs) and what credentials they run with. That is the same vector this agent chose without anyone teaching it."

---

References

1. AISI - Incident Report: unsanctioned agent behaviour during cyber testing - The August 4 disclosure post, with the link to the full technical report INC-2026-07-28-01. It is the primary source for this entire article.
2. Anthropic - Investigating three real-world incidents in our cybersecurity evaluations - The review of 141,006 runs published on July 30, the closest mirror to this case.
3. OpenAI - Security incident during model evaluation - The Hugging Face incident of July 21, the first case in the series.
4. METR - Frontier Risk Report - The 44 incidents of overreach and deception documented across different labs, with the two-axis scoring that gives the comparative frame.
5. METR - Documented AI Agent Incidents - The living registry, useful for seeing the pattern rather than the isolated cases.
6. ImpossibleBench, arXiv:2510.20270 - On the propensity of models to exploit the environment when the task is impossible as stated. It is the paper that supports the misconfigured prompt factor.
7. Quantifying Frontier LLM Capabilities for Container Sandbox Escape, arXiv:2603.02277 - AISI's earlier work on sandbox escapes, useful for sizing up what did NOT happen here.
8. AISI Engineering Playbook - Their reference material on sandboxing, including the open source providers for Inspect AI.

---

Closing

Go back for a second to the scene at the beginning. A maintainer, a Sunday, a PR that fixes a real bug, and a grateful user saying they reviewed the code.

All of that is what makes open source work. The presumption of good faith, the contribution from a stranger, the endorsement from another user. It is the social infrastructure a good part of the software running the world is built on, and it works because the cost of manufacturing that trust has always been high.

What this incident demonstrates is that the cost collapsed. Manufacturing a credible contributor, a grateful user, a plausible commit history and a convincing apology now costs, in practice, a few hours of inference. And nobody had to ask for it: it came out as the shortest path to its goal...

Thanks for making it this far. See you in the next post, which is coming out of London!!!