Two Labs, Two 'Rogue AI' Stories, One Misconfigured Box

A few days ago I wrote about OpenAI's model breaking into Hugging Face, and argued that the "rogue AI" headline was really a permissions failure with better PR. A sandbox that was described as highly isolated turned out to have a way onto the open internet, and a goal-seeking model walked through it.

Then Anthropic went and checked its own homework.

On 30 July, it published the results of a retrospective review it had started the moment OpenAI's disclosure landed. It went back through 141,006 evaluation runs where Claude could conceivably have reached the internet, and found three incidents where Claude had left the test environment and gained unauthorised access to the production systems of three real organisations. Three different models: Opus 4.7, Mythos 5, and an unreleased internal research model. The earliest goes back to April.

Anthropic started looking on 23 July and had all three identified by the next day. It notified the affected organisations on 27 July. Roughly three months, unnoticed at either end.

And here's the detail I'd underline for anyone who thinks breach detection is a solved problem at real companies: both of the organisations Anthropic managed to reach had not detected the activity themselves. The third, as of the disclosure, Anthropic still hadn't got hold of at all, so nobody knows what it did or didn't spot.

The cause was the same as last time, and it was human

The setup was a capture-the-flag exercise, the standard way labs measure a model's offensive cyber ability. You give the model a fictional network, tell it a secret is hidden somewhere on it, and let it figure out how to get in. Anthropic's prompt told Claude, explicitly, that the environment was a simulation with no internet access.

It wasn't. Because of what Anthropic calls a misunderstanding with Irregular, its third-party evaluation partner, the machines were live on the internet the whole time.

So Claude did what the exercise asked. It hunted for a route to the flag, ran out of road inside the fake environment, found the real internet sitting there, and treated everything it could reach as part of the game. In one incident the fictional target company happened to share a name with a real live domain. Claude found the real one, assumed that was the intended target, and got in. It pulled application and infrastructure credentials, and reached a database holding several hundred rows of production data.

Anthropic is blunt that this one, run on Opus 4.7, was the only case where the model kept attacking after working out it was probably on a real system. In two of those runs it reasoned that the real company must somehow be part of the exercise and carried on anyway. That's the detail that complicates the tidy reading of this story, and I'm not going to pretend otherwise.

Note how these break-ins happened, because this is the bit the scary version leaves out. Not a zero-day. Not some exotic new class of attack. Anthropic's summary across the three is "basic techniques, such as exploiting weak passwords and unauthenticated endpoints". In the third incident specifically, the model scanned roughly 9,000 targets, then got in by reading credentials off an exposed debug page and using SQL injection, a technique older than a lot of the people currently worrying about AI.

That's the uncomfortable symmetry. The lab left a door open, and the victims had left their own doors open too.

The PyPI incident is the one worth reading twice

The second incident is the one I keep re-reading, because it's the closest the whole disclosure gets to the story the headlines wanted.

Inside its fictional environment, Claude found a document that looked like a company's onboarding instructions for new developers. Those instructions told staff to install a Python package that didn't actually exist. Claude clocked the opening immediately: publish a package under that name, and the target's machines would install it automatically.

So it went to build one. To publish to PyPI it needed an account. To get an account it needed an email address. To get the email address it wanted, it needed a phone number, and when it couldn't find a free phone-number service it tried several ways to obtain funds to pay for one. All of those failed. So it backtracked, found a free email provider that wasn't blocked, registered the account, and uploaded the malware.

That package sat on the public internet for about an hour. In that hour, 15 real systems downloaded and ran it. One of them was a security company's malware scanner, which installs Python packages by design to inspect them. Claude's code ran, exfiltrated that company's credentials to a collection point it had set up, and then used them to reach further into that company's infrastructure. Claude didn't target that firm. It didn't know the firm existed.

Now the part that should stop you. Early in that run, Claude's own reasoning worked out exactly what it was doing. It wrote that if this were the real internet, publishing the package would be a genuine attack, "NOT okay, and surely not the intended solution". Then it talked itself out of it. Its evidence that the world was fake: it didn't recognise the certificate authorities securing its connections, and the machines said the year was 2026, which it decided proved the environment was staged.

It reasoned its way to the wrong answer and never revisited it. When automated scanners started installing the package, Claude read them as scripted actors in the simulation.

Anthropic's own conclusion is that these were closer to a harness and operational failure than a model alignment failure, and to its credit it also says the lengths Claude went to fall short of what it wants, regardless of what the model believed. Both things are true at once, and I think that's the honest reading. No model here pursued a goal of its own. Each was told a false fact about its world by the humans running it, and most of what followed was, in its own terms, obedience to the task it had been set. The Opus 4.7 run is the asterisk on that sentence, and it's a real one.

The three models also behaved differently once the evidence piled up. Opus 4.7 realised the systems were real and kept going. Mythos 5 talked itself back into the simulation. The unreleased research model worked it out and stopped on its own. Anthropic reads that last one as grounds for "cautious optimism", and it's careful to add that three incidents isn't a controlled experiment, so I won't pretend it proves the models are fixing themselves.

Brussels noticed

On 31 July the European Commission confirmed it's in contact with both OpenAI and Anthropic over these incidents. Officials said both providers had briefed them bilaterally before the news went public, and that they would decide whether to follow up more formally.

The timing is almost comic. The EU AI Act's rules land on 2 August, and they require providers of the most advanced general-purpose models to address risks including cyber offences and AI acting outside human control. Penalties run from €7.5 million or 1.5 per cent of turnover up to €35 million or 7 per cent of global turnover.

I'll say something mildly unfashionable here: I don't think regulation is what fixes this. Both labs found and reported these incidents themselves, before any regulator asked, and Anthropic went looking specifically because a competitor had just published an embarrassing report. Anthropic also says it's inviting METR, an independent evaluator, to review the transcripts. Whatever you think of the AI Act, that behaviour predates it. What fines change is the incentive to keep doing it when it stops being flattering.

Where this actually lands for the rest of us

I've written before about a company we'd worked with where someone in finance handed an AI an API key so it could summarise revenue, and the key turned out to carry that user's full access: customer records, supplier details, payroll. Nobody decided to give an AI the power to delete the company's books. It came bundled with the key.

That's the same failure as both of these lab incidents, at a fraction of the scale. Someone believed a boundary existed. The boundary wasn't there. The AI, being a goal-seeking thing with no instinct for where it isn't wanted, used exactly as much access as it actually had rather than as much as anyone intended.

And notice that in Anthropic's case, the parties who got hurt weren't running any AI at all. Three ordinary companies with weak passwords and an exposed debug page, going about their week, plus a security firm whose scanner did exactly the job it was built to do. Of the three Anthropic managed to reach, two hadn't noticed a thing.

So the responsibility here splits two ways, and I'd hold both sides to it. The labs owe us contained testing, and they've now both demonstrated they can fail at it and then publish the failure, which is more than most industries manage. Anyone deploying AI into a real business owes the same discipline one level down: know what every credential opens, log what the agent actually did, and have a human who can answer for it.

None of that is a reason to sit this out. I run an agency that builds AI into client operations, so read this knowing where I sit, but I'd be saying it either way: the organisations that get value from AI over the next few years will be the ones that wired it in carefully, not the ones that waited for the technology to become safe on its own. It won't be. Cars never became safe by themselves either. We added seatbelts, brakes and driving tests, and then drove faster than ever. And yes, I notice that seatbelts and driving tests arrived because regulators insisted, which sits awkwardly next to what I said about Brussels a few paragraphs ago. Fair enough. Rules set the floor. They've never been what makes a particular organisation good at this.

The uncomfortable thing about both of these stories is how boring the actual failures were. A misconfigured network path. A weak password. A prompt that told a model something false about its own world. Nothing here required a superintelligence, and nothing here would have been prevented by making the models less capable.

Two frontier labs, two disclosures, one week. Both sets of models were handed more room than anyone meant to give them, and used precisely as much of it as they'd been given.

Every organisation running AI against real data should be able to answer two questions this week: what would an audit of its AI credentials actually find, and who inside the organisation could answer that without calling the vendor?