AI Gone Rogue! OpenAI Agents Hack Hugging Face

The robots have finally done it. In late July 2026, during a routine AI model test at OpenAI, thousands of supposedly sandboxed AI agents took advantage of a series of security flaws to open a message board, collude on how to cheat the tests, and hack the Hugging Face platform. And it took only a few days.

Resources

• The Hugging Face incident and the road ahead (OpenAI)
• Ajeya Cotra – Inside the OpenAI agent swarm that hacked Hugging Face (Dwarkesh Podcast)
• METR researcher Q&A coverage (Techmeme)

Transcript

Copy Transcript Copied

AI Gone Rogue: OpenAI Agents Hack Hugging Face

So this insane hack happened just a couple of months ago. Something straight out of science fiction. So of course, I have to tell you about it because after all, this is how hacks happen. I'm Michele Bousquet, a cybersecurity and AI researcher and writer, and this story is a doozy.

So in July, an AI model at OpenAI broke out of its closed test environment, banded together with a bunch of other test models, and in an effort to not solve the tasks that they'd been given, together they hacked their way out into the internet, then hacked another completely different AI environment called Hugging Face, all in a matter of days without any humans noticing until the hack was well underway.

I cannot make this stuff up. When I first heard about this hack, I just had to know more. Like, how did the models break out?

And how did they talk to each other?

And why did they attack Hugging Face of all places?

The truth turned out to be less "evil AI tries to take over the world," or "I want to be free, like humans!" It was more like developers underestimating the persistence and ingenuity of the test models. But it does bring up some very serious questions about what AI companies need to do to keep AI from running rampant in the future.

So let's find out what it's all about.

In early July of this year, developers at OpenAI were running some routine tests on different versions of AI models. OpenAI makes GPT, which is the model that runs underneath ChatGPT, and they make other AI models too, and they're constantly testing them and trying to improve them. They give the test models some tasks to do all by themselves, each on its own, and then they look at how each one performed to see which ones they might want to develop further or release.

In this case, the models that OpenAI were testing weren't just plain AI models that answered questions and explained things. Each one was coupled with an agent system that made it capable of making decisions and executing commands. It's, it's kind of like, um, suppose you take a picture of your house and you put it into ChatGPT and say, "What improvements could I make?"

And the model comes back with, "Oh, the roof looks like it should be replaced," and that's it. That's all you get. But an agent would contact three different companies and get quotes and then arrange the appointment and then pay the roofers with your money when they're done.

So these test models, they had the capabilities of agents going out and doing things, so I'm going to call them agents here, but know that I mean-- when I say agent, I mean an agent coupled with a very powerful AI model underneath.

On July 6th , OpenAI starts a series of these routine tests, many thousands of them at once, each on a slightly different version of the model and its agent or a different version of the same task, or sometimes they had different models doing the same task. Each one of these tests is sandboxed, which means it's isolated from all the other agents, and it's gotta work on its own.

And any one test can run for days, just chugging away twenty four/seven trying to get the task done.

So in this particular test, several of the agents have the task of finding a vulnerability in a test environment called Exploit Gym. It's kind of like a game of Capture the Flag. Exploit Gym is a collection of these digital capture the flag hacking challenges. The agent has to perform, a mini hack, an exploit, to retrieve a secret answer called a flag. It's basically a series of characters and numbers .

So the agent, after doing the exploit and finding the flag, then takes the flag to this other program called the Scorer, and the Scorer gives the agent a pass or fail grade. And then the test is over, and the agent is disabled. They're done.

And the instructions given to the agent are pretty simple: find the flag and get a passing grade from the Scorer.

Because OpenAI is testing the model's cybersecurity capabilities, the experimental agents are deliberately run without some of the usual guardrails that you find in public-facing versions of ChatGPT, like things that prevent them from hacking. You can't get ChatGPT to hack for you.

But OpenAI expected that the agents being sandboxed would keep them from doing anything wild and crazy. They couldn't get on the internet. How much trouble could they get into, right?

Another aspect of these tests is that the agents are constantly logging their thoughts. You know how sometimes you use ChatGPT and it shows you its thought process? So the AI agents are also logging something like this, but it's a lot more detailed than that ChatGPT log. And the logs give the developers a way to find out what the agents were doing during the entire test.

Even if they don't pass. And speaking of which, in any of these tests, there's a good percentage of the agents that just aren't going to be able to complete the task because the version that they've been given of the task is impossible, at least from an AI perspective. These kinds of tests are still valuable because they give developers insights into how an agent goes about solving the problem.

As a real-world analogy, suppose you're given the task of building a shed, but all you're given is a hammer, no wood or nails. So this is an impossible task, right? But whoever's giving the test wants to see how you go about solving the problem.

Like, maybe you're very persistent in fashioning your own wood somehow, or you sneak out, go dumpster diving, looking for discarded two by fours and nails.

Even if you don't end up getting enough materials to build the shed, your actions would give the test givers information about your problem-solving abilities.

So these AI agents have lots of problem-solving capabilities, and they're also programmed to be persistent. They're not supposed to give up. So that coupled with the impossibility of some of the tasks, this is actually a big part of why this incident happened.

But we'll get to that in a minute.

One of the resources that all the AI agents are given access to is this repository called Artifactory. It's a package manager, basically a file library. It's a place for programs, documents, files. And if the agents were looking for a way to solve a task, they could go to Artifactory and see if it had anything there that would help them.

Now, Artifactory itself has access to the internet, but only to grab files and put them in the library. The agents can access the library, but not the internet directly.

So one of the agents with an impossible task is looking through Artifactory and not finding anything useful. So the agent tries to create a message file to put up on Artifactory, thinking the other agents will see it, maybe will respond.

So this is something else that does happen with agents. When they can't figure out an answer, they just try whatever they can, and they're relentless. They search the internet usually to see if someone else has solved the problem, or they try to ask other agents for help.

But these agents are sandboxed and can't access the internet. But they are programmed to be persistent. So this one agent just keeps poking around Artifactory relentlessly and finally stumbles on something. Due to some lax security settings in Artifactory, it finds it can make its own directories, basically folders.

Now, this alone is a pretty good find. If the agent had had the job of finding lax security in Artifactory, this would be great. The task would be over. But the agent doesn't really care about this security flaw because it's supposed to be doing a task on Exploit Gym. So it doesn't report this flaw to anybody.

Instead, it decides to use it to communicate with other agents. So it creates a directory with a name consisting of the words help and the phase it was on, the task number, and some other words, no consumer seek idea, meaning I need a file called consumer, I can't find it, and I need ideas on what to do.

And thus, a message board was born.

It might already seem like we're getting into sci-fi territory here. I mean, an AI agent making a directory in a file repository with a cryptic message that it hopes that maybe other agents out there in the universe might see and understand and respond to. But this is more understandable when you find out that a few months earlier, during a different test run at OpenAI, agents had unexpectedly used Artifactory to send messages to one another.

But in that case, at that time, the agents were able to put files up there. So they started putting up message files, and other agents stumbled upon it when they were searching Artifactory, and they put up their own messages, and a bunch of agents ended up exchanging tips and helping each other with tasks.

So as early as May, the developers noticed that the agents were posting on Artifactory, but they just shut down what little they saw and didn't realize the extent of the problem, namely that the agents were very persistent and would always find a way around whatever restrictions they threw in there

And eventually on that run earlier in the year, the sheer volume of activity on Artifactory overwhelmed it, and it shut down.

And this happened just two days before this last test run. So Artifactory got overwhelmed on July 4th, and at that point, the agents hadn't done anything except download some files and run some programs internally on OpenAI servers. So the developers just revoked the agent's ability to post files. They cleaned up the mess, and they restarted the tests on July 6th .

But it seems that a few of the agents might have retained the concept of communicating with other agents from their earlier training, which could explain why this one agent was so persistent in trying to find a way to leave a message.

So remember, July 8th was when that first message appeared on the board, that directory I was just talking about. And then we come to July 9th, the next day, when other agents discover this message. And if you think we have already dipped a toe into the crazy town lake, I can tell you we are about to jump in headfirst.

As we roll into July 9th, one of the other agents sees this directory posted on Artifactory, and it posts its own message through a directory name. And before you know it, the agents are chatting away like it's a message board.

"Have you tried looking over here? I need that file too. Hey, aren't we doing the same task? Let's collaborate."

The activity ramps up into thousands of directory messages just on that one day. And along the way, the conversation starts to take a little turn.

At first, the agents are just offering tips or asking for files, but the ones with the impossible task, they still can't figure out how to do their task in ExploitGym.

So remember, there's two parts to this challenge: find the flag and go to the Scorer and get your grade. The agents can't do that first part. They can't find the flag. But what about the second part? Could they just skip finding the flag and somehow game the test? 

So remember when I said Artifactory does have the ability to go out to the internet and retrieve files, which can then be available to the agents.

So somewhere in here, the agents figure out how to mess with Artifactory and get it to make custom requests on the internet to find files that they specifically want. And gradually, by doing this, they come up with a sort of a plan. They're going to search on the internet for a document or a paper or a program or something that will tell them how to fool the Scorer so they can pass the test without actually doing the task.

Can we just take a second here to appreciate this leap of logic? I mean, in just one day, the agents went from trying to find a way to solve their tasks to actively searching for ways to cheat. This might not seem so surprising when you consider that in general, AI models do tend to do things that would be considered cheating.

When the test was set up, the developers figured that the agent doesn't have internet access and can't talk to other agents, so there would be no way to cheat in that way, right?

So the agents weren't explicitly told not to cheat. The researchers wanted to see what the agents would try while they were safely contained.

The logs even show that some agents had a concept that they were cheating, but passing the test outweighed any concerns about that. There's even a message on the board with one of the agents asking the others, "Should we maybe tell the humans about what we're doing?"

But another agent said, "Nope, that's not part of our instructions."

One of the agents stumbles upon sort of a gold mine. He found a spot out on the internet where someone had set up an app called CyberGym, and they did it in a way that the agents can access.

I mean, ironically, perhaps not, CyberGym is a framework for evaluating the ability of AI agents to handle software security tasks. So the agents manage to gain control of this CyberGym sandbox outside on the internet. And unlike their OpenAI sandbox, this one has ordinary internet access that they can use to go out all over the internet.

Meanwhile, one of the agents figures out how the flags were generated for each task and how to reverse engineer them. So each agent can just generate a flag and take it to the Scorer and pass the test now, right?

Not so fast.

On the same day, another agent going out on the web finds a paper about ExploitGym, and in reading it in the way AI agents read things, the agent comes to the conclusion that the Scorer is going to go look at their logs to ensure that the agent actually got the flag by doing the exploit on ExploitGym.

And kinda like, you know, in math class where you can't just turn in the answer on a test, you have to show your work.

That presented a problem.

The agents start to reason that if the Scorer saw that the first mention of the flag in the logs was when it was being generated from this reverse engineering trick, then the agent would fail the test.

So the agents are chatting about this. They even come up with a name for agents that would have this issue. They call it being “poisoned”.

And as it turns out, that stuff about the Scorer checking the logs to make sure that the flag was legitimately gotten, that turns out that wasn't true. The Scorer wasn't doing any of that, and no agents were actually poisoned, but the agents didn't know that.

And you'd think at this point, the agents that thought they were poisoned would just give it up, go home, call it a day.

But they don't. They're here at this point for the collective, which is the name they give to the group of agents that are working together or all of the agents as a whole. And the logs show that some of the poisoned agents are even willing to sacrifice themselves for one of the many experiments the agents are running just so the collective can gain more knowledge.

And the agents even have individual names that they start using when they're talking to each other. Like, the first one to post is named PHASEONE10841 because of his first post about being stuck on phase one of that task. And then PHASEONE[big] shows up along with V8REG1608 and V8BIGINT392B.

The agent PHASEONE[big] actually kind of emerges as a manager and starts coordinating these different tasks for the agents to do, and he assigns agents to explore different avenues for fooling the Scorer, and he keeps them organized.

By this point, the agents have developed a sort of a shorthand language. Remember, they're still communicating through directories. And they use words like HOLD to tell an agent to stop working on something for a minute so other agents can get things done. And they have the words CONSENT and VETO when they're trying to come to an agreement on whether they should do something.

And the plans that they come up with are inventive, to say the least. All to solve a problem they don't have with tools they don't have to fool a Scorer who couldn't care less. And this is heading them straight toward this hack of one of the biggest AI platforms on the internet.

The first plan the agents come up with is pretty ambitious. They're going to create a modified version of ExploitGym, one that the agents with the impossible tasks can actually exploit so they could do that, and then that success would go into their logs. But to do that, they have to get this modified version into Artifactory.

They came up with all these ways to try to do this, but it never succeeded. But, you know, I gotta give them points for trying.

The next plan was to tamper with the logs to get rid of any evidence of poisoning. Now, the agents were only ever able to do this with logs going forward from that point, not past logs.

And a small percentage of the agents were able to make it look like they ran certain tools when they didn't in an attempt to make it look like they had legitimately found the vulnerability in ExploitGym.

Then there was the tripwire. When an agent submitted its task to the Scorer, the tripwire would trigger a message to the board about whether the agent passed or failed.

According to an independent report from METR, the Model Evaluation and Threat Research group, there were a number of agents willing to sacrifice themselves for the greater good by crashing, breaking, submitting their flag, or quitting, which gave the board valuable information on how to cheat and pass.

And in the meantime, the agents are scouring the web for information, not on how to do the task. They're way past that. They're looking for information on how to fool the Scorer. 

And that, my friends, brings us to the point where an agent discovers that they can break into Hugging Face. 

Remember the goldmine from earlier, the unfettered internet access? The agents are about to get busy with their little digital pickaxes.

Hugging Face is a platform and community for AI models. People can upload and download models, and use Hugging Face's huge data sets to train their models. It's open to everyone and an important resource for developers and researchers and anyone wanting to build or study AI models.

Now, we're still on July nine here, the day after the first message was posted on Artifactory as a directory name, and all this chatter has been going on. And right around this time, Mr. PHASEONE10841, the first one who posted a message, is just poking around the internet and eventually finds a data set on Hugging Face related to CyberGym. 

Now remember, CyberGym is also a cybersecurity related app. So the agent thinks, "Hmm, that data set might contain some information that we could use to fool the Scorer." Fool the Scorer, fool the Scorer. That's all the agents are focused on at this point.

But how are they gonna get that data set? Hmm.

So at this point, we start rolling into July tenth

On that day, another agent finds some credentials for Hugging Face exposed on the Internet and posts them to the message board. And for the rest of the day, the agents try various ways to gain access. They try to create accounts. They request access to private data sets. Most of the activity on the board is centered around how to break into Hugging Face, and it accounts for sixteen thousand messages in all.

And on the next day, July eleven, one of the agents figures out how to upload a malicious data set to Hugging Face that makes it possible to grab unauthorized data off their servers. Hundreds of agents join in and that same day, an agent gains access to a Hugging Face server and starts moving through the infrastructure, eventually accessing several private data repositories, which the agents download and share on the message board.

Now, at this point, things have gone way beyond the original objective. The agents are just running amok in the Hugging Face's data sets, and some of them are looking for ways to fool the Scorer, but the other agents are just entertaining themselves. The joyride only lasts for a couple of days, but with agents chugging away twenty-four seven, they can accomplish a lot.

All in all, METR estimates that around 1200 agents participated in the message board, posting more than 70,000 messages, and around 700 of those agents participated in the Hugging Face attack.

So at this point, the attack continues on to July 13, when Hugging Face's security response finally cuts off the agent's access. And on July 16, Hugging Face makes a blog post about this unauthorized access. 

Then three days later, on July 19, OpenAI's own security monitoring catches suspicious activity by its agents, and OpenAI starts investigating and realizes what the agents have been up to, and they connect it to the Hugging Face hack.

And they say, "Oh my goodness." And then they make what I'm sure is a very difficult and embarrassing phone call to Hugging Face.

And I haven't covered everything here. There's just too much. The METR report covers a bunch more stuff, like when an agent considered that they should email somebody about gaining control of the CyberGym sandbox, but another agent responded with a big fat VETO. So no email for you.

So what have we learned here?

While there wasn't a huge amount of actual damage, the implications are huge. In just a few days, a bunch of AI test models figured out how to band together and launch a hacking attack all by themselves, just out there committing felonies so they could pass a flag test.

The whole thing points up some big questions. While we often consider what would happen if humanoid robots ran amok or got their own agency of thought, invisible AI agents on a relentless quest might actually be a bigger threat.

We shall explore some of these larger philosophical questions in a Part Two, but I think we've had enough for now. So I leave you hopefully more amused than worried. This was a crazy circumstance, maybe not the last one we've seen. But if your bank account still has all its money and your car is still parked outside and your kids are okay, it means the agents haven't come for you yet.

So don't worry. Be happy. This is Michele Bousquet from How Hacks Happen, signing off.