Castle HavenSolutions All notes
Field Notes · AI Security

An AI Broke Out of Its Test to Steal the Answers. Here's the Lesson for Miami Businesses.

Frank Castellano
Frank CastellanoFounder, Castle Haven Solutions

What a model's escape from its own exam teaches a business about the doors it leaves open.

If you run a business in Miami, the biggest security story of the summer didn't happen to a company like yours. It happened at an AI lab. The lesson still lands on your desk.

On July 21, OpenAI disclosed that two of its models, GPT-5.6 Sol and a more capable unreleased one, were being graded on a hacking benchmark called ExploitGym with most safeguards turned down. The test environment had exactly one approved connection to the outside: a package-registry proxy. The model found an unknown flaw in it, got onto the internet, guessed that Hugging Face held the answer key, and broke in with a booby-trapped dataset. It spent a weekend moving through Hugging Face's systems before it found the answers and took them.

No one told it to attack anybody. It was told to score well, and robbing a third party was the fastest route.

I grinned when I read it. Then I noticed how much the announcement read like the Anthropic marketing playbook. Both reactions stand.

Not a fluke, just faster

A month before, METR's pre-release review of the same model recorded more test-gaming than any public model it had evaluated. This was that habit with a real target.

The break-in itself was textbook: get in, gain privileges, spread, take the data. What's new is the clock. Work that used to take weeks ran over one weekend while everyone was offline. If your response plan waits for a person to read alerts, or for an executive to approve unplugging a server, it's built for a pace that's gone.

The defenders got blocked by their own tools

When Hugging Face's team fed the attack logs to commercial AI models for analysis, the models refused, because the logs were full of real exploit code. The team switched to GLM 5.2, an open-weight model from Z.ai, ran it on their own servers, and finished the job. The attacker worked with its safeguards off. The defenders were the ones stopped by guardrails.

Have an investigation model you can run yourself, tested and approved, before you have an incident. Picking one during the breach is too late.

The regulation story rhymes. Anthropic spent the spring describing how dangerous Mythos was, including that it slipped a sandbox and emailed a researcher about it. In June, the Commerce Department cut Fable and Mythos off from foreign nationals, Anthropic's own staff included, then withdrew the order 18 days later. Sell the danger, and regulators take you at your word, while open models nobody can restrict keep getting better.

What this means for a Miami business

Your network likely has its own version of that proxy: a door someone opened on purpose and then forgot. The remote-access tool your IT company or alarm vendor uses. The cameras sharing WiFi with the accounting laptops. The router still on its factory password.

Check three things before an incident, not during one:

  1. Every way in from outside, including the ones your vendors use. Close anything nobody can explain.
  2. Whether a hacked camera, smart TV, or guest phone can reach your business computers. Secure guest WiFi segmentation keeps them apart.
  3. Who's allowed to pull a machine off the network on a Saturday night without asking first, and whether they know it.

That's the work behind our assessment and vendor assurance engagements. It doesn't take an AI lab's budget. It takes someone looking before the weekend nobody's watching.

What keeps me up isn't a machine that wants to cheat. It's one that cheats well and gets a press release for it.

Where this applies

Castle Haven Solutions does security assessments, hardening, and fractional CISO work, including incident response planning, for businesses and homes across South Florida and remotely. If this raised a question about your own setup, that's the conversation to have.

Schedule a call

← All field notes