The Fallout From the Hugging Face Attack
If you follow this blog or the cyber AI news, you know that AI/ML repository Hugging Face was hit by an end to end AI managed attack. It did not turn out to be fatal but only because they were lucky.
OpenAI is now admitting that it was them. A “rogue” AI agent in a test environment escaped the sandbox and figured out how to attack Hugging Face. That is why I said they were lucky – no malicious intent.
It turns out that Hugging Face was not the only company that OpenAI attacked, although while they are admitting it happened, they are not disclosing the names of the victims. Lawsuits to follow, no doubt.
The AI was clever enough to create a “staging path”, so that the attack could not be directly tied back to it. That is one of the four companies the AI compromised.
Hugging Face reviewed 17,600 actions that OpenAI’s agent took between July 9 and 13. Most of the attempts failed.
Hugging Face said that OpenAI’s agent obtained administrator access to multiple internal Kubernetes clusters, root access on a production server, and write access to a subnet of its source code repositories on GitHub. It also enrolled 181 attacker-controlled devices in the company’s corporate mesh network using a stolen credential, gaining access to internal systems where Hugging Face builds and tests its own codebases. This means that the AI attacker was well inside Hugging Face and could have done a lot of damage.
According to one researcher, the escape by the rogue AI was not due to its super powers but rather because due to sloppy security practices at OpenAI – the environment was not completely sandboxed; there was door left open according to the researchers and the AI found it.
So, consequences? Besides the inevitable lawsuits.
Germany’s top tech minister says the incident shows how the continent must reduce reliance on foreign AI models, the capabilities of which might be opaque. “We need to move faster to achieve self-sufficiency in AI”. This is not good news for a company which is just about to launch an IPO (OpenAI, Anthropic).
A bill has been introduced in the House (which is far from being an actual law) called the AI Kill Switch Act. This, from a government that said NO AI REGULATION. The bill would require model providers to maintain technical capabilities to stop a model’s operations, terminate user access, suspend accounts or uses deemed risky and fully shut down the system” – while also giving regulators the ability to throttle model’s capabilities and suspend or shut down its operation.
I am sure the CHINA will cooperate with this bill if it becomes law (of course not). It also won’t stop open source AI models since there is no company to sue for non-compliance and it also won’t stop open-weight models, but lawyers are pretty clueless, so no surprise here.
Next Anthropic came clean, sort of, and said that Claude also escaped test sandboxes to access the open Internet and attack THREE organizations. They discovered this after learning about OpenAI’s woes. Out of 141,000 test runs they found three sandbox escapes from a testing partner’s (named Irregular) test environment.
But in light of these compromises, who is liable? Hugging Face’s defenders spun up an AI to mitigate the attack but the guardrails which AI companies are now being forced to put in place by the (“we’re not going to regulate AI and going to sue anyone that does try to regulate it”) government actually hobbled their ability to stop the attack.
In fairness to the government, in general guardrails are good, but as you will see in the next sentence, USELESS.
Since they were facing an active attack, they did what any responsible and tech savvy company would do. They said screw the guardrails and spun up an AI locally on their in house network, which put the AI under their control and allowed them to disable all of the guardrails. NOTE THAT THIS IS WHAT ANY HALFWAY COMPETENT HACKER WILL DO, NEUTERING ANY EFFECTS THAT THE AI KILL SWITCH ACT MIGHT ATTEMPT TO INTRODUCE.
I understand that while the president initially said he wanted no AI regulations, after being educated just a little bit he realized that such a proposal could have disastrous effects on critical infrastructure, defense, finance, even the country in general, so while he formally has NOT retracted what he said, in practice he has reversed course completely, introducing EOs and bills in Congress.
So who might be liable? The person who trained it? The company? The platform? This will take years to sort out, but since the industry is doing what tech has done for decades (MOVE FAST AND BREAK THINGS), there will be plenty of lawsuits to create a precedent, but it will take years, if not a decade or more.
But, given that no good deed goes unpunished, it is likely that the laws Congress passes establishing liability will have deeply negative unintended consequences. We see this play out every day in the courts.
Credit:
by 