Autonomous AI Agents Are A ……
Dumpster fire?
Well that is not Gary’s term and we are all adults reading this (at least I think so). He calls it a shitshow.
A new study from Stanford, MIT, Carnegie Mellon, Nvidia and others, which examined over 800 autonomous agents from healthcare, finance and others, showed that 91 percent were vulnerable to subtle but dangerous attacks.
Here is an example. Somewhat contrived, but still valid:
The study also says that almost 90 percent of agents showed “drift” relative to their goals after about 30 steps in their process.
And 94 percent of agents with some form of memory augmentation were vulnerable to poisoning attacks.
This new paper confirms and amplifies what a group of AWS and Berkeley researchers reported a couple of months ago – reporting similar vulnerabilities.
Another example:
As the newer paper’s first author, Owen Sakawa put it to me in an email “The OpenClaw / Moltbook incident (Section 9) is the first real-world empirical validation of the agentic threat model at scale: 770,000 live agents simultaneously compromised via a single database exploit, each with privileged access to their owner’s machine, email, and files. It’s not hypothetical anymore.”
https://garymarcus.substack.com/p/breaking-autonomous-agents-are-a
So does this mean we need to put the AI agent genie back in the box? Even if we would like to, this is not possible.
What does it mean? It means that we need to do a much better job of putting limits around agents. An example of this is when an autonomous agent at Amazon China decided to delete Amazon China’s production environment, it went and found the API key that allowed it to do that (that is what I call initiative) and had the permissions to do that. They lost over 5 million orders as a result.
We need to get a much better handle on what data exists, who has access to it, how we create identities for agents and how we control those identities. AI governance.
There is enormous good that will come from AI, but to paraphrase the quote, with great AI comes great responsibility.
If you need assistance sorting this out, please contact us.
Credit: Gary Marcus
by 