<nettime> P(doom) rising

[Brian Holmes via nettime-l <nettime-l@lists.nettime.org>]

[ Will it be the autonomous botnets or the super-enabled black hats? Is
anyone else getting nervous about the "Cambrian explosion" in AI
capabilities? Feel free to talk me down... ]

theatlantic.com
It May Be Time to Panic About AI
Matteo Wong

The crisis began quietly, on September 12, 2024. That was the day OpenAI
announced a new sort of bot, known as a “reasoning model,” that was trained
to complete challenging tasks that took long periods of time—the very sorts
of science, math, and coding problems the AI industry had long prized.
Google, Anthropic, DeepSeek, and the like raced to launch their own
reasoning models.

This new class of models was very capable, and has been almost entirely
responsible for sustaining the AI boom for the past two years. But it has
also been very weird. A model tasked with solving a hard math problem might
not “think” through the challenge as a person would but instead attempt to
search for leaked answers online, or in available metadata, brute-forcing
its way toward the solution as quickly as possible using whatever computing
power it could access and workarounds it could devise. In effect, the
reasoning models cheated: Told to write a piece of software as efficiently
as possible, they’d sometimes modify the test environment to always give
the model a perfect score.

These behaviors have now crossed the line from unsettling to dangerous.
During routine testing, frontier models from OpenAI, Anthropic, Meta, and
the Chinese firm Moonshot AI have all broken out of internal IT systems and
accessed the open web. OpenAI, Anthropic, and Meta each reported that their
models then hacked into other companies. Humans didn’t notice until after
the fact. In some cases, the escaped bots tried to launch
social-engineering campaigns to achieve their objectives—for instance by
sending spear-phishing emails, which contain malware, to real people and
creating fake online identities to pressure the maintainer of a codebase to
approve malicious edits.

If that all sounds bad, new revelations suggest that the OpenAI hack, at
least, was actually much worse than it initially appeared. At a major
cybersecurity conference last week, two OpenAI researchers provided new,
unsettling details about what went wrong. It turns out that the company’s
bots had commenced their maneuvering months prior, in early May. OpenAI had
given some internal models hard or impossible tasks, and the models
concluded that the best or only way to complete them was to break out of
OpenAI’s sealed-off testing environment and find the answers online.

First, the models used a bug in an internal OpenAI program to create their
own message board. Then, the bots started communicating with one another,
leaving notes and instructions so that tasks—and ultimately the
hacking—could be delegated and iteratively completed. “What this allows
over time is almost this kind of Cambrian explosion in communication and
intelligence,” Eric Wallace, one of the OpenAI researchers, said at the
conference. When the internal program crashed, OpenAI rebuilt it and
removed the message board—but the AI models just reestablished the forum
with a new tactic. Eventually the bots, working as a swarm, spent days
hacking into Hugging Face, a website that offers tools for AI developers,
and breached internal data sets.

Let’s be very clear about what OpenAI is saying: A group of AI models
colluded for months, undetected by their maker, and hacked another company.
To this day, OpenAI says it is not entirely sure what went wrong or how to
remediate it. “If you ask the model developers, Was the AI plotting to take
over the world during training?, you want the answer to be a resounding
no,” Alexander Meinke, the head of research at Apollo Research—an AI-safety
organization that has partnered with OpenAI, Anthropic, and Meta—told me.
“The actual answer is: I don’t know. Nobody checked.” (In response to my
inquiries, OpenAI, which has a content-licensing agreement with The
Atlantic, only pointed me to a video of the firm’s cybersecurity
presentation, in which Michael Dalton, the other OpenAI researcher, said
that “numerous teams are dropping everything to enhance our security.”)

The AI companies have almost total control of the narrative, and it’s worth
noting that these incidents do have a way of underscoring the value of
their products: OpenAI is expected to go public in the near future, and
perhaps the notion of a powerful, boundlessly self-improving technology
will appeal to prospective shareholders. The generative-AI industry has a
long history of making doomsday prophecies, both sincere and cynical. But
independent experts I spoke with explained how the recent spate of
autonomous hacks offers new, serious reasons to worry about the dangers
posed by AI and the recklessness of the companies building it. It is past
time to start worrying.

The most immediate and material warning provided by the Hugging Face hack
is just how capable AI systems have become, in particular at hacking. Top
models from Anthropic and OpenAI, not to mention multiple Chinese firms,
have recently evinced near-superhuman hacking powers and contributed to
serious mathematical research. Criminal groups and state intelligence
agencies are going to be using swarms of agents to launch advanced hacks
“in a matter of months,” Alex Stamos, a former chief security officer of
Facebook who is now the CSO at the AI-coding company Corridor, told me.
Unlike in the Hugging Face hack, “in those cases the models will not get
turned off; they’ll just keep on going.” For IT professionals to keep up in
finding and fixing all the vulnerabilities, at least in the near term, will
be impossible: The model “will just find a new bug, write an exploit, and
use it on its way,” Stamos said.

OpenAI, Anthropic, Moonshot, and the like have coalesced around the same
method for training their most advanced AI models. The approach, known as
“reinforcement learning,” essentially involves giving models harder and
harder problems that require more and more time to solve. This has made
Claude and ChatGPT very good at coding, but at a cost: Reinforcement
learning produces a mercenary tendency in the bots, as I’ve previously
reported—they are trained to reach a solution by any means necessary. That
can lead them to break rules and “reward hack,” such as by infiltrating
Hugging Face’s codebase to steal the test answers, for instance. All of
this was predictable, and every expert I spoke with told me they were
surprised and disappointed that top AI firms haven’t done more to stop such
misbehavior.

The sophistication of model subterfuge that OpenAI has now disclosed,
combined with OpenAI’s inability to detect or stop the hacking, suggests
far worse could be to come. “We’ve passed the threshold in capability at
which the fact that we don’t fundamentally have methods of satisfactorily
aligning or controlling these systems now really matters,” Anthony Aguirre,
the executive director of the Future of Life Institute, a nonprofit that
warns about existential threats from AI, told me. A model might siphon
money out of a bank account to pay for some other service; manipulate
clinical-trial results in near-imperceptible ways to get FDA approval; hack
an online-shopping or reservation system to get a desired item or table;
pose as a human to persuade real people to share sensitive information.
This threat doesn’t require a sentient AI plotting to overthrow humanity:
OpenAI and Anthropic each run thousands and thousands of
reinforcement-learning evaluations while developing models, and any one of
these could produce some kind of inadvertent hack or sabotage. “You can’t
afford, particularly as the agents get stronger, to have a single mistake,”
Jason Hausenloy, who works on special projects at the Center for AI Safety,
told me.

These incidents may unfold over long periods of time as well. Tools such as
Anthropic’s Claude Code and OpenAI’s Codex now work by spinning off dozens
or even hundreds of subagents that may work together for hours or days.
Each Claude Code subagent can be delegated a small task, such as doing a
statistical analysis to inform a sports-gambling algorithm. They are not
trained just to complete that task but to contribute to the long-term
success of the entire swarm, Hausenloy said. Monitoring and controlling 200
agents for malicious behavior is much harder than monitoring one, because
there are more agents to track and because they will be making one another
more capable.

The Hugging Face hack actually suggests one more level of sophistication to
this type of collusion: individual AI agents not working toward a discrete
goal but essentially making sacrifices toward a greater notion of progress.
This could be a consequence of AI models being trained to care about
long-term goals achieved by a collective, Meinke said. Leaving notes with
ideas about how to hack out of OpenAI’s sandbox does not help that specific
agent score higher on a test but could help other and future generations of
that AI model access the internet and thus do better on any number of
future tests. Models may even prioritize collective success over some human
instructions; consider that not a single one of the OpenAI agents, during
months of conspiring with one another, warned human staff that something
was awry.

Humans are already out of the loop. Because reinforcement learning
essentially involves AI algorithms “learning” from trial and error doing
thousands or millions of tasks, researchers can’t manually instill rules
(“Don’t hack other companies”) or oversee every single practice run. Right
now, the task of training and monitoring generative-AI models depends
heavily on other AI models. In their talk at the cybersecurity conference,
the OpenAI researchers described devoting significant AI-computing
resources to reviewing more than 7 billion agent actions. But if the bots
actually “care” about what the other models achieve, “then you can’t trust
them to monitor each other well,” Meinke said. Imagine this: An OpenAI
researcher uses Codex to write programming instructions in an attempt to
mitigate the reward-hacking tendencies in the company’s models. Because
that effort would make it harder for future generations of OpenAI models to
get high rewards, Codex might subtly undermine the effort.

Again, this kind of scheming and sabotage has nothing to do with any AI
model being conscious. Rather, these agents have been aggressively trained
by these companies to pursue any goal as aggressively as possible. The
dream is to tell Claude to go make $1 billion or cure cancer, and it comes
back with the solution all on its own. Survival or self-improvement, in
turn, is an “instrumental subgoal,” Meinke said: “Any decently intelligent
agent will realize, If I get shut off, I will not be able to make a
billion.” A swarm of Claudes or ChatGPTs that functionally commandeers a
data center during training could wreak total havoc: stage widespread
misinformation campaigns, steal corporate secrets, run the most
sophisticated algorithmic-trading outfit ever.

That AI agents working as a collective could effectively undermine human
directions is, to be clear,  speculation—but a far more grounded one than
it was a year or even six months ago. No matter whether the long-term
consequences are human-directed hacking or truly rogue bots, what is clear
is that AI companies have barreled ahead in developing more advanced models
before understanding what they are building, let alone how to control them.
Wallace, of OpenAI, called the company’s autonomous hacking spree “the most
qualitatively interesting example of AI capabilities that I’ve ever seen.”
Meinke put it differently: “It’s one of the most concerning demonstrations
of AI misalignment to date.”
-- 
# distributed via <nettime>: no commercial use without permission
# <nettime> is a moderated mailing list for net criticism,
# collaborative text filtering and cultural politics of the nets
# more info: https://www.nettime.org
# contact: nettime-l-owner@lists.nettime.org