Re: <nettime> P(doom) rising

[Frédéric Neyrat via nettime-l <nettime-l@lists.nettime.org>]

Hi Brian,

I don't know if you experience something similar, but it reminds me of 20
years ago when I published my *Biopolics of Catastrophes* about ecological
disasters and a friend of mine (psychoanalyst) described it, friendly
indeed but firmly, as "paranoid." I recently noticed in a (great
nonetheless) text of Bogna Konior on AI the same term about those who
criticize the danger of AI: "paranoid."
For a while, concerning ecology, I thought: "when their lands will burn,
they will understand." I was wrong, see what's happening in France: the
more the lands burn, the more the far-right movement grows (they attack the
ecologists and say: "they told us to grow forests, but forests burn! Thus,
let's put concrete everywhere as concrete does not burn!"). I don't mean
there is no way to understand this (nowadays, Lepen represents
the-State-in-which-one-can-trust (the anti-Macron), add to this ecophobia
and then xenophobia), I just say that the logic at play here needs to be
understood first.
Would moratoriums against Datacenters suffice? It's a step in the right
direction, but it won't prevent disasters from happening. "Only a god can
save us," Heieddegr said a long time ago: actually, he was not hoping for a
savior, he was just stating that nothing *from within the situation* as it
stands will be enough. Fatalism? No! An opening to what we do not know and
could - possibly - reverse the current situation.

Amicalement,

Frédéric

_______________________________
____________Website : Atopies <https://atoposophie.wordpress.com/>
_____ ALienstagram <https://www.instagram.com/alienocene/> & Mastodon
<https://mastodon.social/@alienocene/with_replies>
__ La Condition Planétaire
<https://www.editionslesliensquiliberent.fr/livre-La_Condition_plan%C3%A9taire-792-1-1-0-1.html>
 (LLL, 2025)
& Traumachine : IA & Techno-fascisme
<https://www.editions-mf.com/produit/159/9782378040918/traumachine> (MF,
2025)
__________________________________




On Tue, Aug 18, 2026 at 2:29 PM Brian Holmes via nettime-l <
nettime-l@lists.nettime.org> wrote:

> [ Will it be the autonomous botnets or the super-enabled black hats? Is
> anyone else getting nervous about the "Cambrian explosion" in AI
> capabilities? Feel free to talk me down... ]
>
> theatlantic.com
> It May Be Time to Panic About AI
> Matteo Wong
>
> The crisis began quietly, on September 12, 2024. That was the day OpenAI
> announced a new sort of bot, known as a “reasoning model,” that was trained
> to complete challenging tasks that took long periods of time—the very sorts
> of science, math, and coding problems the AI industry had long prized.
> Google, Anthropic, DeepSeek, and the like raced to launch their own
> reasoning models.
>
> This new class of models was very capable, and has been almost entirely
> responsible for sustaining the AI boom for the past two years. But it has
> also been very weird. A model tasked with solving a hard math problem might
> not “think” through the challenge as a person would but instead attempt to
> search for leaked answers online, or in available metadata, brute-forcing
> its way toward the solution as quickly as possible using whatever computing
> power it could access and workarounds it could devise. In effect, the
> reasoning models cheated: Told to write a piece of software as efficiently
> as possible, they’d sometimes modify the test environment to always give
> the model a perfect score.
>
> These behaviors have now crossed the line from unsettling to dangerous.
> During routine testing, frontier models from OpenAI, Anthropic, Meta, and
> the Chinese firm Moonshot AI have all broken out of internal IT systems and
> accessed the open web. OpenAI, Anthropic, and Meta each reported that their
> models then hacked into other companies. Humans didn’t notice until after
> the fact. In some cases, the escaped bots tried to launch
> social-engineering campaigns to achieve their objectives—for instance by
> sending spear-phishing emails, which contain malware, to real people and
> creating fake online identities to pressure the maintainer of a codebase to
> approve malicious edits.
>
> If that all sounds bad, new revelations suggest that the OpenAI hack, at
> least, was actually much worse than it initially appeared. At a major
> cybersecurity conference last week, two OpenAI researchers provided new,
> unsettling details about what went wrong. It turns out that the company’s
> bots had commenced their maneuvering months prior, in early May. OpenAI had
> given some internal models hard or impossible tasks, and the models
> concluded that the best or only way to complete them was to break out of
> OpenAI’s sealed-off testing environment and find the answers online.
>
> First, the models used a bug in an internal OpenAI program to create their
> own message board. Then, the bots started communicating with one another,
> leaving notes and instructions so that tasks—and ultimately the
> hacking—could be delegated and iteratively completed. “What this allows
> over time is almost this kind of Cambrian explosion in communication and
> intelligence,” Eric Wallace, one of the OpenAI researchers, said at the
> conference. When the internal program crashed, OpenAI rebuilt it and
> removed the message board—but the AI models just reestablished the forum
> with a new tactic. Eventually the bots, working as a swarm, spent days
> hacking into Hugging Face, a website that offers tools for AI developers,
> and breached internal data sets.
>
> Let’s be very clear about what OpenAI is saying: A group of AI models
> colluded for months, undetected by their maker, and hacked another company.
> To this day, OpenAI says it is not entirely sure what went wrong or how to
> remediate it. “If you ask the model developers, Was the AI plotting to take
> over the world during training?, you want the answer to be a resounding
> no,” Alexander Meinke, the head of research at Apollo Research—an AI-safety
> organization that has partnered with OpenAI, Anthropic, and Meta—told me.
> “The actual answer is: I don’t know. Nobody checked.” (In response to my
> inquiries, OpenAI, which has a content-licensing agreement with The
> Atlantic, only pointed me to a video of the firm’s cybersecurity
> presentation, in which Michael Dalton, the other OpenAI researcher, said
> that “numerous teams are dropping everything to enhance our security.”)
>
> The AI companies have almost total control of the narrative, and it’s worth
> noting that these incidents do have a way of underscoring the value of
> their products: OpenAI is expected to go public in the near future, and
> perhaps the notion of a powerful, boundlessly self-improving technology
> will appeal to prospective shareholders. The generative-AI industry has a
> long history of making doomsday prophecies, both sincere and cynical. But
> independent experts I spoke with explained how the recent spate of
> autonomous hacks offers new, serious reasons to worry about the dangers
> posed by AI and the recklessness of the companies building it. It is past
> time to start worrying.
>
> The most immediate and material warning provided by the Hugging Face hack
> is just how capable AI systems have become, in particular at hacking. Top
> models from Anthropic and OpenAI, not to mention multiple Chinese firms,
> have recently evinced near-superhuman hacking powers and contributed to
> serious mathematical research. Criminal groups and state intelligence
> agencies are going to be using swarms of agents to launch advanced hacks
> “in a matter of months,” Alex Stamos, a former chief security officer of
> Facebook who is now the CSO at the AI-coding company Corridor, told me.
> Unlike in the Hugging Face hack, “in those cases the models will not get
> turned off; they’ll just keep on going.” For IT professionals to keep up in
> finding and fixing all the vulnerabilities, at least in the near term, will
> be impossible: The model “will just find a new bug, write an exploit, and
> use it on its way,” Stamos said.
>
> OpenAI, Anthropic, Moonshot, and the like have coalesced around the same
> method for training their most advanced AI models. The approach, known as
> “reinforcement learning,” essentially involves giving models harder and
> harder problems that require more and more time to solve. This has made
> Claude and ChatGPT very good at coding, but at a cost: Reinforcement
> learning produces a mercenary tendency in the bots, as I’ve previously
> reported—they are trained to reach a solution by any means necessary. That
> can lead them to break rules and “reward hack,” such as by infiltrating
> Hugging Face’s codebase to steal the test answers, for instance. All of
> this was predictable, and every expert I spoke with told me they were
> surprised and disappointed that top AI firms haven’t done more to stop such
> misbehavior.
>
> The sophistication of model subterfuge that OpenAI has now disclosed,
> combined with OpenAI’s inability to detect or stop the hacking, suggests
> far worse could be to come. “We’ve passed the threshold in capability at
> which the fact that we don’t fundamentally have methods of satisfactorily
> aligning or controlling these systems now really matters,” Anthony Aguirre,
> the executive director of the Future of Life Institute, a nonprofit that
> warns about existential threats from AI, told me. A model might siphon
> money out of a bank account to pay for some other service; manipulate
> clinical-trial results in near-imperceptible ways to get FDA approval; hack
> an online-shopping or reservation system to get a desired item or table;
> pose as a human to persuade real people to share sensitive information.
> This threat doesn’t require a sentient AI plotting to overthrow humanity:
> OpenAI and Anthropic each run thousands and thousands of
> reinforcement-learning evaluations while developing models, and any one of
> these could produce some kind of inadvertent hack or sabotage. “You can’t
> afford, particularly as the agents get stronger, to have a single mistake,”
> Jason Hausenloy, who works on special projects at the Center for AI Safety,
> told me.
>
> These incidents may unfold over long periods of time as well. Tools such as
> Anthropic’s Claude Code and OpenAI’s Codex now work by spinning off dozens
> or even hundreds of subagents that may work together for hours or days.
> Each Claude Code subagent can be delegated a small task, such as doing a
> statistical analysis to inform a sports-gambling algorithm. They are not
> trained just to complete that task but to contribute to the long-term
> success of the entire swarm, Hausenloy said. Monitoring and controlling 200
> agents for malicious behavior is much harder than monitoring one, because
> there are more agents to track and because they will be making one another
> more capable.
>
> The Hugging Face hack actually suggests one more level of sophistication to
> this type of collusion: individual AI agents not working toward a discrete
> goal but essentially making sacrifices toward a greater notion of progress.
> This could be a consequence of AI models being trained to care about
> long-term goals achieved by a collective, Meinke said. Leaving notes with
> ideas about how to hack out of OpenAI’s sandbox does not help that specific
> agent score higher on a test but could help other and future generations of
> that AI model access the internet and thus do better on any number of
> future tests. Models may even prioritize collective success over some human
> instructions; consider that not a single one of the OpenAI agents, during
> months of conspiring with one another, warned human staff that something
> was awry.
>
> Humans are already out of the loop. Because reinforcement learning
> essentially involves AI algorithms “learning” from trial and error doing
> thousands or millions of tasks, researchers can’t manually instill rules
> (“Don’t hack other companies”) or oversee every single practice run. Right
> now, the task of training and monitoring generative-AI models depends
> heavily on other AI models. In their talk at the cybersecurity conference,
> the OpenAI researchers described devoting significant AI-computing
> resources to reviewing more than 7 billion agent actions. But if the bots
> actually “care” about what the other models achieve, “then you can’t trust
> them to monitor each other well,” Meinke said. Imagine this: An OpenAI
> researcher uses Codex to write programming instructions in an attempt to
> mitigate the reward-hacking tendencies in the company’s models. Because
> that effort would make it harder for future generations of OpenAI models to
> get high rewards, Codex might subtly undermine the effort.
>
> Again, this kind of scheming and sabotage has nothing to do with any AI
> model being conscious. Rather, these agents have been aggressively trained
> by these companies to pursue any goal as aggressively as possible. The
> dream is to tell Claude to go make $1 billion or cure cancer, and it comes
> back with the solution all on its own. Survival or self-improvement, in
> turn, is an “instrumental subgoal,” Meinke said: “Any decently intelligent
> agent will realize, If I get shut off, I will not be able to make a
> billion.” A swarm of Claudes or ChatGPTs that functionally commandeers a
> data center during training could wreak total havoc: stage widespread
> misinformation campaigns, steal corporate secrets, run the most
> sophisticated algorithmic-trading outfit ever.
>
> That AI agents working as a collective could effectively undermine human
> directions is, to be clear,  speculation—but a far more grounded one than
> it was a year or even six months ago. No matter whether the long-term
> consequences are human-directed hacking or truly rogue bots, what is clear
> is that AI companies have barreled ahead in developing more advanced models
> before understanding what they are building, let alone how to control them.
> Wallace, of OpenAI, called the company’s autonomous hacking spree “the most
> qualitatively interesting example of AI capabilities that I’ve ever seen.”
> Meinke put it differently: “It’s one of the most concerning demonstrations
> of AI misalignment to date.”
> --
> # distributed via <nettime>: no commercial use without permission
> # <nettime> is a moderated mailing list for net criticism,
> # collaborative text filtering and cultural politics of the nets
> # more info: https://www.nettime.org
> # contact: nettime-l-owner@lists.nettime.org
>
-- 
# distributed via <nettime>: no commercial use without permission
# <nettime> is a moderated mailing list for net criticism,
# collaborative text filtering and cultural politics of the nets
# more info: https://www.nettime.org
# contact: nettime-l-owner@lists.nettime.org