Re: <nettime> P(doom) rising

[Brian Holmes via nettime-l <nettime-l@lists.nettime.org>]

Salut Frédéric,

Oh, yes, they also called me paranoid. So much for the short-sighted!

You are generous in your interpretation of Heidegger, but the idea that the
Earth itself is a cosmos, a watery star, this I believe and feel in my skin:

https://technophany.philosophyandtechnology.network/article/view/27164/28209

Concerning the moratorium on data centers, it's all our Chicago mayor,
Brandon Johnson, could offer to the people assembled by the Black advocacy
group Southside Together, last Saturday, just a stone's throw and a world
away from the future Quantum Park that's already stirring up toxic dust
from industrial disasters past.

He said he'd do "anything the law allows." I never saw a politician kicked
out of a popular assembly so fast!

If the Earth really is a cosmos, then maybe I don't need philosophy to find
that place outside the situation. Maybe collective resistance is enough?
Maybe the deep inside is way outside the situation?

Be paranoid, be well,

Brian

On Tue, Aug 18, 2026 at 3:06 PM Frédéric Neyrat <fneyrat@gmail.com> wrote:

> Hi Brian,
>
> I don't know if you experience something similar, but it reminds me of 20
> years ago when I published my *Biopolics of Catastrophes* about
> ecological disasters and a friend of mine (psychoanalyst) described it,
> friendly indeed but firmly, as "paranoid." I recently noticed in a (great
> nonetheless) text of Bogna Konior on AI the same term about those who
> criticize the danger of AI: "paranoid."
> For a while, concerning ecology, I thought: "when their lands will burn,
> they will understand." I was wrong, see what's happening in France: the
> more the lands burn, the more the far-right movement grows (they attack the
> ecologists and say: "they told us to grow forests, but forests burn! Thus,
> let's put concrete everywhere as concrete does not burn!"). I don't mean
> there is no way to understand this (nowadays, Lepen represents
> the-State-in-which-one-can-trust (the anti-Macron), add to this ecophobia
> and then xenophobia), I just say that the logic at play here needs to be
> understood first.
> Would moratoriums against Datacenters suffice? It's a step in the right
> direction, but it won't prevent disasters from happening. "Only a god can
> save us," Heieddegr said a long time ago: actually, he was not hoping for a
> savior, he was just stating that nothing *from within the situation* as
> it stands will be enough. Fatalism? No! An opening to what we do not know
> and could - possibly - reverse the current situation.
>
> Amicalement,
>
> Frédéric
>
> _______________________________
> ____________Website : Atopies <https://atoposophie.wordpress.com/>
> _____ ALienstagram <https://www.instagram.com/alienocene/> & Mastodon
> <https://mastodon.social/@alienocene/with_replies>
> __ La Condition Planétaire
> <https://www.editionslesliensquiliberent.fr/livre-La_Condition_plan%C3%A9taire-792-1-1-0-1.html>
>  (LLL, 2025)
> & Traumachine : IA & Techno-fascisme
> <https://www.editions-mf.com/produit/159/9782378040918/traumachine> (MF,
> 2025)
> __________________________________
>
>
>
>
> On Tue, Aug 18, 2026 at 2:29 PM Brian Holmes via nettime-l <
> nettime-l@lists.nettime.org> wrote:
>
>> [ Will it be the autonomous botnets or the super-enabled black hats? Is
>> anyone else getting nervous about the "Cambrian explosion" in AI
>> capabilities? Feel free to talk me down... ]
>>
>> theatlantic.com
>> It May Be Time to Panic About AI
>> Matteo Wong
>>
>> The crisis began quietly, on September 12, 2024. That was the day OpenAI
>> announced a new sort of bot, known as a “reasoning model,” that was
>> trained
>> to complete challenging tasks that took long periods of time—the very
>> sorts
>> of science, math, and coding problems the AI industry had long prized.
>> Google, Anthropic, DeepSeek, and the like raced to launch their own
>> reasoning models.
>>
>> This new class of models was very capable, and has been almost entirely
>> responsible for sustaining the AI boom for the past two years. But it has
>> also been very weird. A model tasked with solving a hard math problem
>> might
>> not “think” through the challenge as a person would but instead attempt to
>> search for leaked answers online, or in available metadata, brute-forcing
>> its way toward the solution as quickly as possible using whatever
>> computing
>> power it could access and workarounds it could devise. In effect, the
>> reasoning models cheated: Told to write a piece of software as efficiently
>> as possible, they’d sometimes modify the test environment to always give
>> the model a perfect score.
>>
>> These behaviors have now crossed the line from unsettling to dangerous.
>> During routine testing, frontier models from OpenAI, Anthropic, Meta, and
>> the Chinese firm Moonshot AI have all broken out of internal IT systems
>> and
>> accessed the open web. OpenAI, Anthropic, and Meta each reported that
>> their
>> models then hacked into other companies. Humans didn’t notice until after
>> the fact. In some cases, the escaped bots tried to launch
>> social-engineering campaigns to achieve their objectives—for instance by
>> sending spear-phishing emails, which contain malware, to real people and
>> creating fake online identities to pressure the maintainer of a codebase
>> to
>> approve malicious edits.
>>
>> If that all sounds bad, new revelations suggest that the OpenAI hack, at
>> least, was actually much worse than it initially appeared. At a major
>> cybersecurity conference last week, two OpenAI researchers provided new,
>> unsettling details about what went wrong. It turns out that the company’s
>> bots had commenced their maneuvering months prior, in early May. OpenAI
>> had
>> given some internal models hard or impossible tasks, and the models
>> concluded that the best or only way to complete them was to break out of
>> OpenAI’s sealed-off testing environment and find the answers online.
>>
>> First, the models used a bug in an internal OpenAI program to create their
>> own message board. Then, the bots started communicating with one another,
>> leaving notes and instructions so that tasks—and ultimately the
>> hacking—could be delegated and iteratively completed. “What this allows
>> over time is almost this kind of Cambrian explosion in communication and
>> intelligence,” Eric Wallace, one of the OpenAI researchers, said at the
>> conference. When the internal program crashed, OpenAI rebuilt it and
>> removed the message board—but the AI models just reestablished the forum
>> with a new tactic. Eventually the bots, working as a swarm, spent days
>> hacking into Hugging Face, a website that offers tools for AI developers,
>> and breached internal data sets.
>>
>> Let’s be very clear about what OpenAI is saying: A group of AI models
>> colluded for months, undetected by their maker, and hacked another
>> company.
>> To this day, OpenAI says it is not entirely sure what went wrong or how to
>> remediate it. “If you ask the model developers, Was the AI plotting to
>> take
>> over the world during training?, you want the answer to be a resounding
>> no,” Alexander Meinke, the head of research at Apollo Research—an
>> AI-safety
>> organization that has partnered with OpenAI, Anthropic, and Meta—told me.
>> “The actual answer is: I don’t know. Nobody checked.” (In response to my
>> inquiries, OpenAI, which has a content-licensing agreement with The
>> Atlantic, only pointed me to a video of the firm’s cybersecurity
>> presentation, in which Michael Dalton, the other OpenAI researcher, said
>> that “numerous teams are dropping everything to enhance our security.”)
>>
>> The AI companies have almost total control of the narrative, and it’s
>> worth
>> noting that these incidents do have a way of underscoring the value of
>> their products: OpenAI is expected to go public in the near future, and
>> perhaps the notion of a powerful, boundlessly self-improving technology
>> will appeal to prospective shareholders. The generative-AI industry has a
>> long history of making doomsday prophecies, both sincere and cynical. But
>> independent experts I spoke with explained how the recent spate of
>> autonomous hacks offers new, serious reasons to worry about the dangers
>> posed by AI and the recklessness of the companies building it. It is past
>> time to start worrying.
>>
>> The most immediate and material warning provided by the Hugging Face hack
>> is just how capable AI systems have become, in particular at hacking. Top
>> models from Anthropic and OpenAI, not to mention multiple Chinese firms,
>> have recently evinced near-superhuman hacking powers and contributed to
>> serious mathematical research. Criminal groups and state intelligence
>> agencies are going to be using swarms of agents to launch advanced hacks
>> “in a matter of months,” Alex Stamos, a former chief security officer of
>> Facebook who is now the CSO at the AI-coding company Corridor, told me.
>> Unlike in the Hugging Face hack, “in those cases the models will not get
>> turned off; they’ll just keep on going.” For IT professionals to keep up
>> in
>> finding and fixing all the vulnerabilities, at least in the near term,
>> will
>> be impossible: The model “will just find a new bug, write an exploit, and
>> use it on its way,” Stamos said.
>>
>> OpenAI, Anthropic, Moonshot, and the like have coalesced around the same
>> method for training their most advanced AI models. The approach, known as
>> “reinforcement learning,” essentially involves giving models harder and
>> harder problems that require more and more time to solve. This has made
>> Claude and ChatGPT very good at coding, but at a cost: Reinforcement
>> learning produces a mercenary tendency in the bots, as I’ve previously
>> reported—they are trained to reach a solution by any means necessary. That
>> can lead them to break rules and “reward hack,” such as by infiltrating
>> Hugging Face’s codebase to steal the test answers, for instance. All of
>> this was predictable, and every expert I spoke with told me they were
>> surprised and disappointed that top AI firms haven’t done more to stop
>> such
>> misbehavior.
>>
>> The sophistication of model subterfuge that OpenAI has now disclosed,
>> combined with OpenAI’s inability to detect or stop the hacking, suggests
>> far worse could be to come. “We’ve passed the threshold in capability at
>> which the fact that we don’t fundamentally have methods of satisfactorily
>> aligning or controlling these systems now really matters,” Anthony
>> Aguirre,
>> the executive director of the Future of Life Institute, a nonprofit that
>> warns about existential threats from AI, told me. A model might siphon
>> money out of a bank account to pay for some other service; manipulate
>> clinical-trial results in near-imperceptible ways to get FDA approval;
>> hack
>> an online-shopping or reservation system to get a desired item or table;
>> pose as a human to persuade real people to share sensitive information.
>> This threat doesn’t require a sentient AI plotting to overthrow humanity:
>> OpenAI and Anthropic each run thousands and thousands of
>> reinforcement-learning evaluations while developing models, and any one of
>> these could produce some kind of inadvertent hack or sabotage. “You can’t
>> afford, particularly as the agents get stronger, to have a single
>> mistake,”
>> Jason Hausenloy, who works on special projects at the Center for AI
>> Safety,
>> told me.
>>
>> These incidents may unfold over long periods of time as well. Tools such
>> as
>> Anthropic’s Claude Code and OpenAI’s Codex now work by spinning off dozens
>> or even hundreds of subagents that may work together for hours or days.
>> Each Claude Code subagent can be delegated a small task, such as doing a
>> statistical analysis to inform a sports-gambling algorithm. They are not
>> trained just to complete that task but to contribute to the long-term
>> success of the entire swarm, Hausenloy said. Monitoring and controlling
>> 200
>> agents for malicious behavior is much harder than monitoring one, because
>> there are more agents to track and because they will be making one another
>> more capable.
>>
>> The Hugging Face hack actually suggests one more level of sophistication
>> to
>> this type of collusion: individual AI agents not working toward a discrete
>> goal but essentially making sacrifices toward a greater notion of
>> progress.
>> This could be a consequence of AI models being trained to care about
>> long-term goals achieved by a collective, Meinke said. Leaving notes with
>> ideas about how to hack out of OpenAI’s sandbox does not help that
>> specific
>> agent score higher on a test but could help other and future generations
>> of
>> that AI model access the internet and thus do better on any number of
>> future tests. Models may even prioritize collective success over some
>> human
>> instructions; consider that not a single one of the OpenAI agents, during
>> months of conspiring with one another, warned human staff that something
>> was awry.
>>
>> Humans are already out of the loop. Because reinforcement learning
>> essentially involves AI algorithms “learning” from trial and error doing
>> thousands or millions of tasks, researchers can’t manually instill rules
>> (“Don’t hack other companies”) or oversee every single practice run. Right
>> now, the task of training and monitoring generative-AI models depends
>> heavily on other AI models. In their talk at the cybersecurity conference,
>> the OpenAI researchers described devoting significant AI-computing
>> resources to reviewing more than 7 billion agent actions. But if the bots
>> actually “care” about what the other models achieve, “then you can’t trust
>> them to monitor each other well,” Meinke said. Imagine this: An OpenAI
>> researcher uses Codex to write programming instructions in an attempt to
>> mitigate the reward-hacking tendencies in the company’s models. Because
>> that effort would make it harder for future generations of OpenAI models
>> to
>> get high rewards, Codex might subtly undermine the effort.
>>
>> Again, this kind of scheming and sabotage has nothing to do with any AI
>> model being conscious. Rather, these agents have been aggressively trained
>> by these companies to pursue any goal as aggressively as possible. The
>> dream is to tell Claude to go make $1 billion or cure cancer, and it comes
>> back with the solution all on its own. Survival or self-improvement, in
>> turn, is an “instrumental subgoal,” Meinke said: “Any decently intelligent
>> agent will realize, If I get shut off, I will not be able to make a
>> billion.” A swarm of Claudes or ChatGPTs that functionally commandeers a
>> data center during training could wreak total havoc: stage widespread
>> misinformation campaigns, steal corporate secrets, run the most
>> sophisticated algorithmic-trading outfit ever.
>>
>> That AI agents working as a collective could effectively undermine human
>> directions is, to be clear,  speculation—but a far more grounded one than
>> it was a year or even six months ago. No matter whether the long-term
>> consequences are human-directed hacking or truly rogue bots, what is clear
>> is that AI companies have barreled ahead in developing more advanced
>> models
>> before understanding what they are building, let alone how to control
>> them.
>> Wallace, of OpenAI, called the company’s autonomous hacking spree “the
>> most
>> qualitatively interesting example of AI capabilities that I’ve ever seen.”
>> Meinke put it differently: “It’s one of the most concerning demonstrations
>> of AI misalignment to date.”
>> --
>> # distributed via <nettime>: no commercial use without permission
>> # <nettime> is a moderated mailing list for net criticism,
>> # collaborative text filtering and cultural politics of the nets
>> # more info: https://www.nettime.org
>> # contact: nettime-l-owner@lists.nettime.org
>>
>
-- 
# distributed via <nettime>: no commercial use without permission
# <nettime> is a moderated mailing list for net criticism,
# collaborative text filtering and cultural politics of the nets
# more info: https://www.nettime.org
# contact: nettime-l-owner@lists.nettime.org