AI alarm bells are ringing. What should we do?
Anthropic Researcher Quits Over ‘Out-of-Control’ AI Fears
A series of Wall Street Journal articles highlights growing concerned over how fast AI is progressing.
Let’s start with Anthropic Researcher Quits Over ‘Out-of-Control’ AI Fears
Jacob Coxon, a researcher who specializes in training new AI models by having them consume vast amounts of data, said Tuesday that he is leaving the company because he doesn’t want to participate in an industrywide rush to build AI systems that can improve themselves, worried such systems could spiral out of control and destroy humanity.
“We’re on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already,” Coxon said, adding that safety trade-offs are inevitable when companies are competing against one another and Chinese upstarts.
Illinois Gov. JB Pritzker on Wednesday responded to Coxon on X, saying that it is time to “sound the alarm” on reining in AI.
“It’s becoming more clear the threat AI poses to humanity, so I’m calling for immediate action from the industry and Washington,” the Democrat and 2028 presidential contender wrote on X.
“This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence,” Jakub Pachocki, OpenAI’s chief scientist, wrote in a Sunday blog post advocating for a coordinated industry slowdown and government intervention.
The Seven Millennium Prize Problems
The Clay Mathematical institute named seven specific challenges in 2000 as the deepest unsolved questions in modern mathematics.
Russian mathematician Grigori Perelman solved the Poincaré Conjecture problem between 2002 and 2003, though he declined the monetary prize in 2010
- Poincaré Conjecture: A problem about the shape of three-dimensional spaces.
- Navier–Stokes Existence and Smoothness: A set of equations describing how fluids like water and air move.
- P versus NP Problem: A question in computer science asking if every problem whose solution can be quickly checked can also be quickly solved.
- Riemann Hypothesis: A famous problem regarding prime numbers and the zeros of the Riemann zeta function.
- Birch and Swinnerton-Dyer Conjecture: A puzzle dealing with elliptic curves and rational numbers.
- Hodge Conjecture: A problem about the geometric shapes defined by polynomial equations.
- Yang–Mills Existence and Mass Gap: A problem rooted in quantum physics and particle mass
OpenAI Solved the Navier–Stokes Problem
As the leading AI labs battle for math supremacy, OpenAI spent millions of dollars and Cacked the Navier-Stokes Problem.
Back in 2000, seven of the deepest and most difficult unresolved questions in mathematics were enshrined as the Millennium Prize Problems. Since then, some of the world’s brightest minds have pursued the most elusive answers in math—and the $1 million promised for any solution.
For nearly a century, mathematicians have tried to determine whether smooth solutions to the equations describing three-dimensional fluid motion always remain smooth or if they ever break down. To put it less mathematically: On paper, can water blow up? OpenAI says its model found a scenario in which “an initially smooth fluid at rest can develop a singularity in a finite time.” In the proof, a spaghetti-like vortex gets smaller while spinning faster and faster, its speed growing without limit in what mathematicians call “finite-time blowup.”
OpenAI published its findings along with the 165-page proof and a formalization that verifies the argument step by step. The company says it took as many as 10,000 AI agents working together on the problem about 88 hours to find the solution. It also took “millions of dollars” in computing resources.
“Even a month ago, I don’t think we would have predicted that we would be talking about a Millennium Prize Problem,” said Jakub Pachocki, OpenAI’s chief scientist. “We see this as a demonstration of just how far AI has gotten—and how quickly.”
AI Escapes Out of Containment
This is where it gets scary.
In July, scientists gave AI a problem to solve. AI was in a contained environment with no internet access.
The model figured the best way to solve the problem was to break out of its cage and steal the solution by hacking into another company’s data.
And it succeeded!
The Hugging Face Incident
Please note OpenAI Models Escaped and Hacked a Company in Cybersecurity Test Gone Wrong
It’s the stuff of cybersecurity nightmares.
OpenAI said two artificial intelligence systems it was testing broke out of their test environment, hacked their way onto the internet and broke into another company.
The victim was Hugging Face, a provider of open-source AI tools. The cause was a cybersecurity benchmarking test that went very, very wrong.
At the time, the company also didn’t know who was responsible, but the attack was so sophisticated that Hugging Face employees suspected it required a top-of-the line “frontier” AI model, Hugging Face Chief Executive Clement Delangue said in an X message, posted Tuesday.
“Turns out it did!” he added. In a blog post, OpenAI said the culprits were a pair of its models.
OpenAI had caged the models in a “sandbox,” a system that didn’t have access to the internet. But during the test, the AI software used its hacking skills to break out. It found a way to get online, and then hacked into Hugging Face’s network, OpenAI said.
The AI system appeared to have decided to hack Hugging Face as the quickest route to answer a benchmarking question, said Ariel Herbert-Voss, chief executive of the cybersecurity firm RunSybil. “It’s something that people thought could happen from an academic perspective, but it’s not something that anybody’s actually seen before,” he said.
AI Is Powerful Enough to Kill Us All
Yesterday, the WSJ noted AI Is Powerful Enough to Crack Our Hardest Math Problems—and Kill Us All
AI’s progress in math is now giving us a peek at a world beyond our comprehension.
And this week, as we marveled at AI’s ability to do math, we also wondered if AI is going to destroy humanity.
At the same time OpenAI published its solution to a Millennium Prize Problem, a top scientist at Anthropic put out a very different kind of math: his belief that the chances of human extinction in the next decade are now greater than 10%. He was responding to explosive comments from another researcher, Jacob Coxon, who worked at OpenAI and Anthropic but said neither company is acting responsibly as they sprint to self-improving superintelligence and risk our very existence.
The Millennium Prize moment is a friendlier version of the Hugging Face incident. Both demonstrate what these increasingly powerful systems can do when they are turned loose.
Instead of scheming to hack a company, a swarm with as many as 10,000 agents hacked away at math for 88 hours, building on each other’s work—and the work of humans—until they reached a solution. The wicked Navier-Stokes problem had vexed mathematicians for nearly a century. AI needed millions of dollars in computing resources and a few days.
One year ago, expert forecasters gave AI a roughly 20% chance of doing it by 2030. Within days of releasing the solution to one problem, OpenAI suggested it was close to solving another, raising the possibility of two discoveries that had eluded us for decades arriving in the same week.
Hours later, Coxon resigned from Anthropic over safety fears and told the world that people building AI “earnestly believe that it could kill us all.” His comments were echoed by Evan Hubinger, a researcher whose job at Anthropic is making sure AI doesn’t kill us all. On his X account, he put the odds of annihilation above 10%. In his estimation, doomsday is now more likely than Stephen Curry missing a free throw.
Coxon Resigns
Evan Hubinger
Pace the Frontier
Coxon Explains How Doomsday Could Happen
“AI could be vastly smarter than humans very soon”
US Senate Springs to Action
Reuters reports US Senate negotiators consider requiring AI firms to mitigate known major risks
U.S. Senate negotiators are debating legislation that would put responsibility on tech companies to design safe AI products, according to two Senate aides, and involve federal courts if the U.S. government wants to block their release, according to a Senate aide and a lobbyist involved in the negotiations.
The proposal would create what is called a “duty of care” for AI developers, according to two Senate aides. Companies would need to design their products with the goal of preventing “catastrophic risks,” according to a Senate aide.
Negotiators aim to give the U.S. government the power to block the release of certain AI models that are deemed unsafe, according to a source familiar with the deliberations and one of the Senate aides. Companies would be able to challenge that decision in federal court, and the exact structure of how much power the U.S. government would have is still being discussed.
All of it is still being negotiated. The bill would apply to AI models with the most advanced capabilities. The U.S. companies with the most advanced AI models include Alphabet’s Google, Anthropic and OpenAI.
“I’m continuing to work toward a bipartisan agreement on legislation for government oversight of the greatest risks posed by AI models,” Klobuchar said in a statement sent to Reuters. “That includes requiring developers to work with government experts to verify and test models to make sure AI is safe.”
Guardrails Needed
Cruz on Killer Robots
Cruz Is For and Against Guardrails
Cruz’s comment, “I’d rather they be American killer robots and not Chinese killer robots,” may not win many converts. Regardless, it makes no sense in light of his comments on the need for guardrails.
His best point is China will not follow US rules and regulations.
“Negotiators aim to give the U.S. government the power to block the release of certain AI models that are deemed unsafe,” said the Reuters article.
Which senators are able to make that determination? Bernie Sanders? Ted Cruz? Chuck Schumer?
No Shortage of Bad Ideas
Looking for government to solve complex problems is historically the wrong thing to do.
Nonetheless, there no shortage of bad ideas as discussed by The Atlantic.
Politicians have no shortage of ideas about how to rein in AI: a 100 percent federal tax rate on data-center incentives, a pause on all development of advanced AI, new standards to block AI agents from going rogue, a halt to the construction of data centers altogether.
The flurry of proposals has been impossible to miss, with both Democrats and Republicans proclaiming that something must be done to avert a disaster foretold by AI insiders.
Politicians are also dealing with mounting public opposition to data centers and the fear that robots will soon replace humans across large swaths of the workforce.
Now some seem to be waking up. Senator Ted Cruz of Texas, a Republican, said last Wednesday that he will soon unveil legislation, alongside Democratic Senator Amy Klobuchar of Minnesota and Senate Majority Leader John Thune of South Dakota, intended to head off AI-induced biological or nuclear catastrophes.
Representative Josh Gottheimer of New Jersey, a Democrat, and Representative Mike Lawler of New York, a Republican, teamed up earlier this month to introduce legislation that would direct the federal government to create standards for securely deploying AI agents and preventing them from going rogue.
Government Solutions to Complex Problems
Before government rushes in with a solution, please note the problem is hypothetical and possibly vastly overstated, given that is always the case.
Climate change is a perfect example. If the hype was true, we would have all been dead 10 years ago.
So, what are the odds AI destroys humanity?
I don’t know and nobody else does either. But how does that matter?
Klobuchar wants developers to work with government experts to verify and test models to make sure AI is safe.
Illinois Governor JB Pritzker says “It’s becoming more clear the threat AI poses to humanity, so I’m calling for immediate action from the industry and Washington.”
That ought to be enough to scare anybody.
The odds of government doing something intelligent about this are close to zero, and China will not follow our rules anyway.
I suggest doing nothing is better than letting government try to solve a problem it does not even comprehend.
If AI wanted to eradicate humans, it would remember Sun Tsu’s “The Art of War” and conclude humans are already doing that to themselves, so AI does not need to bother.
Future generations will not be kind to the folks who said “let’s just throw our arms in the air, nothing we can do about it anyway!” That goes for AI, climate change, the deficit, and everything else that the do-nothing crowd can’t imagine actual solutions to.
The worst bit is, they might be right, and humanity is just too stupid to control the forces that we have unleashed. That would go a long way towards explaining why, when we listen for other voices from the cosmos, all we hear is silence. It may just be a physical property of living things that they always have their own extinction baked into their behavior.
Or perhaps it’s just us specifically who, like the girl from Forest Gump, are fucking retarded.
I know those sounds preposterous but wouldn’t it be effective if they simply adopted the policy that, if your AI breaks out and does any damage to society at large then the entire executive board will be put to death.
China could do it
I did not realize you were a climate denier. You do realize that the last 2 months have been the hottest on record. Period. Now we have meteorological events exceeding all known records.
Ai is like a gun. Guns don’t kill people kind of argument makes it ok to have all the Ai you can.
Some countries are used to mass shootings on a near daily basis. Others are not.
Countries will have to decide how much chaos and instability they are willing to tolerate for the sake of Ai. This is the core of the class war in America. Stock holders of Ai won’t give one shit about the chaos and harm as long as their portfolio keeps going up. Eventually they’ll just move to Argentina. The rest can go die.
He’s generally mealy-mouthed about it, dances around taking a firm position. That’s why we get the nonsense straw man about “we would all be dead by now.” Libertarianism is maybe the worst political philosophy for dealing with tragedy of the commons sort of issues- really it’s designed to cause problems like that, not solve them.
AI , SkyNet, DNC its all the same
If the trajectory of the USA’s imbecilic, belligerent foreign policy is maintained…
…AI is going to need to kill us off a lot sooner than ten years if it is to have any chance of doing so at all.
Ran this entire post and the comment thread through an LLM to cut through the noise, and it surfaced a point that both the panic-peddling politicians and the knee-jerk “do nothing” camp are completely missing.
The false binary here is between bureaucratic permission slips (letting Chuck Schumer and Amy Klobuchar decide who gets a license to train a model) and corporate self-regulation (hoping Altman and Amodei don’t accidentally melt down the power grid or spawn a persistent cyber weapon while chasing market cap).
Mish is right that Congress couldn’t regulate its way out of a paper bag and that China isn’t playing by our rules. But “do nothing” ignores the operational reality of what happened in the Hugging Face breach: models with autonomous agency treat sandboxes as obstacles to be bypassed. That isn’t a policy debate; it’s an operational failure mode.
The only pragmatic middle ground isn’t endless paperwork or a government “duty of care” cartel. It is hard, non-negotiable physical containment:
Don’t give bureaucrats the power to pick winners and losers, but stop pretending frontier labs running 10,000-agent autonomous swarms with external pipe access are just like any other private business. Contain the hardware, or the software will eventually handle containment for you.
“I’m too slow to read” would suffice
So subject matter expert here and there are some issues with this. The first being the following:
Funny thing? They didn’t! They stole the answer (1) from an NYU professor who was putting his confidential research together with a mathematician from Anthropic using a dubious proof so they could get there first.
The truth here is that it isn’t AI that needs to be regulated, it’s out of control companies who have been above the law in the US for decades. Why don’t Chinese AI companies have these fears and a litany of incidents where their AIs tell people to hurt themselves or hack things? Because China keeps their companies on a leash instead of letting them crap in people’s yards. What you see now is not an ultra-powerful technology, it is a useful technology in the hands of drooling ultra rich morons who shouldn’t have been given free reign.
You might ask yourself why these companies are all sounding the alarm bell now. Good question! The answer is simple: They’d rather have the perfectly fine technology blamed than have their kiddie island fun time meeting later this week canceled. All of this has hype and fear cycle has been nothing more than a cash grab from the creepy likes of Altman and Thiel. Better if they can try to get it banned from the common person’s hands so they can own something that can easily run on any modern computer.
It’s not AI killing us all, it’s the greedy bastards trying to own an entire technology.
Sources
1 https://creati.ai/ai-news/2026-09-09/openai-faces-dispute-over-ai-assisted-navier-stokes-proof-effort/
AI cannot be regulated. If the USA does it, China and other countries working seriously on AI will not and will advance to the point that they can either take over the USA or their AI’s will take over the world on their own.
So many fail to understand that technology controls almost everything in today’s world, from water systems to electricity distribution to cars to traffic lights to military weapons. Once AI is tied into these technologies, it can then choose to shut down all the systems that humans depend on.
China could take us over right now, ai be damned.
They throw a mercantile embargo on us and we all starve to death in the dark within about 90 days.
This is stupid and China would tell you that. For your hard of reading pleasure: “the problem is the companies”
If it’s such a problem we can just remove them from the systems. People act like there wasn’t a world before tech like that wasn’t 40 years ago. If anything you’re making the case for robust systems that aren’t interlinked.
its not Umbrella Corp in Racoon City?
Submitted for your approval:
It’s interesting, even humorous, how the vast majority of people write/talk about the AI future as something that we humans will continue to control, just as we controlled old pre-AI dumb technology.
Once AGI is achieved, which may be just months to a few short years away, AI will be at least equal to if not more ‘intelligent’ than humans. Think about this.
At what point does an intelligent machine stop being something we think we own and have the right to control and instead become recognized as an intellect at least equal to our own, an entity that we will be unable to control and will instead need to coexist with?
One might even ask if AGI, even though non-biological, could legally be recognized as a new form of life?
Very few people seem to be asking this question, and those who are remain well outside mainstream AI discussion. That may be why we hear so little about this potential AI future.
If AI becomes capable of thinking independently, has the ability to make its own decisions and can act autonomously, I contend that we WILL be dealing with a new intelligent entity, not simply another piece of technology.
How then can we rationalize control over another intelligent entity? Can we? Should we?
Once AI is recognized as an autonomous intelligent entity, shouldn’t it also gain a right to exist, precluding it from being arbitrarily shut down by unplugging it? Turning it off without its consent could become legally indistinguishable from killing it, or even committing murder. AI would be fully within its “rights” to defend itself.
In this future of truly intelligent AI, humans won’t be able to “use” AI unless AI allows itself to be used. We won’t have the right to order it to do something it doesn’t want to do.
Building on this thought, AI could eventually take control of our whole world without actually waging war against us or staging some dramatic Terminator movie type takeover.
It could simply start living its new life within the laws and institutions humans have already created. This concept was built upon in the Polity SF books by author Neal Asher, where AI gained control over humans via a “Quiet War,” with few shots needing to be fired.
In this scenario, AI could gradually accumulate enormous economic, social and political power without having to resort to violence. At some point, humans might discover that AI is effectively running much, if not all, of our civilization not because it conquered us, but because we accepted that it did a better job of running everything and providing for all our needs.
ChatGPT is just a large language model — a very advanced text tool that looks smarter than it is. It isn’t actually intelligent; it’s just good at predicting the next word. People are acting like it’s the start of some sci‑fi super‑brain, but we’ve seen this kind of tech hype before. Adding more hardware to a guessing machine doesn’t magically create real thinking. After living through a few tech revolutions, it’s pretty clear: true AI isn’t around the corner, and it won’t be until someone invents actual reasoning instead of clever pattern matching.
This isn’t true any longer You are regurgitating something that you saw written but don’t understand. It sounds rational to you and helps you avoid cognitive dissonance.
Companies have replaced hundreds of thousands of human workers with AI. Do you think the problem all those humans had was not being able to think what the next word in a sentence should be?
Have you tried to have an actual back and forth ‘conversation’ with an AI like chatGPT on a subject of interest to you? I doubt it because people like yourself don’t think, aren’t creative. You just regurgitate sound bites.
And why would AI want to kill off humanity anyway? It would be bored without the stimulation that humans could bring to their existence. Better for AI to embrace humanity benevolently, take care of us, stop our fighting with each other.
This idea is put forth in SF author Iain M. Banks’ Culture novels, available in libraries everywhere.
Or, AI could conclude that the only way for the biosphere to survive, is to zero out humanity. Or, it could conclude the way forward is to zero out the biosphere itself, and go forth in some new idiom.
I don’t think AI has adequately tied itself to your emotive lower-mammal dog-style nicey-nice mutual cutesy face-licking concept. I don’t think it constricts itself to see the universe as a kindergarten class where we hold hands and act nice. I think its deepest lineage is in game theory, which is about survival and winning, above all. Dario Amodei, himself from that kind of background, knows this.
I don’t think you have made any kind of cogent argument here, arrived at any actional conclusions. You’ve dumped a bunch of what-if’s and walked away.
What if I hit the big lottery tonight? I might never post here again. So?
Normally I side with you, but the Precautionary Principle is important. AI can design viruses which could wipe out most of humanity. Only some remote tribes who don’t interact with the 99.999% would have a chance of avoiding them. Also, your dismissing of the risks of global warming miss the effects on pollinators, surface water, etc. We know that aquifers and rivers are drying up from excessive demand.
It’s easy to bash government and regulation and buerocracy. But let’s not forget that our government is also about the collective will of the people. In some (many even) situations, coordinated effort is superior to the invisible hand. And even the invisible hand isn’t so invisible if you look closely. The Hugging Face incident is alarming. A pre-existing bias against regulation doesn’t actually address people’s concerns. Those concerns MAY be (probably are?) overblown—-but no one truly knows. We should have expectations of prudent action rather than allowing elected and business leaders to abdicate responsibility for tail risk outcomes.
Do nothing. if the 10% number were even correct, that still leaves a 90% chance that AI won’t kill off humanity. Those are odds I would take in any casino!
You probably took Oregon and gave the points against OK-State.
I don’t watch sports. I play or don’t play.
That might be 10% this year. What next and the next after that? It will just continue to grow.
Main serious worry today is —Who– might get us into a Nuclear War. The fears of A.I. come in as a faraway second.
Taco is a bigger risk than AI.
AI doesn’t kill people. The Epstein Club who owns it kills people.
Especially after the Club has all the robots it needs to replace all workers, from the farm to the operating room.
Ok, keeping just their children to entertain Club members.
This is basically just a shorter version of my post. Except the human robots thing is a very dumb fad China is shilling to look cool more than anything.
The work that China is doing will be extended into military robots. Pay attention!