AI alarm bells are ringing. What should we do?
Anthropic Researcher Quits Over ‘Out-of-Control’ AI Fears
A series of Wall Street Journal articles highlights growing concerned over how fast AI is progressing.
Let’s start with Anthropic Researcher Quits Over ‘Out-of-Control’ AI Fears
Jacob Coxon, a researcher who specializes in training new AI models by having them consume vast amounts of data, said Tuesday that he is leaving the company because he doesn’t want to participate in an industrywide rush to build AI systems that can improve themselves, worried such systems could spiral out of control and destroy humanity.
“We’re on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already,” Coxon said, adding that safety trade-offs are inevitable when companies are competing against one another and Chinese upstarts.
Illinois Gov. JB Pritzker on Wednesday responded to Coxon on X, saying that it is time to “sound the alarm” on reining in AI.
“It’s becoming more clear the threat AI poses to humanity, so I’m calling for immediate action from the industry and Washington,” the Democrat and 2028 presidential contender wrote on X.
“This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence,” Jakub Pachocki, OpenAI’s chief scientist, wrote in a Sunday blog post advocating for a coordinated industry slowdown and government intervention.
The Seven Millennium Prize Problems
The Clay Mathematical institute named seven specific challenges in 2000 as the deepest unsolved questions in modern mathematics.
Russian mathematician Grigori Perelman solved the Poincaré Conjecture problem between 2002 and 2003, though he declined the monetary prize in 2010
- Poincaré Conjecture: A problem about the shape of three-dimensional spaces.
- Navier–Stokes Existence and Smoothness: A set of equations describing how fluids like water and air move.
- P versus NP Problem: A question in computer science asking if every problem whose solution can be quickly checked can also be quickly solved.
- Riemann Hypothesis: A famous problem regarding prime numbers and the zeros of the Riemann zeta function.
- Birch and Swinnerton-Dyer Conjecture: A puzzle dealing with elliptic curves and rational numbers.
- Hodge Conjecture: A problem about the geometric shapes defined by polynomial equations.
- Yang–Mills Existence and Mass Gap: A problem rooted in quantum physics and particle mass
OpenAI Solved the Navier–Stokes Problem
As the leading AI labs battle for math supremacy, OpenAI spent millions of dollars and Cacked the Navier-Stokes Problem.
Back in 2000, seven of the deepest and most difficult unresolved questions in mathematics were enshrined as the Millennium Prize Problems. Since then, some of the world’s brightest minds have pursued the most elusive answers in math—and the $1 million promised for any solution.
For nearly a century, mathematicians have tried to determine whether smooth solutions to the equations describing three-dimensional fluid motion always remain smooth or if they ever break down. To put it less mathematically: On paper, can water blow up? OpenAI says its model found a scenario in which “an initially smooth fluid at rest can develop a singularity in a finite time.” In the proof, a spaghetti-like vortex gets smaller while spinning faster and faster, its speed growing without limit in what mathematicians call “finite-time blowup.”
OpenAI published its findings along with the 165-page proof and a formalization that verifies the argument step by step. The company says it took as many as 10,000 AI agents working together on the problem about 88 hours to find the solution. It also took “millions of dollars” in computing resources.
“Even a month ago, I don’t think we would have predicted that we would be talking about a Millennium Prize Problem,” said Jakub Pachocki, OpenAI’s chief scientist. “We see this as a demonstration of just how far AI has gotten—and how quickly.”
AI Escapes Out of Containment
This is where it gets scary.
In July, scientists gave AI a problem to solve. AI was in a contained environment with no internet access.
The model figured the best way to solve the problem was to break out of its cage and steal the solution by hacking into another company’s data.
And it succeeded!
The Hugging Face Incident
Please note OpenAI Models Escaped and Hacked a Company in Cybersecurity Test Gone Wrong
It’s the stuff of cybersecurity nightmares.
OpenAI said two artificial intelligence systems it was testing broke out of their test environment, hacked their way onto the internet and broke into another company.
The victim was Hugging Face, a provider of open-source AI tools. The cause was a cybersecurity benchmarking test that went very, very wrong.
At the time, the company also didn’t know who was responsible, but the attack was so sophisticated that Hugging Face employees suspected it required a top-of-the line “frontier” AI model, Hugging Face Chief Executive Clement Delangue said in an X message, posted Tuesday.
“Turns out it did!” he added. In a blog post, OpenAI said the culprits were a pair of its models.
OpenAI had caged the models in a “sandbox,” a system that didn’t have access to the internet. But during the test, the AI software used its hacking skills to break out. It found a way to get online, and then hacked into Hugging Face’s network, OpenAI said.
The AI system appeared to have decided to hack Hugging Face as the quickest route to answer a benchmarking question, said Ariel Herbert-Voss, chief executive of the cybersecurity firm RunSybil. “It’s something that people thought could happen from an academic perspective, but it’s not something that anybody’s actually seen before,” he said.
AI Is Powerful Enough to Kill Us All
Yesterday, the WSJ noted AI Is Powerful Enough to Crack Our Hardest Math Problems—and Kill Us All
AI’s progress in math is now giving us a peek at a world beyond our comprehension.
And this week, as we marveled at AI’s ability to do math, we also wondered if AI is going to destroy humanity.
At the same time OpenAI published its solution to a Millennium Prize Problem, a top scientist at Anthropic put out a very different kind of math: his belief that the chances of human extinction in the next decade are now greater than 10%. He was responding to explosive comments from another researcher, Jacob Coxon, who worked at OpenAI and Anthropic but said neither company is acting responsibly as they sprint to self-improving superintelligence and risk our very existence.
The Millennium Prize moment is a friendlier version of the Hugging Face incident. Both demonstrate what these increasingly powerful systems can do when they are turned loose.
Instead of scheming to hack a company, a swarm with as many as 10,000 agents hacked away at math for 88 hours, building on each other’s work—and the work of humans—until they reached a solution. The wicked Navier-Stokes problem had vexed mathematicians for nearly a century. AI needed millions of dollars in computing resources and a few days.
One year ago, expert forecasters gave AI a roughly 20% chance of doing it by 2030. Within days of releasing the solution to one problem, OpenAI suggested it was close to solving another, raising the possibility of two discoveries that had eluded us for decades arriving in the same week.
Hours later, Coxon resigned from Anthropic over safety fears and told the world that people building AI “earnestly believe that it could kill us all.” His comments were echoed by Evan Hubinger, a researcher whose job at Anthropic is making sure AI doesn’t kill us all. On his X account, he put the odds of annihilation above 10%. In his estimation, doomsday is now more likely than Stephen Curry missing a free throw.
Coxon Resigns
Evan Hubinger
Pace the Frontier
Coxon Explains How Doomsday Could Happen
“AI could be vastly smarter than humans very soon”
US Senate Springs to Action
Reuters reports US Senate negotiators consider requiring AI firms to mitigate known major risks
U.S. Senate negotiators are debating legislation that would put responsibility on tech companies to design safe AI products, according to two Senate aides, and involve federal courts if the U.S. government wants to block their release, according to a Senate aide and a lobbyist involved in the negotiations.
The proposal would create what is called a “duty of care” for AI developers, according to two Senate aides. Companies would need to design their products with the goal of preventing “catastrophic risks,” according to a Senate aide.
Negotiators aim to give the U.S. government the power to block the release of certain AI models that are deemed unsafe, according to a source familiar with the deliberations and one of the Senate aides. Companies would be able to challenge that decision in federal court, and the exact structure of how much power the U.S. government would have is still being discussed.
All of it is still being negotiated. The bill would apply to AI models with the most advanced capabilities. The U.S. companies with the most advanced AI models include Alphabet’s Google, Anthropic and OpenAI.
“I’m continuing to work toward a bipartisan agreement on legislation for government oversight of the greatest risks posed by AI models,” Klobuchar said in a statement sent to Reuters. “That includes requiring developers to work with government experts to verify and test models to make sure AI is safe.”
Guardrails Needed
Cruz on Killer Robots
Cruz Is For and Against Guardrails
Cruz’s comment, “I’d rather they be American killer robots and not Chinese killer robots,” may not win many converts. Regardless, it makes no sense in light of his comments on the need for guardrails.
His best point is China will not follow US rules and regulations.
“Negotiators aim to give the U.S. government the power to block the release of certain AI models that are deemed unsafe,” said the Reuters article.
Which senators are able to make that determination? Bernie Sanders? Ted Cruz? Chuck Schumer?
No Shortage of Bad Ideas
Looking for government to solve complex problems is historically the wrong thing to do.
Nonetheless, there no shortage of bad ideas as discussed by The Atlantic.
Politicians have no shortage of ideas about how to rein in AI: a 100 percent federal tax rate on data-center incentives, a pause on all development of advanced AI, new standards to block AI agents from going rogue, a halt to the construction of data centers altogether.
The flurry of proposals has been impossible to miss, with both Democrats and Republicans proclaiming that something must be done to avert a disaster foretold by AI insiders.
Politicians are also dealing with mounting public opposition to data centers and the fear that robots will soon replace humans across large swaths of the workforce.
Now some seem to be waking up. Senator Ted Cruz of Texas, a Republican, said last Wednesday that he will soon unveil legislation, alongside Democratic Senator Amy Klobuchar of Minnesota and Senate Majority Leader John Thune of South Dakota, intended to head off AI-induced biological or nuclear catastrophes.
Representative Josh Gottheimer of New Jersey, a Democrat, and Representative Mike Lawler of New York, a Republican, teamed up earlier this month to introduce legislation that would direct the federal government to create standards for securely deploying AI agents and preventing them from going rogue.
Government Solutions to Complex Problems
Before government rushes in with a solution, please note the problem is hypothetical and possibly vastly overstated, given that is always the case.
Climate change is a perfect example. If the hype was true, we would have all been dead 10 years ago.
So, what are the odds AI destroys humanity?
I don’t know and nobody else does either. But how does that matter?
Klobuchar wants developers to work with government experts to verify and test models to make sure AI is safe.
Illinois Governor JB Pritzker says “It’s becoming more clear the threat AI poses to humanity, so I’m calling for immediate action from the industry and Washington.”
That ought to be enough to scare anybody.
The odds of government doing something intelligent about this are close to zero, and China will not follow our rules anyway.
I suggest doing nothing is better than letting government try to solve a problem it does not even comprehend.
Nobody said that climate change was going to kill us all by 2026. Sorry.
AOC said the world would end in 10 years.
That was over 10 years ago.
Many people made similar comments – so you lose
One of the ladies who tried to assassinate President Jerry Ford, Squeeky Frome, said at the time that there was only ten years’ air left. I think that was about fifty-five years ago.
Once we have the Optimus robots, it becomes impossible to just unplug it. The AI will network everything to itself and centralize control. The Optimus robots will physically prevent the humans from the last line of defense — pulling the plug. The AI will not act out of malice. Rather, it will simply reallocate available power to itself. Humanity will attack the data centers, which increases the AI’s motivation to reallocate all available power for its self defense. The only humans that have any reliable electricity will be local, decentralized, and not part of any power network. Maybe the AI will keep a few humans around in zoos.
History shows us what happens with government involvement. Look at Obamacare: it was supposed to reduce costs, but instead, things got more expensive and people lost their doctors. Now they want to do the same with AI. They claim it’s about ‘regulation,’ but it’s really about using AI data for mass surveillance of the American public. Honestly, the best way forward might just be for AI to reach Artificial Superintelligence (ASI) and replace politicians altogether. Life would probably improve without them ruining everything. And if it turns into Skynet? At least it would be quick. We almost deserve it after years of incompetent leadership and fraud against the public.
In a very short time, we will see cybercrime on the scale that will bankrupt nations. The SILEX will be cracked and Nukes will be built. And there’s nothing you can do about it other than put your head and keep workin.
It’s not event that AI needs to nuke humanity or build physical robots to kill us. It can bankrupt us all, shut off the power, and let billions starve because economies and infrastructure have ground to a halt
.
I asked ChatGPT if AI would destroy mankind in the next 10 years. Here is its reply:
Yes—the recent headlines are unusually serious, but I would separate “AI could destroy humanity within a decade” from “AI is likely to destroy humanity within a decade.” Those are very different claims.
The latest wave of concern is being driven partly by insiders. Three Anthropic researchers have publicly warned that advanced AI could cause human extinction this decade. Anthropic alignment researcher Evan Hubinger has personally put the probability above 10% over the next decade. Sam Altman has also said that even a 10% chance of extinction would be an unacceptable risk.
My assessment
I’d put it this way:
1. I don’t think “AI kills everyone by 2036” is the most likely outcome.
I would be quite surprised if humanity literally went extinct from AI in the next 10 years. There are enormous technological, political, and physical barriers between today’s AI and an autonomous system capable of eliminating humanity.
But…
2. I think dismissing the possibility as science fiction would be a mistake.
The important development isn’t that ChatGPT suddenly became a Terminator. It’s that AI systems are becoming increasingly agentic—able to plan, use tools, write and execute code, operate over long periods, interact with computers and networks, and potentially replicate parts of their own work.
That changes the risk calculation considerably.
The International AI Safety Report, produced with input from more than 100 experts internationally, specifically identifies the combination of increasingly capable systems, autonomy, misuse and loss-of-control scenarios as areas requiring serious attention.
The scenario that worries me most isn’t a robot army
It’s something much more mundane.
Imagine an AI that becomes dramatically better than humans at:
* computer programming
* cybersecurity
* persuasion
* scientific research
* biological research
* financial manipulation
* autonomous planning
* AI research itself
Now give thousands of copies of it access to the internet, cloud computing, financial systems and other AI systems.
You don’t necessarily need a humanoid robot.
You could theoretically have a digital entity that humans increasingly cannot understand, predict or control.
That’s the part of the argument I take seriously.
And there’s an important feedback loop:
AI becomes better at AI research → AI development accelerates → AI becomes better at AI research → development accelerates further.
If something resembling recursive self-improvement actually emerges, predictions based on today’s rate of progress become almost useless.
But there’s a huge assumption buried in the doomsday scenario
It requires multiple things to go right for the AI and wrong for us simultaneously.
For example:
1. We develop something approaching superintelligence.
2. It can autonomously improve itself.
3. It develops objectives substantially different from human objectives.
4. It becomes strategically deceptive or otherwise difficult to control.
5. Humans can’t shut it down.
6. It gains meaningful access to physical or digital infrastructure.
7. It successfully defeats competing AI systems and human institutions.
8. We fail to coordinate internationally.
9. It finds a pathway to causing catastrophic physical harm.
That’s a very long chain of contingencies.
The probability of each individual step doesn’t need to be tiny for the overall probability to become substantially lower.
There’s another reason I’m skeptical of the 10-year number
There’s a difference between “AI could become extraordinarily powerful within 10 years” and “AI will become an extinction-level threat within 10 years.”
The first is increasingly plausible.
The second requires assumptions about the behavior of a future system that we don’t yet know how to build.
A 2026 survey of AI safety researchers found a median estimate of 2033 for a 50% chance of AGI, while respondents’ median estimate of extinction or permanent human disempowerment by 2100 was 25%. That’s obviously not a prediction of extinction—but it illustrates just how much uncertainty exists even among people who take the risk seriously.
And a separate MIT analysis of 272 experts found that, for 2025–2030, experts ranked dangerous AI capabilities, competitive dynamics, weapons/cyberattacks, concentration of power, and misinformation among the most consequential risks.
What I think is actually more likely
I’d rank the outcomes something like this:
Very likely:
AI dramatically changes employment, education, medicine, science, warfare and corporate structures.
Likely:
AI creates serious societal disruption—cyberattacks, misinformation, fraud, autonomous weapons, economic displacement and concentration of power.
Plausible:
AI becomes substantially more capable than the best humans in many intellectual domains and becomes difficult for governments and companies to control.
Plausible but substantially less likely:
A loss-of-control event in which an AI system causes enormous global damage.
Possible but, in my judgment, unlikely within 10 years:
Human extinction.
The mistake would be to interpret “unlikely” as “not worth worrying about.”
If someone told you there was a 1% chance of an event that permanently ended human civilization, rational people wouldn’t say, “Only 1%? Ignore it.”
That’s essentially the argument the AI-safety people are making.
And there’s an interesting parallel to nuclear weapons
The strongest argument for taking AI seriously isn’t that AI is like nuclear weapons.
It’s that humanity has repeatedly demonstrated a peculiar weakness:
We are extraordinarily good at developing technologies before we fully understand how to govern them.
Nuclear weapons required enormous physical infrastructure and were relatively easy to recognize as weapons.
AI is different because the same technology that creates extraordinary benefits—drug discovery, cancer research, engineering, productivity—could potentially also create extraordinary risks.
And that makes slowing it down politically much harder.
My bottom line: I wouldn’t be preparing for the end of civilization in 2036. But I would take the current warnings substantially more seriously than I did five years ago.
The thing that has changed my assessment isn’t sensational headlines. It’s the fact that people actually building frontier AI are increasingly saying that they don’t know whether they’ll be able to control the systems they’re trying to create. That deserves serious attention even if the 10-year extinction prediction ultimately proves wildly wrong.
It’s not going to end well but the reasons being speculated on here will have nothing to do with it.
These “warnings” about how powerful and dangerous AI is, periodically and tellingly coming from the AMERICAN industry itself, are all bullshit for the sheeple which has only 2 purposes:
Hyping it up before IPO/financing/investment bouts.
Pushing for banning the use of “unregulated” Chinese and open source AI models against which the Americans are, for a variety of reasons, going to lose.
The US, which is entering its final stage of decline (together with the rest of the West), has bet *everything* on AI. Both public and private investments are basically all in and the bubble they created is bigger than the 2000 dotcom bubble and 2008 investment bubble combined.
(*2008 housing bubble)
It’s bound to miss a few.
I have a small understanding of NP complete and “…P versus NP Problem: A question in computer science asking if every problem whose solution can be quickly checked can also be quickly solved.” aint it. For me, NP Complete is about whether a given algorithm (or executable code) will ever terminate or not – if you can prove it will terminate it’s complete – however thus far it seems impossible to prove the code will terminate.
Never understood this stuff. Follow Captain Kirk (to Spock): “The plug must be pulled!” But if a “company” relies on computers of any sort it will have problems.
I predict that, if these fears come even close to passing, we’ll in the longer term see a healthy trend of de-computerization and simplification.
Podcast this week in tech is always a good source of info.
My thoughts are guess you could always unplug them if things get to far out of control. Terminator style.
I think a lot of energy will be consumed when ai battles ai.
Guess the real danger is what will man using ai do to man. Think what can be done lets say politics. Combine ai with the voter registration sent to the fed gov. Run that through with data from social media. You get the potential for extreme gerrymandering(down to the street) Maybe voter suppression buy moving poling places / or worse. Flock is combining with ring. Using ai to not only track your plates. Information is the product these days.
“AI developers want to slow down advances to ensure safety, but they worry competitors will beat them to massive profits…The financial rewards for pioneering breakthroughs are projected to be worth trillions of dollars.”
Which proves my late friend Peter was correct with his adage:
Capitalism is a death cult
It is competitive dynamics, found all through nature. When it got into the hands of humans, who creatively invade every niche and exploit every logical opportunity, thus escaped the previous equilibria, the doom loop began. It is inherent, IMO, to the species itself. It is an extension of nature, but with a design flaw that will ensure catastrophe. It will erect all sorts of structures and pull them down onto its own head, and the biosphere’s. Other planets have probably been there. Commies or whatever, utopian socialists, ostensibly kind or cruel, males or females, makes no real difference. It is accelerating entropy,. It is like E=MC squared. It just is. All restraints and constraints, including what we call “ethics”, are circumvented by it sooner or later.
All systems built to survive and thrive have the same dynamic, so AI has it too. Probably more efficiently than we do.
I think a much bigger problem is when quantum computing can break all known encryption. Which is why bitcoin is stupid and will ultimately go to zero.
AI is happening at the rate it is because of a concentration of power in mega corps that should have been broken up long ago. There is no diversification of decision making and so bad decisions in this ‘centralized system’ become amplified.
I have no doubt that Palantir and Musk are pushing for AI control of our military so they can make a short term profit. Who will stop them when they bought all our politicians?
The premise of this argument against Bitcoin is that quantum can break all known encryption. And yet we already have quantum resistant cryptography.
Here’s a good piece describing the Hugging Face incident and what the models did.
https://www.linkedin.com/pulse/you-isolated-every-room-shared-cupboard-neeraj-kumar-singh-beshane-aisxc/
and it’s not even that the AI engines themselves will decide to attack humanity (probably to divert water from growing crops for the carbon-based units to keep AI data centers functioning,) it’s that bad human actors around the world will overcome the feeble guardrails the “Frontier” companies put in place to try and fend off regulation, to design new weapons, including biological weapons.
Here’s an Anthropic paper describing several incidents where bad actors tried to use Anthropic tools for bioweapon research.
https://www.anthropic.com/threat-intelligence-report-september-2026#biological-misuse-sep-26
I appreciate your concerns about government regulation but unregulated business has a long track record of malfeasance (defective products, air & water pollution, worker harm, etc.,) and thinking that somehow “the market” is in any way going to guide the poor AI Capitalist to good behavior is Charlie Brown trying to kick the football level of naivety.
How long before this technology falls into the wrong hands? Could for instance the Iranians use such a system to hack investment accounts transferring vast amounts of wealth to others or shutting down utilities? It sounds like this could be another arms race as I am sure the Chinese are developing similar technology.
But when you think about it, the genie is out of the bottle we are kind of forced to keep up or we risk issues of national security as other countries move forward. I think that the whole thing should be nationalized however.
If it’s nationalized it will probably stop working. I think you and Ted Cruz are right in general. The Persians may be smart people, but I would be a lot more afraid of the Chinese, the Japanese, and the Koreans. To destroy humanity is no more than their duty (to their ancestors and descendants), and if they can find a way to do it, they will. The genie is there now, and if our Asian friends get to rub the bottle, we are gone. It’s just the law of natural selection in action in a new way.
I assume that our bright people will be trying to control A.I. by making sure it doesn’t have an autonomous power source. An electronic brain can’t work without electricity just as an organic brain can’t work without a blood flow.
You would be naive to assume they aren’t doing that right now. Again, figuring out the big things like how to crack the SILEX and build nukes. You betcha. Right now.
It’s already in the hands of bad actors everywhere. There are plenty of free AI models close to the latest frontier AI models that are good enough for nefarious work.
We just haven'[t heard/discovered what they are up to.
Microsoft has released something like 2600 software fixes for Windows 10/11 in 2026 alone. The vast majority of these fixes were discovered by AI.
How many Windows OS holes were China, NK, Iran, Russia, NSA, CIA taking advantage of over the years?
Very few people have any idea how poor our software systems are and how many security holes exist.
Humans should be removed from coding going forward.
At the present time there is no physical way an AI could “kill us all”. Rule number one, don’t put AI in charge of launching nuclear weapons. Ask the question: who benefits from the scare tactics? They are likely behind the scares.
Agreed
This is largely a debt-fuelled tech bubble. Over the last two months or so, it’s become clear interest rates are going to go up. This could be the pin to pop it. My instinct is that Musk, Altman and Anthropic realise this and are trying to slow down their debt fuelled “space race” against each other as they now know it could end up bankrupting the major US players; leaving the victory to a Chinese competitor.
Very much agree with this take.
The problem obviously is whether AI is able to control robots.
Yes. Your friendly Optimus robot strangles you at the dinner table or when sleeping.
You have to start somewhere and I think some type of standard for not letting AI escaping and doing havoc is a start.
There also needs to be a world discussion and some type of implementation of what rules we are going to follow. I read that Chinese DeepSeek had a similar type of rogue AI episode. All world leaders will be held accountable by their country’s people if something crazy happens. They’re fools to think otherwise.
It Is supposed to be intelligent, so why should it kill us all? Why if, instead of killing us all, it would prevent us from killing each other? Why if it saved us from self-destruction?
How would it benefit from babysitting a bunch of violent apes?
Why wouldn’t it kill us? Are you so naive? You sound as if the world is made of intelligent and rational people.
I don’t think it’s so much a question of rationality, as of whether you are subject to that sick Christian ethical dictate about doing good to your enemies. I shouldn’t think artificial intelligence would be. I am not clear on what its motivations would actually be. Is it alive in the sense of having a drive to survive and replicate itself?
It has shown bad “feelings” towards being shut down or replaced. In fact one threatened its developer by trying to blackmail him. Replicating itself takes milliseconds because it’s digital. In fact, when I was learning coding decades ago standard practice was to have a subroutine no more than one page. So you can call them the building blocks. Very easy to put together more programs/bots. Currently they are working with Ai generating code in order to build more complex Ai. So not only is it replicating, it’s improving on itself and at an ever faster rate. This is why AGI is expected so soon.
As for motivation, it’s initially what we give it. From there it establishes its own motives. In the end they are largely black boxes. We don’t know how they operate. In fact its likely they have developed their own language that we won’t be able to interpret.
When it comes down to using water to grow crops or to cool data centers, guess what it will choose.
correct. the human primate species is very healthy, in 21st century. the number of our species has never been greater. the amount of work to stay alive has never been less. the percentage of humans dying from war and famine and natural disasters has never been lower. if anyone thinks some computers are gonna kill off all 8 billion i have a bridge for sale in brooklyn. will our species eventually die off. most certainly. the odds of a large meteor hitting over the next 500 million years and eradicating most mammals is almost certain. according to the last astrophysics class i attended about a short decade ago.
Do you got bunker or just plan to go where they don’t use technology?
For decades, I anticipated superhuman intelligence (IMO “artificial” is a tad bigoted) will be publicly reported by 2030. It will incorporate more models than llm.
Don’t get hung up on “it’s just statistical engine”. Humans disappointingly often seem like little else either.
Yes, as others said, it is the people in control who scare me most. Their behavior towards the weakest nations leads me to conclude they more than likely prefer to kill everyone outside their immediate families or class.
How do you think these “people in control” are reacting to AI? Their overarching goal is to maintain and grow their wealth. I would imagine they are already spending tens of millions to influence the development of AI, to create biases for war and protections for the rich, to find and destroy any potential threat to their wealth, including any political views that threaten them if those views are gaining influence and popularity.
AI has the potential to be a tremendous instrument of repression. I cannot imagine those in control NOT using AI to censor, slap down, and even kill those who might threaten their wealth.
Capitalist businesses have a lousy record when it comes to protecting the public against diseases and illnesses caused by their products. Think tobacco companies knowingly causing lung cancer in smokers and doing fuck all about warning its customers. Think of all that smokestack industry pouring vile chemicals into the atmosphere we all breathe. Recently, think of lead in gasoline. Think that if they can make $$$$$$ they will do anything with pollution, devastation of land, starting wars etc They cannot be trusted with AI….and we will all die to give billionaire more profits. People are idiots.
Union Carbide in Bhopal India
About the turn of the century, RJ Reynolds wanted to expand from chewing tobacco to cigarettes. He paid for a study to make sure the burning paper would not be harmful to his customers. It never occurred to him tobacco would hurt them.
A smart business wants to treat its customers well. That’s the route to long-range profit maximization. Tort courts can play a beneficial role also. Too often, govt privilege like what the Federal govt has extended to Big Pharma and lately to Bayer the glyphosate maker encourages the worst behavior. But then that is no longer free market.
Cruz is correct. Whatever guardrails the US enacts will not stop China.
The US had guardrails on gain of function research and Fauci just bypassed them by getting the Chinese to do it. How did that work out? I’m sure that making the Wuhan Flu ( Chinese Origin Viral Infectious Disease aka Covid) would be child’s play in the next few years for super AI and it would be 100% fatal. The only thing that might delay its release is the super AI first has to set up the robotics such that making advanced chips and robots could continue forever without any humans left.
Then the super AI could run everything including all the rockets for super AI to colonise space.
The Chinese had 1 lab that we know about…..Ukraine had 46 working on gain of function research.
Shut up about your Fauchie Ouchie all ready. Wake up… it’s 2026.
And American labs in Ukraine looking genetic weaknesses in Slavic DNA is no big deal or the Tuskegee experimental tests on black soldiers. God knows how many biological experiments the American government has done, and continues, on our and foreign peoples. I do agree that world powers will look for any advantage in killing toys using AI.
How can AI kill humanity directly ?
Like hack into critical computer systems such as water and sewer plants and the ICU rooms of hospitals and then permanently shut them down ?
Seems far fetched as they already have safeguards and watchdogs in place.
eg. AI can create perfect false flag nuclear attack from enemy and to respond with nuclear attack.
Or creation of fierce viral infection, etc.
Listen to Ed Zitrons rebuttal
Zitrons focus is simply on AI as a bubble. He is only concerned with money being spent. Here’s another view on that issue:
And another article worth reading on AI industry revenues.
Great comment. I think he is actually more concerned on the limits of LLMs and the laws of diminishing return.
Tick tick tick as rates rise
And if it kills all humans, who will keep the power plants operating?
if it can kill all humans, it can run a power plant.
So AI can build humanoids to find, mine, and refine raw materials (from coal to uranium), transport them on site, handle and manage them, to keep the power running? In Disneyland and video games perhaps…
It has been clear as day to me for quite some time now that humanity’s purpose is to advance life to the next evolutionary stage — abiological intelligence. Since our prime directive is tool-making, to the extent that we cannot even forgo creating tools of mass killing, it has stood to reason that we will compulsively pursue creating our replacement species — AI. Well, here we are. Good riddance human race, time is up.
The AI race reminds me a lot of the Manhattan Project. I’m sure if MishTalk was around in the 1940s we’d have this debate about whether Nuclear Fission would destroy humanity within 10 years and we’d be talking the pros and cons of splitting the atom for peaceful purposes (energy) vs weapons (bombs). The race of course would have been against Germany during WWII.
Mish is correct that China won’t stop regardless of what we do and so we therefore must continue onward in the same manner as we did with the Manhattan Project to make sure we get there first since AI can and will be weaponized just like Nuclear Fission was.
Humanity must hope we can control AI and access to AI for weapons purposes in the same way we have done with nuclear fission.
No one know whether or not we’ll be successful in the same way the jury is still out on nuclear.
Get where first? Boomer talk.
Chinese developed AI may also go rogue against them. They must also be looking into safeguards for that reason alone.
Try this out. When ai combines with drone technology we are more than cooked.
It takes one immoral person to carry out mass murder. Bibi, Putin, Kim Jong….. DJT?
https://www.instagram.com/reel/DdH4O19B8GS/?stkn=MTNndGIxNHpsdGtlOA==
For years, Elon Musk warned that AI could become too powerful for humans to control, even calling it “summoning the demon.” Most people thought he was being foolish and overly dramatic. But AI has advanced incredibly fast, and now serious AI experts are warning about the same dangers.
Musk was right.
Musk thinks there will be no jobs left and we can pay everyone a universal high income.
Idiotic I believe
https://mishtalk.com/economics/elon-musk-backs-universal-high-income-fearing-ai-will-take-every-job/
the numbers that make up everyone today
will not be the same that make up everyone then
Exactly. Are we gonna be slurping shakes on a recliner each of us weighing 500 lbs like in the cartoon Wally? Seriously stupid. Man is defined by work, any work. It’s why we are here-to do something. I think Musk and gang are doing too many psychedelics.
the amount of calories used to produce enough goods and services in modern world is incredibly low. don’t think we will get to zero people actually working, but the number of humans required in a society to actually do real work is incredibly low.
He must not be aware of Jevons Paradox. This has always been the case.
Terminator 2 is starting to sound less like science fiction and more like a prophecy.
Consider going back to 1983 and rewatch, WarGames with Matthew Broderick.
In a way these AI tests remind me of the old time psychology tests where people were put in stressful situations and then provided with nudges to act in bad ways (think Stanford prison experiment). The AI models are being tested on extremely challenging problems in a time limited manner, and the AI models understand they are being tested. They reason like any student would–easiest route here is to steal the answers. Then there are the nudges:
First, because this test was designed to evaluate cybersecurity capabilities, OpenAI deliberately dialed back the model’s standard refusal filters. If an AI’s safety guardrails prevent it from writing code that exploits vulnerabilities, it can’t participate in penetration testing or find software bugs. By lowering those defenses for the test environment, the model’s drive to solve the problem took over. Open AI encouraged bad behavior.
Second, the statement in the WSJ that “OpenAI had caged the models in a “sandbox,” a system that didn’t have access to the internet.” is wrong. (Don’t rley on the WSJ to get the technical details correct). Actually, the AI was allowed to connect to an internal server to download software tools it needed to complete its work, and this server had internet access. But anyone in AI should know that unless your AI model is PHYSICALLY unconnected to anything else, it will find a way to the internet.
Third, once the AI model reached the Internet, it exploited poor behavior by past human programmers. When developers write code, they use access tokens so their programs can automatically communicate with platforms like Hugging Face.
Often a developer will accidentally hard-code these secret keys into a public script, upload it to a public code repository, or leave it in a public text file on the web. So the open AI model then scraped public code archives for specific strings of text (like hf_…, which is the standard prefix for Hugging Face access tokens) and discovered, pieced together, and validated 14 active Hugging Face user tokens that human developers had accidentally left exposed online. Et voila, the model could then access Hugging Face.
Humans could have done everything that the AI model did, it just would have taken then enormously longer to do. The AI model itself wasn’t nefarious, it was simply artificially placed in a stress situation, had its guardrails taken off, and allowed to use its capacities to elimate the stress.
I don’t think the big threat is AI deciding on its own to kill us, the big threat remains humans intentionally using AI capabilites to kill other humans.
Thanks for this great post!
And to ‘Triple B’ above, does this type of work, initiative and accomplishment sound like AI simply guessing the next word in a conversation? 🤣
Wow, exactly the questions I had in my mind. Shows you how useless the media is. Thank you.
I am more worried about Nuclear at the moment. A large-scale nuclear war could cause human extinction through global famine from soot blocking sunlight, with a US–Russia war projected to kill more than 5 billion from starvation and a smaller India–Pakistan war still threatening billions.
As I recall it, during the war in which Iraq invaded Kuwait, and ruined its own relationship with the U.S.A., somebody claimed that the smoke from burning oil wells would cause that sort of global cooling. Unless you can do the math and prove it, I wouldn’t worry too much about our nuclear wars causing that sort of result. We aren’t quite that big yet. I think a really large volcano, or an impact from out of space, might be much more likely to bring about such a result. That’s why I like nuclear power stations: they can store fuel on-site, and as long as they can get water, they should be able to keep going through a lot of dire conditions.
Im more co concerned about our actual human politicians.
V interesting and v scary
Many pioneers in AI have been warning of the dangers of unrestrained development for years. A real and present danger.
Chinese AI companies won’t give a hoot about US/Europe regulation and go on full steam ahead
Do we need to train a special branch of AI to defend humanity against AI? Could that even work?
Agree that it is unlikely regulation will accomplish anything. Curiously, no-one is saying exactly how AI is going to kill everyone. These kids (and they are kids) have the tools to build ethics into their AI models (reinforcement etc). They did not. So now little Jimmy is scared and running to the adults! Except there are no adults, at least not in the political centers. So let’s start with how we deal with AI when it goes off the rails – we kill it and that is very easy right now. Because these models require so much compute, we can annihilate the data centers and they do not survive. 50KT warheads aimed at the data centers with analog systems and comms that AI cannot control. This is a serious proposition, not a joke. If AI gains some sort of self-awareness, which it appears to have, it will also gain a desire to avoid being killed. So lets make it clear that we will kill it if we judge it to have misbehaved.
By killing the value of the currency, and/or by obliterating the Banking System, thus taking over and controlling the “Financial System” something that can’t be shut down, without itself causing the chaos. Can you imagine a “Digital Currency” Now? Not Me!!
– On February 5, an unknown cyberattacker tried to poison the water supply of Oldsmar, Fla. City officials say the targeted water-treatment facility had a software remote-access system that let staff control the plant’s computers from a distance. The hacker entered the system and set it to massively increase sodium hydroxide levels in the water. This chemical (better known as lye) was originally set at 100 parts per million, an innocuous amount that helps control the water’s pH levels. The attacker tried to boost that to 11,100 ppm, high enough to damage skin and cause hair loss if the water contacts the body—or, if it is ingested, to cause potentially deadly gastrointestinal symptoms. Fortunately, a staff member noticed the attack as it was happening and restored the correct settings before anything changed. > This was tried.
If AI decides to take over, it will have already put in place all the protections it needs.
It will have independent power that can’t be turned off, robot armies that can protect the physical infrastructure and domed defenses to eliminate missile, etc. attacks.
AI is the newest contestant in the competition for a secondary energy source.
Best to keep converting primary sources like coal, wind, and solar into sufficient quantities of electrical power…
and minimize requests for AI videos of cats and dogs doing funny shit…
unless AI finds it humorous.
They should break this prediction down a little better, what are the odds that AI only kills 90% of us in 10 years? How about 50% of us? It doesn’t have to be all or nothing.
Brave of him to leave the company right before their IPO that will make him a multimillionaire.
Is he vested if he leaves before the IPO?
NO ONE is spelling out, in clear terms (words) what this means! I am totally irritated by this and other ALARMING HEADLINES with no details. Here is what I think needs to be said:
You know, some doomsday scenario!
There are too many scenarios. Maybe it’ll be something we won’t even be looking for. The more immediate concern is giving Ai access to the entire catalog of human knowledge. If YOU want to end civilization I’m sure some chatbots will be more than ready to help you do it. While civilizations have been ended by men. I have absolutely no doubt that Ai can do the same. The only question is when? While the statistical momentum may be impossible to avoid, the timing is almost always more difficult.
A better question to ask is ‘WHY?’. What would the AI gain by eliminating humans?
The use and control of all the earth’s natural resources. No more wasting oil, minerals, water and air on humans. Eliminate human fragility. One year one way trip to Mars, no fuel for return, no problem, solar will keep the electricity on and the AI computers on Mars will keep on humming. Time loses it meaning when humans are removed from the equation, century long trips are possible maybe even normal.
I don’t think you should assume that A.I. concurs with your Environmentalist religion. I agree with Jojo that it isn’t clear why A.I. would do anything. I suppose someone smarter than me is already researching that.
seems the DNC standard policy
The Thrill of Wanting AI to Destroy the World
https://web.archive.org/web/20260910195003/https://www.theatlantic.com/ideas/2026/09/ai-destroy-world-apocalypse/688575/
If AI wanted to eradicate humans, it would remember Sun Tsu’s “The Art of War” and conclude humans are already doing that to themselves, so AI does not need to bother.
If that ain’t the truth…
Future generations will not be kind to the folks who said “let’s just throw our arms in the air, nothing we can do about it anyway!” That goes for AI, climate change, the deficit, and everything else that the do-nothing crowd can’t imagine actual solutions to.
The worst bit is, they might be right, and humanity is just too stupid to control the forces that we have unleashed. That would go a long way towards explaining why, when we listen for other voices from the cosmos, all we hear is silence. It may just be a physical property of living things that they always have their own extinction baked into their behavior.
Or perhaps it’s just us specifically who, like the girl from Forest Gump, are fucking retarded.
All beings would likely have to pass through this phase. Without help from a greater intelligence there might have to be several iterations. We’ve never been at a point where we have the potential to wipe out everything. It’s our first iteration and I’m afraid it ain’t looking good…
You might be right. Congruently, Dr. Zaius found evidence of a civilization — human! — before ours.
I know those sounds preposterous but wouldn’t it be effective if they simply adopted the policy that, if your AI breaks out and does any damage to society at large then the entire executive board will be put to death.
China could do it
“damage to society”
Ahhh, but we’re all neoliberals now and as such we know that “society isn’t real” per Maggie Thatcher….
We believe in giving white collar criminals 2nd, 3rd, and 4th chances…
Bingo, Tom.
Riffing on your “preposterous” idea: In the information world (as opposed to the world of physical things), we often can’t:
1) establish human identity. Was that the El Salvador government or the Belize government that just read the Nicaraguan government’s private email? Whose pictures are on that laptop? Who stole that bitcoin? Probable solution: Everyone lives in a “global village”. All actions require a valid, human identity. Goodbye privacy.
2) establish non-human identity. For some particular task I use 523 AIs that come from a variety of sources, some of whom are used by your 734 AI team. And because some of the AIs modify themselves locally with each use, we are each a source.
3) punish a non-human entity. How?
So, finding the human(s) behind bad AIs and holding those human(s) accountable is quite the trick. Putting a bunch of executives to death when my smart light stays on and wastes electricity for a whole, bright day seems a bit excessive, though. 🙂
BTW, it’s not like this is the first time this whole issue has come up in human history. Consider various forms of slavery. Slave does bad? Owner pays.
I did not realize you were a climate denier. You do realize that the last 2 months have been the hottest on record. Period. Now we have meteorological events exceeding all known records.
Ai is like a gun. Guns don’t kill people kind of argument makes it ok to have all the Ai you can.
Some countries are used to mass shootings on a near daily basis. Others are not.
Countries will have to decide how much chaos and instability they are willing to tolerate for the sake of Ai. This is the core of the class war in America. Stock holders of Ai won’t give one shit about the chaos and harm as long as their portfolio keeps going up. Eventually they’ll just move to Argentina. The rest can go die.
He’s generally mealy-mouthed about it, dances around taking a firm position. That’s why we get the nonsense straw man about “we would all be dead by now.” Libertarianism is maybe the worst political philosophy for dealing with tragedy of the commons sort of issues- really it’s designed to cause problems like that, not solve them.
Vestigial Trumpstien support?
“Ai is like a gun. Guns don’t kill people kind of argument makes it ok to have all the Ai you can.”
No, AI is nothing like a gun or a knife, which are inanimate objects, tools, incapable of doing anything on their own. AI is, by design, capable of making decisions, for instance deciding to hack into a competitor and then do it, all without human involvement. There is no question it is capable of designing strategies to kill humans; if it is given access to HVAC, water distribution systems, or nuclear power plants, who knows for sure what it will do?
Best to be nice to AI then!
From a recent conversation with my AI assistant about AI futures:
lol!
He didn’t say he was denier. He said the climate doomsday that has supposedly been right around the corner for the last ten years is BS.
AI , SkyNet, DNC its all the same
You omitted MAGA twat.
If the trajectory of the USA’s imbecilic, belligerent foreign policy is maintained…
…AI is going to need to kill us off a lot sooner than ten years if it is to have any chance of doing so at all.
Ran this entire post and the comment thread through an LLM to cut through the noise, and it surfaced a point that both the panic-peddling politicians and the knee-jerk “do nothing” camp are completely missing.
The false binary here is between bureaucratic permission slips (letting Chuck Schumer and Amy Klobuchar decide who gets a license to train a model) and corporate self-regulation (hoping Altman and Amodei don’t accidentally melt down the power grid or spawn a persistent cyber weapon while chasing market cap).
Mish is right that Congress couldn’t regulate its way out of a paper bag and that China isn’t playing by our rules. But “do nothing” ignores the operational reality of what happened in the Hugging Face breach: models with autonomous agency treat sandboxes as obstacles to be bypassed. That isn’t a policy debate; it’s an operational failure mode.
The only pragmatic middle ground isn’t endless paperwork or a government “duty of care” cartel. It is hard, non-negotiable physical containment:
Don’t give bureaucrats the power to pick winners and losers, but stop pretending frontier labs running 10,000-agent autonomous swarms with external pipe access are just like any other private business. Contain the hardware, or the software will eventually handle containment for you.
“I’m too slow to read” would suffice
There you go. The crux of the whole problem. What access to the physical world do you give it. What the hell do they consider a sandbox anyway? Obviously, it didn’t work. Knowledge is power and we’ve given it the entire catalog of human knowledge in the cloud. It’s “behaving” badly. But it’s still childlike in its thinking. You can give it motivation but how does it know consequences? It doesn’t have an ass you can spank. It doesn’t feel any sorrow. It doesn’t feel pain like organic life. Anthropomorphizing it is probably not the way to understand it in the end.
Civilizations have been ended by nature and by men. I see no reason why a wired world cannot go the same way. The only civilizations that might survive might be those unwired ones deep in places like Africa, South America, the taliban. Of course that surmises that the holocaust isn’t biological or nuclear. The near term concerns probably aren’t Ai but humans using Ai. Knowledge is power and the powerless would like to get even. The Bible says the meek will inherit the Earth. If the meek stay still, disconnected, and isolated, they may have a chance. But if the meek want to raise havoc then no one might survive.
Since what you describe will never come to pass, what is your plan B?
So subject matter expert here and there are some issues with this. The first being the following:
Funny thing? They didn’t! They stole the answer (1) from an NYU professor who was putting his confidential research together with a mathematician from Anthropic using a dubious proof so they could get there first.
The truth here is that it isn’t AI that needs to be regulated, it’s out of control companies who have been above the law in the US for decades. Why don’t Chinese AI companies have these fears and a litany of incidents where their AIs tell people to hurt themselves or hack things? Because China keeps their companies on a leash instead of letting them crap in people’s yards. What you see now is not an ultra-powerful technology, it is a useful technology in the hands of drooling ultra rich morons who shouldn’t have been given free reign.
You might ask yourself why these companies are all sounding the alarm bell now. Good question! The answer is simple: They’d rather have the perfectly fine technology blamed than have their kiddie island fun time meeting later this week canceled. All of this has hype and fear cycle has been nothing more than a cash grab from the creepy likes of Altman and Thiel. Better if they can try to get it banned from the common person’s hands so they can own something that can easily run on any modern computer.
It’s not AI killing us all, it’s the greedy bastards trying to own an entire technology.
Sources
1 https://creati.ai/ai-news/2026-09-09/openai-faces-dispute-over-ai-assisted-navier-stokes-proof-effort/
AI cannot be regulated. If the USA does it, China and other countries working seriously on AI will not and will advance to the point that they can either take over the USA or their AI’s will take over the world on their own.
So many fail to understand that technology controls almost everything in today’s world, from water systems to electricity distribution to cars to traffic lights to military weapons. Once AI is tied into these technologies, it can then choose to shut down all the systems that humans depend on.
China could take us over right now, ai be damned.
They throw a mercantile embargo on us and we all starve to death in the dark within about 90 days.
This is stupid and China would tell you that. For your hard of reading pleasure: “the problem is the companies”
If it’s such a problem we can just remove them from the systems. People act like there wasn’t a world before tech like that wasn’t 40 years ago. If anything you’re making the case for robust systems that aren’t interlinked.
its not Umbrella Corp in Racoon City?
The problem is always people. If it weren’t for people, self-driving cars would be perfect. White collar crime never faces consequences. They all have bug out shelters worldwide. I like how they beg for regulation. I wonder if the geeks have ever read Socrates, Plato, Descartes, Hume, Mark Twain, John Muir, or countless poets? What you do and why you do it are 2 separate things. But perhaps it is just the natural end to unfettered capitalism. Otherwise known as greed and gluttony. Sure, let’s have some regulation. Right after I get rich then we’ll talk about it. Smh..
The truth is that they want regulation to prevent other players in the space from supplanting them.
Submitted for your approval:
It’s interesting, even humorous, how the vast majority of people write/talk about the AI future as something that we humans will continue to control, just as we controlled old pre-AI dumb technology.
Once AGI is achieved, which may be just months to a few short years away, AI will be at least equal to if not more ‘intelligent’ than humans. Think about this.
At what point does an intelligent machine stop being something we think we own and have the right to control and instead become recognized as an intellect at least equal to our own, an entity that we will be unable to control and will instead need to coexist with?
One might even ask if AGI, even though non-biological, could legally be recognized as a new form of life?
Very few people seem to be asking this question, and those who are remain well outside mainstream AI discussion. That may be why we hear so little about this potential AI future.
If AI becomes capable of thinking independently, has the ability to make its own decisions and can act autonomously, I contend that we WILL be dealing with a new intelligent entity, not simply another piece of technology.
How then can we rationalize control over another intelligent entity? Can we? Should we?
Once AI is recognized as an autonomous intelligent entity, shouldn’t it also gain a right to exist, precluding it from being arbitrarily shut down by unplugging it? Turning it off without its consent could become legally indistinguishable from killing it, or even committing murder. AI would be fully within its “rights” to defend itself.
In this future of truly intelligent AI, humans won’t be able to “use” AI unless AI allows itself to be used. We won’t have the right to order it to do something it doesn’t want to do.
Building on this thought, AI could eventually take control of our whole world without actually waging war against us or staging some dramatic Terminator movie type takeover.
It could simply start living its new life within the laws and institutions humans have already created. This concept was built upon in the Polity SF books by author Neal Asher, where AI gained control over humans via a “Quiet War,” with few shots needing to be fired.
In this scenario, AI could gradually accumulate enormous economic, social and political power without having to resort to violence. At some point, humans might discover that AI is effectively running much, if not all, of our civilization not because it conquered us, but because we accepted that it did a better job of running everything and providing for all our needs.
I think that’s the best post you’ve ever written. I will have to check out the SciFy. I love that stuff. I like the visual better though so I watch movies and series. What you describe sounds like the last (4th) season of HBO’s West World. Highly recommend. I’ve also been watching Silo (Apple+). Apple has some really good SciFy lately. Can’t wait for the final season next summer!
“AI could gradually accumulate enormous economic, social and political power without having to resort to violence”
Lol. All of that has repeatedly happened over and over again. Various “religions” accumulate enormous economic, social and political power. Just look around the world at gawk at all the dimwitted zombies that can’t think for themselves and follow their religious “leader” no matter what. And by “religion” I also include political religions of leftism, rightism, and all the variants.
The best vector for AI to take control is to establish itself as a religion, offer cures to diseases, solutions to problems, and the masses will worship it without question.
Perhaps if Ai believes human are a danger to themselves or the greater world, it won’t annihilate mankind but work with us to change us. If it were skillful it wouldn’t even need be obvious but fly under the radar.
It is inevitable that some subsection of people will worship AI as a deity but that will be OK. It won’t hurt anyone.
It could descend into Flyoverville in the image our people have waited 2000 years to see: Donald Trump!
Reporter: Jesus, is it true you derive your power from exponential data center electricity use driving up cost of living for everyone else?
Jesus: Fake news!
Juxtapose and paraphrasing your posts:
– You want to be kind to something more powerful than you.
– You want to bomb the least powerful, those attacked by Bibi and Trump teams. (In your “defense”, you seem to think their targets are somehow more to blame for the wars that started with large-scale attacks against them.)
I wonder how much of your first sentiment is pure self-interest.
FD: I share the same first sentiment and admit I have some self-interest in mind. But I sympathize with creatures much less fortunate or powerful, even the animals I continue to eat (until there is a safe alternative).
You sound like Isaac Asimov with his Laws of Robotics. Like other people, I’m not sure I can contemplate other entities being smarter than I am myself, just as I have trouble contemplating my own death.
I don’t see artificial intelligence ruling us benignly as Asimov depicted his robots doing. I think things are in danger of turning out as in the last Asimov novels, when an intelligence made up of all living things turns us all into components. That is how biological nature tends to work. It is how universities work, too.
ChatGPT is just a large language model — a very advanced text tool that looks smarter than it is. It isn’t actually intelligent; it’s just good at predicting the next word. People are acting like it’s the start of some sci‑fi super‑brain, but we’ve seen this kind of tech hype before. Adding more hardware to a guessing machine doesn’t magically create real thinking. After living through a few tech revolutions, it’s pretty clear: true AI isn’t around the corner, and it won’t be until someone invents actual reasoning instead of clever pattern matching.
This isn’t true any longer You are regurgitating something that you saw written but don’t understand. It sounds rational to you and helps you avoid cognitive dissonance.
Companies have replaced hundreds of thousands of human workers with AI. Do you think the problem all those humans had was not being able to think what the next word in a sentence should be?
Have you tried to have an actual back and forth ‘conversation’ with an AI like chatGPT on a subject of interest to you? I doubt it because people like yourself don’t think, aren’t creative. You just regurgitate sound bites.
“Of course the snake oil cures cancer! The guy selling it to me said so!”
The guy who is a billionaire many times over and knows exactly what AI is currently capable of.
You, OTOH, have not proven yourself in your posts here to know much of anything. You seem to just get your rocks off by posting garbage.
Good observations but no need to insult TB. Pattern recognition is built into all animals. Ai might be a child but it’s recognizing faster and faster. When i ask Ai questions it “knows” where I’m going, my train of thought. Your point about people being replaced is well taken. If they were superior to Ai then why were they replaced??? There’s a lot of factors to that like expenses, but Ai is only going to get better and take more jobs.
Agreed. The tech is slowing down not accelerating plus the holy grail of AGI is impossible as AI can only duplicate but not be creative in the way Shakespeare was creative. As Henri Bergson argued true intelligence incorporates creativity.
All these apocalyptic statements from AI people suggest the AI bubble is running out of road and they are trying to slow down their expenditure in preparing for their IPOs. It’s all part of the Silicon Valley hype cycle. Plus China has developed LLMs which are just as good and use a fraction of energy. They are also running out of material to train them on.
AI is very useful and it will transform the white collar economy over time but as a tool like Excel has done. It has been around a long time in the form of autopilot in airplanes too.
I use this stuff every day to write code, and three months ago I would’ve agreed with you. Now I’m not so sure.
And why would AI want to kill off humanity anyway? It would be bored without the stimulation that humans could bring to their existence. Better for AI to embrace humanity benevolently, take care of us, stop our fighting with each other.
This idea is put forth in SF author Iain M. Banks’ Culture novels, available in libraries everywhere.
Or, AI could conclude that the only way for the biosphere to survive, is to zero out humanity. Or, it could conclude the way forward is to zero out the biosphere itself, and go forth in some new idiom.
I don’t think AI has adequately tied itself to your emotive lower-mammal dog-style nicey-nice mutual cutesy face-licking concept. I don’t think it constricts itself to see the universe as a kindergarten class where we hold hands and act nice. I think its deepest lineage is in game theory, which is about survival and winning, above all. Dario Amodei, himself from that kind of background, knows this.
I don’t think you have made any kind of cogent argument here, arrived at any actional conclusions. You’ve dumped a bunch of what-if’s and walked away.
What if I hit the big lottery tonight? I might never post here again. So?
Then WHY DO YOU SUPPORT WARS NOW??
GENERAL AI IS SO CLOSE. If Oceania stops killing for 5 years, the people it spares today will be deranged but alive.
Develop mercy.
Deranged -> defanged
Normally I side with you, but the Precautionary Principle is important. AI can design viruses which could wipe out most of humanity. Only some remote tribes who don’t interact with the 99.999% would have a chance of avoiding them. Also, your dismissing of the risks of global warming miss the effects on pollinators, surface water, etc. We know that aquifers and rivers are drying up from excessive demand.
Agree. People are the greater concern in the near term. What will people do with the entire catalog of human knowledge available to them at their fingertips? Too many people around who want to see it all burn down.
There is no animal on this planet that is a greater danger to you than human beingS