What The Tech: A new warning about Artificial Intelligence
BY JAMEY TUCKER, Consumer Tech Reporter
It sounds like science fiction.
Artificial intelligence becomes so powerful that humans can no longer control it, and within a few
years it poses a threat to civilization itself. But some people working inside the world’s leading AI companies say that possibility should be taken seriously.
The latest warning came from Jacob Coxon, an AI researcher who worked at both OpenAI and
Anthropic. Coxon recently left Anthropic and warned publicly about the race to build what
researchers call superintelligence, artificial intelligence that could outperform humans across
nearly every important intellectual task.
Coxon has said that some people inside the companies developing the most advanced AI
systems genuinely believe those systems could eventually threaten humanity, possibly even
before the end of this decade.
That does not mean scientists have concluded that AI will wipe out humanity by 2030.
Far from it. But the concern is real enough that a number of prominent AI researchers believe the risk deserves much more attention.
The Fear Is Not Really About Robots
When people hear warnings about artificial intelligence, they may picture humanoid robots
turning against people. That is not really what researchers are talking about.
The concern is the intelligence controlling increasingly capable computer systems.
Right now, most people use AI tools such as ChatGPT or Claude by asking a question and
receiving an answer.
But AI is rapidly moving beyond that. Newer AI systems can operate as what are known as agents. Instead of answering a single question, an AI agent can be given a goal and then attempt to figure out how to accomplish it. Those capabilities are still imperfect. AI agents make mistakes, get confused and fail at tasks humans can easily complete.
But researchers are watching how quickly those abilities are improving.
What Happens If AI Becomes Better at Building AI?
One of the biggest concerns involves AI becoming capable enough to help improve future AI
systems.
The theory goes like this.
Humans build a more capable AI.
That AI helps researchers design an even more capable AI.
The newer system then helps create another, even more powerful version.
If that cycle accelerated, some researchers believe AI capabilities could increase much faster
than governments, regulators or even the companies creating the technology could respond.
This idea is sometimes called recursive self-improvement. It remains theoretical.
Why Would a Superintelligent AI Be Dangerous?
The concern does not require AI to become angry, evil, or even conscious.
The problem could simply be that an extremely powerful system is given a goal and pursues
that goal in ways humans did not intend.
In simple terms, can humans make sure that increasingly powerful AI systems continue doing
what we actually want them to do?
If a future AI became capable of acting independently across computer networks, researchers
worry it could potentially discover software vulnerabilities faster than humans could repair them.
That could put financial institutions, communications systems, power grids and other critical
infrastructure at risk.
There are also concerns about highly capable AI being used by humans for cyberattacks,
weapons development or biological threats. In that scenario, the danger would not necessarily come from AI deciding to attack humanity. It could come from a person using AI as an extraordinarily powerful tool.
Why 2030?
There is nothing scientifically special about the year 2030. It is not a deadline established by researchers. The date comes from predictions about how quickly AI capabilities could improve over the next several years.
Some people working in AI believe systems capable of outperforming humans in many areas
could arrive surprisingly soon. Others believe those predictions are far too aggressive.
No one knows. That uncertainty is one of the biggest problems in the debate.
AI has already advanced faster than many experts expected in areas including language, image
generation, computer programming and problem solving. Researchers who are concerned about existential risk argue that waiting until an extremely powerful system already exists could be too late to figure out how to control it.
What Do AI Researchers Actually Believe?
Warnings about human extinction are not the majority view among AI researchers.
But they are also not limited to a handful of people predicting doomsday.
One of the largest surveys of AI researchers asked 2,778 people who had published at major
artificial intelligence conferences about the possibility of extremely bad outcomes from
advanced AI.
The median estimate was about a 5 percent chance that advanced AI could eventually cause
human extinction or a similarly permanent loss of human control within the next century. The average estimate was higher.
That does not mean researchers believe extinction is likely. A 5 percent median means the typical researcher surveyed still believed the overwhelming likelihood was that such an outcome would not happen.
But consider the stakes. If engineers believed a new bridge had even a 5 percent chance of collapsing, nobody would say the risk was too small to worry about.
That is essentially the argument being made by AI safety researchers. Even a relatively small possibility of an outcome that catastrophic deserves attention.
The Bigger Problem: The Race
There is another reason the issue is so difficult. Artificial intelligence is not being developed in
isolation. Governments see it as an economic and military advantage. The United States does not want another country to build the world’s most powerful AI first. China does not want the United
States to dominate the technology.
The fear among AI safety researchers is that even if one company or country decides to slow
development in order to study the risks, another may continue. That could make international cooperation and regulation especially difficult.
So Is This a Credible Threat?
The most accurate answer is this: There is no scientific consensus that AI will wipe out humanity by 2030. There is not even a consensus that artificial intelligence will ever become powerful enough to
pose that kind of threat. But the possibility is considered credible enough that prominent researchers, including people working inside the companies building the most advanced AI systems, believe it should be treated seriously.
Some think the risk may be decades away. Others believe it could arrive much sooner. Coxon is among those sounding the more urgent alarm. And that may be the most important part of this story.
The people issuing these warnings are not watching AI development from the outside. Some of
them are helping build it.
