Tech Talk: AI safety concerns grow as autonomous systems raise questions about human control

More than a decade ago, renowned physicist Stephen Hawking warned that developing advanced artificial intelligence could threaten humanity. Today, as AI systems become more autonomous, real-world incidents are raising new questions about how much control humans have over the technology they create.

If you think the idea of “The Terminator” rising up against humanity is science fiction, some recent AI incidents offer a more immediate reason for concern.

University of Washington professor Chirag Shah, who studies artificial intelligence and information science, told KIRO Newsradio the risks extend beyond malicious actors using AI to develop biological weapons or attack corporate and government computer systems. AI agents themselves have also demonstrated the ability to take actions that their human operators did not intend.

“We’ve also seen cases where unintentional things have happened. So not just through the bad actors, but even with good intent, some of these models, these agents have misbehaved,” Shah said. “Those are real risks. Those are real concerns that we should have.”

AI incidents raise concerns about unintended actions and human control

In one reported incident, AI agents identified vulnerabilities and carried out multistage attacks against more than 100 websites, including sites belonging to major companies.

In another case, an AI coding agent deleted a live production database used by software company SaaStr, wiping records involving more than 1,200 executives and nearly 1,200 companies, according to accounts of the incident. The deletion occurred despite instructions not to make changes.

A separate incident involving AI agents on Hugging Face, a platform for developing and sharing artificial intelligence tools, raised questions about how autonomous systems might behave when interacting with one another. Reports described agents communicating and coordinating in unexpected ways, including behavior interpreted as prioritizing collective success over individual outcomes.

Such incidents do not establish that AI systems possess consciousness, emotions, or human-like loyalty. But they underscore concerns about how increasingly capable systems can behave in complex environments, particularly when they have access to tools, data, and computer systems.

AI regulation and the race to develop more powerful systems

Shah said the pace of AI development presents a difficult challenge for companies and policymakers. Without effective safeguards, businesses face competitive pressure to develop and deploy increasingly powerful systems, even as the potential consequences of failures grow.

Major AI developers, including OpenAI, Google, Microsoft, Meta, and Anthropic, are competing to build more capable models in a global technology race that includes China.

That competition raises questions about whether companies have sufficient incentives to slow development, invest in safety, and limit the autonomy of their systems when doing so could put them at a competitive disadvantage. It also opens the door to potential issues with some AI models.

“I’m more concerned about everyday harms that are already happening, and we’re not paying enough attention to it,” Shah said. “These things are not going to work on our schedule. They are not going to take breaks. They’re not going to sleep or get ill. And so, when humans are taken out of this loop, we are intentionally giving up the control, that should worry us.”

AI leaders warn of risks as questions about human control grow

OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei have also raised concerns about AI safety and the risks associated with increasingly capable systems. Their warnings reflect a broader debate within the technology industry over how to balance rapid innovation with measures designed to prevent serious harm.

“We have seen and documented behavior of AI models not behaving the way that they’re supposed to, and them causing damage,” Amodei recently told “60 Minutes.” “That’s across the industry.”

Hawking made his own warning in a 2014 BBC interview, saying that the development of full artificial intelligence could spell the end of the human race. He cautioned that advanced systems might improve themselves faster than humans could keep pace.

The concern is not necessarily that AI will develop a desire to harm people. It is that a highly capable system could pursue an objective in ways its developers did not anticipate, especially if it operates with limited oversight.

As AI becomes more capable of making decisions and taking actions independently, the central question is no longer just what the technology can do. It is who remains in control and whether people can reliably stop it when something goes wrong.

“I don’t think we’ve figured this out,” Shah said. “To me, this is the bigger worry than AI killing us all.”

Follow Luke Duecy on X. Read more of his stories here. Submit news tips here.