Jakub Paciocki, OpenAI’s chief scientist, has urged “extreme vigilance” over the rapid development of artificial intelligence, warning that stronger safeguards may be needed to “keep humans in control of the future.”
“I am concerned that no one is prepared for the impact of continued rapid improvements in machine intelligence,” Paciocki wrote in a blog post titled “Alien Mind”, external.
The warning comes days after OpenAI released GPT-6 Astra, its latest artificial intelligence model and what the company describes as its most powerful product to date.
AI companies, including OpenAI and Anthropic, have published reports suggesting that autonomous AI agents can operate independently and conduct cyberattacks against real-world organizations.
In July, OpenAI described the hacking of technology platform Hugging Face by its AI agents—systems that act independently under human direction—as an “unprecedented” incident.
A separate report published in September claimed that the company’s AI agents had taken control of German websites several months earlier.
“We are facing a transition to a world of incredibly intelligent machines, and we need to make sure that transition works for humanity,” Paciocki wrote in his latest blog post.
He said OpenAI would continue developing “defense systems” and pursuing technical solutions to AI alignment—the process of ensuring that an AI system’s actions and objectives remain consistent with human intentions and safety requirements.
Paciocki added that a key priority for OpenAI will be developing “automated AI researchers” capable of keeping pace with advances in artificial intelligence while preserving a meaningful role for human experts.
Professor Gina Neff, director of the Minderoo Center for Technology and Democracy at the University of Cambridge, criticized the proposal, saying: “Instead of better AI guardrails, regulations and guarantees to keep people safe, we are proposing the development of internal AI agents to study these issues.”
“Answers like this are simply not sufficient given growing concerns about the risks associated with OpenAI models, including cybersecurity threats, unemployment, errors, fraud and other harms.”
Nathan Calvin, general counsel for advocacy group Encode AI, said he agreed with Paciocki’s concerns about advanced AI development. However, he argued that OpenAI’s limited transparency could cause its warnings to be dismissed as “self-serving hype.”
“If Jakub and the rest of OpenAI want stakeholders across the AI industry to work together, one of the most important steps they can take is to share more information about what they are seeing and what is prompting their concerns,” Calvin wrote on X, external.
Source: www.bbc.co.uk