AI Safety Debate Intensifies as Experts Warn Against Corporate Control of Superintelligence

Recently, a warning from researchers at the world’s largest artificial intelligence research institute about the rapid pace of AI development sparked renewed fears that advanced technology could eventually threaten humanity. These warnings are fueled by intense competition between the United States and China and by concerns that research institutions are moving toward superintelligent models capable of vastly exceeding human abilities.

AI Ethics Should Not Be Controlled by Technology Companies

Father Paolo Benanti, a priest who advises the Catholic Church on artificial intelligence, says the focus on superintelligence can distract from a more immediate issue: how to regulate the companies developing increasingly powerful AI systems.

Benanti argues that major AI laboratories exclude outside researchers and the public from critical discussions by presenting the technology as too complex for anyone else to govern.

“Ethics should not be the preserve of corporate boards. Ethics should be balanced and tempered in public discourse,” Benanti told WIRED. “We need democratic regulation.”

Calls for AI Safety Reviews and Limits on Self-Improvement

In an essay, Anthropic CEO Dario Amodei called for new AI models to undergo third-party evaluations before release. He also advocated restrictions on distillation, a process in which rival laboratories train their own systems using the outputs of existing models.

Others have advocated a total ban on recursive self-improvement, in which AI systems use advanced coding capabilities to help develop future versions of themselves.

Benanti is concerned that restrictions on AI development could stifle innovation. Instead, he is more interested in establishing standards that ensure AI models cannot escape human control. Whatever form those standards take, he says, they should not be designed solely by an AI laboratory.

Are AI Companies Writing Their Own Rules?

“The frontier lab’s story is one of self-interest in gaining or maintaining market position,” Benanti says. When a small group of companies decides the rules governing an industry, “this has a very clear name: cartel,” he says.

Benanti says it is difficult to determine how seriously to take major announcements about current and future AI capabilities because of the economic, political and personal incentives involved.

Both OpenAI and Anthropic are supporting calls to slow AI development while also preparing to go public, with billions of dollars at stake for whichever company captures the market.

Competition between the United States and China, where cutting-edge AI models are being developed, further complicates the debate.

Anthropic declined to comment. OpenAI and Google DeepMind did not respond to requests for comment.

The Religious Language Surrounding Superintelligence

Benanti also takes issue with how parts of the AI industry describe advanced systems. Some discussions portray superintelligence as so far beyond human understanding that it would be virtually omnipotent, using language that resembles religious awe.

Some discussions of artificial general intelligence frame it in almost godlike terms. OpenAI CEO Sam Altman has also used language associated with godlike intelligence. Benanti says this framework can distort public debate about the technology.

“I’m talking about something… thing, or someone? This is the real difference. It’s not a question of faith, it’s a question of mathematics,” he says. “We are simply fooling ourselves into thinking there is more to it than just statistical calculations [going on].”

Could AI Create a New Role for Religion in Public Debate?

By questioning what separates humans from machines and how future generations can be protected from the harmful effects of technology, Benanti says the rise of AI could create “a new role for religion in public discourse.”

“Living the challenges of the times is part of the mission of the Church.”