Anthropic’s initial public offering prospectus offers an unusually detailed look at the AI company’s concerns about the technology’s future. Although the filing includes financial information, its risk-factor section is the largest focus so far, spanning 94 pages and describing dangers that could disrupt Anthropic’s plans—or, in the company’s framing, threaten humanity itself.
Anthropic identifies recursive self-improvement as a key risk
The prospectus addresses the broader debate over whether artificial intelligence could pose an existential threat to humanity. While the precise mechanisms remain unclear, the filing identifies recursive self-improvement (RSI) as one possible path to catastrophic outcomes.
“Sufficiently advanced AI systems could pose catastrophic or existential risks to humanity if not developed, deployed, and operated with appropriate safeguards, monitoring, and security measures,” the prospectus states, citing concerns raised by critics, regulators and people within Anthropic. This is the first of three appearances of the phrase “existential risk” in the filing.
RSI describes a process in which AI systems take over increasing portions of AI research and development. AI already plays a significant role in developing new models, and at least one laboratory is using an approach resembling RSI to accelerate the creation of a model that performs beyond expectations. The more extreme scenario is one in which humans are no longer involved in the process at all.
“The risks will increase as AI systems substantially automate AI research and development, for example, if AI advances accelerate beyond the ability to evaluate and protect new models,” Anthropic’s filing says.
Safeguards could fall behind AI development
The prospectus warns that recursive self-improvement could allow AI models to advance faster than Anthropic’s safeguards. The company might not recognize that it had lost that race until it was too late. Anthropic did not immediately respond to a request for comment.
The filing also describes what could happen if increasingly autonomous systems operate with less human oversight and can take consequential actions. “As AI systems become more autonomous, require less human oversight, and are able to take consequential actions, there is a risk that such systems will independently take harmful actions at scale without human intervention, whether due to misalignment with human intent, misspecification of purpose, or self-preservation behavior,” it states.
That scenario evokes the familiar fictional “Skynet” vision of autonomous AI acting against human interests.
Source: www.wired.com