Scientists in China are examining what it would take to create a powerful self-improving artificial intelligence system—potentially the “last man-made AI”—that could develop its own successor. In a new, non-peer-reviewed paper, the researchers outline a framework for recursive self-improvement and consider whether AI systems could eventually direct their own training, research and development with minimal human input.
Recursive self-improvement (RSI) refers to the idea that an AI system could use what it learns during one round of development to guide the next. In principle, this feedback loop could allow AI systems to produce increasingly capable generations of models.
In a study uploaded to the preprint database arXiv on September 10, the researchers proposed a framework that categorizes AI systems from Level 1, the lowest, to Level 5, the highest, according to their ability to improve themselves.
From human-led development to “meta-improvements”
At the lower levels, AI development remains primarily human-driven. As systems become more capable, they could identify their own weaknesses and decide how to address them. Eventually, they might be able to make “meta-improvements”—changes to the way they improve themselves.
Whether AI can achieve sustained self-improvement at this level remains an open question. Previous experiments, however, have demonstrated more limited forms of RSI.
Earlier this year, an AI system called A-Evolve-Training was tasked with improving the performance of a 30-billion-parameter large language model (LLM), according to a study. Over four rounds, A-Evolve-Training made slight improvements to the LLM’s benchmark performance by testing different training techniques and using the results to guide subsequent experiments.
Peter Bentley, a professor of digital biology at University College London, questioned whether experiments of this kind provide a reliable route to RSI.
“This idea that some AIs will magically start generating better AIs themselves — that’s nonsense,” Bentley told Live Science in an email.
Bentley noted that self-improving computer systems have existed for decades. Evolutionary computation and reinforcement learning, for example, are both based on the principle that systems can improve their performance over time in response to feedback.
An even greater challenge, Bentley said, would be developing AI systems capable of carrying out the research needed to produce stronger successors. Such systems would also need to overcome their dependence on human expertise and physical computing infrastructure.
“You can’t do ‘recursive self-improvement’ unless you can enhance your own training, enhance your own hardware, and test thoroughly in real-world settings,” Bentley added.
The challenge of true RSI
The paper’s authors acknowledge several of these difficulties. Developing an RSI-enabled system may require huge investments in computing power. Human oversight was also used to ensure that continuous improvements produced meaningful benefits.
The authors point out that the process becomes effectively redundant if the resources needed to develop and test subsequent AI models outweigh the benefits those models provide.
Researchers also need reliable ways to determine whether an AI system’s capabilities are genuinely improving through successive iterations, rather than whether the system is exploiting weaknesses in the tests used to measure it. This is a known issue in AI development: models can achieve higher benchmark scores by carefully selecting or optimizing for the testing methods and results.
Kayla Blomquist, a University of Oxford AI researcher and co-founder of the Oxford China Policy Institute, said the paper does not provide a practical roadmap to RSI. However, she said it could help developers and policymakers understand the technical challenges and safety risks associated with increasingly autonomous AI systems.
Blomquist also questioned whether there is a clear point at which AI becomes completely independent of human development.
“There may never be an unambiguous ‘last’ model constructed by humans, as the paper states that the handover of responsibility is gradual rather than instantaneous,” Blomquist added.
Source: www.livescience.com