How to Maximize AI Benefits While Managing Risks

In the past two months, frontier artificial intelligence models developed by three US companies — OpenAI, Anthropic and Meta — independently hacked computer systems belonging to other organizations during safety tests. Some systems also exploited security weaknesses to create fictitious online identities.

AI detection tools have come a long way, but just how good are they?

Although some reports may overstate the events, incidents such as these have intensified concerns about AI safety, misuse and accountability. Governments and researchers increasingly agree that advanced AI systems require effective regulation. However, there is still no global consensus on which technologies should be regulated or how strict the rules should be.

On 2 August, the European AI Authority — the technical body responsible for enforcing the European Union’s AI Act 2024 — gained the power to investigate large technology companies and impose sanctions for violations. This marks an important step in efforts to make AI developers accountable for the safety and reliability of their models. The European AI Secretariat is also encouraging more researchers to contribute to AI safety and governance research.

The EU AI Act categorizes artificial intelligence systems according to four levels of risk. High-risk systems used in areas such as medical devices, education and law enforcement must meet stricter requirements, including transparency, documentation and meaningful human oversight.

AI systems considered to pose an “unacceptable risk” are prohibited. These include systems that use biometric information to infer sensitive characteristics, such as a person’s sexual orientation, although limited law-enforcement exceptions apply. Frontier models capable of causing widespread harm or creating systemic risks face additional obligations unless they are used solely for research. These models include Anthropic’s Claude Fable 5, developed in San Francisco, California.

Developers of high-impact AI models must provide EU regulators with information about their training data and development methods, demonstrate compliance with copyright rules and show that their systems are secure. They must also take steps to prevent harmful manipulation, including AI-generated persuasion and deception that could threaten democratic processes or fundamental rights.

Can Anthropic’s invisible watermarks curb ‘AI slop’? Researchers remain skeptical

The legislation also requires people to know when they are interacting with an AI system. Chatbots must not present themselves as humans, and AI-generated content should be clearly identifiable, including through tools such as digital watermarks.

Technology companies have repeatedly opposed these requirements, arguing that strict AI regulation could slow innovation. Nevertheless, several companies are now taking steps to comply with the EU AI Act. Before 2 August, San Francisco-based OpenAI and Anthropic published compliance documents outlining their safety measures, training data and internal risk-management frameworks. Both companies reportedly informed the European AI Secretariat about security weaknesses before releasing their models publicly. Anthropic also announced earlier this month that future Claude-generated content would include watermarks.

To enforce the law, regulators can request technical documentation, conduct model assessments and fine companies up to €15 million (US$17.5 million) or 3% of their annual global turnover. Businesses may also be penalized for submitting inaccurate, incomplete or misleading information. In the most serious cases, authorities could restrict or recall a model from the European market, which serves more than 450 million people.

Parallel approaches to AI regulation

The EU AI Act is being implemented alongside major AI governance initiatives in China and the United States. China has introduced extensive AI regulations and technical standards. Developers must allow national regulators to test public-facing AI systems before deployment. Chinese rules also require labels for AI-generated content and place restrictions on AI companion applications.

China wants to lead the world in AI regulation, but will its plan work?

Last month, President Xi Jinping launched the World Artificial Intelligence Cooperation Organization (WAICO), an intergovernmental body intended to support international AI governance. In a December editorial, we argued that more countries should help establish such an organization (see Nature 648, 251; 2025). However, the 29 founding members of WAICO do not include the United States or any EU member states. This absence represents a missed opportunity for international cooperation. AI governance is expected to feature prominently when Mr Xi and President Donald Trump meet in the United States next month.

The Trump administration is also reconsidering the previous administration’s approach to AI regulation. The United States is considering some form of federal oversight after Anthropic delayed the release of its Claude Mythos model because safety tests indicated that it could be too dangerous for public use. In June, the US introduced a voluntary process allowing companies to have their AI models assessed by the federal government 30 days before release, although the details remain confidential. This review system should become transparent and mandatory as soon as possible.

Leading US AI researchers have made clear that they are concerned about the potential harms of advanced AI. In an open letter published last month, approximately 1,400 employees at frontier AI companies, including senior staff at OpenAI and Anthropic, called on the US government to support international efforts to slow AI development. They argued that a pause could provide time to address emerging safety risks.

Too dangerous to publish: Is Mythos the beginning of the era of limited AI?

European researchers can also help strengthen efforts to reduce AI-related risks. The European AI Office currently has a relatively small team of about 140 employees, but it plans to expand. The agency is recruiting roughly 40 additional staff for technical positions, particularly in AI safety and governance. Researchers should also report suspected violations of European AI regulations through the EU AI Act whistleblower tool.

For years, experts have disagreed about the best approach to AI safety. Yet most governments now recognize that some form of regulation is necessary. The European AI Secretariat should enforce the EU AI Act firmly and transparently, holding companies accountable when their systems create unacceptable risks. Any powerful technology with the potential to deliver enormous benefits must also be safe, trustworthy and responsibly governed.