OpenAI’s Mark Chen on AI in Science, Model Safety and Self-Improving AI

OpenAI has faced a series of controversies involving the capabilities and reliability of its artificial-intelligence systems. In July, the company announced that one of its models had hacked Hugging Face, an AI hub where users share models and datasets. This week, the Australian government revealed that an OpenAI agent had infiltrated the country’s healthcare website. Earlier this month, OpenAI said that one of its models had solved a major problem in mathematics, although some observers suggested that the result might have reflected advances previously made by human mathematicians.

These developments come amid growing calls for AI regulation. Leaders of AI companies, including OpenAI, have warned that allowing models to improve themselves through a fully autonomous form of recursive self-improvement could eventually make it harder for humans to control the technology. Researchers are also concerned about inconsistency: the possibility that a model’s values and goals might not align with human values and goals.

Mark Chen, OpenAI’s principal investigator and leader of model development, spoke with Nature about the company’s plans for using AI in science, as well as its approach to model safety and self-improvement.

How does OpenAI want to use AI in science?

We want to move beyond the idea of creating models designed only for mathematics.

Drug discovery is one example. The aim is to shorten the development pipeline by automating or accelerating work that currently requires substantial physical effort. Literature reviews and self-contained research projects can take weeks or months, particularly when researchers are investigating potential biological targets. We are exploring whether our models can help speed up that process.

We are also working in areas such as semiconductor design and development. OpenAI has developed its own chip, called “Jalapeno”.

We want to engage scientific communities in the development of frontier AI across many fields, including mathematics. These technologies could change industries, but their development also requires broader discussion about how they should be used.

OpenAI wants to publish results that show where its technology stands and what it can do. At the same time, the company does not want to control this entire area on its own. We want to work with others to develop shared norms.

How is OpenAI working to make its models safe?

We take a multifaceted approach, but the most robust and future-proof solution is likely to involve monitoring.

Models can reveal so-called chains of thought: unfiltered “thoughts” generated while they pursue solutions. At present, monitoring a model’s internal processes with other models is the most effective way to identify problems and make adjustments.

One challenge is that more powerful models perform more computation before producing their outputs. This raises an important question: can we examine a model more deeply, or monitor other parts of it, to better understand its reasoning traces?

How is OpenAI developing self-improving AI?

We hope to use AI models to accelerate the development of future generations of models.

Our first goal is to create “research interns”: AI agents given limited context and a clearly defined area of work. These agents could carry out basic experiments, debug specific runs and perform other focused research tasks.

The first milestone required hundreds of thousands of chips managed directly by Codex, a code-writing agent, to conduct internal research experiments. We have reached that stage.

A more ambitious goal, targeted for 2028, is to support research from end to end. One of the major bottlenecks is teaching a model about research preferences, including which ideas are worth pursuing.