How AI Reconstructs Images From Brain Scans More Accurately

Several teams are developing tools that recreate images from brain-scan data. However, Irani says existing approaches are not accurate enough. If a person looks at a banana, for example, these models may generate an image of a banana—but the structure and location of the object may be different from what the person actually saw.

Building a More Accurate Brain Decoder

The team set out to reconstruct previously viewed images more accurately. To train the AI model, they used existing data from eight people who each viewed approximately 9,000 images while undergoing high-resolution fMRI scans.

Their “brain decoder” has two branches. One predicts the structure of an image, such as where its colors appear. The other predicts the image’s content, such as a bunch of bananas on a plate. These predictions help diffusion models—AI systems known for creating images and videos by gradually removing noise from pixels—produce a more accurate representation of what a person sees.

However, improving the model required far more data than was available from the fMRI scans.

Using an Encoder to Expand the Training Data

To address this limitation, Irani and her colleagues trained a second model in the opposite direction: an encoder that predicts brain activity from an image. The team then used the encoder and decoder together to improve both tools.

For example, they could begin with a new image of a leopard. The encoder predicts what a person’s fMRI scan might look like while viewing that image. The decoder then uses that predicted brain activity to reconstruct the image. At first, the result may not look much like a leopard, Irani says. But repeatedly training the models in this way eventually leads to dramatic improvements.

This method also allows the team to train the system on as many images as needed, even when those images were never shown to a person inside an fMRI scanner. About 70% of the training data consists of images that were not originally paired with fMRI scans.

What Brain Image Reconstruction Could Reveal

By combining data from multiple studies, the researchers also identified brain regions that appear to share similar functions across individuals. Some regions seemed to respond to images of food, while others responded to images of sports.

Irani, a computer scientist, is now working with neuroscientists to determine whether these tools can help reveal new information about how the brain processes visual images.

Other research teams are exploring similar brain-decoding tools, including one approach, another method and a related tool.