Published in eLife, the study could help scientists better understand how visual information is processed in the brain. In the future, the method may also allow researchers to compare how different species perceive the same environment.

Deciphering What the Brain Sees

Scientists have spent years investigating how the brain interprets signals from the eyes. In human studies, researchers have shown people images and movies while recording brain activity with functional magnetic resonance imaging (fMRI). They have then used those measurements to reconstruct visual information, sometimes down to individual pixels.

The new UCL study takes a different approach. Instead of relying on broad brain-imaging signals, the researchers recorded activity from individual brain cells in mice. These single-cell measurements provide a more detailed view of how visual information is represented in the brain.

Using activity recorded from the visual cortex, the researchers generated high-quality reconstructions of videos shown to the mice.

Lead author Dr Joel Bauer of UCL’s Sainsbury Wellcome Centre said the team wanted a better way to investigate how the brain interprets what it sees. Existing methods do not always generalize well to situations that have not been specifically tested, so the researchers aimed to develop a method that could capture what is represented in the brain and compare it with reality.

The researchers are particularly interested in the difference between what is physically in front of an animal’s eyes and how that information is represented in the brain. Studying this difference may reveal which visual features the brain emphasizes, modifies, or excludes.

How Neuron Activity Becomes Video

To reconstruct the videos, Dr Bauer and his colleagues used a dynamic neural encoding model originally developed by another research team for the 2023 Sensorium competition. The model is designed to predict how individual neurons respond while mice watch a movie. It also takes other factors into account, including animal movement and changes in pupil diameter.

The UCL researchers refined the approach using the same dataset. First, they calculated how the neurons were expected to behave while the mouse looked at a blank screen. They then compared those predictions with the actual neural activity recorded while the mice watched movies.

The researchers measured neural activity using microscopic imaging. This technique identifies active brain cells by detecting increases in local calcium levels.

Using the difference between the predicted and measured activity, the algorithm gradually changed the pixels of the initially blank movie. With each adjustment, the reconstructed video became more similar to the video viewed by the mouse.

Reconstructing an Unseen 10-Second Movie

After training the model, the researchers conducted more challenging tests. They recorded brain activity from mice watching videos that had not been included in the model’s training data.

Using only that neural activity, the system reconstructed a 10-second movie that was similar to the unseen video.

Dr Bauer said the approach achieved a high-quality reconstruction of a 10-second video clip. He added that including data from more individual neurons increased the accuracy of the reconstruction, highlighting the importance of comprehensive neural recordings.

The result suggests that the method was not simply memorizing videos shown during training. Instead, patterns of neural activity could be used to infer visual information from new scenes.

How Researchers Compared the Original and Reconstructed Videos

To evaluate the reconstruction, the researchers used a measurement called pixel correlation. This method compares corresponding pixels in the original and reconstructed movies.

The analysis found only a small difference in the timing of the two videos. However, the researchers said there is still considerable room to improve the image resolution and expand the range of visual scenes that can be reconstructed.

Future research will focus on collecting data that supports sharper reconstructions and represents a larger portion of what the animal sees.

Why the Brain Does Not Simply Record Reality

The researchers plan to use the technology to investigate a deeper question about vision: how different the brain’s internal representation of the world is from the physical scene in front of us.

Vision is more than a recording process like a camera. The brain continually interprets, filters, and modifies incoming sensory information. Understanding where and how these changes occur could reveal important principles of perception.

Dr Bauer explained that people do not have a perfect representation of the world in their heads. The visual processing system changes information as it builds an internal representation of reality. According to the researchers, this misalignment is not necessarily an error. It may be a feature of perception that reflects how the brain interprets and extends sensory information.