LightTok: A New AI Vision Chip Could Help Drones Use Less Energy

Chinese researchers have developed a sensor that could reduce the energy autonomous systems use to process visual data. The new two-dimensional chip, called LightTok, converts light directly into the digital “tokens” used by artificial intelligence (AI) models.

In a study published Aug. 19 in Nature Sensors, scientists described a chip architecture designed to eliminate several energy-intensive steps used by traditional image-processing systems. The researchers say LightTok could eventually help drones, robots and other edge devices process visual information more efficiently.

“Our design idea was to move token generation to the sensor itself, allowing the chip to directly generate tokens that the AI model can process when light reaches the sensor,” said Miao Feng, director of the Brain-Inspired Intelligence Institute at Nanjing University. “These tokens contain complete image information.”

How traditional visual recognition wastes energy

In a conventional visual-recognition system, incoming light passes through several stages before an AI model can interpret it. An image sensor first captures the light, while an analog-to-digital converter transforms the signal into pixels. The data is then temporarily stored and transferred to another chip, where the image is divided into square sections, or “patches.” Each patch is converted into a token for the AI model to analyze.

A widely cited study found that analog-to-digital converters account for an average of 66% of an image sensor’s energy consumption. Moving visual data to the cloud for processing can increase total energy use even further.

How the LightTok chip works

LightTok combines five processing stages by integrating sensing, memory and computation within the same pixel. The researchers built the chip using an array based on single-layer molybdenum disulfide floating-gate phototransistors.

These components can detect light, retain information about what they detect and use that information in calculations. By processing data at the sensor, LightTok physically eliminates the need to move large amounts of visual information between separate components—a major source of energy waste.

Molybdenum disulfide is a two-dimensional material that is highly sensitive to light and can be produced in sheets only one atom thick. A phototransistor converts incoming photons into electrical current, while a floating gate can trap and retain electrical charge after the light is switched off.

“Light comes in and tokens come out,” said Liang Shijun, a professor of physics at Nanjing University, in comments summarized by China’s state news agency, Xinhua. “This chip physically eliminates data movement, which is a major source of energy waste.”

LightTok reaches 87.3% image-recognition accuracy

During testing, LightTok achieved 87.3% accuracy in image recognition. The researchers also reported that the chip was 10 times more energy efficient at converting light into tokens than the traditional multistep process described in the study.

That efficiency could be especially valuable for robots, drones and other autonomous devices that operate with limited battery capacity. Processing visual information locally may reduce the need to transfer data to another chip or remote cloud server.

The chip is still far below smartphone-camera resolution

LightTok currently has a maximum resolution of just 32 × 32 light-sensitive pixels. That is far lower than the resolution of modern smartphone cameras and would not yet meet the visual requirements of many drones or autonomous systems.

However, Miao said the technology could potentially be scaled using complementary metal-oxide semiconductor manufacturing processes—the same general manufacturing technology used for chips in smartphones and laptops, as well as sensors in drones.

Could LightTok extend drone flight times?

If the technology can be scaled successfully, the researchers believe it could improve remote-sensing systems. For example, drones surveying disaster areas or remote locations might be able to operate for longer periods because they would spend less energy processing visual data.

Kumar Sokka, CEO of Acre Security, a company that provides real-world sensing technology for critical infrastructure, described the work as a “small-scale demonstration.” He said the direction of the research is important for companies working with physical-world AI.

Sokka, who was not involved in the study and previously worked for industrial automation company Rockwell Automation, said AI research has largely focused on models. A major challenge, he explained, is converting what a sensor sees into a format an AI model can use at the location where the sensing occurs.

Turning raw data such as light into tokens can require substantial energy, which is inefficient for robots and edge devices operating with tight power budgets. Processing information directly at the point of detection could support the growth of physical AI, Sokka said, although it would not solve every energy-related challenge.