Brain-Computer Interface Decodes Speech and Gestures Simultaneously

Human communication involves more than speech. We nod, shrug, wave, point and use other gestures to express meaning. For people with severe paralysis, brain-computer interfaces (BCIs) can help convey intended sounds and movements. Now, researchers have shown that a single brain implant can decode both at the same time.

The system uses artificial intelligence to translate electrical activity in the brain into text displayed on a screen while also controlling movements of a personalized animated avatar. It can process verbal and non-verbal communication within seconds of the user’s intended action.¹

Recent BCI research has enabled people with paralysis to communicate² and control a computer cursor³ or robotic arm.⁴ However, most studies have focused on restoring one of these functions at a time, says study co-author Samantha Brosler, a doctoral student in bioengineering at the University of California, San Francisco.

Although the system is still a proof of concept, it could represent a step towards a more versatile BCI that captures more of the expressiveness involved in human communication, Brosler says.

“This gives us hope for neuroprostheses in everyday use,” says Christian Helf, a computational neuroscientist at Maastricht University in the Netherlands.

Why decoding speech and gestures is difficult

Decoding gestures and speech simultaneously is challenging because some of the neural signals involved overlap. Neural activity can also differ when a person speaks and gestures at the same time compared with when they perform either action alone.

To distinguish between the signals, Brosler and her colleagues worked with two participants. The BCI used an array of 253 electrodes implanted on the surface of the brain’s cortex. The array covered a broad area of the sensorimotor cortex, allowing the researchers to record signals associated with both speech and body movement.

Testing the brain-computer interface

The first participant had speech and motor disabilities resulting from a brain-stem stroke. The researchers asked him to silently attempt five phrases and perform gestures including waving, nodding, shaking hands and clapping.

The second participant had amyotrophic lateral sclerosis, a type of motor-neuron disease that caused progressive speech paralysis and movement disorders. At the time of the study, he could type messages on a computer and phone. The researchers asked him to imagine saying 10 phrases and making 10 gestures, such as shrugging or giving a thumbs-up, while keeping his body still apart from his facial muscles.

The BCI recorded neural activity as the participants attempted to speak or gesture separately and as they combined speech with gestures. Examples included waving while saying “hello” and nodding while saying “yes.”