Dr. Mario did not prepare us for this. In a pioneering effort, researchers from UC San Francisco and UC Berkeley, in partnership with Edinburgh-based Speech Graphics, have devised a groundbreaking communications system that allows a woman, paralyzed by stroke, to speak freely through a digital avatar she controls with a brain-computer interface.
Brain-Computer Interfaces (BCIs) are devices that monitor the analog signals produced by your gray matter and convert them into the digital signals that computers understand — like a mixing soundboard’s DAC unit but what fits inside your skull. This technology has the potential to transform the lives of individuals with paralysis or communication disorders.
For this particular study, researchers led by Dr. Edward Chang, chair of neurological surgery at UCSF, implanted a 253-pin electrode array into the speech center of the patient’s brain. These probes monitored and captured the electrical signals that would have otherwise driven the muscles in her jaw, lips, and tongue. Instead of controlling her physical movements, these signals were transmitted through a cabled port in her skull to a bank of processors. Within this computing stack, a machine learning AI was able to recognize and interpret the patient’s electrical signal patterns for more than 1,000 words after a few weeks of training.
But that’s only half of the breakthrough. Through the AI interface, the patient can now write out her responses, similar to how Synchron’s system works for individuals suffering from locked-in syndrome. However, she can also “speak” using a synthesized voice trained on recordings of her natural voice from before she was paralyzed. This allows her to communicate more dynamically and express herself beyond just written communication.
Related Post
To further enhance the patient’s ability to communicate, the researchers collaborated with Speech Graphics, the same company that developed photorealistic facial animation technology for popular video games. Speech Graphics’ technology “reverse engineers” the necessary musculoskeletal movements a face would make based on analysis of the audio input. This data is then fed in real-time to a game engine, which animates the patient’s digital avatar. The mental signals from the patient are mapped directly to the avatar, enabling her to not only speak but also express emotions and communicate nonverbally.
Creating a digital avatar that can speak and emote in real-time, connected directly to the subject’s brain, shows the potential for AI-driven faces well beyond video games. Facial communication is intrinsic to being human, and restoring this ability through the use of technology provides a sense of embodiment and control to patients who have lost it.
BCI technology has been slowly developing since the early 1970s, but recent advancements in processing and computing systems have reinvigorated the field. Several well-funded startups are currently vying to be the first through the FDA’s regulatory device approval process. Synchron made headlines last year when it successfully implanted a BCI in a human patient, while Elon Musk’s Neuralink entered restricted FDA trials earlier this year. The field is rapidly progressing, and these advancements have the potential to significantly improve the quality of life for individuals with communication disabilities.
In conclusion, the collaboration between researchers from UC San Francisco, UC Berkeley, and Speech Graphics has led to the development of a groundbreaking communications system. By utilizing a brain-computer interface, a paralyzed stroke patient can now speak freely through a digital avatar. This technology not only restores the ability to communicate verbally but also allows for nonverbal expressions and emotions. With further advancements in BCI technology, we may witness an entirely new way for individuals with communication disabilities to express themselves and regain control over their lives.
