Neuroengineers at Columbia University have developed an AI-enabled technology that can translate thoughts into intelligible and recognizable speech, a breakthrough that could eventually help people who cannot speak.
Thoughts To Speech System
For the study published in the journal Scientific Reports on Jan. 29, Nima Mesgarani, from Columbia University's Mortimer B. Zuckerman Mind Brain Behavior Institute, and colleagues involved epilepsy patients who undergo regular brain surgeries to train a vocoder.
A vocoder is a computer algorithm that can synthesize speech after being trained on recordings of people talking. It is the same technology used by Apple's Siri and Amazon Echo, which can learn to recreate speech from recording, and give verbal responses to questions.
Instead of learning from recordings, however, the study's vocoder needed to learn from brain activity. To do this, the researchers asked the patients to listen to sentences spoken by different people while they measured the patterns of their brain activity.
Once the system became familiar with the brain activities, the researchers asked the patients to listen to a recording of a person reciting numbers zero through nine, recorded their brain activity and ran this through the vocoder.
Using neural networks, a type of artificial intelligence that mimics the structures of the human brain cells, the researchers analyzed and cleaned up the sound produced by the vocoder in response to the signals. The resulting sound was a robotic voice that recited a sequence of numbers.
System To Benefit People Who Lost Ability To Speak
The researchers want to test more complicated words and sentences next. They also want to run tests on brain signals produced when a person speaks or imagines speaking.
Their ultimate hope is for the system to become part of an implant, similar to those used by epilepsy patients, that can translate the wearer's thoughts into words.
"In this scenario, if the wearer thinks 'I need a glass of water,' our system could take the brain signals generated by that thought, and turn them into synthesized, verbal speech," Mesgarani said. "This would be a game changer. It would give anyone who has lost their ability to speak, whether through injury or disease, the renewed chance to connect to the world around them."
