He has not been able to speak since 2003 when he was paralysed at the age of 20 by a severe stroke after a terrible car crash.
Now, in a scientific milestone, researchers have tapped into the speech areas of his brain— allowing him to produce comprehensible words and sentences simply by trying to say them. When the man, known by his nickname, Pancho, tries to speak, electrodes implanted in his brain transmit signals to a computer that displays them on the screen.
His first recognisable sentence, researchers said, was, “My family is outside.”
The achievement, published recently in the New England Journal of Medicine, could eventually help many patients with conditions that steal their ability to talk.
“This is farther than we’ve ever imagined we could go,” said Melanie Fried-Oken, a professor of neurology and paediatrics at Oregon Health & Science University, who was not involved in the project.
Three years ago, when Pancho, now 38, agreed to work with neuroscience researchers, they were unsure if his brain had even retained the mechanisms for speech.
“That part of his brain might have been dormant, and we just didn’t know if it would ever really wake up in order for him to speak again,” said Dr Edward Chang, chairman of neurological surgery at the University of California, who led the research.
The team implanted a rectangular sheet of 128 electrodes, designed to detect signals from speech-related sensory and motor processes linked to the mouth, lips, jaw, tongue and larynx. In 50 sessions over 81 weeks, they connected the implant to a computer by a cable attached to a port in Pancho’s head, and asked him to try to say words from a list of 50 common ones he helped suggest, including “hungry,” “music” and “computer.”
As he did, electrodes transmitted signals through a form of artificial intelligence that tried to recognise the intended words.
“Our system translates the brain activity that would have normally controlled his vocal tract directly into words and sentences,” said David Moses, a postdoctoral engineer who developed the system with Sean Metzger and Jessie R Liu, graduate students. The three are lead authors of the study.
Pancho (who asked to be identified only by his nickname to protect his privacy) also tried to say the 50 words in 50 distinct sentences like “My nurse is right outside” and “Bring my glasses, please” and in response to questions like “How are you today?”
His answer, displayed on-screen: “I am very good.”
In nearly half of the 9,000 times Pancho tried to say single words, the algorithm got it right. When he tried saying sentences written on the screen, it did even better.
By funnelling algorithm results through a kind of autocorrect language-prediction system, the computer correctly recognised individual words in the sentences nearly three-quarters of the time and perfectly decoded entire sentences more than half the time.
“To prove that you can decipher speech from the electrical signals in the speech motor area of your brain is groundbreaking,” said Fried-Oken, whose own research involves trying to detect signals using electrodes in a cap placed on the head, not implanted.
After a recent session, observed by The New York Times, Pancho, wearing a black fedora over a white knit hat to cover the port, smiled and tilted his head slightly with the limited movement he has. In bursts of gravelly sound, he demonstrated a sentence composed of words in the study: “No, I am not thirsty.”
In interviews over several weeks for this article, he communicated through email exchanges using a head-controlled mouse to painstakingly type key-by-key, the method he usually relies on.
The brain implant’s recognition of his spoken words is “a life-changing experience,” he said.
“I just want to, I don’t know, get something good, because I always was told by doctors that I had 0 chance to get better,” Pancho typed during a video chat from the Northern California nursing home where he lives.
Later, he emailed: “Not to be able to communicate with anyone, to have a normal conversation and express yourself in any way, it’s devastating, very hard to live with.”
During research sessions with the electrodes, he wrote, “It’s very much like getting a second chance to talk again.”
Pancho was a healthy field worker in California’s vineyards until a car crash after a soccer game one summer Sunday, he said. After surgery for serious damage to his stomach, he was discharged from the hospital, walking, talking and thinking he was on the road to recovery.
But the next morning, he was “throwing up and unable to hold myself up,” he wrote. Doctors said he experienced a brainstem stroke, apparently caused by a post-surgery blood clot.
A week later, he woke up from a coma in a small, darkroom. “I tried to move, but I couldn’t lift a finger, and I tried to talk, but I couldn’t spit out a word,” he wrote. “So, I started to cry, but as I couldn’t make any sound, all I made were some ugly gestures.”
It was terrifying. “I wished I didn’t ever come back from the coma I was in,” he wrote.
The new approach, called a speech neuroprosthesis, is part of a surge of innovation aimed at helping tens of thousands of people who lack the ability to talk, but whose brains contain neural pathways for speech, said Dr Leigh Hochberg, a neurologist with Massachusetts General Hospital, Brown University and the Department of Veterans Affairs, who was not involved in the study but co-wrote an editorial about it.
That could include people with brain injuries or conditions like amyotrophic lateral sclerosis (also known as Lou Gehrig’s disease) or cerebral palsy, in which patients have the insufficient muscle control to speak.
“The urgency can’t be overstated,” said Hochberg, who directs a project called BrainGate that implants tinier electrodes to read signals from individual neurons; it recently decoded a paralysed patient’s attempted handwriting motions.
“It’s now only a matter of years,” he said, “before there will be a clinically useful system that will allow for the restoration of communication.”
Pancho’s buoyant personality has helped the researchers navigate challenges, but also occasionally makes speech recognition uneven.
“I sometimes can’t control my emotions and laugh a lot and don’t do too good with the experiment,” he emailed.
Before his stroke, Pancho had attended school only up to sixth grade in his native Mexico. With remarkable determination, he has since earned a high school diploma, taken college classes, received a web developer certificate and begun studying French.
“I think the car wreck got me to be a better person, and smarter too,” he emailed.
With his restricted wrist movement, Pancho can manoeuvre an electric wheelchair, pressing the joystick with a stuffed sock tied around his hand with rubber bands. At stores, he’ll hover near something until cashiers decipher what he wants, like a cup of coffee.
“They place it in my wheelchair, and I bring it back to my home so I can get help drinking it,” he said. “The people here at the facility find themselves surprised, they always asked me, ‘HOW DID YOU BUY THAT, AND HOW DID YOU TELL THEM WHAT YOU WANTED!?’”
The speech study is the culmination of over a decade of research, in which Chang’s team mapped brain activity for all vowel and consonant sounds and tapped into the brains of healthy people to produce computerised speech.
Researchers emphasise that the electrodes are not reading Pancho’s mind, but detecting brain signals corresponding to each word he tries to say.
“He is thinking the word,” Fried-Oken said. “It’s not random thoughts that the computer is picking up.”
Chang said “in the future, we might be able to do what people are thinking,” which raises “some really important questions about the ethics of this kind of technology.” But this, he said, “is really just about restoring the individual’s voice.”
The team also wants to engineer implants with more sensitivity and make it wireless for complete implantation to avoid infection, said Chang. “I just wanted to somehow be able to do something for myself, even a tiny bit,” Pancho said, “but now I know, I’m not doing it just for myself.”