How brain-guided hearing aids may one day help users

Hearing aids amplify everything — "indiscriminately" — wrote a group of scientists working on a system that may make brain-controlled hearing aids possible.

"Current hearing aids are good at amplifying sounds and voices, but they struggle with the classic 'cocktail party problem' — deciding which voice matters to the listener," said Vishal Choudhari*, lead author on their study published in Nature Neuroscience.

It can take a lot of effort to focus your attention on one voice in a crowded room. "Hearing is not only about whether words are understood correctly," Choudhari told DW. "Two people may both understand [what they're saying], but one person may need far more mental effort to follow the conversation. That can become exhausting over time."

As a result, many people stop using hearing aids just when they need them most — in restaurants, cafeterias, parties, or busy social spaces. 

So, Choudhari and his colleagues are trying to develop a smart technology that knows what a hearing aid user is listening to. They want to enhance that one sound or voice, while at the same time reducing the volume of any other sounds, voice or background noise.

And to do that, they designed a system that reads brain waves and, using artifical intelligence, interprets what a listener is listening to.

"Many hearing aids use beamforming, which enhances sounds coming from a certain direction, usually in front of the listener. But real conversations are dynamic," said Choudhari. "People turn their heads, switch attention, or even listen to someone without directly looking at them."

Brain-controlled hearing from theory to practice

Under Nima Mesgarani, a professor and principal investigator at Columbia's Zuckerman Institute, Choudhari and the team developed real-time machine learning algorithms that could examine brain waves and identify the conversations that four normal-hearing test participants were listening to.

The researchers called it a closed-loop auditory attention decoding (AAD) system. They wanted to find out whether AAD could be accurate and fast enough to selectively amplify the voice of an individual speaker while suppressing background voices.

While the idea sounds amazing, it's far from ready for general use. For now, the concept relies on electrodes being attached to the brain in a clinical setting.

In the study, the four test patients were undergoing brain monitoring for epilepsy — so, they already had intracranial electrodes and that was convenient for the researchers.

The participants were presented with recordings of two competing sound sources, coming from small speakers, positioned left and right.

The recordings featured people of various gender mixes, in conversation about food, travel and exercise. Their words were interspersed with what the researchers described as "multi-talker babble" and pedestrian noise.

Mesgarani and colleagues had already discovered in 2012 that brain waves rise and fall depending on a person's attention. They display peaks and troughs, the timing of which lines up with the sounds and silences in a conversation. 

In the new study, "the loudness of the competing conversations was adjusted dynamically in real time based on the decoded brain signals," explained Choudhari. "The attended conversation became louder while the competing conversation became quieter."

The system showed good results, according to the researchers, whether they guided participants to listen to a certain conversation and then asked them to switch their attention, as well as when the participants chose a conversation freely.

Challenges and promises ahead for brain-controlled hearing

Other experts in the field acknowledge the advances that Choudhari and Mesgarani's work has revealed.

Volker Hohmann, a professor of Auditory Signal Processing at the University of Oldenburg's Cluster of Excellence Hearing4all, highlighted "the impressive accuracy with which auditory attention can be decoded from brain signals, especially when intracranial electrodes are used."

However, in an email to DW, Hohmann said the system was not yet useful in everyday situations — a point that the researchers also noted.

The acoustic conditions were "completely fixed, with the listener not moving [...] Acoustic communication in daily life is far more dynamic," said Hohmann.

"The issue is that if one source is enhanced, the other becomes harder to hear, making it hard to switch attention to the quieter source and decode the change," said Bernhard Seeber, a professor of Audio Information Processing at the Technical University of Munich in an email to DW.

Seeber credited the team in the US with showing that the system could react in real-time when the listener switches attention, but said further research was needed "to achieve reliable real-time attention decoding from skin-electrode signals" — that is, a less invasive method to monitor brain signals than intracranial electrodes.

That is also where Choudhari sees promise for the system, and, ultimately, its being incorporated into smart, wearable technology: "Imagine smart glasses or earbuds that know what conversation you are listening to with your brain signals, can help summarize important information, or even assist with memory and note-taking in noisy environments," he said.

*Vishal Choudhari conceived the research as a PhD candidate under Nima Mesgarani at Columbia University's Zuckerman Mind Brain Behavior Institute. Choudhari is now a Founding Research Scientist at an AI company in Seattle, US.