How your eyes change the sounds you hear

HostThere's this strange video you can find online where a person is clearly saying the sound bah, bah, bah. But if you watch their mouth, they seem to be making the shape for gah, gah, gah. The weird thing is, as you watch it, you don't hear bah or gah. You hear a totally new sound, like dah, dah, dah. But the second you close your eyes, the sound flips back to bah. It's like your eyes are lying to your ears and winning the fight.

GuestIt's one of the most famous tricks in the world of brain science. Two researchers named Harry McGurk and John MacDonald stumbled onto it back in 1976. They were actually trying to study something else, but they noticed that when what the eyes see clashes with what the ears hear, the brain doesn't just pick a side. It doesn't go with the sound or the sight. Instead, it does a bit of math. It takes both pieces of data and builds a new sound that fits both. We call this the McGurk effect. It shows us that hearing isn't just a straight line from your ears to your mind. It's a mix of everything your senses can find at that moment. What you think you're hearing is actually a guess your brain makes based on all the proof it has.

HostThat feels a bit scary, to be honest. I want to believe that when I hear a sound, I'm just hearing the world as it is. But you're saying my brain is just making a calculation and might even be making up sounds that aren't even there?

GuestWell, think of it less like a lie and more like your brain trying to be helpful. The brain knows that both your eyes and your ears have strengths and weaknesses. Hearing is amazing at timing. Your ears can tell exactly when a sound starts and they're great at feeling the vibration of your vocal cords. But your ears are actually not that great at figuring out the shape of a mouth. Vision is much better at that. Your eyes can see exactly where the tongue and lips are moving. This is what experts call the place of articulation. Because your brain knows your eyes are better at spotting the shape of a word, it lets the eyes lead the way on that part of the job.

HostBut my ears are right there. If the sound waves hitting my eardrums say one thing, why would my brain let my eyes override them? Surely the sound itself is the best proof of what was said.

GuestNot always. Think about sounds like pea, bee, or em. To your ears, those can sound quite similar if there's any noise in the room. But to your eyes, they look very different because of how your lips meet. Because these visual cues are so clear, your brain treats them as the main proof for what word is being formed. If the mouth makes a certain shape, your brain will actually go back and re-edit the sound signals it just got from the ears to make sure they match. It's like a film editor changing the sound of a scene at the last minute because the actor's lips don't match the script.

HostSo if I'm aware that this is happening, can I stop it? If I know the video is a trick and I tell myself to just listen to the sound, can I force my brain to hear the real bah sound while my eyes are open?

GuestYou really cannot. That's the wildest part. You can be the world expert on this effect and know exactly how the trick works, but you'll still hear the fake sound. This is because the mixing happens in a specific part of the brain called the Superior Temporal Sulcus. Think of this spot as a hub where all your senses meet to trade notes. The work this hub does is what we call pre-attentive. That just means it happens way before the information ever reaches the part of your mind that thinks and makes choices. By the time you're even aware of the sound, the mixing is already done. Your brain has already baked the cake, and you can't pull the eggs back out of the batter.

HostThat sounds like a glitch in our system. Is this just a weird fluke that only happens in a lab, or does this actually change how we talk to each other every day?

GuestIt's actually a vital tool for survival. We're all lip-readers, even if we don't know it. When you're in a loud, crowded restaurant, your brain is working overtime to hear your friend across the table. In that noise, seeing their mouth move gives your brain a massive boost. It's like turning up the volume by about fifteen decibels. Your eyes are watching the rhythm of their lips and telling your ears which specific sounds to pay attention to. It helps you filter out the dishes clattering and other people talking so you can focus on the one person you care about.

HostIt's amazing to think that my eyes are basically acting like a volume knob for my ears just to help me get through a dinner conversation.

GuestYour brain is constantly using that visual rhythm to decide which frequencies in a noisy room are part of the story you're trying to follow and which ones are just junk to be tossed away.

HostSo even when I think I'm just listening, I'm actually watching a silent movie and using it to write the script of what I hear.

GuestThat's exactly it.

HostThe next time I'm at a loud party and I find myself staring at someone's lips just to understand them, I'll know my brain is just trying to do the math to keep me in the loop.