The real difference between AR and VR under the hood

HostI was thinking the other day about how we talk about these new digital worlds. We have these two big names, AR and VR, and they always get lumped together like they're just two versions of the same thing. But when you actually put the gear on, they feel worlds apart. Is the difference just about what we see on the screen, or is there something deeper going on with the tech itself?

GuestIt helps to think about it as the difference between a blindfold and a pair of glasses. At its heart, VR, or virtual reality, wants to cut you off from the world. It replaces everything you see with a digital world. AR, or augmented reality, wants to stay in the world with you but add a new layer on top of it. On the outside, they both look like goggles, but under the hood, they're trying to solve two completely different problems. With VR, the computer is the boss. It creates every leaf on every tree and tells your eyes exactly what to look at. But with AR, the computer is more like a guest in your house. It has to watch what you're doing, learn the shape of your room, and try to fit its digital toys into your real space without bumping into the furniture.

HostI saw a video where someone wore a VR headset at a coffee shop and used the cameras on the front to see their latte. That felt like AR to me because they were seeing the real world with digital screens floating in it. Does that mean the line between them is basically gone?

GuestThat's what a lot of people call passthrough. It's a clever trick, but it's still fundamentally VR. The cameras on the front of the headset take a video of the room and show it to you on a screen an inch from your eyes. You aren't actually looking at your coffee; you're looking at a movie of your coffee. True AR doesn't use a video feed. It uses clear glass or plastic lenses. You see the real light from the real world, and the headset uses tiny projectors to bounce digital light into your eyes. The reason this matters is that video always has a tiny bit of lag. If you move your head fast in a passthrough headset, the world might feel like it's swimming or dragging behind you.

HostBut a tiny bit of lag can't be that big of a deal. I play games with lag all the time and it doesn't bother me. Why does it matter so much when it's on your face?

GuestIt matters because your inner ear and your eyes are always talking to each other. When you move your head, your inner ear feels that motion instantly. If the video of the world takes even a few extra milliseconds to catch up to that movement, your brain gets confused. It thinks you might have eaten something bad because your senses don't match up, and that's why people get that sick, dizzy feeling. In true AR with clear glass, that lag doesn't exist for the real world because you're looking right through it. The challenge there's making the digital objects stay still. If you put a digital cup on a real table and you move your head, that cup has to stay glued to that spot. If it slides around even a hair, the illusion breaks.

HostThat sounds like a lot of math. Is that why AR headsets always seem so much more expensive or bulky?

GuestThe math is wild. To make AR work, the headset has to do something called mapping. It uses sensors to bounce light off the walls and the floor to build a 3D map of the room in real time. It needs to know where the edge of the table is so it can hide a digital character behind it. VR doesn't have to do nearly as much of that. In VR, the computer already knows where everything is because it built the world from scratch. AR has to constantly guess and check its work as you walk around. And it has to do all that while staying small enough to fit on your face.

HostIf the math is so much harder, why not just stick with the video version? If we make the cameras and the screens fast enough, won't the swimming feeling go away?

GuestWe can get close, but there's another big hurdle, which is how your eyes focus. When you look at a screen in a VR headset, your eyes are focused on a fixed point just a couple of inches away, even if the movie shows a mountain in the distance. This can make your eyes very tired after a while. In true AR, because you're looking through glass, your eyes can focus on a real object across the room and then shift to a digital object right in front of you. That's much more natural for the body. But the trade-off is light. It's really hard to project digital light that can compete with a bright sunny day. That's why digital objects in AR often look a bit like ghosts or shadows, while in VR they can look solid and bright.

HostSo we're basically choosing between a perfect digital world that makes us a bit sick, or a real world with faint digital ghosts that are really hard to compute.

GuestThat's the trade-off. Right now, VR is great for games because we can control the whole world. AR is aiming for something much bigger, like replacing your phone or your laptop. But to do that, we have to solve the heat problem. All those sensors and chips for mapping the room get very hot. If you put a powerful computer right against your forehead, it's going to get uncomfortable fast. The real wall we're hitting is that we're trying to make a tiny pair of glasses do the work of a huge gaming computer without burning your face off.

HostThe blindfold is easy to build because it owns your whole world, but those clear glasses have to earn their spot by playing nice with the room you're already standing in.