Magic Leap promises live-streamed holograms within five years

Yesterday, Magic Leap CEO Rony Abovitz and NBA commissioner Adam Silver announced plans to eventually bring Turner-owned NBA content to Magic Leap’s elusive AR goggles, by casting virtual screens in front of a wearer’s face through the goggles. It wasn’t an earth-shattering announcement, but it was an indication that Magic Leap’s tech could be more than billion-dollar vaporware, and that the company is thinking about the kind of content needed to convince people to wear light-field face computers.

What’s more interesting is that Abovitz believes volumetric video — a way of capturing video as 3D objects and scenes — will be live-streamed to people’s faces within a relatively short period of time, and that the kind of cameras used to make this happen will eventually replace traditional broadcast cameras.

In an interview with The Verge, Abovitz said that within “two to five years,” it will be technically possible for people wearing Magic Leap goggles to watch an NBA game (or other media) live, but in a holographic, interactive form.

“You can stream over the top and to the screens, the virtual screens — you can do that now,” he said. “We’re looking at, how do you derive the information to move the volumetric stuff from that? And then, how do you do volumetric live-streaming as well … if you time where processing power is going, particularly backends, you’re single-digit years away from that happening.”

Abovitz said that in order for it to work, it would require multiple fixed camera sensors to be placed around a venue. He also said that instead of using “30 big TV cameras shooting from different angles,” broadcasters would instead use a larger number of much smaller, super high-resolution cameras that would send all the data to a backend computing stack.

“[It will] move from standard 2D television broadcasting to full volumetric capture. That’s probably going to take a decade before you see it happening across multiple sports and news, but you’ll see early adopters,” he said.

Right now, making volumetric video is a production-intensive process. It often requires cylindrical green screen stages, like the kind Microsoft and 8i use; a multi-camera setup; and plenty of video processing and rendering time. The end result is video assets that you can walk up to, walk around, and even talk to, whether in a VR headset or through the lens of a smartphone. Many of the examples that exist to date have been avatars created for marketing or educational purposes, like this Buzz Aldrin hologram.

What Abovitz is proposing is not only something that requires more computational power, but also means outfitting everyday spaces or venues with these kinds of cameras, and on the consumer end, getting people to buy and wear the headset. Abovitz pointed out that one of the benefits of working with a sports league is that it’s a fixed physical space, which could make the production challenges easier to tackle.

Abovitz also shared more about how apps on Magic Leap’s goggles and its proprietary operating system will work, claiming that the number of possible interactions on the developer version of the goggles is “insanely large.” The system is capable of interpreting commands from someone’s head, eyes, voice, gestures, and a physical control, he said, which developers can currently customize. “Because it’s the creator edition [of the goggles], if you want to blink twice and use your voice, we’re not stopping you.”

App icons will also appear in volumetric format on the app launcher, and by just looking at the icon, a preview of the app will be revealed, whether it’s “an NBA experience, a Mario Kart, or whatever it is you want to do,” he said.