Why You Can Connect with Tree Bark

Justine:          Hi, it’s Justine Harcourt-de-Tourville. We are poolside at South by Southwest. We’re at the JW Marriott, home of the virtual cinema, which is just a few floors down. And some of the entries that are there, there are 25, one of them is Ahorse! from WeMakeVR. And we’re going to be speaking with WwMakeVR’s Avinash Changa. He’s also going to be talking with us about some of the evolutions in VR and some of the new research that he’s been delving into. So, stay tuned and let’s hear it for Avinash.

(INTRO MUSIC)

Justine:          Welcome, Avinash.Thank you for joining us here at South by Southwest.

Avinash:         My pleasure.

Justine:          And it’s a great conference. What have you seen so far that has really triggered you?

Avinash:         Oh, good question. I saw the installation for -27 Celsius which is an experience made by [1:04 Young Cuna] and Amaury, who also worked on Notes on Blindness. So, this is basically their next piece, little side notes, Notes of Blindness was just voted as one of the top 10 pieces that’s actually meaningful or that actually has had an impact.

                        And I was excited to see their next piece. You can’t go into too much detail. With immersive, you have to experience it for yourself. But it’s great to see that people are queueing up for these news type of experiences that are not just about the gimmick of VR or the hype or big explosions or Michael Bay VR. But actually, not just tell the story, but give people the experience that triggers thinking.

Justine:          And it’s memorable.

Avinash:         Absolutely.

Justine:          And yeah, you feel it. You’ve been in VR for a while. I hear it’s 2012.

Avinash:         Yes.

Justine:          That is seven years ago now.

Avinash:         Yes.

Justine:          How things changed for you?

Avinash:         Well, prior to VR, back in 2007, we already explored AR, augmented reality, which everyone kind of forgot. And then, on the wake of the rise of VR, AR came back.

                        So, what’s changed overtime is that…

Justine:          Wait. In 2007, what was AR like? What would have that looked like?

Avinash:         There was a lot of I would say, actually it’s like layer and there were engines like Metaio. So, basically, what it did was we would use the webcam on your laptop or on your desktop computer and you would, for example, take up a print magazine and kind of recognize a print ad. And if you would hold that in front of your webcam, on your screen, you would see a little video display with the magazine and then, we’re going to project a character or a 3D object on top of that. So, we would track it in 3D space.

Justine:          Okay, that’s what the primitive technology was then in 2007.

Avinash:         Yes.

Justine:          Alright. You are doing in the AR space in 2007 and…

Avinash:         Well, we had a 3D or CGI animation studio, so we did visual effects for feature films, short films and advertising, you name it. And that means that we had a team of creatives that we’re very much native to digital production techniques.

                        And in our vision, or in my vision, to me, it doesn’t matter what the end platform is. The production process of creating a 3D model or filming something or doing visual effects remains the same regardless if you’re going to show that on a mobile phone or on a cinema screen, largely. I mean, once you get that final stage, then it changes.

                        So, because of that being natives to that technology, we got a very early prototype of the Oculus DK1 with their requests like, “Hey, can you guys make digital content for this?” I’m like, “Okay, sure we could.” After playing around with that for a couple of weeks, I was like, “Okay, this is fun, but hang on. We’ve grown up with films like Existence and The Matrix. And in pop culture, we have this very different vision of VR, these realistic immersive worlds.” So, I was like, “Can we make that?”

                        So, we ended up inventing the first proper natural stereoscopic VR camera back in 2012, and things took off from that. At that point, everyone was focused on the technology on cameras, on headsets and all of that, whereas it’s not really about that. But that is something that happens with emerging technologies. Everyone looks at the hardware, whereas we were like, no, we want to create experiences. And I think that’s the thing that has changed over time.

                        So, now, we’re getting to the stage where the average consumer is focusing less and less on the hardware, on the headsets, even though they still do, but you hear conversations that are more about immersive experiences, about an interesting game or an interesting 360 video or something that is neither of those and that they can’t really put a tag on. But it’s something that they get either an emotional response or form a passion about. And that’s interesting. That, I think, has changed in the last seven years.

                        But I’d say it’s more true for the professional side. When you look at consumers, they are now getting through the stage like, “Alright, I’ve heard about VR. It’s the thing from Oculus, right or that little [5:26 Ray box], which I can buy for 200 bucks, or I can watch Netflix on that thing. So, that’s interesting that you see that the average consumer is using a VR headset like an Oculus Gold, and that 80 percent of those users are using it for Netflix, to watch Netflix in bed.

                        So, non-360 video or non-VR games, but at least those devices are getting out there. And that’s very different from three years ago.

Justine:          Okay. So, people are using headsets. That’s a very good thing. And when did you first get into VR…

Avinash:         Well, I grew up…

Justine:          …from AR?

Avinash:         I mean, well, it actually goes further back than that, because as a kid, I was always fascinated by other worlds, videogames and comic books and sci-fi movies. So, I grew up with a desire like, “Oh, could I venture into other worlds?” That desire, like, “Oh, I’m playing a video game. What if I could step into that screen and be part of that?”

                        So, growing up, as most VR pioneers have done, I tried all the different VR headsets which were all pieces of crap. But then, when we heard about the Oculus Kickstarter, I was like, “Is this is really going to make a big of a deal?” Initially, I think that this is yet another attempt at a VR headset. But no, it was actually more than that.

                        So, I would say, a year after we got the DK1 and in the process of developing that initial camera, we got like, “Okay, we’re going to close shop on our CGI studio and we’re going to launch WeMakeVR and we’re going to go all in.” VR is going to be the future. Not in its form as it was at the time as in 360 videos, or very basic animated games but more as… what convinced me was I saw the potential and the way that this new immersive technology can and will fundamentally change the world, change our way of life, change the way that we communicate, change the way we educate ourselves, change the way we improve quality of life.

                        And because that is close to my heart, that desire to improve the world, it wasn’t even a decision. It just happened naturally like, “Okay, this is where I need to go.” Everything I’ve done in my life has kind of stirred me here and just culminates in immersive. I wouldn’t say VR. I mean even though we’re called WeMakeVR, we’re trying to get rid of the term VR or XR or AR or MR or all those Rs. We are crafting immersive experiences and we don’t have a good terminology for that yet.

Justine:          We’re not calling ourselves MX or something yet.

Avinash:         Oh my god, I mean, every time someone tries to come up with this new term, it still has to find its right terminology, something that people will remember and something that makes sense. But we always keep trying to, being humans, just trying to shoehorn it into a little box somewhere. And that’s another big change.

Justine:          So, you’re literally trying to come out of the box. No pun intended. But tell me then. So, you’ve had a life on passion for immersive and it led you down to form a company called WeMakeVR. What was the first thing you made?

Avinash:         The first thing we made was a tour through the Canals of Amsterdam. Because when you live in Amsterdam, everyone knows the Canals and a lot of people have boats as do I.

                        So, when we had that first camera, which was literally hot glue and blocks of wood and rubber band just held together, we put the camera on that boat and took a little tour through the Canals of Amsterdam. And then, we spent, I’d say, three or four months manually frame by frame stitching the whole thing together, because there was no stitching software.

Justine:          Wow.

Avinash:         And I mean this was pre auto panel and video stitching. So, back at the office, we put people inside our prototype headsets, and they would look around and be amazed like, “Oh, I can aww other boats. I can see people on bikes.” And interesting thing happened. That was one of our interesting first discoveries.

                        While we were filming, there was a boat coming in the opposite direction and I asked people on boats so that they waved to each other. So, someone waved at us while we were filming. In the office, every single person would instinctively wave back to the person on the boat, even though they are fully aware they’re sitting in the office, wearing a piece of plastic on their head.

Justine:          That’s a really good marker of a really immersive experience, isn’t it? You could feel like you’re in it and you could see you’re on automatic pilot, so to speak

Avinash:         Exactly, you have an instinctive natural response. But it would even be further than that and that is for me personally was the biggest eye opener and tipping point.

                        This experience took about four minutes. They would take off the headset and then they would look around in the studio. And actually, they would ask the host like, “Alright, that was really cool. But where is the fan and where’s the heat blower?” And we were initially confused because we had none. So, they said, “No, you’re pulling my leg because there has to be something hidden here because I felt the wind in my face, and I felt the heat from the sun.”

                        And then, we realized “Hang on. If the experience is complete enough and realistic enough, your brain becomes quite malleable and starts to fill in the blanks.” And that was, for us, to the main reason to partner up with researchers in universities. And we had the question like “Okay. We now know that we can make all this stuff. Why does it work the way it does? Why does certain experiences work and why do other experiences do not work, even though we instinctively think it’s going to work?”

                        And it turns out that this entire industry for us as makers is extremely counterintuitive. Anything that we think is going to work, doesn’t. For example, everyone initially in that first VR wave was like, “Oh, we’re going to do first person shooting in VR. Everyone’s going to be playing COD and other first-person shooters in VR headsets.” It turns out that that is still, to this day, a really hard genre to track because people get obnoxious. It doesn’t translate well.

Justine:          Will you explain COD for anyone who may not know?

Avinash:         Oh, call of duty. Call of duty is a first-person shooter and there are many like it. Oh, sorry for the COD fans. But the whole aspect of running around and shooting stuff, that initially is like, “Oh, yes.” Everyone was like, “If you can run around in VR, that’s going to make a lot of sense.” So, then we saw things like the Virtuix Omni and other platforms that you could run on in place. Because if you would be running around wearing a VR headset in your living room, you’re going to bump into walls.

                        But even with those kinds of technologies, it doesn’t really work. The rules for creating immersive experiences are fundamentally different. So, we set out, seven years ago, on a mission to explore these rules. Why do certain things work and why don’t they work? And how can we share that? How can we share that knowledge with other makers?

Justine:          So, you partnered with universities that discover this research?

Avinash:         Yes. Basically, we would discover, “Oh hey, we have an issue here. We know that something works but we don’t know why.” For example, we know that people now experience heat or cold or wind. Why do they do that? Is that frame rate?

Justine:          But it’s not happening so the brain fills it in.

Avinash:         Well, yeah, I mean it’s not real. It’s your brain that tells you, “Hey, you’re feeling heat, but we don’t know why.” Because we still associate the medium with a primarily visual medium. And then, we’re like, “Well, if you’re in a cinema, I mean you can sort of get that idea of “Oh, I’m feeling the cold, because we’re seeing a movie about the North Pole,” but you know the difference.

                        And with VR, you start not knowing the difference anymore. You start getting confused or your brain becomes confused between what is simulated and what isn’t, which kind of makes sense because everything that you experience as a human being is dictated by your brain. If your brain tells you that you have an ouch, you have an ouch, even though it’s not there.

                        So, when we encounter these things, we brief that to the university like, “Hey, could you research if this sensation of heat has to do with the brightness of the image or frame rate or the setting or is it sound?” And slowly but surely, we are getting the grips with more of these. Basically, we’re expanding the vocabulary of this new language that we’re just exploring.

                        So that, on its own, of course, is not what drives us. What drives me personally and by effect as a company is, I want to create experiences that kind of, for example, improve quality of life, that can help you conquer a fear or that can help you get over agoraphobia. Then, we work back from that.

                        So, if we then want to create an experience, what ingredients do we need? Do we need someone to be visually engaged? Do we need stereoscopic imaging to get that emotional connection? Do we need high framerate? I mean, these are, of course, the technical things that the researchers or the universities help us out with. But it’s always experience-driven and the goal is, in that sense, we want to make someone happy. How do we do that? We want to…

Justine:          And what did you find? So, does frame rate matter? Or does the stereoscopic imagery do more?

Avinash:         Absolutely. And that is one of the things where our previous experience as the effects company makes a lot of difference. We have been creating stereoscopic visuals for, I would say, a decade and a half. So, that helped us out at the beginning.

                        So, the first camera that we’ve created for VR was built as a stereoscopic camera, because we noticed that if you do a 360 monoscopic panoramic image, people are initially excited like, “Oh, I can look around” but they don’t get that sensation of being there. The brain never feels that you’re there, and that’s something that pops up in, for example, a functional MRI.

                        If you have a 360 video, your visual cortex is engaged, and you get the sensation that you’re watching a movie that’s projected around you. When you have proper, human natural stereoscopic imaging, your brain actually fires in different regions, and your brain perceives that as an image that you are seeing in real life.

                        Now, if you would do post conversion stereo, which is what we see in Hollywood a lot, your brain doesn’t do that. If you shoot it in proper, native, stereo that mimics what the human eye does, then you get that people, for example, can develop memories, and that’s something where the research comes in. When you see this, you see actual new synaptic connections being made by your brain, so your brain cannot distinguish this new memory from a real memory.

Justine:          So, the simulated memory is just as powerful or just as much the same as a real experience memory.

Avinash:         Well, from a physiological sense, there is no difference.

Justine:          Of course.

Avinash:         It is that same synaptic connection.

Justine:          Right.

Avinash:         And then, there’s a little interesting anecdote. A couple of years back, we were hosting an event and we’re doing demos. And one of our immersive directors was running or showering a VR demo. And there was this lady and there was her son, standing there and looking at the demo and he was like, “Oh, I know her,” but he can’t recall her name. An hour later, she was still walking around, and he started feeling bad, because not only was he convinced that he knew her, he had a conversation with her, but he could not recall her name.

                        And then, today, it turns out that we have created a VR experience where this lady was in with you. You were both on a swing, on a super high skyscraper and it was a scary moment. And she looks at the camera and shares her anxiety with you. He had seen that, and he was fully convinced that this was an actual experience that he had.

                        Now, when I speak at conferences, I often talk about this and he, this director, who works for me, always consider that a marketing spiel like, “Oh yeah.” This was the first time that he had actually experienced this sensational false memory, and this is something that works in the VR space on a daily basis. So, that was very interesting to say the least because once you get to the stage where you can realize that you can actually implant false memories, you get into a moral and ethical gray area.

                        So, that’s another thing that I often like to talk about you to the more experienced makers. Not that we have the technological capabilities to create this kind of really strong immersive experiences, what do we do in terms of morals and ethics and guidelines? I mean, is that up to us? Should we come up with guidelines as a community or as makers? Most of them get too excited and just want to make cool stuff, but these are important conversations to have at this stage of the industry.

Justine:          Well, at this stage of the industry, VR is basically… we’re all pirates, right? We’re trying to figure it out. The rules aren’t written. There’s no standard at this point, so everyone’s kind of navigate the open waters and maybe say, this is the best route here and this is the best route, you know, talking to each other.

Avinash:         I would argue that in the last two and a half or three years, we are actually seeing standards emerge. We are actually getting slightly past that pirating, initial pioneering, fully uncharted territory. I mean this is the tip of the iceberg that we’re hitting, but we are finding some common grounds. For example, at this stage, we know that there is a fundamental difference between a 360 video and 360 3D video. And a couple of years ago, there was still a lot of debate like, “Oh, it’s still the same thing” or the value of, for example, using smartphones in a cardboard or cardboard-like format. Two years ago…

Justine:          What’s the verdict?

Avinash:         We’ve abandoned smartphone formats and cardboards. It was great at the time. It was great to sort of get this initial wow factor, but the moment an average consumer gets past that wow factor, there needs to be some substance. And in my personal opinion, the smartphones and cardboard type format will not deliver that substance simply because issues like visual fidelity, frame rate, etc.

Justine:          That’s not a holistic experience.

Avinash:         No, it’s not.

Justine:          It’s more a viewing. You’re still kind of an observer of VR.

Avinash:         Absolutely, yeah. And it plays on the gimmicky angle because you get someone that… it’s like the old View-Master. You look at it, and they pass it onto your buddy who’s sitting next to you, like “Oh, look at this. I found a porn video in VR. Isn’t that cool?”

                        And then, you know, everyone lasts for a couple of minutes, and then it’s put on the shelf and then no one considers it again. Whereas if you put someone in a basic headset like an Oculus Gold for 200 bucks and that person has seen something like for example Notes on Blindness, that person will have a very profound experience and not think about just technology anymore but think about the experience they had.

                        And that’s what we are seeing. That’s where standards are emerging. I think that realization that cardboard doesn’t cut it anymore, but better headsets are now affordable like the Go and the upcoming Quest and the Vive Focus. That’s going to basically up the ante a little bit. We know the expectations for makers change and consumers and companies that commission content are getting more educated.

Justine:          Well, let me go back and find out, in some of the research that you’ve been doing, what research have you learned or gleaned rather that is really applicable for VR creators that work in narrative or cinematic? 

Avinash:         So, one interesting, very recent discovery or something that we’ve been able to prove a little bit better is frame rate. In that first wave of immersive content, a lot of people making 30 frames per second 360 videos. Yup, you will get that initial wow. Someone hasn’t seen anything before, but then we started upping to 60 FPS. Alright, that negates nausea. We can do a bit more high action stuff. And some people kind of oppose that because it doesn’t have that cinematic feel that we’re used to.

                        So, that’s a decision that makers have to make a call on like, “Do I want to create a piece that is like Hollywood cinematic narrative piece or do I want to create something that will instill a memory or that is focused on transference of knowledge?”

                        Something magical happens once you start hitting 120 frames per second and up. And it’s not just that the image is smoother, it’s that your brain requires less effort to accept something as being real.

                        The lower the frame rate, the more your brain has to work to bridge that gap between fiction and reality.

Justine:          So, 120 is something that you’ve found?

Avinash:         Yes.

Justine:          Anything else that you’ve found?

Avinash:         When it comes to social connections, a couple of years back, we were like, “Okay, if we want people to connect on an emotional level in a virtual space, we need super high-resolution cameras on both sides, let’s say this is a multi-user experience, on both sides of the field.” And we discovered that that is not true.

                        If both avatars have animated pupils and if you can hear someone’s voice, and if you can have a basic animation rate that mimics your physical emotions to your avatar, that’s enough. If on the other side, you have someone that you know, you will recognize their voice. You will recognize very subtle physical emotions that are translated to the avatar. You will see their pupillary gaze. And it doesn’t matter if that character looks like a Lego character or Donald Duck or a piece of tree bark. You connect.

                        And again, that’s one of these things that is super counterintuitive. The expectation was…

Justine:          I want to just reiterate that it is definitely counterintuitive to connect with a piece of tree bark.

Avinash:         It is. And it’s weird that it ends up working. And that is, I think, where advancements in technology do help us. For example, getting to the stage where we can do pupillary tracking and then translate that to animating your avatar. That is a big difference.

                        I mean, at the same time, we’ve seen the latest hype. No, I wouldn’t say hype but latest development. Over the past 18 months, we’ve seen a big insurgence of volumetric capturing. Everyone’s playing around with connects and point clouds. I’m like, “Oh, it’s not just a 360 video, but it is a volumetric character that I can walk around.” So, it’s therefore going to be more real. And I think we’ve seen examples of that in, for example, the Blade Runner promotional experience. It has a lot of a budget. It was commercially commissioned.

                        So, what happens in Blade Runner, you put on the headset and there’s this volumetric actress and nurse standing right in front of you and she looks you right in the eyes. You’re like, “Oh, this is very realistic.” The moment you take a step to the left and to the right, you can. She stays in place. It’s not a 360 video. You can actually walk around, but she keeps staring into me there, in the place where you were.

                        The moment that happens, in my opinion, your suspicion of this belief breaks. Your sense of immersion breaks. It doesn’t work. Even though you’re like, “Oh, this is the new frontier of immersive experiences,” on an emotional level, you do not connect anymore.

                        And that’s something that we started playing around with like, “Oh, can we rotate the avatar? Can we do something with gaze control? Can we animate the pupils?” And it turns out that, right now, I would say trying to achieve what we’ve done in high quality 360 stereoscopic video in volumetric and trying to do the same thing on a narrative level, I don’t think that right now is the road forward, because there’s a technological gap there that we right now cannot solve.

                        Even when your budget and talent is unlimited, if you look at Carne y Arena… Carne y Arena, not much of your audience will probably know it. It’s an immersive experience about crossing the Mexican-US border with a family of refugees.

Justine:          With your feet in the sand and a backpack and optic devices so you get the…

Avinash:         Exactly, yes. So, it was made by Alejandro Iñárritu, an Oscar-winning moviemaker. The technical team was ILMxLAB. Amazing guys, they do stunning work. Then, I’m not going to the narrative per se but once you are in that desert or the sand under your feet and you see those volumetric captured actors who form their family around you, again, don’t make eye contact. They are there. You can walk around them, but they don’t acknowledge your presence. They don’t connect to you emotionally. So, for a lot of people, it’s a huge wow factor and it makes people think, but not because that part of your VR experience was so solid.

                        And one thing that we keep bumping into is that people get caught up in that initial wow factor. If someone or an average consumer has never seen or experienced a proper VR experience, the moment they put on a VR and see something new, they’re like, “Oh, wow. This is really cool.” And then project on sensation of “Oh, this is something that’s super innovative” on the content, regardless of what that content is.

                        So, there’s a lot of poor-quality content that still gets high praise. Iñárritu’s is an amazing piece. I’m super happy that it exists because it exposes millions of people who are not interested in technology and not interested in VR to an immersive experience. It was an eye opener. And because of the name, because of the production and the quality, this piece has been shown around across the globe for the past two years.

Justine:          And let’s not forget that actual subject matter, the fact did not only introduce technology and immersive, but introduced the powerful, controversial subject to experience in a whole different level. So, it was incredible on so many fronts and checks a lot of boxes.

Avinash:         Yes. Incredible, groundbreaking, it paved the way for a lot of new makers. But it’s a double-edges sword because you got such a high name. So, it’s such a big name.

                        There’s a lot of venues and museums that they’re like, “Oh, this immersive world is really cool. We want another installation, so we’re going to look for the next super big name that we know from the film industry and hope that they are going to make something immersive, and we’re going to program that.” Whereas from a storytelling standpoint, from a narrative perspective, it’s a level playing field.

                        There are tons of new makers, tons of talents who have no history in cinema, but can make something that is equally as impactful and as worthwhile as Carne y Arena. These makers have a hard time getting exposure for their pieces because they don’t have that name and the people that program the event don’t have the technical experience or knowledge to make that distinction.

                        So, well, as a programmer, you’re going to play it safe and that is something that is great about South Bay. You see projects that are programmed here that are not necessarily associated or made by these big names that have budgets that are in the tens of millions of dollars. There are pieces here that have been made for $50,000 or even less, that are getting a platform.

                        And that’s something that I think for everyone that’s making content now. Not the people that are still learning how to stitch, but that are getting more into the actual content side of things. It’s easy to get discouraged like, “Oh, we can’t compete with those big names.” I would say, “Yeah, you can.” You just have to get out there and make these pieces that you are passionate about. I mean if you do this, if you get into the industry now because you’re like, “Oh, let’s do a quick cashcrab,” you’re going to be so disappointed. We are running a marathon, not a sprint. If you actually start a company now, if you become a maker, you need to have that longterm vision.

Justine:          Well, that’s a good point because I’d like to end on, what kind of advice do you have for young upcoming, or even older people that want to make the transition from classic to VR? what kind of advice would you give them?

Avinash:         You need to do this for the right reasons. Get into this immersive format or world or family, because you have a deep, personal desire to make a difference, to make beautiful art or to make an impact. Don’t get into this industry for money at this stage. I mean it’ll get there, but just in simple market dynamics, we don’t have a hundred million active VR user to have a headset in their homes right now, who are willing to pay for content.

                        At the same time, there is hope because Netflix just evaluated the success of Bandersnatch, which was their interactive next piece. And even though it’s an old concept, that usually you defend your own concept, again the same as Iñárritu’s piece, it has exposed millions of people who are not interested in tech but are interested in cinematic or narrative experiences to this format. They have doubled down. They are going to do more investments in that area, which is going to help consumer adoption. This is going to open up consumers to their curiosity, to explore more immersive experiences.

                        And that’s, I think, where the opportunities are for the new makers per se. If you want to do something new or if you want to get into this industry, don’t look at the 360 videos etc. Learn from it but look beyond that. Look at the uncharted territory. It is, and I cannot say this is enough, a level playing field. The guys that have millions of dollars in production value don’t have more knowledge than you have.

                        You can come up with a unique idea that no one has come up with, and it can be a monster hit. Look at Beat Saber. You know, three guys coming up with a very simple game that is not being played by millions and millions of people. Okay, it’s not narrative, but it is content you can make.

                        And I would say, in terms of narratives, in terms of stories, we’re just seeing the tip of the iceberg. I think my strongest belief, especially for people that come from a traditional filmmaking industry, let go of what you know. Embrace the fact that you don’t know anything and that none of us knows anything. Every single person in this industry that that now says we know how this works, pardon my French, is full of shit, because we don’t.

Justine:          Well, with that, we are going to take the cue from the music that we hear because it wouldn’t be South Bay if there wasn’t music and excitement all around us. And thanks so much again for joining us. We appreciate it.

Avinash:         My pleasure.

(OUTRO MUSIC)