When Time and Space Collide, There’s Sound

(INTRO MUSIC)

Justine:          Well, hello.

Billy:               Hello, Justine.

Justine:          Hello, Billy. So, let me ask you to introduce yourself if you don’t mind.

Billy:               Okay. First of all, thanks for having me here on your amazing podcast.

Justine:          Inaugural podcast.

Billy:               Yeah, that’s great.

Justine:          The very first one.

Billy:               Cool one. Thank you very much. I’m Billy Mello. I’m a sound designer, musical producer, DJ, and everything audio-related guy. I’ve been doing this for, like, 27 years.

Justine:          Oh my gosh.

Billy:               And yeah, it’s been a while. And recently, for like three years from now, I graduated in audio for video games in Berklee College of Music in the US. And then, I started working, of course, if you started working with video games, you get interested into VR, and then I started working with VR audio. That’s what I’ve been doing now here in Berlin.

Justine:          Oh, great and what company is that?

Billy:               My company is called Lucha Libre Audio. We have three studios, one in Berlin, one in Sao Paulo, Brazil, and another one in Los Angeles. So, we do audio, in general, as I have been doing for forever, advertising and everything, but the VR audio is always focused here in Berlin.

Justine:          Terrific. Well, thank you and welcome.

Billy:               Thank you.

Justine:          I’ll see what kind of questions we have prepared for you. Alright. Can you develop further your idea, like what you think about placing music in space?

Billy:               Yeah, this is really important because when we work with regular audio like movies, or music and everything, it’s always linear. It’s time-based and yeah, it’s basically based on time, and then it will listen to linear audio. When you when you shift it to games and to VR, it’s a different perception. So, it’s also based on time, of course, because you have the duration of things. But it’s also placement, which is a totally different approach to different concepts and a totally different way of doing this. Because when you do spatial audio, you have to really think about the space and what the space represents to you, and what you can do to achieve the best audio inside that space. So, this is a really tricky and interesting way of working with audio.

Justine:          Can you give me an example of something that maybe the right ear might hear easily or the left ear? Are there differences there? Because I know like for example, if people are walking behind you, it’s harder to hear? Are there little factoid?

Billy:               No. The thing is, since we don’t have earlids like we have eyelids, we cannot shut down our ears. So, we listen to things 24 hours a day, seven days a week. But our brain kind of cut out the non-important things like okay, we’re here in a calm environment, we don’t have to worry about somebody walking around us because we are safe here.

Justine:          So, the film is like a filter rather.

Billy:               Those are brain filters. When we are hearing the footsteps of people coming in and everything, the brain says, “Yeah, it’s not important so cut it out.” So, but yeah, the main important thing is to make the audio work in a way that your brain perceives this in an important and something that makes sense for you inside the VR.

So, there’s also another concept called head-related transfer function, which is HRTF. It’s that the name for this, which is the difference of the sound coming to your left ear, then to your right ear when you hear something on your left. And then, from the right ear to the left ear, when you hear something on your right. So, your brain perceives the difference as milliseconds because of the distance between the ears, but your brain perceives this as something happening on your right, on your left, behind and in front. So, this is something evolutionary and it really helps you to understand how to work with audio in VR environment.

Justine:          So, we’re a little bit like Barnell’s?

Billy:               Yeah, exactly. I think this is the main idea.

Justine:          Alright. Well, good. And what is the added value then of six degrees of freedom in your musical work and songs?

Billy:               Yeah, it’s good because when we listen to songs, we go to Spotify or you listen to music on YouTube, and it’s just a linear audio stereo. You can have something left, something on the right, but it’s always linear. And when you have six degrees of freedom in hearing, and then on walking around the space, it’s really cool because you can separate the whole song in the six degrees of freedom we have. So, you can put the voice in front of you, the drums behind you, I don’t know, the keyboards up left, and you can really separate the sound.

And it’s such a different and interesting perspective of listening to music because when you listen to the regular linear stereo music, it mainly focuses on vocals, and something that is happening right in front of you. Of course, you get the rhythm and everything, but the main focus of it is the voice and the song and everything happening there. But when you split it into six degrees of freedom, in audio, you can actually “Okay, I’m just going to go there to that corner. I’m just going to listen to the percussion”. And then, you keep on listening to the percussion. And it’s really interesting to have this different approach of music, which gives a new perspective and you can listen to the music a hundred times. And the hundred times will be different because you go near to one instrument to the other instrument. You can move around each instrument, so it’s really interesting to do this.

Justine:          That is interesting, because traditionally, if you’ve grown up with speakers, you know just the right and the left. That’s your sound. And now, you’re really getting this immersive effect as the big word in VR immersive. I don’t know if you’ve heard it.

Billy:               I’ve heard it somewhere, but I don’t know where.

Justine:          So, in terms of like mastering a soundtrack, is there any difference spatially in 360 video or 360 gaming that you have to think about?

Billy:               Yeah, for sure. If you master regular audio, you have the proper mastering tools and everything. But if you master for 360 videos, you have to work with loudness, which is a different way of mixing and having the proper way of arranging the sounds. It’s just a totally different concept. It’s based on TV, but it also works very well on 360 videos.

And if you’re mastering for games, it’s a different perspective also. Because the thing is, we don’t quite know, but if you have a game, the regular game and a TV or a movie, you expect music on that. You actually expect music to happen there. But in virtual reality, we’re kind of in between the game and the movie and the real life. In real life, you don’t have like, “Oh, I’m so happy today” and then, the orchestra starts playing. It doesn’t happen.

Justine:          It would be so great.

Billy:               It will be amazing. If your life would have a soundtrack, it would be amazing. But unfortunately, not, so we have to have a real balance between the real life and the music game/movie thing in VR. So, the mastering is different. If you do something in games, you have to rely on resonance or oculus or whatever SDK you want to use in audio. If you’re doing 360 videos, then you have to rely on loudness but specifically for Facebook or whatever. So, it’s a totally different approach.

Justine:          Wow, that’s very good. Are there any kind of plugins or musical software or things that you recommend or use that you love?

Billy:               Yeah, what I’ve been working a lot in terms of 360 videos is Reaper, which is a free DAW. Yeah, it’s a reaper.fm. You go there and download it. It’s a Pro Tools with lasers. Actually, Pro Tools is too expensive and Reaper’s free, and you have all this coding thing you can do really fun stuff in Reaper, which is kind of amazing to work with.

And inside Reaper, I generally use the Facebook-specialized audio and I use blue ripple or use sphere, depending on what the project is. If it’s music, it’s sphere. If it’s something that really relies on reverberation, I use the blue ripple. And if it’s something to be really fast, I have already something that came to me in ambisonic, I use the Facebook.

Justine:          Well, it’s terrific.

Billy:               And in games, units. That’s what I’m used to do work with, with the Google resonance audio. It’s tested with steam VR. It’s kind of okay, but Google resonance delivers for all platforms you want to.

Justine:          Is there a difference from the different link headsets in terms of…?

Billy:               Yeah, if you deliver 360 video, you have to deliver it for YouTube, for Facebook, for Samsung gear, for Oculus, for vive and for everybody else. And if you deliver game in units, you have to make it for Mac, for PC, for steam, for Samsung mobile or whatever.

Justine:          And so, does that mean you have to know 12 different ways of doing sound?

Billy:               The 12 different, no. Actually, some things are kind of different in delivery because we the space problem specifically for audio or for games because they have like if you do something for vive or Steam, you can have lots of space for delivering everything for the quality everything. But if you’re doing for something mobile, you have to think smaller. So, we have to manage these issues also. And there are ways to deliver specifically for each one of the platforms.

Justine:          Wow. So, there’s a lot of conceiving of space before you even begin.

Billy:               Sure, that’s really important.

Justine:          Okay, do you see any differences in the spatial audio workflow and the spatial audio acceptance between Brazilian agencies and European agencies, and I’ll throw in American agencies?

Billy:               I think that, not actually, because we’re such in an open space right now. We’re such in an open environment right now because we want experiences. We think that the main thing of delivering 360 videos or games or whatever is to have a cool experience.

So, Brazilians think of a cool experience as something, I don’t know, more adrenaline-related. And maybe cool experiences here in Europe are more like, I don’t know, intellectually-related. On my vision, what I’ve been working with is like, here in Europe, I’m doing much more intellectually-driven experiences like in 360 videos and in games. Like you have to really think about it and really to give you some new content.

And for Brazil, we actually deliver some fun experiences, something like, I just delivered a Heineken experience now, which is Formula One running through the streets in Rio de Janeiro right next to the beach and it’s really amazing. We had a camera in the Formula One and Philippe Massa and Rubens Barrichello were driving the car and they want this adrenaline. How does it feel to be on a Formula One car, this experience? And here in Europe, I delivered one experience for kids that needs to be managed by the government because they have some problems at home. So, you know, the balance is quite different, but in the end, they’re all experiences. They’re all interesting stuff to learn and to see and to take you apart from your regular, normal view of life.

Justine:          That’s fantastic. And on that note, what a great note to end.

Billy:               Yeah, but I think the main idea of VR is taking you out of your regular space and putting you somewhere different, yeah?

Justine:          Yeah, but there are tendencies, right? There are cultural tendencies and you address that. But they’re still open to us, that’s the great thing about a global platform.

Billy:               Yeah, for sure.

Justine:          We can try the Brazilian or we can try the European.

Billy:               Yeah, and it doesn’t take off that maybe Brazilians want some intellectual stuff and maybe the Europeans want some fun stuff. It’s totally open, but what I’ve been working with is what happens.

Justine:          Well, you’re just seeing patterns or patterns to merge or something like that.

Billy:               Yeah.

Justine:          Well, thanks so much, Billy. I appreciate you doing this inaugural one. Bye.

Billy:               Bye.

(OUTRO MUSIC)