How to record spatial sound in a VR/360 film set
If you were able to go back in time and be a part of something big, to be Google’s first employee or work with Steve Jobs in his garage, would you do it if you knew the business would succeed? Probably yes. Yet in the present, we can’t say for sure what’s a good investment or what’s not. We have to follow our interests and passions, hoping things will turn out. Well, spatial sound is one of those investments for me. When I think about the intersection between virtual reality and spatial sound, my mind explodes with creativity. I feel like I found a promised land that is still empty, and I hope others will join along and start building in it with me. So, if you are a musician, sound engineer, or filmmaker- I hope you will see the potential here too.
What is spatial sound?
Spatial sound, also known as sound in 3D, or ambisonic sound (although it’s a name of only one format within spatial sound) is a sound in 360 degrees. Meaning if something explodes behind you in VR and you turn around, you will hear it in front of you as you turn. Just like the visual you see in the headset, expect sound is all around you, positioned in space like in real life. This is the first time we interact with sound in such a way. Spatial audio is a key solution to attract and direct the attention of the viewer to the story unfolding around him. Let’s say a demon is launching at you from the back, you will for sure turn around. How scary could that be, right? You hear something. You turn and explore. This ability might be familiar to you from video games. But when we are talking about movies, or music videos in VR, this is a new craft that has just been born.
A little history about spatial sound
In 2016 Facebook acquired a start up called 2 Big Ears, which developed a plug-in for DAW stations to composite sound in 360 and render it in a format which can be read as spatial by 360 video players. Facebook released it for free as Facebook360 workstation, to encourage creators to come into the space. But since the concept and the medium at large are so new, we haven’t seen it explode… yet.
In addition, the technology and tools for creating spatial sound are still limited, but with a few tricks and creative thinking, you can achieve amazing results.
Spatial sound is being distributed by two main formats- Ambisonic and FuMa. (Ambisonic is also as Ambix). There isn’t much difference in the quality between them, only the way the sound location is being coded. Or in other words, the HRTF (Head-related transfer function) are different. In the beginning FuMa was more commonly used by the cinematic VR community, but after both Facebook and Youtube chose to adopt the Ambix format, FuMa is no longer being commonly used.
The new Rule Book on Set
The birth of virtual reality requires us to dumb the rule book from film school and come up with new solutions to the most basic elements in the filmmaking craft. From how to light the set, to where the director stands and also how to record sound on set. The classic boom and LAV mics aren’t applicable anymore in the VR world. I mean, obviously you can use them, but then you would have to composite out the soundman from the panorama, which creates a lot of unnecessary work. Also, you will get a clear sound but it will not be directional or fit for mixing spatially later on. So how do we record sound on a 360 set?
LEARN HOW TO CREATE SPATIAL SOUND FOR IMMERSIVE EXPERIENCES
START LEARNING RIGHT NOW
THROUGH VIDEO TUTORIALS
- Spot microphones
- LAV microphones
- Under camera microphones
Let’s start with identify three different types of microphones we can use:
Under camera Mic
These spatial mics are positioned under the 360 camera to give us a spatial baseline to the scene. But do not think that only having a spatial microphone under the camera is enough. Spatial microphones are recording EVERYTHING around. Meaning, you are grabbing all the information around you on the same volume. Therefore if there is a dialogue in front of you, it will be recorded on the same volume as any other noise around. Therefore, we need more mics and information in order to have an excellent spatial mix.
Spot Mics
One of the core elements of this new craft is to layer and add sound recording on top of one another to create more full audio in 3D. Let’s say that we have a scene; It’s 2 people sitting and talking by the beach. Since we do want a clear recording of the waves, we have the base spatial microphone under the camera. But in order to get a clear audio of the dialogue, we can hide a spot mic near the actors. It can be the Zoom H2N hidden inside a purse, positioned near the characters. Or it can also be an iPhone recording as it’s placed on the table. Later, in post-production, we will take that sound and position where the purse was on the 360 video. This is the best and most effective way to design 360 audio since the microphone isn’t moving and can easily be positioned in its location. To add to that, After I place the spot mics around the set, I come close to the microphones and say to the mic “this mic is positioned inside the purse,” so later in post-production it will be obvious how to the recordings.
LAV Mics
We still use LAV mics in VR, but a lot less. In fact, it’s the last in priority and I will use it only if the character isn’t moving or if it’s really windy or loud on set and I want a clear pickup from the character. The problems with LAV mics in VR are few. First, if the actor moves a lot in space, you would have to track his movement down manually in post-production, and the tools to do that, either with Hear360 or FB360 workstation, are limited. And even if you did a perfect job tracking the actor’s movement and assigning the LAV to pick up sound to it, it will never sound natural. Therefore spot mics are the way to go with VR filmmaking.
I hope this brief description helped you understand your sound tools on set better. Please feel free to subscribe to our channels and the VRTL newsletter. We will post more articles about spatial sound every week. And if you would like to dive into the depth of the spatial sound craft, consider downloading our full course on spatial sound design. In the course, we will go over all the information you need to become a professional in the space. We design the course in a way so you could understand how to work with all the tools available in the market. Now all the bugs, workarounds, and work methods, so you will have a quick up to speed on the subject and start working in the field within a month or two. Thank you for reading and welcome to the VR creators community.
This article was written by Nir Netzer. Nir is a VR director who is working out of Los Angeles. He directed VR projects for DreamworksTV, NBC, and competed in Oculus Launchpad 2018. Nir is VRTL’s instructor for Spatial Sound. You can download his course on this LINK and learn more about Nir’s VR content studio at https://www.netzerland.com.
