Free Tools for Live Unlock 3D Spatial Audio, VR, AR

Envelop began life by opening a space for exploring 3D sound. Now, the nonprofit, directed by artist Christopher Willits, has released a set of free spatial sound tools you can use in Live 10.

What is spatial audio?

First, let’s back up. Listening to sound in three dimensions is not just some high-tech gimmick. It’s how you hear naturally with two ears. The way that actually works is complex – the Wikipedia overview alone is dense – but close your eyes, tilt your head a little, and listen to what’s around you. Space is everything.

And just as in the leap from mono to stereo, space can change a musical mix – it allows clarity and composition of sonic elements in a new way, which can transform its impact. So it really feels like the time is right to add three dimensions to the experience of music and sound, personally and in performance.

Intuitively, 3D sound seems even more natural than visual counterparts. You don’t need to don weird new stuff on your head, or accept disorienting inputs, or rely on something like 19th century stereoscopic illusions. Sound is already as ephemeral as air (quite literally), and so, too, is 3D sound.

So, what’s holding us back? Well, stereo sound required a chain of gear, from delivery to speaker. But those delivery mechanisms are fast evolving for 3D, and not just in terms of proprietary cinema setups.

But stereo audio also required something else to take off: mixers with pan pots. Stereo only happened because tools made its use accessible to musicians.

Looking at something like Envelop’s new tools for Live 10, you see something like the equivalent of those first pan pots. Add some free devices to Live, and you can improvise with space, hear the results through headphones, and scale up to as many speakers as you want, or deliver to a growing, standardized set of virtual reality / 3D / game / immersive environments.

And that could open the floodgates for 3D mixing music. (Maybe even it could open your own floodgates there.)

Envelop tools for Live

Envelop for Live (E4L) has hit GitHub. It’s a free set of tools – but you do need the full version of Ableton Live 10 Suite (since it provides the requisite set of multi-point audio plumbing). Provided you’re working from that as a base, though, musicians get a set of Max for Live-powered devices for working with spatial audio production and live performance, and developers get a set of tools for creating their own effects.

Start here for the download:

www.envelop.us/software

See also the more detailed developer site:

github.com/EnvelopSound/EnvelopForLive

Read an overview of the system, and some basic explanations of how it works (including some definitions of 3D sound terminology):

github.com/EnvelopSound/EnvelopForLive/wiki/System-Overview

And then find a getting started guide, routing, devices, and other reference materials on the wiki:

github.com/EnvelopSound/EnvelopForLive/wiki

It’s beautiful, elegant software – the friendliest I’ve seen yet to take on spatial audio. Kudos to core developers Mark Slee, Roddy Lindsay, and Rama Gotfried.

Here’s the basic idea of how the whole package works.

Output. There’s a Master Bus device that stands in for your output buses. It decodes your spatial audio, and adapts routing to however many speakers you’ve got connected – whether that’s just your headphones or four speakers or a huge speaker array. (That’s the advantage of having a scalable system – more on that in a moment.)

Sources. Live 10’s Mixer may be built largely with the idea of mixing tracks down to stereo, but you probably already think of it sort of as a set of particular musical materials – as sources. The Source Panner device, added to each track, lets you position that particular musical/sonic entity in three-dimensional space.

Processors. Any good 3D system needs not only 3D positioning, but also separate effects and tools – because normal delays, reverbs, and the like presume left/right or mid/side stereo output. (Part of what completes the immersive effect is hearing not only the positioning of the source, but reflections around it.)

In this package, you get:

Spinner: automates motion in 3D space horizontally and with vertical oscillations

B-Format Sampler: plays back existing Ambisonics wave files (think samples with spatial information already encoded in them)

B-Format Convolution Reverb: imagine a convolution reverb that works with three-dimensional information, not just two-dimensional – in other words, exactly what you’d want from a convolution reverb

Multi-Delay: cascading, three-dimensional delays out of a mono source

HOA Transform: without explaining Ambisonics, this basically molds and shapes the spatial sound field in real-time

Meter: Spatial metering. Cool.

All of this spatial information is represented via a technique called Ambisonics. Basically, any spatial system – even stereo – involves applying some maths to determine relative amplitude and timing of a signal to create particular impressions of space and depth. What sets Ambisonics apart is, it represents the spatial field – the sphere of sound positions around the listener – separately from the individual speakers. So you can imagine your sound positions existing in some perfect virtual space, then being translated back to however many speakers are available.

This scalability really matters. Just want to check things out with headphones? Set your master device to “binaural,” and you’ll get a decent approximation through your headphones. Or set up four speakers in your studio, or eight. Or plug into a big array of speakers at a planetarium or a cinema. You just have to route the outputs, and the software decoding adapts.

Envelop is by no means the first set of tools to help you do this – the technique dates back to the 70s, and various software implementations have evolved over the years, many of them free – but it is uniquely easy to use inside Ableton Live.

Open source and standards

It’s significant that Envelop’s tools are available as free and open source. Max/MSP, Max for Live, and Ableton Live are of course proprietary tools, but the patches and externals exist independently, and a free license means you’re free to learn from or modify the code and patches. Plus, you can share your projects across machines and users, provided everybody’s on Live 10 Suite.

The underlying techniques here are all fully open and standardized. Ambisonics work with a whole lot of different 3D use cases, from personal VR to big live performances. By definition, they don’t define the sound space in a way that’s particular to any specific set of speakers, so they’re mobile by design. That means if you’re learning spatial sound as a kind of instrument, here you don’t have to learn each new system as if it’s a new instrument, or remake your music to move from one setting to another.

Advanced Max/MSP users will probably already be familiar with the basic tools on which the Envelop team have built. They’re the work of the Institute for Computer Music and Sound Technology, at the Zürcher Hochschule der Künste in Zurich, Switzerland. ICST have produced a set of open source externals for Max/MSP.

Venues, VR, AR and Beyond

You do need compelling venues to make spatial sound’s payoff apparent – and Envelop are building their own venues for musicians. Their Envelop SF venue is a permanent space in San Francisco, dedicated to spatial listening and research. Envelop Satellite is a mobile counterpart to that, which can tour festivals and so on. Because Envelop SF and Envelop Satellite also have some LED effects, you’ll find some devices for controlling those in the E4L toolkit (these might also be useful templates for stuff you’re doing).

In addition to venues, there’s also a growing ecosystem of products for production and delivery, one that spans musical venues and personal immersive media. To put that more simply: after well over a century of recording devices and production products assuming mono or stereo, now they’re also accommodating the three dimensions your two ears and brain have always been able to perceive. And you’ll be able to enjoy the results whether you’re on your couch with a headset on, or whether you prefer to go out to a live venue.

Ambisonics-powered products now include Facebook 360, Google VR, Waves, GoPro, and others, with more on the way, for virtual and augmented reality. So you can use Live 10 and Envelop for Live as a production tool for making music and sound design for those environments too.

Text: Peter Kirn

A longer version of this article appeared on Create Digital Music.