Visualizing Sound: A Beginner’s Guide to Using TouchDesigner with Live
An audio wheel by Bileam Tschepe showing a visual representation of the song, "Says" by Nils Frahm
Throughout the history of electronic music, audiovisual art has been integral to its aesthetic identity, both in live performance settings and more recently online. As a result, ever more music makers are incorporating visual media into their work, leading to a new generation of artists who continue to push the boundaries of music’s visual representation.
While there are many different ways to create audiovisual art, we’ve been hearing from a growing number of people who’ve been turning to the node-based programming language TouchDesigner to augment their music with real-time visuals and other media-based works. Developed by the Toronto-based company Derivative, TouchDesigner was created primarily for artists who want to work with image, light and sound. The application is typically used to craft live performances, installations, music visuals and other multi-sensory experiences — although use cases extend far beyond creative arts. Whether it's through projection mapping, virtual reality, or physical computing, TouchDesigner helps people bring their ideas to life in numerous ways.
In this beginner's guide, we'll introduce you to TouchDesigner and explore some of the ways in which you can use the platform to create various kinds of audiovisual art. Along the way, we'll be joined by Derivative’s co-founder Greg Hermanovic and Berlin-based A/V artists Bileam Tschepe and Ioana Bîlea. We’re grateful to them for talking with us and sharing their expertise and insights into the history of TouchDesigner, the available resources for learning the platform, and its creative potential when used with Live.
Greg Hermanovic
From early on in his working life, Greg Hermanovic was always drawn to the idea of creating interactive tools for artists. Prior to his career in graphic software, he worked in aerospace, where he developed real-time simulators for pilot training and for the US Space Shuttle robot arm. Hermanovic later co-founded Side Effects Software, the makers of Houdini, and received two Academy Awards for his work in procedural visual effects. In 2000, he branched off to start Derivative, where he currently leads the team that makes TouchDesigner.
“A lot of the roots of TouchDesigner are in modular and analog synthesizers,” Hermanovic says. “The first synths I got to put my hands on were the EMS Synthi, the Arp 2600, and a bit of Moog. And at the same time, I was really interested in experimental films like “Synchromy” by Norman McClaren. At the time McClaren was creating soundtracks from images drawn directly onto film. So the soundtrack was the image and the image was the soundtrack. This really excited me because I realized, OK, here’s a marriage of the two.”
Elekktronaut
Bileam Tschepe - also known as Elekktronaut - uses TouchDesigner to create interactive, organic, audiovisual art, as well as installations and live performances. He also hosts TouchDesigner workshops and meetups at Music Hackspace and has a YouTube channel featuring easy-to-follow TouchDesigner tutorials. Tschepe describes how he sees images, forms, and colors when he listens to music. “It's very abstract,” he says. “I can’t really explain it. It can be really hard to realize what I see. But, I think TouchDesigner is the perfect tool to do it.”
Spherical Aberration
For Ioana Bîlea, aka Spherical Aberration, getting into audiovisual arts came out of frustration. “I was coordinating a music festival in Romania,” she remembers. “But I always wanted to make music. It wasn't until I moved to Berlin, however, that I found a community who were really supporting female artists. I found the VJ community was generally more open though, so I eventually started VJing with Resolume. But that got boring for me, especially for bigger events where I would have to be there all night. I was looking for a tool that was more entertaining and fun. One night a friend showed me TouchDesigner and I immediately realized that's what I was looking for.”
Artists working with TouchDesigner
Throughout his career in visual effects, Hermanovic has always been inspired and influenced by music culture. In the mid-'90s, he regularly moonlighted as an audiovisual artist for some of Toronto’s early rave parties. These experiences would ultimately contribute to shaping and inspiring many of the features we see in TouchDesigner today. “I was doing visuals voluntarily for this rave company called Chemistry,” he recalls. “I remember trying to project pictures onto the wall in real-time while doing live editing, live coding, and all that kind of stuff. At one event, I was typing in this super long expression of code, trying to get something to wobble on the screen with some decay and stuff like that. And I thought this was insane. I was trying to type parenthesis, going nuts, during a performance, with people stumbling, and dropping their drinks on my keyboard. And so that was the moment I said, OK, we need to put a new system of Operators into our software which we called CHOPS. So CHOPS were born through a lot of pain and inconvenience”.
In the late ‘90s, a fortuitous encounter between Hermanovic and Richie Hawtin paved the way for their first audiovisual collaboration at the Mutek festival in Montreal. At the time Hawtin was using an early version of Ableton Live with a custom MIDI controller, while at Derivative, Hermanovic and his team were working on early builds of TouchDesigner. “Richie and I pushed the technologies together and made a couple of songs,” remembers Hermanovic. “Some worked well, others didn’t. Later, we did the Plastikman 2010 tour with a lot more time, better organization, and technology behind us.”
Since the days of Hermanovic and Hawtin’s pioneering collaborations the number of people using TouchDesigner to combine audio, video, graphics, and data has grown exponentially worldwide. Some notable names using the program include Christopher Bauder, Ben Heim, CLAUDE, Markus Heckmann, and Carsten Nicolai aka Alva Noto. These artists have used TouchDesigner to create a wide range of projects. Many of them also use TouchDesigner as a platform for prototyping and developing new ideas.
“Christopher Bauder’s studio used TouchDesigner for a collaboration with Robert Henke called DEEP WEB,” remembers Bîlea. “They had a giant audiovisual installation at Kraftwerk in Berlin. And there are people like 404zero who use TouchDesigner to generate controlling signals for a modular synth setup. And there’s NONOTAK Studio, who use it to control their light installations with sound.”
Below is a YouTube playlist featuring the work of various artists using TouchDesigner. This playlist showcases a range of creative possibilities which may spark inspiration for those interested in using the platform in their own work.
Learning TouchDesigner
TouchDesigner, like any programming language, has a learning curve, but those with experience in synthesis, Live, or Max/MSP may find some familiar concepts. There are many online resources available to help individuals at any level of experience learn TouchDesigner. “You’ve got to start somewhere and get your feet wet,” Hermanovic advises. “So try something easy first. Then you’ll get instant results which will motivate you to keep building.”
Music Hackspace, in partnership with Derivative, offers a free online course for those just getting started with TouchDesigner. In addition, they hold monthly meetups featuring presentations and discussions on TouchDesigner projects. “The course is designed for beginners or people who’ve never programmed before,” says Tschepe. “Everyone has to learn the same basics at first. Then you can branch out and focus on different parts that interest you. At our meetups, there's always a chance for a Q&A followed by an open discussion. And usually, someone from Derivative joins.”
“One of the things I really like about Music Hackspace meetups and other events we’re at is the opportunity to watch people use our software over their shoulders,” says Hermanovic. “Because quite often what our users really need isn’t apparent from what they say. Also, our users often miss the new tricks and shortcuts we add to TouchDesigner. So for us, the meetups provide good lessons because we can learn how to make new features more apparent to people”.
TouchDesigner also has a thriving community of users who share inspiration and know-how via Facebook groups, the Derivative forum and Discord servers. The community is where users who have skills can connect with those who either need to learn or want to find others to collaborate with.
“It was rough at first because there were no tutorials and the community was super small,” says Bîlea. “But we had a Facebook group and a GitHub page. The people on Derivative’s forum were so nice. You could ask a question, and someone would always jump in and try to help you. Aurelian Ionus aka Paketa 12 has been one of the main contributors to the community. We’ve all been inspired by him. He gives good advice and a lot of it for free. I don't think I would have gotten to where I am now without his coaching.”
“When I started there weren't that many tutorials and no beginner courses,” remembers Tschepe. “Without Mathew Ragan, I wouldn't have known where to begin. There are a few of his university classes on YouTube. These videos were definitely helpful for me when starting out.”
TouchDesigner has a free non-commercial license for users who are getting started. While the non-commercial license limits the resolution, all basic operations and functions remain accessible for those wanting to use it for their personal learning and non-paying projects.
“We really wanted to get TouchDesigner in the hands of as many people as possible so they could see if they like it or not,” Hermanovic mentions. “And we wanted to give access in countries where money is harder to come by, so there’s affordability. People can then make some art that is lower resolution but not watermarked. Some projects you can even do end to end, non-commercially without any limitations really.”
Useful links on getting started with TouchDesigner
Understanding Operators
Operators in TouchDesigner can be thought of as the building blocks or ingredients that make up your project. They are essentially small nodes that perform specific tasks, such as generating images, processing audio, or controlling data. You can use these Operators by themselves or combine them in different ways to create more complex behaviors and effects. To help explain the concept further, Hermanovic likens TouchDesigner’s family of Operators to the Periodic Table of elements. “Imagine every element does something different, there's no commonality,” he says. “And the elements combine to form molecules. And the molecules are these powerful things that form life. That's how I wanted TouchDesigner to work at the basic level. All of the Operators are bite-sized enough that each one has a good job, and can do lots of interesting things without being overloaded with features.”
TouchDesigner has a collection of sixteen Operators that are considered to be the most fundamental and essential for building projects. These are referred to as the “Sweet 16”. “They are the most popular and useful Operators which everyone should get to know,” Hermanovic suggests.
TouchDesigner also features a Palette Browser with pre-constructed networks of Operators, called Components. Beginners can quickly get started by dragging and dropping these Components into their projects. Additional Components can be downloaded from AlphaMoonbase's OLIB library page and ChopChopChop.org.
Algorhythm
For those not wanting to dive too deeply into node-based programming, Tschepe is currently developing Algorhythm, a real-time interface for TouchDesigner that allows users to easily play and manipulate generative visuals and media using audio and other inputs. Algorhythm allows users to fade between visuals and add post-effects without needing to program large networks of Operators. It is currently available through Tschepe's Patreon.
Making audio-reactive visuals with TouchDesigner and Ableton Live
There are several ways that TouchDesigner can be integrated with Live to create audio-reactive visuals and other kinds of interactive experiences. One approach is to use TouchDesigner to control audio and video clips in Live. This can be done by sending OSC (Open Sound Control) messages from TouchDesigner to Live. Another method is to use TouchDesigner to visualize audio data in real-time, such as waveforms or spectrums, using the audio input from Live. This can be done using TouchDesigner's audio analysis tools which convert sound frequencies into numerical data. The data can then be connected to any number of TouchDesigner’s Operators opening up infinite possibilities for real-time music visualization.
TouchDesigner's Ableton Link CHOP can also sync Live Sets using timing information on a Link-enabled network. You can take it further with the TDAbleton tool which offers tight, two-way integration and full access to almost everything going on in a Live Set. The recently released TDAbleton 2.0 update also includes access to Clip information in Live 11 arrangements and allows users to trigger Macro Variations and access all 16 Rack Marcos.
Alternatively, OSC signals can be sent between Live and TouchDesigner using the LiveGrabber Max for Live plugins.
“I’m happy there are great audio tools already out there through Ableton, VST and stuff like that,” says Hermanovic. “So we don’t really have to go there. But just being on the periphery of audio and being able to talk two ways with it is really good. I’m excited about the development of TDAbleton. The feedback I’m getting is that its link is pretty tight, flexible and open”.
Useful links on audio-reactive visuals and TouchDesigner’s integration with Live
Choosing the right computer
When choosing a computer for use with TouchDesigner, it is advisable to opt for the most powerful configuration that your budget allows. This will provide the best performance and enable you to tackle more complex projects. Factors to consider when determining the spec of your computer include:
Processor: TouchDesigner can be a CPU-intensive application, so it's important to have a powerful processor. Look for a processor with a high number of cores and a high clock speed.
Graphics card: TouchDesigner is also graphics-intensive, so a good graphics card is important. Look for a card with a high amount of dedicated video memory and support for GPU acceleration. It’s also worth noting, certain TouchDesigner features will only work with Nvidia GPUs.
Memory: TouchDesigner benefits from a lot of memory, so look for a computer with at least 16GB of RAM.
Storage: TouchDesigner files can be quite large, so it's important to have a fast and large hard drive or SSD to store them on.
Portability: If you need to take your computer with you on the go, you'll want to consider its size and weight.
“Technically for the basic stuff, you don't really need a good computer,” says Tschepe. “But, if you’re doing some high-end 3D work you’re going to have issues. You’ll need a really powerful computer. Or if you want to render an 8K live projection, obviously you have to get a really good graphics card.”
“I bought a gaming laptop,” says Bîlea. “When I bought it three years ago it was one of the most powerful of its kind. Now it’s already falling behind. But it’s still working well for what I need. The Nvidia graphics cards upgrade pretty fast though. You could change your laptop every year if you wanted.”
“At one time I had to ask Silicon Graphics to borrow one of their workstations,” Hermanovic reminisces. “It took four people to move, they were so heavy. They cost $250,000. There were these poor guys pulling it up stairways into the space. Fortunately, the affordability of hardware has come down a lot. We’re piggybacking off the gaming industry because it’s driven the price of graphic chips right down.”
Finding inspiration
There are many ways to find inspiration for audiovisual projects in TouchDesigner. One way is to attend events such as music festivals, and conferences to see what other people are doing with audiovisual technology. You can also look at the portfolios of other artists and designers to see what they have created. And of course, looking to other art forms and occasionally stepping away from the computer and into the world can provide endless ideas and inspiration.
“Personally I think I pick ideas more from non-TouchDesigner-related things,” says Hermanovic. “There’s so much amazing art in many other fields spanning hundreds of years. I’m influenced by painting and photography from way back. I really liked going to NYC in the ‘70s because of the explosion in the Soho area of art galleries and large-format art and installations. That was a hotbed of creativity back then.”
“My visuals are inspired by nature and organic textures,” says Bîlea. “Everything is evolving and slow-moving. I don't usually work with sharp edges or bright colors. I think my music has the same source of inspiration, so it's very ambient.”
“I also like to recreate things I see in nature,” says Tschepe. “I can be inspired by anything really. Anywhere I see natural patterns. It could even be the inside of a coffee mug.”
Forms in Wood, by Bileam Tschepe
We hope that this beginner’s guide has given you a taste of the creative possibilities of using TouchDesigner together with Live and some ideas on where to start. Whether you're an experienced A/V artist looking to expand your skills, or a newcomer to the field looking to explore the intersections of sound, music, and visuals, TouchDesigner offers a wealth of opportunities for exploration and expression. There are no strict rules for using these tools, and the only limit is your imagination. We look forward to hearing – and seeing – what you come up with.
Keep up with Elekktronaut and Spherical Aberration. Follow Derivative on YouTube
Text and interviews by Joseph Joyce and Phelan Kane
Artwork by Bileam Tschepe (Elekktronaut)