AVSynthesis: Blending Light and Sound with OpenGL and Csound5

The artistic combination of sound and image is a common enough phenomenon. Movies, television and various Internet channels demonstrate the happy results from the blend of recorded sight and sound. However, these examples typically utilize sound in the role of an accompanist, perhaps greatly significant but still primarily an accompanist.

There is another way to consider the role of music and sound in video production—a way in which the sound itself informs the flow of images and their transformations. Although not a novel concept (see the Wikipedia entry on John Whitney), the practice has taken on a new richness of possibilities with the use of computers in the recording and editing of digital son et lumière.

Jean-Pierre Lemoine has been exploring these new riches at least since the late 1990s. I profiled his HPKComposer (coauthored with Didiel Debril) in my Book of Linux Music & Sound, which was written in 1999, and even then the HPKComposer Web page stated that the program was “... a 3D art composition tool for Csound”. At that time, the authors chose to use the Virtual Reality Modeling Language (VRML) for its graphics engine. I could meet the program's Java requirements and work with its Csound side, but I was unable to work with VRML under Linux then. Nevertheless, the Web site's screenshots made quite an impression, and I hoped that someday such a program would become useful under Linux.

Cut to the work of Csound developer Gabriel Maldonado: his CsoundAV for Windows is a true fork from the canonical Csound source tree, but Gabe is a genial fellow who freely offers all his code extensions to the community. Recent developments in canonical Csound have facilitated the adoption of some CsoundAV opcodes, though we await the inclusion of the CsoundAV opcodes for OpenGL, and this situation brings us to the latest work of Jean-Pierre Lemoine, titled simply AVSynthesis (Figure 1).

Figure 1. AVSynthesis in Play

AVSynthesis embraces and extends many of the design concepts behind HPKComposer. The program blends sound and images to produce abstract non-representational works of art. It's written in Java, and Csound is still the audio engine of choice, but the VRML interface has been replaced by a set of image controls based on the OpenGL shading language (GLSL). The program creates radical associations and correspondences between image and sound, leveraging the powers of Csound and OpenGL for the arbitrary manipulation of digital audio and digital images.

Like many experimental applications, AVSynthesis is not a perfectly packaged program, and it is not ready for use right out of the box. It is a unique program, and as such, it has some unique requirements that may not be met by your distribution's package repositories. Building the required dependencies is not especially difficult, as long as you have a typical Linux development environment installed and properly configured for your system. I include here the particular instructions for compiling Csound and configuring AVSynthesis, with some notes on the requirements for building the application on a 64-bit system.

AVSynthesis demands a specific set of dependencies:

Where they are noted, the versions are critical, and each component is subject to its own build prerequisites. As mentioned, Csound needs some special attention in order to use it with AVSynthesis.

Csound has its own set of necessary dependencies, but space restrictions here forbid a complete description of the program and its requirements. Fortunately, thorough and excellent documentation is available from www.csounds.com, so I focus here only on the configuration needed to compile the program for use with AVSynthesis.

The following options configure and compile the csound binary for double-precision floating-point numerics and create lib_jcsound.so, a Java “wrapper” library for Csound's audio synthesis and processing services:

scons useDouble=1 install=1 buildPythonOpcodes=1 buildInterfaces=1 
 ↪buildJavaWrapper=1 dynamicCsoundLibrary=1

The Python opcodes are not required by AVSynthesis, but I include the option for use with Steven Yi's blue, a superb environment for working with Csound. All other options in this build configuration must be included for work with AVSynthesis. If the build is successful, the lib_jcsound.so library will be at the top level of the Csound source tree. Install Csound (scons install), then copy lib_jcsound.so to the AVSynthesis native directory. That's it; you're finished with setting up the audio side of AVSynthesis.

The OpenGL and LWJGL libraries provide the interface's visual components and style. The various parameter control screens resemble the control panels seen in many OpenGL-based games, with visual effects, such as animated icons and mobile transparencies—niceties that liven the appearance of the program and improve its work flow.

The LWJGL libraries present a minor difficulty. The AVSynthesis package includes the LWJGL libraries as Windows-format DLLs but not the required native Linux libraries (that is, in .so format). The package includes these DLLs:

Those files must be replaced by the following native Linux equivalents:

The lib_jcsound.so library comes from the Csound build described above; the others come from the LWJGL binary package (downloaded from lwjgl.org). Alas, 64-bit users will need to build and install the LWJGL and the IL libraries themselves. As far as I could tell, packages for these libraries are not readily available in 64-bit format, but building them is trivial and requires no special instructions beyond adding --with-pic to the configuration step (./config --with-pic). After building or downloading the libraries, they must be copied to the AVSynthesis native directory. You then can move or delete the DLL versions.

Neither Java nor OpenGL requires any rebuilding or special runtime options. These are common packages now, so if you don't have them installed already, summon your package manager and install the latest versions (Java must be 1.5 or higher). AVSynthesis itself is launched from a .jar file that works equally well in a 32-bit or 64-bit environment.

In addition to these software requirements, your computer should have a fast CPU and a video system capable of accelerated 3-D graphics. I tested AVSynthesis on two machines: a 32-bit box with an AMD64 3800+ CPU (a 2.4GHz chip) and a 64-bit machine powered by an AMD64 3200+ CPU (2GHz). Both systems include NVIDIA graphics boards (GeForce 7300GS and GeForce 7600GS, respectively), with xorg.conf configured for NVIDIA's proprietary nvidia driver (that is, not the open-source nv module). The 32-bit iron runs the JAD distribution, based on OpenSUSE 10.2, and my 64-bit box runs 64 Studio, a Debian-based distro. Both systems are optimized for multimedia and include kernels optimized for real-time performance. However, programs such as AVSynthesis want resources, lots of them, and I consider my machines as rather low-end for AVSynthesis. Your mileage may vary, of course, but for the best results from this program, I recommend a 3GHz CPU, at least 2GB of RAM, a fast 3-D graphics card and a large, fast hard disk.

I also recommend a high-quality audio system. Cheaper desktop speaker arrays may be suitable for watching DVDs, but Csound is capable of audiophile-quality output, so you'll want a sound system as powerful as your graphics system. Here at Studio Dave, I have my JAD box connected to a relatively low-end 5.1 sound system (a combination of Creative Labs and Peavey hardware), while the 64 Studio machine is hooked up to a conventional small studio audio system with a Yamaha digital mixer, a standalone 100-watt power amplifier and a pair of high-quality monitor speakers.