AVSynthesis: Blending Light and Sound with OpenGL and Csound5

Figure 3 shows a default empty layer. When the mouse pointer stays on the layer image, a transparent overlay appears with various controls for managing the layer. Click on the icon in the lower-left corner of the overlay to invoke the Layer Editor shown in Figure 4. The icons across the top of the screenshot represent, from left to right, the transformed image, the base image selector, the modulating image selector, the GL shader effect editor, the envelope curve editor and the audio system editor. Let's start our movie-making by selecting our base and modulator images to create an image for treatment by the GL shaders. Next, click on that image (it's the largest of the top three) to invoke the GLSL shader selector, then set the light source, contrast and effect processor for your blended image. Each shader has its own set of performance controls, some of which are shared by all the shaders, while others are unique to the particular effects you've chosen. Figure 4 displays the results of such a process after adding the Wobble shader.

Figure 3. A Blank Layer

Figure 4. The Layer Editor

At this point, you can call the GL shader editor for further finessing of the transformation. Note that the transparency that appears over the blended image includes a play control for testing your later transforms at any point in the process, so feel free to bend, fold, staple and mutilate to whatever degree necessary. Set constraint ranges, apply envelope curves and specify single values. Experiment, experiment, experiment. Be aware, however, that AVSynthesis is short on safeguards, so save your work frequently. There's also no undo/redo, and you receive no warnings about anything except when you decide to quit the program.

Figure 5 shows the control panel for the Wobble effect. The shader's unique controls are at the bottom of the panel and consist of a start slider and two sliders apiece for controlling the frequency and amplitude parameters of the effect. The remaining controls are, as mentioned, common to all the shaders. They include texture managers, a transparency slider, color controls, and eye and light positioners. These common controls can be augmented by extensions required by a particular shader.

Figure 5. GL Shader Controls

A parameter value can be set explicitly with its slider, or you can define a range of values with the constraint mask (the black and gray bars shown in Figure 5) to limit the possible values only to the range covered by the mask. This range can be modified further by one of the envelopes defined in the Curves screen.

The icon at the top-right corner of Figure 4 invokes the AVSynthesis audio system editors. When the icon is selected, a column of new icons appears at the screen's left (Figure 6). From top to bottom, these icons represent the audio sequencer, three synthesizers, three processing modules and the audio mixer. They are all external representations of the Csound engine within AVSynthesis. We'll consider each of these components in turn, but only briefly.

Figure 6. The AVSynthesis Sequencer

The sequencer manages the flow of time for the evolution of both the sound and the video transformations. Lower values represent slower speeds, and higher values make things happen faster. However, time distortion possibilities are rampant in AVSynthesis, and it is not always a simple matter to predict exactly how long a composition will last.

The controls in the synthesis, processing and mixing screens behave exactly like their video counterparts (Figure 7). Values are defined with sliders and masks, envelopes can be placed over ranges and so forth.

Figure 7. A Csound Synthesizer

Incidentally, Csound's deployment is completely concealed to the normal user, and no prior knowledge of Csound or any other programming language is necessary in order to use AVSynthesis.

The test play function is available here too. When you are satisfied with the sound, save the layer, then click the mini-image of the composition editor (at the top-left corner of the Layer Editor) to return to that screen.