The Power of the Incredible Hulk—the ILM Linux Death Star

The story in the July 2002 issue of Linux Journal about the Linux desktop and renderfarm conversion at Industrial Light & Magic (ILM) drew record interest from readers. Linux has advanced at ILM in the past year, meeting the special challenges ILM faced in creating the movie The Hulk (release June 2003).

The Star Wars Death Star exists! But, not as a menacing planet-killing weapon. ILM's Death Star renderfarm is the computing power behind the many motion pictures produced at Industrial Light & Magic. “Linux will just continue to grow”, says ILM CTO Cliff Plumer. “Our renderfarm has over 1,500 processors currently, and almost 1,000 more are added every evening with desktops.” The renderfarm utilizes both dedicated CPUs and the computing power of idle desktops.

The Death Star Renderfarm

“Our core renderfarm is comprised almost entirely of Linux boxes since we switched over from SGI machines”, says ILM Systems Developer Mike Thompson. “We have about 750 nodes—1,500 CPUs.” The renderfarm exists as a row of computing towers made up of 1U rackmount dual-processor PCs. However, this isn't a supercomputer in the classic sense. Each machine operates semi-independently in a grid configuration, not bound together as a supercomputer running a single job. At ILM, a proprietary batch-scheduling program called ObaQ manages the workload across machines.

Figure 2. ILM RackSaver Linux renderfarm has 1,500 processors now, which will double for Star Wars Episode III.

As the raw horsepower of systems increased, so did the demand for electricity and cooling in ILM's machine room. “It's important to reduce power and heat”, says Thompson. “We went with AMD Athlon 1600 CPUs, a low-power variant that can be a little difficult to find.” Each node has 2GB of memory, expandable to 4GB, and each node usually runs two jobs at once to utilize the dual processors, but sometimes this is reduced to a single job for full use of memory.

The AMD-powered nodes are RS-1100 units produced by RackSaver. RackSaver caught ILM's attention at a National Association of Broadcasters (NAB) convention and was given the opportunity to bid on building the renderfarm. RackSaver competes mostly with heavyweights IBM and Dell. RackSaver CEO David Diggers says RackSaver's advantage is servers with double the density of competitors. “We're very strong in this vertical market with sales to ILM, Pixar and Warner Brothers”, he said.

The RackSaver renderfarm servers are connected via 100BASE-TX into a Foundry 8000 switch that aggregates network traffic into a gigabit link into the network core. “Just recently we added a 10-gigabit link into our core which helped a lot”, says Thompson. “We have a file server core and 2,500 rendering cores with total aggregate traffic of about 70TB a day.”

Why Green Is Scary

The Hulk is not a typical comic book movie”, says Technical Director Doug Sutton. “It is some of the most challenging work we've done in years. Making a 15-foot-tall green guy look real is an incredible challenge!” The Hulk is a computer-generated (CG) digital actor with emotions and complex green skin.

“Film is designed to make people not look green, to push green away”, says Principal Software Engineer Rod Bogart. “We're using Kodak Premiere print stock that has deeper color than Kodak Vision. If the character is green, as with Hulk, it's hard to make it look like green skin. The green dinosaurs of Jurassic Park are not as hard. Dinosaurs don't have the same sort of skin highlights as people.” The audience is less forgiving about human faces—even green ones. “Green passes through yellow as it goes to white”, adds Sutton. “You need to see that on a monitor in order to counteract or accept it.”

Live action was shot in the streets of San Francisco, California. Jennifer Connelly would be in the middle of a street acting to a big green guy who isn't there. A grip holding a pole with green head on top would be her only cue. “They did a few really cool practical effects, explosions, but mainly it is CG”, says Sutton, “a combination of Maya and SOFTIMAGE”.

Cineon and OpenEXR

Capturing film images demands higher dynamic range than the typical JPEG or PNG supports. Kodak Cineon has long been the standard for digitized film. Cineon is a 10-bit logarithmic format. Compared to JPEG, which is 8-bit linear (that is, 24-bit RGB), a Cineon image has more dynamic range and is especially rich in colors near black. As computing power has increased, most computation has switched to floating point, and working in 10-bit log has become a limitation. OpenEXR is a new floating-point image format.

“The OpenEXR file format is a better digital representation of film because it has a dynamic range of over 30 f-stops without loss of precision”, says Plumer. “Previous 8-bit file formats have the dynamic range of only around seven to ten f-stops and cannot accurately reproduce images with extreme contrast.” ILM created the EXR format in the summer of 2000 and has used it on Harry Potter and the Sorcerer's Stone, Men in Black II, Gangs of New York, Signs, Dreamcatcher, The Hulk, Van Helsing, Peter Pan, Timeline and Pirates of the Caribbean.

OpenEXR uses lossless compression like PNG, not lossy compression like JPEG. Actually, there is an unused Piz12 lossy compression option. Unlike PNG or JPEG, OpenEXR uses wavelet encoding—basically a tree structure containing the signed differences between pixels. Because the magnitude of the numbers is smaller and there are fewer unique values, Huffman encoding can compress that more efficiently. EXR supports 32-bit float, 32-bit integer and 16-bit float to any number of channels. Channels can be different depths; for instance, RGBAZ images need more precision in Z depth—with 16-16-16-16-32 typical. The Z channel is physical depth, not a color or alpha mask—think of Z sort of like sonar. In January 2003, ILM released open-source OpenEXR. The first open-source application to support it was CinePaint.