DreamWorks Animation "Shrek the Third": Linux Feeds an Ogre

DreamWorks Animation pushes the limits of CG filmmaking with Linux.
Let There Be Global Illumination

“Global illumination is something lighters everywhere love”, says Pearce. “That bounces the light off all the surfaces.” The challenge is global illumination, often implemented as ray tracing, can be expensive. “It's brilliant for architecture fly-throughs where the lighting is static and baked onto the building”, says Pearce. “Our problem is everything in Shrek 3 is moving.” The DreamWorks software “bakes” (calculates ahead of time) what's not moving and then tries to do the parts that are moving efficiently. “Global illumination is in almost every scene in Shrek 3”, says Pearce. “Scene complexity is what caps us now, such as forests. Ray tracing there is not reasonable.” Advances in Linux computing power and multicore chips and software are pushing render capability higher and higher.

“We use global illumination like a DP does”, says Art Department Production Designer Guillaume Aretos. “We use bounce cards and colored bounce cards. The moviemaking process for us is very close to live action where we use the light that bounces. Shrek 2 was a south of Italy feel, kind of Beverly Hills turned into Italy.” Shrek 3 moves away from the eternal spring of first two movies, with a more Northern European look. “It's very hard to get an overcast effect with bouncing backlight”, says Aretos, “and bright light shining through dark clouds.”

“I'm the head of layout”, says Nick Walker, “which makes me the DP.” Layout is the group that figures out where the virtual actors will stand in virtual sets. “Shrek is seven feet tall and all torso”, says Walker. “Shrek could swallow Fiona's head when moving in for a kiss. Puss n' Boots and Donkey, the two sidekicks, are different sizes. It's difficult to get a two-shot.”

Figure 2. Donkey with Shrimp Skewers on Fire—Final Lit Version of the Scene.

The Shrek 3 Linux Pipeline

The production moves from story and concept artwork into 3-D modeling and eventually render. DreamWorks Animation uses the popular Linux Maya commercial package for 3-D modeling. Layout positions the characters in the scenes and determines overall lighting. Models are “rigged” with internal skeletons by the Character TDs, then given to the scene animators. Because of the complexity, Shrek 3 animators were assigned in pairs to each of the hundreds of scenes. In the past, it was one animator per scene. Lighting and any special effects are added, such as cloth or flames. Then, the scene is rendered frame by frame on a 3,000+ CPU Linux renderfarm.

Figure 3. Storyboard: a Story Artist's Rendition of the Scene

Each frame is assigned to a different node of the renderfarm by grid software (using Platform LSF, a commercial Linux package), so that many frames can be output simultaneously. The frames are edited into a movie using Avid software (not on Linux). Early in the process, hand-drawn storyboard images are scanned, and a scratch audio track is edited together creating a rough video representation of the movie. As each sequence is completed, it replaces the rough storyboard footage, building the fully rendered movie scene by scene.

Figure 4. Layout: the Layout Artist's Blocking of the Camera Moves and Character Poses

“The fact that our main production pipeline is all on Linux is pretty interesting”, says head of Production Technology Darin Grant. “We've been on Linux for years, but I'm still amazed. Looking back, when using Linux was a radical concept for Digital Domain's renderfarm during Titanic, it wasn't that long till the industry moved toward wholehearted adoption.” The studio Digital Domain, where Grant formerly worked, built the first Linux renderfarm for Titanic (released in 1997). Grant now works at the PDI DreamWorks facility in northern California and commutes to Glendale each week to ensure that his cross-site team is in sync. He also uses VSC, an immersive video teleconferencing system developed by DreamWorks Animation that HP has since taken to market as the Halo room. HAVEN is the DS3 45mbs Halo video exchange network, used extensively to connect between DreamWorks facilities and with HP's desktop division and with AMD.

Figure 5. Animation: at this stage, the animators create the character's performance for the scene.

“The issues with maintaining a large Linux-based pipeline are the same as maintaining a large pipeline on any operating system”, says Grant. “We unified the studio on one standard pipeline a while ago, and now we have all productions at all times using the same pipeline. They stress, push and develop the pipeline in different ways on each production. Linux provides us with many advantages. Solid support for threading, NFS and LAMP toolsets are big pluses for us. Managing developers gets easier each year as the quality of the development tools and IDEs available on Linux improves.”

Figure 6. Lighting: Final Version of the Scene with Lighting and Textures Added to the Frame

“At the desktop, we use HP xw9300 workstations running RHEL 4”, says head of Digital Operations Derek Chan. “The renderfarm uses HP DL145 G2 servers. Our standard for memory is to have 2GB per core. Servers have four cores, so that's 8GB.” Chan says DreamWorks Animation has a good relationship with Red Hat, working closely to ensure that HP workstations work with Red Hat Linux for DreamWorks.

Figure 7. The Huge Server Farm at DreamWorks Animation

“A challenge we overcame on Shrek 3 was the integration of metadata into our pipeline”, says Grant. “In a cross-team effort between R&D and Production Technology, we put in place a system that maintains historical version information, render statistics and other really valuable data in each and every file we produce. That we were able to do this shows one of the key advantages of having a proprietary toolset and file formats.” DreamWorks uses its own TIFF-like file format based on 16-bit binary fixed point, a limited High Dynamic Range (HDR) image format with a color range of 0 to 2.0. Letting images go whiter than white leaves headroom for image adjustments.

DreamWorks Animation technology is organized into three core groups: R&D to create new technology, Production Technology that oversees the production pipeline and Digital Operations that's responsible for the compute, network and storage infrastructure. The production pipeline has hundreds of small tools and applets that form the other glueware, which enables 200 people to work as an orchestrated team. Most legacy pipeline code is written in Perl, and most of the newer code is being written in Python. “We'd love to be all Python”, says Leonard, “but today we still have lots of Perl”.

“Our team has been spearheading the transition from Perl to Python at the facility”, says Grant. “There are three primary reasons for this. The creation of Python bindings to a C++ library is very easy and allows us to utilize core R&D libraries in the rest of the pipeline more quickly. The object-oriented nature of Python is very attractive given our new asset model and should allow us to make changes to that asset model much more easily in the future. And, Python is a first-class citizen in many of the third-party software applications that are used in our industry.”

At the Linux Movies Conference, an all-day conference for motion picture technologists, last held in 2005 [that I chaired], the consensus of studio technologists was that the constraint on renderfarm size was heat. “The wall is still heat and power”, says Chan, “and a little bit of floor space”. DreamWorks Animation is into dual-core and about to go quad-core. “In the next eight months, we'll switch to eight computing cores per desktop”, says Chan. The studio is interested in accelerated computing with GP-GPU. That has the potential to move render processes from overnight to interactive. But, there are daunting technical barriers in that GPU programming is so alien, and there are bandwidth limitations going between CPUs and GPUs on graphics cards. “There are significant performance gains to be had”, says Chan, “especially if we could keep it all on die with an AMD-integrated GPU”.

Shrek 3 consumes 24TB of storage, out of an allocation of 30TB. DreamWorks likes to keep all its movies in near-line storage on big arrays of spinning disks. “People refer to previous movies all the time”, says Chan. “We have the three Shrek movies. Madagascar is getting a sequel.” Although everything is pretty much kept on-line, DreamWorks archives to tape sent off-site for disaster recovery.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

The improvements being made

Miami DUI Lawyer's picture

The improvements being made in CGI for movies and video games is apparent with each new film or game that is released. The characters are looking more and more life-like. I think the most incredible part is the attention that graphic artists give even to the background characters. It's not just stagnant background people anymore like in earlier versions of CGI productions. The background characters move around and do things in the same great detail as the main character of movies and games. It's amazing what has already come out and I'm excited to see what CGI artists will continue to come out with.

paslanmaz sac

paslanmaz sac's picture

Ty

Thank You

RHM's picture

Thank You

i watched the movie, but

Tarihte Bugun Bir Sey Oldu's picture

i watched the movie, but other Shrek movies were better i think..

i watched the movie, but

Tarihte Bugun Bir Sey Oldu's picture

i watched the movie, but other Shrek movies were better i think..

Thanks

Anonymous's picture

Thank You

Good info

Pharmf866's picture

Hello! dfekkbk interesting dfekkbk site!

I'm surprised, Imm good. My

Pisadam's picture

I'm surprised, Imm good.

My Page : www.forumline.net

Can you come pls ?

I'm surprised, Imm good. My

Pisadam's picture

I'm surprised, Imm good.

My Page : www.forumline.net

Can you come pls ?

türkçe rap müzik

rap's picture

Rendering time...

tomaxxi's picture

So, what is real time to render movie like "Shrek the Third"?
Two days? Two weeks? Two months?

tomaxxi

It is norly

Anonymous's picture

In 2008, Intel had at their disposal 9,000 servers used for verification of their cpus in development. Now in 2009 I expect that number to be even larger. http://7daysport.blogspot.com

Daz Software or E-fronteir Poser?

Michael REMY's picture

hi !

it's strange...in a documentary, i heard that was Poser7 software (not an open-source software one) which was majoritaly used to produce Shrek3 face animation and characters movings.....

so, what 's the true ?

A movie from Linux, but not for Linux

Juanjo's picture

Yesterday my girlfriend gave me a Shrek the Third DVD as a Chrismas gift, I wan't it because I already have the first and the sendcond movie in DVD, but, at this moment I'm trying to see it on my desktop computer, and on my laptop, both running linux, with updated mplayer an xine, and it doesn't work, maybe it's a problem with anticopy protection, it have a notice saying it's only playable from a windows computer. I've also tried it on my legacy XBOX with XMMP, where I get the sound, but not the image.

Dreamworks movie staff does an incredible job making the movies with Linux, but Dreamworks packagers are promoting piracy, because xvid encoded movies work anywhere and original DVDs doesn't.

I wan't to play my expensive DVDs on my linux powered computers. Now I have to download it, because I can't play original DVDs.

Thanks Dreamworks for promoting Linux, but you are also promoting piracy.

How to play encrypted DVD's on xine using linux ?

William's picture

Refer this URL

http://spreadlinux.org.in/HowToCompileXine.html

Section:
Steps to install the multi-media codes and libdvdcss rpm (for playing encryped dvd's)

In short you need to install libdvdcss rpm to play encrypted DVD's
I have watched lots of encrypted DVD's particularly from Dreamworks and Pixar using this rpm installed with xine video player.

Thanks

Dreamworks rock!

kidnames's picture

Dreamworks have revolutionized the movie industry, and mostly with open source, this guys rock.

The moviemaking

Ann's picture

I'm so glad I read this article. I had no idea how it was made until reading this. The process of creating this animation is simply amazing from sketch to miracle ;)

Free for me, None for you!

filasopher's picture

I find it curious when I read articles like this regarding studios whose foundation is touted as "Linux" and the "outstanding" role they play as pioneers in this frontier.
All too often I read that these monster money makers are happy to gleen off of the Open source community, but flinch at the idea of offering something back! These companies to me are nothing more than Judas in corporate atire.
I get frustrated by the copouts given by these organizations. The fact is anything that doesn't directly or indirectly put money in their pocket isn't worth pursuing. I applaud any of the so called advancements that they may have made, but I often wonder how many of these advancements are due to some young upstart college individual who'll never be named.
These articles always read "free for me, cost for you!" Coming from the perspective of the Corporations. It is like that little kid who never learned to share. "My toys! Mine!"
I agree that the article was good. I agree that Linux only made sense coming from IRIX. I also agree that IRIX may have been a pinnacle we didn't realize! Finally, I also agree that it will be a great day when Linux has formidible repetoire of video applications! I mean AVID? Come on!

cool

remix's picture

The reason linux is widely used by film studios is because all their software used to run on irix-workstations. Linux is very close to irix featurewise

Shrek III

Seduzione's picture

I tried the videogame on XBOX360 on Shrek The Third and it's really good! :)

Humble servants ought to be promoted.

M. Miyojim's picture

I see the higher level artistic work for movies being done on proprietary computer platforms and the humble, weight-lifting jobs done by GNU/Linux render farms, as if the tools originally created by the open-source community were only good for secondary roles.

The movie studios should set apart a budget for a small team to extract and wrap byproducts useful for general PCs out there, as payment for the benefits obtained with the success of the movies.

I guess 20% of the savings made by not using proprietary licenses for the thousands of CPUs in the render farm would be enough to pay for this little team. And they should spend some effort to flesh out multiplatform applications replacing the proprietary ones in use, or just kickoff the open-source development for them. It is a fair payback to the community.

3000 CPUs is not really impressive nowadays

Ju's picture

For "who's got the biggest ?" contest, you may check the top500 list
... HPC nowadays uses systems with 10^5 CPUs.

The reason linux is widely used

Anonymous's picture

The reason linux is widely used by film studios is because all their software used to run on irix-workstations. Linux is very close to irix featurewise.

Linux its a probed tech

Luis Medina's picture

This is a great news for me, I do use Linux on Desktop but read this and see where Linux its used say to me what i already know Linux rocks!

Largest commercial linux installation

Daniel's picture

I'm pretty sure Google has them beat

http://en.wikipedia.org/wiki/Google_platform

At least Grid Engine can be

Grid's picture

At least Grid Engine can be used to replace LSF:

Grid Engine Project Home

It's opensource and free.

If you use SGE, you still

Anonymous's picture

If you use SGE, you still have to pay to get "official" support though. That can surpass the cost of LSF.

are they using 5Dwm (SGI clone desktop on linux)

Eric Masson's picture

Just wondering if they were using the Indigo Magic Desktop for Linux a.k.a 5dwm ?

cheers,

EM

http://5dwm.org

Credit to Linux

Rick Stanley's picture

I would like to encourage the Film Industry to recognize Linux and Open Source in the Film Credits. If Linux is contributing so much to the advancements in animation, then they should give credit, and thanks! Let them show their support. They might even give a cameo role to Tux sometime too! ;^)

To Rick Stanley

Fernando Barajas's picture

AFAIK, "Wheezy", the penguin on Toy Story II is a homage to Tux :-)

Sadly no lite versions

Trizt's picture

I know it's would take a lot time from the big companies and cost them money, but it had been nice if they could have released some lite version of their software for home users, not necessarily as opensource but at least as precompiled version for x86, x86_64, PPC, PPC64. Another option could be to release those codes they think are obsolete and release those as they are for the public to use as they wish.

I have seen quite a lot of people asking for this kind of software for Linux, there are those heavy and expensive programs that no normal end-user will afford and sadly they can't get rid of the last dependency they have to microsoft.

I know I'm wishing for too much...

Film making

chris's picture

I do agree with you. I wish I could get rid of Microsoft for film editing. I am curently using Première and Canopus professional boards and codec. Linux software on the subject is very poor.

linux is better than microsoft

grace's picture

with your comment, I respect it, but you just said that so because you were just using microsoft, have you tried linux? It's better to do it in linux.... try to have one..

What Makes Linux Better

ADAC's picture

Getting past the usual religious "Microsoft is better" "No Linux is Better".

I use strictly MS and Photoshop products for my graphics. Is there a good reason why I would want to switch to, or at least have one box operating on Linux? would there really be a tangible effort that would be worth the expense or retraining?

Large commercial linux installations

Anonymous's picture

I'm curious to know why Dreamworks was investigated over Pixar. From all I've been able to glean, Pixar's pipeline runs non-commercialized linux, and has more processors in its farm. "Perhaps no commercial Linux installation is larger than DreamWorks Animation" indeed.

For that matter, I expect Oracle's Austin Datacenter which two years ago housed over 10,000 servers would have eclipsed Dreamworks' as well.

DreamWorks Animation has a

Anonymous's picture

DreamWorks Animation has a lot more CPU's than they are reporting in this article (certainly over double and possibly quadruple).

Thanks for a very well written article!

For shear computing power.....

Anonymous's picture

Anyone interested in this may want to see: "At the heart of the matter" Eweek magazine (trade journal - www.eweek.com), May 21, 2007, pages 25 - 30. They examine the computing grid put together at CERN for the LHC (Large Hadron Collider). A quote from the story: "About 200 institutions in 80 countries - some with their own large data centers - will participate in the grid to process an expected 15 petabytes of data generated per year by the LHC."
No numbers are tallied for the whol grid, but "CERN itself will contribute about 10 percent of the 100,000 or so processors needed for the job. In all, CERN will provide about 8,000 systems - using both single and dual-core chips - to the task".
Presumably this would be the largest NON-commercial concentration of raw computing power using commodity (off-the-shelf) hardware?

In 2004, Intel had at their

Adam Krolnik's picture

In 2004, Intel had at their disposal 6,000 servers used for verification of their cpus in development. Now in 2007 I expect that number to be even larger.

4K rendering, compositing, editing, transfer. whats the pipeline

siddharth's picture

not a discussion on linux as microsoft 64 and vista are going multi processor so its yet to be found who the winner would be. ofcourse savings from not having to use properitery software will be a major consideration.

besides the discussion i had a question...

i wish to find out the production pipeline, the software and equipment used for 3D 4K movie making > Rendering > Compositing > Editing > Transfer to Films > Backup on DLT etc...

my knowledge is 3ds max / maya / ZBrush for 3D >>> Combustion for compositing but not for 4K >>> Premiere for editing but not for 4K >>> what comes next

i need it for 4K. even if the matter is not completely relevant but if someone can take the trouble to answer then i'll be really thankful

thanks and regards

siddharth

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix