Advanced Video Coding on Linux

Use H.264 to create high-quality, low-bitrate digital video with currently available tools on Linux.
QuickTime and H.264

It is nice that QuickTime 7 supports H.264-encoded video. Apple itself encodes all of its movie trailers on-line using H.264. Although this is good, and fosters the adoption of this codec, the QuickTime implementation has some limitations, most notably with B-Frames and Profile support. We need a short detour to explain what this means for our encoding project.

The MPEG standard for H.264 includes a number of profiles, such as Baseline, Main, Extended and High. These profiles delineate different technical capabilities that a decoder may need to possess. As its name suggests, the Baseline profile is the simplest and least-demanding profile, and Main, Extended and High require more processing power and the interpretation of more technical features in order to decode properly. QuickTime 7 supports Baseline and parts of the Main profiles; however, it chokes on features of the Extended and High profiles.

B-Frames are a type of storage format for digital video. These types of frames reference information from other previously decoded frames in order for the decoder to do its job properly, which is to decode the video. B-Frames are interleaved amongst other frame types known as I-Frames and P-Frames. It's a technical detail, but the QuickTime 7 H.264 decoder can support only up to two B-Frames, no more. This is unfortunate, because using more B-Frames would let us increase quality under some circumstances.

To remain QuickTime-compatible, we need to keep these limitations in mind. However, the quality of our low-bitrate encoding will not really suffer that much, even with these limitations. And, we can enable a few additional options to improve things quite a bit. The first is the subpixel motion estimation (--subme) size, which controls the precision of motion estimation calculations used by x264 during the encoding process. By increasing this to 6, the maximum, we gain a lot of visual quality at the cost of some additional encoding time, but it is worth it. We also can configure how x264 analyzes frames to perform better motion estimation (--analyse), which leads to higher-quality encodes. Note that some types of analysis are for High profile encodings only, such as 8x8 DCT, which are not supported by QuickTime, so we avoid those settings. We also can disable PSNR calculations (--no-psnr) to buy back a little speed during the encode. PSNR is simply a quality measurement and has no effect on the actual encoding quality.

Putting all this together, we can now output a high-quality, low-bitrate, QuickTime-compatible and standards-compliant video encoding using H.264:

mkfifo tmp.fifo.yuv
mencoder -vf format=i420 -nosound -ovc raw -of \
    rawvideo -ofps 23.976 -o tmp.fifo.yuv \
    max.dv 2>&1 > /dev/null &
x264 -o max-video.mp4 --fps 23.976 --bframes 2 \
    --progress --crf 26 --subme 6 --analyse \
    p8x8,b8x8,i4x4,p4x4 --no-psnr tmp.fifo.yuv 720x480
rm tmp.fifo.yuv

We can make further improvements. Because this video file is destined for the Web, we most likely want to reduce the frame size to something more friendly, possibly crop out unwanted areas, and make other adjustments. For example, to reduce the frame size, run the following commands:

mkfifo tmp.fifo.yuv
mencoder -vf scale=480:320,format=i420 -nosound -ovc \
    raw -of rawvideo -ofps 23.976 -o tmp.fifo.yuv \
    max.dv 2>&1 > /dev/null &
x264 -o max-video.mp4 --fps 23.976 --bframes 2 \
    --progress --crf 26 --subme 6 --analyse \
    p8x8,b8x8,i4x4,p4x4 --no-psnr tmp.fifo.yuv 480x320
rm tmp.fifo.yuv

Here we instruct mencoder to scale the output to 480x320 and also tell x264 to accept that frame size. This will further reduce the file size, which is appropriate for video on the Web.

Final Steps

Based on the QuickTime format, the .mp4 container format can store many types of media and is also the MPEG standard for storing H.264 video and AAC audio, which is how we will be using it. Use MP4Box, which is part of the gpac project, to combine the audio and video streams we've just created:

MP4Box -add max-video.mp4 -add audiodump.aac \
   -fps 23.976 max-x264.mp4

This produces the final output file max-x264.mp4. You can play back the file with MPlayer, or with Apple's QuickTime player on a non-Linux OS. You also can embed this file into a Web page for playback from a browser by using Apple's instructions for embedding QuickTime movies (see Resources). Free software tools such as the mplayer-plugin can be used to play this file from within Firefox on Linux.

By way of comparison, here are the file sizes and bitrates of the original raw DV file max.dv, our H.264-encoded file max-x264.mp4 and a corresponding XviD encoding max-xvid.avi, which was created from the same source video (see Resources):

mencoder max.dv -vf scale=480:320 -ovc xvid -xvidencopts \
   fixed_quant=7:qpel:nopacked -oac mp3lame \
   -ofps 24000/1001 -o max-xvid.avi

Table 1. Table of Results

FileFile SizeVideo Bitrate

And here are accompanying screenshots for each sample.

Figure 1. DV

Figure 2. XviD

Figure 3. x264

As you can see, the visual quality of the H.264-encoded file is just as high as the XviD version, arguably higher, but at a lower bitrate and file size. This shows that you can achieve similar results in less space, or much better results in the same space, with H.264 compared to other codecs such as XviD. In addition, the work flow and options for encoding with x264 are similar to XviD, but with greatly improved output. So, if you are used to encoding with XviD, many of the concepts and options should be familiar to you when working with x264.

Figure 4. XviD Detail

Figure 5. x264 Detail

The more you experiment with x264, the more you will discover the amazing savings in bitrates and file sizes while still maintaining an extremely high visual quality. The world of video encoding is definitely a black art, as there are hundreds of variables and options that can be brought to bear in any particular encoding project. There is no one-size-fits-all method of video encoding. However, the technical superiority of H.264 over XviD or regular MPEG-2-encoded video is too great not to take advantage of it. And, you can start taking advantage of it today, using the tools described above. Because H.264 is an MPEG-standard encoding, used with an MPEG-standard audio codec inside of an MPEG-standard container format, all the work you invest in using these tools to encode your video will be future-proof as well as high-quality. Use the techniques outlined above as a starting point for your own H.264-encoding projects, and you'll discover why H.264 is becoming the next standard for video encoding.

Resources for this article: /article/9197.

Dave Berton is a professional programmer. He can be contacted at



Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

A quick ending.

Anonymous's picture

mplayer -ao pcm -vc null -vo null max.dv
MPlayer SVN-r29237-4.4.1 (C) 2000-2009 MPlayer Team
mplayer: could not connect to socket
mplayer: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.

Playing max.dv.
File not found: 'max.dv'
Failed to open max.dv.

Exiting... (End of file)

Also tried sudo with the same result.

Son :) max.dv is the name of

Anonymous's picture

Son :) max.dv is the name of the file you're processing. That the name of his (author's) file. Maybe your file has a different name?

Why don't you try replacing the max.dv with the name of your file!?


H264 coding/decoding in hardware?

Vassili Leonov's picture

Hi everybody,

It is very pleasing to see solid and rapidly growing Free Software / open source support for H264 video encoding standard. There are multiple options for doing H264 under GNU/Linux. The completeness of implementation is also getting better - for example handling of interlaced video had been finally improved in 2009 for libavcodec - it decodes *supposedly* H264 specification compliant Mobilygen MG1264 audio/video codec chip produced streams now - you can check MG1264 MP4 samples here - - if D1 (full resolution) samples are NOT playing back for you properly it means that your version of libavcodec / mplayer is not recent enough :-) These are full D1 2000 Kbps samples and one 500 Kbps CIF sample, which should playback well everywhere, except from QT player maybe.

Although this article shows the way to use H264 software encoding based on x264 tool, I think it's a good place to ask author and readers about other known Linux H264 hardware devices - for either PCI / PCI-E bus or USB bus - any H264 done in hardware and usable under Linux for integration - with GNU/GPLed drivers like LML26415 PCI card has, or even binary-only non-free drivers or transparent interfaces like RTP/RTSP streams etc. We are building various multi channel video processing systems and having additional choices for H264 standard (720x480/576) and especially high definition video (1920x1080) done in *hardware* for very low CPU overhead is very important. Although hardware design and OEMing is fun thing to do and we enjoy every moment of it :-), sometimes it's desirable to use something that is already available - if only it's Linux friendly. I would appreciate any feedback here or sent to my email


avi to mp4

Shinji's picture

I have one question. Is posible to improve the quality of a avi raw, compressing this in mp4? I have seen in some places taht people download the avi raw and convert it to mp4 codec and the result is better than the source (avi raw). I tried to do this with x624 and similar options like you, but the result is the same than the raw.
Thanks and sorry for my english :S

no it is not unless you use

Anonymous's picture

no it is not unless you use some filters, but you have to remember that h264 and xvid aren't lossless codecs so each time you re-encode something using these codecs you lose quality

The two wires crossing the top of the clock tower ?

UEKAE's picture


What happened to the two wires which were crossing the top of the clock tower ? You can see them in the DV image but missing in the xvid and x264.



Interlaced Video

fearofcarpet's picture

I generally use mencoder to convert interlaced 720:480 m2v transport streams to x264 AVIs. I would like to use this method to make Quicktime friendly files, but mencoder says "Skipping Frame!" (which it always does with this video source) which results in mp4 files in which every other frame is blank. Is there an easy way around this? I've played around with the usual pullup and deinterlace filters that I use to make AVIs, but no luck...

Interlaced Video

fearofcarpet's picture

A solution is to use -of rawvideo with mencoder's native x264 encoder and then mux it all up with MP4Box... If I use the same quicktime-friendly settings I get a quicktime-friendly MP4 out in the end.

An aspect ratio problem

Horace Mitchell's picture

I enjoyed your article, but I think there is a subtle problem with your examples. You start with a DV stream whose natural resolution is 720x480. This seems to be 3:2 aspect ratio, but the stream is designed to play back at 4:3 aspect ratio or 640x480 resolution. This has to do with the legacy of NTSC video having more pixels per inch along a horizontal scan line than vertically across scan lines. I'm not sure, but I don't think H.264 has an aspect ratio flag to account for this issue (MPEG-2 does have such a flag). As a result, I think all of your H.264 output files are stretched horizontally by 720/640. To get the correct result, you probably have to tell the encoder to output a file with a 4:3 aspect ratio. You don't seem to be doing that in the examples.

Specifying aspect ratio, and issues with Quicktime playback

Anonymous's picture

You can set this ratio by specifying the "sample aspect ratio" to x264 using --sar, and also by specifying the "pixel aspect ratio" to MP4Box using :par. I believe these both effect the aspect ratio in the same way. The calculation of the aspect ratio will depend on the properties of the source video clip. More discussion of aspect ratios here. H.264 does have support for aspect ratio signalling, but it of course depends on the decoder to interpret it correctly. For small web clips, it is often best to just resize the source video so that the aspect ratio is 1:1, which will be displayed correctly by any decoder which understands h264.

As one example, Quicktime 7 will not respect these aspect ratio settings, and part of the article deals with issues of compatibility with Quicktime. Although you can set the aspect ratio in the .mp4 file, it will still appear incorrectly when played in the Quicktime player. To fix this problem, open the properties of the video file in Quicktime and manually specify the correct horizontal and vertical dimensions, then save the file (the ability to do this may be restricted to Quicktime Pro). This (non-standard) method will correct the aspect ratio (for Quicktime-playback only).