Editors' Note: This article has been updated since its original posting.

Seven years ago, when I was still a clueless kid, I played my first video clip on the Internet using a 56k connection. It was in Real Media format and contained a video track in 11Kbps and a mono audio track. All the buffering drove me crazy. Today, with all sorts of high-speed lines, you seldom come across network congestion during playback of a streamed video clip. This new speed has enabled rapid development of multimedia on the Internet. One of them is video streaming.

What is video streaming? Traditionally, we must download the whole video file in order to play it. With streaming, video data sent to the streaming client is decoded and played immediately, as it is received. Implementations such as video conferencing, surveillance systems and video on demand (VOD) all are based on video streaming technology.

When people talk about video streaming, the first word that comes to mind is "expensive". Even if you are only converting existing media content, the software required for conversion might not be affordable for normal home users. One of the major reasons to use Linux for video streaming is it is inexpensive. Most Linux video encoding and streaming tools are free, and they are easy to set up. In less than an hour, one could have a streaming server serving a dozen of MPEG-4 clips. MPEG-4 allows encoding to happen at different bit rates and resolutions, while these are limited in some other codecs. The utmost advantage of using MPEG-4, of course, is it offers a standard format that is becoming more popular. Unlike proprietary formats, MPEG-4 is an open standard, so adding MPEG-4 support is easy. A few MPEG-4 audio and video players already are on the market, and more of them will be released in the future.

If you are streaming MPEG-4 clips, the setup is simple. You need only an MPEG-4 streaming server with a fast connection. A DSL line would be enough if you are serving only a few buddies. If you want to implement video conferencing or surveillance systems, though, you need a live MPEG-4 encoder and a compatible video capturing device under Linux. Of course, boards are on the market that encode video to MPEG-4 on-fly, but they are generally quite expensive for home users. Under Linux, the MPEG4IP suite, the FFmpeg multimedia system and Apple's Darwin Streaming Server offer an inexpensive way to stream and create MPEG-4 content.

The MPEG4IP project began in summer 2000 by David Mackie, who works on multimedia streaming for Cisco Systems. He is no longer active on the project and Bill May is now the main developer. The project is licensed under Mozilla Public License 1.1, but the codecs used in MPEG4IP are subject to patent royalties depending on how they are used. You should read the COPYING file in the package before using them. Licensing fees also are associated with MPEG-4, including a per-stream charge. People who plan to go commercial should check it out first.

In the MPEG4IP box are handy tools that can help you encode and play MPEG-4 clips. You may get your copy from mpeg4ip.sourceforge.net/ or from its CVS on SourceForge. While I was writing this article, 0.9.7 was the latest stable release. If you would like to try out new features, get release 0.9.7.7 from the CVS repository. You also need the LAME package if you are compiling mp4live (enabled by default), the live MPEG-4 encoder for live broadcasting. Extract the package and begin compiling:

tar zxvf mpeg4ip-0.9.7.tar.gz
./bootstrap
make
make install

To compile without mp4live (and LAME) add --disable-mp4live after ./bootstrap. MPEG4IP uses a modified version of the SDL library, so you do not need to bother installing it. You do need the GTK+ library, though, if you want to compile the GUI player.

The FFmpeg Multimedia System itself is a complete solution for creating and streaming MPEG-4. When this article was written, the stable release was 0.4.6. The streaming server supports only HTTP streaming at this stage, so we take advantage of FFmpeg, the encoder, at the moment. Get the source from SourceForge and begin compilation:

tar zxvf ffmpeg-0.4.6.tar.gz
./configure
make
make install

This configuration is sufficient for encoding MPEG-4 video. If you want MP3 and OggVorbis encoding support, add --enable-mp3lame and --enable-vorbis and install their respective libraries beforehand.

Here at my school video clips are stored in different formats. This is likely to be the case in most situations--you do not need to convert only one format to MPEG-4 but many formats. Some of my fellow video team members store the clips in MPEG-1 and 2, some of them in DivX or XviD AVI and some use DV AVI if they do not have time to encode. Here, we discuss how we encode these formats to MPEG-4, as they are likely to be the most common formats we might come across.

No single tool with a single button can do all the encoding work well for you. You need to encode the video and audio separately, so the first thing to do is split up the video and audio. Under Linux, FFmpeg is a Swiss knife for video processing. It encodes and decodes various video formats and supports some basic video processing, such as de-interlacing. To extract the audio track from the source, type:

ffmpeg -i <input_file> -vn <output_file>

The command can be applied to both DV/DivX/XviD AVI and MPEG files. The vn switch disables video in the output file. The resulting file would be a raw PCM audio file.

To generate an MPEG-4 complaint clip, you need MPEG-4 AAC audio. The MPEG4IP suite includes a redistributed AAC encoder called FAAC. At the of this writing, FAAD2 is available on SourceForge and uses the version 1.1 library for encoding, which is a bit faster. If you are interested, you can grab the source from faac.sourceforge.net. A copy of the FAAC binary should have been installed properly when you were installing the MPEG4IP suite.

To encode the raw PCM audio track split by FFmpeg, type:

faac -m4 -b64 -r48000 -pLC <input_file> <output_file>

The -m flag specifies the AAC MPEG version. We are creating MPEG-4 AAC so we use -m4. The -b flag specifies the bit rate for the encoded output. A value of 32 or 64 produces satisfactory quality and is an economical size for streaming. The -r flag specifies the sampling rate of the input file; otherwise, FAAC would not read the file. Instead, it would produce an error notifying the user that the file format is not supported. The last -p flag, which specifies the profile, is a bit more complicated. The Main profile includes all the coding tools (an MPEG term understood as "features"). The LC profile, which stands for low complexity, reduces decoding complexity on the client side (for slow devices or machines). The LTP profile, which stands for long term prediction, provides significant improvement in quality but also increases decoding complexity. It is best used for audio tracks with clear pitch. I use LC most of the time for my school's MPEG-4 clips.

MPEG4IP suite comes with two encoders, mp4encode and xvidenc. mp4encode encodes an AVI file to an MPEG-4 video clip with its built-in OpenDivX encoder or ISO MPEG-4 encoder, if specified. xvidenc, on the other hand, encodes raw video file with the XviD encoder. If you encode with these utilities, the input AVI file must contain a video track in raw YV12 (YUV12) format, and the audio track must be in raw PCM format. In this article, instead, I show how we can use FFmpeg to simplify the video encoding process. Because FFmpeg decodes DV, MPEG-1 and 2, DivX and XviD, we can skip the step of preparing YV12 raw for mp4encode or xvidenc. The command for encoding from these formats to ISO MPEG-4 video is the same:

ffmpeg -i <input_file> -an -b 300 -vcodec mpeg4 <output_file>

FFmpeg then encodes the input file without audio (the -an flag) at 300 kilobits/second, using the ISO MPEG-4 encoder to an AVI video file containing a single MPEG-4 video track. If you want to perform inverse telecine (IVTC) for NTSC source under Linux, you would need transcode, another nice program suite that provides useful video processing functionalities. It also is possible to encode the video in 2-pass mode to get the best result.

1st pass:

ffmpeg -i <input_file> -an -pass 1 -passlogfile <log_file>
-qscale 2 -vcodec mpeg4 <output_file>

2nd pass:

ffmpeg -i <input_file> -an -pass 2 -passlogfile <log_file>
-b 300 -vcodec mpeg4 <output_file>

The input file, the output file and the log file in both passes must be the same. Also, you should specify your desired bit rate during the second pass instead of the first.

After encoding the audio and video, it is time to combine them to create a real MPEG-4 clip. The MPEG4IP suite provides a nice tool called mp4creator for this purpose. It can add or remove tracks in an MPEG-4 file. But, let's first create the MPEG-4 clip by adding a video track:

mp4creator -c myvideo.avi -hint mytest.mp4

The -hint flag tells mp4creator to add an additional hint track for the video track. Hinting is required if you are streaming a clip with Darwin Streaming Server. If the mytest.mp4 file does not exist already, mp4creator creates it for you. To add an audio track, enter:

mp4creator -c myaudio.aac -hint -interleave mytest.mp4

Both commands perform a similar action. The second command introduced the -interleave flag, which creates an interleaved RTP payload format for the audio hint track. If you are using MP3 for your audio track (possibly not), ignore this flag as it supports only AAC. Finally, optimize the clip:

mp4creator -optimize mytest.mp4

This command optimizes the file layout, hence improving its speed during streaming. The rearranged MPEG-4 file contains control information at the beginning of the file, thereby enabling HTTP streaming of the file. To see a list of tracks in an MPEG-4 file, use:

mp4creator -list mytest.mp4

If you have a successful MPEG-4 clip, the list should look something like this (a clip currently being streamed at my school):

Before putting the clip up to the streaming server, you should test it. The MPEG4IP suite comes with mp4player and gmp4player for playing locally stored or streamed MPEG-4 files. You should use gmp4player, as the console player provides little control of video playback. If everything looks fine, upload the MPEG-4 file to your streaming server's media directory. Hurray! Your video is ready to be streamed over the net!

Starting from its fourth version, the Darwin Streaming Server does not stream only QuickTime movies but also MPEG-4 ones. The benefit is Darwin Streaming Server is completely free for non-commercial use, and it runs under Linux smoothly with a small load.

To get a copy of the streaming server, go to the Darwin project page. You are required to have a valid APSL (Apple Public Source License) registration in order to download the binary or the source. Follow the instructions on the web site for details on how to obtain an APSL registration. Get the server binary for Linux (the web site shows Red Hat Linux 7.x), and proxy if you need to serve clients behind firewall. Uncompress the package to a local directory and begin the installation:

tar zxvf DarwinStreamingSrvr4.1.2-Linux.tar.gz
cd DarwinStreamingSrvr4.1.2-Linux
./Install

Make sure you have Perl installed for the web-based administration interface. Install also Net::SSLeay and OpenSSL with a valid certificate if you want SSL support. The installation script automatically copies the files to suitable places and displays where they are copied to. It then asks you for an administrative user name and password, which is used later to log in to the web-based administration interface, where you can access most of the server's features.

Next, launch the administration interface using the Perl script streamingadminserver.pl. It launches Darwin Streaming Server if it is not already running. Point your favourite browser to the video server's port 1220, 1240 if you enabled SSL support. Log in with the account you set during the installation. As this is your first time logging in, you are presented with the Setup Assistant. It guides you through the basic settings with detailed descriptions for each item. You need a valid SSL certificate to enable SSL support, unless you modify the script. After completing the Setup Assistant, you will have a running Darwin Streaming Server ready to flood the line with MPEG-4 videos.

Now you have an MPEG-4 clip on hand and ready to stream. One option is to stream it over HTTP; simply put the clip in a publicly accessible directory on your web server and point your player to the clip's URL. If you want to stream over RTSP, direct your player to rtsp://<server_name>/<clip_name>. (For example, rtsp://video.wyk.edu.hk/sample_100kbit.mp4 is a sample clip provided by Darwin Streaming Server.) Streaming over RTSP allows random seeking without waiting for the clip to be downloaded progressively.

If you want to create a playlist or make a live broadcast, you would need to create an SDP (session description protocol) file. To create one, log into your video server's administration interface and then go to the Playlists section. Create a new movie playlist and drag the accompanying media to the playlist. Give the playlist a name and a mount point, which is an SDP file. Save the list, and do not forget to start the list afterwards. Instead of pointing your player to the MPEG-4 file, supply the new URL, rtsp://<server_name>/<sdp_name> to receive the broadcast. Congratulate yourself! You have successfully created a streaming MPEG-4 system and your first MPEG-4 video clip.

If you want to stream live video from your capturing device, use mp4live which comes with MPEG4IP. You need a Video4Linux-compatible capturing device, Video4Linux and I2C support in the kernel for live broadcasting. The user interface of mp4live is quite self-explanatory. You can record the video to an MPEG-4 file or stream it to clients with unicasting or multicasting. Click the Generate button to generate an SDP file, and copy it to your video server's media directory.

If you are within a local LAN or enterprise, multicasting would be the best method. In this method (on both QuickTime, Real and gmp4player), you pass the SDP file directly to the program. Set up a web server on the broadcasting machine, then point the player to http://<server machine>/capture.sdp, where capture.sdp is the aforementioned generated SDP file. With unicasting, you would use rtsp://<server machine>/capture.sdp instead. The server machine must be running Darwin Streaming Server, and the result is the streams are unicasted and repeated. This should be used only where multicast is not available.

People who try encoding the video may complain about the video quality. The MPEG-4 codecs used in Windows and Linux are the same, so what is the difference? The key component that determines the ultimate quality of the video is the rate controller. The fact is rate controllers in Linux still need some work. I recommend using the famous VirtualDub (free and licensed under the GPL) if you can, because it is more powerful in video encoding and processing and significantly enhances video quality.

Most people prefer using QuickTime, RealPlayer or Windows Media Player to play streaming video. Starting with version 6, QuickTime has built-in support for MPEG-4, so what you need to do is upgrade your QuickTime Player. For RealPlayer and Windows Media Player, you must install the EnvivioTV plug-in, which can be freely downloaded and used. You may get your copy at www.envivio.com.

Many discussions of QuickTime 6's MPEG-4 support can be found on the Net. In short, a bunch of users cannot play streamed MPEG-4 with QuickTime 6, and sometimes they receive strange error messages. The problem lies in the implementation of MPEG-4 in QuickTime 6. It is known that MPEG-4 video encoded with GMC (global motion compensation), quarterpixels (qpel) and chroma motion cannot be played with QuickTime 6. QuickTime 6 also uses an MPEG quantizer, which improves cripsness and is better than H.263 for preserving quality. Extensive tests in my school show that MPEG-4 AAC audio with bit rates under 64 kilobits/second generate strange error messages in QuickTime 6. The players provided by MPEG4IP, however, do not have the same problems with QuickTime 6.

QuickTime 6:

MPEG4IP mp4player:

Before encoding an MPEG-4 streaming video clip, you must consider a few things. Do you want to keep the maximum quality or compatibility? Who is the intended audience for the clip? If you are showing the clip to a large group of people or the general public, it is best to encode the clip so it may be played by common video players. If you want to preserve quality, you can use an MPEG quantizer during encoding or the MPEG-4 encoding tools discussed above. You should provide download of the MPEG4IP player so users are able to play them properly. The MPEG4IP suite includes Visual Studio workspace files, which let you compile Windows version of the player, wmp4player. Provide it as well, as most of your clients are probably running Windows.

MPEG-4 will be the trend in video streaming in the coming years. The MPEG-4 implementations we have today are still not the full MPEG-4 toolbox. For example, integration of natural and synthetic content and broadcast-grade synchronization need to be advanced. With the technology becoming more mature, MPEG-4 will soon become the standard for internet video.

I would like to thank Bill May, the main developer for the MPEG4IP project, for his comments and suggestions on this article.

MPEG4IP Project

FFmpeg Multimedia System

Apple Darwin Streaming Server

transcode