Compression Tools Compared
Data compression works so well that popular backup and networking tools have some built in. Linux offers more than a dozen compression tools to choose from, and most of them let you pick a compression level too. To find out which perform best, I benchmarked 87 combinations of tools and levels. Read this article to learn which compressor is a hundred times faster than the others and which ones compress the most.
The most popular data compression tool for Linux is gzip, which lets you choose a compression level from one to nine. One is fast, and nine compresses well. Choosing a good trade-off between speed and compression ratio becomes important when it takes hours to handle gigabytes of data. You can get a sense of what your choices are from the graph shown in Figure 1. The fastest choices are on the left, and the highest compressing ones are on the top. The best all-around performers are presented in the graph's upper left-hand corner.

Figure 1. Increasing the compression level in gzip increases both compression ratio and time required to complete.
But many other data compression tools are available to choose from in Linux. See the comprehensive compression and decompression benchmarks in Figures 2 and 3. As with gzip, the best performers are in the upper left-hand corner, but these charts' time axes are scaled logarithmically to accommodate huge differences in how fast they work.
The Benchmarks
How compactly data can be compressed depends on what type of data it is. Don't expect big performance increases from data that's already compressed, such as files in Ogg Vorbis, MP3 or JPEG format. On the other hand, I've seen data that allows performance increases of 1,000%!
All benchmarks in this article used the same 45MB of typical Linux data, containing:
24% ELF 32-bit LSB
15% ASCII C program
11% gzip compressed data
8% ASCII English text
7% binary package
4% directory
2% current ar archive
2% Texinfo source text
2% PostScript document text
2% Bourne shell script
2% ASCII text
21% various other data types
This data set was chosen because it is more representative of the demands made on today's Linux systems than the data used in the traditional Canterbury and Calgary test data, because this data set is bigger and contains Linux binaries.
I used the same lightly loaded AMD Athlon XP 1700+ CPU with 1GB of RAM and version 2.4.27-1-k7 of the Linux kernel for all tests. Unpredictable disk drive delays were minimized by pre-loading data into RAM. Elapsed times were measured in thousandths of a second. I'm not affiliated with any of the tools, and I strove to be objective and accurate.
The tools that tend to compress more and faster are singled out in the graphs shown in Figures 4 and 5. Use these for backups to disk drives. Remember, their time axes are scaled logarithmically. The red lines show the top-performing ones, and the green lines show the top performers that also can act as filters.
Trending Topics
| The Linux powered LAN Gaming House | Feb 08, 2012 |
| Creating a vDSO: the Colonel's Other Chicken | Feb 06, 2012 |
| Your CMS Is Not Your Web Site | Feb 01, 2012 |
| Casper, the Friendly (and Persistent) Ghost | Jan 31, 2012 |
| Razor-qt 0.4 - Qt based Desktop Environment | Jan 30, 2012 |
| Using Plop Boot Manager for USB Boot | Jan 25, 2012 |
- The Linux powered LAN Gaming House
- Readers' Choice Awards 2011
- Validate an E-Mail Address with PHP, the Right Way
- Creating a vDSO: the Colonel's Other Chicken
- Boot with GRUB
- Building a Two-Node Linux Cluster with Heartbeat
- Python for Android
- Why Python?
- Your CMS Is Not Your Web Site
- Writing a Simple USB Driver
- Lubuntu on a USB key
6 hours 58 min ago - Because XFCE is neither fish
22 hours 13 min ago - Because XFCE is neither fish
22 hours 14 min ago - KDE Bloat
1 day 4 hours ago - My C-64 Memories
1 day 5 hours ago - Spam
1 day 6 hours ago - Ooops....
1 day 11 hours ago - ----- http://ai.vc/zd
1 day 21 hours ago - ----- http://ai.vc/zd
1 day 21 hours ago - ----- http://ai.vc/zd
1 day 21 hours ago










Comments
How to improve backups?
Hi. Congratulations for this very useful article... I use a script for backup made by myself which use tar +gzip... switching to tar - lsop backup time takes less than half time, increasing the backup size by about 25%.
An idea to improve speed is to replace tar with a more intelligent tool.
Infact, tar simply "cat all files to stdout" and then gzip or lsop compress this huge stream of data, but some data is already compressed (images, movies, open document files) and don't need to be recompressed!
The idea is to have an archiver (like tar) which compress each file by itself, storing the original file in case of images, movies, archives, already compressed files.
Is there any tool that can do this, and save all priviledges (owner, group, mode) associated to each file like tar does?
Thank you. Paolo
(1) Found a typo: "On the
(1) Found a typo:
"On the other hand, if you have a 1GHz network, but only a 100MHz CPU"
1 GHz network? Should maybe be 1 Gbps.
(2) Suggestion:
Multi-Core CPUs are the big thing today, compression tools that could utilise multiple cores can run 2, 4 or soon even 8 times faster on "normal" desktop PCs...not even speaking of the servers...which compression tools can utilise this CPU power?
multi-core CPU support
Multi-Core CPUs are the big thing today, compression tools that could utilise multiple cores can run 2, 4 or soon even 8 times faster on "normal" desktop PCs...not even speaking of the servers...which compression tools can utilise this CPU power?
http://compression.ca/pbzip2/
There's parallel bzip2, very good but not pipe support.
HTH,
mfg zmi
Very nice information
Very nice information provided.Thanks!!!
Excelent article.
Thanks very much for this article. I really enjoyed it, and will be helpfull for my daily work.
1. how about another part
how about another part with specific data - like 90+% text? for mysql dumps & dbmail scenarios etc.
and 45MB does not sound as sufficient test data size for rzip to test it's speed.
Compression on Windows
First of all excellent test!.
Believe it or not, but compression is one of those application types where all research takes place on Windows Pc's. The last couple of years there were some major breakthroughs in compression caused by the new PAQ context modeling algorithms. Have a look at this site for some results. Programs like gzip, rzip 7-zip and lzop are tested here too, so it should be easy to compare results.
http://www.maximumcompression.com/