Archiving and Compression
Although the differences are sometimes made opaque in casual conversation, there is in fact a complete difference between archiving files and compressing them. Archiving means that you take 10 files and combine them into one file, with no difference in size. If you start with 10 100KB files and archive them, the resulting single file is 1000KB. On the other hand, if you compress those 10 files, you might find that the resulting files range from only a few kilobytes to close to the original size of 100KB, depending upon the original file type.
Note - In fact, you might end up with a bigger file during compression! If the file is already compressed, compressing it again adds extra overhead, resulting in a slightly bigger file.
All of the archive and compression formats in this chapter — zip, gzip, bzip2, and tar — are popular, but zip is probably the world's most widely used format. That's because of its almost universal use on Windows, but zip and unzip are well supported among all major (and most minor) operating systems, so things compressed using zip also work on Linux and Mac OS. If you're sending archives out to users and you don't know which operating systems they're using, zip is a safe choice to make.
gzip was designed as an open-source replacement for an older Unix program, compress. It's found on virtually every Unix-based system in the world, including Linux and Mac OS X, but it is much less common on Windows. If you're sending files back and forth to users of Unix-based machines, gzip is a safe choice.
The bzip2 command is the new kid on the block. Designed to supersede gzip, bzip2 creates smaller files, but at the cost of speed. That said, computers are so fast nowadays that most users won't notice much of a difference between the times it takes gzip or bzip2 to compress a group of files.
Note - Linux Magazine published a good article comparing several different compression formats, which you can find at www.linux-mag.com/content/view/1678/43/.
zip, gzip, and bzip2 are focused on compression (although zip also archives). The tar command does one thing — archive — and it has been doing it for a long time. It's found almost solely on Unix-based machines. You'll definitely run into tar files (also called tarballs) if you download source code, but almost every Linux user can expect to encounter a tarball some time in his career.
zip
zip both archives and compresses files, thus making it great for sending multiple files as email attachments, backing up items, or for saving disk space. Using it is simple. Let's say you want to send a TIFF to someone via email. A TIFF image is uncompressed, so it tends to be pretty large. Zipping it up should help make the email attachment a bit smaller.
Note - When using ls -l, I'm only showing the information needed for each example.
$ ls -lh -rw-r--r-- scott scott 1006K young_edgar_scott.tif $ zip grandpa.zip young_edgar_scott.tif adding: young_edgar_scott.tif (deflated 19%) $ ls -lh -rw-r--r-- scott scott 1006K young_edgar_scott.tif -rw-r--r-- scott scott 819K grandpa.zip _grandpa.zip
In this case, you shaved off about 200KB on the resulting zip file, or 19%, as zip helpfully informs you. Not bad. You can do the same thing for several images.
$ ls -l -rw-r--r-- scott scott 251980 edgar_intl_shoe.tif -rw-r--r-- scott scott 1130922 edgar_baby.tif -rw-r--r-- scott scott 1029224 young_edgar_scott.tif $ zip grandpa.zip edgar_intl_shoe.tif edgar_baby.tif young_edgar_scott.tif adding: edgar_intl_shoe.tif (deflated 4%) adding: edgar_baby.tif (deflated 12%) adding: young_edgar_scott.tif (deflated 19%) $ ls -l -rw-r--r-- scott scott 251980 edgar_intl_shoe.tif -rw-r--r-- scott scott 1130922 edgar_baby.tif -rw-r--r-- scott scott 2074296 grandpa.zip -rw-r--r-- scott scott 1029224 young_edgar_scott.tif
It's not too polite, however, to zip up individual files this way. For three files, it's not so bad. The recipient will unzip grandpa.zip and end up with three individual files. If the payload was 50 files, however, the user would end up with files strewn everywhere. Better to zip up a directory containing those 50 files so when the user unzips it, he's left with a tidy directory instead.
$ ls -lF drwxr-xr-x scott scott edgar_scott/ $ zip grandpa.zip edgar_scott adding: edgar_scott/ (stored 0%) adding: edgar_scott/edgar_baby.tif (deflated 12%) adding: edgar_scott/young_edgar_scott.tif (deflated 19%) adding: edgar_scott/edgar_intl_shoe.tif (deflated 4%) $ ls -lF drwxr-xr-x scott scott 160 edgar_scott/ -rw-r--r-- scott scott 2074502 grandpa.zip
Whether you're zipping up a file, several files, or a directory, the pattern is the same: the zip command, followed by the name of the Zip file you're creating, and finished with the item(s) you're adding to the Zip file.
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Sponsored by AMD
If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.
Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.
Sponsored by ActiveState
| Speed Up Your Web Site with Varnish | Jun 19, 2013 |
| Non-Linux FOSS: libnotify, OS X Style | Jun 18, 2013 |
| Containers—Not Virtual Machines—Are the Future Cloud | Jun 17, 2013 |
| Lock-Free Multi-Producer Multi-Consumer Queue on Ring Buffer | Jun 12, 2013 |
| Weechat, Irssi's Little Brother | Jun 11, 2013 |
| One Tail Just Isn't Enough | Jun 07, 2013 |
- Speed Up Your Web Site with Varnish
- Containers—Not Virtual Machines—Are the Future Cloud
- Linux Systems Administrator
- Lock-Free Multi-Producer Multi-Consumer Queue on Ring Buffer
- Non-Linux FOSS: libnotify, OS X Style
- Senior Perl Developer
- Technical Support Rep
- UX Designer
- Android's Limits
- Web & UI Developer (JavaScript & j Query)
- Reply to comment | Linux Journal
1 hour 24 min ago - Yeah, user namespaces are
2 hours 41 min ago - Cari Uang
6 hours 12 min ago - user namespaces
9 hours 6 min ago - yea
9 hours 31 min ago - One advantage with VMs
12 hours 26 sec ago - about info
12 hours 33 min ago - info
12 hours 34 min ago - info
12 hours 35 min ago - info
12 hours 37 min ago
Featured Jobs
| Linux Systems Administrator | Houston and Austin, Texas | Host Gator |
| Senior Perl Developer | Austin, Texas | Host Gator |
| Technical Support Rep | Houston and Austin, Texas | Host Gator |
| UX Designer | Austin, Texas | Host Gator |
| Web & UI Developer (JavaScript & j Query) | Austin, Texas | Host Gator |
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?



Comments
Unzipping Password Protected Zips
You left out how to unzip ZIP files that are password protected in Linux. I'm searching for this elusive bit of information on the internet right now...
Password protectedly adding files by PHP code was not found
Password protectedly adding files by PHP code was not found on the internet when i was searching for it... so i come across your article and it gave me the idea to why not issue a system command by php to add files in zip and even protect the files by password ;)
RAR
RAR is good and free too. It supports passwords and can make SFX archives.
No mention of lzma?
How about rzip or lzma? I recall an article in the print edition within the last ten or eleven issues that compared the cpu overhead of each compression method against compression ratios (and possibly other parameters). Anyways, rzip is memory and cpu intensive, IIRC, but has the potential to make enormous savings. I think it's the same as burrows-wheeler over larger data sets, possibly. Worthwhile for stuff that won't be frequently decompressed, IMO.
rzip
actually rzip levels are in search buffer sizes:
-0 = 100MB
-1 = 100MB
-x = x00MB for x>0 and x<=9
cpu intensive? well depends. I hacked bzip2 compression hooks out of the rzip and it's one of the fastest pre archiving filters with best compression ratio for mysql dump of dbmail database.
yup found bug but only in decompression algorithm - not the data itself. yes - made Andrew to fix it.
Correction to wording
Scott,
In the section "Archive Files with tar", paragraph 3, you state that tar is "designed to compress entire directory structures". I think this should read "designed to archive...", since this section deals only with tar's standalone use as an archival tool and since this article/chapter is intended to highlight the difference between archiving and compressing. Other than that, this is a very handy primer on archiving and compressing in *nix.
bzip2 -9
The article states that the default block size for bzip2 is -6. The man page for my system (Ubuntu 6.06) states that -9 is the default, and I am unaware of any system where -6 is the default.
TROGDOR STRIKES AGAIN!
TROGDOR STRIKES AGAIN!
http://news.bbc.co.uk/1/hi/england/cornwall/6088008.stm
Making -9 the default
An easier way to default to the best (-9) compression level would be to export GZIP='-9' and ZIPOPTS='-9' into your environment.