Handle Compressed and Uncompressed Files Uniformly
When looking at log files or other files that are compressed and rotated automatically, it's useful to be able to deal with them in a uniform fashion. The following bash function does that:
function data_source ()
{
local F=$1
# strip the gz if it's there
F=$(echo $F | perl -pe 's/.gz$//')
if [[ -f $F ]] ; then
cat $F
elif [[ -f $F.gz ]] ; then
nice gunzip -c $F
fi
}
Now, when you want to process the files, you can use:
for file in * ; do data_source $file | ... done
If you have bzip2 files, just modify the data_source function to check for that also.
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Sponsored by AMD
Built-in forensics, incident response, and security with Red Hat Enterprise Linux 6
Every security policy provides guidance and requirements for ensuring adequate protection of information and data, as well as high-level technical and administrative security requirements for a system in a given environment. Traditionally, providing security for a system focuses on the confidentiality of the information on it. However, protecting the data integrity and system and data availability is just as important. For example, when processing United States intelligence information, there are three attributes that require protection: confidentiality, integrity, and availability.
Learn more about catching the bad guy in this free white paper.
Sponsored by DLT Solutions
| Designing Electronics with Linux | May 22, 2013 |
| Dynamic DNS—an Object Lesson in Problem Solving | May 21, 2013 |
| Using Salt Stack and Vagrant for Drupal Development | May 20, 2013 |
| Making Linux and Android Get Along (It's Not as Hard as It Sounds) | May 16, 2013 |
| Drupal Is a Framework: Why Everyone Needs to Understand This | May 15, 2013 |
| Home, My Backup Data Center | May 13, 2013 |
- Designing Electronics with Linux
- New Products
- Making Linux and Android Get Along (It's Not as Hard as It Sounds)
- Dynamic DNS—an Object Lesson in Problem Solving
- Using Salt Stack and Vagrant for Drupal Development
- Validate an E-Mail Address with PHP, the Right Way
- Tech Tip: Really Simple HTTP Server with Python
- Why Python?
- Build a Skype Server for Your Home Phone System
- Drupal Is a Framework: Why Everyone Needs to Understand This
- Reply to comment | Linux Journal
50 min 18 sec ago - Reply to comment | Linux Journal
1 hour 40 min ago - Not free anymore
5 hours 42 min ago - Great
9 hours 29 min ago - Reply to comment | Linux Journal
9 hours 37 min ago - Understanding the Linux Kernel
11 hours 52 min ago - General
14 hours 22 min ago - Kernel Problem
1 day 24 min ago - BASH script to log IPs on public web server
1 day 4 hours ago - DynDNS
1 day 8 hours ago
Enter to Win an Adafruit Pi Cobbler Breakout Kit for Raspberry Pi

It's Raspberry Pi month at Linux Journal. Each week in May, Adafruit will be giving away a Pi-related prize to a lucky, randomly drawn LJ reader. Winners will be announced weekly.
Fill out the fields below to enter to win this week's prize-- a Pi Cobbler Breakout Kit for Raspberry Pi.
Congratulations to our winners so far:
- 5-8-13, Pi Starter Pack: Jack Davis
- 5-15-13, Pi Model B 512MB RAM: Patrick Dunn
- 5-21-13, Prototyping Pi Plate Kit: Philip Kirby
- Next winner announced on 5-27-13!
Featured Jobs
| Linux Systems Administrator | Houston and Austin, Texas | Host Gator |
| Senior Perl Developer | Austin, Texas | Host Gator |
| Technical Support Rep | Houston and Austin, Texas | Host Gator |
| UX Designer | Austin, Texas | Host Gator |
| Web & UI Developer (JavaScript & j Query) | Austin, Texas | Host Gator |
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?



Comments
GPL tool to handle this
The big reason why zcat shouldn't be used for this is that zcat will fail if you give it uncompressed data. The whole point is to have a cat that will handle _both_ compressed and uncompressed data. Also, I don't think that checking the filename extension is a valid way to determine file contents.
I wrote ccat to handle not only plaintext and gzipped data, but also bzipped data, and is extensible to handle other compression methods, like compress(1). ccat is at www.administra.tion.ca
Stripping suffix
Why not just execute
X=${Y%.gz}
?
a blast from the past?
Wasn't this same thing done a few months ago? It sure seems familiar.
Function broken
This function will do the wrong thing if you have "file.gz" and "file" in the same directory and try to do something with "file.gz".
$ echo "uncompressed file" >file
$ echo "COMPRESSED file" | gzip >file.gz
$ data_source file.gz
uncompressed file
$
Also, if you make a mistake with the filename, the function just does nothing:
$ data_source fiel.gz
$
so you might not notice your typo.
overkill, redondant
1/ as indicated, zcat does the trick very well already.
2/ using perl to do a simple substitution is a huge overkill
3/ detection the type of a file is probably better served with the 'type' command:
$> type -ib foo.tar.bz2
application/x-bzip2
$> type -ib foo.gz
application/x-gzip
$ mv foo.gz foo.bz2; type -ib foo.bz2
application/x-gzip
Why function?
Hello,
try "man zgrep" instead of using this frantic_function and piping it.
I miss the point or it lacks the point
I don't see the point, if this is meant as a scripting exercise: shooting with a Perl cannon to kill a mosquito when Bash itself and basename, that would make the trick altogether.
I guess zcat, zless, zgrep and z"whatever" have been created for that purpose.
Diseducative, after all.
mixing bash scripts w/perl
IMHO, it seems like bit of overkill to use perl (and subprocess/pipe/etc) for a search/replace inside a bash script. I tend to think that if I'm going to use perl in the script, it's probably best just to write the entire script in perl.
While I admittedly haven't gone to the full length of replacing (or even running) the exact piece of code presented, I did throw together a proof-of-concept piece of code that works on fc6 (bash --version: 3.1.17(1)-release):
#!/bin/bash file="test.gz" # file with a .gz extension file2="test" # file without a .gz extension for comparison nogzfile=${file/%.gz/} # replace the post string (%) .gz with nothing nogzfile2=${file2/%.gz/} # same replacement on the other file # look at the output echo "file: $file" # test.gz echo "file2: $file2" # test echo "nogzfile: $nogzfile" # test echo "nogzfile: $nogzfile2" # testYou may want to incorporate this search/replace inside your script something like (not tested):
local F=${1/%.gz/}
Incase you're interested, the pattern matching part I pulled from my favorite bash scripting resource, The Advanced Bash Scripting Guide:
Reference page used in this example
all-in-all, nice function. I'm sure it will provide many people with a useful snippet. keep up the good work.