Username/Email:  Password: 
TwitterFacebookFlickrRSS
Home ›

Treating Compressed and Uncompressed Data Sources the Same

Dec 19, 2008  By David Sinck
 in

Occasionally, you need to process a number of files—some of which have been compressed and some which have not (think log files). Rather than running two variations, one compressed and one not, wrap it in a bash function:

function data_source ()
{
 local F=$1

 # strip the gz if it's there
 F=$(echo $F | perl -pe 's/.gz$//')

 if [[ -f $F ]] ; then
   cat $F
 elif [[ -f $F.gz ]] ; then
   nice gunzip -c $F
 fi
}

which nicely allows:

for file in * ; do
 data_source $file | ...
done

Whether you're dealing with gzip'd files or uncompressed, you no longer have to treat them differently mentally. With a little more effort, bzip files also could be detected and handled.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Remember the -exec action of find

Martijn Ras's picture
Submitted by Martijn Ras on Mon, 05/04/2009 - 15:21.

I don't like using 'for file in *; do ...; done' as it fails on filenames that contain spaces, using the -exec action of find makes avoiding this problem very easy. Also it might be a good idea to at least print the name of each file before outputting the content of each file in the loop.

I'd use the following:

find . -maxdepth 1 -type f -exec sh -c "echo '{}:' ; bzcat '{}' 2>/dev/null || zcat '{}' 2>/dev/null || cat '{}' 2>/dev/null" \;

Or, if you are inclined to use a shell script create "data_source.sh" and 'chmod +x' it:

#!/bin/sh
echo "$1:"
bzcat "$1" 2>/dev/null || zcat "$1" 2>/dev/null || cat "$1" 2>/dev/null

and use the following:

find . -maxdepth 1 -type f -exec ./data_source.sh '{}' \;

I'd prefer using a case

john.mckown's picture
Submitted by john.mckown on Tue, 12/30/2008 - 12:15.

I'd prefer using a case stucture similar to:

function data_source ()
{
local F=$1

case $F in
*.gz) zcat $F;;
*.bz2) bzcat $F;;
*) cat $F;;
}

This only invokes one external command to do the "cat" function. And it is very easy to extend to other possible suffixes that may be special.

Decompression in a pipe

roaima's picture
Submitted by roaima on Fri, 12/19/2008 - 19:16.

If you're processing files in a pipe, try this:

bzcat -f "${FILE}" | zcat -f | ...

It doesn't matter whether ${FILE} is compressed with bzip2, gzip, or even not compressed at all. It just works. (Assuming you have bzcat and zcat installed, of course.)

Chris

I don't see an -f switch for

john.mckown's picture
Submitted by john.mckown on Tue, 12/30/2008 - 12:22.

I don't see an -f switch for bzcat on my system. And I don't understand the piping. Do you mean something like:

bzcat $F || zcat $F || cat $F

?

I don't like the above as it is inefficient. If bzcat fails, then zcat is tried and if it fails, then normal cat is done. Of course, the good part is that this is not dependent on a file suffix.

Another way to decompression in a pipe

Anonymous's picture
Submitted by Anonymous (not verified) on Thu, 12/25/2008 - 17:51.

Another way if bzcat is not installed...

gzip -dc "${FILE}" | ...

Perl?

Ian's picture
Submitted by Ian (not verified) on Fri, 12/19/2008 - 10:27.

Reaching for the Advanced Bash Scripting Guide...

Check out Table B-5 String Operations and

${string%%substring} Strip longest match of $substring from back of $string

Re: Perl?

Vance's picture
Submitted by Vance (not verified) on Sat, 12/20/2008 - 19:54.

${string%%substring} Strip longest match of $substring from back of $string

Great tip. Or if you want to kick it old-school UNIX style:

F=`dirname $F`/`basename $F .gz`