Handle Compressed and Uncompressed Files Uniformly

When looking at log files or other files that are compressed and rotated automatically, it's useful to be able to deal with them in a uniform fashion. The following bash function does that:

function data_source ()
 local F=$1

 # strip the gz if it's there
 F=$(echo $F | perl -pe 's/.gz$//')

 if [[ -f $F ]] ; then
  cat $F
 elif [[ -f $F.gz ]] ; then
  nice gunzip -c $F

Now, when you want to process the files, you can use:

for file in * ; do
 data_source $file | ...

If you have bzip2 files, just modify the data_source function to check for that also.

Load Disqus comments

Community Events

Park City, UT, USA
New Orleans, LA, USA
Copenhagen, Denmark
Austin, TX, USA
Austin, TX, USA