Work the Shell - Breaking Numbers Down

Last month, we continued our journey into the dark caverns of Apache Web logs, examining how relatively simple shell scripts can be utilized to produce useful and important data. The specific script we created searched a log file for traffic that occurred the previous day, summarizing the number of bytes transmitted.

That's all well and good, but as with many shell scripts, there's a bit of a problem with this one, which was immediately obvious when my busy site produced an estimated monthly data transfer rate of 2346990660 bytes.

Clearly that's a very human-unfriendly number, and doubly so without any commas to break it up into thousands, millions and so on. More important, when talking about data transfer, we're used to thinking in terms of powers of two, so 1 kilobyte is 1024 bytes of data, not 1000 bytes of data, and 1 megabyte is 1024 kilobytes of data, and so on.

Unfortunately, the expr command that we're using for the mathematical calculations doesn't have the ability to work with these powers of two, so we're going to have to do the work ourselves, converting massive numbers into more readable KB, MB or GB values, as appropriate.

The basics are pretty easy:

   kilo="$(( $value / 1024 ))"
   mega="$(( $kilo / 1024 ))"
   giga="$(( $mega / 1024 ))"

Given a nice huge number like 2346990660, the results are then quickly calculated:

   $ sh -x convert.sh 2346990660
   + value=2346990660
   + kilo=2291983
   + mega=2238
   + giga=2
   + exit 0

(Helpful tip: the -x option lets you debug shell scripts by showing, line by line, what command is being executed.)

The problem with this approach is immediately obvious when we switch from a huge number, more than 2GB, to a smaller value:

   $ sh -x convert.sh 5000
   + value=5000
   + kilo=4
   + mega=0
   + giga=0
   + exit 0

We don't want zero values; we want to see the fractional decimal values, which means not only that we can't use the built-in mathematical capabilities of the shell, but we also can't use expr. Instead, we need to move into the crufty, ancient world of bc, the binary calculator.

Now, bc isn't for the faint of heart, but to save you from reading the man page, here's how you can force four digits after the decimal point on the result of a division that results in a value less than 1.0:

   $ echo "scale=2 ; 3000 / 30001" | bc
   .0999

Can you see how to put these together? Here's a new, far-improved way to calculate kilo, mega and giga:

   $ sh -x convert.sh 5000
   + value=5000
   ++ echo 'scale=2; 5000 / 1024'
   ++ bc
   + kilo=4.88
   ++ echo 'scale=2; 4.88 / 1024'
   ++ bc
   + mega=0
   ++ echo 'scale=2; .00 / 1024'
   ++ bc
   + giga=0
   + exit 0

The debug output from the -x option is getting a bit confusing here, I admit, but you now can see that kilo is set to 4.88 when given the initial value of 5000 bytes, and that both mega and giga are zero.

Let's try again (and I'll clean up some of the spurious debug output from this point on, for clarity) with the initial really big value:

   $ convert.sh 2346990660
   value=2346990660
   kilo=2291983.06
   mega=2238.26
   giga=2.18

Cool. Now we can finally see that we're talking about 2.18GB of data being transferred off the site each month—far more coherent than the huge value shown earlier.

Now, let's figure out how to show always the most logical of these values, rather than all of them.

The easiest way to figure out which value is best is simply to ascertain where the value drops below 1.0. In the case of 5000 bytes, that'd be best displayed as 4.88KB, and in the case of the bignum value, that's 2.18GB.

To figure out when the value drops below zero, we'd love to have a floating-point numeric comparison, but sadly, the shell can't manage it. If you try it, you'll just get the error “integer expression expected”.

There are a number of ways to get the “floor” of the value, but I use bc again here to do the job by calculating the division once more, this time without any scale value at all:

   kiloint=$( echo "$value/1024" | bc)"

Doing this gets just the integer portion of the $kilo value, and that can indeed be tested in a conditional statement:

   if [ $kiloint -lt 1 ] ; then

Now, put it all together, and here's how the script looks:

   kilo=$( echo "scale=2; $value / 1024" | bc )
   kiloint=$( echo "$value / 1024" | bc )

   mega=$( echo "scale=2; $kilo / 1024" | bc )
   megaint=$( echo "$kilo / 1024" | bc )

   giga=$( echo "scale=2; $mega / 1024" | bc )
   gigaint=$( echo "$mega / 1024" | bc )

   if [ $kiloint -lt 1 ] ; then
   echo "$value bytes"
   elif [ $megaint -lt 1 ] ; then
   echo "${kilo}KB"
   elif [ $gigaint -lt 1 ] ; then
   echo "${mega}MB"
   else
   echo "${giga}GB"
   fi

A little funky, but it certainly works exactly as we'd hope:

   $ sh convert.sh 5000000000
   4.65GB
   $ sh convert.sh 5000000
   4.76MB
   $ sh convert.sh 50000
   48.82KB
   $ sh convert.sh 50
   50 bytes

The final step is to make it a function so we can include it in other shell scripts and access it as desired. This is done within the Bourne Shell by giving it a unique name and then wrapping the functional code in braces:

   kmg()
   {
      code for function goes here, params are $1, $2, etc.
   }

This can then be invoked within a shell script by name (k=kilo, m=mega, g=giga):

   kmg 500000

More important, you can embed it within a line by using a subshell notation, so given the kmg() function, the following two-line script works splendidly:

   echo given value is $1
   echo which converts to $(kmg $1)

That's nice and short, and if the kmg function is dropped into its own file, you also can use the . command to include another file in the shell script, meaning that the entire test script is now:

   #!/bin/sh

   . kmg.sh

   echo The given value $1 bytes = $(kmg $1)

   exit 0

I'm out of space here, but I hope you can see how this approach can be applied to a wide variety of different shell tasks, making your shell scripts far more efficient and faster to write too!