Work the Shell - Breaking Numbers Down

A kilo of information on how to represent even giga numbers in a mega-useful way.

Last month, we continued our journey into the dark caverns of Apache Web logs, examining how relatively simple shell scripts can be utilized to produce useful and important data. The specific script we created searched a log file for traffic that occurred the previous day, summarizing the number of bytes transmitted.

That's all well and good, but as with many shell scripts, there's a bit of a problem with this one, which was immediately obvious when my busy site produced an estimated monthly data transfer rate of 2346990660 bytes.

Clearly that's a very human-unfriendly number, and doubly so without any commas to break it up into thousands, millions and so on. More important, when talking about data transfer, we're used to thinking in terms of powers of two, so 1 kilobyte is 1024 bytes of data, not 1000 bytes of data, and 1 megabyte is 1024 kilobytes of data, and so on.

Unfortunately, the expr command that we're using for the mathematical calculations doesn't have the ability to work with these powers of two, so we're going to have to do the work ourselves, converting massive numbers into more readable KB, MB or GB values, as appropriate.

Converting Numeric Values

The basics are pretty easy:

   kilo="$(( $value / 1024 ))"
   mega="$(( $kilo / 1024 ))"
   giga="$(( $mega / 1024 ))"

Given a nice huge number like 2346990660, the results are then quickly calculated:

   $ sh -x convert.sh 2346990660
   + value=2346990660
   + kilo=2291983
   + mega=2238
   + giga=2
   + exit 0

(Helpful tip: the -x option lets you debug shell scripts by showing, line by line, what command is being executed.)

The problem with this approach is immediately obvious when we switch from a huge number, more than 2GB, to a smaller value:

   $ sh -x convert.sh 5000
   + value=5000
   + kilo=4
   + mega=0
   + giga=0
   + exit 0

We don't want zero values; we want to see the fractional decimal values, which means not only that we can't use the built-in mathematical capabilities of the shell, but we also can't use expr. Instead, we need to move into the crufty, ancient world of bc, the binary calculator.

Now, bc isn't for the faint of heart, but to save you from reading the man page, here's how you can force four digits after the decimal point on the result of a division that results in a value less than 1.0:

   $ echo "scale=2 ; 3000 / 30001" | bc
   .0999

Can you see how to put these together? Here's a new, far-improved way to calculate kilo, mega and giga:

   $ sh -x convert.sh 5000
   + value=5000
   ++ echo 'scale=2; 5000 / 1024'
   ++ bc
   + kilo=4.88
   ++ echo 'scale=2; 4.88 / 1024'
   ++ bc
   + mega=0
   ++ echo 'scale=2; .00 / 1024'
   ++ bc
   + giga=0
   + exit 0

The debug output from the -x option is getting a bit confusing here, I admit, but you now can see that kilo is set to 4.88 when given the initial value of 5000 bytes, and that both mega and giga are zero.

Let's try again (and I'll clean up some of the spurious debug output from this point on, for clarity) with the initial really big value:

   $ convert.sh 2346990660
   value=2346990660
   kilo=2291983.06
   mega=2238.26
   giga=2.18

Cool. Now we can finally see that we're talking about 2.18GB of data being transferred off the site each month—far more coherent than the huge value shown earlier.

Now, let's figure out how to show always the most logical of these values, rather than all of them.

Displaying the Simplest Answer Only

The easiest way to figure out which value is best is simply to ascertain where the value drops below 1.0. In the case of 5000 bytes, that'd be best displayed as 4.88KB, and in the case of the bignum value, that's 2.18GB.

To figure out when the value drops below zero, we'd love to have a floating-point numeric comparison, but sadly, the shell can't manage it. If you try it, you'll just get the error “integer expression expected”.

There are a number of ways to get the “floor” of the value, but I use bc again here to do the job by calculating the division once more, this time without any scale value at all:

   kiloint=$( echo "$value/1024" | bc)"

Doing this gets just the integer portion of the $kilo value, and that can indeed be tested in a conditional statement:

   if [ $kiloint -lt 1 ] ; then

Now, put it all together, and here's how the script looks:

   kilo=$( echo "scale=2; $value / 1024" | bc )
   kiloint=$( echo "$value / 1024" | bc )

   mega=$( echo "scale=2; $kilo / 1024" | bc )
   megaint=$( echo "$kilo / 1024" | bc )

   giga=$( echo "scale=2; $mega / 1024" | bc )
   gigaint=$( echo "$mega / 1024" | bc )

   if [ $kiloint -lt 1 ] ; then
   echo "$value bytes"
   elif [ $megaint -lt 1 ] ; then
   echo "${kilo}KB"
   elif [ $gigaint -lt 1 ] ; then
   echo "${mega}MB"
   else
   echo "${giga}GB"
   fi

A little funky, but it certainly works exactly as we'd hope:

   $ sh convert.sh 5000000000
   4.65GB
   $ sh convert.sh 5000000
   4.76MB
   $ sh convert.sh 50000
   48.82KB
   $ sh convert.sh 50
   50 bytes

The final step is to make it a function so we can include it in other shell scripts and access it as desired. This is done within the Bourne Shell by giving it a unique name and then wrapping the functional code in braces:

   kmg()
   {
      code for function goes here, params are $1, $2, etc.
   }

This can then be invoked within a shell script by name (k=kilo, m=mega, g=giga):

   kmg 500000

More important, you can embed it within a line by using a subshell notation, so given the kmg() function, the following two-line script works splendidly:

   echo given value is $1
   echo which converts to $(kmg $1)

That's nice and short, and if the kmg function is dropped into its own file, you also can use the . command to include another file in the shell script, meaning that the entire test script is now:

   #!/bin/sh

   . kmg.sh

   echo The given value $1 bytes = $(kmg $1)

   exit 0

I'm out of space here, but I hope you can see how this approach can be applied to a wide variety of different shell tasks, making your shell scripts far more efficient and faster to write too!

Dave Taylor is a 26-year veteran of UNIX, creator of The Elm Mail System, and most recently author of both the best-selling Wicked Cool Shell Scripts and Teach Yourself Unix in 24 Hours, among his 16 technical books. His main Web site is at www.intuitive.com.

______________________

Dave Taylor has been hacking shell scripts for over thirty years. Really. He's the author of the popular "Wicked Cool Shell Scripts" and can be found on Twitter as @DaveTaylor and more generally at www.DaveTaylorOnline.com.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Function for bigger numbers

Simon Sweetman's picture

Here is a little Korn Shell implementation based on your article that supports up to Yotta Bytes Yikes!

I thought I'd share it with your readers.

#!/bin/ksh
bytestohr()
{
    # Convert input parameter (number of bytes) 
    # to Human Readable form
    #
    SLIST="bytes,KB,MB,GB,TB,PB,EB,ZB,YB"

    POWER=1
    VAL=$( echo "scale=2; $1 / 1" | bc)
    VINT=$( echo $VAL / 1024 | bc )
    while [ $VINT -gt 0 ]
    do
        let POWER=POWER+1
        VAL=$( echo "scale=2; $VAL / 1024" | bc)
        VINT=$( echo $VAL / 1024 | bc )
    done

    echo $VAL$( echo $SLIST | cut -f$POWER -d, )
}

NOTE: It's tempting to use the shell internal maths instead of bc for more performance, but values over 8Gb aren't supported by the shell's 32bit integers.

Oops mistake above

Simon Sweetman's picture

As soon as you make a post you find a mistake, the above while statement should be changed to

while [ ! $VINT = "0" ]

To avoid numeric overflows in some Korn shell implementations.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix