"Dogs" of the Linux Shell

One incarnation of the so called 80/20
rule has been associated with software systems. It has been
observed that 80% of a user population regularly uses only 20% of a
system's features. Without backing this up with hard statistics, my
20+ years of building and using software systems tells me that this
hypothesis is probably true. The collection of Linux command-line
programs is no exception to this generalization. Of the dozens of
shell-level commands offered by Linux, perhaps only ten commands
are commonly understood and utilized, and the remaining majority
are virtually ignored.Which of these dogs of the Linux shell have the most value to
offer? I'll briefly describe ten of the less popular but useful
Linux shell commands, those which I have gotten some mileage from
over the years. Specifically, I've chosen to focus on commands that
parse and format textual content.The working examples presented here assume a basic
familiarity with command-line syntax, simple shell constructs and
some of the not-so-uncommon Linux commands. Even so, the
command-line examples are fairly well commented and
straightforward. Whenever practical, the output of usage examples
is presented under each command-line execution.The following eight commands parse, format and display
textual content. Although not all provided examples demonstrate
this, be aware that the following commands will read from standard
input if file arguments are not presented.Table 1. Summary of
CommandsHead/TailAs their names imply, head
and tail are used to display some
amount of the top or bottom of a text block.
head presents beginning of a file
to standard output while tail does the same with the end of a file.
Review the following commented examples:

## (1) displays the first 6 lines of a file
   head -6 readme.txt
## (2) displays the last 25 lines of a file
   tail -25 mail.txt

Here's an example of using head and tail in concert to
display the 11th through 20th line of a file.

# (3)
head -20 file | tail -10 

Manual pages show that the tail command has more command-line
options than head. One of the more useful tail option is -f. When
it is used, tail does not return when end-of-file is detected,
unless it is explicitly interrupted. Instead, tail sleeps for a
period and checks for new lines of data that may have been appended
since the last read.

## (4) display ongoing updates to the given
##     log file 
tail -f /usr/tmp/logs/daemon_log.txt

Imagine that a dæmon process was continually appending
activity logs to the /usr/adm/logs/daemon_log.txt file. Using
tail -f at a console window, for example, will
more or less track all updates to the file in real time. (The -f
option is applicable only when tail's input is a file).If you give multiple arguments to tail, you can track several
log files in the same window.

## track the mail log and the server error log
## at the same time.
tail -f /var/log/mail.log /var/log/apache/error_log

tac--Concatenate in ReverseWhat is cat spelled backwards? Well, that's what tac's
functionality is all about. It concatenates file order and their
contents in reverse. So what's its usefulness? It can be used on
any task that requires ordering elements in a last-in, first-out
(LIFO) manner. Consider the following command line to list the
three most recently established user accounts from the most recent
through the least recent.

# (5) last 3 /etc/passwd records - in reverse
$ tail -3 /etc/passwd | tac
curly:x:1003:100:3rd Stooge:/homes/curly:/bin/ksh
larry:x:1002:100:2nd Stooge:/homes/larry:/bin/ksh
moe:x:1001:100:1st Stooge:/homes/moe:/bin/ksh

nl--Numbered Line Outputnl is a simple but useful
numbering filter. I displays input with each line numbered in the
left margin, in a format dictated by command-line options.
nl provides a plethora of options
that specify every detail of its numbered output. The following
commented examples demonstrate some of of those options:

# (6) Display the first 4 entries of the password
#     file - numbers to be three columns wide and 
#     padded by zeros.
$ head -4 /etc/passwd | nl -nrz -w3
001     root:x:0:1:Super-User:/:/bin/ksh
002     daemon:x:1:1::/:
003     bin:x:2:2::/usr/bin:
004     sys:x:3:3::/:
#
# (7) Prepend ordered line numbers followed by an
#     '=' sign to each line -- start at 101.
$ nl -s= -v101 Data.txt
101=1st Line ...
102=2nd Line ...
103=3rd Line ...
104=4th Line ...
105=5th Line ...
  .......

fmt--FormatThe fmt command is a simple text formatter that focuses on
making textual data conform to a maximum line width. It
accomplishes this by joining and breaking lines around white space.
Imagine that you need to maintain textual content that was
generated with a word processor. The exported text may contain
lines whose lengths vary from very short to much longer than a
standard screen length. If such text is to be maintained in a text
editor (like vi), fmt is the command of choice to transform the
original text into a more maintainable format. The first example
below shows fmt being asked to reformat file contents as text lines
no greater than 60 characters long.

# (8) No more than 60 char lines
$ fmt -w 60 README.txt > NEW_README.txt
# 
# (9) Force uniform spacing:
#     1 space between words, 2 between sentences
$ echo "Hello   World. Hello Universe." | \
fmt -u -w80 
Hello World.  Hello Universe.

fold--Break Up Inputfold is similar to fmt but
is used typically to format data that will be used by other
programs, rather than to make the text more readable to the human
eye. The commented examples below are fairly easy to follow:

# (10) Format text in 3 column width lines
$ echo oxoxoxoxo | fold -w3 
oxo
xox
oxo
# (11) Parse by triplet-char strings - 
#      search for 'xox'
$ echo oxoxoxoxo | fold -w3 | grep "xox"
xox
# (12) One way to iterate through a string of chars
$ for i in $(echo 12345 | fold -w1)
> do
> ### perform some task ...
> print $i
> done
1
2
3
4
5

trtr is a simple pattern
translator. Its practical application overlaps a bit with other,
more complex tools, such as sed and awk [with larger binary
footprints]. tr is quite useful
for simple textual replacements, deletions and additions. Its
behavior is dictated by "from" and "to" character sets provided as
the first and second argument. The general usage syntax of tr is as
follows:

# (12)  tr usage
tr [options] "set1" ["set2"] < input > output

Note that tr does not accept file arguments; it reads from
standard input and writes to standard output. When two character
sets are provided, tr operates on the characters contained in
"set1" and performs some amount of substitution based on "set2".
Listing 1 demonstrates some of the more common tasks performed with
tr.Listing 1. Common Tasks with
trprpr shares features with
simpler commands like nl and fmt, but its command-line options make
it ideal for converting text files into a format that's suitable
for printing. pr offers options
that allow you to specify page length, column width, margins,
headers/footers, double line spacing and more.Aside from being the best suited formatter for printing
tasks, pr also offers other useful features. These features include
allowing you to view multiple files vertically in adjacent columns
or columnizing a list in a fixed number of columns (see Listing
2).Listing 2. Using prMiscellaneousThe following two commands are specialized parsers used to
pick apart file path pieces.Basename/DirnameThe basename and dirname commands are useful for presenting
portions of a given file path. Quite often in scripting situations,
it's convenient to be able to parse and capture a file name or the
containing-directory name portions of a file path. These commands
reduce this task to a simple one-line command. (There are other
ways to approach this using the Korn shell or sed "magic", but
basename and dirname are more portable and straightforward).basename is used to strip
off the directory, and optionally, the file suffix parts of a file
path. Consider the following trivial examples:

:# (21) Parse out the Java Class name
$ basename
/usr/local/src/java/TheClass.java .java 
TheClass 
# (22) Parse out the file name.  
$ basename srcs/C/main.c 
main.c

dirname is used to display
the containing directory path, as much of the path as is provided.
Consider the following examples:

# (23) absolute and relative directory examples
$ dirname /homes/curly/.profile 
/homes/curly 
$ dirname curly/.profile
curly 
# 
# (24) From any korn-shell script, the following
#  line will assign the directory from where 
#  the script was launched 
SCRIPT_HOME="$(dirname $(whence $0))" 
# 
# (25)
# Okay, how about a non-trivial practical example?
#  List all directories (under $PWD that contain a  
#  file called 'core'.
$ for i in $(find $PWD -name core )^
> do 
> dirname $i
> done | sort -u
bin 
rje/gcc 
src/C

ConclusionThe multiple commands in this article are presented in
support of a hypothesis claiming the lion's share of a given
system's feature set goes unnoticed. My goal here was to increase
the awareness of several of the lesser utilized and showcased
utilities that offer some value. If you ever think, "there must be
a easier way to accomplish 'X'", while writing a script or while
struggling with something at the command-line prompt, perhaps there
is. Do some digging. One of the better sources for such digging is
the O'Reilly Linux in a Nutshell book--a well
organized, quick reference. I also would encourage you to examine
the installed manual and info-based pages--not all command-line
options were covered here.Louis Iacona has been
designing and developing applications on UNIX/Linux since 1982.
Most recently, his efforts have been directed at applying
leading-edge design/development techniques to the enterprise needs
of fortune 2000 companies. He is currently a Senior Consulting
Engineer at OmniE Labs, Inc.
(www.omnie.com).

email: lji@omnie.com