Home ›

Scripting GNU in the 21st Century

May 27, 2004  By Nick Moffitt
 in
Scripting in the GNU environment and parsing HTML in bash.
Command-Line Options

In our usage function, we mentioned a number of command-line options. For one thing, we promised that we would allow the user to list station codes. Alas, the function to do this is nowhere near as elegant as the rest of the script: 


	function liststations {
	    wget -O - -o /dev/null 'http://bart.gov/index.asp?ct=1' | \
	    html2 2> /dev/null | \
	    sed '/\/html\/body\/table\/tbody\/tr\/td\/table\/tr\/td\/form\/select\/@name=origin/,/\/html\/body\/table\/tbody\/tr\/td\/table\/tr\/td\/form\/br/p;d;'  | \
	    cut -d = -f 2 | grep -v ^/ | tail +6 | \
	    while read i; do read j; echo -e "$i\t$j"; read blank; done
	}

The command-line arguments to a bash script are stored in numbered variables. Recall that we used $0 to get the name of the script in the usage function. The rest of the arguments are likewise stored in $1, $2 and so on. The number of arguments is stored in the $# variable.

The end of our script now reads: 


	case $1 in
		-h|--help)
			usage
			exit 0;;
		-l|--list)
			liststations
			exit 0;;
		-*|--*)
			usage
			exit 1;;
	esac

	if [ $# = 1 ]
	then
		BARTDEST=$1
	elif [ $# = 2 ]
	then
		BARTDEST=$2
		BARTSTART=$1
	fi

	extractdata | formatdata
Other Features

The full version I use contains all sorts of extra features, including color escape sequences, return trips with a specified delay and the ability to simply spit out the URL to be pasted into a Web browser. That version will be available here for the foreseeable future.

Nick Moffitt is a free software enthusiast living in Oakland, California, where he maintains a multiuser community shell server. He is a member of the LNX-BBC Project and maintains GAR, nwall and the popular game robotfindskitten.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Re: Scripting GNU in the 21st Century

Anonymous's picture
Submitted by Anonymous on Mon, 05/31/2004 - 02:00.

It should be noted that the LSB is POSIX-based.

Re: Scripting GNU in the 21st Century

Anonymous's picture
Submitted by Anonymous on Mon, 05/31/2004 - 02:00.

Overall, for someone learning bash, this is probably a reasonable example. I have many similar ones myself (grabbing satellite wildfeed data, for example).

However, as a means of introducing a newcomer to bash, or as a convincing description of why the newcomer should be using bash, I feel it falls short.

For example, some simple timing shows that his "$(basename $0)" construct is almost 100x slower than using "${0##*/}", although he does use another version of the same construct later, meaing that he is aware of it!

His repeated use of the backslash character as a line continuation does not improve the readability of the script; in fact, it makes it worse. Leave it out! Yes, it still works -- if the line ends with a character which indicates there's more needed, the line continuation character is redundant. The pipe symbol (vertical bar) is such a character.

In general, all variable usage should be enclosed in double quotes (ie, "Dollar signs in Double quotes"). This technique is only wrong 1 time out of 100, so the programmer will be correct 99% of the time. :) Yes, it may mean the double quotes are redundant in some cases, but there's a lot to be said for consistency, and hence, readability. Only when dealing with word splitting (where you want the word split), will the double quotes be incorrect.

He appears to use brackets in "if" statements when the POSIX (?) technique would be double parentheses (brackets are for string comparisons, parens are for numeric comparisons, and the doubled form of each is recommended since they turn off I/O redirection, wildcarding, and word splitting). Maybe he doesn't know?

Lastly, the "liststations" function seems overly complex to me. First, is it really necessary to specify the entire XPath all the way from the "html" element down to the "select" element and its attribute?? I haven't seen the data, but I'd be willing to bet that just the "select" element and attribute would be enough (since select is non-functional outside of forms anyway, and the attribute specifies the name of the select element!). Regardless of whether simplification is possible, change the delimiter of the regexpr! Use something other than a slash and avoid LTS ("leaning toothpick syndrome", per Larry Wall). With the text thus cleaned up visually, maybe let sed also do the elimination of text up to and including the equals sign? That eliminates the need for cut, although it may hurt readability. Additionally, sed can also replace the while loop; tell sed to match on the "select" element and attribute, then read three more lines into the holding space, appending them to whats already there. Now run a substitution on the hold space and print the result. (Or if you're not comfortable with sed, use awk, and you still eliminate the cut and while loop.) YMMV.

Overall, I agree with the other comment posted here that the standard for scripts should be POSIX, not a particular tool. Of course, POSIX has its own problems (quite a few, actually!), but that's a decision that individual organizations need to make: portability vs. speed/usability.

The sh POSIX standard

Anonymous's picture
Submitted by Anonymous on Fri, 05/28/2004 - 02:00.

The sha-bang ( #!) at the head of a script tells your system that this file is a set of commands to be fed to the command interpreter indicated. The #! is actually a two-byte [1] "magic number", a special marker that designates a file type, or in this case an executable shell script (see man magic for more details on this fascinating topic). Immediately following the sha-bang is a path name. This is the path to the program that interprets the commands in the script, whether it be a shell, a programming language, or a utility. This command interpreter then executes the commands in the script, starting at the top (line 1 of the script), ignoring comments. [2]

#!/bin/sh
#!/bin/bash
#!/usr/bin/perl
#!/usr/bin/tcl
#!/bin/sed -f
#!/usr/awk -f

Each of the above script header lines calls a different command interpreter, be it /bin/sh, the default shell (bash in a Linux system) or otherwise. [3] Using #!/bin/sh, the default Bourne Shell in most commercial variants of Unix, makes the script portable to non-Linux machines, though you may have to sacrifice a few Bash-specific features. The script will, however, conform to the POSIX [4] sh standard.

REF:

1) Advanced Bash-Scripting Guide
2) The Single UNIX Specification, Version 3

The Horror

Anonymous's picture
Submitted by Anonymous on Fri, 05/28/2004 - 02:00.

While it's a neat hack to parse HTML using bash, and I respect the authors significant contributions to Free Software (LNX-BBC, GAR - "We're not worthy!"), isn't this really a sign that scripting activities on GNU/Linux (and UNIX systems, if you must) should really be employing proper languages like Python and [insert favourite "agile" language here]?

Re: The Horror

Anonymous's picture
Submitted by Anonymous on Mon, 05/31/2004 - 02:00.

No. These days anything goes and Bash is appropriately qualified. Respect is due to people who enjoy time-tested languages.

Re: Scripting GNU in the 21st Century

Anonymous's picture
Submitted by Anonymous on Fri, 05/28/2004 - 02:00.

Somebody go tell those who are making the 'GNU' autoconf and automake??

Re: Scripting GNU in the 21st Century

Anonymous's picture
Submitted by Anonymous on Fri, 05/28/2004 - 02:00.

.. of course, the enitre point of GNU autotools is to enable you to code programs wuch that they compile regardless of what is and isn't avaliable on the build, host and target platforms, and if we start assuming they're fully GNU compatible it sort of defeats the point a bit, no?

Caution: Theater-Wide Monitor Required (NT)

Anonymous's picture
Submitted by Anonymous on Thu, 05/27/2004 - 02:00.

.

Webcast
More Resources
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers

Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.

Learn More

Sponsored by AMD

White Paper
More Resources
Red Hat White Paper: Using an Open Source Framework to Catch the Bad Guy

Built-in forensics, incident response, and security with Red Hat Enterprise Linux 6

Every security policy provides guidance and requirements for ensuring adequate protection of information and data, as well as high-level technical and administrative security requirements for a system in a given environment. Traditionally, providing security for a system focuses on the confidentiality of the information on it. However, protecting the data integrity and system and data availability is just as important. For example, when processing United States intelligence information, there are three attributes that require protection: confidentiality, integrity, and availability.

Learn more about catching the bad guy in this free white paper.

Learn More

Sponsored by DLT Solutions