Work the Shell - Looking More Closely at Letter and Word Usage

 in
More examples of using the shell to find the frequency of letters used in the English language.

Time and again I have entreated you, dear readers, with my plea for “A letter! My column, nay, my kingdom for a reader letter!” And lo, the miracle occurred, the heavens parted, the angels sang and a letter arrived:

In addition to the letter and word frequency, how about looking at how frequently a letter appears as the first letter of a word? Just to make things more interesting, what is the frequency of two-letter combinations? For instance, if the first letter of a two-letter combination is a t, what is the most frequent second letter? Thanks for the article in Linux Journal. It was a good read and nice scripts.—Mike Short

Quando omni flunkus moritati.

First off, before I even read the letter, I was intrigued by the closing quote. Latin? Isn't that, like, a dead language? Turns out the quote's a good one though, especially for IT admins in big companies. It roughly translates to “when all else fails, play dead”, and it comes from the Red Green Show, a Canadian comedy. (Thanks Google.)

Now, on to the heart of the letter. Mike's referring to an earlier column where we looked at how to use shell scripts to ascertain letter and word usage, using three books as source material: Dracula, A History of the United States and Pride and Prejudice, all downloaded from Project Gutenberg.

In that series of columns, we ascertained that the ten most common letters in the English language are e, t, a, o, n, i, s, r, h and d. Are they the same if we constrain it to just the first letter of words? Let's find out.

Extracting Just the First Letter of Words

Once we have a corpus of writing and the ability to break it down by words, so that the input stream to the counting script:

is
like
this

it's done like so:

$ cat dracula.txt | tr ' ' '\
' | grep -v '[^[:alpha:]]' | grep -v "^$"

That'll turn Dracula into the world's narrowest book, with one word per line.

Now we simply can add to it to axe all but the first letter by appending cut -c1. The result looks like one of those streams of letters in The Matrix, but that's another story.

So, all that's left is to translate uppercase into lowercase, sort, and then use our friend uniq -c to tally up the results:

tr '[:upper:]' '[:lower:]' | sort | uniq -c | sort -rn | head

And, the resultant top ten are:

20648 t
15787 a
11110 i
10655 w
9906 h
9030 s
7618 o
5720 m
5411 b
4597 f

Quite different! Now, the question is, does it change based on the type of content? Let's do the same command, but this time, let's feed in all three of our books, not just Dracula (though with the rabid <cough cough> popularity of Twilight, maybe Linux Journal would do well to stick with a vampire theme for a few issues?):

34359 t
27053 a
18212 w
18119 h
17854 i
15746 s
13614 o
10076 b
9792 m
7712 f

It's not exactly the same. Isn't that interesting? I'm not sure what to make of it, but as you can see, a good grasp of shell script commands makes finding out this sort of fairly goofy information interesting.

Calculating Digraphs

But, we're not quite done, because Mike also wondered about two-letter combinations. It's this sort of query that really shows just how helpful becoming savvy on the command line can be. To calculate that requires only one character to be changed in the command invoked above. Do you know what it is?

It's the cut command. Above, we're specifying that we want only the very first character of each line of input with cut -c1. If we want the first two, we simply can tweak that command flag as appropriate.

But, -c2 won't work, because that'll give us only the second letter of each word (and the most common second letter in the English language is o, followed by h, e, a and n).

Instead, we need to use a letter range, which looks like this: -c1-2. The result of that invocation is:

22100 th
10168 an
9138 to
7508 he
7100 of
5873 i<space>
5517 in
5332 ha
5157 be
4664 wh

There ya go, Mike. The most common two-letter combination in the English language is th, which actually makes some sense, followed as a distant second by an.

I hope it's trivially obvious how you could use this to calculate the most common three-letter combinations (it should be no surprise at all that the is the most common three letter combo, followed by and).

I'll wrap up here, but again, I invite you to send me your letters and queries so we can explore various ways to use shell scripts.

Dave Taylor has been involved with UNIX since he first logged in to the on-line network in 1980. That means that, yes, he's coming up to the 30-year mark now. You can find him just about everywhere on-line, but start here: www.DaveTaylorOnline.com.

______________________

Dave Taylor has been hacking shell scripts for over thirty years. Really. He's the author of the popular "Wicked Cool Shell Scripts" and can be found on Twitter as @DaveTaylor and more generally at www.DaveTaylorOnline.com.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

missing the n in the newline substitution

Anonymous's picture

$ cat dracula.txt | tr ' ' '\
' | grep -v '[^[:alpha:]]' | grep -v "^$"

shoule be:

$ cat dracula.txt | tr ' ' '\n
' | grep -v '[^[:alpha:]]' | grep -v "^$"

unless I am missing something ;)

Bash is not C or Python or ...

Mitch Frazier's picture

Ending a line with a backslash is the correct way to escape a newline in bash. In many languages, such as C, the backslash gives the next character special significance (eg \n causes the n to become a newline in the string). In bash, on the other hand, the backslash removes any special significance that the next character has. So a backslash as the last character on the line causes the next character (a real newline) to lose its special significance (which is to end a command line) and become a regular character.

The newline case is a bit strange for sure and causes some funny looking code sometimes, I tend to do this instead:

newline='
'

...

cat dracula.txt | tr ' ' "$newline" | grep ...

That makes it a bit more explicit as to what's happening and you only have to see the "weirdness" once at the top of your script.

Also note, that in Dave's example the newline is not actually required since it appears inside a quoted string. Bash won't recognize a newline as the end of a command if there is an open quoted string, so it just puts the newline into the string. So, the following will work also:

$ cat dracula.txt | tr ' ' '
' | grep -v '[^[:alpha:]]' | grep -v "^$"

Mitch Frazier is an Associate Editor for Linux Journal.

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState