Work the Shell - Making a <emphasis>Mad Libs</emphasis> Generator
My son is at the age when he's decomposing sentences, diagramming them and learning about the parts of speech. Me? I couldn't differentiate between an adverb and an adjective if a wet, smelly red ball smacked me in the head. That's why I have an editor!
There are games for everything, however, and one of the best games for learning the parts of speech is a simple one that's been around since I was a kid: Mad Libs. You know what I'm talking about, it takes simple sentences like: “When my dog is happy, he jumps and barks, his tail wagging a mile a minute.” and transforms them into: “When my [ noun ] is [ adjective ], he [ verb ] and [ verb ], his [ noun ] wagging a mile a [ noun ].”
The question is, can we write a shell script that can perform this sort of transformation? The answer, of course, is yes.
There are two challenges with this project: figuring out which words to replace with their parts of speech and figuring out the part of speech of a given word. Let's tackle these in reverse order.
It turns out that a number of different Web sites let you look up a word and offer its definition and part of speech. The one I use for this exercise is from Princeton, because it's fast, easy to parse and easy to submit queries.
To look up the part of speech of, say, “dog”, the URL to invoke is simply wordnetweb.princeton.edu/perl/webwn?s=dog.
The result highlights the part of speech as an h3 line, so isolating that element is a breeze:
curl --silent "lookup$word" | grep '<h3>'
This particular word demonstrates one of the nuances of the problem: many words have more than one part of speech, demonstrated by the difference between a pet dog and someone who is dogging your every footstep. Sure enough, the result:
For simplicity's sake, let's just take the first match, easily done by adding | head -1 to the pipe. Next, let's drop it all into lowercase and strip out the HTML:
| tr '[:upper:]' '[:lower:]' | sed 's/<h3>//;s/<\/h3>//'
Both of these are worth a bit of explanation. You might well have seen tr '[A-Z]' '[a-z]' as the more common way to transliterate uppercase to lowercase, and that works just fine, if you're working in English. Using the character sets “:upper:” and “:lower:” is a more portable alternative that's preferred.
The sed command also lets you specify more than one command argument to apply by simply separating them with a semicolon. What we have here is a substitution of <h3> to a null string (for example, removing it), followed by the same thing for </h3>.
That's all we need to get the part of speech. For example:
$ lookup="http://wordnetweb.princeton.edu/perl/webwn?s=" $ word="happy" $ curl --silent "$lookup$word" | grep '<h3>' | tr '[:upper:]' '[:lower:]' | sed 's/<h3>//;s/<\/h3>//' adjective
And, the hard part's done!
For this article, let's use a replacement density constant to figure out whether any given word should be replaced. The higher the density, the more likely a given word in the input stream will be replaced by its part of speech.
This is lazy and not a great solution, because it can match “is” or “the” just as easily as “dog” or “tail”, but let's go with it for now to get a sense of how it'll all fit together. We'll come back to it and improve the sophistication of the selection criteria later. With me? Good!
For a given word, deciding whether to substitute its part of speech can be calculated as follows, assuming we have a variable called density that has a nonzero integer value:
if [ $(( $RANDOM % $density )) = 1 ] ; then
$RANDOM is one of those cool magic variables in the Bourne shell that has a different value each time you reference it—handy!
Let's put these together and see what we get. We'll use an initial density of 5, which theoretically should mean that if we have a properly random $RANDOM, each word should have a 1:5 chance of being replaced.
The script needs to read the input word by word, testing each word as it goes. This can be done easily with the following loop structure, assuming that the text input comes from stdin:
while read sentence ; do for word in $sentence ; do
Now, we add the random conditional and have a skeleton ready to test:
while read sentence ; do for word in $sentence ; do if [ $(( $RANDOM % $density )) -eq 1 ] ; then echo "(($word))" else echo $word fi done done
You can see that at this stage we're going to output the words we're planning on replacing with “(())”. Here's a quick test:
echo this is a test mad-lib input | sh make-madlib.sh this is ((a)) test ((mad-lib)) input
One tiny tweak before I wrap it up for the month—how do we get the words to appear on the same line? It's easy. Remember that each of the code loops is essentially a little script of its own, so this task can be accomplished by adding four characters to the very end of the outermost loop:
done done | fmt
That's all you have to do—add the |fmt after the second done statement. Now when it's run:
echo this is a test mad-lib input | sh make-madlib.sh this is a ((test)) ((mad-lib)) input
Next month, we'll add the part of speech lookup code into the conditional and then spend some time exploring a more sophisticated word choice algorithm. Clearly, random isn't as beneficial.
Dave Taylor has been hacking shell scripts for a really long time, 30 years. He's the author of the popular Wicked Cool Shell Scripts and can be found on Twitter as @DaveTaylor and more generally at www.DaveTaylorOnline.com.
Dave Taylor has been hacking shell scripts for over thirty years. Really. He's the author of the popular "Wicked Cool Shell Scripts" and can be found on Twitter as @DaveTaylor and more generally at www.DaveTaylorOnline.com.
|Dynamic DNS—an Object Lesson in Problem Solving||May 21, 2013|
|Using Salt Stack and Vagrant for Drupal Development||May 20, 2013|
|Making Linux and Android Get Along (It's Not as Hard as It Sounds)||May 16, 2013|
|Drupal Is a Framework: Why Everyone Needs to Understand This||May 15, 2013|
|Home, My Backup Data Center||May 13, 2013|
|Non-Linux FOSS: Seashore||May 10, 2013|
- Dynamic DNS—an Object Lesson in Problem Solving
- Making Linux and Android Get Along (It's Not as Hard as It Sounds)
- Using Salt Stack and Vagrant for Drupal Development
- New Products
- A Topic for Discussion - Open Source Feature-Richness?
- Drupal Is a Framework: Why Everyone Needs to Understand This
- Validate an E-Mail Address with PHP, the Right Way
- RSS Feeds
- Readers' Choice Awards
- Tech Tip: Really Simple HTTP Server with Python
3 hours 33 min ago
- Reply to comment | Linux Journal
4 hours 6 min ago
- All the articles you talked
6 hours 29 min ago
- All the articles you talked
6 hours 32 min ago
- All the articles you talked
6 hours 34 min ago
10 hours 58 min ago
- Keeping track of IP address
12 hours 49 min ago
- Roll your own dynamic dns
18 hours 3 min ago
- Please correct the URL for Salt Stack's web site
21 hours 14 min ago
- Android is Linux -- why no better inter-operation
23 hours 30 min ago
Enter to Win an Adafruit Pi Cobbler Breakout Kit for Raspberry Pi
It's Raspberry Pi month at Linux Journal. Each week in May, Adafruit will be giving away a Pi-related prize to a lucky, randomly drawn LJ reader. Winners will be announced weekly.
Fill out the fields below to enter to win this week's prize-- a Pi Cobbler Breakout Kit for Raspberry Pi.
Congratulations to our winners so far:
- 5-8-13, Pi Starter Pack: Jack Davis
- 5-15-13, Pi Model B 512MB RAM: Patrick Dunn
- 5-21-13, Prototyping Pi Plate Kit: Philip Kirby
- Next winner announced on 5-27-13!
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?