Work the Shell - Generating Turn-by-Turn Driving Directions

 in
Maps? We don't need no stinking maps. With bash and a few command-line utilities, we can extract the textual driving instructions from an on-line map source and dispense with all those pesky GUI maps.

I'm happy to report that this month, I'm answering a reader's question about how to script something. Dunno what's up with the rest of you readers, but apparently writing to me with your weird and challenging shell-scripting puzzles isn't making the short list right now. Reader Paul M. asks:

Is there a way to screen-scrape Google Maps direction results? I'm after the text (turn left at Ho-ho-kus Blvd), not the maps. When I look at a saved results page, all I can see is CSS and JavaScript code. If I do a manual copy and paste of the directions, however, the turn-by-turn directions appear. Got any suggestions on how to grab turn-by-turn driving directions automatically, Dave?

Ah, those tricky programmers over at Google Maps make this pretty darn difficult! Poke around at the source pages generated by maps.google.com looking for directions, and it's clear that they're using a method=post or other advanced way to hide the starting and ending points from the URL itself, along with some very fancy coding to make the Web pages highly interactive. So to heck with it!

After much digging around and looking at how the different mapping sites work, I settled on Expedia.com as the best place to get driving directions so that we'll be able to specify start and stop points via URL and also understand the output. To get started, check out Expedia's interactive driving directions in your Web browser at www.expedia.com/Directions.

On Expedia, enter a starting and ending address for directions, and you'll find that it's all stored in a scary-complex URL like this: www.expedia.com/pub/agent.dll?qscr=mrdr&rtyp=0&unit=0&lats1=38.89872&lons1=-77.036379&alts1=5&strt1=1600+Pennsylvania+Ave+NW&city1=Washington&stnm1=DC&zipc1=20006&regn1=0&labl1=1600+Pennsylvania+Ave+NW%2C%0AWashington%2C+DC+20006&lats2=28.393142483519902&lons2=-81.57198620078931&alts2=5&strt2=N+World+Dr&city2=Orlando&stnm2=FL&zipc2=32830&regn2=0&labl2=World+Dr%2C%0AOrlando%2C+FL+32830&. (Eagle-eyed readers will notice that I'm offering the Obama family driving directions to Disney World.)

You can strip some of the superfluous information out of the URL and create a simple command-line call to get the map and directions:

start="strt1=1600+Pennsylvania+Ave+NW&city1=Washington&stnm1=DC"
dest="strt2=N+World+Dr&city2=Orlando&stnm2=FL&zipc2=32830"

curl --silent "http://www.expedia.com/pub/agent.dll?$start&
 ↪$dest&qscr=mrdr&rtyp=0&unit=0"

You can see that Expedia wants an address unwrapped and split by street address, city, state and zip code (though if it can figure out the location, it appears you can skip the zip code, as shown in start above).

Now that we have that, let's use sed to extract just the table of results, without the other superfluous information. This is done by manual analysis of the source file and noting that it's all in a table that starts with this HTML line:


<TABLE BORDER=1 BORDERCOLOR=#E4E4E4 CELLSPACING=0 CELLPADDING=4>

Not surprisingly, the line we seek that denotes the end of the table is </TABLE>. Here's the code that lets you slice things as desired:


sed -n '/BORDERCOLOR=#E4E4E4/,/<\/TABLE>/p'

Put them all together and save the output to a temp file. After that, the next challenge is to turn that HTML table into something you actually can read.

To do that, we're going to turn to a great open-source utility called Lynx. You might already have Lynx on your system, but if you don't, grab a copy of the Lynx text-based Web browser from lynx.isc.org. We'll use that to interpret and convert the HTML markup to raw text.

Fortunately, Lynx excels at this kind of challenge, as demonstrated by the working code:

curl --silent "http://www.expedia.com/pub/agent.dll?$start&
 ↪$dest&qscr=mrdr&rtyp=0&unit=0"| \
   sed -n '/BORDERCOLOR=#E4E4E4/,/<\/TABLE>/p' | \
   lynx -dump -stdin

Yup, that's it. Specify a correct start and destination, make sure that the script knows where to find Lynx on your system, and the output will look like this:


Directions Distance Time
Start: Depart Start on Local road(s) (East) 0.1 < 1min
    1: Turn RIGHT (South) onto E Executive Ave NW 0.1 0:01
    2: Turn LEFT (East) onto Alexander Hamilton Pl NW, then
       immediately turn RIGHT (South) onto 15th St NW 0.1 0:01
    3: Turn LEFT (East) onto Pennsylvania Ave NW, then immediately
       turn RIGHT (South) onto 14th St NW 0.3 0:02
    4: Keep STRAIGHT onto US-1 [14th St NW] 1.1 0:02
  ...
   22: Take Ramp (LEFT) onto Western Way (Disney World) 1.9 0:02
   23: Turn LEFT (North) onto Bear Island Rd 2.1 0:03
   24: Turn RIGHT (East) onto Floridian Way 0.3 0:01
   25: Keep STRAIGHT onto World Dr 0.4 0:01
  End: Arrive End < 0.1 < 1min
Total Route 881 mi 13 hrs  2 mins

I'll leave it as an exercise to you, dear reader, to create a wrapper that prompts people for starting and ending addresses and then uses the curl invocation to Expedia and subsequent invocation of Lynx to display turn-by-turn driving directions.

Dave Taylor has been hacking shell scripts for a really long time, 30 years. He's the author of the popular Wicked Cool Shell Scripts and can be found on Twitter as @DaveTaylor and more generally at www.DaveTaylorOnline.com.

______________________

Dave Taylor has been hacking shell scripts for over thirty years. Really. He's the author of the popular "Wicked Cool Shell Scripts" and can be found on Twitter as @DaveTaylor and more generally at www.DaveTaylorOnline.com.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Expedia has changed to use

Anonymous's picture

Expedia has changed to use POST method '-d' option with curl. You'd have to test if the same fields still work but I suspect they would.

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState