Extending Web Services Using Other Web Services

Last month, we looked at the latest incarnation of Web services offered by on-line giant Amazon. Amazon was one of the first companies to embrace Web services, and although some of its newer offerings require payments on a monthly or per-query basis, basic catalog searches are still available free of charge.

If we think of each individual Web service as a function call, we can think of a collection of Web services, such as Amazon Web Services (AWS), as a software library. And although we can certainly create interesting applications with such libraries, it is often useful to create new libraries that sit on top of the existing ones. In many ways, the history of software is the history of creating increasingly powerful abstractions by stacking libraries on top of one another. Outside of the classroom, most of us haven't ever had to implement a sort algorithm or create a buffered I/O library, simply because such things have been written and optimized by previous generations of programmers.

I thus believe that it's useful for us to consider AWS not as a set of routines that we can incorporate into end-user programs, but rather as a set of low-level libraries on top of which we can (and should) create new libraries appropriate for our specific needs.

This month, we look at a simple example of what I mean. The project will reflect my love of books. The Internet has made it difficult for me to stop buying used books, because so many are available at low prices. But, I'm fortunate to be spending several years in Skokie, Illinois, which has an excellent public library. Skokie's library has not only an extensive collection, but it also has a Web-based interface to the book catalog. Our project for this month, thus, is to create a Web service that integrates Amazon's catalog with the information from the Skokie public library. In other words, we're going to write a Web service that itself relies upon another Web service. The input to our service will be an International Standard Book Number (ISBN); the output will be an indication of the book's availability and price at Amazon and the Skokie library.

In some ways, this Web service will duplicate the excellent Book Burro plugin for the Firefox Web browser, which I often use to find the best bargains. And indeed, Book Burro looks at both bookstores and public libraries in order to find books. I recommend Book Burro to everyone who uses Firefox. But, I believe that building your own simple Web service, even if it duplicates the functionality of another program, is a worthwhile endeavor.

Moreover, Web services have the advantage of being available from any programming language and any application. I can implement my Web service using Ruby, and people will still be able to access it from Java, Python, Perl or virtually any other language. In many ways, this achieves what object broker middleware services like CORBA had promised, only without the baggage that made CORBA a more complex (but arguably more secure and rich) programming platform. It makes a Web service more powerful than a simple software library, because it can be accessed from any platform or language, so long as the requesting computer is connected to the Internet.

In order to integrate an ISBN search for the Skokie library, we're going to need a way to query the library for information about book availability. Unfortunately, my library doesn't have a Web services API for querying its database. But, it does have the next-best thing, namely a simple Web interface that we can query.

There are several ways to look through the output from a Web page. Because many sites now use HTML that can be parsed as if it were XML, we might want to use an XML-parsing library to read through the response from the library's Web site, looking for particular text in specific places.

Much as I might like the idea of such an approach, I'm probably not the only Web developer who takes a more practical, quick-and-dirty look. I have used my library's Web site enough times to know that there is a limited number of responses it might send back to me. As a result, I'll use the reliable, if somewhat stupid, approach of looking for particular cues in the HTTP response.

Our program (skokie-lookup.rb, Listing 1) is written in Ruby, a language I have grown to enjoy more and more over the past few months. We begin by importing the included Net::HTTP module, which defines classes and methods that provide HTTP-based communication.

#!/usr/bin/ruby

require 'net/http'

if ARGV.length == 0
  puts "#{$0}: You must enter at least one argument."
  exit
end

output = ""

# Set up our regular expressions
not_in_collection_re = /class="yourEntryWouldBeHereData"/ix
on_shelf_re = /CHECK SHELF/ix
checked_out_re = /DUE /ix

# Iterate through each of our arguments
ARGV.each do |isbn|

  # Ignore non-ISBN arguments
  if not isbn.match(/[0-9xX]{10}/)
    output << "ISBN #{isbn} is invalid.\n"
    next
  end

  # Ask the library what it knows about our ISBN
  response = Net::HTTP.get_response('catalog.skokie.lib.il.us',
                                    "/search~S4/i?SEARCH=#{isbn}")

  # Check our regular expressions against the HTML response
  if not_in_collection_re.match(response.body)
    output << "ISBN #{isbn} is not in the Skokie collection.\n"
  elsif on_shelf_re.match(response.body)
    output << "ISBN #{isbn} is on the shelf.\n"
  elsif checked_out_re.match(response.body)
    output << "ISBN #{isbn} is currently checked out.\n"
  else
    output << "ISBN #{isbn} response: Unparseable!\n"
  end
end

# Show everyone what we've learned
puts output

We then check to make sure that we have at least one command-line argument, by looking at the built-in ARGV array. If the length of ARGV is 0, we know we weren't passed any arguments, and we should give the user a brief indication of how the program should be used.

Then, we set up a number of variables that will be needed later on. The output variable is a string to which we will add any output we need to send to the user. We also create three Regexp (regular expression) objects, which we will use in our loop.

Next comes the meat of the program. We iterate over each element of ARGV, first checking that it is a ten-character ISBN containing only numbers and the letter X. We then query the Skokie library's Web site for that ISBN, passing Net::HTTP.get_response the hostname and path to the program we want. The HTTP response, including its headers and body, is then available in our response variable.

Now we compare the response body against our three regular expressions, checking which it matches. Using Ruby's << operator for concatenation, we add an appropriate message to the output variable for each ISBN. Finally, just before the program exits, it gives a full report of ISBNs.