Introduction to Lisp-Stat

Although I installed Linux on a 80 Meg partition on my Gateway 33Mhz PC over a year ago, I really did not make serious use of Linux for my scientific work, mostly because I lacked disk space. Recently, I bought a new 1-Gig drive and that excuse went away. So I decided to install Lisp-Stat, a program that I use most for my statistical Computing.

Written by Luke Tierney at the University of Minnesota, Lisp-Stat is a powerful, interactive, object-oriented statistical computing environment based on the the Xlisp dialect of Lisp. It runs on under Microsoft Windows, Macs, and Unix based X11 systems almost uniformly. It has good graphical facilities for both static and dynamic graphics along with functions for common statistical computations.

Furthermore, using the foreign function interface, one can call C and Fortran programs from within Lisp-Stat. A byte-code compiler is available to speed up your programs once you have debugged them. Of course, one needn't use Lisp-Stat for statistical computing alone; I routinely use for all kinds of things: as a calculator, for figuring out grades of my students, as an engine for hypertext illustrations as well as matrix manipulations.

In this article, I will introduce you to some of the capabilities of Lisp-Stat. Although I use Lisp in this article, I will not get into the details of Lisp programming unless it impinges on our discussion. If you are new to Lisp, you might want to read article on Scheme by Robert Sanders, Linux Journal, March 1995, as Lisp and Scheme are closely related. Some of the comments I make apply actually to Lisp, but it serves no useful purpose to delineate what is the Lisp and what is the Stat part. You need not know Lisp to use any of the examples or to follow the article. If you get seriously interested in Lisp-Stat you should probably get a copy of Tierney's book titled Lisp-Stat, ISBN 0-471-50916-7, published by John Wiley. Besides being the canonical reference for Lisp-Stat, it provides a quick and practical introduction to Lisp.

Figure 1: An example of a Lisp-Stat session

Assuming that you have installed Lisp-Stat successfully, just type xlispstat to invoke the program. To quit the program, just type (exit). Figure 1 shows a simple session. Case does not matter and the > you see in the figure is Lisp-Stat's prompt. The data I have used is the number of requests a WWW server honored during each of the 24 hours in a day. The def macro binds a variable name requests to the list of values.

As the example shows, calculation of summary statistics like the mean and standard deviation are trivial. Since Lisp-Stat is based on Lisp, you have all the power of Lisp for data manipulation. A rich set of data types is available including vectors, sequences, strings, matrices. In figure 1 the variable A is defined to be a 3x3 matrix and B is a list of three numbers. The example solves Ax=b for x by computing A<+>-1<+>b yielding the solution [2, -6, 1]. Note that b is a list while A<+>-1<+> is a matrix, yet the Lisp interpreter takes care of the types and in effect computes the product of a matrix and a vector.

Many of Lisp-Stat's functions operate on sequences which may be lists or vectors and they are vectorized, meaning that these functions can be applied to arguments that are lists and the result is a list of the results of applying the function to each element of the list. Some other functions are vector reducing, meaning that they can be applied to a list of arguments but they return a single number. In figure 1, the function mean is an example of a vector-reducing function; it treated the list of lists as a single long list and returned the mean of the long list. On the other hand, the function normal-cdf is a vectorized function and invoking it on a list of three numbers produces a list of three answers. Of course, if we do wish mean to behave in a vectorized fashion, the statement (mapcar #'mean (list (normal-rand 10) (normal-rand 20))) will do it.

Figure 2: A histogram and a Line Plot

A picture is worth a thousand words, particularly in statistics. Lisp-Stat boasts excellent graphical tools. The graphical system is based on an object-oriented paradigm. Functions that create graphical windows or plots return an object as the result. The returned object is just another data type much like a number or a list and it can be used in appropriate computations.

Commonly used graphical functions are histogram for constructing histograms, plot-points for plotting (x,y) pairs, plot-lines for joining (x,y) pairs by means of lines, plot-function for plotting a function of one variable, spin-function for plotting a function of two variables, and spin-plot for 3-d plots.

All the spin functions provide controls for yawing, pitching and rolling in the graph they create. Figure 2 shows a plot of the number of requests versus each of the 24 hours. The plots were produced using the following lines of code.

(histogram requests)
(def time (iseq 24))
(plot-lines time requests)
(send * :add-points time requests)
(send ** :point-symbol (iseq 24) 'diamond)

Just drawing the lines alone is less than satisfactory since the exact location of the points is lost. So, after constructing the plot, we send a “message” to the plot using the send function asking the object to add-points to the graph resulting in the graph shown. The * in the send function refers to the result of the previous command, i.e., the plot object. The ** refers to the result of the command before the previous one. In the example, I have asked that the plotting symbol be a diamond instead of the default circle. The user has a choice of quite a few plotting symbols.

In each plot there is a menu button that has further useful options. One can select or deselect points with the mouse, highlight certain points, save the plot as a postscript file etc. I will only discuss a single feature, that of linking. Linked plots are a way of sharing information between plots. Consider for example, figure 2, where we have a plot of requests versus time as well as a histogram of requests.

If you enable linking by choosing the Link View item in the menu in each plot, selecting a vertical bar in the histogram by dragging the mouse with the button pressed causes the corresponding points in the line plot to be highlighted. You might have to peer at the figure to see that the point where the highest peak occurs is highlighted since it corresponds to the highlighted histogram bar. Linking is extremely useful in viewing multidimensional data since one can get a better idea of how the same group of points can be projected in different views.

Online documentation for Lisp-Stat is available via the functions help, help* and apropos. For help on the mean function, type (help 'mean). The use of the quote is essential, otherwise the interpreter would assume that mean is a variable and try to evaluate it. However, in many situations, one does not know what the function is named.

For example, is the function that multiples two matrices mat-mult or matrix-multiply? Typing (apropos 'mult) will print a list of all symbols that have the word “mult” in them. This might help you narrow down the search. On the other hand, if you know that the function you are looking for contains the word matrix in it, (help* 'matrix) will return help on all symbols that contain the word matrix. The help facility as it exists now is less than optimal and several people are developing a more elaborate help system.

I usually read the newsgroup sci.stat.math and almost always there is someone out there who wants to know how to calculate an F-probability or how to generate a normal random variable. Lisp-Stat has distribution functions and generators for all of the commonly used distributions. For example (normal-cdf 1.645) will give you the probability to the left of 1.645 which is about 0.95. The statement(def x (normal-rand 100)) will define x to be a list of 100 standard normal variates. Similar functions exist for Students-T, Gamma, Beta, Chi-squared and F distributions as (help* 'cdf) or (help* 'rand) will show.

Lisp-Stat has many functions for input and output. For dealing with files, I've rarely needed to go beyond using the two functions read-data-file and read-data-columns. The statement (read-data-file "foo.dat") returns the whole file contents as one long list, while (read-data-columns "foo.dat") returns a list of columns of the file. One can specify the number of columns in the data file as a second argument to read-data-columns. Otherwise, it guesses the number of columns based on the first line. The function format, which is similar to C's sprintf(), is a versatile function for formatted printing.