How to Index Anything

 in
You probably have search on your web site, but how about a search engine for the man pages on your system or even your mail? Try this simple indexing package.

You might want to build custom indices of documents for many reasons. A widely cited one is to supply search functionality to a web site, but you also may want to index your e-mail or technical documents. Anyone who has looked into implementing such a functionality has probably found it's not as easy as it might seem. Various factors conspire to make searching difficult.

The venerable and indispensable grep and its ilk are effective for scanning through lines of text. But grep, egrep and their relations won't do everything for you. They won't search across lines, they won't show search results in a ranked order and their linear search algorithms don't lend themselves to searching larger volumes of data.

HTML doesn't help the situation either. Its display-oriented features, idiosyncratic grammar and bevy of formatting and entity tags make it fairly difficult to parse correctly.

At the other end of the data storage spectrum is data slotted into a database. The ubiquitous example is that of the SQL database, which allows somewhat sophisticated search facilities but usually is not particularly fast for searching. Some database engines, notably MySQL 4, address this issue by allowing fast and ranked searches, but they may not be as customizable as desired.

In this article, we explore ways to create custom indices using SWISH-E, Perl and XML on Linux. Through examples, we show how SWISH-E can be used to build indices of HTML files, PDF files and man pages.

SWISH-E (simple web indexing system for humans—enhanced) is a descendant of SWISH, which was created in 1994 by Kevin Hughes. SWISH was transferred in 1996 to the UC Berkeley Library to fix bugs and add features, and the result was licensed under the GPL and renamed SWISH-E. Development continues, spearheaded by current project maintainer Bill Moseley and assisted by a team of developers.

Here at SkateboardDirectory.com, we happened upon SWISH-E when researching indexing toolkits. We found that it offers a unique combination of features that make it attractive for our purposes. Not only does SWISH-E offer a fast and robust toolkit with which to build and query indices, but it is also well documented, undergoes active development and bug fixes and includes a Perl interface. We also liked that maintainer Moseley and other experienced SWISH-E users and developers are usually prompt when addressing questions and bugs brought up on the SWISH-E mailing list.

Installing SWISH-E

For our examples, we started with a stock Red Hat 7.3 workstation with the Software Development bundle of packages installed. We also tested the examples on a Red Hat 6.2 workstation and a Debian Woody.

Currently, installing SWISH-E on Red Hat means installing from source, and the zlib and libxml2 libraries are required to build SWISH-E fully. If you find you need to install either, you probably can find packages provided with your distribution. We also use the xpdf package in our examples, so you may want to install that now if it isn't already. Our reference Red Hat 7.3 workstation setup had all of SWISH-E's prerequisites installed.

Here, we describe the use of SWISH-E 2.4, which according to the development team, should be released by the time you read this article. You can fetch and set up SWISH-E with the following sequence of commands, substituting the current version for (x.x):

% wget \
  http://swish-e.org/Download/swish-e-x.x.tar.gz
% tar zxf swish-e-x.x.tar.gz
% cd swish-e-x.x
% ./configure
% make
% make test

To install the SWISH-E binary, C libraries and man pages into their default locations in /usr/local, type make install as root. This installs the SWISH-E executable into /usr/local/bin. If this directory isn't in your PATH, either change your appropriate dot file to include /usr/local/bin in your PATH, or always call the swish-e executable by full pathname, like /usr/local/bin/swish-e.

Now, let's build and install the SWISH::API Perl module from the Perl directory in the source. We'll need it later when we build a Perl client for our index of man pages. SWISH::API is set up by the normal Perl module install process:

% cd perl
% perl Makefile.PL
% make
% make test

Then, install the SWISH-E Perl module by typing make install as root.

Now that SWISH-E and the SWISH::API Perl module are installed fully, let's build a simple index of HTML files to test SWISH-E. For this example, we index the HTML, one-page-per-section versions of the Linux Documentation Project (LDP) HOWTOs, which we've unpacked into ~/HOWTO-htmls/. The tarballs of LDP documents used in this article come from www.tldp.org/docs.html.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Re: How to Index Anything

Mengucino's picture

In this page you will find the way to do this. Also it has a lot of tips for webmaster and search engines.

Bye

Megucino from
______

Re: How to Index Anything

Anonymous's picture

Does anyone know how to indexing a Dinamic page? such as a php/jsp page?

Re: How to Index Anything

Vuelos's picture

For PHP you don´t have to do nothing expecial. The server will return to the bot HTML code not PHP or JSP code. The problem for a robot is to know if a page for example /pages.php?id=23 is diferent from /pages.php?id=24. The robot can´t index every page with different parameters so it must to implement an algorithm that allow to determine if pages are similar or equal and in this case it shouldn´t be indexed.

Re: How to Index Anything

augur's picture

Does anyone know how to indexing a Dynamic page? such as a php/jsp page?

If you spider the site (i.e. -S http) then you don't need to do anything special as long as the PHP/JSP code results in Text, HTML, or XML.
If you use FS method then, at least for PHP, you can have SWISH-E use the PHP cgi executable to process each document into Text, HTML, or XML. In your index configuration add something like this:

IndexContents HTML* .php
FileFilter .php /usr/bin/php "-q '%p'"

Can Swish deal separately with META?

Anonymous's picture

Can Swish deal separately with the META element? It would be very useful to be able to search arbitrary metadata such as authors, keywords or abstracts.

Re: Can Swish deal separately with META?

Anonymous's picture

yes, SWISH-E will automatically parse META tags in HTML/XML docs,
as per the current SWISH-E 2.4.0 documentation here.

SMAN Project RELEASED: search on man pages

Anonymous's picture

Hello All,
The SMAN project, is now publicly available from
http://joshr.com/src/sman.

SMAN is an enhanced version of the unix standbys 'man -k' and 'apropos,' as discussed in Josh Rabinowitz's "How To Index Anything" article in the July 2003 issue of Linux Journal.

Please test it out and let Josh know what you think!

From the SMAN README:

Sman is the Searcher for Man pages. Based on the example of the
same name in Josh Rabinowitz's article "How To Index Anything"
in the July, 2003 issue of Linux Journal
(http://www.linuxjournal.com/article.php?sid=6652), sman is
an enhanced version of 'apropos' and 'man -k'. Sman adds
several key abilities over its predecessors:

* Supports complex natural language text searches such as
"(linux and kernel) or (mach and microkernel)"

* Shows results in a ranked order

* Allows for searches by manpage section, title,
body, or filename

* Uses a prebuilt index to perform fast searches

* Performs 'stemming' so that a search for "searches"
will match a document with the word "searching"

Again, SMAN is available from available from
http://joshr.com/src/sman.

Posted on Tuesday, July 01, 2003?

Anonymous's picture

Posted on Tuesday, July 01, 2003?

Re: Posted on Tuesday, July 01, 2003?

Anonymous's picture

I tried the man page index example and got errors when I entered

swish-e -c sman-index.conf -S prog

I got many warnings like this:

Warning: Unknown header line: ...

Here are the first few and the last couple:

$ swish-e -c sman-index.conf -S prog
Indexing Data Source: "External-Program"
Indexing "./sman-index-prog.pl"
10373 man pages to index...

Warning: Unknown header line: 'll>' from program ./sman-index-prog.pl
:385: warning [p 2, 9.8i]: can't break line
:391: warning [p 2, 10.8i]: can't break line
:399: warning [p 3, 0.8i]: can't break line

Warning: Unknown header line: 'ntains spaces.' from program ./sman-index-prog.pl

Warning: Unknown header line: 'Tcl 8.1 Tcl(n)' from program ./sman-index-prog.pl

Warning: Unknown header line: '' from program ./sman-index-prog.pl

[snip]

Warning: Unknown header line: '>' from program ./sman-index-prog.pl

Warning: Unknown header line: '>' from program ./sman-index-prog.pl
err: External program failed to return required headers Path-Name: & Content-Length:
.

Re: setenv LANG C to work around UTF issues

Anonymous's picture

I was able to get around this by setting the environment variable LANG to "C" like this (adjust for your shell);

setenv LANG C

I think this only needs to be done before indexing with sman-update, and not for sman itself.

Re: Posted on Tuesday, July 01, 2003?

Anonymous's picture

The author says the code was tested on RH6.2, RH7.3, and Debian Woody. Maybe you made a typo, or you have multibyte man pages on your system (which the article and code mention that SWISH-E will gak on?)

I just tried the sman example above and it worked for me on RH6.2:

% swish-e -c sman-index.conf -S prog
Indexing Data Source: "External-Program"
Indexing "./sman-index-prog.pl"
4803 man pages to index...
processing 20
....

There's an enhanced version of SMAN in development at http://joshr.com/src/sman. This version should work better, since it's not shortened to fit in an article.

Re: swan from joshr.com still gives errors

Anonymous's picture

# rpm -q libxml2
libxml2-2.5.4-1
# uname -a
Linux localhost 2.4.20-8 #1 Thu Mar 13 17:54:28 EST 2003 i686 i686 i386 GNU/Linux

# sman-update --verbose --warn --debug

[snip maybe valuable information ?]

**==== END XML of /usr/share/man/mann/Tcl.n.gz =========

** working on /usr/share/man/mann/after.n.gz
** Running man /usr/share/man/mann/after.n.gz...

Warning: Unknown header line: 'd, even if the vari-' from program stdin

Warning: Unknown header line: 'able

Re: swan from joshr.com still gives errors

Anonymous's picture

Some people report that setting the environment variable LANG=en_US might help this issue. I've also heard that a new release of sman is coming that will make it easier to pinpoint the source of errors like this. Let us know if this works!

Re: sman from joshr.com still gives errors

Anonymous's picture

There is a new release at http://joshr.com/src/sman .
Please let us know if this solves your problem.

Sman rocks! Works for me!

Anonymous's picture

I've been using sman for a while on my systems with no problems. It even works on OS X now. There's a new version at http://joshr.com/src/sman

Sman Rocks, and it's on CPAN and Freshmeat

Anonymous's picture

You can now find the latest versions of Sman on Freshmeat at
http://freshmeat.net/projects/sman/
and on CPAN at
http://search.cpan.org/~joshr/Sman/

Re: How to Index Anything

Anonymous's picture

This is cool. But how does Google or Yahoo read through any file type for content. I have done searches for linux and these sites have returned pdf, word, html, excel, powerpoint, text, and even an microsoft project file. How can these sites run such massive searches?

Re: How to Index Anything

Anonymous's picture

Google is massive array of computers. That's why it is fast.
They have filters for those types. As long as you have filters, you can do it too.

sure, swish-e is not google,

Anonymous's picture

sure, swish-e is not google, and never will be, but

it can also index MsWord, OpenOffice, PDF, RTF (apart from standard xml, html, txt) - PPT filter is also available by now (see swish-e.org).

and last, but not least - I run swish-e on Windows and Linux too, almost everything described in this good article is possible with Windows version of swish-e (yes, you don't have man pages there :)

cheers

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState