How to Index Anything

 in
You probably have search on your web site, but how about a search engine for the man pages on your system or even your mail? Try this simple indexing package.
Indexing HTML on the Filesystem

The first step in building an index with SWISH-E is writing a configuration file. Create a directory like ~/indices, cd into it and create the file ./howto-html.conf with the following contents:

# howto-html.conf
IndexDir  ../HOWTO-htmls/
IndexOnly .html
IndexFile ./howto-html.index

The IndexDir directive specifies the directory in which SWISH-E should look for files to be indexed. The IndexOnly directive requests that only files ending in .html be indexed. Finally, the location of the index to be created is specified with the IndexFile directive.

Our First Index

Now, let's build our index of HTML files with the command:

% swish-e -c howto-html.conf

The -c option specifies which SWISH-E configuration file to use. On an older system, building this index may take a few minutes or so; on a contemporary one, it should take under a minute. Figure 1 illustrates the process of indexing HTML files on the filesystem with SWISH-E.

Figure 1. Indexing HTML on the Filesystem with SWISH-E

Searching the Index

Let's test our first index by doing a simple search for HTML files relevant to the term NFS. You can test SWISH-E indices quickly using the swish-e executable by specifying an index with the -f option, and the text to be searched with the -w option; searches on SWISH-E indices are case-insensitive. Because we expect a lot of pages (or hits) to include the word NFS, we use the -m 3 option to request only three:

% swish-e -f howto-html.index -m 3 -w nfs

This returns (abridged and reformatted):

1000 ../HOWTO-htmls/NFS-HOWTO/performance.html
        "Optimizing NFS Performance" 33288
998 ../HOWTO-htmls/NFS-HOWTO/intro.html
       "Introduction" 10966
993 ../HOWTO-htmls/NFS-HOWTO/security.html
       "Security and NFS" 35968

Not bad—those pages are definitely about NFS, and the output is intuitive. The first column is the rank SWISH-E gives each hit—the hits considered most relevant always are ranked 1000, with less-relevant files ranked in descending order. The second column shows the name of the file, the third gives the page's title and the fourth shows the byte count of the indexed data. SWISH-E determines the title of each page from the HTML tags in each file using one of its HTML parsing engines.

The built-in SWISH-E parsing engines are called TXT, HTML and XML, and each is designed to parse the corresponding type of content. Recent versions of SWISH-E also can use the libxml2 library for the HTML2 and XML2 parsing back ends. Both the XML2 and HTML2 parsers are preferable to their built-in counterparts—especially HTML2. This is why a recent version of libxml2, though technically optional when building SWISH-E, probably should be considered a prerequisite.

Basic SWISH-E Search Syntax

SWISH-E supports a full-featured text retrieval search language with syntax including AND, OR, NOT and parenthetic grouping that all work predictably. For example, the following searches all have the expected semantics:

% swish-e -f howto-html.index -w nfs AND tcp
% swish-e -f howto-html.index -w nfs OR tcp
% swish-e -f howto-html.index \
        -w '(gandalf OR frodo) OR (lord AND rings)'
The Configuration File

SWISH-E configuration files are simple text files in which each line is either a directive or a comment. Any line in which the first non-whitespace character is a # is ignored by SWISH-E as a comment. All other non-empty lines should be in the form:

Directive Options [Options] ...

If you need to specify an option with spaces embedded, you can use quotation marks:

Directive "Option With Spaces!"

If the option has single quotation marks within it, you can quote it with the double quote character and vice versa, for example:

Directive "Fred's Index Option"
Directive 'By Josh "joshr" Rabinowitz'

Dozens of directives can be applied to SWISH-E configuration files. An exhaustive reference can be found in the SWISH-E documentation.

The Index

Each SWISH-E index is stored in a pair of files. One is named as specified in the IndexFile directive, and the other is called indexname.prop. When talking about a SWISH-E index, we mean this pair of files.

The indices can get large. In our example index of HTML files, the index occupies about 11MB, about one-fourth the size of the original files indexed.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Re: How to Index Anything

Mengucino's picture

In this page you will find the way to do this. Also it has a lot of tips for webmaster and search engines.

Bye

Megucino from
______

Re: How to Index Anything

Anonymous's picture

Does anyone know how to indexing a Dinamic page? such as a php/jsp page?

Re: How to Index Anything

Vuelos's picture

For PHP you don´t have to do nothing expecial. The server will return to the bot HTML code not PHP or JSP code. The problem for a robot is to know if a page for example /pages.php?id=23 is diferent from /pages.php?id=24. The robot can´t index every page with different parameters so it must to implement an algorithm that allow to determine if pages are similar or equal and in this case it shouldn´t be indexed.

Re: How to Index Anything

augur's picture

Does anyone know how to indexing a Dynamic page? such as a php/jsp page?

If you spider the site (i.e. -S http) then you don't need to do anything special as long as the PHP/JSP code results in Text, HTML, or XML.
If you use FS method then, at least for PHP, you can have SWISH-E use the PHP cgi executable to process each document into Text, HTML, or XML. In your index configuration add something like this:

IndexContents HTML* .php
FileFilter .php /usr/bin/php "-q '%p'"

Can Swish deal separately with META?

Anonymous's picture

Can Swish deal separately with the META element? It would be very useful to be able to search arbitrary metadata such as authors, keywords or abstracts.

Re: Can Swish deal separately with META?

Anonymous's picture

yes, SWISH-E will automatically parse META tags in HTML/XML docs,
as per the current SWISH-E 2.4.0 documentation here.

SMAN Project RELEASED: search on man pages

Anonymous's picture

Hello All,
The SMAN project, is now publicly available from
http://joshr.com/src/sman.

SMAN is an enhanced version of the unix standbys 'man -k' and 'apropos,' as discussed in Josh Rabinowitz's "How To Index Anything" article in the July 2003 issue of Linux Journal.

Please test it out and let Josh know what you think!

From the SMAN README:

Sman is the Searcher for Man pages. Based on the example of the
same name in Josh Rabinowitz's article "How To Index Anything"
in the July, 2003 issue of Linux Journal
(http://www.linuxjournal.com/article.php?sid=6652), sman is
an enhanced version of 'apropos' and 'man -k'. Sman adds
several key abilities over its predecessors:

* Supports complex natural language text searches such as
"(linux and kernel) or (mach and microkernel)"

* Shows results in a ranked order

* Allows for searches by manpage section, title,
body, or filename

* Uses a prebuilt index to perform fast searches

* Performs 'stemming' so that a search for "searches"
will match a document with the word "searching"

Again, SMAN is available from available from
http://joshr.com/src/sman.

Posted on Tuesday, July 01, 2003?

Anonymous's picture

Posted on Tuesday, July 01, 2003?

Re: Posted on Tuesday, July 01, 2003?

Anonymous's picture

I tried the man page index example and got errors when I entered

swish-e -c sman-index.conf -S prog

I got many warnings like this:

Warning: Unknown header line: ...

Here are the first few and the last couple:

$ swish-e -c sman-index.conf -S prog
Indexing Data Source: "External-Program"
Indexing "./sman-index-prog.pl"
10373 man pages to index...

Warning: Unknown header line: 'll>' from program ./sman-index-prog.pl
:385: warning [p 2, 9.8i]: can't break line
:391: warning [p 2, 10.8i]: can't break line
:399: warning [p 3, 0.8i]: can't break line

Warning: Unknown header line: 'ntains spaces.' from program ./sman-index-prog.pl

Warning: Unknown header line: 'Tcl 8.1 Tcl(n)' from program ./sman-index-prog.pl

Warning: Unknown header line: '' from program ./sman-index-prog.pl

[snip]

Warning: Unknown header line: '>' from program ./sman-index-prog.pl

Warning: Unknown header line: '>' from program ./sman-index-prog.pl
err: External program failed to return required headers Path-Name: & Content-Length:
.

Re: setenv LANG C to work around UTF issues

Anonymous's picture

I was able to get around this by setting the environment variable LANG to "C" like this (adjust for your shell);

setenv LANG C

I think this only needs to be done before indexing with sman-update, and not for sman itself.

Re: Posted on Tuesday, July 01, 2003?

Anonymous's picture

The author says the code was tested on RH6.2, RH7.3, and Debian Woody. Maybe you made a typo, or you have multibyte man pages on your system (which the article and code mention that SWISH-E will gak on?)

I just tried the sman example above and it worked for me on RH6.2:

% swish-e -c sman-index.conf -S prog
Indexing Data Source: "External-Program"
Indexing "./sman-index-prog.pl"
4803 man pages to index...
processing 20
....

There's an enhanced version of SMAN in development at http://joshr.com/src/sman. This version should work better, since it's not shortened to fit in an article.

Re: swan from joshr.com still gives errors

Anonymous's picture

# rpm -q libxml2
libxml2-2.5.4-1
# uname -a
Linux localhost 2.4.20-8 #1 Thu Mar 13 17:54:28 EST 2003 i686 i686 i386 GNU/Linux

# sman-update --verbose --warn --debug

[snip maybe valuable information ?]

**==== END XML of /usr/share/man/mann/Tcl.n.gz =========

** working on /usr/share/man/mann/after.n.gz
** Running man /usr/share/man/mann/after.n.gz...

Warning: Unknown header line: 'd, even if the vari-' from program stdin

Warning: Unknown header line: 'able

Re: swan from joshr.com still gives errors

Anonymous's picture

Some people report that setting the environment variable LANG=en_US might help this issue. I've also heard that a new release of sman is coming that will make it easier to pinpoint the source of errors like this. Let us know if this works!

Re: sman from joshr.com still gives errors

Anonymous's picture

There is a new release at http://joshr.com/src/sman .
Please let us know if this solves your problem.

Sman rocks! Works for me!

Anonymous's picture

I've been using sman for a while on my systems with no problems. It even works on OS X now. There's a new version at http://joshr.com/src/sman

Sman Rocks, and it's on CPAN and Freshmeat

Anonymous's picture

You can now find the latest versions of Sman on Freshmeat at
http://freshmeat.net/projects/sman/
and on CPAN at
http://search.cpan.org/~joshr/Sman/

Re: How to Index Anything

Anonymous's picture

This is cool. But how does Google or Yahoo read through any file type for content. I have done searches for linux and these sites have returned pdf, word, html, excel, powerpoint, text, and even an microsoft project file. How can these sites run such massive searches?

Re: How to Index Anything

Anonymous's picture

Google is massive array of computers. That's why it is fast.
They have filters for those types. As long as you have filters, you can do it too.

sure, swish-e is not google,

Anonymous's picture

sure, swish-e is not google, and never will be, but

it can also index MsWord, OpenOffice, PDF, RTF (apart from standard xml, html, txt) - PPT filter is also available by now (see swish-e.org).

and last, but not least - I run swish-e on Windows and Linux too, almost everything described in this good article is possible with Windows version of swish-e (yes, you don't have man pages there :)

cheers

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState