Gnu Queue: Linux Clustering Made Easy

Farm those jobs out with Gnu queue!
Obtaining and Installing GNU Queue: A Quick Look

All of this sounds great, you say. How do I obtain and install GNU Queue? You can download the latest release of GNU Queue from its web site at http://www.gnuqueue.org/. It's a participating project on SourceForge, and you can find all sorts of discussion forums, support forums and bug-tracking databases. Download the program from the web site, unpack it and then run:

 ./configure<\n>
 make install

from the top-level directory. Run make install and fire up the dæmon with queued -D & on each machine in your cluster.

For a quick reference on using the queue command to farm jobs out to the network, visit the GNU Queue home page. That's all there is to it!

Detailed Instructions on Installing GNU Queue

Before installing GNU Queue on your cluster, you have to make a decision that is basically guided by whether you have root (administrative) privileges on your cluster. If you do, you'll probably want to install GNU Queue in a manner that makes it available to all the users on your site. This is the --enable-root option. On the other hand, if you're just an ordinary Jane or Joe on your cluster or want to see what the fuss is all about without giving away privileges, you can install GNU Queue as an ordinary user, the default mode of installation.

Yes, ordinary users can install GNU Queue as a batch processing system on your cluster! But, if another user wants to run GNU Queue, he'll have to change the port numbers in the source code to insure no one else is running GNU Queue. That's why it's better to let the system administrator install GNU Queue (with --enable-root option to the configure script) if you expect a lot of users will want to run GNU Queue on your cluster.

Once you've downloaded GNU Queue off the Net, the first thing to do is to unpack it using the tar command. Under Linux, this is just tar xzf filename, where file name is the name of the file (compressed with gzip and having either the .tar.gz or .tgz file extensions. On other systems it's a little bit more involved, since the tar installed by default is not GNU tar and doesn't support the zdecompression option. You'll need to explicitly run the gunzip decompression program: gunzip filename.tar.gz; tar xf filename.tar, where filename.tar.gz is the file, with .tar.gz extension, that you obtained from the network. (Savvy users might want to use the zcat filename.tar.gz|tar tf - trick, but this assumes the zcat program installed on your system can handle GNU zipped file. gunzip is part of the GNU gzip package; you can obtain it from ftp://ftp.gnu.org/.

So you've unpacked the distribution and you're sitting in the distribution's top-level directory. Now what? Well, if you're an ordinary Jane or Joe you install the program into the distribution directory by running ./configure followed by make install on each machine in your cluster. Then, fire up the dæmon with queued -D & on each machine in your cluster. If you want more details (or you're a system administrator), continue reading.

Installation by Plain Folks

Run ./configure. If you're installing it on a system where you're not a superuser but an ordinary peon, configure sets the makefile to install GNU Queue into the current directory. queue will go into ./bin; queued dæmon will go into ./sbin; ./com/queue will be the shared spool directory; the host access control list file will go into ./share; and the queued pid files will go into ./var . If you want things to go somewhere else, run ./configure --prefix=dir, where dir is the top-level directory where you want things to be installed.

System-Wide Installation by Superusers

The default ./configure option is to install GNU Queue in the local directory for use by a single user only. System administrators should run the command ./configure --enable-root instead. When installing with the --enable-root option, configure sets the makefile to install GNU Queue under the /usr/local prefix. queue will go in /usr/local/bin; queued dæmon will go into /usr/local/sbin; /usr/local/com/queue will be the shared spool directory; the host access control list file will go into /usr/local/share; and the queued pid files will go into /usr/local/var. If you want things to go somewhere else, run the following:

./configure --enable-root<\n>
--prefix=dir

where dir is the top-level directory where you want things to be installed.

./configure takes a number of additional options that you may wish to be aware of, including options for changing the paths of the various directories. ./configure --help gives a full listing of them. Here are a few examples, --bindir specifies where queue goes; --sbindir specifies where queued goes; --localstatedir states where the spool directory and queued pid files go; and --datadir lists where the host access control file goes. If ./configure fails inelegantly, make sure lex is installed. GNU flex is an implementation of lex available from the FSF, http://www.gnu.org/.

Now, run make to compile the programs. If your make complains about a syntax error in the Makefile, you'll need to run GNU Make which is hopefully already installed on your machine (perhaps as gmake or gnumake), but, if not, you can obtain it from the FSF at http://www.gnu.org/.

If all goes well, make install will install the programs into the directory you specified with ./configure. Missing directories will be created. The host name of the node make install is being run on will be added to the host access control list if it is not already there.

Now, try running Queue. Start up ./queued -D & on the local machine. (If you did a make install on the node, the host name should already be in the host access control list file.)

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

cluster

abner's picture

me podria ayudar a configurar un cluster en linux, los pasos a seguir

alternative to GNU Queue

Roger's picture

Beware that this is a pre-alpha release. The original GNU Queue code hadn't been supported in some time - the project was restarted last summer with a new re-write which is this current release. I would therefore be leery using this current code for anything more than experimentation, certainly not in a production environment.

An alternative, also free and open source, is Sun's Grid Engine project (http://gridengine.sunsource.net/). It works on Linux, Solaris, Windows, Mac OS X, and probably most versions of Unix. Grid Engine is probably an order of magnitude more complex to configure than GNU Queue but it has many more capabilities. For your home computer network it is undoubtedly overkill but for any other environment it is seriously worth considering.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix