Gnu Queue: Linux Clustering Made Easy
All of this sounds great, you say. How do I obtain and install GNU Queue? You can download the latest release of GNU Queue from its web site at http://www.gnuqueue.org/. It's a participating project on SourceForge, and you can find all sorts of discussion forums, support forums and bug-tracking databases. Download the program from the web site, unpack it and then run:
./configure<\n> make install
from the top-level directory. Run make install and fire up the dæmon with queued -D & on each machine in your cluster.
For a quick reference on using the queue command to farm jobs out to the network, visit the GNU Queue home page. That's all there is to it!
Before installing GNU Queue on your cluster, you have to make a decision that is basically guided by whether you have root (administrative) privileges on your cluster. If you do, you'll probably want to install GNU Queue in a manner that makes it available to all the users on your site. This is the --enable-root option. On the other hand, if you're just an ordinary Jane or Joe on your cluster or want to see what the fuss is all about without giving away privileges, you can install GNU Queue as an ordinary user, the default mode of installation.
Yes, ordinary users can install GNU Queue as a batch processing system on your cluster! But, if another user wants to run GNU Queue, he'll have to change the port numbers in the source code to insure no one else is running GNU Queue. That's why it's better to let the system administrator install GNU Queue (with --enable-root option to the configure script) if you expect a lot of users will want to run GNU Queue on your cluster.
Once you've downloaded GNU Queue off the Net, the first thing to do is to unpack it using the tar command. Under Linux, this is just tar xzf filename, where file name is the name of the file (compressed with gzip and having either the .tar.gz or .tgz file extensions. On other systems it's a little bit more involved, since the tar installed by default is not GNU tar and doesn't support the zdecompression option. You'll need to explicitly run the gunzip decompression program: gunzip filename.tar.gz; tar xf filename.tar, where filename.tar.gz is the file, with .tar.gz extension, that you obtained from the network. (Savvy users might want to use the zcat filename.tar.gz|tar tf - trick, but this assumes the zcat program installed on your system can handle GNU zipped file. gunzip is part of the GNU gzip package; you can obtain it from ftp://ftp.gnu.org/.
So you've unpacked the distribution and you're sitting in the distribution's top-level directory. Now what? Well, if you're an ordinary Jane or Joe you install the program into the distribution directory by running ./configure followed by make install on each machine in your cluster. Then, fire up the dæmon with queued -D & on each machine in your cluster. If you want more details (or you're a system administrator), continue reading.
Run ./configure. If you're installing it on a system where you're not a superuser but an ordinary peon, configure sets the makefile to install GNU Queue into the current directory. queue will go into ./bin; queued dæmon will go into ./sbin; ./com/queue will be the shared spool directory; the host access control list file will go into ./share; and the queued pid files will go into ./var . If you want things to go somewhere else, run ./configure --prefix=dir, where dir is the top-level directory where you want things to be installed.
The default ./configure option is to install GNU Queue in the local directory for use by a single user only. System administrators should run the command ./configure --enable-root instead. When installing with the --enable-root option, configure sets the makefile to install GNU Queue under the /usr/local prefix. queue will go in /usr/local/bin; queued dæmon will go into /usr/local/sbin; /usr/local/com/queue will be the shared spool directory; the host access control list file will go into /usr/local/share; and the queued pid files will go into /usr/local/var. If you want things to go somewhere else, run the following:
./configure --enable-root<\n> --prefix=dir
where dir is the top-level directory where you want things to be installed.
./configure takes a number of additional options that you may wish to be aware of, including options for changing the paths of the various directories. ./configure --help gives a full listing of them. Here are a few examples, --bindir specifies where queue goes; --sbindir specifies where queued goes; --localstatedir states where the spool directory and queued pid files go; and --datadir lists where the host access control file goes. If ./configure fails inelegantly, make sure lex is installed. GNU flex is an implementation of lex available from the FSF, http://www.gnu.org/.
Now, run make to compile the programs. If your make complains about a syntax error in the Makefile, you'll need to run GNU Make which is hopefully already installed on your machine (perhaps as gmake or gnumake), but, if not, you can obtain it from the FSF at http://www.gnu.org/.
If all goes well, make install will install the programs into the directory you specified with ./configure. Missing directories will be created. The host name of the node make install is being run on will be added to the host access control list if it is not already there.
Now, try running Queue. Start up ./queued -D & on the local machine. (If you did a make install on the node, the host name should already be in the host access control list file.)
|Using Salt Stack and Vagrant for Drupal Development||May 20, 2013|
|Making Linux and Android Get Along (It's Not as Hard as It Sounds)||May 16, 2013|
|Drupal Is a Framework: Why Everyone Needs to Understand This||May 15, 2013|
|Home, My Backup Data Center||May 13, 2013|
|Non-Linux FOSS: Seashore||May 10, 2013|
|Trying to Tame the Tablet||May 08, 2013|
- RSS Feeds
- Making Linux and Android Get Along (It's Not as Hard as It Sounds)
- Using Salt Stack and Vagrant for Drupal Development
- New Products
- Validate an E-Mail Address with PHP, the Right Way
- Drupal Is a Framework: Why Everyone Needs to Understand This
- A Topic for Discussion - Open Source Feature-Richness?
- Download the Free Red Hat White Paper "Using an Open Source Framework to Catch the Bad Guy"
- Tech Tip: Really Simple HTTP Server with Python
- New Products
- Android is Linux -- why no better inter-operation
1 hour 47 min ago
- Connecting Android device to desktop Linux via USB
2 hours 15 min ago
- Find new cell phone and tablet pc
3 hours 13 min ago
4 hours 42 min ago
- Automatically updating Guest Additions
5 hours 51 min ago
- I like your topic on android
6 hours 37 min ago
- Reply to comment | Linux Journal
6 hours 59 min ago
- This is the easiest tutorial
13 hours 13 min ago
- Ahh, the Koolaid.
18 hours 52 min ago
- git-annex assistant
1 day 51 min ago
Enter to Win an Adafruit Pi Cobbler Breakout Kit for Raspberry Pi
It's Raspberry Pi month at Linux Journal. Each week in May, Adafruit will be giving away a Pi-related prize to a lucky, randomly drawn LJ reader. Winners will be announced weekly.
Fill out the fields below to enter to win this week's prize-- a Pi Cobbler Breakout Kit for Raspberry Pi.
Congratulations to our winners so far:
- 5-8-13, Pi Starter Pack: Jack Davis
- 5-15-13, Pi Model B 512MB RAM: Patrick Dunn
- 5-21-13, Prototyping Pi Plate Kit: Philip Kirby
- Next winner announced on 5-27-13!
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?