OpenSSH is a standard part of the system administrator's toolbox. Although sshd, the OpenSSH daemon, is versatile, sometimes, you need more than one to meet your security goals.

First of all, there's the emergency scenario. You mess something up, try to restart sshd, and it doesn't come back. That means either go visit the system in person, and it may be a long ways away, or if you have another sshd running, use that to get back in and fix the regular one.

Second, there are security policy considerations. You can always run one sshd and have it listen on two interfaces, but that's probably going to be inappropriate for a "bastion host" system where you want to be as secure as possible for the outward-facing network interface and give more features to people coming in from the inward-facing one.

For example, you might want everyone to be able to ssh to the shell box from their desktop systems, but only allow the sysadmin and webmaster to get in from the outside. Unfortunately, you can't do an AllowUsers that only applies to one interface. If you want two sets of AllowUsers users, you'll need two sshds.

The same is true for PermitRootLogin.

Another simple but useful security measure to protect daemons that send sensitive information as plaintext is to configure them to listen only on the loopback interface, so only localhost can connect to it. What good does this do, if you want to provide services to the network? Simple. By making the daemon bind only to the loopback interface (with Apache's BindAddress or similar configuration directives) you mandate that remote users have to use an ssh tunnel. At the same time, though, you might want to prevent ordinary users from getting a shell on that host.

You can prevent users from setting up ssh port forwarding with AllowTcpForwarding no in sshd_config. (As the man page mentions, this isn't foolproof if the user can install his or her own port forwarding software on the system.) But how do you make it so that a user can only do port forwarding, not log in?

Use the old Un*x trick of setting the user's shell to /bin/false. You can create as many tunnel-only accounts as you like. To allow a user access to one, just copy that user's ssh public key into the tunnel-only account's .ssh/authorized_keys.

The simplest way to have two copies of sshd running is to have two init scripts and two configuration files. Fortunately, this is simple. Just cd to /etc/ssh, or whereever your sshd_config file lives, and cp it to /etc/ssh/extra_sshd_config, or something descriptive like admin_group_sshd_config. In the new file, change either the Port line or the ListenAddress line so your extra sshd won't be on the same port/interface combination as your regular one. (Here, I use port 22222, but you can also use the hopefully unused telnet port, port 23) If your system has two or more network interfaces, you can have two sshd instances, both on Port 22, but different interfaces. You can test this step with sshd -f /etc/ssh/extra_sshd_config. Now make a copy of your ssh init script, which might live in /etc/init.d/ or /etc/rc.d/init.d. Call the copy extra_sshd, or something descriptive like admin_group_sshd. In the copy, add the command-line option -f /etc/ssh/extra_sshd_config to any mention of sshd. Run the script with the argument of "start" and you should be able to do an ssh -p 22222 localhost.

You can now use your distribution's standard tools to add the new init script to the appropriate runlevels, or just do it by hand with ln -s.

Jim Dennis of Linux Gazette has pointed out that on a seriously messed-up system, the "dreaded OOM killer", a kernel feature that kills off processes to try to recover from an out of memory condition, might take out the sshd that gets started from the init script in /etc/init.d/ or /etc/init.d/rc.d. This leaves you with no way to get into the messed-up system that needs it most. The answer is to start a second, emergency sshd, directly from /etc/inittab. If a process started from /etc/iniittab dies, init will restart it.

In order to run sshd directly from inittab, add a line like this: ss:12345:respawn:/usr/sbin/sshd -D -f /etc/ssh/extra_sshd_config

and do a kill -HUP 1 to make init reread /etc/inittab.

Wait a minute! Why the -D for "don't become a daemon?" When a process started from an init script becomes a daemon, it goes through a little dance, in which the original sshd process exits, and one of its descendants runs in the background as a child of init. Since we're starting sshd directly from /etc/inittab, the original process is already a child of init, so we don't need to do this. Doing so will confuse init horribly, when the original process it's keeping track of dies and it thinks it has to keep restarting sshd.

The advantage of this approach is that init will restart the extra sshd if the OOM killer kills it, or if by some chance there's a bug that causes it to die. The latter has never happened to me, but you never know. To stop this started-from-inittab sshd, just comment out the /etc/inittab line and do a kill -HUP 1.

By now you might have some ideas for interesting things you could do if you just had another sshd running on your system. The important thing is to start with the security policy you want to enforce, and use ssh as a tool to help do that, not just tweak it until it works.