Sharing Admin Privileges for Many Hosts Securely

The problem: you have a large team of admins, with a substantial turnover rate. Maybe contractors come and go. Maybe you have tiers of access, due to restrictions based on geography, admin level or even citizenship (as with some US government contracts). You need to give these people administrative access to dozens (perhaps hundreds) of hosts, and you can't manage all their accounts on all the hosts.

This problem arose in the large-scale enterprise in which I work, and our team worked out a solution that:

  • Does not require updating accounts on more than one host whenever a team member arrives or leaves.

  • Does not require deletion or replacement of Secure Shell (SSH) keys.

  • Does not require management of individual SSH keys.

  • Does not require distributed sudoers or other privileged-access management tools (which may not be supported by some Linux-based appliances anyway).

  • And most important, does not require sharing of passwords or key passphrases.

It works between any UNIX or Linux platforms that understand SSH key trust relationships. I personally have made use of it on a half-dozen different Linux distros, as well as Solaris, HP-UX, Mac OS X and some BSD variants.

In our case, the hosts to be managed were several dozen Linux-based special-purpose appliances that did not support central account management tools or sudo. They are intended to be used (when using the shell at all) as the root account.

Our environment also (due to a government contract) requires a two-tier access scheme. US citizens on the team may access any host as root. Non-US citizens may access only a subset of the hosts. The techniques described in this article may be extended for N tiers without any real trouble, but I describe the case N == 2 in this article.

The Scenario

I am going to assume you, the reader, know how to set up an SSH trust relationship so that an account on one host can log in directly, with no password prompting, to an account on another. (Basically, you simply create a key pair and copy the public half to the remote host's ~/.ssh/authorized_keys file.) If you don't know how to do this, stop reading now and go learn. A Web search for "ssh trust setup" will yield thousands of links—or, if you're old-school, the AUTHENTICATION section of the ssh(1) man page will do. Also see ssh-copy-id(1), which can greatly simplify the distribution of key files.

Steve Friedl's Web site has an excellent Tech Tip on these basics, plus some material on SSH agent-forwarding, which is a neat trick to centralize SSH authentication for an individual user. The Tech Tip is available at http://www.unixwiz.net/techtips/ssh-agent-forwarding.html.

I describe key-caching below, as it is not very commonly used and is the heart of the technique described herein.

For illustration, I'm assigning names to players (individuals assigned to roles), the tiers of access and "dummy" accounts.

Hosts:

  • darter — the hostname of the central management host on which all the end-user and utility accounts are active, all keys are stored and caching takes place; also, the sudoers file controlling access to utility accounts is here.

  • n1, n2, ... — hostnames of target hosts for which access is to be granted for all team members ("n" for "non-special").

  • s1, s2, ... — hostnames of target hosts for which access is to be granted only to some team members ("s" for "special").

Accounts (on darter only):

  • univ — the name of the utility account holding the SSH keys that all target hosts (u1, u2, ...) will trust.

  • spec — the name of the utility account holding the SSH keys that only special, restricted-access, hosts (s1, s2, ...) will trust.

  • joe — let's say the name of the guy administering the whole scheme is "Joe" and his account is "joe". Joe is a trusted admin with "the keys to the kingdom"—he cannot be a restricted user.

  • andy, amy — these are users who are allowed to log in to all hosts.

  • alice

  • ned, nora — these are users who are allowed to log in only to "n" (non-special) hosts; they never should be allowed to log in to special hosts s1, s2, ...

  • nancy

You will want to create shared, unprivileged utility accounts on darter for use by unrestricted and restricted admins. These (per our convention) will be called "univ" and "rstr", respectively. No one should actually directly log in to univ and rstr, and in fact, these accounts should not have passwords or trusted keys of their own. All logins to the shared utility accounts should be performed with su(1) from an existing individual account on darter.

The Setup

Joe's first act is to log in to darter and "become" the univ account:


$ sudo su - univ

Then, under that shared utility account, Joe creates a .ssh directory and an SSH keypair. This key will be trusted by the root account on every target host (because it's the "univ"-ersal key):


$ mkdir .ssh    # if not already present
$ ssh-keygen -t rsa -b 2048 -C "universal access 
 ↪key gen YYYYMMDD" -f
.ssh/univ_key
   Enter passphrase (empty for no passphrase):

Very important: Joe assigns a strong passphrase to this key. The passphrase to this key will not be generally shared.

(The field after -C is merely a comment; this format reflects my personal preference, but you are of course free to develop your own.)

This will generate two files in .ssh: univ_key (the private key file) and univ_key.pub (the public key file). The private key file is encrypted, protected by the very strong passphrase Joe assigned to it, above.

Joe logs out of the univ account and into rstr. He executes the same steps, but creates a keypair named rstr_key instead of univ_key. He assigns a strong passphrase to the private key file—it can be the same passphrase as assigned to univ, and in fact, that is probably preferable from the standpoint of simplicity.

Joe copies univ_key.pub and rstr_key.pub to a common location for convenience.

For every host to which access is granted for everyone (n1, n2, ...), Joe uses the target hosts' root credentials to copy both univ_key.pub and rstr_key.pub (on separate lines) to the file .ssh/authorized_keys under the root account directory.

For every host to which access is granted for only a few (s1, s2, ...), Joe uses the target hosts' root credentials to copy only rstr_key.pub (on a single line) to the file .ssh/authorized_keys under the root account directory.

______________________

J.D. Baldwin has been a UNIX, Linux and Web user and administrator going back to SunOS 1.1 (1984), Slackware 3.0 (1995) and Apache 1.2 (1997). He currently works in network security for a large multinational company.