Use SSH to Cross a Suspect Host Securely

SSH tunnels can be used to connect securely to a server behind a firewall in such a way that the firewall can't observe the data. This is also true and useful if you know an attacker has gained control of the firewall.

Recently at our local (Phoenix) Free Software Stammtisch, we were talking about security. I was surprised to find that no one else realized you can ssh safely across a compromised host to another machine behind it. A common example of “crossing” a host to get to another machine is accessing a machine on a private network protected by a firewall. The firewall is on the edge of the private network and provides the only means to get to the private network. So, you have to pass through the firewall to get to the private network.

But, what happens if the firewall has (or you believe it has) been compromised? How do you get to the private network without more security problems? It turns out that the answer is the same. You go through the firewall, but you do it in such a way that your connection remains secure even when the firewall itself may no longer be. The connection to the machine on the private network still can be made securely, even if you know the host you're passing through has been compromised and some rogue element has taken control of it.

As an example, let's say you need to connect from your Netbook to a server on your work's network. Let's also say the only way to get to your server is via a connection to your work's SSH/VPN gateway, but you think someone has broken into the gateway (yes, yes, get it off-line as soon as possible, but let's do that, and continue to be able to work and recover the gateway).

Now, let's consider an example scenario with three machines, named corresponding to what they are: Netbook, Gateway and Server. If you already understand SSH tunneling and want the short story, see the Short Story sidebar.

For the long story, let's start with a description of some simple tunneling. The -L option on the command line allows you to create a tunnel:

ssh -N -f -L 2222:localhost:22 Gateway

The -L option allows you to specify the entry and exit points for a secure tunnel created by SSH. In the examples used in this article, it gets an argument that has three fields. The fields are separated by colons. The first field is the local port to bind (the local port to listen on). The next field is the host to connect to. This field is interpreted by the remote machine in the SSH connection, not by the local machine. The computer initiating the SSH connection doesn't need to know how to get to it. The third field is the port to connect to on the far end of the tunnel.

If you run this first command from Netbook, the command creates an SSH connection to Gateway and builds a tunnel that forwards port 2222 on Netbook to port 22 (the standard SSH port) on Gateway. The local port, 2222, can be almost any value as long as it's not already in use, although it must be above 1023 if you're not connecting as root. Now you can connect to SSH on Gateway by using that tunnel. Again, it's important to note that the tunnel destination hostname (the second field) is interpreted by Gateway, so “localhost” is instructing Gateway to connect to itself. localhost is not Netbook; it's Gateway. For more examples of using -L to create tunnels, see the SSH -L Examples sidebar.

The -N and -f options are useful when just creating a tunnel and not needing a remote shell. The -N option tells SSH not to run a remote command. The -f option puts the ssh command in the background, so you can regain control of the shell that ran the ssh command.

Now, run the following command on Netbook:

ssh -p 2222 localhost

This second command creates an SSH connection to port 22 on Gateway by using the tunnel created in the first command. In the second command, localhost is the destination machine, so it's interpreted by the SSH client on Netbook, which means the ssh command running on Netbook connects to Netbook but on a nonstandard port. The tunnel is entered at port 2222 on Netbook and comes out at port 22 on Gateway. Because sshd is listening on port 22, the tunnel connects to the SSH dæmon on Gateway.

Presuming Gateway can connect to Server even though Netbook can't (remember Server is “firewalled” behind Gateway), it's also possible to create a tunnel to Server using a command such as:

ssh -a -N -f -L 3333:Server:22 Gateway

In this case, the tunnel from port 3333 on Netbook doesn't connect to Gateway, rather it connects to port 22 on Server. Gateway is essentially forwarding packets from Netbook to Server. Like the first command, the tunnel destination (the second field, Server) is interpreted by Gateway, so this tunnel connects to Server. Netbook doesn't need to know how to get to Server; Gateway handles the routing.

The -a option here makes sure authentication agent connections are not forwarded to Gateway. If you are concerned that Gateway is compromised, you don't want the attacker to gain control of your authentication agent connections. For more, see the Authentication Agents sidebar.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

"you can ssh safely across a

Anonymous's picture

"you can ssh safely across a compromised host"

You keep using that word. I do not think it means what you think it means.

Not so troubling

lufthans's picture

WARNING: Anything communicated to authenticate to a compromized host could potentially be captured by the attacker and used against other hosts.

As the author, I do not mean for my article to dismiss the importance of shutting down a compromised host. The article explains that you might need to temporarily hop through the compromized host and describes how to do so safely.

I agree in pulling a machine off the air and even specify doing so in the 3rd paragraph. As a sysadmin I realize there are often steps to take before doing so. This article describes a way to safely connect to the internal network if needed, even if it's not the best position to be in.

The authentication credentials for the bastion host and the chewy center of your network should be different :).

I warn against forwarding agent connections through the compromized host. I also cover safely verifying the internal server's host key.

I see that I did leave out specific warnings about authentication credential ( such as personal key and password ) reuse. You are correct, if they are the same on both Gateway and Server the attacker could gain the credentials when you connect with Gateway and then use them on Server. I should have covered that in the article.

WARNING: Anything communicated to authenticate to the Gateway could potentially be captured by the attacker and used against other hosts.

Better?

The article does not pretend there are no security risks at all and specifically warns against a couple.

I generally believe bastion hosts on the edge of the network should be treated as if they could be compromized any minute. The methods in this article allow access to network resources without giving the bastion host access to those resources even if it's the gateway to reach them.

Very Troubling Article

Anonymous's picture

This article does a fine job of explaining ssh tunneling. However, I am troubled by two issues with the technique suggested by the author.

First, that the author simply dismisses the importance of immediately removing a suspect host from the network. This is a red flag to be highly skeptical of any security advice to follow. Best practices are ignored at one's own peril.

Second, the omission of any discussion of the risks of establishing an ssh connection to the compromised host.

Yes, the tunneled ssh connection is not much different than an ssh session crossing unknown internet hosts. However in order to create that tunnel, one must first authenticate to the compromised host. This risks of this cannot be understated.

Well known are many rootkits which alter or replace sshd in very nasty ways. At a minimum, you are handing over your authentication credentials. By establishing a tunnel to an internal host you are potentially alerting your attacker to another target of interest which may accept those credentials.

The entire tone of the article presents the author's tunneling technique as if it involved no security risks at all. This is simply not true. The security best practice in this case is well established: Never open a network login to a compromised host.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix