Use SSH to Cross a Suspect Host Securely
Recently at our local (Phoenix) Free Software Stammtisch, we were talking about security. I was surprised to find that no one else realized you can ssh safely across a compromised host to another machine behind it. A common example of “crossing” a host to get to another machine is accessing a machine on a private network protected by a firewall. The firewall is on the edge of the private network and provides the only means to get to the private network. So, you have to pass through the firewall to get to the private network.
But, what happens if the firewall has (or you believe it has) been compromised? How do you get to the private network without more security problems? It turns out that the answer is the same. You go through the firewall, but you do it in such a way that your connection remains secure even when the firewall itself may no longer be. The connection to the machine on the private network still can be made securely, even if you know the host you're passing through has been compromised and some rogue element has taken control of it.
As an example, let's say you need to connect from your Netbook to a server on your work's network. Let's also say the only way to get to your server is via a connection to your work's SSH/VPN gateway, but you think someone has broken into the gateway (yes, yes, get it off-line as soon as possible, but let's do that, and continue to be able to work and recover the gateway).
Now, let's consider an example scenario with three machines, named corresponding to what they are: Netbook, Gateway and Server. If you already understand SSH tunneling and want the short story, see the Short Story sidebar.
Short Story
The short description is that you initiate a connection to Gateway. With that connection, you create a tunnel to port 22 on Server using the -L option to ssh:
-L $local_port:Server:22
You then can connect to a local port on Netbook that is the entry point for a tunnel that comes out at port 22 of the destination machine, which is Server. The tunneled connection is never decrypted on Gateway, so it stays secure.
For the long story, let's start with a description of some simple tunneling. The -L option on the command line allows you to create a tunnel:
ssh -N -f -L 2222:localhost:22 Gateway
The -L option allows you to specify the entry and exit points for a secure tunnel created by SSH. In the examples used in this article, it gets an argument that has three fields. The fields are separated by colons. The first field is the local port to bind (the local port to listen on). The next field is the host to connect to. This field is interpreted by the remote machine in the SSH connection, not by the local machine. The computer initiating the SSH connection doesn't need to know how to get to it. The third field is the port to connect to on the far end of the tunnel.
If you run this first command from Netbook, the command creates an SSH connection to Gateway and builds a tunnel that forwards port 2222 on Netbook to port 22 (the standard SSH port) on Gateway. The local port, 2222, can be almost any value as long as it's not already in use, although it must be above 1023 if you're not connecting as root. Now you can connect to SSH on Gateway by using that tunnel. Again, it's important to note that the tunnel destination hostname (the second field) is interpreted by Gateway, so “localhost” is instructing Gateway to connect to itself. localhost is not Netbook; it's Gateway. For more examples of using -L to create tunnels, see the SSH -L Examples sidebar.
The -N and -f options are useful when just creating a tunnel and not needing a remote shell. The -N option tells SSH not to run a remote command. The -f option puts the ssh command in the background, so you can regain control of the shell that ran the ssh command.
SSH -L Examples
The following command connects to remote_host and builds a tunnel from port 8888 of your desktop to port 80 of my Web server. You then can point a Web browser at http://localhost:8888/ to read my home page:
ssh -L 8888:www.LuftHans.com:80 remote_host
The following command connects to remote_host and builds a tunnel from port 9993 of your desktop to port 993 of mail_server's IMAP over SSL server. You then can configure your mail client to connect to port 9993 on your local system to read your mail:
ssh -L 9993:mail_server:993 remote_host
In both cases, the remote servers see the connection coming from remote_host, and in both cases, it doesn't matter whether your desktop can talk directly to the server at the far end of the tunnel.
Now, run the following command on Netbook:
ssh -p 2222 localhost
This second command creates an SSH connection to port 22 on Gateway by using the tunnel created in the first command. In the second command, localhost is the destination machine, so it's interpreted by the SSH client on Netbook, which means the ssh command running on Netbook connects to Netbook but on a nonstandard port. The tunnel is entered at port 2222 on Netbook and comes out at port 22 on Gateway. Because sshd is listening on port 22, the tunnel connects to the SSH dæmon on Gateway.
Presuming Gateway can connect to Server even though Netbook can't (remember Server is “firewalled” behind Gateway), it's also possible to create a tunnel to Server using a command such as:
ssh -a -N -f -L 3333:Server:22 Gateway
In this case, the tunnel from port 3333 on Netbook doesn't connect to Gateway, rather it connects to port 22 on Server. Gateway is essentially forwarding packets from Netbook to Server. Like the first command, the tunnel destination (the second field, Server) is interpreted by Gateway, so this tunnel connects to Server. Netbook doesn't need to know how to get to Server; Gateway handles the routing.
The -a option here makes sure authentication agent connections are not forwarded to Gateway. If you are concerned that Gateway is compromised, you don't want the attacker to gain control of your authentication agent connections. For more, see the Authentication Agents sidebar.
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Sponsored by AMD
Built-in forensics, incident response, and security with Red Hat Enterprise Linux 6
Every security policy provides guidance and requirements for ensuring adequate protection of information and data, as well as high-level technical and administrative security requirements for a system in a given environment. Traditionally, providing security for a system focuses on the confidentiality of the information on it. However, protecting the data integrity and system and data availability is just as important. For example, when processing United States intelligence information, there are three attributes that require protection: confidentiality, integrity, and availability.
Learn more about catching the bad guy in this free white paper.
Sponsored by DLT Solutions
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?
| Designing Electronics with Linux | May 22, 2013 |
| Dynamic DNS—an Object Lesson in Problem Solving | May 21, 2013 |
| Using Salt Stack and Vagrant for Drupal Development | May 20, 2013 |
| Making Linux and Android Get Along (It's Not as Hard as It Sounds) | May 16, 2013 |
| Drupal Is a Framework: Why Everyone Needs to Understand This | May 15, 2013 |
| Home, My Backup Data Center | May 13, 2013 |
- New Products
- Linux Systems Administrator
- Senior Perl Developer
- Technical Support Rep
- UX Designer
- Web & UI Developer (JavaScript & j Query)
- Designing Electronics with Linux
- Dynamic DNS—an Object Lesson in Problem Solving
- Using Salt Stack and Vagrant for Drupal Development
- Making Linux and Android Get Along (It's Not as Hard as It Sounds)
- another very interesting
1 hour 32 min ago - Reply to comment | Linux Journal
3 hours 26 min ago - Reply to comment | Linux Journal
10 hours 20 min ago - Reply to comment | Linux Journal
10 hours 36 min ago - Favorite (and easily brute-forced) pw's
12 hours 27 min ago - Have you tried Boxen? It's a
18 hours 19 min ago - seo services in india
22 hours 51 min ago - For KDE install kio-mtp
22 hours 51 min ago - Evernote is much more...
1 day 51 min ago - Reply to comment | Linux Journal
1 day 9 hours ago




Comments
"you can ssh safely across a
"you can ssh safely across a compromised host"
You keep using that word. I do not think it means what you think it means.
Not so troubling
WARNING: Anything communicated to authenticate to a compromized host could potentially be captured by the attacker and used against other hosts.
As the author, I do not mean for my article to dismiss the importance of shutting down a compromised host. The article explains that you might need to temporarily hop through the compromized host and describes how to do so safely.
I agree in pulling a machine off the air and even specify doing so in the 3rd paragraph. As a sysadmin I realize there are often steps to take before doing so. This article describes a way to safely connect to the internal network if needed, even if it's not the best position to be in.
The authentication credentials for the bastion host and the chewy center of your network should be different :).
I warn against forwarding agent connections through the compromized host. I also cover safely verifying the internal server's host key.
I see that I did leave out specific warnings about authentication credential ( such as personal key and password ) reuse. You are correct, if they are the same on both Gateway and Server the attacker could gain the credentials when you connect with Gateway and then use them on Server. I should have covered that in the article.
WARNING: Anything communicated to authenticate to the Gateway could potentially be captured by the attacker and used against other hosts.
Better?
The article does not pretend there are no security risks at all and specifically warns against a couple.
I generally believe bastion hosts on the edge of the network should be treated as if they could be compromized any minute. The methods in this article allow access to network resources without giving the bastion host access to those resources even if it's the gateway to reach them.
Very Troubling Article
This article does a fine job of explaining ssh tunneling. However, I am troubled by two issues with the technique suggested by the author.
First, that the author simply dismisses the importance of immediately removing a suspect host from the network. This is a red flag to be highly skeptical of any security advice to follow. Best practices are ignored at one's own peril.
Second, the omission of any discussion of the risks of establishing an ssh connection to the compromised host.
Yes, the tunneled ssh connection is not much different than an ssh session crossing unknown internet hosts. However in order to create that tunnel, one must first authenticate to the compromised host. This risks of this cannot be understated.
Well known are many rootkits which alter or replace sshd in very nasty ways. At a minimum, you are handing over your authentication credentials. By establishing a tunnel to an internal host you are potentially alerting your attacker to another target of interest which may accept those credentials.
The entire tone of the article presents the author's tunneling technique as if it involved no security risks at all. This is simply not true. The security best practice in this case is well established: Never open a network login to a compromised host.