Hack and / - Linux Troubleshooting, Part II: Local Network

If bill is responding to ping, the next step is to test whether port 25 is even open on bill. There are a few different methods for doing this, but telnet is one of the easiest and is likely already to be installed on your machines. Let's assume bill has an IP of 10.1.1.17; I would type:

$ telnet 10.1.1.17 25
Trying 10.1.1.17...
telnet: Unable to connect to remote host: Connection refused

If telnet doesn't complain about Connection refused, but instead starts outputting SMTP commands, then congratulations, you don't have a networking problem! On the downside, this means you probably have some sort of SMTP problem, which might be more of a pain to troubleshoot. If telnet complains with Connection refused, either port 25 is down on the remote machine (possibly the SMTP service on bill isn't running or isn't listening on that port), or a firewall is blocking you. This is where a tool like nmap can be handy, and it's one of the reasons I often use nmap instead of telnet when I want to test whether a port is available.

You see, many firewalls are configured to block ports by dropping packets with no reply. Because normally a server would send a basic reply back to let you know the port is closed, if the packet is dropped instead, nmap will flag it as filtered instead of closed:

$ nmap -p 25 10.1.1.17

Starting Nmap 5.00 ( http://nmap.org ) at 2010-01-04 20:20 PST
Interesting ports on 10.1.1.17:
PORT   STATE  SERVICE
25/tcp filtered smtp

In this case, nmap says the port is filtered, which tells me there is a firewall blocking this port. If these machines were on different subnets, there might be a firewall in between the networks restricting access. Because I know these machines are on the same subnet, I would assume that there is some iptables firewall configured on bill that needs to be checked.

Let's assume we think the problem is on bill. After I've performed the same network troubleshooting on bill that I have on shawn, the next step is to log in to bill and test whether port 25 is open and listening for connections. For this, I will use the netstat tool. netstat can be used to output all sorts of information about network connections on the machine. In this case though, I will just use the -lnp options to list listening ports and the processes that have the ports open, then I will grep for the port I'm interested in, port 25:

$ sudo netstat -lnp | grep :25
tcp        0      0 0.0.0.0:25    0.0.0.0:*     LISTEN   1878/master

The column I want to pay the most attention to here is the fourth column that lists what local address is open on port 25. In this case, I can see it is set to 0.0.0.0:25, which means bill is listening to port 25 connections on all available interfaces. If I had set up the mail server to listen only on eth0, this would be set to 10.1.1.17:25. If, on the other hand, I saw this was set to 127.0.0.1:25, I might have found the cause of the problem: the mail server was set to listen only to the localhost address (127.0.0.1) and isn't listening for any connections from the outside network. In that situation, I would reconfigure my mail server so that it listens on eth0. If I got no output from the above command, I would know my problem is that my server isn't running at all (or isn't set to listen on port 25). Then, I'd need to start my mail server and troubleshoot why it stopped running to begin with, or why it isn't listening on the right port.

As you can see, network troubleshooting can lead you in all sorts of interesting directions. Even now I've barely scratched the surface. In my next column, I'll extend network troubleshooting beyond the local network and touch on how to track down routing and DNS problems from your local networks to the Internet itself.