Hack and / - Linux Troubleshooting, Part II: Local Network

Last month, I discussed localhost troubleshooting, and this month, I extend troubleshooting to your local network. Find out why shawn can't talk to bill.
Next: Probe bill's Ports

If bill is responding to ping, the next step is to test whether port 25 is even open on bill. There are a few different methods for doing this, but telnet is one of the easiest and is likely already to be installed on your machines. Let's assume bill has an IP of 10.1.1.17; I would type:

$ telnet 10.1.1.17 25
Trying 10.1.1.17...
telnet: Unable to connect to remote host: Connection refused

If telnet doesn't complain about Connection refused, but instead starts outputting SMTP commands, then congratulations, you don't have a networking problem! On the downside, this means you probably have some sort of SMTP problem, which might be more of a pain to troubleshoot. If telnet complains with Connection refused, either port 25 is down on the remote machine (possibly the SMTP service on bill isn't running or isn't listening on that port), or a firewall is blocking you. This is where a tool like nmap can be handy, and it's one of the reasons I often use nmap instead of telnet when I want to test whether a port is available.

You see, many firewalls are configured to block ports by dropping packets with no reply. Because normally a server would send a basic reply back to let you know the port is closed, if the packet is dropped instead, nmap will flag it as filtered instead of closed:

$ nmap -p 25 10.1.1.17

Starting Nmap 5.00 ( http://nmap.org ) at 2010-01-04 20:20 PST
Interesting ports on 10.1.1.17:
PORT   STATE  SERVICE
25/tcp filtered smtp

In this case, nmap says the port is filtered, which tells me there is a firewall blocking this port. If these machines were on different subnets, there might be a firewall in between the networks restricting access. Because I know these machines are on the same subnet, I would assume that there is some iptables firewall configured on bill that needs to be checked.

Test bill Directly

Let's assume we think the problem is on bill. After I've performed the same network troubleshooting on bill that I have on shawn, the next step is to log in to bill and test whether port 25 is open and listening for connections. For this, I will use the netstat tool. netstat can be used to output all sorts of information about network connections on the machine. In this case though, I will just use the -lnp options to list listening ports and the processes that have the ports open, then I will grep for the port I'm interested in, port 25:

$ sudo netstat -lnp | grep :25
tcp        0      0 0.0.0.0:25    0.0.0.0:*     LISTEN   1878/master

The column I want to pay the most attention to here is the fourth column that lists what local address is open on port 25. In this case, I can see it is set to 0.0.0.0:25, which means bill is listening to port 25 connections on all available interfaces. If I had set up the mail server to listen only on eth0, this would be set to 10.1.1.17:25. If, on the other hand, I saw this was set to 127.0.0.1:25, I might have found the cause of the problem: the mail server was set to listen only to the localhost address (127.0.0.1) and isn't listening for any connections from the outside network. In that situation, I would reconfigure my mail server so that it listens on eth0. If I got no output from the above command, I would know my problem is that my server isn't running at all (or isn't set to listen on port 25). Then, I'd need to start my mail server and troubleshoot why it stopped running to begin with, or why it isn't listening on the right port.

As you can see, network troubleshooting can lead you in all sorts of interesting directions. Even now I've barely scratched the surface. In my next column, I'll extend network troubleshooting beyond the local network and touch on how to track down routing and DNS problems from your local networks to the Internet itself.

Kyle Rankin is a Systems Architect in the San Francisco Bay Area and the author of a number of books, including The Official Ubuntu Server Book, Knoppix Hacks and Ubuntu Hacks. He is currently the president of the North Bay Linux Users' Group.

______________________

Kyle Rankin is a director of engineering operations in the San Francisco Bay Area, the author of a number of books including DevOps Troubleshooting and The Official Ubuntu Server Book, and is a columnist for Linux Journal.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix