Hack and / - Linux Troubleshooting, Part II: Local Network
This column is the second in a series dedicated to one of my favorite subjects: troubleshooting. Because my column is generally aimed more at tips and tricks and less on philosophy and design, I'm not going to talk much about overall approaches to problem solving. Instead, in this series, I describe some general classes of problems you might find on a Linux system, and then I discuss how to use common tools, most of which probably already are on your system, to isolate and resolve each class of problem.
In the first column, I talked about how to diagnose high-load issues on a server, but the fact is that these days, just about every Linux computer is connected to a network, and a large number of the problems you have are based in the network. This month, I focus on local network troubleshooting, and although I am writing from the perspective of servers, most of these steps will apply to any Linux machine on a network. Also, because the goal of this article is to show how to become better at troubleshooting, I list each step from the lowest level on up. In real life, I'd probably skip ahead here and there to make the troubleshooting process faster.
The generic problem I cover here is how to track down the root cause when one machine can't communicate with another machine on the same network. For this example, let's assume I have two servers named bill and shawn. The server shawn is trying to communicate with bill over port 25 (port 25 is used for sending e-mail over SMTP), but wouldn't you know it, bill isn't responding.
One of the first things I might do in a scenario like this is find another machine on the same network and try to connect with bill from there. If I can talk to bill from another machine on the same network, the problem is most likely with shawn or with the network in between shawn and bill. If I have the same problem from another machine on the same network, it's more likely that the problem is with bill, so I would start troubleshooting from there. Just so I can discuss more troubleshooting steps, let's start troubleshooting from shawn.
One of the most embarrassing things in troubleshooting is to waste an hour only to find out that something wasn't plugged in. So the first step I perform is to make sure that shawn is plugged in to the network. Although I could inspect the port physically on the server, if the server were in a different city, I might run a program like ethtool. ethtool gives you a lot of different diagnostics on your Ethernet devices. By default, all you have to do is run ethtool as root and pass the Ethernet device you want to check as an argument. In many cases this will be eth0:
$ sudo ethtool eth0 Settings for eth0: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full Advertised auto-negotiation: Yes Speed: 100Mb/s Duplex: Full Port: Twisted Pair PHYAD: 0 Transceiver: internal Auto-negotiation: on Supports Wake-on: pg Wake-on: d Current message level: 0x000000ff (255) Link detected: yes
As you can see, ethtool gives all sorts of information, including the fact that this machine supports 10 base T, 100 base T and gigabit networking speeds, but it currently communicates at 100 base T, full duplex. To check for a link, just look at the very last line that says “Link detected”. As you can see in my example, link is detected, so my cable is plugged in and I can move on.
Before I move past ethtool completely, it's worth mentioning that it does a lot more than just diagnose link problems. A common problem I've found on networks is a host with slower-than-normal network speeds. Often you'll see this crop up after a reboot or a power outage. What often happens is that when the interface connects to the network, it will try to auto-negotiate the fastest speed it can. Sometimes auto-negotiation doesn't work correctly, in which case the interface might fail back to half duplex mode or might even fail back to 10 base T! If you know that your network can support 100 base T at full duplex, you can use ethtool to disable auto-negotiation and force full duplex. To do this for eth0, you would type:
$ sudo ethtool -s eth0 autoneg off duplex full
Kyle Rankin is a systems architect; and the author of DevOps Troubleshooting, The Official Ubuntu Server Book, Knoppix Hacks, Knoppix Pocket Reference, Linux Multimedia Hacks, and Ubuntu Hacks.
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?
|Designing Electronics with Linux||May 22, 2013|
|Dynamic DNS—an Object Lesson in Problem Solving||May 21, 2013|
|Using Salt Stack and Vagrant for Drupal Development||May 20, 2013|
|Making Linux and Android Get Along (It's Not as Hard as It Sounds)||May 16, 2013|
|Drupal Is a Framework: Why Everyone Needs to Understand This||May 15, 2013|
|Home, My Backup Data Center||May 13, 2013|
- Designing Electronics with Linux
- New Products
- Making Linux and Android Get Along (It's Not as Hard as It Sounds)
- Dynamic DNS—an Object Lesson in Problem Solving
- Using Salt Stack and Vagrant for Drupal Development
- Validate an E-Mail Address with PHP, the Right Way
- Why Python?
- Tech Tip: Really Simple HTTP Server with Python
- Build a Skype Server for Your Home Phone System
- Linux Systems Administrator
- Dynamic DNS
18 min 37 sec ago
- Reply to comment | Linux Journal
1 hour 17 min ago
- Reply to comment | Linux Journal
2 hours 7 min ago
- Not free anymore
6 hours 9 min ago
9 hours 56 min ago
- Reply to comment | Linux Journal
10 hours 4 min ago
- Understanding the Linux Kernel
12 hours 19 min ago
14 hours 48 min ago
- Kernel Problem
1 day 51 min ago
- BASH script to log IPs on public web server
1 day 5 hours ago