Hack and / - Linux Troubleshooting, Part III: Remote Networks
Another common nslookup error you might run into is this:
$ nslookup web1 Server: 10.2.2.2 Address: 10.2.2.2#53 ** server can't find web1: NXDOMAIN
Here my name server at 10.2.2.2 responded to me but told me it couldn't find the record for server web1. This error could mean that I don't have web1's proper domain name in my DNS search path. If you don't specify a host's fully qualified domain name (for instance, web1.mysite.com) but instead use the shorthand form of the hostname, your system will check /etc/resolv.conf for domains in your DNS search path. It then will add those domains one by one to the end of your hostname to see if it resolves. The DNS search path is the line in /etc/resolv.conf that starts with the word search:
search example.net example2.net nameserver 10.2.2.2
In my case, when I search for web1's IP address, my system will first search for web1.example.net, and if that has no records, it will search for web1.example2.net. If you want to test whether this is the problem, simply run nslookup again but with the fully qualified domain name (such as web1.mysite.com). If it resolves, either make sure you always use the fully qualified domain name when you access that server, or add that domain to the search path in /etc/resolv.conf.
If you try nslookup against the fully qualified domain name and you still get the same NXDOMAIN error above, your problem is with the name server itself. Troubleshooting the full range of DNS server problems is a bit beyond what I could reasonably fit in this column, but here are a few steps to get you started. If you know your DNS server is configured to have the record you are looking for itself, you need to examine its zone records to make sure that particular hostname exists. If, on the other hand, you are searching for a domain for which you know it doesn't have a record (say, www.linuxjournal.com), it's possible your DNS server isn't allowing recursive queries from your host or at all. You can test that by trying to resolve some other remote host on the Internet. If it doesn't resolve, it's probably a recursion setting. If it does resolve, the problem might very well be with that remote site's DNS server.
If after all these tests you find that your DNS servers are working fine, but you still can't access the remote server, the final step is to perform another traceroute like above, only directly against the remote server. So for instance, if you wanted to test your route to www.linuxjournal.com, the traceroute might look like the following:
$ traceroute www.linuxjournal.com traceroute to www.linuxjournal.com (188.8.131.52), 30 hops max, ↪60 byte packets 1 10.1.1.1 (10.1.1.1) 1.016 ms 2.222 ms 2.308 ms 2 75-101-46-1.dsl.static.sonic.net (184.108.40.206) 6.916 ms ↪7.389 ms 8.386 ms 3 921.gig0-3.gw.sjc2.sonic.net (220.127.116.11) 11.265 ms ↪12.435 ms 13.050 ms 4 108.ae0.gw.equinix-sj.sonic.net (18.104.22.168) 13.846 ms ↪15.233 ms 15.390 ms 5 GIG2-0.sea-dis-2.peer1.net (22.214.171.124) 35.149 ms ↪36.272 ms 36.944 ms 6 oc48.so-2-1-0.sea-coloc-dis-1.peer1.net (126.96.36.199) ↪37.340 ms 27.884 ms 27.266 ms 7 10ge.ten1-2.sj-mkp16-dis-1.peer1.net (188.8.131.52) ↪28.421 ms 29.014 ms 29.688 ms 8 10ge.ten1-2.sj-mkp2-dis-1.peer1.net (184.108.40.206) ↪30.903 ms 31.015 ms 31.804 ms 9 10ge-ten1-3.la-600w-cor-1.peer1.net (220.127.116.11) ↪40.840 ms 41.279 ms 42.069 ms 10 10ge.ten1-1.la-600w-cor-2.peer1.net (18.104.22.168) ↪42.587 ms 43.710 ms 44.921 ms 11 10ge-ten1-2.dal-eqx-cor-1.peer1.net (22.214.171.124) ↪81.702 ms 82.959 ms 83.934 ms 12 10ge-ten1-1.dal-eqx-cor-2.peer1.net (126.96.36.199) ↪74.876 ms 72.454 ms 72.798 ms 13 10ge-ten1-3.sat-8500v-cor-2.peer1.net (188.8.131.52) ↪80.224 ms 81.872 ms 82.569 ms 14 184.108.40.206 (220.127.116.11) 83.499 ms 84.162 ms ↪85.048 ms 15 www.linuxjournal.com (18.104.22.168) 85.484 ms 86.461 ms ↪87.153 ms
In this example, I'm 15 hops (or routers) away from the www.linuxjournal.com server. This is an example of a successful query, but if you ran the same query and noticed a number of rows of asterisks that never made it to your destination and you couldn't ping www.linuxjournal.com directly, the problem could be an Internet routing issue between you and the remote network. Unfortunately, it's probably something outside your control, but fortunately, these sorts of problems tend to resolve themselves pretty quickly, so just keep trying.
If, on the other hand, your traceroute command was successful, but the remote site still didn't work, go back to the steps I discussed in my previous column on how to use telnet and nmap to test whether a remote port is open. It actually could be that the remote server is down (hey, it happens to the best of us) or that someone has configured a firewall to block you from that remote server.
I hope this series has kindled (or rekindled) your interest in troubleshooting under Linux. One of the things I love about Linux is how little it hides from you about how it works and how many troubleshooting tools it provides when things do go wrong. If this has piqued your interest, there are many more troubleshooting avenues for you to explore—from DNS servers like I mentioned above, to troubleshooting just about any type of service. Also, if you have any other great tools or techniques you use to track down these problems, drop me a line. I'm always on the lookout for tools to solve problems faster.
Kyle Rankin is a Systems Architect in the San Francisco Bay Area and the author of a number of books, including The Official Ubuntu Server Book, Knoppix Hacks and Ubuntu Hacks. He is currently the president of the North Bay Linux Users' Group.
Kyle Rankin is a systems architect; and the author of DevOps Troubleshooting, The Official Ubuntu Server Book, Knoppix Hacks, Knoppix Pocket Reference, Linux Multimedia Hacks, and Ubuntu Hacks.
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?
|Designing Electronics with Linux||May 22, 2013|
|Dynamic DNS—an Object Lesson in Problem Solving||May 21, 2013|
|Using Salt Stack and Vagrant for Drupal Development||May 20, 2013|
|Making Linux and Android Get Along (It's Not as Hard as It Sounds)||May 16, 2013|
|Drupal Is a Framework: Why Everyone Needs to Understand This||May 15, 2013|
|Home, My Backup Data Center||May 13, 2013|
- Linux Systems Administrator
- New Products
- Senior Perl Developer
- Technical Support Rep
- UX Designer
- Designing Electronics with Linux
- Dynamic DNS—an Object Lesson in Problem Solving
- Using Salt Stack and Vagrant for Drupal Development
- Making Linux and Android Get Along (It's Not as Hard as It Sounds)
- Have you tried Boxen? It's a
4 hours 58 min ago
- seo services in india
9 hours 30 min ago
- For KDE install kio-mtp
9 hours 31 min ago
- Evernote is much more...
11 hours 31 min ago
- Reply to comment | Linux Journal
20 hours 16 min ago
- Dynamic DNS
20 hours 50 min ago
- Reply to comment | Linux Journal
21 hours 49 min ago
- Reply to comment | Linux Journal
22 hours 39 min ago
- Not free anymore
1 day 2 hours ago
1 day 6 hours ago