Tales from the Server Room - Panic on the Streets of London

Here I was, thousands of miles away from home, breathing in the warm exhaust from a rack full of servers, trying to bring a stubborn server back to life. I wasn't completely without options just yet. I had a hunch the problem was related to DHCP, so I pored through the logs on my DHCP server and confirmed that, yes, I could see leases being granted to the server, and, yes, there were ample spare leases to hand out. I even restarted the DHCP service for good measure.

Finally, I decided to watch the DHCP logs during a kickstart. I would start the kickstart process, see the machine gets its lease, either the first time or when I told it to retry, then fail later on in the install. I had a log full of successful DHCP requests with no explanation of why it didn't work. Then I had my first real clue: during one of the kickstarts, I noticed that the server had actually requested a DHCP lease multiple times.

Even with this clue, I started running out of explanations. The DHCP server seemed to be healthy. After all, my laptop was able to use it just fine, and I had a log file full of successful DHCP requests. Here I turned to the next phase of troubleshooting: the guessing game. I swapped cables, changed what NIC was connected and even changed the switch port. After all of that, I still had the same issue. I had kickstarted the machine so many times now, I had the entire list of arguments memorized. I was running out of options, patience and most important, time.

[Bill: I remember seeing an e-mail or two about this. I was comfortably ensconced at the corporate HQ in California, and you were working on this while I was asleep. I'm sure I'd have been able to help more if I'd been awake. I'm glad you were on the case though.]

I was now at the next phase of troubleshooting: prayer. Somewhere around this time, I had my big breakthrough. While I was swapping all the cables around, I noticed something interesting on the switch—the LEDs for the port I was using went amber when I first plugged in the cable, and it took quite a bit of time to turn green. I noticed that the same thing happened when I kickstarted my machine and again later on during the install. It looked as though every time the server brought up its network interface, it would cause the switch to reset the port. When I watched this carefully, I saw during one install that the server errored out of the install while the port was still amber and just before it turned green!

What did all of this mean? Although it was true that the DHCP server was functioning correctly, DHCP requests themselves typically have a 30-second timeout before they give an error. It turned out that this switch was just hovering on the 30-second limit to bring a port up. When it was below 30 seconds I would get a lease; when it wasn't, I wouldn't. Even though I found the cause of the problem, it didn't do me much good. Because the installer appeared to reset its port at least three times, there was just about no way I was going to be able to be lucky enough to get three consecutive sub-30-second port resets. I had to figure out another way, yet I didn't manage the networking gear, and the people who did wouldn't be awake for hours (see sidebar).

[Bill: One of the guys I worked with right out of college always told me “Start your troubleshooting with the cabling.” When troubleshooting networking issues, it's easy to forget about things that can affect the link-layer, so I check those as part of the cabling now. It doesn't take long and can save tons of time.]

I started reviewing my options. I needed some way to take the switch out of the equation. In all of my planning for this trip, I happened to bring quite a toolkit of MacGyver sysadmin gear, including a short handmade crossover cable and a coupler. I needed to keep the original kickstart server on the network, but I realized if I could clone all of the kickstart configurations, DHCP settings and package repositories to my laptop, I could connect to the machine with a crossover cable and complete the kickstart that way.

After a few apt-gets, rsyncs, and some tweaking and tuning on the server room floor, I had my Frankenstein kickstart server ready to go. Like I had hoped, the kickstart completed without a hitch. I was then able to repeat the same task on the other two servers in no time and was relieved to send the e-mail to the rest of the team saying that all of their servers were ready for them, right on schedule. On the next day of the trip, I was able to knock out all of my tasks early so I could spend the final provisional day sightseeing around London. It all goes to show that although a good plan is important, you also should be flexible for when something inevitably goes outside your plan.

[Bill: I'm glad you planned like you did, but it also highlights how important being observant and having a good troubleshooting methodology are. Although you were able to duct-tape a new kickstart server out of your laptop, you could have spent infinitely longer chasing the issue. It's just as important to know when to stop chasing a problem and put a band-aid in place as it is to fix the problem in the first place.]