Tales from the Server Room - Panic on the Streets of London
I've always thought it's better to learn from someone else's mistakes than from my own. In this column, Kyle Rankin or Bill Childers will tell a story from their years as systems administrators while the other will chime in from time to time. It's a win-win: you get to learn from our experiences, and we get to make snide comments to each other. Kyle tells the first story in this series.
I was pretty excited about my first trip to the London data center. I had been to London before on vacation, but this was the first time I would visit our colocation facility on business. What's more, it was the first remote data-center trip I was to take by myself. Because I still was relatively new to the company and the junior-most sysadmin at the time, this was the perfect opportunity to prove that I knew what I was doing and could be trusted for future trips.
The maintenance was relatively straightforward. A few machines needed a fresh Linux install, plus I would troubleshoot an unresponsive server, audit our serial console connections, and do a few other odds and ends. We estimated it was a two-day job, but just in case, we added an extra provisional day.
[Bill: If I remember right, I had to fight to get that extra day tacked onto the trip for you. We'd learned from past experience that nothing at that place seemed easy at face value.]
Even with an extra day, I wanted this trip to go smoothly, so I came up with a comprehensive plan. Each task was ordered by its priority along with detailed lists of the various commands and procedures I would use to accomplish each task. I even set up an itemized checklist of everything I needed to take with me.
[Bill: I remember thinking that you were taking it way too seriously—after all, it was just a kickstart of a few new machines. What could possibly go wrong? In hindsight, I'm glad you made all those lists.]
The first day I arrived at the data center, I knew exactly what I needed to do. Once I got my badge and was escorted through multiple levels of security to our colocation cages, I would kickstart each of the servers on my list one by one and perform all the manual configuration steps they needed. If I had time, I could finish the rest of the maintenance; otherwise, I'd leave any other tasks for the next day.
Now, it's worth noting that at this time we didn't have a sophisticated kickstart system in place nor did we have advanced lights-out management—just a serial console and a remotely controlled power system. Although our data center did have a kickstart server with a package repository, we still had to connect each server to a monitor and keyboard, boot from an install CD and manually type in the URL to the kickstart file.
[Bill: I think this experience is what started us down the path of a lights-out management solution. I remember pitching it to the executives as “administering from the Bahamas”, and relaying this story to them was one of the key reasons that pitch was successful.]
After I had connected everything to the first server, I inserted the CD, booted the system and typed in my kickstart URL according to my detailed plans. Immediately I saw the kernel load, and the kickstart process was under way. Wow, if everything keeps going this way, I might even get this done early, I thought. Before I could start making plans for my extra days in London though, I saw the kickstart red screen of death. The kickstart logs showed that for some reason, it wasn't able to retrieve some of the files it needed from the kickstart server.
Great, now I needed to troubleshoot a broken kickstart server. Luckily, I had brought my laptop with me, and the troubleshooting was straightforward. I connected my laptop to the network, eventually got a DHCP lease, pointed the browser to the kickstart server, and sure enough, I was able to see my kickstart configuration files and browse through my package repository with no problems.
I wasn't exactly sure what was wrong, but I chalked it up to a momentary blip and decided to try again. This time, the kickstart failed, but at a different point in the install. I tried a third time, and it failed at the original point in the install. I repeated the kickstart process multiple times, trying to see some sort of pattern, but all I saw was the kickstart fail at a few different times.
The most maddening thing about this problem was the inconsistency. What's worse, even though I had more days to work on this, the kickstart of this first server was the most important task to get done immediately. In a few hours, I would have a team of people waiting on the server so they could set it up as a database system.
Kyle Rankin is a VP of engineering operations at Final, Inc., the author of a number of books including DevOps Troubleshooting and The Official Ubuntu Server Book, and is a columnist for Linux Journal. Follow him @kylerankin.
Practical Task Scheduling Deployment
July 20, 2016 12:00 pm CDT
One of the best things about the UNIX environment (aside from being stable and efficient) is the vast array of software tools available to help you do your job. Traditionally, a UNIX tool does only one thing, but does that one thing very well. For example, grep is very easy to use and can search vast amounts of data quickly. The find tool can find a particular file or files based on all kinds of criteria. It's pretty easy to string these tools together to build even more powerful tools, such as a tool that finds all of the .log files in the /home directory and searches each one for a particular entry. This erector-set mentality allows UNIX system administrators to seem to always have the right tool for the job.
Cron traditionally has been considered another such a tool for job scheduling, but is it enough? This webinar considers that very question. The first part builds on a previous Geek Guide, Beyond Cron, and briefly describes how to know when it might be time to consider upgrading your job scheduling infrastructure. The second part presents an actual planning and implementation framework.
Join Linux Journal's Mike Diehl and Pat Cameron of Help Systems.
Free to Linux Journal readers.Register Now!
- SUSE LLC's SUSE Manager
- My +1 Sword of Productivity
- Non-Linux FOSS: Caffeine!
- Managing Linux Using Puppet
- Control Your Linux Desktop with D-Bus
- Download "Linux Management with Red Hat Satellite: Measuring Business Impact and ROI"
- Doing for User Space What We Did for Kernel Space
- SuperTuxKart 0.9.2 Released
- Murat Yener and Onur Dundar's Expert Android Studio (Wrox)
- Google's SwiftShader Released
With all the industry talk about the benefits of Linux on Power and all the performance advantages offered by its open architecture, you may be considering a move in that direction. If you are thinking about analytics, big data and cloud computing, you would be right to evaluate Power. The idea of using commodity x86 hardware and replacing it every three years is an outdated cost model. It doesn’t consider the total cost of ownership, and it doesn’t consider the advantage of real processing power, high-availability and multithreading like a demon.
This ebook takes a look at some of the practical applications of the Linux on Power platform and ways you might bring all the performance power of this open architecture to bear for your organization. There are no smoke and mirrors here—just hard, cold, empirical evidence provided by independent sources. I also consider some innovative ways Linux on Power will be used in the future.Get the Guide