Loading
Rebooting the Magic Way
If you have ever had a hard drive fail on a remote server you may remember the feeling you had after trying to issue the following commands:
# reboot
bash: /sbin/reboot: Input/output error
# shutdown -r now
bash: /sbin/shutdown: Input/output error
Obviously, there is a problem with your drive. These commands are failing because the kernel is unable to load the /sbin/reboot and /sbin/shutdown binaries from the disk so that it can execute them.
A fsck on the next boot might be able to correct whatever is wrong with the disk, but first you need to get the system to reboot. If your machine is located at a managed hosting provider then you could submit a reboot ticket, but you'll have to wait for someone to take responsibility.
Wouldn't it be nice if there was a way to ask the kernel to reboot without needing to access the failing drive? Well, there is a way, and it is remarkably simple.
The "magic SysRq key" provides a way to send commands directly to the kernel through the /proc filesystem. It is enabled via a kernel compile time option, CONFIG_MAGIC_SYSRQ, which seems to be standard on most distributions. First you must activate the magic SysRq option:
echo 1 > /proc/sys/kernel/sysrq
When you are ready to reboot the machine simply run the following:
echo b > /proc/sysrq-trigger
This does not attempt to unmount or sync filesystems, so it should only be used when absolutely necessary, but if your drive is already failing then that may not be a concern.
In addition to rebooting the system the sysrq trick can be used to dump memory information to the console, sync all filesystems, remount all filesystems in read-only mode, send SIGTERM or SIGKILL to all processes except init, or power off the machine entirely, among other things.
Also, instead of echoing into /proc/sys/kernel/sysrq each time you can activate the magic SysRq key at system boot time using sysctl, where supported:
echo "kernel.sysrq = 1" >> /etc/sysctl.conf
If you would like to learn more about magic SysRq you can read the sysrq.txt file in the kernel documentation.
______________________
White Paper
Fabric-Based Computing Enables Optimized Hyperscale Data Centers
Today’s modular x86 servers are compute-centric, designed as a least common denominator to support a wide range of IT workloads. Those generic, virtualized IT workloads have much different resource optimization requirements than hyperscale and cloud applications. They have resulted in a “one size fits all” enterprise IT architecture that is not optimized for a specific set of IT workloads, and especially not emerging hyperscale workloads, such as web applications, big data, and object storage. In this report, you will learn how shifting the focus from traditional compute-centric IT architectures to an innovative disaggregated fabric-based architecture can optimize and scale your data center.
Sponsored by AMD
White Paper
Red Hat White Paper: Using an Open Source Framework to Catch the Bad Guy
Built-in forensics, incident response, and security with Red Hat Enterprise Linux 6
Every security policy provides guidance and requirements for ensuring adequate protection of information and data, as well as high-level technical and administrative security requirements for a system in a given environment. Traditionally, providing security for a system focuses on the confidentiality of the information on it. However, protecting the data integrity and system and data availability is just as important. For example, when processing United States intelligence information, there are three attributes that require protection: confidentiality, integrity, and availability.
Learn more about catching the bad guy in this free white paper.
Sponsored by DLT Solutions
- New Products
- Making Linux and Android Get Along (It's Not as Hard as It Sounds)
- Drupal Is a Framework: Why Everyone Needs to Understand This
- A Topic for Discussion - Open Source Feature-Richness?
- Home, My Backup Data Center
- RSS Feeds
- New Products
- Trying to Tame the Tablet
- What's the tweeting protocol?
- Dart: a New Web Programming Experience
- Drupal is an Awesome CMS and a Crappy development framework
3 hours 34 min ago - IT industry leaders
5 hours 57 min ago - Reply to comment | Linux Journal
22 hours 45 min ago - Reply to comment | Linux Journal
1 day 1 hour ago - Reply to comment | Linux Journal
1 day 2 hours ago - great post
1 day 3 hours ago - Google Docs
1 day 3 hours ago - Reply to comment | Linux Journal
1 day 8 hours ago - Reply to comment | Linux Journal
1 day 9 hours ago - Web Hosting IQ
1 day 10 hours ago
Enter to Win an Adafruit Prototyping Pi Plate Kit for Raspberry Pi
It's Raspberry Pi month at Linux Journal. Each week in May, Adafruit will be giving away a Pi-related prize to a lucky, randomly drawn LJ reader. Winners will be announced weekly.
Fill out the fields below to enter to win this week's prize-- a Prototyping Pi Plate Kit for Raspberry Pi.
Congratulations to our winners so far:
- 5-8-13, Pi Starter Pack: Jack Davis
- 5-15-13, Pi Model B 512MB RAM: Patrick Dunn
- Next winner announced on 5-21-13!
Free Webinar: Linux Backup and Recovery
Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.
In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.
Developer Poll
Comments
echo 1 >
echo 1 > /proc/sys/kernel/sysrq
bash: /proc/sys/kernel/sysrq: No such file or directory
:'(
and that's how you learn not to be such a dick when building your own kernel.
Heartfelt thanks
I was butting head for a long time to see how i can do it remotely without going to the site. Thanks once again.. you saved me some green and greened the earth.. :)
Thanks!
Thanks a lot! My disk is going to hell but important data was saved. :-) You save me from 100 km long trip to telehouse.
Than you so much for this
Than you so much for this trick. It should come in handy when rebooting my vps! for me site
morgellons
printer problems
Do you happen to know a magic way to reboot my printers? That would be magic trick which I would love to learn... I just hate when printers start behaving very weird and ruining my whole day.
I like this blog.Nice work.
I like this blog.Nice work.
Less typing
If one is truely looking to do less typing and reboot, I suggest:
"init 6"
This however, does require the file system to be functioning (since it will walk thru the /etc/rc?.d/ shutdown scripts).
But really, how often does one need to shut down a system (or reboot) with no filesystem?
# init 6 bash:
# init 6 bash: /sbin/init: Input/output error:)
Also, as several people have already commented, there are lot's of cases where you may need to reboot without accessing the filesystem. I've had to deal with the Dell RAID issue myself.
Excellent information!
To Mr. Miller's question: "But really, how often does one need to shut down a system (or reboot) with no filesystem?", I would have to reply more often then we wished. It may just be this particular application, but we have 1500+ instances of a system running Linux offering file shares via nfs with which we encounter this very problem. Being able to remotely reboot such a system so that it can run fsck upon reboot is potentially a great time saver. Without this functionality, the time needed to coordinate with remote sites to physically reboot the boxes may lead to days when this service is unavailable. This also allows us to make a quicker determination of whether we merely have file system corruption, or hardware failure, and dispatch maintenance when required.
Not that I'm a Linux guru or
Not that I'm a Linux guru or anything...but if the system drive is failing, Why would you want to reboot the server/system? Wouldn't that render it totally unusable? After all, you're still linked in remotely, so you could rebuild it onto a working drive or at least move the system to a new drive. Or was this implied?
*scratching head in uncertainty*
I'll say its an interesting article and I'm sure there's a use for this for this.
TsueDesu a.k.a This is stick!
Hmm. I was stupid enough to
Hmm. I was stupid enough to make my a and b nameservers the same host. Now the disk is failing but I can't get to the host until monday at the earliest. This is great. It might just work. ;-)
It didn't work BTW. I have to visit the server on monday anyhow. Ho hum.
Dell RAID Problems
I've had Dell servers with PERC RAID cards crap out under heavy IO and cause the filesystem to become unusable until a reboot. The cards and the disks were fine, but the driver wasn't. Rebooting resolved the issue until we were able to upgrade to better hardware and drivers.
-- Will Reese
sohpet
thanks for sites
Sync and umount with magic sysrq keys
With the Magic SysRQ keys is also possibile to umount and sync all the mounted disk.
If you issue the following key:
R S E I U B
You will: Sync the disk, Term and the Kill all the process, Umount all the mounted disk and the Reboot the system; so you can "emlate" a correct shoutdown process.
I use
alt-sysrq-S, U, S, B. Sync, remount all filesystems R/O, sync again, hard reboot. I typically let around half a second between those. There's some standard sequence involving elephants, but I figured mine out when magic syswreck was first introduced, and stuck with it. (Hint: That means parent's sequence is probably better.)
FYI, from /usr/src/linux*/Documentation/sysrq.txt:
'b' - Will immediately reboot the system without syncing or unmounting
your disks.
'c' - Will perform a kexec reboot in order to take a crashdump.
'd' - Shows all locks that are held.
'e' - Send a SIGTERM to all processes, except for init.
'f' - Will call oom_kill to kill a memory hog process.
'g' - Used by kgdb on ppc and sh platforms.
'h' - Will display help (actually any other key than those listed
above will display help. but 'h' is easy to remember :-)
'i' - Send a SIGKILL to all processes, except for init.
'k' - Secure Access Key (SAK) Kills all programs on the current virtual
console. NOTE: See important comments below in SAK section.
'm' - Will dump current memory info to your console.
'n' - Used to make RT tasks nice-able
'o' - Will shut your system off (if configured and supported).
'p' - Will dump the current registers and flags to your console.
'q' - Will dump a list of all running timers.
'r' - Turns off keyboard raw mode and sets it to XLATE.
's' - Will attempt to sync all mounted filesystems.
't' - Will dump a list of current tasks and their information to your
console.
'u' - Will attempt to remount all mounted filesystems read-only.
'v' - Dumps Voyager SMP processor info to your console.
'w' - Dumps tasks that are in uninterruptable (blocked) state.
'x' - Used by xmon interface on ppc/powerpc platforms.
Not that easy
# echo 1 > /proc/sys/kernel/sysrq
bash: /bin/echo: Input/output error
Shell built-in
'echo' is also a shell built-in for any decent shell.
Shell built-in
'echo' is also a shell built-in for any decent shell.
echo IS built into bash, but
echo IS built into bash, but in some distros there's also a /bin/echo which is in the path. Try PATH="" echo > /proc/yourmom
In Bash, the precedence
In Bash, the precedence order is:
1. Alias
2. Function (not sure of the order of the first two)
3. Built-in
4. Program found via a $PATH search
The built-in will get used first. Most Linux distros run with Bash as the primary shell now.