Rebooting the Magic Way

Aug 21, 2008 By Cory Wright

HOW-TOs

If you have ever had a hard drive fail on a remote server you may remember the feeling you had after trying to issue the following commands:

    # reboot
    bash: /sbin/reboot: Input/output error
    # shutdown -r now
    bash: /sbin/shutdown: Input/output error

Obviously, there is a problem with your drive. These commands are failing because the kernel is unable to load the /sbin/reboot and /sbin/shutdown binaries from the disk so that it can execute them.

A fsck on the next boot might be able to correct whatever is wrong with the disk, but first you need to get the system to reboot. If your machine is located at a managed hosting provider then you could submit a reboot ticket, but you'll have to wait for someone to take responsibility.

Wouldn't it be nice if there was a way to ask the kernel to reboot without needing to access the failing drive? Well, there is a way, and it is remarkably simple.

The "magic SysRq key" provides a way to send commands directly to the kernel through the /proc filesystem. It is enabled via a kernel compile time option, CONFIG_MAGIC_SYSRQ, which seems to be standard on most distributions. First you must activate the magic SysRq option:

    echo 1 > /proc/sys/kernel/sysrq

When you are ready to reboot the machine simply run the following:

    echo b > /proc/sysrq-trigger

This does not attempt to unmount or sync filesystems, so it should only be used when absolutely necessary, but if your drive is already failing then that may not be a concern.

In addition to rebooting the system the sysrq trick can be used to dump memory information to the console, sync all filesystems, remount all filesystems in read-only mode, send SIGTERM or SIGKILL to all processes except init, or power off the machine entirely, among other things.

Also, instead of echoing into /proc/sys/kernel/sysrq each time you can activate the magic SysRq key at system boot time using sysctl, where supported:

    echo "kernel.sysrq = 1" >> /etc/sysctl.conf

If you would like to learn more about magic SysRq you can read the sysrq.txt file in the kernel documentation.

______________________

Cory Wright is a blogger for LinuxJournal.com

Comments

echo 1 >

Submitted by asr (not verified) on Jan 08, 2010.

echo 1 > /proc/sys/kernel/sysrq
bash: /proc/sys/kernel/sysrq: No such file or directory

:'(

and that's how you learn not to be such a dick when building your own kernel.

Heartfelt thanks

Submitted by baski (not verified) on Nov 04, 2009.

I was butting head for a long time to see how i can do it remotely without going to the site. Thanks once again.. you saved me some green and greened the earth.. :)

Thanks!

Submitted by Jan Angelovic (not verified) on Mar 09, 2009.

Thanks a lot! My disk is going to hell but important data was saved. :-) You save me from 100 km long trip to telehouse.

Than you so much for this

Submitted by morgellons (not verified) on Nov 16, 2008.

Than you so much for this trick. It should come in handy when rebooting my vps! for me site
morgellons

printer problems

Submitted by Cristian (not verified) on Nov 03, 2008.

Do you happen to know a magic way to reboot my printers? That would be magic trick which I would love to learn... I just hate when printers start behaving very weird and ruining my whole day.

I like this blog.Nice work.

Submitted by bilety lotnicze (not verified) on Sep 05, 2008.

I like this blog.Nice work.

Less typing

Submitted by Craig MIller (not verified) on Aug 25, 2008.

If one is truely looking to do less typing and reboot, I suggest:
"init 6"

This however, does require the file system to be functioning (since it will walk thru the /etc/rc?.d/ shutdown scripts).

But really, how often does one need to shut down a system (or reboot) with no filesystem?

# init 6 bash:

Submitted by Cory Wright on Aug 26, 2008.

    # init 6
    bash: /sbin/init: Input/output error

Also, as several people have already commented, there are lot's of cases where you may need to reboot without accessing the filesystem. I've had to deal with the Dell RAID issue myself.

Cory Wright is a blogger for LinuxJournal.com

Excellent information!

Submitted by Anonymous (not verified) on Aug 26, 2008.

To Mr. Miller's question: "But really, how often does one need to shut down a system (or reboot) with no filesystem?", I would have to reply more often then we wished. It may just be this particular application, but we have 1500+ instances of a system running Linux offering file shares via nfs with which we encounter this very problem. Being able to remotely reboot such a system so that it can run fsck upon reboot is potentially a great time saver. Without this functionality, the time needed to coordinate with remote sites to physically reboot the boxes may lead to days when this service is unavailable. This also allows us to make a quicker determination of whether we merely have file system corruption, or hardware failure, and dispatch maintenance when required.

Not that I'm a Linux guru or

Submitted by TsueDesu (not verified) on Aug 25, 2008.

Not that I'm a Linux guru or anything...but if the system drive is failing, Why would you want to reboot the server/system? Wouldn't that render it totally unusable? After all, you're still linked in remotely, so you could rebuild it onto a working drive or at least move the system to a new drive. Or was this implied?

*scratching head in uncertainty*

I'll say its an interesting article and I'm sure there's a use for this for this.

TsueDesu a.k.a This is stick!

Hmm. I was stupid enough to

Submitted by Mad Cow (not verified) on Oct 25, 2008.

Hmm. I was stupid enough to make my a and b nameservers the same host. Now the disk is failing but I can't get to the host until monday at the earliest. This is great. It might just work. ;-)

It didn't work BTW. I have to visit the server on monday anyhow. Ho hum.

Dell RAID Problems

Submitted by Will Reese on Aug 25, 2008.

I've had Dell servers with PERC RAID cards crap out under heavy IO and cause the filesystem to become unusable until a reboot. The cards and the disks were fine, but the driver wasn't. Rebooting resolved the issue until we were able to upgrade to better hardware and drivers.

-- Will Reese

sohpet

Submitted by Anonymous (not verified) on Aug 23, 2008.

thanks for sites

Sync and umount with magic sysrq keys

Submitted by messageid (not verified) on Aug 22, 2008.

With the Magic SysRQ keys is also possibile to umount and sync all the mounted disk.

If you issue the following key:

R S E I U B

You will: Sync the disk, Term and the Kill all the process, Umount all the mounted disk and the Reboot the system; so you can "emlate" a correct shoutdown process.

I use

Submitted by cuervo (not verified) on Sep 06, 2008.

alt-sysrq-S, U, S, B. Sync, remount all filesystems R/O, sync again, hard reboot. I typically let around half a second between those. There's some standard sequence involving elephants, but I figured mine out when magic syswreck was first introduced, and stuck with it. (Hint: That means parent's sequence is probably better.)

FYI, from /usr/src/linux*/Documentation/sysrq.txt:

'b' - Will immediately reboot the system without syncing or unmounting
your disks.

'c' - Will perform a kexec reboot in order to take a crashdump.

'd' - Shows all locks that are held.

'e' - Send a SIGTERM to all processes, except for init.

'f' - Will call oom_kill to kill a memory hog process.

'g' - Used by kgdb on ppc and sh platforms.

'h' - Will display help (actually any other key than those listed
above will display help. but 'h' is easy to remember :-)

'i' - Send a SIGKILL to all processes, except for init.

'k' - Secure Access Key (SAK) Kills all programs on the current virtual
console. NOTE: See important comments below in SAK section.

'm' - Will dump current memory info to your console.

'n' - Used to make RT tasks nice-able

'o' - Will shut your system off (if configured and supported).

'p' - Will dump the current registers and flags to your console.

'q' - Will dump a list of all running timers.

'r' - Turns off keyboard raw mode and sets it to XLATE.

's' - Will attempt to sync all mounted filesystems.

't' - Will dump a list of current tasks and their information to your
console.

'u' - Will attempt to remount all mounted filesystems read-only.

'v' - Dumps Voyager SMP processor info to your console.

'w' - Dumps tasks that are in uninterruptable (blocked) state.

'x' - Used by xmon interface on ppc/powerpc platforms.