Rebooting the Magic Way

 in

If you have ever had a hard drive fail on a remote server you may remember the feeling you had after trying to issue the following commands:

    # reboot
    bash: /sbin/reboot: Input/output error
    # shutdown -r now
    bash: /sbin/shutdown: Input/output error

Obviously, there is a problem with your drive. These commands are failing because the kernel is unable to load the /sbin/reboot and /sbin/shutdown binaries from the disk so that it can execute them.

A fsck on the next boot might be able to correct whatever is wrong with the disk, but first you need to get the system to reboot. If your machine is located at a managed hosting provider then you could submit a reboot ticket, but you'll have to wait for someone to take responsibility.

Wouldn't it be nice if there was a way to ask the kernel to reboot without needing to access the failing drive? Well, there is a way, and it is remarkably simple.

The "magic SysRq key" provides a way to send commands directly to the kernel through the /proc filesystem. It is enabled via a kernel compile time option, CONFIG_MAGIC_SYSRQ, which seems to be standard on most distributions. First you must activate the magic SysRq option:

    echo 1 > /proc/sys/kernel/sysrq

When you are ready to reboot the machine simply run the following:

    echo b > /proc/sysrq-trigger

This does not attempt to unmount or sync filesystems, so it should only be used when absolutely necessary, but if your drive is already failing then that may not be a concern.

In addition to rebooting the system the sysrq trick can be used to dump memory information to the console, sync all filesystems, remount all filesystems in read-only mode, send SIGTERM or SIGKILL to all processes except init, or power off the machine entirely, among other things.

Also, instead of echoing into /proc/sys/kernel/sysrq each time you can activate the magic SysRq key at system boot time using sysctl, where supported:

    echo "kernel.sysrq = 1" >> /etc/sysctl.conf

If you would like to learn more about magic SysRq you can read the sysrq.txt file in the kernel documentation.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

echo 1 >

asr's picture

echo 1 > /proc/sys/kernel/sysrq
bash: /proc/sys/kernel/sysrq: No such file or directory

:'(

and that's how you learn not to be such a dick when building your own kernel.

Heartfelt thanks

baski's picture

I was butting head for a long time to see how i can do it remotely without going to the site. Thanks once again.. you saved me some green and greened the earth.. :)

Thanks!

Jan Angelovic's picture

Thanks a lot! My disk is going to hell but important data was saved. :-) You save me from 100 km long trip to telehouse.

Than you so much for this

morgellons's picture

Than you so much for this trick. It should come in handy when rebooting my vps! for me site
morgellons

printer problems

Cristian's picture

Do you happen to know a magic way to reboot my printers? That would be magic trick which I would love to learn... I just hate when printers start behaving very weird and ruining my whole day.

I like this blog.Nice work.

bilety lotnicze's picture

I like this blog.Nice work.

Less typing

Craig MIller's picture

If one is truely looking to do less typing and reboot, I suggest:
"init 6"

This however, does require the file system to be functioning (since it will walk thru the /etc/rc?.d/ shutdown scripts).

But really, how often does one need to shut down a system (or reboot) with no filesystem?

# init 6 bash:

Cory Wright's picture
    # init 6
    bash: /sbin/init: Input/output error

:)

Also, as several people have already commented, there are lot's of cases where you may need to reboot without accessing the filesystem. I've had to deal with the Dell RAID issue myself.

Excellent information!

Anonymous's picture

To Mr. Miller's question: "But really, how often does one need to shut down a system (or reboot) with no filesystem?", I would have to reply more often then we wished. It may just be this particular application, but we have 1500+ instances of a system running Linux offering file shares via nfs with which we encounter this very problem. Being able to remotely reboot such a system so that it can run fsck upon reboot is potentially a great time saver. Without this functionality, the time needed to coordinate with remote sites to physically reboot the boxes may lead to days when this service is unavailable. This also allows us to make a quicker determination of whether we merely have file system corruption, or hardware failure, and dispatch maintenance when required.

Not that I'm a Linux guru or

TsueDesu's picture

Not that I'm a Linux guru or anything...but if the system drive is failing, Why would you want to reboot the server/system? Wouldn't that render it totally unusable? After all, you're still linked in remotely, so you could rebuild it onto a working drive or at least move the system to a new drive. Or was this implied?

*scratching head in uncertainty*

I'll say its an interesting article and I'm sure there's a use for this for this.

TsueDesu a.k.a This is stick!

Hmm. I was stupid enough to

Mad Cow's picture

Hmm. I was stupid enough to make my a and b nameservers the same host. Now the disk is failing but I can't get to the host until monday at the earliest. This is great. It might just work. ;-)

It didn't work BTW. I have to visit the server on monday anyhow. Ho hum.

Dell RAID Problems

Will Reese's picture

I've had Dell servers with PERC RAID cards crap out under heavy IO and cause the filesystem to become unusable until a reboot. The cards and the disks were fine, but the driver wasn't. Rebooting resolved the issue until we were able to upgrade to better hardware and drivers.

-- Will Reese

sohpet

Anonymous's picture

thanks for sites

Sync and umount with magic sysrq keys

messageid's picture

With the Magic SysRQ keys is also possibile to umount and sync all the mounted disk.

If you issue the following key:

R S E I U B

You will: Sync the disk, Term and the Kill all the process, Umount all the mounted disk and the Reboot the system; so you can "emlate" a correct shoutdown process.

I use

cuervo's picture

alt-sysrq-S, U, S, B. Sync, remount all filesystems R/O, sync again, hard reboot. I typically let around half a second between those. There's some standard sequence involving elephants, but I figured mine out when magic syswreck was first introduced, and stuck with it. (Hint: That means parent's sequence is probably better.)

FYI, from /usr/src/linux*/Documentation/sysrq.txt:

'b' - Will immediately reboot the system without syncing or unmounting
your disks.

'c' - Will perform a kexec reboot in order to take a crashdump.

'd' - Shows all locks that are held.

'e' - Send a SIGTERM to all processes, except for init.

'f' - Will call oom_kill to kill a memory hog process.

'g' - Used by kgdb on ppc and sh platforms.

'h' - Will display help (actually any other key than those listed
above will display help. but 'h' is easy to remember :-)

'i' - Send a SIGKILL to all processes, except for init.

'k' - Secure Access Key (SAK) Kills all programs on the current virtual
console. NOTE: See important comments below in SAK section.

'm' - Will dump current memory info to your console.

'n' - Used to make RT tasks nice-able

'o' - Will shut your system off (if configured and supported).

'p' - Will dump the current registers and flags to your console.

'q' - Will dump a list of all running timers.

'r' - Turns off keyboard raw mode and sets it to XLATE.

's' - Will attempt to sync all mounted filesystems.

't' - Will dump a list of current tasks and their information to your
console.

'u' - Will attempt to remount all mounted filesystems read-only.

'v' - Dumps Voyager SMP processor info to your console.

'w' - Dumps tasks that are in uninterruptable (blocked) state.

'x' - Used by xmon interface on ppc/powerpc platforms.

Not that easy

Anonymous's picture

# echo 1 > /proc/sys/kernel/sysrq
bash: /bin/echo: Input/output error

Shell built-in

Anonymous's picture

'echo' is also a shell built-in for any decent shell.

Shell built-in

Anonymous's picture

'echo' is also a shell built-in for any decent shell.

echo IS built into bash, but

Anonymous's picture

echo IS built into bash, but in some distros there's also a /bin/echo which is in the path. Try PATH="" echo > /proc/yourmom

In Bash, the precedence

Anonymous's picture

In Bash, the precedence order is:
1. Alias
2. Function (not sure of the order of the first two)
3. Built-in
4. Program found via a $PATH search

The built-in will get used first. Most Linux distros run with Bash as the primary shell now.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix