Home ›

Rebooting the Magic Way

Aug 21, 2008  By Cory Wright
 in

If you have ever had a hard drive fail on a remote server you may remember the feeling you had after trying to issue the following commands:

    # reboot
    bash: /sbin/reboot: Input/output error
    # shutdown -r now
    bash: /sbin/shutdown: Input/output error

Obviously, there is a problem with your drive. These commands are failing because the kernel is unable to load the /sbin/reboot and /sbin/shutdown binaries from the disk so that it can execute them.

A fsck on the next boot might be able to correct whatever is wrong with the disk, but first you need to get the system to reboot. If your machine is located at a managed hosting provider then you could submit a reboot ticket, but you'll have to wait for someone to take responsibility.

Wouldn't it be nice if there was a way to ask the kernel to reboot without needing to access the failing drive? Well, there is a way, and it is remarkably simple.

The "magic SysRq key" provides a way to send commands directly to the kernel through the /proc filesystem. It is enabled via a kernel compile time option, CONFIG_MAGIC_SYSRQ, which seems to be standard on most distributions. First you must activate the magic SysRq option:

    echo 1 > /proc/sys/kernel/sysrq

When you are ready to reboot the machine simply run the following:

    echo b > /proc/sysrq-trigger

This does not attempt to unmount or sync filesystems, so it should only be used when absolutely necessary, but if your drive is already failing then that may not be a concern.

In addition to rebooting the system the sysrq trick can be used to dump memory information to the console, sync all filesystems, remount all filesystems in read-only mode, send SIGTERM or SIGKILL to all processes except init, or power off the machine entirely, among other things.

Also, instead of echoing into /proc/sys/kernel/sysrq each time you can activate the magic SysRq key at system boot time using sysctl, where supported:

    echo "kernel.sysrq = 1" >> /etc/sysctl.conf

If you would like to learn more about magic SysRq you can read the sysrq.txt file in the kernel documentation.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

echo 1 >

asr's picture
Submitted by asr (not verified) on Fri, 01/08/2010 - 07:18.

echo 1 > /proc/sys/kernel/sysrq
bash: /proc/sys/kernel/sysrq: No such file or directory

:'(

and that's how you learn not to be such a dick when building your own kernel.

Heartfelt thanks

baski's picture
Submitted by baski (not verified) on Wed, 11/04/2009 - 15:26.

I was butting head for a long time to see how i can do it remotely without going to the site. Thanks once again.. you saved me some green and greened the earth.. :)

Thanks!

Jan Angelovic's picture
Submitted by Jan Angelovic (not verified) on Mon, 03/09/2009 - 14:01.

Thanks a lot! My disk is going to hell but important data was saved. :-) You save me from 100 km long trip to telehouse.

Than you so much for this

morgellons's picture
Submitted by morgellons (not verified) on Sun, 11/16/2008 - 20:58.

Than you so much for this trick. It should come in handy when rebooting my vps! for me site
morgellons

printer problems

Cristian's picture
Submitted by Cristian (not verified) on Mon, 11/03/2008 - 07:57.

Do you happen to know a magic way to reboot my printers? That would be magic trick which I would love to learn... I just hate when printers start behaving very weird and ruining my whole day.

I like this blog.Nice work.

bilety lotnicze's picture
Submitted by bilety lotnicze (not verified) on Fri, 09/05/2008 - 05:14.

I like this blog.Nice work.

Less typing

Craig MIller's picture
Submitted by Craig MIller (not verified) on Mon, 08/25/2008 - 09:29.

If one is truely looking to do less typing and reboot, I suggest:
"init 6"

This however, does require the file system to be functioning (since it will walk thru the /etc/rc?.d/ shutdown scripts).

But really, how often does one need to shut down a system (or reboot) with no filesystem?

# init 6 bash:

Cory Wright's picture
Submitted by Cory Wright on Tue, 08/26/2008 - 10:41. 
    # init 6
    bash: /sbin/init: Input/output error

:)

Also, as several people have already commented, there are lot's of cases where you may need to reboot without accessing the filesystem. I've had to deal with the Dell RAID issue myself.

Excellent information!

Anonymous's picture
Submitted by Anonymous (not verified) on Tue, 08/26/2008 - 08:42.

To Mr. Miller's question: "But really, how often does one need to shut down a system (or reboot) with no filesystem?", I would have to reply more often then we wished. It may just be this particular application, but we have 1500+ instances of a system running Linux offering file shares via nfs with which we encounter this very problem. Being able to remotely reboot such a system so that it can run fsck upon reboot is potentially a great time saver. Without this functionality, the time needed to coordinate with remote sites to physically reboot the boxes may lead to days when this service is unavailable. This also allows us to make a quicker determination of whether we merely have file system corruption, or hardware failure, and dispatch maintenance when required.

Not that I'm a Linux guru or

TsueDesu's picture
Submitted by TsueDesu (not verified) on Mon, 08/25/2008 - 06:27.

Not that I'm a Linux guru or anything...but if the system drive is failing, Why would you want to reboot the server/system? Wouldn't that render it totally unusable? After all, you're still linked in remotely, so you could rebuild it onto a working drive or at least move the system to a new drive. Or was this implied?

*scratching head in uncertainty*

I'll say its an interesting article and I'm sure there's a use for this for this.

TsueDesu a.k.a This is stick!

Hmm. I was stupid enough to

Mad Cow's picture
Submitted by Mad Cow (not verified) on Sat, 10/25/2008 - 14:30.

Hmm. I was stupid enough to make my a and b nameservers the same host. Now the disk is failing but I can't get to the host until monday at the earliest. This is great. It might just work. ;-)

It didn't work BTW. I have to visit the server on monday anyhow. Ho hum.

Dell RAID Problems

Will Reese's picture
Submitted by Will Reese on Mon, 08/25/2008 - 14:47.

I've had Dell servers with PERC RAID cards crap out under heavy IO and cause the filesystem to become unusable until a reboot. The cards and the disks were fine, but the driver wasn't. Rebooting resolved the issue until we were able to upgrade to better hardware and drivers.

-- Will Reese

sohpet

Anonymous's picture
Submitted by Anonymous (not verified) on Sat, 08/23/2008 - 21:06.

thanks for sites

Sync and umount with magic sysrq keys

messageid's picture
Submitted by messageid (not verified) on Fri, 08/22/2008 - 05:59.

With the Magic SysRQ keys is also possibile to umount and sync all the mounted disk.

If you issue the following key:

R S E I U B

You will: Sync the disk, Term and the Kill all the process, Umount all the mounted disk and the Reboot the system; so you can "emlate" a correct shoutdown process.

I use

cuervo's picture
Submitted by cuervo (not verified) on Sat, 09/06/2008 - 17:10.

alt-sysrq-S, U, S, B. Sync, remount all filesystems R/O, sync again, hard reboot. I typically let around half a second between those. There's some standard sequence involving elephants, but I figured mine out when magic syswreck was first introduced, and stuck with it. (Hint: That means parent's sequence is probably better.)

FYI, from /usr/src/linux*/Documentation/sysrq.txt:

'b' - Will immediately reboot the system without syncing or unmounting
your disks.

'c' - Will perform a kexec reboot in order to take a crashdump.

'd' - Shows all locks that are held.

'e' - Send a SIGTERM to all processes, except for init.

'f' - Will call oom_kill to kill a memory hog process.

'g' - Used by kgdb on ppc and sh platforms.

'h' - Will display help (actually any other key than those listed
above will display help. but 'h' is easy to remember :-)

'i' - Send a SIGKILL to all processes, except for init.

'k' - Secure Access Key (SAK) Kills all programs on the current virtual
console. NOTE: See important comments below in SAK section.

'm' - Will dump current memory info to your console.

'n' - Used to make RT tasks nice-able

'o' - Will shut your system off (if configured and supported).

'p' - Will dump the current registers and flags to your console.

'q' - Will dump a list of all running timers.

'r' - Turns off keyboard raw mode and sets it to XLATE.

's' - Will attempt to sync all mounted filesystems.

't' - Will dump a list of current tasks and their information to your
console.

'u' - Will attempt to remount all mounted filesystems read-only.

'v' - Dumps Voyager SMP processor info to your console.

'w' - Dumps tasks that are in uninterruptable (blocked) state.

'x' - Used by xmon interface on ppc/powerpc platforms.

Not that easy

Anonymous's picture
Submitted by Anonymous (not verified) on Fri, 08/22/2008 - 02:06.

# echo 1 > /proc/sys/kernel/sysrq
bash: /bin/echo: Input/output error

Shell built-in

Anonymous's picture
Submitted by Anonymous (not verified) on Fri, 08/22/2008 - 03:20.

'echo' is also a shell built-in for any decent shell.

Shell built-in

Anonymous's picture
Submitted by Anonymous (not verified) on Fri, 08/22/2008 - 03:19.

'echo' is also a shell built-in for any decent shell.

echo IS built into bash, but

Anonymous's picture
Submitted by Anonymous (not verified) on Fri, 08/22/2008 - 07:11.

echo IS built into bash, but in some distros there's also a /bin/echo which is in the path. Try PATH="" echo > /proc/yourmom

In Bash, the precedence

Anonymous's picture
Submitted by Anonymous (not verified) on Fri, 08/22/2008 - 09:11.

In Bash, the precedence order is:
1. Alias
2. Function (not sure of the order of the first two)
3. Built-in
4. Program found via a $PATH search

The built-in will get used first. Most Linux distros run with Bash as the primary shell now.

Webcast
More Resources
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers

Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.

Learn More

Sponsored by AMD

White Paper
More Resources
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState