Monitoring Hard Disks with SMART

One of your hard disks might be trying to tell you it's not long for this world. Install software that lets you know when to replace it.

The final part of the smartctl output (Listing 5) is a report of the self-tests run on the disk. These show two types of self-tests, short and long. (ATA-6/7 disks also may have conveyance and selective self-tests.) These can be run with the commands smartctl -t short /dev/hda and smartctl -t long /dev/hda and do not corrupt data on the disk. Typically, short tests take only a minute or two to complete, and long tests take about an hour. These self-tests do not interfere with the normal functioning of the disk, so the commands may be used for mounted disks on a running system. On our computing cluster nodes, a long self-test is run with a cron job early every Sunday morning. The entries in Listing 5 all are self-tests that completed without errors; the LifeTime column shows the power-on age of the disk when the self-test was run. If a self-test finds an error, the Logical Block Address (LBA) shows where the error occurred on the disk. The Remaining column shows the percentage of the self-test remaining when the error was found. If you suspect that something is wrong with a disk, I strongly recommend running a long self-test to look for problems.

The smartctl -t offline command can be used to carry out off-line tests. These off-line tests do not make entries in the self-test log. They date back to the SFF-8035i standard, and update values of the Attributes that are not updated automatically under normal disk operation (see the UPDATED column in Listing 3). Some disks support automatic off-line testing, enabled by smartctl -o on, which automatically runs an off-line test every few hours.

The SMART standard provides a mechanism for running disk self-tests and for monitoring aspects of disk performance. Its main shortcoming is that it doesn't provide a direct mechanism for informing the OS or user if problems are found. In fact, because disk SMART status frequently is not monitored, many disk problems go undetected until they lead to catastrophic failure. Of course, you can monitor disks on a regular basis using the smartctl utility, as I've described, but this is a nuisance.

The remaining part of the smartmontools package is the smartd dæmon that does regular monitoring for you. It monitors the disk's SMART data for signs of problems. It can be configured to send e-mail to users or system administrators or to run arbitrary scripts if problems are detected. By default, when smartd is started, it registers the system's disks. It then checks their status every 30 minutes for failing Attributes, failing health status or increased numbers of ATA errors or failed self-tests and logs this information with SYSLOG in /var/log/messages by default.

You can control and fine-tune the behavior of smartd using the configuration file /etc/smartd.conf. This file is read when smartd starts up, before it forks into the background. Each line contains Directives pertaining to a different disk. The configuration file on our computing cluster nodes look like this:

# /etc/smartd.conf config file
/dev/hda -S on -o on -a -I 194 -m sense@phys.uwm.edu
/dev/hdc -S on -o on -a -I 194 -m sense@phys.uwm.edu

The first column indicates the device to be monitored. The -o on Directive enables the automatic off-line testing, and the -S on Directive enables automatic Attribute autosave. The -m Directive is followed by an e-mail address to which warning messages are sent, and the -a Directive instructs smartd to monitor all SMART features of the disk. In this configuration, smartd logs changes in all normalized attribute values. The -I 194 Directive means ignore changes in Attribute #194, because disk temperatures change often, and it's annoying to have such changes logged on a regular basis.

Normally smartd is started by the normal UNIX init mechanism. For example, on Red Hat distributions, /etc/rc.d/init.d/smartd start and /etc/rc.d/init.d/smartd stop can be used to start and stop the dæmon.

Further information about the smartd and its config file can be found in the man page (man smartd), and summaries can be found with the commands smartd -D and smartd -h. For example, the -M test Directive sends a test e-mail warning message to confirm that warning e-mail messages are delivered correctly. Other Directives provide additional flexibility, such as monitoring changes in raw Attribute values.

What should you do if a disk shows signs of problems? What if a disk self-test fails or the disk's SMART health status fails? Start by getting your data off the disk and on to another system as soon as possible. Second, run some extended disk self-tests and see if the problem is repeatable at the same LBA. If so, something probably is wrong with the disk. If the disk has failing SMART health status and is under warranty, the vendor usually will replace it. If the disk is failing its self-tests, many manufacturers provide specialized disk health programs, for example, Maxtor's PowerMax or IBM's Drive Fitness Test. Sometimes these programs actually can repair a disk by remapping bad sectors. Often, they report a special error code that can be used to get a replacement disk.

This article has covered the basics of smartmontools. To learn more, read the man pages and Web page, and then write to the support mailing list if you need further help. Remember, smartmontools is no substitute for backing up your data. SMART cannot and does not predict all disk failures, but it often provides clues that something is amiss and has helped to keep many computing clusters operating reliably.

Several developers are porting smartmontools to FreeBSD, Darwin and Solaris, and we recently have added extensions to allow smartmontools to monitor and control the ATA disks behind 3ware RAID controllers. If you would like to contribute to the development of smartmontools, write to the support mailing list. We especially are interested in information about the interpretation and meaning of vendor-specific SMART Attribute and raw values.

Bruce Allen is a professor of Physics at the University of Wisconsin - Milwaukee. He does research work on gravitational waves and the very early universe, and he has built several large Linux clusters for data analysis use.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

re:S.M.A.R.T

Rick Paste's picture

Isn't the BIOS just activating the function and then some other piece of software has to do the checking. I am not sure, if the BIOS is giving you an alert, when harddisks are getting damaged.

The BIOS does alert you if

Anonymous's picture

The BIOS does alert you if the drive is about to fail. However, if the machine is rarely restarted, we still need the monitoring software of course :-/

re:S.M.A.R.T

Thomas Rice's picture

Oh yes - thanks a lot for the informaton.

Smart Tool - Active

Anonymous's picture

Dear Mr. Bruce,

we are using the smart tools to test the 2.5" Fugitsu HD on one of embedded cPCI dual P-III board. We used it due to a problem we met - freezed "black screen" after POST (happened during the OS loading). It happened only with XP OS. The "Black screen" lead the user to manualy reset the card and hope it will not happened again in the next power cycle.

The command line used to run the test was smartctl -t long –d ata /dev/hda - the test run 40 minutes ! and fix the problem - but we have no idea what cause the problem and how this tool solve it if it should be only a TEST TOOL.

Can you assist with the following:
A. What tests is it running? read only? writes? does it change the HD controller working parameters?
Can you specifiy locations of the strings written/readden ?or this is random ?
B. Does it change something in the operating system? if it does, then what?
C. Does it change the Disk structure? if yes, how ?

Thank you
Shahar

A: SMART extended self test.

ballen's picture

A: SMART extended self test. No it does not change drive parameters.
B. No, it does not change something in the OS.
C. It does not change the disk structure. However as the previous responder says, when the self-test is run, the disk firmware may find and correct some types of problems on the disk surface.

Bruce Allen

As far as I know the given co

Anonymous's picture

As far as I know the given command line runs the diagnostic procedure *embedded* inside the hd firmware so it is to hd manufacturers discretion what is actually done. What may be possible in your case is that there was a bad sector in the disk that couldn't be read most of the times. When you run the procedure it might have happened that the sector could have been read one time and it have been immediately remaped (i.e. its data was moved to the disk spare area and the sector itself marked as unavailable).

Otherwise than that the tool doesn't affect HD state other than smart monitoring enable/disable flag and heath attributes values.

Regards,
DS.

Thanks for the article. Great

Anonymous's picture

Thanks for the article. Great stuff!

Thanks

RalphGL's picture

Thanks for this very useful article!

Maxtor HD: Smart not be enabled

peter farge's picture

Hello Bruce,

thanks for your great work. I have this Maxtor HD:

=== START OF INFORMATION SECTION ===
Device Model: MAXTOR 4K040H2
Firmware Version: A08.1500
User Capacity: 40,037,760,000 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 5
ATA Standard is: ATA/ATAPI-5 T13 1321D revision 1
SMART support is: Available - device has SMART capability.
SMART support is: Disabled

And I cant enable SMART? I use the HD under Win2000. I have tried:

C:\s\bin>smartctl -s on /dev/hda
smartctl version 5.33 [i386-pc-mingw32] Copyright (C) 2002-4 Bruce Allen
Home page is http://smartmontools.sourceforge.net/
=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Enabled.

Its only a note. The HD works fine and I only want to be informed about the status...

Please try a more recent

ballen's picture

Please try a more recent release of smartmontools. There were some teething problems with the initial Windows version, but those have been fixed.

Bruce

Have you checked that it is e

Chris Clemson's picture

Have you checked that it is enabled in the BIOS?
Usually there is a setting which controls if SMART is enabled or not.

Re: Monitoring Hard Disks with SMART

Anonymous's picture

I have a drive that I can't even get into with fdisk, but I can access mounted file systems. It's temperature is indicating 53 celsius, but there is no max-min. It's in my machine next to another drive, and I'm going to try moving them apart. But any thoughts on fdisk not being able to get into the partition table even? It reads the other drive fine.

Re: Monitoring Hard Disks with SMART

ballen's picture

I don't have any idea why fdisk can't provide information. What's the command line to and error message from fdisk?

The temperature min/max isn't provided by all vendors. IBM/Hitachi and Toshiba (recent disks) have this, but many other vendors and older disks either have no temperature information, or (as with your disk) just the current temperature.

Re: Monitoring Hard Disks with SMART

Anonymous's picture

excellent tool, using this now on a possibly faulty hd from our debian server

Re: Monitoring Hard Disks with SMART

ballen's picture

Thank you!

Monitoring Hard Disks with SMART

Anonymous's picture

how are the following 2 situations are dealt by SMART

1. If the server needs to be shutdown and the disks have to be intentionally unmounted when SMART is doing its job ? can it be terminated abruptly ?

2. what happens when the SMART is trying to access a disk thats not responding over NFS ?

Re: Monitoring Hard Disks with SMART

ballen's picture

Here are the answers to your questions:

(1) You can unmount the disk while SMART activity is taking place, with no harmful effects. In fact you can get all smart data (run smartctl for example, or smartd) on an unmounted disk. This is because SMART operates at the level of the disk hardware, whereas 'mounting' refers to file systems on the disk and is at a much higher level. When you ask 'can it be terminated abruptly?' I assume that you mean, can you power down or unmount a disk which is carrying out SMART operations? The anwer is yes.

(2) This question does not make sense. SMART does not 'access a disk'. A utility such as smartctl or smartd 'accesses a disk'. These utilities can only access a disk connected to the local machine, such as /dev/hda or /dev/sda. These utilities can not access a disk 'over the network': see my response (1) above. Linux only allows you to access FILE SYSTEMS over a network (eg, using NFS), not 'raw disks'.

smartd may spin up the disk

Anonymous's picture

Today I finally found out why hda was always spinning despite a
hdparm -S 241 /dev/hda
command, and no file systems mounted on that disk.

It's the smartd that comes with Fedora Core 1. Whenever smartd checks
the disk, the disk spins up (in my case it's a Seagate ST360020A, but
this seems to be a general problem). Newer versions of smartd have
the -n option that allows suppressing the check for disks that are in
various power-saving modes, but not the version that comes with Fedora
Core 1. The workarounds are: use the newer smartd with the
appropriate option, don't use smartd for this particular disk, or
disable smartd altogether. Being lazy, I chose the latter, and now
hda spins down and stays that way.

Re: smartd may spin up the disk

ballen's picture

This makes sense.

As you said, smartd can be told not to spin up non-spinning disk. To implement this for your Fedora system, you need to:

(0) save a copy of smartd.conf:
cp /etc/smartd.conf /etc/smartd.conf.save
(1) download the current stable release (5.30)
(2) install it (over the Fedora smartmontools which are part of kernel-utils):
rpm -ivh --nodeps --force smartmontools-5.30-1.i386.rpm
(3) cp /etc/smartd.conf.save /etc/smartd.conf
(4) add '-n standby' after /dev/hda in smartd.conf
(5) restart smartd:
/etc/rc.d/init.d/smartd restart

Bruce

Re: Monitoring Hard Disks with SMART

Anonymous's picture

I read this article with great interest, then I went and tried it out, and viola! one of my servers reported a problem. With the long diagnostics output in hand, the vendor replaced the disk, no questions. I then deployed this across all my machines, including a lab, plus client machines and bingo! found another one.

As before, the vendor ponied up the replacement disk without any difficulties.

It was a little awkward explaining to the client that I needed to replace the disk BEFORE it died. That's a new concept, one they are not familiar with, to say the least.

I now run the smartctrl demon and feel much better knowing it's keeping tabs on my disks. Prior to this, ignorance was NOT bliss, it was just this nagging feeling lurking in the back of my mind, waiting for a disk failure.

Excellent article! Thanks for bringing this to our attention. This one article alone more than paid for my whole subscription. As is typical of every issue of LJ, I always find several articles I just have to try out. A real 'hands-on' magazine.

-tim
timbornATdeltacfax.com

Re: Monitoring Hard Disks with SMART

ballen's picture

Tim,

thanks for your comment - I'm glad this has been useful.

If your disks are failing their self-tests, but still have 'passing' SMART health status, it may just be that they have one or more sectors that are unreadable. Next time this happens, you can probably fix it by forcing the disk to reallocate the sectors. Two ways to do this are:

(1) with a scalpel: see
http://smartmontools.sourceforge.net/BadBlockHowTo.txt

(2) with a hatchet:
dd if=/dev/zero of=/dev/hda bs=4k
but note that this will DESTROY ALL INFORMATION ON THE DISK!
(sorry for shouting.)

This may be easier than returning the disk to the vendor for replacement.

Bruce

Re: Monitoring Hard Disks with SMART

setyoko's picture

A good monitoring for disk, and it will be more powerful when smartmontools on one server can read all disk health on a network. :)

Re: Monitoring Hard Disks with SMART

Anonymous's picture

Well, indeed you can get that functionality by way of redirecting syslog output on each host to a master server.

Cheers,

Pancho

Re: Monitoring Hard Disks with SMART

Anonymous's picture

I'm not sure if your comment about smartmontools on one server reading the health of all disks on a network is serious or not! It might be possible if
the raw disk SMART data were made available in some exportable file-system like way, but I doubt that this will happen anytime soon. (I did post a patch to
the linux kernel mailing list for this purpose, but didn't get any response.)

Bruce

Re: Monitoring Hard Disks with SMART

setyoko's picture

Of course i'm serious, in my mind if smartmontools can work just like snmpd with its fork, it would be better.

--Aria--

Re: Monitoring Hard Disks with SMART

Anonymous's picture

I picked up a copy of the Linux Journal in an airport in Nashville, TN. I was impressed! This is a great publication.

Few weeks later, and I'm wondering, "how's it that all my values are over the threshold..." and I remembered the article.

But I had lost my magazine! I was distressed! Then I found this web site, and lo and behold! The article was printed here too! Thats great! Not all publications will do such a nice service.

Thanks for a great article, and for posting it on the net!

Re: Monitoring Hard Disks with SMART

Anonymous's picture

I'm also happy that Linux Journal allowed me to post a copy of the article here! -- Bruce

Re: Monitoring Hard Disks with SMART

motie's picture

Thank you for turning me on to and maintaining a very useful tool. This was my first introduction to SMART and smartmontools.

Re: Monitoring Hard Disks with SMART

Anonymous's picture

You're welcome!

Re: Monitoring Hard Disks with SMART

Anonymous's picture

Thanks for a great article explaining something I'd always been suspicious of until I started having an HD fail this morning.

Would be interesting to see the smartd tools integrated in with the DBUS work.
http://www.freedesktop.org/software/dbus

Re: Monitoring Hard Disks with SMART

Anonymous's picture

how are the following 2 situations are dealt by SMART

1. If the server needs to be shutdown and the disks have to be intentionally unmounted when SMART is doing its job ? can it be terminated abruptly ?

2. what happens when the SMART is trying to access a disk thats not responding over NFS ?

Re: Monitoring Hard Disks with SMART

Anonymous's picture

Could you please be a more explicit about how you'd like
for smartmontools to integrate with DBUS?

It would also be helpful if you could send these suggestions to the
smartmontools support mailing list: smartmontools-support@lists.sourceforge.net

Additonal information about the mailing list is at:
http://lists.sourceforge.net/lists/listinfo/smartmontools-support

Sourceforge.net is not access

Anonymous's picture

Sourceforge.net is not accessible. How can I get source for SMART tools ?

smartmontools

Anonymous's picture

On my system, Mandriva 2006, it is part of this package:
smartmontools

You can try to find any similar precompiled package for your system.

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState