Monitoring Hard Disks with SMART

One of your hard disks might be trying to tell you it's not long for this world. Install software that lets you know when to replace it.

It's a given that all disks eventually die, and it's easy to see why. The platters in a modern disk drive rotate more than a hundred times per second, maintaining submicron tolerances between the disk heads and the magnetic media that store data. Often they run 24/7 in dusty, overheated environments, thrashing on heavily loaded or poorly managed machines. So, it's not surprising that experienced users are all too familiar with the symptoms of a dying disk. Strange things start happening. Inscrutable kernel error messages cover the console and then the system becomes unstable and locks up. Often, entire days are lost repeating recent work, re-installing the OS and trying to recover data. Even if you have a recent backup, sudden disk failure is a minor catastrophe.

Many users and system administrators don't know that Self-Monitoring, Analysis and Reporting Technology systems (SMART) are built in to most modern ATA and SCSI hard disks. SMART disk drives internally monitor their own health and performance. In many cases, the disk itself provides advance warning that something is wrong, helping to avoid the scenario described above. Most implementations of SMART also allow users to perform self-tests on the disk and to monitor a number of performance and reliability attributes.

By profession I am a physicist. My research group runs a large computing cluster with 300 nodes and 600 disk drives, on which more than 50TB of physics data are stored. I became interested in SMART several years ago when I realized it could help reduce downtime and keep our cluster operating more reliably. For about a year I have been maintaining an open-source package called smartmontools, a spin-off of the UCSC smartsuite package, for this purpose.

In this article, I explain how to use smartmontools' smartctl utility and smartd dæmon to monitor the health of a system's disks. See smartmontools.sourceforge.net for download and installation instructions and consult the WARNINGS file for a list of problem disks/controllers. Additional documentation can be found in the man pages (man smartctl and man smartd) and on the Web page.

Versions of smartmontools are available for Slackware, Debian, SuSE, Mandrake, Gentoo, Conectiva and other Linux distributions. Red Hat's existing products contain the UCSC smartsuite versions of smartctl and smartd, but the smartmontools versions will be included in upcoming releases.

To understand how smartmontools works, it's helpful to know the history of SMART. The original SMART spec (SFF-8035i) was written by a group of disk drive manufacturers. In Revision 2 (April 1996) disks keep an internal list of up to 30 Attributes corresponding to different measures of performance and reliability, such as read and seek error rates. Each Attribute has a one-byte normalized value ranging from 1 to 253 and a corresponding one-byte threshold. If one or more of the normalized Attribute values less than or equal to its corresponding threshold, then either the disk is expected to fail in less than 24 hours or it has exceeded its design or usage lifetime. Some of the Attribute values are updated as the disk operates. Others are updated only through off-line tests that temporarily slow down disk reads/writes and, thus, must be run with a special command. In late 1995, parts of SFF-8035i were merged into the ATA-3 standard.

Starting with the ATA-4 standard, the requirement that disks maintain an internal Attribute table was dropped. Instead, the disks simply return an OK or NOT OK response to an inquiry about their health. A negative response indicates the disk firmware has determined that the disk is likely to fail. The ATA-5 standard added an ATA error log and commands to run disk self-tests to the SMART command set.

To make use of these disk features, you need to know how to use smartmontools to examine the disk's Attributes (most disks are backward-compatible with SFF-8035i), query the disk's health status, run disk self-tests, examine the disk's self-test log (results of the last 21 self-tests) and examine the disk's ATA error log (details of the last five disk errors). Although this article focuses on ATA disks, additional documentation about SCSI devices can be found on the smartmontools Web page.

To begin, give the command smartctl -a /dev/hda, using the correct path to your disk, as root. If SMART is not enabled on the disk, you first must enable it with the -s on option. You then see output similar to the output shown in Listings 1–5.

The first part of the output (Listing 1) lists model/firmware information about the disk—this one is an IBM/Hitachi GXP-180 example. Smartmontools has a database of disk types. If your disk is in the database, it may be able to interpret the raw Attribute values correctly.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

re:S.M.A.R.T

Rick Paste's picture

Isn't the BIOS just activating the function and then some other piece of software has to do the checking. I am not sure, if the BIOS is giving you an alert, when harddisks are getting damaged.

The BIOS does alert you if

Anonymous's picture

The BIOS does alert you if the drive is about to fail. However, if the machine is rarely restarted, we still need the monitoring software of course :-/

re:S.M.A.R.T

Thomas Rice's picture

Oh yes - thanks a lot for the informaton.

Smart Tool - Active

Anonymous's picture

Dear Mr. Bruce,

we are using the smart tools to test the 2.5" Fugitsu HD on one of embedded cPCI dual P-III board. We used it due to a problem we met - freezed "black screen" after POST (happened during the OS loading). It happened only with XP OS. The "Black screen" lead the user to manualy reset the card and hope it will not happened again in the next power cycle.

The command line used to run the test was smartctl -t long –d ata /dev/hda - the test run 40 minutes ! and fix the problem - but we have no idea what cause the problem and how this tool solve it if it should be only a TEST TOOL.

Can you assist with the following:
A. What tests is it running? read only? writes? does it change the HD controller working parameters?
Can you specifiy locations of the strings written/readden ?or this is random ?
B. Does it change something in the operating system? if it does, then what?
C. Does it change the Disk structure? if yes, how ?

Thank you
Shahar

A: SMART extended self test.

ballen's picture

A: SMART extended self test. No it does not change drive parameters.
B. No, it does not change something in the OS.
C. It does not change the disk structure. However as the previous responder says, when the self-test is run, the disk firmware may find and correct some types of problems on the disk surface.

Bruce Allen

As far as I know the given co

Anonymous's picture

As far as I know the given command line runs the diagnostic procedure *embedded* inside the hd firmware so it is to hd manufacturers discretion what is actually done. What may be possible in your case is that there was a bad sector in the disk that couldn't be read most of the times. When you run the procedure it might have happened that the sector could have been read one time and it have been immediately remaped (i.e. its data was moved to the disk spare area and the sector itself marked as unavailable).

Otherwise than that the tool doesn't affect HD state other than smart monitoring enable/disable flag and heath attributes values.

Regards,
DS.

Thanks for the article. Great

Anonymous's picture

Thanks for the article. Great stuff!

Thanks

RalphGL's picture

Thanks for this very useful article!

Maxtor HD: Smart not be enabled

peter farge's picture

Hello Bruce,

thanks for your great work. I have this Maxtor HD:

=== START OF INFORMATION SECTION ===
Device Model: MAXTOR 4K040H2
Firmware Version: A08.1500
User Capacity: 40,037,760,000 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 5
ATA Standard is: ATA/ATAPI-5 T13 1321D revision 1
SMART support is: Available - device has SMART capability.
SMART support is: Disabled

And I cant enable SMART? I use the HD under Win2000. I have tried:

C:\s\bin>smartctl -s on /dev/hda
smartctl version 5.33 [i386-pc-mingw32] Copyright (C) 2002-4 Bruce Allen
Home page is http://smartmontools.sourceforge.net/
=== START OF ENABLE/DISABLE COMMANDS SECTION ===
SMART Enabled.

Its only a note. The HD works fine and I only want to be informed about the status...

Please try a more recent

ballen's picture

Please try a more recent release of smartmontools. There were some teething problems with the initial Windows version, but those have been fixed.

Bruce

Have you checked that it is e

Chris Clemson's picture

Have you checked that it is enabled in the BIOS?
Usually there is a setting which controls if SMART is enabled or not.

Re: Monitoring Hard Disks with SMART

Anonymous's picture

I have a drive that I can't even get into with fdisk, but I can access mounted file systems. It's temperature is indicating 53 celsius, but there is no max-min. It's in my machine next to another drive, and I'm going to try moving them apart. But any thoughts on fdisk not being able to get into the partition table even? It reads the other drive fine.

Re: Monitoring Hard Disks with SMART

ballen's picture

I don't have any idea why fdisk can't provide information. What's the command line to and error message from fdisk?

The temperature min/max isn't provided by all vendors. IBM/Hitachi and Toshiba (recent disks) have this, but many other vendors and older disks either have no temperature information, or (as with your disk) just the current temperature.

Re: Monitoring Hard Disks with SMART

Anonymous's picture

excellent tool, using this now on a possibly faulty hd from our debian server

Re: Monitoring Hard Disks with SMART

ballen's picture

Thank you!

Monitoring Hard Disks with SMART

Anonymous's picture

how are the following 2 situations are dealt by SMART

1. If the server needs to be shutdown and the disks have to be intentionally unmounted when SMART is doing its job ? can it be terminated abruptly ?

2. what happens when the SMART is trying to access a disk thats not responding over NFS ?

Re: Monitoring Hard Disks with SMART

ballen's picture

Here are the answers to your questions:

(1) You can unmount the disk while SMART activity is taking place, with no harmful effects. In fact you can get all smart data (run smartctl for example, or smartd) on an unmounted disk. This is because SMART operates at the level of the disk hardware, whereas 'mounting' refers to file systems on the disk and is at a much higher level. When you ask 'can it be terminated abruptly?' I assume that you mean, can you power down or unmount a disk which is carrying out SMART operations? The anwer is yes.

(2) This question does not make sense. SMART does not 'access a disk'. A utility such as smartctl or smartd 'accesses a disk'. These utilities can only access a disk connected to the local machine, such as /dev/hda or /dev/sda. These utilities can not access a disk 'over the network': see my response (1) above. Linux only allows you to access FILE SYSTEMS over a network (eg, using NFS), not 'raw disks'.

smartd may spin up the disk

Anonymous's picture

Today I finally found out why hda was always spinning despite a
hdparm -S 241 /dev/hda
command, and no file systems mounted on that disk.

It's the smartd that comes with Fedora Core 1. Whenever smartd checks
the disk, the disk spins up (in my case it's a Seagate ST360020A, but
this seems to be a general problem). Newer versions of smartd have
the -n option that allows suppressing the check for disks that are in
various power-saving modes, but not the version that comes with Fedora
Core 1. The workarounds are: use the newer smartd with the
appropriate option, don't use smartd for this particular disk, or
disable smartd altogether. Being lazy, I chose the latter, and now
hda spins down and stays that way.

Re: smartd may spin up the disk

ballen's picture

This makes sense.

As you said, smartd can be told not to spin up non-spinning disk. To implement this for your Fedora system, you need to:

(0) save a copy of smartd.conf:
cp /etc/smartd.conf /etc/smartd.conf.save
(1) download the current stable release (5.30)
(2) install it (over the Fedora smartmontools which are part of kernel-utils):
rpm -ivh --nodeps --force smartmontools-5.30-1.i386.rpm
(3) cp /etc/smartd.conf.save /etc/smartd.conf
(4) add '-n standby' after /dev/hda in smartd.conf
(5) restart smartd:
/etc/rc.d/init.d/smartd restart

Bruce

Re: Monitoring Hard Disks with SMART

Anonymous's picture

I read this article with great interest, then I went and tried it out, and viola! one of my servers reported a problem. With the long diagnostics output in hand, the vendor replaced the disk, no questions. I then deployed this across all my machines, including a lab, plus client machines and bingo! found another one.

As before, the vendor ponied up the replacement disk without any difficulties.

It was a little awkward explaining to the client that I needed to replace the disk BEFORE it died. That's a new concept, one they are not familiar with, to say the least.

I now run the smartctrl demon and feel much better knowing it's keeping tabs on my disks. Prior to this, ignorance was NOT bliss, it was just this nagging feeling lurking in the back of my mind, waiting for a disk failure.

Excellent article! Thanks for bringing this to our attention. This one article alone more than paid for my whole subscription. As is typical of every issue of LJ, I always find several articles I just have to try out. A real 'hands-on' magazine.

-tim
timbornATdeltacfax.com

Re: Monitoring Hard Disks with SMART

ballen's picture

Tim,

thanks for your comment - I'm glad this has been useful.

If your disks are failing their self-tests, but still have 'passing' SMART health status, it may just be that they have one or more sectors that are unreadable. Next time this happens, you can probably fix it by forcing the disk to reallocate the sectors. Two ways to do this are:

(1) with a scalpel: see
http://smartmontools.sourceforge.net/BadBlockHowTo.txt

(2) with a hatchet:
dd if=/dev/zero of=/dev/hda bs=4k
but note that this will DESTROY ALL INFORMATION ON THE DISK!
(sorry for shouting.)

This may be easier than returning the disk to the vendor for replacement.

Bruce

Re: Monitoring Hard Disks with SMART

setyoko's picture

A good monitoring for disk, and it will be more powerful when smartmontools on one server can read all disk health on a network. :)

Re: Monitoring Hard Disks with SMART

Anonymous's picture

Well, indeed you can get that functionality by way of redirecting syslog output on each host to a master server.

Cheers,

Pancho

Re: Monitoring Hard Disks with SMART

Anonymous's picture

I'm not sure if your comment about smartmontools on one server reading the health of all disks on a network is serious or not! It might be possible if
the raw disk SMART data were made available in some exportable file-system like way, but I doubt that this will happen anytime soon. (I did post a patch to
the linux kernel mailing list for this purpose, but didn't get any response.)

Bruce

Re: Monitoring Hard Disks with SMART

setyoko's picture

Of course i'm serious, in my mind if smartmontools can work just like snmpd with its fork, it would be better.

--Aria--

Re: Monitoring Hard Disks with SMART

Anonymous's picture

I picked up a copy of the Linux Journal in an airport in Nashville, TN. I was impressed! This is a great publication.

Few weeks later, and I'm wondering, "how's it that all my values are over the threshold..." and I remembered the article.

But I had lost my magazine! I was distressed! Then I found this web site, and lo and behold! The article was printed here too! Thats great! Not all publications will do such a nice service.

Thanks for a great article, and for posting it on the net!

Re: Monitoring Hard Disks with SMART

Anonymous's picture

I'm also happy that Linux Journal allowed me to post a copy of the article here! -- Bruce

Re: Monitoring Hard Disks with SMART

motie's picture

Thank you for turning me on to and maintaining a very useful tool. This was my first introduction to SMART and smartmontools.

Re: Monitoring Hard Disks with SMART

Anonymous's picture

You're welcome!

Re: Monitoring Hard Disks with SMART

Anonymous's picture

Thanks for a great article explaining something I'd always been suspicious of until I started having an HD fail this morning.

Would be interesting to see the smartd tools integrated in with the DBUS work.
http://www.freedesktop.org/software/dbus

Re: Monitoring Hard Disks with SMART

Anonymous's picture

how are the following 2 situations are dealt by SMART

1. If the server needs to be shutdown and the disks have to be intentionally unmounted when SMART is doing its job ? can it be terminated abruptly ?

2. what happens when the SMART is trying to access a disk thats not responding over NFS ?

Re: Monitoring Hard Disks with SMART

Anonymous's picture

Could you please be a more explicit about how you'd like
for smartmontools to integrate with DBUS?

It would also be helpful if you could send these suggestions to the
smartmontools support mailing list: smartmontools-support@lists.sourceforge.net

Additonal information about the mailing list is at:
http://lists.sourceforge.net/lists/listinfo/smartmontools-support

Sourceforge.net is not access

Anonymous's picture

Sourceforge.net is not accessible. How can I get source for SMART tools ?

smartmontools

Anonymous's picture

On my system, Mandriva 2006, it is part of this package:
smartmontools

You can try to find any similar precompiled package for your system.

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState