RAID-1, Part 1

Part 1 of this two-part series describes
RAID, in which cases RAID-1 is useful, the RAID-1 installation
requirements and how to install RAID-1 when you have an existing
ext2 filesystem. Part 2 covers how to set up RAID-1 with an
existing and a new swap partition, how to boot from a RAID-1
device, how to use a RAID-1 array to facilitate backing up a busy
filesystem or a database that cannot be taken off line for long and
how to set up monitoring scripts that will notify you of
problems.We wrote this article because we could not find a complete
description of the setup process, and we wanted to document what we
learned to make it easier for others to implement RAID-1. We
learned how to implement RAID by reviewing the "Software-RAID
HOWTO" and Usenet correspondence as well as through trial and
error. This article includes information from the HOWTO and Usenet
archives. Please read the HOWTO and the other resources called out
in the references section for a lot more information about
RAID.This article focuses on RAID-1. There are five RAID levels:
Linear mode, RAID-0, RAID-1, RAID-4 and RAID-5. RAID-1 maintains an
exact mirror of the data on one disk on another disk. If one of the
active RAID disks is removed (or fails), the data are still intact.
If there are spare disks available, and if the system survived the
crash, reconstruction of the mirror will begin immediately on one
of the spare disks. If there is no spare disk, the system will
continue to run on the remaining good disk, until you can obtain
and install a replacement disk. RAID-1 is an effective, inexpensive
way to help to ensure that your system stays up when you have a
hard disk failure. One also could use a RAID-1 device to facilitate
backing up a busy filesystem.A RAID-1 device (e.g., /dev/md0) maintains an exact copy
(mirror) of the files in a given partition (e.g. /dev/hda2) on a
separate partition (e.g./dev/hdc2). The Linux RAID code mirrors
partitions, not entire disks. The partitions that make up a RAID
device set should be on separate hard disks. Write performance is
slightly worse than on a single device, because identical copies of
the data written must be sent to every disk in the array. The write
is not complete until all disk writes are finished. Reading may be
faster than without RAID-1, depending on the read-balancing
strategy that is implemented. We did not benchmark our RAID setups.
We are using RAID to provide data redundancy, not to improve disk
performance.We suffered a hard disk failure on our production web server
a few months after we installed RAID-1 on the system. We noticed
that a hard disk partition failed during a routine review of our
system logs (the RAID code is truly transparent, we didn't notice
the failure until three days after it happened). If we were not
using RAID-1, we would have found out when the system crashed. We
have about twelve staff members who work on the server and hundreds
of users who access our web site on a daily basis. They would have
lost time had server gone down. We replaced the failed disk during
scheduled downtime. A second (identical) disk failed a few weeks
later with similar results. This was a much better and less
stressful outcome for all concerned. We have since set up scripts
that monitor the status of the RAID devices and send e-mail alerts
when there are problems.There are three requirements for RAID-1: the kernel must
support RAID-1, the RAID device driver must be compiled into the
kernel or be available as a module and raidtools must be installed.
Technically, you can use RAID with just one hard disk but most
installations use more than one disk.If /proc/mdstat exists, RAID support was compiled into the
kernel. The Personalities listing indicates which RAID devices are
available. For example:

     more /proc/mdstat
     Personalities : [RAID1]
     read_ahead not set
     unused devices: <none>

indicates that the kernel supports RAID and the RAID-1 device
driver is loaded (RAID-1 may have been compiled into the kernel or
have been loaded as a module).
We've set up RAID-1 arrays on Red Hat 7.0, 7.1 and 7.2 and
Debian potato using the 2.4.4, 2.4.9, 2.4.12 and 2.4.17 kernels
that include RAID support, and the 2.2.16-22 kernel with the md
driver 0.90.0 patch. If you are using a kernel that does not
include RAID support, you may use the RAID-patches and raidtools
found at
people.redhat.com/mingo.If you're using IDE for your RAID devices, you should install
them on different controllers. SCSI RAID setups can get away with
using the same bus, but they have a greater chance of a broken disk
taking down the whole machine. And, consider using different
manufacturers (or at least different lots) for the disks in your
RAID array. Our initial array was created with two identical hard
disks. The disks failed within a month of each other.The partition that will be used for the RAID-1 array on the
second disk should be about the same size as that on the first
disk. It must be at least as large as the first disk's partition.
If the second disk's partition is larger, the extra space will not
be used by the RAID-1 device. The smallest partition determines the
size of the RAID-1 device./usr on a RAID-1 DeviceBefore you begin, if the /usr partition is in use and
contains data that you do not want to lose, back up your data. The
following list outlines the steps used to configure /usr as a
RAID-1 array:

1. Partition the second hard disk, if needed.
   Partition sizes should be calculated based on sectors, not
   cylinders. If using fdisk to partition the disks, use the -u
   option. The smallest partition of a mirror set determines that
   metadevice's maximum size. Getting the partition sizes to match up
   is almost impossible. Just try to get them close while making sure
   the second disk's partitions are the same size or larger. You may
   ignore any fdisk warnings about cylinder boundaries, they are a
   problem only if you'll be accessing the disk with DOS/Windows. If
   you want the RAID-1 devices to be mounted during boot, change the
   filesystem type to "fd" (Linux RAID autodetect).
2. Go to single-user mode using the command
   init 1 so that /usr can be unmounted.
   Alternatively, you could stop all processes using the /usr
   partition. The fuser program can help you if you wish to take this
   route.
3. Copy the contents of /usr to /var/usr.save or
   another partition with enough room so that you can restore the
   files after you make the RAID. You will have to make a new ext2
   filesystem on the original /usr (in this case /dev/hda2)
   partition.
4. umount /usr. When you went to single-user mode, the
   processes that were using the /usr partition should have been
   stopped. If you can't unmount /usr, determine which processes are
   still using /usr and stop them.
5. Use fdisk or an equivalent partition manager to
   toggle the file system type of the /usr partition on the original
   disk to fd (Linux RAID autodetect). The new disk was set to type fd
   in step 1.
6. Create /etc/raidtab with an entry for the /usr
   partition array (Listing 1). (Note that if you use Emacs, or some
   other editor that lives in /usr, you'll have to make do with vi or
   something else that lives in /bin.)
7. Make the new RAID-1 /usr array, using
   mkraid /dev/md0. Now type cat
   /proc/mdstat, which should indicate md0 : active
   RAID1 hdc2[1] hda2[0]. You will see a progress meter
   measuring the state of the mirroring process. The RAID code knows
   nothing about what is actually on its disks, so it is now copying
   all the data from /dev/hda2 to /dev/hdc2. This copying happens
   transparently in the background, so you can now go to step 8 and
   format the device.
8. Create an ext2 filesystem on /dev/md0 using the
   command mke2fs /dev/md0. Do not
   mke2fs on the RAID-1 component partitions, in
   this case /dev/hda2 and /dev/hdc2. If you do not create an ext2
   filesystem on /dev/md0, then e2fsck /dev/md0
   will return an error message, something like this:

The filesystem size (according to the superblock) is
2104483 blocks. The physical size of the device is 2104384 blocks.
Either the superblock or the partition table is likely to be corrupt.

This is because mkraid writes the RAID superblock near the
end of the component partitions.
e2fsck does not recognize the RAID
superblock that has caused the physical size to be smaller. You can
mount /dev/md0 at this point, and even use /usr, but the ext2
filesystem superblock contains incorrect information. You may not
notice problems but you should not use the filesystem in this
state. You will not be able to boot and mount /dev/md0 unless you
turn off the filesystem checking by making the appropriate entry in
fstab (e.g., /dev/md0 /usr ext2 defaults 1 0).
The 0 at the end of the line causes e2fsck to be skipped. Do not do
this unless you have to fix your RAID. Make /dev/md0 an ext2
filesystem.

md: RAID1 personality
registered as nr 3', 'md: md driver 0.90.0 MAX_MD_DEVS=256,
MD_SB_DISKS=27', 'md: Autodetecting RAID arrays.', 'md: considering
hdc2 ...', 'md: adding hdc2 ...', 'md: adding hda2 ...', 'md: created
md0', 'RAID1: RAID set md0 active with 2 out of 2 mirrors', '......',
'fsck: /dev/md0: clean', 'rc.sysinit: Mounting local filesystems:
succeeded.

Instead of copying the /usr partition to a temporary
location, you may rely on the failed-disk directive and mark the
existing disk (the one with the data) as failed-disk 0 and the new
disk as RAID-disk 1. Then mkraid and make an
ext2 filesystem on the RAID array. The failed disk will be ignored
by the RAID code. Then mount the RAID device on a temporary
mountpoint as a normal ext2 partition, and copy over the /usr
content. Umount the failed disk, and use
raidhotadd to add it to the new RAID array.
Mount the RAID device as /usr, and go back to multi-user status.
This is a more risky procedure than copying the files to a
different partition. If you make a mistake in raidtab, you could
overwrite the existing partition. And if the disk is completely
full, an undesirable situation in any case, you may have problems
when the RAID superblock is written; we did not test this
case.Listing 1. The /etc/raidtab with an entry for the /usr
partition raiddev /dev/md0

        RAID-level      1
        nr-RAID-disks   2
        persistent-superblock 1
        chunk-size      4
        device          /dev/hda2
        RAID-disk       0
        device          /dev/hdc2
        RAID-disk       1
mdctl

The above example uses raidtools version 0.90.0 and relies on
/proc/mdstat and the system logs to monitor the RAID-1 arrays.
mdctl is another tool that can be
used to create, control and monitor Linux md devices (aka RAID
arrays) . We did not use mdctl to create our RAID-1 arrays, but we
did use it to learn more about our RAID devices. You can get
mdctl-0.5 at
www.cse.unsw.edu.au/~neilb/source/mdctl.
To learn more about mdctl search http://groups.google.com/ using
the query +author:neilb@cse.unsw.edu.au +mdctl.Use the command mdctl --examine /dev/hda2
to print the content of the md superblock on the device. Use the
command mdctl --detail /dev/md0 to print the
details of a given md array.To deactivate the RAID-1 array, use the command
raidstop /dev/md0. In some cases this does not
work. For example, /dev/md0 is not mounted, but an error message
says device busy. In this case, mdctl
--stop /dev/md0 may work. Note, unmount the RAID-1 device
before you stop the array. If you don't, your system may hang.
Deactivating the RAID device halts all synchronization. An inactive
RAID device cannot be mounted.ReferencesThe
Software-RAID HOWTO"Kernel Korner: The Linux RAID-1,
4, 5 Code", Linux Journal, December
1997.Usenet; one of the archives is
groups.google.com.
The following search queries may help to get you started: +RAID1,
+failed-disk +linux, +RAID1 +swap +linux and +linux +RAID
+superblock.If you're curious about the RAID superblock, you can find a
description in the mdctl-0.5 source code. Take a look at the file
md_p.h. You also can take a look at the kernel mddriver source code
files including /usr/src/linux/drivers/md/md.c.AcknowledgementsThanks to those who developed the Linux RAID code (see
drivers/md/md.c for names), Jakob Østergaard for the "The
Software-RAID HOWTO", the Usenet correspondents and Niel Brown for
mdctl.Joe Edwards, PE, PhD wrote
his first useful program using FORTRAN on an IBM 370 almost 30
years ago. The program performed forensic analysis of X-ray
diffraction data. He started using Linux in 1995. He is the lead
programmer, sysadmin and dba for the
GeneTests-GeneClinics
Projects at the University of Washington.Audin Malmin is a programmer
and sysadmin for the GeneTests-GeneClinics Projects at the
University of Washington. His first programming experiences were on
his dad's Timex Sinclair 1000. He first experimented with Linux in
1996 on his 386sx16 with 3MB of RAM.Ron Shaker is the lead
programmer on the GeneSeek Project at the University of Washington.
He has worked as a sysadmin, dba and systems engineer over the past
13 years and began using UNIX in 1988.