High Availability Linux with Software RAID

Turn your machine into an HA server after you test it out on a VMware setup.

RAID, redundant array of independent (or inexpensive) disks, is a system that employs two or more disk drives in combination, through hardware or software, for performance and fault tolerance. RAID has a number of different configurations referred to as levels. The most common RAID levels and their functions are:

  • Level 0: data striping, no redundancy.

  • Level 1: disk mirroring.

  • Level 3: similar to 0 but one specific disk is used to stripe data.

  • Level 5: low-level data striping across all disks with stripe error correction.

For more information, refer to www.acnc.com/04_01_00.html for a thorough discussion on the various RAID levels.

Data striping is the ability to spread disk writes across multiple disks. This alone can result in improved performance as well the ability to create one large volume from multiple disks. For instance, if you had nine 6GB drives, you ordinarily would be forced to create at least nine partitions when configuring your system. This partitioning scheme, however, may not make sense for your situation. If you created a RAID 0 out of the nine drives, it would appear to the system as one 54GB drive, which you could then partition as you saw fit. In this scenario, though, if one disk fails then the entire array would fail.

Disk mirroring uses two drives at a time and duplicates exactly one drive to the other. This duplication provides hardware redundancy; if one drive fails, the other can continue to operate independently. Software errors can propagate across the mirror, however, corrupting both disks.

Level 3 assigns one disk in the array to be used for parity (error correction), and the data is then striped across all the other disks in the array. The advantage here is any one disk in the array can fail without any data loss. However, you must give up one disk's worth of space for error correction. Level 3 does not work well with a software RAID solution, and it also has performance drawbacks, as compared to Level 5.

Level 5 stripes data and the error correcting parity data across all the disks in the array. As a result, one disk can fail without loss of data. When this happens, the RAID is said to be operating in degraded mode. If more than one disk fails at the same time, though, the entire array will fail.

This article focuses on using software RAID Level 5 under a fresh installation of Red Hat 8.0 and testing the fault tolerance of the RAID. RAID support for Linux has matured over the years, and the ability to install a system that can boot into a RAID-configured set of disks is standard.

Before actually rebuilding my server with RAID 5, I wanted to be able to test out the installation, tools and failure modes in a safe environment. I also wanted the tests to be as close as possible to the real configuration of my physical hardware.

I have been using VMware (www.vmware.com) since its first beta release in the late 1990s. I highly recommend it for anyone who has to develop on multiple platforms or who needs to do any type of testing on multiple platforms. Using VMware, I was able to set up a Linux virtual machine with six 9GB SCSI drives (as are found on my server) on a machine with only one real physical IDE drive.

As we will see, creating a high availability (HA) Linux server using RAID 5 is a pretty straightforward process. There is one catch, however; you must have at least one native partition that contains the /boot directory. This has to do with the kernel needing to load the drivers that support RAID from a native disk before it can actually mount the RAID. This little detail makes things interesting. Namely, affects the way the drives in the RAID are partitioned and how you recover from a failure of the particular drive that contains the native partition.

Installing with RAID

In order to configure my Linux VM to match my physical machine as closely as possible, I created six 9GB SCSI drives (Figure 1). One of the nice things about creating the virtual drives is they do not initially take up as much space as you assign to them. Instead, the files grow to accommodate data placed on them inside the VM. So, as far as the VM is concerned, it has 54GB at its disposal. But the complete test installation takes up about 1GB of physical space from my actual hard drive.

Figure 1. VM Configuration

After configuring VMware to reference the physical CD-ROM drive for the VM's CD-ROM drive, I placed the first Red Hat 8.0 disk into the drive and powered on the VM. There are a few partitioning requirements at this step. First, each partition used in a RAID volume should be the same size. Second, one partition on one of the drives should be native and should mount at /boot. Third, for RAID 5, one partition's worth of space in a RAID volume needs to be "sacrificed" to account for parity (error correction) data. Because my physical machine has 512MB of RAM, I wanted to have 1GB of swap space. Table 1 shows my partitioning scheme.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

backup of /boot partition and MBR on second disk rather than CD?

Anonymous's picture

Is there any reason that the following wouldn't work as ab
alternative to using a boot CD to back up the /boot partition and master boot record in case of a failure on the first disk:

  • Maintain a copy of the boot partition on the second disk and a copy of the boot manager MBR in the MBR area of the second disk (presumably configured to use the boot partition on the second disk).
  • In case of a failure of the first disk, use the BIOS to switch to booting from the second disk by toggling "bootable" bits on the partitions?

    (Can BIOSes typically boot from a second disk?)

edit which /etc/fstab?

Anonymous's picture

In your second test and its recovery steps, you say to edit
/etc/fstab to comment out the /boot entry.

Does the boot fail after the RAID drivers/modules are
loaded, so that the volume containing /etc/fstab is available?

Re: High Availability Linux with Software RAID

Anonymous's picture

using soft raid for swap is waist of CPU,

linux can do the same without soft raid:

just append to all swap partitions "priority=1"

and linux will use them as they were a part of striped soft raid.

Re: High Availability Linux with Software RAID

Anonymous's picture

In case of _real_ drive fail, Linux can (and, imho, will in 99.99%) panic.

Why?

In our case drive didn't responded, stupid scsi driver tried to reset scsi adapter, then kernel died...

Certanly, this is far better that lost of _full_ filesystem, but..

Hardware raid is _only_ choise for servers ...

Re: High Availability Linux with Software RAID

Anonymous's picture

I've used RAID and forced failures in dozens of ways and NEVER had a kernel panic. This is with adaptec and Sumbios controllers, and with basically unplugging the drive from a hot shoe while the server was running and serving requests (test environment, as well as actual failures in the real environment)

I did however, know a coworker using a HW RAID controller who had it mark two disks bad because the cable to them had slipped off while the server was being moved. Guess who had to rebuild and restore his whole RAID array because his $1000 RAID card wouldn't let him restart the RAID5 in place due to two bad drives.

P.s. the CPU load on my dual PIII 750 running flat out accessing it's raid arrays is about 1% of a CPU. If you have to worry about 1% of your CPU you have a lot of things on your plate ahead of that.

Re: High Availability Linux with Software RAID

Anonymous's picture

I don't think thats what the author was intending to achieve max performance. But more guarenteed availablity.

If you use the partitions directly in the fstab with priority=1 and a drive fails then the mache will probally go down since a portion of the swap space is now corrupt. However if they are on a RAID 5 setup the machine will just keep on humming. Assuming you don't have a 2nd drive failure.

Re: High Availability Linux with Software RAID

Anonymous's picture

Yes, You could do that, but then You loose HA, because swap will fail,

as soon a disk with a swap partition fails.

Performance wise it would be better to use raid 1 than raid 5 for swap.

Re: High Availability Linux with Software RAID

Anonymous's picture

Anybody has info of how to do this using User Mode Linux?

Re: High Availability Linux with Software RAID

Anonymous's picture

UML is part of the kernel, so is not affected by the RAID subsystem underneath of it. You just need to set up the RAID Disk system as explained, and then install a UML kernel, and way you go.

Re: High Availability Linux with Software RAID

Anonymous's picture

thats not totally true ...
a bug in the ubd driver in uml prevents raidhotadd from working correctly. the bug is known, and a patch is available to fix it (it will be in the next uml release)

greetings,
frank

Re: High Availability Linux with Software RAID

Anonymous's picture

If one is looking to truely run a HA server, would it not be better to make /boot a RAID-1 array, and use a Ramdisk to boot the machine and allow access to the Software RAID. Also, for better performance of the swap partition, rather than creating a software RAID disk for swap, set all the relevant partitions to swap space and set them to equal priority in /etc/fstab so that they are used as a RAID-0 array, without the overhead of the Software RAID system running.

Re: High Availability Linux with Software RAID

Anonymous's picture

Having swap on RAID is a good idea, otherwise a single disk error

can make your machine crash.

I would tend to disagree with

Anonymous's picture

I would tend to disagree with the whole concept of placing your swap on a raid partition.

See line #18 in the link below for more information:

http://linas.org/linux/Software-RAID/Software-RAID-8.html

We're not talking about strip

Anonymous's picture

We're not talking about striping though, but mirroring, so if one drive dies, all the data written to swap doesn't go down with it, as that would be double plus ungood.

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState