DRBD in a Heartbeat

After you've done that, you have to set up DRBD before moving forward with Heartbeat. In my setup, the configuration file is /etc/drbd.conf, but that can change depending on distribution and compile time options, so try to find the file and open it now so you can follow along. If you can't find it, simply create one called /etc/drbd.conf.

Listing 1 is my configuration file. I go over it line by line and add explanations as comments that begin with the # character.

# Each resource is a configuration section for a
# mirrored disk.
# The drbd0 is the name we will use to refer
# to this disk when starting or stopping it.

resource drbd0 {
  protocol C;
  handlers {
    pri-on-incon-degr "echo 'DRBD: primary requested but inconsistent!'
    ↪| wall; /etc/init.d/heartbeat stop"; #"halt -f";
    pri-lost-after-sb "echo 'DRBD: primary requested but lost!'
    ↪| wall; /etc/init.d/heartbeat stop"; #"halt -f";
  }

  startup {
    degr-wfc-timeout 120;    # 2 minutes.
  }

  disk {
    on-io-error   detach;
  }
# These are the network settings that worked best for me.
# If you want to play around with them, go
# ahead, but take a look in the man pages of drbd.conf
# and drbdadm to see what each does.

  net {
    timeout 120;
    connect-int 20;
    ping-int 20;
    max-buffers     2048;
    max-epoch-size  2048;
    ko-count 30;

# Remember to change this shared-secret on both the master
# and slave machines.

    cram-hmac-alg "sha1";
    shared-secret "FooFunFactory";
  }

  syncer {
    rate 10M;
    al-extents 257;
  }

# This next block defines the settings for the server
# labeled as server1.  This label should be in your
# /etc/hosts file and point to a valid host.

  on server1 {

# The following device will be created automatically by
# the drbd kernel module when the DRBD
# partition is in master mode and ready to write.
# If you have more than one DRBD resource, name
# this device drbd1, drbd2 and so forth.

    device     /dev/drbd0

# Put the partition device name you've prepared here.

    disk	     /dev/sda5;

# Now put the IP address of the primary server here.
# Note: you will need to use a unique port number for
# each resource.

    address   192.168.1.3:7788;
    meta-disk  internal;
  }

# This next block is identical to that of server1 but with
# the appropriate settings of the server called
# server2 in our /etc/hosts file.

  on server2 {
    device    /dev/drbd0;
    disk      /dev/sda5;
    address   192.168.1.2:7788;
    meta-disk internal;
  }
}

Now, let's test it by starting the DRBD driver to see if everything works as it should. On your command line on both servers type:

drbdadm create-md drbd0; /etc/init.d/drbd restart; cat /proc/drbd

If all goes well, the output of the last command should look something like this:

0: cs:Connected st:Secondary/Secondary ds:Inconsistent/Inconsistent r---
   ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
       resync: used:0/7 hits:0 misses:0 starving:0 dirty:0 changed:0
       act_log: used:0/257 hits:0 misses:0 starving:0 dirty:0 changed:0

Note: you always can find information about the DRBD status by typing:

cat /proc/drbd

Now, type the following command on the master system:

drbdadm -- --overwrite-data-of-peer primary drbd0; cat /proc/drbd

The output should look something like this:

0: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent r---
   ns:65216 nr:0 dw:0 dr:65408 al:0 bm:3 lo:0 pe:7 ua:6 ap:0
       [>...................] sync'ed:  2.3% (3083548/3148572)K
       finish: 0:04:43 speed: 10,836 (10,836) K/sec
       resync: used:1/7 hits:4072 misses:4 starving:0 dirty:0 changed:4
       act_log: used:0/257 hits:0 misses:0 starving:0 dirty:0 changed:0

This means it is syncing your disks from the master computer that is set as the primary one to the slave computer that is set as secondary.

Next, create the filesystem by typing the following on the master system:

mkfs.ext3 /dev/drbd0

Once that is done, on the master computer, go ahead and mount the drive /dev/drbd0 on the /replicated directory we created for it. We'll have to mount it manually for now until we set up Heartbeat.

An important part of any redundant solution is properly preparing your services so that when the master machine fails, the slave machine can take over and run those services seamlessly. To do that, you have to move not only the data to the replicated DRBD disk, but also move the configuration files.

Let me show you how I've got Sendmail set up to handle the mail and store it on the replicated drives. I use Sendmail for this example as it is one step more complicated than the other services, because even if the machine is running in slave mode, it may need to send e-mail notifications from internal applications, and if Sendmail can't access the configuration files, it won't be able to do this.

On the master machine, first make sure Sendmail is installed but stopped. Then create an etc directory on your /replicated drive. After that, copy your /etc/mail directory into the /replicated/etc and create a symlink from /replicated/etc/mail to /etc/mail.

Next, make a var directory on the /replicated drive, and copy /var/mail, /var/spool/mqueue and any other mail data folders into that directory. Then, of course, create the appropriate symlinks so that the new folders are accessible from their previous locations.

Your /replicated directory structure should now look something like:

/replicated/etc/mail
/replicated/var/mail
/replicated/var/spool/mqueue
/replicated/var/spool/mqueue-client
/replicated/var/spool/mail

And, on your main drive, those folders should be symlinks and look something like:

/etc/mail -> /replicated/etc/mail
/var/mail -> /replicated/var/mail
/var/spool/mqueue -> /replicated/var/spool/mqueue
/var/spool/mqueue-client -> /replicated/var/spool/mqueue-client
/var/spool/mail -> /replicated/var/spool/mail

Now, start Sendmail again and give it a try. If all is working well, you've successfully finished the first part of the setup.

The next part is to make sure it runs, even on the slave. The trick we use is copying the Sendmail binary onto the mounted /replicated drive and putting a symlink to the binary ssmtp on the unmounted /replicated folder.

First, make sure you have ssmtp installed and configured on your system. Next, make a directory /replicated/usr/sbin, and copy /usr/sbin/sendmail to that directory. Then, symlink from /usr/sbin/sendmail back to /replicated/usr/sbin/sendmail.

Once that's done, shut down Sendmail and unmount the /replicated drive. Then, on both the master and slave computers, create a folder /replicated/usr/sbin and a symlink from /usr/sbin/ssmtp to /replicated/usr/sbin/sendmail.

After setting up Sendmail, setting up other services like Apache and PostgreSQL will seem like a breeze. Just remember to put all their data and configuration files on the /replicated drive and to create the appropriate symlinks.