Home ›

Recovery of RAID and LVM2 Volumes

Issue 146

From Issue #146
June 2006

Apr 28, 2006  By Richard Bulling...
 in
Raid and Logical Volume Managers are great, until you lose data.

Listing 6. Modified Volume Group Configuration File 

VolGroup01 {
id = "xQZqTG-V4wn-DLeQ-bJ0J-GEHB-4teF-A4PPBv"
seqno = 1
status = ["RESIZEABLE", "READ", "WRITE"]
extent_size = 65536
max_lv = 0
max_pv = 0

physical_volumes {

pv0 {
id = "tRACEy-cstP-kk18-zQFZ-ErG5-QAIV-YqHItA"
device = "/dev/md2"

status = ["ALLOCATABLE"]
pe_start = 384
pe_count = 2365
}
}

# Generated by LVM2: Sun Feb  5 22:57:19 2006

Once you have a volume group configuration file, migrate the volume group to this system with vgcfgrestore, as Listing 7 shows.

Listing 7. Activating the Recovered LVM2 Volume 

[root@recoverybox ~]# vgcfgrestore -f VolGroup01 VolGroup01
[root@recoverybox ~]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "VolGroup01" using metadata type lvm2
  Found volume group "VolGroup00" using metadata type lvm2
[root@recoverybox ~]# pvscan
  PV /dev/md2    VG VolGroup01   lvm2 [73.91 GB / 32.00 MB free]
  PV /dev/hda2   VG VolGroup00   lvm2 [18.91 GB / 32.00 MB free]
  Total: 2 [92.81 GB] / in use: 2 [92.81 GB] / in no VG: 0 [0   ]
[root@recoverybox ~]# vgchange VolGroup01 -a y
  1 logical volume(s) in volume group "VolGroup01" now active
[root@recoverybox ~]# lvscan
  ACTIVE            '/dev/VolGroup01/LogVol00' [73.88 GB] inherit
  ACTIVE            '/dev/VolGroup00/LogVol00' [18.38 GB] inherit
  ACTIVE            '/dev/VolGroup00/LogVol01' [512.00 MB] inherit

At this point, you can now mount the old volume on the new system, and gain access to the files within, as shown in Listing 8.

Listing 8. Mounting the Recovered Volume 

[root@recoverybox ~]# mount /dev/VolGroup01/LogVol00 /mnt
[root@recoverybox ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       19G  4.7G   13G  28% /
/dev/hda1              99M   12M   82M  13% /boot
none                  126M     0  126M   0% /dev/shm
/dev/mapper/VolGroup01-LogVol00
                       73G  2.5G   67G   4% /mnt
# ls -l /mnt
total 200
drwxr-xr-x   2 root root  4096 Feb  6 02:36 bin
drwxr-xr-x   2 root root  4096 Feb  5 18:03 boot
drwxr-xr-x   4 root root  4096 Feb  5 18:03 dev
drwxr-xr-x  79 root root 12288 Feb  6 23:54 etc
drwxr-xr-x   3 root root  4096 Feb  6 01:11 home
drwxr-xr-x   2 root root  4096 Feb 21  2005 initrd
drwxr-xr-x  11 root root  4096 Feb  6 02:36 lib
drwx------   2 root root 16384 Feb  5 17:59 lost+found
drwxr-xr-x   3 root root  4096 Feb  6 22:12 media
drwxr-xr-x   2 root root  4096 Oct  7 09:03 misc
drwxr-xr-x   2 root root  4096 Feb 21  2005 mnt
drwxr-xr-x   2 root root  4096 Feb 21  2005 opt
drwxr-xr-x   2 root root  4096 Feb  5 18:03 proc
drwxr-x---   5 root root  4096 Feb  7 00:19 root
drwxr-xr-x   2 root root 12288 Feb  6 22:37 sbin
drwxr-xr-x   2 root root  4096 Feb  5 23:04 selinux
drwxr-xr-x   2 root root  4096 Feb 21  2005 srv
drwxr-xr-x   2 root root  4096 Feb  5 18:03 sys
drwxr-xr-x   3 root root  4096 Feb  6 00:22 tftpboot
drwxrwxrwt   5 root root  4096 Feb  7 00:21 tmp
drwxr-xr-x  15 root root  4096 Feb  6 22:33 usr
drwxr-xr-x  20 root root  4096 Feb  5 23:15 var

Now that you have access to your data, a prudent final step would be to back up the volume group information with vcfgbackup, as Listing 9 shows.

Listing 9. Backing Up Recovered Volume Group Configuration 

[root@teapot-new ~]# vgcfgbackup
Volume group "VolGroup01" successfully backed up.
Volume group "VolGroup00" successfully backed up.
[root@teapot-new ~]# ls -l /etc/lvm/backup/
total 24
-rw-------  1 root root 1350 Feb 10 09:09 VolGroup00
-rw-------  1 root root 1051 Feb 10 09:09 VolGroup01
Conclusion

LVM2 and Linux software RAID make it possible to create economical, reliable storage solutions with commodity hardware. One trade-off involved is that some procedures for recovering from failure situations may not be clear. A tool that reliably extracted old volume group information directly from the disk would make recovery easier. Fortunately, the designers of the LVM2 system had the wisdom to keep plain-text backup copies of the configuration on the disk itself. With a little patience and some research, I was able to regain access to the logical volume I thought was lost; may you have as much success with your LVM2 and RAID installation.

Resources for this article: /article/8948.

Richard Bullington-McGuire is the Managing Partner of PKR Internet, LLC, a software and systems consulting firm in Arlington, Virginia, specializing in Linux, Open Source and Java. He has been a Linux sysadmin since 1994. You can reach him at rbulling@pkrinternet.com.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Thanks

Anonymous's picture
Submitted by Anonymous (not verified) on Sun, 12/10/2006 - 19:16.

Count another life saved. In spite of destroying one HD out of a two HD LVM set, we still recovered some data thanks to these tips. Not too shabby.

Alternate recovery method

Garth Webb's picture
Submitted by Garth Webb (not verified) on Sun, 11/05/2006 - 11:51.

In my situation I did not have any explicit RAID arrays. I just had the standard RedHat FC5 configuration of a single VolGroup00 volume group with two logical volumes. I pulled that 20GB drive from my system recently and installed a 320GB drive in its place and reinstalled FC5. Because this new drive had the same VolGroup00 volume group created, I could not mount the 20 GB drive I had in a USB enclosure.

It seemed that most of this article was aimed at teasing out the lvm metadata and rewriting it to affect a volume group name change. Since all of the lvm tools require that you address the volume groups on your physical drives by their name, you enconter the naming conflict (how hard would it have been to include a rename command that took a physical path and renamed the volume there?).

Rather than fight that battle, I booted off my FC5 rescue CD (or any bootable tools CD with the LVM tools on it) and did a:

vgrename VolGroup00 Seagate320

The naming conflict didn't really matter here it seems. It just renamed whatever VolGroup00 it found first, which happened to be my new 320 G drive. I could then activate both with:

vgchange -ay

and then mount the volumes and copy, etc.

Nice recovery method

rbulling's picture
Submitted by rbulling on Wed, 11/22/2006 - 20:24.

This looks like a much simpler way to do things, as long as you are not dealing with software RAID.

One thing you'd need to be careful about is making sure that you leave the new VolGroup00 named VolGroup00 at the end of the recovery process.

I suspect that using vgrename / vgscan / vgchange in combination would allow you to rename both volume groups to something else, then rename the newest VolGroup00 back to VolGroup00, so that the system would continue to work on boot.

You could probably use the same technique after you recovered the RAID configuration, and avoid the messy surgery on raw disk information. Next time I encounter this issue, I'll give that a try.

With my distro (Ubuntu), and

Anonymous's picture
Submitted by Anonymous on Mon, 03/15/2010 - 09:56.

With my distro (Ubuntu), and I suspect many others these days, drives are actually mounted by UUID, not by device name *or* by VG/LV name. So renaming the VG, even the one containing root, should *) not be a problem.

What I did to recover with a RAID1/LVM stack which blew one of the drives, was boot the ubuntu desktop liveCD, sudo apt-get install mdadm lvm2, mdadm --assemble /dev/mdX (be careful of raid devnames that occur in multiples -- ie run this on recoveryPC with old_drives, and disconnected recoveryPC_drives -- but it will simply scan the drives attached for appropriate partitions), then the lvm stuff will simply magically appear and you can vgrename easily if necessary. Once you've done that, connect all the drives and reboot, and after the mdadm.conf magic you will have all your data.

Incidentally, on my home server, raid1 OS disks, Raid5 storage disks, and LVM over the top, hell yeah that makes sense. On my desktop, not so much.

* Yeah. I haven't tested this explicitly.

Thanks!

Juan's picture
Submitted by Juan (not verified) on Sat, 10/21/2006 - 00:15.

This article save my day!

Thanks!

Anders Båtstrand's picture
Submitted by Anders Båtstrand (not verified) on Fri, 10/06/2006 - 06:41.

This worked great for me. Thanks for putting it together!

Doesn't quite work for me

jweage's picture
Submitted by jweage (not verified) on Thu, 09/21/2006 - 19:26.

I just ran into a similar problem attempting to move a disk from one machine to another, with both disks configured as VolGroup00. I worked through your example, but when it came to restoring VolGroup01 (Listing 6), vgcfgrestore refused claiming it couldn't find a contents line. In my dump, there are 5 additional header lines before the VolGroup01 { line, which vgcfgrestore requires.

After I figured this out and restored the volume group, I could not get any logical volumes to show up. lvscan did not pick up the three logical volumes on the volume group! Those were also in the dd extracted file, so I had to add all of that back into the config file and do another vgcfgrestore, vgactivate.

This is really disconcerting, as this is likely to be a common problem. Unfortunately it seems that LVM is NOT the way to go for the typical workstation, unless someone really needs the ability to resize a volume.

Correction to Listing 6

rbulling's picture
Submitted by rbulling on Wed, 11/22/2006 - 20:06.

It appears that Listing 6 got truncated somewhere along the line before publication.

The full Listing 6 should be:


Listing 6: Modified Volume Group Configuration File

VolGroup01 {
id = "xQZqTG-V4wn-DLeQ-bJ0J-GEHB-4teF-A4PPBv"
seqno = 1
status = ["RESIZEABLE", "READ", "WRITE"]
extent_size = 65536
max_lv = 0
max_pv = 0

physical_volumes {

pv0 {
id = "tRACEy-cstP-kk18-zQFZ-ErG5-QAIV-YqHItA"
device = "/dev/md2"

status = ["ALLOCATABLE"]
pe_start = 384
pe_count = 2365
}
}

# Generated by LVM2: Sun Feb 5 22:57:19 2006
logical_volumes {

LogVol00 {
id = "i17qXJ-Blzu-u1Dr-bSlR-0kNC-yuBH-lnbkSi"
status = ["READ", "WRITE", "VISIBLE"]
segment_count = 1

segment1 {
start_extent = 0
extent_count = 2364

type = "striped"
stripe_count = 1 # linear

stripes = [
"pv0", 0
]
}
}
}
}

contents = "Text Format Volume Group"
version = 1

description = ""

creation_host = "localhost.localdomain" # Linux localhost.localdomain 2.6.9-11.EL #1 Wed Jun 8 20:20:13 CDT 2005 i686
creation_time = 1139180239 # Sun Feb 5 22:57:19 2006

what if the machine your

dave's picture
Submitted by dave (not verified) on Wed, 08/16/2006 - 08:25.

what if the machine your using for recovery has raid itself?? when you append to mdadm.conf can md0,1,2 be renumbered to 3,4,5?

Renumbering md0 to md3, for example, works

rbulling's picture
Submitted by rbulling on Wed, 11/22/2006 - 20:16.

You should be able to do that without any problems, as long as you explicitly keep the UUID signature in the renamed device line.

Experienced this exact

Neekofab's picture
Submitted by Neekofab (not verified) on Wed, 09/13/2006 - 06:55.

Experienced this exact problem. moved a md0/md1 disk to a recovery workstation that already had an md0/md1 device. they could not coexist, and I could not find a way to move the additional md0/md1 devices to md2/md3. I ended up disconnecting the system md0/md1 devices, booting up with sysresccd and shoving the data over the network.

bleah

I ran into the same issue

Anonymous's picture
Submitted by Anonymous (not verified) on Wed, 05/09/2007 - 19:51.

I ran into the same issue and solved it with a little reading about mdadm. All you have to do is create a new array from the old disks.

# MAKEDEV md1
# mdadm -C /dev/md1 -l 1 -n 2 missing /dev/sdb1

Voila. Your raid array has now been moved from md0 to md1.

I ran into the same issue

Anonymous's picture
Submitted by Anonymous (not verified) on Wed, 05/09/2007 - 19:50.

I ran into the same issue and solved it with a little reading about mdadm. All you have to do is create a new array from the old disks.

# MAKEDEV md1
# mdadm -C /dev/md1 -l 1 -n 2 missing /dev/sdb1

Voila. Your raid array has now been moved from md0 to md1.

Recovery Linux Data

Anonymous's picture
Submitted by Anonymous (not verified) on Tue, 05/29/2007 - 08:05.

Kernel Linux - Ext2 & Ext3 Data Recovery Software supports LVM Partition PARTIALLY
LVM Recovery

LVM Data Recovery

Chris - Armor-IT's picture
Submitted by Chris - Armor-IT (not verified) on Sun, 08/01/2010 - 11:52.

I review data recovery software for my employer. I would like an opportunity to test your software if possible.

Data Recovery Software for Linux

Unistal Data Recovery's picture
Submitted by Unistal Data Recovery (not verified) on Thu, 08/23/2007 - 09:26.

If you need efficient linux data recovery software, which runs on Linux then download Linux Data Recovery Software.

LVM Recovery

Recover Data's picture
Submitted by Recover Data (not verified) on Tue, 06/05/2007 - 00:37.

Recover Data for Linux also supports LVM Recovery. Download the software DEMO and scan you disk for lost data and files.

Visit : Recover Data Software

White Paper
More Resources
Fabric-Based Computing Enables Optimized Hyperscale Data Centers

Today’s modular x86 servers are compute-centric, designed as a least common denominator to support a wide range of IT workloads. Those generic, virtualized IT workloads have much different resource optimization requirements than hyperscale and cloud applications. They have resulted in a “one size fits all” enterprise IT architecture that is not optimized for a specific set of IT workloads, and especially not emerging hyperscale workloads, such as web applications, big data, and object storage. In this report, you will learn how shifting the focus from traditional compute-centric IT architectures to an innovative disaggregated fabric-based architecture can optimize and scale your data center.

Learn More

Sponsored by AMD

White Paper
More Resources
Red Hat White Paper: Using an Open Source Framework to Catch the Bad Guy

Built-in forensics, incident response, and security with Red Hat Enterprise Linux 6

Every security policy provides guidance and requirements for ensuring adequate protection of information and data, as well as high-level technical and administrative security requirements for a system in a given environment. Traditionally, providing security for a system focuses on the confidentiality of the information on it. However, protecting the data integrity and system and data availability is just as important. For example, when processing United States intelligence information, there are three attributes that require protection: confidentiality, integrity, and availability.

Learn more about catching the bad guy in this free white paper.

Learn More

Sponsored by DLT Solutions