[linux-lvm] Removing a failed PV from VG/LV

[linux-lvm] Removing a failed PV from VG/LV

[Tom Wizetek]

Can someone please outline the process of removing a failed PV
(without replacing it) from a single VG / single LV? Let's say we just
want to continue using what's left of the LV and accept the data loss.

I'm going to throw some ideas here, so forgive my ignorance if this
sounds utterly stupid. My vague understanding of how it would be done
is as follows (assuming ext2):

# vgreduce --removemissing --force /dev/vg1/lv1
...but this would also remove the one and only LV

# lvcreate -n lv1 vg1

# vgchange -ay vg1

# mount /dev/vg1/lv1 /mnt/lv1
...this would fail because the FS doesn't know about the missing drive
(blocks count would be off)

# fsadm check /dev/vg1/lv1
...would fail being unable to determine FSTYPE of /dev/dm-0

# e2fsck /dev/vg1/lv1
...would produce far too many errors but would eventually complete

# fsadm resize /dev/vg1/lv1
...would fail but at least will give us the current blocks count

# resize2fs /dev/vg1/lv1 1234567890
...fails as well since superblock is fubar

# debugfs -w /dev/vg1/lv1
debugfs: set_super_value blocks_count 1234567890
...fail

# mke2fs -n /dev/vg1/lv1
...to get superblock backups locations

# e2fsck -f -y -b 32768 /dev/vg1/lv1
...check using alternative superblock = fail

# debugfs -w -s 98304 -b 4096 /dev/vg1/lv1
...fail

# mke2fs -S -b 4096 /dev/vg1/lv1
...LAST RESORT (and a really bad idea): rewrite superblock and group
descriptors without touching the inode table and block and inode
bitmaps
# e2fsck -fy dev/vg1/lv1
# mount /dev/vg1/lv1 /mnt/lv1
...now no errors but as expected everything is gone

So, is it at all possible to get the LV back up without an actual
replacement PV? What is the right way to do this? Could we "simulate"
the missing PV (if so, how?) and run vgcfgrestore, then fsck, then
reduce FS and finally reduce VG?

Please provide some suggestions or, if it has been covered before, a
link to a solution. I searched the list archives and found two threads
(one in 2006 and the other in 2009) describing a similar scenario but
no clear instructions on how this should be handled were posted.

Many thanks in advance for any input on this subject.

-- 
TW

Re: [linux-lvm] Removing a failed PV from VG/LV

[Bryn M. Reeves]

On 09/02/2010 06:56 AM, Tom Wizetek wrote:
> Can someone please outline the process of removing a failed PV
> (without replacing it) from a single VG / single LV? Let's say we just
> want to continue using what's left of the LV and accept the data loss.

You can activate the VG in partial mode, allowing you to access what's
left of the data on the LVs. This lets you recover as much as remains on
the surviving disks.

E.g.:

Set up some test devices & a test VG/LV:

# pvcreate /dev/loop0
  Physical volume "/dev/loop0" successfully created
# pvcreate /dev/loop1
  Physical volume "/dev/loop1" successfully created
# vgcreate tvg1 /dev/loop0 /dev/loop1
  Volume group "tvg1" successfully created
# lvcreate -L 80M -n l0 tvg1
  Logical volume "l0" created
# lvs tvg1
  LV   VG      Attr   LSize  Origin Snap%  Move Log Copy%  Convert
  l0   tvg1    -wi-a- 80.00M

De-activate the VG and "hide" a PV:

# vgchange -an tvg1
  0 logical volume(s) in volume group "tvg1" now active
# losetup -d /dev/loop1

Try to re-activate:

# vgchange -ay tvg1
  Couldn't find device with uuid 'YU47yZ-CWBj-dANw-kc4Y-gLEt-KcqK-yWXWEn'.
  Couldn't find device with uuid 'YU47yZ-CWBj-dANw-kc4Y-gLEt-KcqK-yWXWEn'.
  Refusing activation of partial LV l0. Use --partial to override.
  0 logical volume(s) in volume group "tvg1" now active

Add the partial flag:

# vgchange -ay --partial tvg1
  Partial mode. Incomplete volume groups will be activated read-only.
  Couldn't find device with uuid 'YU47yZ-CWBj-dANw-kc4Y-gLEt-KcqK-yWXWEn'.
  Couldn't find device with uuid 'YU47yZ-CWBj-dANw-kc4Y-gLEt-KcqK-yWXWEn'.
  1 logical volume(s) in volume group "tvg1" now active

Now we can get at what's left:

# dd if=/dev/tvg1/l0 of=/dev/null
dd: reading `/dev/tvg1/l0': Input/output error
122880+0 records in
122880+0 records out
62914560 bytes (63 MB) copied, 1.04186 s, 60.4 MB/s

To see what's happened here we can look at the device-mapper tables for
the device:

# dmsetup table tvg1-l0
0 122880 linear 7:0 384      # this is the surviving PV
122880 40960 linear 253:7 0  # this is the missing PV

The device 253:7 is device dm-7 which has been created by the vgchange
--partial to fill in the missing space:

# dmsetup table tvg1-l0-missing_1_0
0 40960 error

The error target is a special target that just returns I/O errors - you
can't (obviously) read the data that is not there but you can still
attempt to recover whatever's left on the surviving disks.

Older versions of the tools needed you to set up the "missing stripe
filler" device manually but since about 2.02.40 this is automatic.

It's a bit dated now but you might also find some of this presentation
from a couple of years back helpful:

http://people.redhat.com/breeves/talks/iotc-2008-recovering-lvm2/index.html

Regards,
Bryn.

Re: [linux-lvm] Removing a failed PV from VG/LV

[Tom Wizetek]

This is very useful information. I appreciate it Bryn. The link to the
presentation was also particularly valuable.

Now, having read your post, I need to ask: instead of
recovering/moving data off the partial VG, is there a way to just
shrink the existing VG by the size of the missing PV and carry on?

If I set up a loop device and did a pvcreate --uuid, then
vgcfgrestore, vgscan and finally activated the VG and fsck'd the LV,
could I then shrink the filesystem and remove the replacement loop VG?

P.S.
Using LVM2 2.02.70 here and I can't seem to pvcreate on loop:
# pvcreate /dev/loop0
Device /dev/loop0 not found (or ignored by filtering).
# ls -l /dev/loop*
brw-rw---- 1 root disk 7, 0 Sep  2 11:24 /dev/loop0
brw-rw---- 1 root disk 7, 1 Sep  2 09:31 /dev/loop1
brw-rw---- 1 root disk 7, 2 Sep  2 09:31 /dev/loop2
brw-rw---- 1 root disk 7, 3 Sep  2 09:31 /dev/loop3
brw-rw---- 1 root disk 7, 4 Sep  2 09:31 /dev/loop4
brw-rw---- 1 root disk 7, 5 Sep  2 09:31 /dev/loop5
brw-rw---- 1 root disk 7, 6 Sep  2 09:31 /dev/loop6
brw-rw---- 1 root disk 7, 7 Sep  2 09:31 /dev/loop7
# grep -w filter /etc/lvm/lvm.conf
    # A filter that tells LVM2 to only use a restricted set of devices.
    # The filter consists of an array of regular expressions.  These
    # Don't have more than one filter line active at once: only one gets used.
    filter = [ "a/.*/" ]
    # filter = [ "r|/dev/cdrom|" ]
    #filter = [ "a/loop/", "r/.*/" ]
    # filter =[ "a|loop|", "r|/dev/hdc|", "a|/dev/ide|", "r|.*|" ]
    # filter = [ "a|^/dev/hda8$|", "r/.*/" ]

Re: [linux-lvm] Removing a failed PV from VG/LV

[Tom Wizetek]

Please disregard my previous inquiry about pvcreate on loop. I got it
to work by doing 'losetup /dev/loop0 disk-image'. That was off-topic
anyway.

To state what information I'm looking for in a more concise form:
How to go about restoring a VG without having to physically replace a
failed PV? Is it possible to create a "fake" PV, restore LVM metadata,
fsck the filesystem and finally remove the PV?

-- 
TW