[linux-lvm] VG Recovery off multiple PV's

[linux-lvm] VG Recovery off multiple PV's

[dGenus Mailing List]

Well... I've got myself in a bit of a bind, and spending close to 24
hours trying to fix a problem I have with LVM, and browsing the
internet, I've come to the conclusion that I need some expert help!!

Problem: Disk died in LVM... I madly tried to move the PE's off the
drive, and managed to get all but teh offending PE's off to new drives i
added to the system. (no 42 wouldn't you know it... thought it was the
answer not the question...) .  Thought I was ok, until the drive locked
again, and I had to reboot.

When it came back, I had a vg which wouldn't be recognised after
vgchange and vgscan, with lots of various errors, most of which I've
seen similar examples for here.   I upgraded the lvm to lvm2 and kernel
to 2.6 to see whether I could get it all going using the partial mount.
Anyways, here is what I get from the various outputs.

[root@wren pictures]# lvm pvscan
  4 PV(s) found for VG data: expected 5
  VG data differs between PVs /dev/hde1 and /dev/hdg2
  VG data differs between PVs /dev/hde1 and /dev/hda8
  VG data differs between PVs /dev/hde1 and /dev/hdh1
  2 PV(s) found for VG pictures: expected 3
  Logical volume (lv_pictures) contains an incomplete mapping table.
  PV /dev/hdg1   VG data       lvm1 [60.50 GB / 0    free]
  PV /dev/hde2   VG data       lvm1 [68.75 GB / 0    free]
  PV /dev/hdb    VG data       lvm1 [55.84 GB / 0    free]
  PV /dev/hdf2   VG data       lvm1 [53.06 GB / 0    free]
  PV /dev/hde1   VG pictures   lvm1 [2.75 GB / 0    free]
  PV /dev/hdf1   VG pictures   lvm1 [2.75 GB / 0    free]
  Total: 6 [0   ] / in use: 6 [0   ] / in no VG: 0 [0   ]

Now I can quite happily see the volumes in the missing vg (in this case
pictures).

# lvm pvdisplay /dev/hde1
  --- Physical volume ---
  PV Name               /dev/hde1
  VG Name               pictures
  PV Size               2.79 GB / not usable 45.55 MB
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              88
  Free PE               0
  Allocated PE          88
  PV UUID               uJqHOZ-puzh-6m8D-04dP-v3np-0Fj5-dh1CDY

# lvm pvdisplay /dev/hdf1
  --- Physical volume ---
  PV Name               /dev/hdf1
  VG Name               pictures
  PV Size               2.79 GB / not usable 45.55 MB
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              88
  Free PE               0
  Allocated PE          88
  PV UUID               FJh4Dl-2KPf-LyHK-8I3F-LAsk-hJ0x-0svzu4

# lvm pvdisplay /dev/hdh1
  --- Physical volume ---
  PV Name               /dev/hdh1
  VG Name               pictures
  PV Size               4.66 GB / not usable 32.77 MB
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              148
  Free PE               0
  Allocated PE          148
  PV UUID               63LnM2-4MsC-Lg7I-st7o-xKcX-LHNy-BFYPtz

# lvm pvdisplay /dev/hdg2

  --- Physical volume ---
  PV Name               /dev/hdg2
  VG Name               pictures
  PV Size               13.99 GB / not usable 51.58 MB
  Allocatable           yes
  PE Size (KByte)       32768
  Total PE              446
  Free PE               445
  Allocated PE          1
  PV UUID               KW087O-c77n-k7Hf-4TCn-6MpM-Kf6L-izuYBv

And when I try a vgchange this is what happens:

# lvm vgchange -P -a y pictures
  Partial mode. Incomplete volume groups will be activated read-only.
  VG data differs between PVs /dev/hde1 and /dev/hdg2
  VG data differs between PVs /dev/hde1 and /dev/hda8
  VG data differs between PVs /dev/hde1 and /dev/hdh1
  2 PV(s) found for VG pictures: expected 3
  Logical volume (lv_pictures) contains an incomplete mapping table.
  VG data differs between PVs /dev/hde1 and /dev/hdg2
  VG data differs between PVs /dev/hde1 and /dev/hda8
  VG data differs between PVs /dev/hde1 and /dev/hdh1
  2 PV(s) found for VG pictures: expected 3
  Logical volume (lv_pictures) contains an incomplete mapping table.
  VG data differs between PVs /dev/hde1 and /dev/hdg2
  VG data differs between PVs /dev/hde1 and /dev/hda8
  VG data differs between PVs /dev/hde1 and /dev/hdh1
  2 PV(s) found for VG pictures: expected 3
  Logical volume (lv_pictures) contains an incomplete mapping table.
  1 logical volume(s) in volume group "pictures" now active

But I can't get it mounted, although the entry exists in /dev/mapper

ls -l /dev/mapper
total 0
crw-------    1 root     root      10,  63 Jan 24 13:49 control
brw-------    1 root     root     254,   0 Jan 24 14:11 data-lv_data
brw-------    1 root     root     254,   1 Jan 24 14:11 pictures-lv_pictures

The logical link is connected to /dev/data/lv_data, but none for
/dev/pictures/lv_pictures

Now my main question is, and I'm hoping this is possible.  Can I list
the PE's on the drives in question and then rebuild the VG in some way
by manually specifying these LE/PE's?  When I did a vgcfgbackup I got a
file with a lot of missing segments.  I'd like to manually specify the
extents and rebuild the vg.

The reason for persisting??  My wife is a photographer and you can
imagine what's on this lvm.....

Hoping someone can help I can provide further information if need be.

Re: [linux-lvm] VG Recovery off multiple PV's

[dGenus Mailing List]

dGenus Mailing List wrote:

> Well... I've got myself in a bit of a bind, and spending close to 24
> hours trying to fix a problem I have with LVM, and browsing the
> internet, I've come to the conclusion that I need some expert help!!

Anybody able to help??  Please?? :D  I pay with cookies!!!

[linux-lvm]VG Recovery off multiple PV's

[tom]

I saw your message in January but I have been working on the "mysteries 
of LVM on-disk layout". Here is what I "think" I've found.
1) Even if your lvm is screwed up, your ext2/3 data is probably still 
there since that volume wasn't the one with physical problems.
2) If you want to "backup" for safety's sake your ext2 data then you 
need to do a pvdata for each of your partitions. The developers have a 
better handle on this, maybe one day they will post to your problem.
The Ext2 filesystem is placed on a hard disk starting at 1024 bytes in 
from the beginning of the partition. This is probably for MBR areas 
(which are usually 512 bytes I believe).
There is a 56 byte data area (mine always start with 0x00)
There is the magic number for the filesystem type 53EF (this is byte 
reversed order of 0xEF53. Magic numbers found in /usr/share/misc/magic)
The there is your ext2 partition metadata and data (I don't care of the 
layout and neither do you, only the size).
Question is where is this on the partition? And the answer is, this is 
roughly dynamically assigned space based on the size of your physical 
volume (I am guessing, you have to look at the lvm code). I used dd to 
find out using hexedit what is going on here :
I will use your first partition /dev/hdg1 to work on all examples.
dd if=/dev/hdg1 of=(any filename on a partition with space to hold it 
lets call it hdg1.e2fs for example) ibs=512 obs=1M count=(supposed to be 
ibs * this count=total bytes but I didn't have the space for it on mine 
so I can't be sure)
Your data is saved. Now the hard part is to either mount it in the 
loopback mount, note: the directory name can be anything.
mkdir /mnt/widget  
mount -t ext3 -o ro,errors=continue,loop  (full path)/hdg1 /mnt/widget
Now mine says the standard error of bad superblock, etc... And this is 
as far as I got. Maybe you won't have my problems.
Also, my theory is that if you have a comparable disk and lets say it is 
hdf then you should be able to dd directly to that disk and it should 
mount ok (my theory only but I am going to try it when my drive 
arrives). the command would be :
dd if=/dev/hdg1 of=/dev/hdf ibs=512 obs=1M count=60G
About 45mins later roughly it should be done. There is NO status info in 
dd saying nn% complete or anything like that, just let it run.

After all data is accounted for and backed up then you can restart the 
LVM process.
I hope this helps a little, sorry I haven't the experience (yet) to tell 
you how to finish the recovery.

I want to say though, that as you start your LVM init process for the 
disks that the lvm dir should be backed up at each step then reboot to 
make sure everything is ok. Short of this and you will get yourself into 
my situation, where there is data and preferences that have accummulated 
over time that I never thought to backup, I copied the data to the lvm, 
everything looked ok, booted ok, then BLAM primary boot disk dies and 
takes all of the LVM data with it, with nearly no chance of recovery. 
C'est la vie!

Good luck. Happy hunting.
-T-

Re: [linux-lvm]VG Recovery off multiple PV's

[Tom Hollins]

Whoops! skipped the step of figuring out where the partition starts (SORRY!). I was putting the kids to bed.
Load the dd'd partition file (hdg1.e2fs) into hexedit (hexedit hdg1.e2fs) 
Hold <CTRL> key and press <S>
Enter 53EF  press <return>
You will need to eyeball this. Whereever you end up, back up 56 bytes. Most likely this 
will be a 0x00 and previous area for the previous 0x400 (1024 bytes) should be zeroes (this would have been your
MBR area). Mark the start of this (Address of 53ef) minus (56 bytes) minus (0x440 bytes) and this will be the start of your ext2 partition. That will be used to help calculate the skip value for dd.
So the address is in hex, multiply by 16 to turn into decimal divide by 512 (your block size) and this
is the number of input blocks to skip before dd starts copying.

Additional data I found about the LVM partition on the disk itself:
1) 0x00 - 0x1FFFF - LVM data
2) 0x20000 - 0x2E50F - Physical extents region as reported by pvdata. Notice all of the 0200s, remember this is in byte reversed order, so this is the vg2 physical extents data, then vg3 etc...
3) 0x2E510-0x401FF - all zeroes (probably the end of the physical extents table
4) 0x40200-0x41fff - un-initialized part of the disk.
5) 0x525000 -        My ext3 partition. Yours might be the same since I had a 60G disk also.

I am sure the LVM programmers will get a recovery utility going when time permits or someone else will write one, but for now this is all I gleaned from 16 hrs searching the net, reading the entire lvm-dev mailing list and all.

-T-

On Sun, 08 Feb 2004 20:35:19 -0500
tom <tom@tomhollins.com> wrote:

> I saw your message in January but I have been working on the "mysteries 
> of LVM on-disk layout". Here is what I "think" I've found.
> 1) Even if your lvm is screwed up, your ext2/3 data is probably still 
> there since that volume wasn't the one with physical problems.
> 2) If you want to "backup" for safety's sake your ext2 data then you 
> need to do a pvdata for each of your partitions. The developers have a 
> better handle on this, maybe one day they will post to your problem.
> The Ext2 filesystem is placed on a hard disk starting at 1024 bytes in 
> from the beginning of the partition. This is probably for MBR areas 
> (which are usually 512 bytes I believe).
> There is a 56 byte data area (mine always start with 0x00)
> There is the magic number for the filesystem type 53EF (this is byte 
> reversed order of 0xEF53. Magic numbers found in /usr/share/misc/magic)
> The there is your ext2 partition metadata and data (I don't care of the 
> layout and neither do you, only the size).
> Question is where is this on the partition? And the answer is, this is 
> roughly dynamically assigned space based on the size of your physical 
> volume (I am guessing, you have to look at the lvm code). I used dd to 
> find out using hexedit what is going on here :
> I will use your first partition /dev/hdg1 to work on all examples.
> dd if=/dev/hdg1 of=(any filename on a partition with space to hold it 
> lets call it hdg1.e2fs for example) ibs=512 obs=1M count=(supposed to be 
> ibs * this count=total bytes but I didn't have the space for it on mine 
> so I can't be sure)
> Your data is saved. Now the hard part is to either mount it in the 
> loopback mount, note: the directory name can be anything.
> mkdir /mnt/widget  
> mount -t ext3 -o ro,errors=continue,loop  (full path)/hdg1 /mnt/widget
> Now mine says the standard error of bad superblock, etc... And this is 
> as far as I got. Maybe you won't have my problems.
> Also, my theory is that if you have a comparable disk and lets say it is 
> hdf then you should be able to dd directly to that disk and it should 
> mount ok (my theory only but I am going to try it when my drive 
> arrives). the command would be :
> dd if=/dev/hdg1 of=/dev/hdf ibs=512 obs=1M count=60G
> About 45mins later roughly it should be done. There is NO status info in 
> dd saying nn% complete or anything like that, just let it run.
> 
> After all data is accounted for and backed up then you can restart the 
> LVM process.
> I hope this helps a little, sorry I haven't the experience (yet) to tell 
> you how to finish the recovery.
> 
> I want to say though, that as you start your LVM init process for the 
> disks that the lvm dir should be backed up at each step then reboot to 
> make sure everything is ok. Short of this and you will get yourself into 
> my situation, where there is data and preferences that have accummulated 
> over time that I never thought to backup, I copied the data to the lvm, 
> everything looked ok, booted ok, then BLAM primary boot disk dies and 
> takes all of the LVM data with it, with nearly no chance of recovery. 
> C'est la vie!
> 
> Good luck. Happy hunting.
> -T-
> 
> 
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 

[linux-lvm] VG Recovery off multiple PV's

[dGenus Mailing List]

Well... I've got myself in a bit of a bind, and spending close to 24 
hours trying to fix a problem I have with LVM, and browsing the 
internet, I've come to the conclusion that I need some expert help!!

Problem: Disk died in LVM... I madly tried to move the PE's off the 
drive, and managed to get all but teh offending PE's off to new drives i 
added to the system. (no 42 wouldn't you know it... thought it was the 
answer not the question...) .  Thought I was ok, until the drive locked 
again, and I had to reboot. 

When it came back, I had a vg which wouldn't be recognised after 
vgchange and vgscan, with lots of various errors, most of which I've 
seen similar examples for here.   I upgraded the lvm to lvm2 and kernel 
to 2.6 to see whether I could get it all going using the partial mount.  
Anyways, here is what I get from the various outputs.

[root@wren pictures]# lvm pvscan
  4 PV(s) found for VG data: expected 5
  VG data differs between PVs /dev/hde1 and /dev/hdg2
  VG data differs between PVs /dev/hde1 and /dev/hda8
  VG data differs between PVs /dev/hde1 and /dev/hdh1
  2 PV(s) found for VG pictures: expected 3
  Logical volume (lv_pictures) contains an incomplete mapping table.
  PV /dev/hdg1   VG data       lvm1 [60.50 GB / 0    free]
  PV /dev/hde2   VG data       lvm1 [68.75 GB / 0    free]
  PV /dev/hdb    VG data       lvm1 [55.84 GB / 0    free]
  PV /dev/hdf2   VG data       lvm1 [53.06 GB / 0    free]
  PV /dev/hde1   VG pictures   lvm1 [2.75 GB / 0    free]
  PV /dev/hdf1   VG pictures   lvm1 [2.75 GB / 0    free]
  Total: 6 [0   ] / in use: 6 [0   ] / in no VG: 0 [0   ]

Now I can quite happily see the volumes in the missing vg (in this case 
pictures).

# lvm pvdisplay /dev/hde1  
  --- Physical volume ---
  PV Name               /dev/hde1
  VG Name               pictures
  PV Size               2.79 GB / not usable 45.55 MB
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              88
  Free PE               0
  Allocated PE          88
  PV UUID               uJqHOZ-puzh-6m8D-04dP-v3np-0Fj5-dh1CDY
  
# lvm pvdisplay /dev/hdf1
  --- Physical volume ---
  PV Name               /dev/hdf1
  VG Name               pictures
  PV Size               2.79 GB / not usable 45.55 MB
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              88
  Free PE               0
  Allocated PE          88
  PV UUID               FJh4Dl-2KPf-LyHK-8I3F-LAsk-hJ0x-0svzu4
  
# lvm pvdisplay /dev/hdh1
  --- Physical volume ---
  PV Name               /dev/hdh1
  VG Name               pictures
  PV Size               4.66 GB / not usable 32.77 MB
  Allocatable           yes (but full)
  PE Size (KByte)       32768
  Total PE              148
  Free PE               0
  Allocated PE          148
  PV UUID               63LnM2-4MsC-Lg7I-st7o-xKcX-LHNy-BFYPtz
  
# lvm pvdisplay /dev/hdg2

  --- Physical volume ---
  PV Name               /dev/hdg2
  VG Name               pictures
  PV Size               13.99 GB / not usable 51.58 MB
  Allocatable           yes
  PE Size (KByte)       32768
  Total PE              446
  Free PE               445
  Allocated PE          1
  PV UUID               KW087O-c77n-k7Hf-4TCn-6MpM-Kf6L-izuYBv

And when I try a vgchange this is what happens:

# lvm vgchange -P -a y pictures
  Partial mode. Incomplete volume groups will be activated read-only.
  VG data differs between PVs /dev/hde1 and /dev/hdg2
  VG data differs between PVs /dev/hde1 and /dev/hda8
  VG data differs between PVs /dev/hde1 and /dev/hdh1
  2 PV(s) found for VG pictures: expected 3
  Logical volume (lv_pictures) contains an incomplete mapping table.
  VG data differs between PVs /dev/hde1 and /dev/hdg2
  VG data differs between PVs /dev/hde1 and /dev/hda8
  VG data differs between PVs /dev/hde1 and /dev/hdh1
  2 PV(s) found for VG pictures: expected 3
  Logical volume (lv_pictures) contains an incomplete mapping table.
  VG data differs between PVs /dev/hde1 and /dev/hdg2
  VG data differs between PVs /dev/hde1 and /dev/hda8
  VG data differs between PVs /dev/hde1 and /dev/hdh1
  2 PV(s) found for VG pictures: expected 3
  Logical volume (lv_pictures) contains an incomplete mapping table.
  1 logical volume(s) in volume group "pictures" now active

But I can't get it mounted, although the entry exists in /dev/mapper

ls -l /dev/mapper
total 0
crw-------    1 root     root      10,  63 Jan 24 13:49 control
brw-------    1 root     root     254,   0 Jan 24 14:11 data-lv_data
brw-------    1 root     root     254,   1 Jan 24 14:11 pictures-lv_pictures

The logical link is connected to /dev/data/lv_data, but none for 
/dev/pictures/lv_pictures

Now my main question is, and I'm hoping this is possible.  Can I list 
the PE's on the drives in question and then rebuild the VG in some way 
by manually specifying these LE/PE's?  When I did a vgcfgbackup I got a 
file with a lot of missing segments.  I'd like to manually specify the 
extents and rebuild the vg.

The reason for persisting??  My wife is a photographer and you can 
imagine what's on this lvm.....

Hoping someone can help I can provide further information if need be.