* [linux-lvm] VG Recovery off multiple PV's
@ 2004-01-23 21:19 dGenus Mailing List
0 siblings, 0 replies; 5+ messages in thread
From: dGenus Mailing List @ 2004-01-23 21:19 UTC (permalink / raw)
To: linux-lvm
Well... I've got myself in a bit of a bind, and spending close to 24
hours trying to fix a problem I have with LVM, and browsing the
internet, I've come to the conclusion that I need some expert help!!
Problem: Disk died in LVM... I madly tried to move the PE's off the
drive, and managed to get all but teh offending PE's off to new drives i
added to the system. (no 42 wouldn't you know it... thought it was the
answer not the question...) . Thought I was ok, until the drive locked
again, and I had to reboot.
When it came back, I had a vg which wouldn't be recognised after
vgchange and vgscan, with lots of various errors, most of which I've
seen similar examples for here. I upgraded the lvm to lvm2 and kernel
to 2.6 to see whether I could get it all going using the partial mount.
Anyways, here is what I get from the various outputs.
[root@wren pictures]# lvm pvscan
4 PV(s) found for VG data: expected 5
VG data differs between PVs /dev/hde1 and /dev/hdg2
VG data differs between PVs /dev/hde1 and /dev/hda8
VG data differs between PVs /dev/hde1 and /dev/hdh1
2 PV(s) found for VG pictures: expected 3
Logical volume (lv_pictures) contains an incomplete mapping table.
PV /dev/hdg1 VG data lvm1 [60.50 GB / 0 free]
PV /dev/hde2 VG data lvm1 [68.75 GB / 0 free]
PV /dev/hdb VG data lvm1 [55.84 GB / 0 free]
PV /dev/hdf2 VG data lvm1 [53.06 GB / 0 free]
PV /dev/hde1 VG pictures lvm1 [2.75 GB / 0 free]
PV /dev/hdf1 VG pictures lvm1 [2.75 GB / 0 free]
Total: 6 [0 ] / in use: 6 [0 ] / in no VG: 0 [0 ]
Now I can quite happily see the volumes in the missing vg (in this case
pictures).
# lvm pvdisplay /dev/hde1
--- Physical volume ---
PV Name /dev/hde1
VG Name pictures
PV Size 2.79 GB / not usable 45.55 MB
Allocatable yes (but full)
PE Size (KByte) 32768
Total PE 88
Free PE 0
Allocated PE 88
PV UUID uJqHOZ-puzh-6m8D-04dP-v3np-0Fj5-dh1CDY
# lvm pvdisplay /dev/hdf1
--- Physical volume ---
PV Name /dev/hdf1
VG Name pictures
PV Size 2.79 GB / not usable 45.55 MB
Allocatable yes (but full)
PE Size (KByte) 32768
Total PE 88
Free PE 0
Allocated PE 88
PV UUID FJh4Dl-2KPf-LyHK-8I3F-LAsk-hJ0x-0svzu4
# lvm pvdisplay /dev/hdh1
--- Physical volume ---
PV Name /dev/hdh1
VG Name pictures
PV Size 4.66 GB / not usable 32.77 MB
Allocatable yes (but full)
PE Size (KByte) 32768
Total PE 148
Free PE 0
Allocated PE 148
PV UUID 63LnM2-4MsC-Lg7I-st7o-xKcX-LHNy-BFYPtz
# lvm pvdisplay /dev/hdg2
--- Physical volume ---
PV Name /dev/hdg2
VG Name pictures
PV Size 13.99 GB / not usable 51.58 MB
Allocatable yes
PE Size (KByte) 32768
Total PE 446
Free PE 445
Allocated PE 1
PV UUID KW087O-c77n-k7Hf-4TCn-6MpM-Kf6L-izuYBv
And when I try a vgchange this is what happens:
# lvm vgchange -P -a y pictures
Partial mode. Incomplete volume groups will be activated read-only.
VG data differs between PVs /dev/hde1 and /dev/hdg2
VG data differs between PVs /dev/hde1 and /dev/hda8
VG data differs between PVs /dev/hde1 and /dev/hdh1
2 PV(s) found for VG pictures: expected 3
Logical volume (lv_pictures) contains an incomplete mapping table.
VG data differs between PVs /dev/hde1 and /dev/hdg2
VG data differs between PVs /dev/hde1 and /dev/hda8
VG data differs between PVs /dev/hde1 and /dev/hdh1
2 PV(s) found for VG pictures: expected 3
Logical volume (lv_pictures) contains an incomplete mapping table.
VG data differs between PVs /dev/hde1 and /dev/hdg2
VG data differs between PVs /dev/hde1 and /dev/hda8
VG data differs between PVs /dev/hde1 and /dev/hdh1
2 PV(s) found for VG pictures: expected 3
Logical volume (lv_pictures) contains an incomplete mapping table.
1 logical volume(s) in volume group "pictures" now active
But I can't get it mounted, although the entry exists in /dev/mapper
ls -l /dev/mapper
total 0
crw------- 1 root root 10, 63 Jan 24 13:49 control
brw------- 1 root root 254, 0 Jan 24 14:11 data-lv_data
brw------- 1 root root 254, 1 Jan 24 14:11 pictures-lv_pictures
The logical link is connected to /dev/data/lv_data, but none for
/dev/pictures/lv_pictures
Now my main question is, and I'm hoping this is possible. Can I list
the PE's on the drives in question and then rebuild the VG in some way
by manually specifying these LE/PE's? When I did a vgcfgbackup I got a
file with a lot of missing segments. I'd like to manually specify the
extents and rebuild the vg.
The reason for persisting?? My wife is a photographer and you can
imagine what's on this lvm.....
Hoping someone can help I can provide further information if need be.
^ permalink raw reply [flat|nested] 5+ messages in thread
* [linux-lvm] VG Recovery off multiple PV's
@ 2004-01-23 21:22 dGenus Mailing List
2004-01-27 1:03 ` dGenus Mailing List
0 siblings, 1 reply; 5+ messages in thread
From: dGenus Mailing List @ 2004-01-23 21:22 UTC (permalink / raw)
To: linux-lvm
Well... I've got myself in a bit of a bind, and spending close to 24
hours trying to fix a problem I have with LVM, and browsing the
internet, I've come to the conclusion that I need some expert help!!
Problem: Disk died in LVM... I madly tried to move the PE's off the
drive, and managed to get all but teh offending PE's off to new drives i
added to the system. (no 42 wouldn't you know it... thought it was the
answer not the question...) . Thought I was ok, until the drive locked
again, and I had to reboot.
When it came back, I had a vg which wouldn't be recognised after
vgchange and vgscan, with lots of various errors, most of which I've
seen similar examples for here. I upgraded the lvm to lvm2 and kernel
to 2.6 to see whether I could get it all going using the partial mount.
Anyways, here is what I get from the various outputs.
[root@wren pictures]# lvm pvscan
4 PV(s) found for VG data: expected 5
VG data differs between PVs /dev/hde1 and /dev/hdg2
VG data differs between PVs /dev/hde1 and /dev/hda8
VG data differs between PVs /dev/hde1 and /dev/hdh1
2 PV(s) found for VG pictures: expected 3
Logical volume (lv_pictures) contains an incomplete mapping table.
PV /dev/hdg1 VG data lvm1 [60.50 GB / 0 free]
PV /dev/hde2 VG data lvm1 [68.75 GB / 0 free]
PV /dev/hdb VG data lvm1 [55.84 GB / 0 free]
PV /dev/hdf2 VG data lvm1 [53.06 GB / 0 free]
PV /dev/hde1 VG pictures lvm1 [2.75 GB / 0 free]
PV /dev/hdf1 VG pictures lvm1 [2.75 GB / 0 free]
Total: 6 [0 ] / in use: 6 [0 ] / in no VG: 0 [0 ]
Now I can quite happily see the volumes in the missing vg (in this case
pictures).
# lvm pvdisplay /dev/hde1
--- Physical volume ---
PV Name /dev/hde1
VG Name pictures
PV Size 2.79 GB / not usable 45.55 MB
Allocatable yes (but full)
PE Size (KByte) 32768
Total PE 88
Free PE 0
Allocated PE 88
PV UUID uJqHOZ-puzh-6m8D-04dP-v3np-0Fj5-dh1CDY
# lvm pvdisplay /dev/hdf1
--- Physical volume ---
PV Name /dev/hdf1
VG Name pictures
PV Size 2.79 GB / not usable 45.55 MB
Allocatable yes (but full)
PE Size (KByte) 32768
Total PE 88
Free PE 0
Allocated PE 88
PV UUID FJh4Dl-2KPf-LyHK-8I3F-LAsk-hJ0x-0svzu4
# lvm pvdisplay /dev/hdh1
--- Physical volume ---
PV Name /dev/hdh1
VG Name pictures
PV Size 4.66 GB / not usable 32.77 MB
Allocatable yes (but full)
PE Size (KByte) 32768
Total PE 148
Free PE 0
Allocated PE 148
PV UUID 63LnM2-4MsC-Lg7I-st7o-xKcX-LHNy-BFYPtz
# lvm pvdisplay /dev/hdg2
--- Physical volume ---
PV Name /dev/hdg2
VG Name pictures
PV Size 13.99 GB / not usable 51.58 MB
Allocatable yes
PE Size (KByte) 32768
Total PE 446
Free PE 445
Allocated PE 1
PV UUID KW087O-c77n-k7Hf-4TCn-6MpM-Kf6L-izuYBv
And when I try a vgchange this is what happens:
# lvm vgchange -P -a y pictures
Partial mode. Incomplete volume groups will be activated read-only.
VG data differs between PVs /dev/hde1 and /dev/hdg2
VG data differs between PVs /dev/hde1 and /dev/hda8
VG data differs between PVs /dev/hde1 and /dev/hdh1
2 PV(s) found for VG pictures: expected 3
Logical volume (lv_pictures) contains an incomplete mapping table.
VG data differs between PVs /dev/hde1 and /dev/hdg2
VG data differs between PVs /dev/hde1 and /dev/hda8
VG data differs between PVs /dev/hde1 and /dev/hdh1
2 PV(s) found for VG pictures: expected 3
Logical volume (lv_pictures) contains an incomplete mapping table.
VG data differs between PVs /dev/hde1 and /dev/hdg2
VG data differs between PVs /dev/hde1 and /dev/hda8
VG data differs between PVs /dev/hde1 and /dev/hdh1
2 PV(s) found for VG pictures: expected 3
Logical volume (lv_pictures) contains an incomplete mapping table.
1 logical volume(s) in volume group "pictures" now active
But I can't get it mounted, although the entry exists in /dev/mapper
ls -l /dev/mapper
total 0
crw------- 1 root root 10, 63 Jan 24 13:49 control
brw------- 1 root root 254, 0 Jan 24 14:11 data-lv_data
brw------- 1 root root 254, 1 Jan 24 14:11 pictures-lv_pictures
The logical link is connected to /dev/data/lv_data, but none for
/dev/pictures/lv_pictures
Now my main question is, and I'm hoping this is possible. Can I list
the PE's on the drives in question and then rebuild the VG in some way
by manually specifying these LE/PE's? When I did a vgcfgbackup I got a
file with a lot of missing segments. I'd like to manually specify the
extents and rebuild the vg.
The reason for persisting?? My wife is a photographer and you can
imagine what's on this lvm.....
Hoping someone can help I can provide further information if need be.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [linux-lvm] VG Recovery off multiple PV's
2004-01-23 21:22 [linux-lvm] VG Recovery off multiple PV's dGenus Mailing List
@ 2004-01-27 1:03 ` dGenus Mailing List
0 siblings, 0 replies; 5+ messages in thread
From: dGenus Mailing List @ 2004-01-27 1:03 UTC (permalink / raw)
To: linux-lvm
dGenus Mailing List wrote:
> Well... I've got myself in a bit of a bind, and spending close to 24
> hours trying to fix a problem I have with LVM, and browsing the
> internet, I've come to the conclusion that I need some expert help!!
Anybody able to help?? Please?? :D I pay with cookies!!!
^ permalink raw reply [flat|nested] 5+ messages in thread
* [linux-lvm]VG Recovery off multiple PV's
@ 2004-02-08 20:35 tom
2004-02-08 22:09 ` Tom Hollins
0 siblings, 1 reply; 5+ messages in thread
From: tom @ 2004-02-08 20:35 UTC (permalink / raw)
To: linux-lvm
I saw your message in January but I have been working on the "mysteries
of LVM on-disk layout". Here is what I "think" I've found.
1) Even if your lvm is screwed up, your ext2/3 data is probably still
there since that volume wasn't the one with physical problems.
2) If you want to "backup" for safety's sake your ext2 data then you
need to do a pvdata for each of your partitions. The developers have a
better handle on this, maybe one day they will post to your problem.
The Ext2 filesystem is placed on a hard disk starting at 1024 bytes in
from the beginning of the partition. This is probably for MBR areas
(which are usually 512 bytes I believe).
There is a 56 byte data area (mine always start with 0x00)
There is the magic number for the filesystem type 53EF (this is byte
reversed order of 0xEF53. Magic numbers found in /usr/share/misc/magic)
The there is your ext2 partition metadata and data (I don't care of the
layout and neither do you, only the size).
Question is where is this on the partition? And the answer is, this is
roughly dynamically assigned space based on the size of your physical
volume (I am guessing, you have to look at the lvm code). I used dd to
find out using hexedit what is going on here :
I will use your first partition /dev/hdg1 to work on all examples.
dd if=/dev/hdg1 of=(any filename on a partition with space to hold it
lets call it hdg1.e2fs for example) ibs=512 obs=1M count=(supposed to be
ibs * this count=total bytes but I didn't have the space for it on mine
so I can't be sure)
Your data is saved. Now the hard part is to either mount it in the
loopback mount, note: the directory name can be anything.
mkdir /mnt/widget
mount -t ext3 -o ro,errors=continue,loop (full path)/hdg1 /mnt/widget
Now mine says the standard error of bad superblock, etc... And this is
as far as I got. Maybe you won't have my problems.
Also, my theory is that if you have a comparable disk and lets say it is
hdf then you should be able to dd directly to that disk and it should
mount ok (my theory only but I am going to try it when my drive
arrives). the command would be :
dd if=/dev/hdg1 of=/dev/hdf ibs=512 obs=1M count=60G
About 45mins later roughly it should be done. There is NO status info in
dd saying nn% complete or anything like that, just let it run.
After all data is accounted for and backed up then you can restart the
LVM process.
I hope this helps a little, sorry I haven't the experience (yet) to tell
you how to finish the recovery.
I want to say though, that as you start your LVM init process for the
disks that the lvm dir should be backed up at each step then reboot to
make sure everything is ok. Short of this and you will get yourself into
my situation, where there is data and preferences that have accummulated
over time that I never thought to backup, I copied the data to the lvm,
everything looked ok, booted ok, then BLAM primary boot disk dies and
takes all of the LVM data with it, with nearly no chance of recovery.
C'est la vie!
Good luck. Happy hunting.
-T-
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [linux-lvm]VG Recovery off multiple PV's
2004-02-08 20:35 [linux-lvm]VG " tom
@ 2004-02-08 22:09 ` Tom Hollins
0 siblings, 0 replies; 5+ messages in thread
From: Tom Hollins @ 2004-02-08 22:09 UTC (permalink / raw)
To: linux-lvm
Whoops! skipped the step of figuring out where the partition starts (SORRY!). I was putting the kids to bed.
Load the dd'd partition file (hdg1.e2fs) into hexedit (hexedit hdg1.e2fs)
Hold <CTRL> key and press <S>
Enter 53EF press <return>
You will need to eyeball this. Whereever you end up, back up 56 bytes. Most likely this
will be a 0x00 and previous area for the previous 0x400 (1024 bytes) should be zeroes (this would have been your
MBR area). Mark the start of this (Address of 53ef) minus (56 bytes) minus (0x440 bytes) and this will be the start of your ext2 partition. That will be used to help calculate the skip value for dd.
So the address is in hex, multiply by 16 to turn into decimal divide by 512 (your block size) and this
is the number of input blocks to skip before dd starts copying.
Additional data I found about the LVM partition on the disk itself:
1) 0x00 - 0x1FFFF - LVM data
2) 0x20000 - 0x2E50F - Physical extents region as reported by pvdata. Notice all of the 0200s, remember this is in byte reversed order, so this is the vg2 physical extents data, then vg3 etc...
3) 0x2E510-0x401FF - all zeroes (probably the end of the physical extents table
4) 0x40200-0x41fff - un-initialized part of the disk.
5) 0x525000 - My ext3 partition. Yours might be the same since I had a 60G disk also.
I am sure the LVM programmers will get a recovery utility going when time permits or someone else will write one, but for now this is all I gleaned from 16 hrs searching the net, reading the entire lvm-dev mailing list and all.
-T-
On Sun, 08 Feb 2004 20:35:19 -0500
tom <tom@tomhollins.com> wrote:
> I saw your message in January but I have been working on the "mysteries
> of LVM on-disk layout". Here is what I "think" I've found.
> 1) Even if your lvm is screwed up, your ext2/3 data is probably still
> there since that volume wasn't the one with physical problems.
> 2) If you want to "backup" for safety's sake your ext2 data then you
> need to do a pvdata for each of your partitions. The developers have a
> better handle on this, maybe one day they will post to your problem.
> The Ext2 filesystem is placed on a hard disk starting at 1024 bytes in
> from the beginning of the partition. This is probably for MBR areas
> (which are usually 512 bytes I believe).
> There is a 56 byte data area (mine always start with 0x00)
> There is the magic number for the filesystem type 53EF (this is byte
> reversed order of 0xEF53. Magic numbers found in /usr/share/misc/magic)
> The there is your ext2 partition metadata and data (I don't care of the
> layout and neither do you, only the size).
> Question is where is this on the partition? And the answer is, this is
> roughly dynamically assigned space based on the size of your physical
> volume (I am guessing, you have to look at the lvm code). I used dd to
> find out using hexedit what is going on here :
> I will use your first partition /dev/hdg1 to work on all examples.
> dd if=/dev/hdg1 of=(any filename on a partition with space to hold it
> lets call it hdg1.e2fs for example) ibs=512 obs=1M count=(supposed to be
> ibs * this count=total bytes but I didn't have the space for it on mine
> so I can't be sure)
> Your data is saved. Now the hard part is to either mount it in the
> loopback mount, note: the directory name can be anything.
> mkdir /mnt/widget
> mount -t ext3 -o ro,errors=continue,loop (full path)/hdg1 /mnt/widget
> Now mine says the standard error of bad superblock, etc... And this is
> as far as I got. Maybe you won't have my problems.
> Also, my theory is that if you have a comparable disk and lets say it is
> hdf then you should be able to dd directly to that disk and it should
> mount ok (my theory only but I am going to try it when my drive
> arrives). the command would be :
> dd if=/dev/hdg1 of=/dev/hdf ibs=512 obs=1M count=60G
> About 45mins later roughly it should be done. There is NO status info in
> dd saying nn% complete or anything like that, just let it run.
>
> After all data is accounted for and backed up then you can restart the
> LVM process.
> I hope this helps a little, sorry I haven't the experience (yet) to tell
> you how to finish the recovery.
>
> I want to say though, that as you start your LVM init process for the
> disks that the lvm dir should be backed up at each step then reboot to
> make sure everything is ok. Short of this and you will get yourself into
> my situation, where there is data and preferences that have accummulated
> over time that I never thought to backup, I copied the data to the lvm,
> everything looked ok, booted ok, then BLAM primary boot disk dies and
> takes all of the LVM data with it, with nearly no chance of recovery.
> C'est la vie!
>
> Good luck. Happy hunting.
> -T-
>
>
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2004-02-09 3:10 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2004-01-23 21:22 [linux-lvm] VG Recovery off multiple PV's dGenus Mailing List
2004-01-27 1:03 ` dGenus Mailing List
-- strict thread matches above, loose matches on Subject: below --
2004-02-08 20:35 [linux-lvm]VG " tom
2004-02-08 22:09 ` Tom Hollins
2004-01-23 21:19 [linux-lvm] VG " dGenus Mailing List
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.