[linux-lvm] Repair inconsistent PV

[linux-lvm] Repair inconsistent PV

[Andreas Pakulat]

Hi,

yesterday I lost my / partition due to hardware problem. Now to reinstall
the system I had to reduce the size of my VG, i did that using the
KNOPPIX CD and first all went Ok. But during the Removal of one LV one
harddisk "crashed", had several read errors. I could fix this, as It was
due to a broken connection to the drive. But now the PV that was on that
drive is inconsisten:

root@tty2[/]# vgscan
vgscan -- reading all physical volumes (this may take a while...)
vgscan -- found inactive volume group "vgmatrix"
vgscan -- ERROR "pv_check_consistency_all_pv(): PE" volume group
"vgmatrix" is inconsistent
vgscan -- ERROR: unable to do a backup of volume group "vgmatrix"
vgscan -- ERROR "lvm_tab_vg_remove(): unlink" removing volume group
"vgmatrix" from "/etc/lvmtab"
vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created
vgscan -- WARNING: This program does not do a VGDA backup of your volume
group
root@tty2[/]#

If i do a pvcreate -ff /dev/hdc2 the inconsistency is gone, but also the
metadata and as my / is also gone, I have no backup of the
Metadata. So the Question is, can I somehow repair the PV Metadata or am
I lost? It is also hard for me at the moment to get a Kernel with
device-mapper support, so using LVM2 is only the last option for me.

Please say if you need any additional Information, here is pvdisplay 
of the problematic PV:

root@tty2[/]# pvdisplay -v /dev/hdc2
--- Physical volume ---
PV Name               /dev/hdc2
VG Name               vgmatrix
PV Size               5.84 GB [12241530 secs] / NOT usable 4.19 MB [LVM:
133 KB]
PV#                   2
PV Status             available
Allocatable           NO
Cur LV                2
PE Size (KByte)       4096
Total PE              1493
Free PE               444
Allocated PE          1049
PV UUID               v7FOLU-uCS4-ZD8R-2z6S-GIKR-RCbf-RI2Nax


Andreas

-- 
Wie man sein Kind nicht nennen sollte: 
  Harry Bo 

Re: [linux-lvm] Repair inconsistent PV

[Andreas Pakulat]

[-- Attachment #1: Type: text/plain, Size: 2681 bytes --]

On 16.Aug 2003 - 17:45:56, Andreas Pakulat wrote:
> Hi,
> 
> yesterday I lost my / partition due to hardware problem. Now to reinstall
> the system I had to reduce the size of my VG, i did that using the
> KNOPPIX CD and first all went Ok. But during the Removal of one LV one
> harddisk "crashed", had several read errors. I could fix this, as It was
> due to a broken connection to the drive. But now the PV that was on that
> drive is inconsisten:
> 
> root@tty2[/]# vgscan
> vgscan -- reading all physical volumes (this may take a while...)
> vgscan -- found inactive volume group "vgmatrix"
> vgscan -- ERROR "pv_check_consistency_all_pv(): PE" volume group
> "vgmatrix" is inconsistent
> vgscan -- ERROR: unable to do a backup of volume group "vgmatrix"
> vgscan -- ERROR "lvm_tab_vg_remove(): unlink" removing volume group
> "vgmatrix" from "/etc/lvmtab"
> vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created
> vgscan -- WARNING: This program does not do a VGDA backup of your volume
> group
> root@tty2[/]#
> 
> If i do a pvcreate -ff /dev/hdc2 the inconsistency is gone, but also the
> metadata and as my / is also gone, I have no backup of the
> Metadata. So the Question is, can I somehow repair the PV Metadata or am
> I lost? It is also hard for me at the moment to get a Kernel with
> device-mapper support, so using LVM2 is only the last option for me.
> 
> Please say if you need any additional Information, here is pvdisplay 
> of the problematic PV:
> 
> root@tty2[/]# pvdisplay -v /dev/hdc2
> --- Physical volume ---
> PV Name               /dev/hdc2
> VG Name               vgmatrix
> PV Size               5.84 GB [12241530 secs] / NOT usable 4.19 MB [LVM:
> 133 KB]
> PV#                   2
> PV Status             available
> Allocatable           NO
> Cur LV                2
> PE Size (KByte)       4096
> Total PE              1493
> Free PE               444
> Allocated PE          1049
> PV UUID               v7FOLU-uCS4-ZD8R-2z6S-GIKR-RCbf-RI2Nax

I now installed a 2.6.0-test2 Kernel and tried to get further using
lvm2. From that I know that the VG Metadata differs between the PV on
hdc2 and the PV on hda7. I also experimented with the Partial-Option but
the LV I need cannot be mounted ;( Is there a way to create the
VG-Metadata of one PV from the Data on the other? How? I could create a
text-file using vgcfgbackup from lvm2, the interesting thing is that it
contains the PV-Data from hdc2, but I think that is the PV with the
wrong Information. Anyway I hope someone here can help me soon.

Andreas

-- 
Verwirrung ist die höchste Form der Kommunikation.
		-- Dali

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [linux-lvm] Repair inconsistent PV

[Andreas Pakulat]

Hi,

only wanted to notify everyone, that I could now solve my problem with
hexedit and lvm.h. I changed the Major/Minor Numbers in both hdXX and
then looked at the VG Data, I found that the one in hdc2 seemed OK,
while the one in hda7 looked like random Data was written in it. So I
copied the VG Metadata into hda7 using dd. A vgscan marked lvtmp as
problematic, it was in hda7 but in the vgmetadata were only 3 LV's, so I
changed that and threw out the LV Metadata of that particular lv. It
is also the one I was deleting when the hardware error occured. Now I
can read my data again.

Andreas

Re: [linux-lvm] Repair inconsistent PV

[Scott Mcdermott]

Andreas Pakulat on Mon 18/08 23:42 +0200:
> only wanted to notify everyone, that I could now solve my
> problem with hexedit and lvm.h. I changed the Major/Minor
> Numbers in both hdXX and then looked at the VG Data, I
> found that the one in hdc2 seemed OK, while the one in
> hda7 looked like random Data was written in it. So I
> copied the VG Metadata into hda7 using dd. A vgscan marked
> lvtmp as problematic, it was in hda7 but in the vgmetadata
> were only 3 LV's, so I changed that and threw out the LV
> Metadata of that particular lv. It is also the one I was
> deleting when the hardware error occured. Now I can read
> my data again.

that's cool investigative work...

just wondering out of curiosity...did you have a backup when
you did this? :)

Re: [linux-lvm] Repair inconsistent PV

[Andreas Pakulat]

On 18.Aug 2003 - 18:08:05, Scott Mcdermott wrote:
> Andreas Pakulat on Mon 18/08 23:42 +0200:
> > only wanted to notify everyone, that I could now solve my
> > problem with hexedit and lvm.h. I changed the Major/Minor
> > Numbers in both hdXX and then looked at the VG Data, I
> > found that the one in hdc2 seemed OK, while the one in
> > hda7 looked like random Data was written in it. So I
> > copied the VG Metadata into hda7 using dd. A vgscan marked
> > lvtmp as problematic, it was in hda7 but in the vgmetadata
> > were only 3 LV's, so I changed that and threw out the LV
> > Metadata of that particular lv. It is also the one I was
> > deleting when the hardware error occured. Now I can read
> > my data again.
> 
> that's cool investigative work...

With a bit knowledge of C and the size of integer-datatypes it is not
that hard. Even though I had to play a little, to find out which
metadata block is the right one. And the lvm.h file is very well written
and documented, it's relatively easy to use it to understand the bytes
on the device.

> just wondering out of curiosity...did you have a backup when
> you did this? :)

You mean a backup of my original VG-Metadata? No, I explained that in
the original post, I lost my / partition and wanted to reinstall my
system, during the reorganization of the VG I lost some metadata due to
a hardware error. If I had a backup of the LVM-Metadata the whole
process would have been much easier - only a pvcreate -ff and
vgcfgrestore. 

From the raw devices I made a backup before starting of doing anything
to the PV's, I did not want to loose more data.

And I did not have a system backup, as this system is not that
important, but it would have costed a reasonable amount of time to get
the system back working without the data from the lv.

Andreas