[linux-lvm] recover lost VG (VG data differs between PVs)

[linux-lvm] recover lost VG (VG data differs between PVs)

[Jelle Herold]

Help! :)

I think I messed up my VG trying to recover it... :( this is what I did.

First, I believe grub erased the first sector of a whole-disk PV
(/dev/hde). luckily I had a lvm1 backup in /etc/lvmconf. Using (lvm1's)
pvcreate and vgcfgrestore I managed to get my VG back online. Good.

Then I wanted to move the data from the PV on /dev/hde to a new PV, so I
created a new PV (/dev/hdf3) and used vgextend to extend the VG. Without
thinking I rebooted the computer and worse: I managed to overwrite that
first sectore of /dev/hde again! This time I don't have any backups of
the new configuration (still have the old ones), but I didn't move the
PV or resized the fs on the vg.

I removed the newly added VG and using vgcfgrestore to recover the
metadata on /dev/hde. but now vgdisplay tells me

 VG data differs between PVs /dev/hde and /dev/hdf3
 Volume group "datavg" doesn't exist

Any ideas on how to proceed next are highly appreciated...

Jelle

Re: [linux-lvm] recover lost VG (VG data differs between PVs)

[Alasdair G Kergon]

On Sun, Nov 14, 2004 at 11:58:53PM +0100, Jelle Herold wrote:
>  VG data differs between PVs /dev/hde and /dev/hdf3
>  Volume group "datavg" doesn't exist
 
So hde should be in the VG at the moment, but not hdf3?
Add a filter to lvm.conf so lvm only looks at hde.
Run vgscan and see check if things look OK
eg vgcfgbackup and read the text backup file.
If things *are* OK like that, then pvremove hdf3,
remove the filter & vgscan.

Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] recover lost VG (VG data differs between PVs)

[Jelle Herold]

On Sun, Nov 14, 2004 at 11:21:26PM +0000, Alasdair G Kergon wrote:
> >  VG data differs between PVs /dev/hde and /dev/hdf3
> >  Volume group "datavg" doesn't exist
>  
> So hde should be in the VG at the moment, but not hdf3?

Yes.

> Add a filter to lvm.conf so lvm only looks at hde.  Run
> vgscan and see check if things look OK eg vgcfgbackup and
> read the text backup file.  If things *are* OK like that,
> then pvremove hdf3, remove the filter & vgscan.

OK, I shall try that tomorrow. But a quick look at
vgdisplay -P -v shows:

 VG data differs between PVs /dev/hde and /dev/hdf3
 Partial mode. Incomplete volume groups will be activated read-only.
    Finding all volume groups
    Finding volume group "datavg"
  --- Volume group ---
  VG Name               datavg
  System ID             trigger1078793806
  Format                lvm1
  VG Access             read
  VG Status             resizable
  MAX LV                256
  Cur LV                1
  Open LV               0
  Max PV                256
  Cur PV                3
  Act PV                3
  VG Size               284.09 GB
  PE Size               32.00 MB
  Total PE              9091
  Alloc PE / Size       9091 / 284.09 GB
  Free  PE / Size       0 / 0   
  VG UUID               f49snm-PqWg-hfnp-Ms2k-dgyC-M72S-YqScuJ
   
  --- Logical volume ---
  LV Name                /dev/datavg/datalv
  VG Name                datavg
  LV UUID                000000-0000-0000-0000-0000-0000-000000
  LV Write Access        read/write
  LV Status              NOT available
  LV Size                284.09 GB
  Current LE             9091
  Segments               3
  Allocation             normal
  Read ahead sectors     1024
   
  --- Physical volumes ---
  PV Name               /dev/hde     
  PV UUID               ifxE7h-Z1c6-9mox-VYz5-XTFy-aBw2-vAkdk4
  PV Status             allocatable
  Total PE / Free PE    1831 / 0
   
  PV Name               /dev/hdg3     
  PV UUID               3AnIvc-DdzR-JqtO-hYkV-3vqw-4Qm0-FBKsCS
  PV Status             allocatable
  Total PE / Free PE    3580 / 0
   
  PV Name               /dev/hdh     
  PV UUID               oxjwHR-Cm3S-d0GZ-5LJy-xqa7-X4X6-RAzX02
  PV Status             allocatable
  Total PE / Free PE    3680 / 0

All PVs that should be in the VG are there (hdf3 was the one
I added (and removed again) later to move hde to).

So, *this* output looks OK to me as far as the PVs are
concerned, but I shall have a look at the text backup file
as you suggested. 

thanks for the help
Jelle.

Re: [linux-lvm] recover lost VG (VG data differs between PVs)

[Jelle Herold]

Hi Alasdair,

On Sun, Nov 14, 2004 at 11:21:26PM +0000, Alasdair G Kergon wrote:
> >  VG data differs between PVs /dev/hde and /dev/hdf3
> >  Volume group "datavg" doesn't exist
>  
> So hde should be in the VG at the moment, but not hdf3?

precisely.

> Add a filter to lvm.conf so lvm only looks at hde.  Run vgscan and see
> check if things look OK eg vgcfgbackup and read the text backup file.
> If things *are* OK like that, then pvremove hdf3, remove the filter &
> vgscan.

Wow! *phew* that did the job! THANKS A LOT!!

Just a question out of curiosity, what does "VG data differs..." mean,
and why does filtering hdf3 help (I thought vgreduce removed that PV
from the VG)?

well, again, thanks a zillion
Jelle.

Re: [linux-lvm] recover lost VG (VG data differs between PVs)

[Alasdair G Kergon]

On Mon, Nov 15, 2004 at 11:24:00AM +0100, Jelle Herold wrote:
> Just a question out of curiosity, what does "VG data differs..." mean,
LVM2 checks its metadata is consistent on disk every time you access
a VG.   LVM1 caches its metadata and has less-stringent checks.

That message means it found an anomaly that it couldn't
fix - the metadata on 3 of your devices formed a consistent VG
after you restored the backup, but the metadata on the 4th device hdf3 
still said it was part of the VG.

Filtering tells LVM2 to ignore hdf3 completely, then
pvremove hdf3 wipes LVM metadata from it.

Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] recover lost VG (VG data differs between PVs)

[Jelle Herold]

On Mon, Nov 15, 2004 at 02:15:10PM +0000, Alasdair G Kergon wrote:
> > Just a question out of curiosity, what does "VG data differs..."
> > mean,
>
> LVM2 checks its metadata is consistent on disk every time you access a
> VG.   LVM1 caches its metadata and has less-stringent checks.
> 
> That message means it found an anomaly that it couldn't fix - the
> metadata on 3 of your devices formed a consistent VG after you
> restored the backup, but the metadata on the 4th device hdf3 still
> said it was part of the VG.
> 
> Filtering tells LVM2 to ignore hdf3 completely, then pvremove hdf3
> wipes LVM metadata from it.

Nice system, this LVM. Easy to use, very stable, fool proof. I love it!

Thanks a lot for the clean explanation and the help with getting my VG
back online. You don't know how happy I am with the positive outcome of
this accident :-)

cheers!
Jelle