[linux-lvm] Strange PV metadata issue

[linux-lvm] Strange PV metadata issue

[Alex Brett]

I have a single volume group (unimaginatively named vg), made up of a
single physical volume. I noticed earlier this evening that for some
reason when I created the partition, I'd only created it at about 750G,
rather than the whole 2TB or so that is available to it (it's on a 3ware
hardware raid card if that makes any difference).

To fix this, I deleted the existing partition, and created a new one,
starting at the same place and ending at the end of the disk, all seemed
well. For some reason it couldn't update the partition table without
rebooting - at this point I may have done something stupid, and tried to
do a pvresize anyway. This came back and told me it had quite happily
resized the volume, but on checking pvdisplay, I saw that it was in fact
still the same size.

So, I then rebooted, and did a pvresize, and everything appeared to be
good - I had a 2TB PV, which mapped through to the volume group etc
correctly. I created a new 500G LV, which mkfs'd correctly and seemed
happy...

Later on, I tried running lvdisplay, which is now failing with:
myth ~ # lvdisplay
  Incorrect metadata area header checksum
  Incorrect metadata area header checksum
  Incorrect metadata area header checksum
  Volume group "vg" inconsistent
  Incorrect metadata area header checksum
  Incorrect metadata area header checksum
  WARNING: Inconsistent metadata found for VG vg - updating to use version 5
  Incorrect metadata area header checksum
  Automatic metadata correction failed
  Internal error: Volume Group vg was not unlocked
  Device '/dev/sda4' has been left open.
  Device '/dev/sda4' has been left open.
  Device '/dev/sda4' has been left open.

After some experimenting with vgcfgrestore, and re-creating the PV etc,
the error remains. The existing LVs seem to be perfectly fine, I can
mount them with no problems, and data seems intact etc. I ran pvck,
which shows that I have two metadata areas, one roughly where expected,
and another at roughly the 750G mark, which I didn't expect (I hadn't
specified how many copies of the metadata I wanted, so I would only
expect one):
myth ~ # pvck /dev/sda4
  Incorrect metadata area header checksum
  Found label on /dev/sda4, sector 1, type=LVM2 001
  Found text metadata area: offset=4096, size=192512
  Found text metadata area: offset=788529872896, size=172032
  Incorrect metadata area header checksum

Examing the data on the disk directly, I do seem to have a correct
metadata copy at the 4096 point (it matches what I have in
/etc/lvm/backup/vg), but at the 788529.. point it's just full of zeros.

I assume what's happened (I'm no expert in lvm at all, so a lot of this
is guessing) is that when I (stupidly) ran pvresize on the disk before
rebooting, it somehow created this second metadata area, which the mkfs
of my new LV overwrote in some way.

Looking at the LVs that exist and their stripes, I'm fairly sure that
this metadata area is *inside* the new 500G LV that I created, since
that shows up as:

                        segment1 {
                                start_extent = 0
                                extent_count = 128000   # 500 Gigabytes

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = [
                                        "pv0", 128000
                                ]
                        }
which obviously seems wrong to me!

Can anybody suggest a way of dealing with this - can I manually edit the
data on the partition to remove any reference to this second metadata
area some how? Ideally I'd copy all the data off, and start from
scratch, but unfortunately I don't have enough storage space to do that!

Thanks in advance,
Alex Brett
alex@alexbrett.net

Re: [linux-lvm] Strange PV metadata issue

[Alex Brett]

Alex Brett wrote:
> I assume what's happened (I'm no expert in lvm at all, so a lot of this
> is guessing) is that when I (stupidly) ran pvresize on the disk before
> rebooting, it somehow created this second metadata area, which the mkfs
> of my new LV overwrote in some way.
> 
> Looking at the LVs that exist and their stripes, I'm fairly sure that
> this metadata area is *inside* the new 500G LV that I created, since
> that shows up as:
> 
>                         segment1 {
>                                 start_extent = 0
>                                 extent_count = 128000   # 500 Gigabytes
> 
>                                 type = "striped"
>                                 stripe_count = 1        # linear
> 
>                                 stripes = [
>                                         "pv0", 128000
>                                 ]
>                         }
> which obviously seems wrong to me!
> 
> Can anybody suggest a way of dealing with this - can I manually edit the
> data on the partition to remove any reference to this second metadata
> area some how? Ideally I'd copy all the data off, and start from
> scratch, but unfortunately I don't have enough storage space to do that!
> 

For anybody interested - I managed to fix this by forcing the pvcreate
with -ff, specifying only 1 metadata copy (not sure if I needed this bit
or not) and then a vgcfgrestore did the trick - all seems happy now :)

Alex

Re: [linux-lvm] Strange PV metadata issue

[Bryn M. Reeves]

Alex Brett wrote:
> For anybody interested - I managed to fix this by forcing the pvcreate
> with -ff, specifying only 1 metadata copy (not sure if I needed this bit
> or not) and then a vgcfgrestore did the trick - all seems happy now :)

The same situation/recovery steps were discussed on the list last week:

https://www.redhat.com/archives/linux-lvm/2009-January/msg00014.html

Bryn.