[linux-lvm] pvcreate & vgcfgrestore won't work

[linux-lvm] pvcreate & vgcfgrestore won't work

[Ralf Sparr]

Hi list,

as mentioned in my earlier post, LVM complains about
"Incorrect metadata area header checksum". As this occurs
often (many questions, rarely answers in the list) I tried to create
some 'new' via

# pvcreate --force --uuid 7vFXd3-MYnt-TVdG-luYr-rDPn-o8a1-vjewWk
--restorefile /etc/lvm/archive/alice_00038.vg alice
  Incorrect metadata area header checksum
  uuid 7vFXd3-MYnt-TVdG-luYr-rDPn-o8a1-vjewWk already in use on
"/dev/loop53"

--> also '-ff' doesn't do the trick

# pvscan -vvv
        Processing: pvscan -vvv
        O_DIRECT will be used
      Setting global/locking_type to 1
      File-based locking selected.
      Setting global/locking_dir to /var/lock/lvm
      Locking /var/lock/lvm/P_global WB
    Wiping cache of LVM-capable devices
[...]
    Wiping internal VG cache
        lvmcache: initialised VG #orphans_lvm1
        lvmcache: initialised VG #orphans_pool
        lvmcache: initialised VG #orphans_lvm2
    Walking through all physical volumes
        /dev/hda: Skipping (regex)
[...]
      /dev/loop53: size is 13674639104 sectors
        Closed /dev/loop53
      /dev/loop53: size is 13674639104 sectors
        Opened /dev/loop53 RW O_DIRECT
        /dev/loop53: block size is 4096 bytes
        Closed /dev/loop53
        Using /dev/loop53
        Opened /dev/loop53 RW O_DIRECT
        /dev/loop53: block size is 4096 bytes
      /dev/loop53: lvm2 label detected
        lvmcache: /dev/loop53: now in VG #orphans_lvm2 (#orphans_lvm2)
        /dev/loop53: Found metadata at 49152 size 3734 (in area at 4096
size 192512) for alice (xIDosE-M2TQ-l9oj-PyAU-BZHY-XgV2-ZeasFv)
        lvmcache: /dev/loop53: now in VG alice
        lvmcache: /dev/loop53: setting alice VGID to
xIDosEM2TQl9ojPyAUBZHYXgV2ZeasFv
        lvmcache: /dev/loop53: VG alice: Set creation host to
xen-master.
  Incorrect metadata area header checksum
        Closed /dev/loop53
        Opened /dev/loop54 RO
[...]
        Using cached label for /dev/loop53
        Opened /dev/loop53 RW O_DIRECT
        /dev/loop53: block size is 4096 bytes
        Using cached label for /dev/loop53
        Read alice metadata (19) from /dev/loop53 at 49152 size 3734
        Closed /dev/loop53
        Opened /dev/loop53 RW O_DIRECT
        /dev/loop53: block size is 4096 bytes
  Incorrect metadata area header checksum
        Closed /dev/loop53
        /dev/loop53 0:      0 1048576: alice_storage_data(0:0)
        /dev/loop53 1: 1048576   2560: alice_storage(0:0)
        /dev/loop53 2: 1051136   2560: alice_fuppes(0:0)
        /dev/loop53 3: 1053696   2560: alice_game(0:0)
        /dev/loop53 4: 1056256   2560: alice_openvpn(0:0)
        /dev/loop53 5: 1058816   2560: alice_primary(0:0)
        /dev/loop53 6: 1061376  12800: alice_apache(0:0)
        /dev/loop53 7: 1074176  12800: alice_xen(0:0)
        /dev/loop53 8: 1086976  51200: alice_game_data(0:0)
        /dev/loop53 9: 1138176  25600: alice_xp(0:0)
        /dev/loop53 10: 1163776   5120: alice_x264(0:0)
        /dev/loop53 11: 1168896   2560: alice_firewall(0:0)
        /dev/loop53 12: 1171456 393216: alice_storage_data(1048576:0)
        /dev/loop53 13: 1564672 104595: NULL(0:0)
  WARNING: Volume Group alice is not consistent
  PV /dev/loop53   VG alice   lvm2 [6.37 TB / 408.57 GB free]
  Total: 1 [376.57 GB] / in use: 1 [376.57 GB] / in no VG: 0 [0   ]
      Unlocking /var/lock/lvm/P_global

-> I think 'Total' is messed up, there 6TB missing :(


Is there anything I can do to recover my data? Any  clue is very
welcome!

Ralf

Re: [linux-lvm] pvcreate & vgcfgrestore won't work

[Bryn M. Reeves]

Ralf Sparr wrote:
> Hi list,
> 
> as mentioned in my earlier post, LVM complains about
> "Incorrect metadata area header checksum". As this occurs
> often (many questions, rarely answers in the list) I tried to create
> some 'new' via

Running pvcreate followed by vgcfgrestore is the correct way to fix a 
broken MDA checksum - I've used this many times to recover from these 
situations.

> # pvcreate --force --uuid 7vFXd3-MYnt-TVdG-luYr-rDPn-o8a1-vjewWk
> --restorefile /etc/lvm/archive/alice_00038.vg alice

Your syntax is wrong. You've got:

pvcreate -f -u=$UUID --restorefile=/path $VGNAME

But pvcreate is expecting a PV path as the last argument (the device to 
be initialised as a PV), not a VG name (are you confusing this with the 
vgcfgrestore arguments?).

This works just fine for me:

# pvcreate -f --uuid 6cxF7v-f0XU-3EK1-QhA2-YSVT-yEe3-vQnE09 
--restorefile /etc/lvm/archive/l0_00000.vg /dev/loop0
   Can't initialize physical volume "/dev/loop0" of volume group "l0" 
without -ff

# pvcreate -ff --uuid 6cxF7v-f0XU-3EK1-QhA2-YSVT-yEe3-vQnE09 
--restorefile /etc/lvm/archive/l0_00000.vg /dev/loop0
Really INITIALIZE physical volume "/dev/loop0" of volume group "l0" [y/n]? y
   WARNING: Forcing physical volume creation on /dev/loop0 of volume 
group "l0"
   Physical volume "/dev/loop0" successfully created

# vgcfgrestore -f /etc/lvm/archive/l0_00000.vg l0
   Restored volume group l0

Regards,
Bryn.

Re: [linux-lvm] pvcreate & vgcfgrestore won't work

[Bryn M. Reeves]

SoulSweeper wrote:
> On Tue, 2009-01-06 at 12:08 +0000, Bryn M. Reeves wrote:
>> Ralf Sparr wrote:
>>> Hi list,
>>>
>>> as mentioned in my earlier post, LVM complains about
>>> "Incorrect metadata area header checksum". As this occurs
>>> often (many questions, rarely answers in the list) I tried to create
>>> some 'new' via
>> Running pvcreate followed by vgcfgrestore is the correct way to fix a 
>> broken MDA checksum - I've used this many times to recover from these 
>> situations.
> 
> Hmmm, doesn\t sound very promising. Why does this happen? The error
> directly occured after shutting the lvm down. I wanted to reboot the
> server to check a new network-configuration (no LVM changes at all).
> loop-aes & md were still up.

It's hard to say - when I've encountered this in the past, I've usually 
been doing things that I'd expect might introduce a bit of "excitement" 
to the system, e.g. reproducing bugs that involve crashing the kernel 
with a lot of I/O happening.

I am not aware of any deterministic steps that would trigger this under 
normal circumstances (and I'd consider it a bug if I was :) - when I've 
needed to demonstrate how to recover from this in the past I've had to 
resort to writing code to deliberately trash the checksums.

You mention that you're using loop-aes and md, I wonder if maybe 
something's being cached or not flushed to disk correctly when you 
reboot that could be causing corruption?

Regards,
Bryn.

Re: [linux-lvm] pvcreate & vgcfgrestore won't work

[Ralf Sparr]

On Tue, 2009-01-06 at 17:15 +0000, Bryn M. Reeves wrote:
> SoulSweeper wrote:
> > On Tue, 2009-01-06 at 12:08 +0000, Bryn M. Reeves wrote:
> >> Ralf Sparr wrote:
> >>> Hi list,
> >>>
> >>> as mentioned in my earlier post, LVM complains about
> >>> "Incorrect metadata area header checksum". As this occurs
> >>> often (many questions, rarely answers in the list) I tried to create
> >>> some 'new' via
> >> Running pvcreate followed by vgcfgrestore is the correct way to fix a 
> >> broken MDA checksum - I've used this many times to recover from these 
> >> situations.
> > 
> > Hmmm, doesn\t sound very promising. Why does this happen? The error
> > directly occured after shutting the lvm down. I wanted to reboot the
> > server to check a new network-configuration (no LVM changes at all).
> > loop-aes & md were still up.
> 
> It's hard to say - when I've encountered this in the past, I've usually 
> been doing things that I'd expect might introduce a bit of "excitement" 
> to the system, e.g. reproducing bugs that involve crashing the kernel 
> with a lot of I/O happening.
> 
> I am not aware of any deterministic steps that would trigger this under 
> normal circumstances (and I'd consider it a bug if I was :) - when I've 
> needed to demonstrate how to recover from this in the past I've had to 
> resort to writing code to deliberately trash the checksums.
> 
> You mention that you're using loop-aes and md, I wonder if maybe 
> something's being cached or not flushed to disk correctly when you 
> reboot that could be causing corruption?

Very unlikely. The system itself is located on a mirror. loop-aes is on
my 'data-raid', then lvm is accessing the loop-device. LVM hosts my
xen-domu's and data-partitions.
I run shutdown on every domU and did a 'vgchange -a y'. Then the error
became visible. LVM doesn't shutdown, access to the LVs were still
posible. Then the server was rebooted, but this time without any
accessible LV.
After all, I don't really care about the 'why'. My data is back online!

Again, many thanks to you.

rgds
Ralf


> 
> Regards,
> Bryn.
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/