[linux-lvm] Instability with 2.6, recovering (meta)data?

[linux-lvm] Instability with 2.6, recovering (meta)data?

[Toad]

[-- Attachment #1: Type: text/plain, Size: 1422 bytes --]

Each time I have deleted an LV on LVM2 on 2.6, it has corrupted my
metadata. Last time I did a vgcfgrestore on 2.4 and it worked, however I
have recently been bitten by some sort of incompatibility bug and my LVM
now only works on 2.6 (it might work on a 2.4+devmapper kernel, I'll
look into that). Here is what happens when I try to get my metadata
back:
root@amphibian:/usr/src/linux# vgdisplay -d -v
    Finding all volume groups
    Finding volume group "vg-big"
  invalid lv in extent map
  Couldn't fill logical volume maps.
  Volume group "vg-big" doesn't exist
root@amphibian:/usr/src/linux# pvscan -d -v
    Wiping cache of LVM-capable devices
    Wiping internal cache
    Walking through all physical volumes
  invalid lv in extent map
  Couldn't fill logical volume maps.
  No matching physical volumes found
root@amphibian:/usr/src/linux# vgcfgrestore -v -d vg-big
  PV /dev/hda is a different format (lvm1)
  Restore failed.

That last one is particularly perverse; I was always told that LVM1 and
LVM2 are compatible. Of course later I discovered this not to be the
case; after some operation or other (creating an LV? resizing one?) on
LVM on 2.6, I could no longer access my LVs under 2.4/LVM1.

Any help?
-- 
Matthew J Toseland - toad@amphibian.dyndns.org
Freenet Project Official Codemonkey - http://freenetproject.org/
ICTHUS - Nothing is impossible. Our Boss says so.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [linux-lvm] Instability with 2.6, recovering (meta)data?

[Toad]

With agk's help, I eventually found that this worked:
vgcfgrestore vg-big -M1 -f /etc/lvm/archive/vg-big_00003.vg

However, the LV that I deleted that originally caused the problem,
/dev/vg-big/slowtemp (~ 10GB), doesn't exist, and yet it's space usage 
appears to still be used:
root@amphibian:~# vgdisplay
...
VG Size               212.81 GB
PE Size               32.00 MB
Total PE              6810
Alloc PE / Size       6704 / 209.50 GB
Free  PE / Size       106 / 3.31 GB

As I was originally trying to free up enough space to vgmove off one of
the disks, in order to replace it and then vgmove off another of the
disks (IDE madness...), this is still a problem.

On Sun, Nov 09, 2003 at 09:38:21PM +0000, Toad wrote:
> Each time I have deleted an LV on LVM2 on 2.6, it has corrupted my
> metadata. Last time I did a vgcfgrestore on 2.4 and it worked, however I
> have recently been bitten by some sort of incompatibility bug and my LVM
> now only works on 2.6 (it might work on a 2.4+devmapper kernel, I'll
> look into that). Here is what happens when I try to get my metadata
> back:
> root@amphibian:/usr/src/linux# vgdisplay -d -v
>     Finding all volume groups
>     Finding volume group "vg-big"
>   invalid lv in extent map
>   Couldn't fill logical volume maps.
>   Volume group "vg-big" doesn't exist
> root@amphibian:/usr/src/linux# pvscan -d -v
>     Wiping cache of LVM-capable devices
>     Wiping internal cache
>     Walking through all physical volumes
>   invalid lv in extent map
>   Couldn't fill logical volume maps.
>   No matching physical volumes found
> root@amphibian:/usr/src/linux# vgcfgrestore -v -d vg-big
>   PV /dev/hda is a different format (lvm1)
>   Restore failed.
> 
> That last one is particularly perverse; I was always told that LVM1 and
> LVM2 are compatible. Of course later I discovered this not to be the
> case; after some operation or other (creating an LV? resizing one?) on
> LVM on 2.6, I could no longer access my LVs under 2.4/LVM1.
> 
> Any help?
> -- 
> Matthew J Toseland - toad@amphibian.dyndns.org
> Freenet Project Official Codemonkey - http://freenetproject.org/
> ICTHUS - Nothing is impossible. Our Boss says so.



-- 
Matthew J Toseland - toad@amphibian.dyndns.org
Freenet Project Official Codemonkey - http://freenetproject.org/
ICTHUS - Nothing is impossible. Our Boss says so.

Re: [linux-lvm] Instability with 2.6, recovering (meta)data?

[Toad]

[-- Attachment #1: Type: text/plain, Size: 3584 bytes --]

This is 100% reproducible on my current system.
root@amphibian:~# lvremove /dev/vg-big/slowtemp
Do you really want to remove active logical volume "slowtemp"? [y/n]: y
  Logical volume "slowtemp" successfully removed
root@amphibian:~# vgdisplay -v
    Finding all volume groups
    Finding volume group "vg-big"
  invalid lv in extent map
  Couldn't fill logical volume maps.
  Volume group "vg-big" doesn't exist

root@amphibian:~# vgcfgrestore vg-big -M1 -f /etc/lvm/archive/vg-big_00003.vg
  Restored volume group vg-big
root@amphibian:~# vgdisplay
[ worked perfectly ]

After restoring the old metadata, /dev/vg-big/slowtemp still doesn't
exist, but when I reboot, it comes back.

If there is anything within reason I can do to assist your debugging of
this otherwise excellent software please contact me.

On Mon, Nov 10, 2003 at 05:46:15PM +0000, toad wrote:
> With agk's help, I eventually found that this worked:
> vgcfgrestore vg-big -M1 -f /etc/lvm/archive/vg-big_00003.vg
> 
> However, the LV that I deleted that originally caused the problem,
> /dev/vg-big/slowtemp (~ 10GB), doesn't exist, and yet it's space usage 
> appears to still be used:
> root@amphibian:~# vgdisplay
> ...
> VG Size               212.81 GB
> PE Size               32.00 MB
> Total PE              6810
> Alloc PE / Size       6704 / 209.50 GB
> Free  PE / Size       106 / 3.31 GB
> 
> As I was originally trying to free up enough space to vgmove off one of
> the disks, in order to replace it and then vgmove off another of the
> disks (IDE madness...), this is still a problem.
> 
> On Sun, Nov 09, 2003 at 09:38:21PM +0000, Toad wrote:
> > Each time I have deleted an LV on LVM2 on 2.6, it has corrupted my
> > metadata. Last time I did a vgcfgrestore on 2.4 and it worked, however I
> > have recently been bitten by some sort of incompatibility bug and my LVM
> > now only works on 2.6 (it might work on a 2.4+devmapper kernel, I'll
> > look into that). Here is what happens when I try to get my metadata
> > back:
> > root@amphibian:/usr/src/linux# vgdisplay -d -v
> >     Finding all volume groups
> >     Finding volume group "vg-big"
> >   invalid lv in extent map
> >   Couldn't fill logical volume maps.
> >   Volume group "vg-big" doesn't exist
> > root@amphibian:/usr/src/linux# pvscan -d -v
> >     Wiping cache of LVM-capable devices
> >     Wiping internal cache
> >     Walking through all physical volumes
> >   invalid lv in extent map
> >   Couldn't fill logical volume maps.
> >   No matching physical volumes found
> > root@amphibian:/usr/src/linux# vgcfgrestore -v -d vg-big
> >   PV /dev/hda is a different format (lvm1)
> >   Restore failed.
> > 
> > That last one is particularly perverse; I was always told that LVM1 and
> > LVM2 are compatible. Of course later I discovered this not to be the
> > case; after some operation or other (creating an LV? resizing one?) on
> > LVM on 2.6, I could no longer access my LVs under 2.4/LVM1.
> > 
> > Any help?
> > -- 
> > Matthew J Toseland - toad@amphibian.dyndns.org
> > Freenet Project Official Codemonkey - http://freenetproject.org/
> > ICTHUS - Nothing is impossible. Our Boss says so.
> 
> 
> 
> -- 
> Matthew J Toseland - toad@amphibian.dyndns.org
> Freenet Project Official Codemonkey - http://freenetproject.org/
> ICTHUS - Nothing is impossible. Our Boss says so.

-- 
Matthew J Toseland - toad@amphibian.dyndns.org
Freenet Project Official Codemonkey - http://freenetproject.org/
ICTHUS - Nothing is impossible. Our Boss says so.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [linux-lvm] Instability with 2.6, recovering (meta)data?

[Alasdair G Kergon]

On Sun, Nov 09, 2003 at 09:38:21PM +0000, Toad wrote:
> after some operation or other (creating an LV? resizing one?) on
> LVM on 2.6, I could no longer access my LVs under 2.4/LVM1.

Likely to have been creating an LV.

The LVM2 CVS head contains a workaround for this; I'll release a new
LVM2 tarball shortly.
 
Alasdair
-- 
agk@uk.sistina.com

Re: [linux-lvm] Instability with 2.6, recovering (meta)data?

[Alasdair G Kergon]

On Mon, Nov 10, 2003 at 05:46:15PM +0000, Toad wrote:
> With agk's help, I eventually found that this worked:
> vgcfgrestore vg-big -M1 -f /etc/lvm/archive/vg-big_00003.vg

LVM2 tools default to using a new improved metadata format.
To tell some tools to use the old LVM1 format of metadata, you need -M1.
(Most tools work it out for themselves, but some, like vgcreate
& vgcfgrestore still need to be told).

You can change the default choice for all tools back to the LVM1 format 
in the config file with:
  global { format = "lvm1" }
 
Alasdair
-- 
agk@uk.sistina.com

Re: [linux-lvm] Instability with 2.6, recovering (meta)data?

[Alasdair G Kergon]

On Mon, Nov 10, 2003 at 05:55:29PM +0000, Toad wrote:
>   invalid lv in extent map

Thanks for sending me a copy of the metadata.

I've reproduced the problem and found the bug when LVM2
writes out LVM1-formatted VG metadata after an LV
has been deleted.  Working on a fix...

Alasdair

Re: [linux-lvm] Instability with 2.6, recovering (meta)data?

[Jose Luis Domingo Lopez]

On Monday, 10 November 2003, at 18:55:03 +0000,
Alasdair G Kergon wrote:

> The LVM2 CVS head contains a workaround for this; I'll release a new
> LVM2 tarball shortly.
>  
Not trying to force anyone to publish code before it is ready ;-), but is
there an aproximate timeframe for the availability of "pvmove" for Linux
kernel version 2.6.x ?

Thank you.

-- 
Jose Luis Domingo Lopez
Linux Registered User #189436     Debian Linux Sid (Linux 2.6.0-test9-mm2)

Re[2]: [linux-lvm] Instability with 2.6, recovering (meta)data?

[Spam]

> On Monday, 10 November 2003, at 18:55:03 +0000,
> Alasdair G Kergon wrote:

>> The LVM2 CVS head contains a workaround for this; I'll release a new
>> LVM2 tarball shortly.
>>  
> Not trying to force anyone to publish code before it is ready ;-), but is
> there an aproximate timeframe for the availability of "pvmove" for Linux
> kernel version 2.6.x ?

  I think it is already working. I have done pvmove in LVM2 for 2.6.0-test9

> Thank you.

Re: [linux-lvm] Instability with 2.6, recovering (meta)data?

[Alasdair G Kergon]

On Mon, Nov 10, 2003 at 11:08:57PM +0000, Alasdair Kergon wrote:
> On Mon, Nov 10, 2003 at 05:55:29PM +0000, Toad wrote:
> >   invalid lv in extent map

Fix is in CVS now.  
I'll release a fresh tarball shortly including this.

The space used by deleted LVs wasn't getting cleared correctly in VGs
with lots of LVs when LVM2 wrote the LVM1-formatted metadata back to
disk.

vgmknodes (which makes / on an LV easier) is also in CVS.

Alasdair
-- 
agk@uk.sistina.com