[linux-lvm] LVM Problems when moving data off a PV

[linux-lvm] LVM Problems when moving data off a PV

[Nick Gushlow]

I've got a volume group (vg2) that spans 3 partitions on two disks;
hdb1, hdb5, hdc.  hdb1 and hdb5 are the only partitions on disk hdb.

Disk hdb is failing and I've got an identical replacement from Maxtor,
so I stuck the new disk in and tried to follow the instructions on the
LVM howto for removing a disk. 

I've successfully transfered hdb5 to the new disk (hdd) but I got an
error when trying to move hdb1; I can't remember that error now but I am
assuming that there may be damage to that part of the disk as the disk
is failing.

After trying a few things I found on the lvm lists and other places and
I'm now at a stage where a vgscan reports

------------------------------------------------------------------------
vgscan -- only found 4253 of 4267 LEs for LV /dev/vg2/storage (0) 
vgscan -- ERROR "vg_read_with_pv_and_lv(): allocated LE of LV" can't get
data of volume group "vg2" from physical volume(s)
------------------------------------------------------------------------

I tried a couple of things from very old posts I found on the lvm
mailing list including compiling LVM2 from cvs so I could do a vgreduce
--removemissing but that didn't work (I assume because they are lvm1) 

Can anyone tell me what might be wrong, and more importantly how I can
get the volume group back up. 

I am assuming that as the data is spread across 3 disks if I can't
recover from this error then I will have lost everything on them. 

Unfortunately I don't have a backup because I have nothing large enough
to backup the data in the volume group. 

TIA,

Nick Gushlow.

Re: [linux-lvm] LVM Problems when moving data off a PV

[Heinz Mauelshagen]

On Sun, Mar 14, 2004 at 01:32:09PM +0000, Nick Gushlow wrote:
> I've got a volume group (vg2) that spans 3 partitions on two disks;
> hdb1, hdb5, hdc.  hdb1 and hdb5 are the only partitions on disk hdb.
> 
> Disk hdb is failing and I've got an identical replacement from Maxtor,
> so I stuck the new disk in and tried to follow the instructions on the
> LVM howto for removing a disk. 
> 
> I've successfully transfered hdb5 to the new disk (hdd) but I got an
> error when trying to move hdb1; I can't remember that error now but I am
> assuming that there may be damage to that part of the disk as the disk
> is failing.
> 
> After trying a few things I found on the lvm lists and other places and
> I'm now at a stage where a vgscan reports

Nick,

what exactly were those 'things' ?

> 
> ------------------------------------------------------------------------
> vgscan -- only found 4253 of 4267 LEs for LV /dev/vg2/storage (0) 
> vgscan -- ERROR "vg_read_with_pv_and_lv(): allocated LE of LV" can't get
> data of volume group "vg2" from physical volume(s)
> ------------------------------------------------------------------------

You need to select the last valid metadata backup of vg2 from /etc/lvmconf
(check it with vgcfgrestore -ll -f /etc/lvmconf/vg2.conf[.#.old]; see
'man vgcfgrestore for metadata archive file names) and restore it.

Say vg.conf.1.old is the valid one:

for dev in AllPVsOfVG2 # list all PVs reported by vgcfgrestore -ll above
do
	pvcreate -ff $dev
	vgcfgrestore -f /etc/lvmconf/vg.conf.1.old $dev
done
vgscan
vgchange -ay vg2

> 
> I tried a couple of things from very old posts I found on the lvm
> mailing list including compiling LVM2 from cvs so I could do a vgreduce
> --removemissing but that didn't work (I assume because they are lvm1) 

The LVM2 tools support LVM1 metadata format.
What went wrong running vgreduce --removemissing ?

In order to activate LVM1 VGs with LVM2, you need a kernel with device-mapper.

> 
> Can anyone tell me what might be wrong, and more importantly how I can
> get the volume group back up. 

See vgcfgrestore above.

> 
> I am assuming that as the data is spread across 3 disks if I can't
> recover from this error then I will have lost everything on them. 
> 
> Unfortunately I don't have a backup because I have nothing large enough
> to backup the data in the volume group. 

<preach>
You need backups in order to be able to survive user/sw/hw failure.
</preach>

> 
> TIA,
> 
> Nick Gushlow.
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Red Hat GmbH
Consulting Development Engineer                   Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@RedHat.com                            +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] LVM Problems when moving data off a PV

[Nick Gushlow]

Thanks for your reply Heinz.

On Mon, 2004-03-15 at 12:34, Heinz Mauelshagen wrote:
> what exactly were those 'things' ?

I can't remember :-( but there was very little of help, most of it was
running some of the tools with different switches to try to find the
error being reported in more detail, but none of it made much sense to
me! :-)


> 
> You need to select the last valid metadata backup of vg2 from /etc/lvmconf
> (check it with vgcfgrestore -ll -f /etc/lvmconf/vg2.conf[.#.old]; see
> 'man vgcfgrestore for metadata archive file names) and restore it.
> 
> Say vg.conf.1.old is the valid one:
> 
> for dev in AllPVsOfVG2 # list all PVs reported by vgcfgrestore -ll above
> do
> 	pvcreate -ff $dev
> 	vgcfgrestore -f /etc/lvmconf/vg.conf.1.old $dev
> done
> vgscan
> vgchange -ay vg2
> 

I'll give that a go, a quick scan of the lvmconf directory shows two
files with recent timestamps, I assume that vg2.conf is not good
otherwise all would work, so vg2.conf.1.old must be the one.  All the
others are dated months ago before I transferred one of the partitions.

Time to hit those man pages! :-)

> The LVM2 tools support LVM1 metadata format.
> What went wrong running vgreduce --removemissing ?
> 
> In order to activate LVM1 VGs with LVM2, you need a kernel with device-mapper.

Off the top of my head I don't remember, but a quick look at my kernel
config shows no device mapper support, so I guess that's it.


What would you recommend doing first? your suggestion about the metadata
backup or recompiling the kernel and running the lvm2 tools?

> <preach>
> You need backups in order to be able to survive user/sw/hw failure.
> </preach>

Yeah I know, fortunately the critical drives are backed up, this is none
important stuff that can be replaced (but would be a pain in the
backside.)


Say I don't manage to recover the vg, is it possible to create a new vg
out of the pvs on the good disks and recover the data that's on those?

Nick.

Re: [linux-lvm] LVM Problems when moving data off a PV

[Nick Gushlow]

[-- Attachment #1: Type: text/plain, Size: 1198 bytes --]

On Mon, 2004-03-15 at 12:34, Heinz Mauelshagen wrote:

> 
> You need to select the last valid metadata backup of vg2 from /etc/lvmconf
> (check it with vgcfgrestore -ll -f /etc/lvmconf/vg2.conf[.#.old]; see
> 'man vgcfgrestore for metadata archive file names) and restore it.
> 
> Say vg.conf.1.old is the valid one:
> 
> for dev in AllPVsOfVG2 # list all PVs reported by vgcfgrestore -ll above
> do
> 	pvcreate -ff $dev
> 	vgcfgrestore -f /etc/lvmconf/vg.conf.1.old $dev
> done
> vgscan
> vgchange -ay vg2
> 


That did the trick thanks. :-)

Of course in a moment of not concentrating I messed things up again.  I
still had pvremove on the disk from when I previously installed the lvm2
tools, and although I wanted to do a vgreduce I did a pvremove.  Even
when it kicked up an error I just followed what it said and typed
pvremove -ff. oops :-/

Now I get a a similar error as before with vgscan, and other tools give
a 'VGDA in kernel and lvmtab are NTO consistent: please run vgscan'
error.

I tried re-running what you suggested above but vgcfgrestore gives a
'can't restore part of active volume group "vg2"' error.

Is there anyway to recover from my stupidity (other than a lobotomy!)?




[-- Attachment #2: Type: text/html, Size: 1550 bytes --]

Re: [linux-lvm] LVM Problems when moving data off a PV

[Heinz Mauelshagen]

On Tue, Mar 16, 2004 at 11:04:09PM +0000, Nick Gushlow wrote:
> On Mon, 2004-03-15 at 12:34, Heinz Mauelshagen wrote:
> 
> > 
> > You need to select the last valid metadata backup of vg2 from /etc/lvmconf
> > (check it with vgcfgrestore -ll -f /etc/lvmconf/vg2.conf[.#.old]; see
> > 'man vgcfgrestore for metadata archive file names) and restore it.
> > 
> > Say vg.conf.1.old is the valid one:
> > 
> > for dev in AllPVsOfVG2 # list all PVs reported by vgcfgrestore -ll above
> > do
> > 	pvcreate -ff $dev
> > 	vgcfgrestore -f /etc/lvmconf/vg.conf.1.old $dev
> > done
> > vgscan
> > vgchange -ay vg2
> > 
> 
> 
> That did the trick thanks. :-)
> 
> Of course in a moment of not concentrating I messed things up again.  I
> still had pvremove on the disk from when I previously installed the lvm2
> tools, and although I wanted to do a vgreduce I did a pvremove.  Even
> when it kicked up an error I just followed what it said and typed
> pvremove -ff. oops :-/
> 
> Now I get a a similar error as before with vgscan, and other tools give
> a 'VGDA in kernel and lvmtab are NTO consistent: please run vgscan'
> error.
> 
> I tried re-running what you suggested above but vgcfgrestore gives a
> 'can't restore part of active volume group "vg2"' error.
> 
> Is there anyway to recover from my stupidity (other than a lobotomy!)?

:)

Deactivate the VG and rerun pvcreate/vgcfgrestore on the pvremoved disk,
vgscan and "vgchange -ay vg2" afterwards.

> 
> 
> 

> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Red Hat GmbH
Consulting Development Engineer                   Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@RedHat.com                            +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] LVM Problems when moving data off a PV

[Nick Gushlow]

[-- Attachment #1: Type: text/plain, Size: 463 bytes --]

On Wed, 2004-03-17 at 14:31, Heinz Mauelshagen wrote:

> 
> Deactivate the VG and rerun pvcreate/vgcfgrestore on the pvremoved disk,
> vgscan and "vgchange -ay vg2" afterwards.
> 

Excellent that did the trick. :-) 

The only way I found to get rid of the "VGDA in kernel and lvmtab are
NOT consistent: please run vgscan" error when running 'vgchange -an vg2"
was to reboot.

For future reference, is there a better/proper method to get rid of that
error?

Nick.

[-- Attachment #2: Type: text/html, Size: 814 bytes --]

Re: [linux-lvm] LVM Problems when moving data off a PV

[Heinz Mauelshagen]

On Wed, Mar 17, 2004 at 08:03:04PM +0000, Nick Gushlow wrote:
> On Wed, 2004-03-17 at 14:31, Heinz Mauelshagen wrote:
> 
> > 
> > Deactivate the VG and rerun pvcreate/vgcfgrestore on the pvremoved disk,
> > vgscan and "vgchange -ay vg2" afterwards.
> > 
> 
> Excellent that did the trick. :-) 
> 
> The only way I found to get rid of the "VGDA in kernel and lvmtab are
> NOT consistent: please run vgscan" error when running 'vgchange -an vg2"
> was to reboot.
> 
> For future reference, is there a better/proper method to get rid of that
> error?

If you are able to deactivate the VG without a reboot, which is rather unlikely
because you've got filesystems on LVs mounted etc., do that.

Unless the LVM metadata gets overwritten externally,
LVM will keep both (metadata on disk and in kernel) in sync.

> 
> Nick.

> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Red Hat GmbH
Consulting Development Engineer                   Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@RedHat.com                            +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-