[linux-lvm] Hard disk failure, restore problem

[linux-lvm] Hard disk failure, restore problem

[Thomas Berton]

Hi,
I've recently encountered some problems with my LVM.

I had (and still have) a bad hd /dev/hda (used part. /dev/hda1)

I started:
    pvmove /dev/hda1 /dev/hdf1
yet the hd gave an error and I had to reboot.

I have a metadata backup made before I've started the move.

The volume group name is bigspace, yet the 'dir' /dev/bigspace/bigspace1 
is also gone...

now I got
pvscan -- reading all physical volumes (this may take a while...)
pvscan -- inactive PV "/dev/hdg1" is in no VG  [74.53 GB]
pvscan -- inactive PV "/dev/hde1" is in no VG  [149.05 GB]
pvscan -- inactive PV "/dev/hdf1" is in no VG  [149.05 GB]
pvscan -- inactive PV "/dev/hdc1" is in no VG  [111.79 GB]
pvscan -- inactive PV "/dev/hdd1" is in no VG  [74.53 GB]
pvscan -- inactive PV "/dev/hda1" is in no VG  [74.53 GB]
pvscan -- inactive PV "/dev/hdb1" is in no VG  [74.53 GB]
pvscan -- total: 7 [708 GB] / in use: 0 [0] / in no VG: 7 [708 GB]

yet if I do the restore I get

I also did a pvcreate -ff /dev/hdx1 for every disk... I've read 
somewhere that should fix it. yet id did not.

pvscan -- inactive PV "/dev/hda1"  is associated to unknown VG 
"bigspace" (run vgscan)

vgscan gives
vgscan -- reading all physical volumes (this may take a while...)
vgscan -- ERROR "vg_read_with_pv_and_lv(): current PV" can't get data of 
volume group "bigspace" from physical volume(s)
vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created
vgscan -- WARNING: This program does not do a VGDA backup of your volume 
group

there is a LOT of data on this array of disk (as you can imagine)

sometimes, hda is working good, so i hope it want to work long enough to 
restore the data.

My guess now is to make a new vg and add those disks... but I want to be 
sure, so I don't lose the data.

Any suggestions to recover the data or to replace the broken harddisk 
with a new one with as little data loss as possible.

Regards,
Thomas.

Re: [linux-lvm] Hard disk failure, restore problem

[Heinz Mauelshagen]

On Fri, Mar 12, 2004 at 11:22:44AM -0800, Thomas Berton wrote:
> Hi,
> I've recently encountered some problems with my LVM.
> 
> I had (and still have) a bad hd /dev/hda (used part. /dev/hda1)
> 
> I started:
>    pvmove /dev/hda1 /dev/hdf1
> yet the hd gave an error and I had to reboot.
> 
> I have a metadata backup made before I've started the move.
> 
> The volume group name is bigspace, yet the 'dir' /dev/bigspace/bigspace1 
> is also gone...
> 
> now I got
> pvscan -- reading all physical volumes (this may take a while...)
> pvscan -- inactive PV "/dev/hdg1" is in no VG  [74.53 GB]
> pvscan -- inactive PV "/dev/hde1" is in no VG  [149.05 GB]
> pvscan -- inactive PV "/dev/hdf1" is in no VG  [149.05 GB]
> pvscan -- inactive PV "/dev/hdc1" is in no VG  [111.79 GB]
> pvscan -- inactive PV "/dev/hdd1" is in no VG  [74.53 GB]
> pvscan -- inactive PV "/dev/hda1" is in no VG  [74.53 GB]
> pvscan -- inactive PV "/dev/hdb1" is in no VG  [74.53 GB]
> pvscan -- total: 7 [708 GB] / in use: 0 [0] / in no VG: 7 [708 GB]

Thomas,

you ran 'pvcreate -ff' *and* vgcfgrestore ... on all PVs ok ?
pvscan output looks like the vgcfgrestore went wrong.

> 
> yet if I do the restore I get
> 
> I also did a pvcreate -ff /dev/hdx1 for every disk... I've read 
> somewhere that should fix it. yet id did not.

Yes, you need "pvcreate -ff ..." before you vgcfgrestore to the devices.

> 
> pvscan -- inactive PV "/dev/hda1"  is associated to unknown VG 
> "bigspace" (run vgscan)
> 
> vgscan gives
> vgscan -- reading all physical volumes (this may take a while...)
> vgscan -- ERROR "vg_read_with_pv_and_lv(): current PV" can't get data of 
> volume group "bigspace" from physical volume(s)
> vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created
> vgscan -- WARNING: This program does not do a VGDA backup of your volume 
> group
> 
> there is a LOT of data on this array of disk (as you can imagine)
> 
> sometimes, hda is working good, so i hope it want to work long enough to 
> restore the data.

If you want to activate "bigspace" just in order to backup date *and*
remove/recreate it afterwards, you might want to use the following workaround:

o identify your actual metadata backup (say, bigspace.conf)

o mkdir /etc/lvmtab.d

o cp bigspace.conf /etc/lvmtab.d/bigspace

o echo -ne "bigspace\0" > /etc/lvmtab

o vgchange -ay

> 
> My guess now is to make a new vg and add those disks... but I want to be 
> sure, so I don't lose the data.

That won't work, because the PVs are falgged *new* after pvcreate -ff
and you will just have empty PVs that way.

> 
> Any suggestions to recover the data or to replace the broken harddisk 
> with a new one with as little data loss as possible.
> 
> Regards,
> Thomas.
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Red Hat GmbH
Consulting Development Engineer                   Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@RedHat.com                            +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] Hard disk failure, restore problem

[Thomas Berton]

Hello,

I've done

 > o identify your actual metadata backup (say, bigspace.conf)
 >
 > o mkdir /etc/lvmtab.d
 >
 > o cp bigspace.conf /etc/lvmtab.d/bigspace
 >
 > o echo -ne "bigspace\0" > /etc/lvmtab
 >
 > o vgchange -ay


after being sure to restore the metadata I have

pvscan -- reading all physical volumes (this may take a while...)
pvscan -- inactive PV "/dev/hdg1" of VG "bigspace" [74.47 GB / 768 MB free]
pvscan -- inactive PV "/dev/hde1" of VG "bigspace" [149 GB / 0 free]
pvscan -- inactive PV "/dev/hdf1" of VG "bigspace" [149 GB / 139.16 GB free]
pvscan -- inactive PV "/dev/hdc1" of VG "bigspace" [111.75 GB / 0 free]
pvscan -- inactive PV "/dev/hdd1" of VG "bigspace" [74.47 GB / 0 free]
pvscan -- inactive PV "/dev/hda1" of VG "bigspace" [74.47 GB / 8.22 GB free]
pvscan -- inactive PV "/dev/hdb1" of VG "bigspace" [74.47 GB / 0 free]
pvscan -- total: 7 [708 GB] / in use: 7 [708 GB] / in no VG: 0 [0]


vgscan -- reading all physical volumes (this may take a while...)
vgscan -- only found 17904 of 17876 LEs for LV /dev/bigspace/bigspace1 (0)
vgscan -- ERROR "vg_read_with_pv_and_lv(): allocated LE of LV" can't get
data of volume group "bigspace" from physical volume(s)
vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created
vgscan -- WARNING: This program does not do a VGDA backup of your volume
group

=> after the vgscan the echo -ne "bigspace\0" > /etc/lvmtab is gone gone
again....

[root@ment root]# lvscan
lvscan -- ERROR: VGDA in kernel and lvmtab are NOT consistent; please
run vgscan

after doing the vgscan, I get

pvscan -- reading all physical volumes (this may take a while...)
pvscan -- inactive PV "/dev/hdg1"  is associated to unknown VG
"bigspace" (run vgscan)
pvscan -- inactive PV "/dev/hde1"  is associated to unknown VG
"bigspace" (run vgscan)
pvscan -- inactive PV "/dev/hdf1"  is associated to unknown VG
"bigspace" (run vgscan)
pvscan -- inactive PV "/dev/hdc1"  is associated to unknown VG
"bigspace" (run vgscan)
pvscan -- inactive PV "/dev/hdd1"  is associated to unknown VG
"bigspace" (run vgscan)
pvscan -- inactive PV "/dev/hda1"  is associated to unknown VG
"bigspace" (run vgscan)
pvscan -- inactive PV "/dev/hdb1"  is associated to unknown VG
"bigspace" (run vgscan)
pvscan -- total: 7 [708 GB] / in use: 7 [708 GB] / in no VG: 0 [0]

again.


the /dev/bigspace/bigspace1 doesn't exists, so I can't mount.

should I recreate bigspace and force an import ?

I 've alse rebooted once after the vgchange -ay, but he also didn't find
the bigspace.

thanx again for your help!!

kind regards,
Thomas.

Re: [linux-lvm] Hard disk failure, restore problem

[Heinz Mauelshagen]

Thomas,

don't run vgscan after the procedure below.

Remember this is a workaround *just* to get you to the point to be able
to backup your data and recreate your VG.

Run vgmknodes to create your device nodes in /dev/bigspace/
after step "vgchange -ay" below.

Regards,
Heinz    -- The LVM Guy --


On Mon, Mar 15, 2004 at 09:52:02PM +0100, Thomas Berton wrote:
> Hello,
> 
> I've done
> 
> > o identify your actual metadata backup (say, bigspace.conf)
> >
> > o mkdir /etc/lvmtab.d
> >
> > o cp bigspace.conf /etc/lvmtab.d/bigspace
> >
> > o echo -ne "bigspace\0" > /etc/lvmtab
> >
> > o vgchange -ay
> 
> 
> after being sure to restore the metadata I have
> 
> pvscan -- reading all physical volumes (this may take a while...)
> pvscan -- inactive PV "/dev/hdg1" of VG "bigspace" [74.47 GB / 768 MB free]
> pvscan -- inactive PV "/dev/hde1" of VG "bigspace" [149 GB / 0 free]
> pvscan -- inactive PV "/dev/hdf1" of VG "bigspace" [149 GB / 139.16 GB free]
> pvscan -- inactive PV "/dev/hdc1" of VG "bigspace" [111.75 GB / 0 free]
> pvscan -- inactive PV "/dev/hdd1" of VG "bigspace" [74.47 GB / 0 free]
> pvscan -- inactive PV "/dev/hda1" of VG "bigspace" [74.47 GB / 8.22 GB free]
> pvscan -- inactive PV "/dev/hdb1" of VG "bigspace" [74.47 GB / 0 free]
> pvscan -- total: 7 [708 GB] / in use: 7 [708 GB] / in no VG: 0 [0]
> 
> 
> vgscan -- reading all physical volumes (this may take a while...)
> vgscan -- only found 17904 of 17876 LEs for LV /dev/bigspace/bigspace1 (0)
> vgscan -- ERROR "vg_read_with_pv_and_lv(): allocated LE of LV" can't get
> data of volume group "bigspace" from physical volume(s)
> vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created
> vgscan -- WARNING: This program does not do a VGDA backup of your volume
> group
> 
> => after the vgscan the echo -ne "bigspace\0" > /etc/lvmtab is gone gone
> again....
> 
> [root@ment root]# lvscan
> lvscan -- ERROR: VGDA in kernel and lvmtab are NOT consistent; please
> run vgscan
> 
> after doing the vgscan, I get
> 
> pvscan -- reading all physical volumes (this may take a while...)
> pvscan -- inactive PV "/dev/hdg1"  is associated to unknown VG
> "bigspace" (run vgscan)
> pvscan -- inactive PV "/dev/hde1"  is associated to unknown VG
> "bigspace" (run vgscan)
> pvscan -- inactive PV "/dev/hdf1"  is associated to unknown VG
> "bigspace" (run vgscan)
> pvscan -- inactive PV "/dev/hdc1"  is associated to unknown VG
> "bigspace" (run vgscan)
> pvscan -- inactive PV "/dev/hdd1"  is associated to unknown VG
> "bigspace" (run vgscan)
> pvscan -- inactive PV "/dev/hda1"  is associated to unknown VG
> "bigspace" (run vgscan)
> pvscan -- inactive PV "/dev/hdb1"  is associated to unknown VG
> "bigspace" (run vgscan)
> pvscan -- total: 7 [708 GB] / in use: 7 [708 GB] / in no VG: 0 [0]
> 
> again.
> 
> 
> the /dev/bigspace/bigspace1 doesn't exists, so I can't mount.
> 
> should I recreate bigspace and force an import ?
> 
> I 've alse rebooted once after the vgchange -ay, but he also didn't find
> the bigspace.
> 
> thanx again for your help!!
> 
> kind regards,
> Thomas.

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Red Hat GmbH
Consulting Development Engineer                   Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@RedHat.com                            +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-