All of lore.kernel.org
 help / color / mirror / Atom feed
* [linux-lvm] LVM1 and a dying disk
@ 2003-09-03  3:55 Rickard Olsson
  2003-09-03  4:23 ` Heinz J . Mauelshagen
  0 siblings, 1 reply; 6+ messages in thread
From: Rickard Olsson @ 2003-09-03  3:55 UTC (permalink / raw)
  To: linux-lvm

One of the disks in my /dev/files_vg vg have been making weird noises 
for a while. This vg is a JBOD with the following disks in it:


majestix root # pvscan
pvscan -- reading all physical volumes (this may take a while...)
pvscan -- ACTIVE   PV "/dev/ide/host2/bus1/target0/lun0/disc" of VG "f2"
      [149 GB / 0 free]
pvscan -- inactive PV "/dev/ide/host2/bus0/target0/lun0/disc"  is in 
EXPORTED VG
  "files_vg" [149.03 GB / 0 free]
pvscan -- inactive PV "/dev/ide/host2/bus0/target1/lun0/disc"  is in 
EXPORTED VG
  "files_vg" [111.77 GB / 0 free]
pvscan -- inactive PV "/dev/ide/host0/bus1/target1/lun0/disc"  is in 
EXPORTED VG
  "files_vg" [93.14 GB / 0 free]
pvscan -- inactive PV "/dev/ide/host0/bus0/target0/lun0/disc"  is in 
EXPORTED VG
  "files_vg" [111.77 GB / 0 free]
pvscan -- inactive PV "/dev/ide/host0/bus0/target1/lun0/disc"  is in 
EXPORTED VG
  "files_vg" [93.14 GB / 0 free]
pvscan -- total: 6 [708 GB] / in use: 6 [708 GB] / in no VG: 0 [0]


Exported, you say? Yep. The machine that this vg was in suffered a fatal 
root filesystem corruption last friday - not related to this disk (I 
name it Black Friday, that same day my VW Passat dropped the rear 
muffler and my upstream ISP had some major network issues that still 
isn't resolved).
The f2 vg is the new vg I'm building to move data over to.


majestix root # vgscan
vgscan -- reading all physical volumes (this may take a while...)
vgscan -- found active volume group "f2"
vgscan -- ERROR "vg_read_with_pv_and_lv(): current PV" can't get data of 
volume
group "files_vgPV_EXP" from physical volume(s)
vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created
vgscan -- WARNING: This program does not do a VGDA backup of your volume 
groups


And devfs? Gentoo. Devfs is a major and total pain when trying to do a 
vgimport, listing 6 of those long paths on the command line is no fun. 
And, it doesn't work:


vgimport -- ERROR "pv_read(): PV identifier invalid" reading physical 
volume "/d
ev/ide/host0/bus1/target0/lun0/disc"


I did manage to export the vg before the machine died permanently. The 
backups I had of the root drive are older then the last major change to 
the vg (addition of the Seagate 160GB that shows up as 149.03 GB above). 
The troublesome disk is not showing in the pvscan, it's currently on 
/dev/ide/host0/bus1/target0/lun0/disc and a pvdata for it reveals:


majestix root # pvdata /dev/ide/host0/bus1/target0/lun0/disc
--- Physical volume ---
PV Name               /dev/ide/host0/bus1/target0/lun0/disc
VG Name               ÇD?
PV Size               111.79 GB [234441648 secs] / NOT usable 16.19 MB 
[LVM: 155 KB]
PV#                   0
PV Status             NOT available
Allocatable           yes (but full)
Cur LV                9306113
PE Size (KByte)       16384
Total PE              7153
Free PE               0
Allocated PE          7153
PV UUID               rDBb94-Kb8v-GMDa-HxIi-???

Segmentation fault


As you can see, not all fun and games.

Now, is there any way to re-initialize the pv data on the failing disk 
long enough to get it back into the vg and extract the data? The data 
left in the vg is not critically important (I started backing it up as 
soon as I noticed the noises), but it would be really nice to get it back.

I have considered pvcreating it, but I'm not sure how I'd get it into 
the vg afterwards - would a vgscan/vgchange suffice? Would a reiserfsck 
--rebuild_tree then prune the fs of anything that was on this disk?

    / Rickard Olsson,IT-Konsult/
   / Telefon: +46 70 635 01 42/
  / http://www.webhackande.se/

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [linux-lvm] LVM1 and a dying disk
  2003-09-03  3:55 [linux-lvm] LVM1 and a dying disk Rickard Olsson
@ 2003-09-03  4:23 ` Heinz J . Mauelshagen
  2003-09-03  4:40   ` Rickard Olsson
  2003-09-03  5:53   ` Rickard Olsson
  0 siblings, 2 replies; 6+ messages in thread
From: Heinz J . Mauelshagen @ 2003-09-03  4:23 UTC (permalink / raw)
  To: linux-lvm

Rickard,

you say that you don't have a recent LVM metadata backup reflecting
your VG with /dev/ide/host0/bus1/target0/lun0/disc in it ?

If you had, normal procedure would be to

# pvcreate -ff /dev/ide/host0/bus1/target0/lun0/disc

and to

# vgcfgrestore -n YourVGName /dev/ide/host0/bus1/target0/lun0/disc
# vgscan
# vgchange -ay

(Because you exported, you need to pvcreate+vgcfgrestore to all PVs)

If /dev/ide/host0/bus1/target0/lun0/disc took a serious overwrite
(and it looks like it eyeing the pvdata output below), there's almost no
chance to cook up the correct metadata by hand I'm afraid.

People, please take LVM metadata backups!

Regards,
Heinz    -- The LVM Guy --


On Wed, Sep 03, 2003 at 10:53:12AM +0200, Rickard Olsson wrote:
> One of the disks in my /dev/files_vg vg have been making weird noises 
> for a while. This vg is a JBOD with the following disks in it:
> 
> 
> majestix root # pvscan
> pvscan -- reading all physical volumes (this may take a while...)
> pvscan -- ACTIVE   PV "/dev/ide/host2/bus1/target0/lun0/disc" of VG "f2"
>       [149 GB / 0 free]
> pvscan -- inactive PV "/dev/ide/host2/bus0/target0/lun0/disc"  is in 
> EXPORTED VG
>   "files_vg" [149.03 GB / 0 free]
> pvscan -- inactive PV "/dev/ide/host2/bus0/target1/lun0/disc"  is in 
> EXPORTED VG
>   "files_vg" [111.77 GB / 0 free]
> pvscan -- inactive PV "/dev/ide/host0/bus1/target1/lun0/disc"  is in 
> EXPORTED VG
>   "files_vg" [93.14 GB / 0 free]
> pvscan -- inactive PV "/dev/ide/host0/bus0/target0/lun0/disc"  is in 
> EXPORTED VG
>   "files_vg" [111.77 GB / 0 free]
> pvscan -- inactive PV "/dev/ide/host0/bus0/target1/lun0/disc"  is in 
> EXPORTED VG
>   "files_vg" [93.14 GB / 0 free]
> pvscan -- total: 6 [708 GB] / in use: 6 [708 GB] / in no VG: 0 [0]
> 
> 
> Exported, you say? Yep. The machine that this vg was in suffered a fatal 
> root filesystem corruption last friday - not related to this disk (I 
> name it Black Friday, that same day my VW Passat dropped the rear 
> muffler and my upstream ISP had some major network issues that still 
> isn't resolved).
> The f2 vg is the new vg I'm building to move data over to.
> 
> 
> majestix root # vgscan
> vgscan -- reading all physical volumes (this may take a while...)
> vgscan -- found active volume group "f2"
> vgscan -- ERROR "vg_read_with_pv_and_lv(): current PV" can't get data of 
> volume
> group "files_vgPV_EXP" from physical volume(s)
> vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created
> vgscan -- WARNING: This program does not do a VGDA backup of your volume 
> groups
> 
> 
> And devfs? Gentoo. Devfs is a major and total pain when trying to do a 
> vgimport, listing 6 of those long paths on the command line is no fun. 
> And, it doesn't work:
> 
> 
> vgimport -- ERROR "pv_read(): PV identifier invalid" reading physical 
> volume "/d
> ev/ide/host0/bus1/target0/lun0/disc"
> 
> 
> I did manage to export the vg before the machine died permanently. The 
> backups I had of the root drive are older then the last major change to 
> the vg (addition of the Seagate 160GB that shows up as 149.03 GB above). 
> The troublesome disk is not showing in the pvscan, it's currently on 
> /dev/ide/host0/bus1/target0/lun0/disc and a pvdata for it reveals:
> 
> 
> majestix root # pvdata /dev/ide/host0/bus1/target0/lun0/disc
> --- Physical volume ---
> PV Name               /dev/ide/host0/bus1/target0/lun0/disc
> VG Name               ÇD?
> PV Size               111.79 GB [234441648 secs] / NOT usable 16.19 MB 
> [LVM: 155 KB]
> PV#                   0
> PV Status             NOT available
> Allocatable           yes (but full)
> Cur LV                9306113
> PE Size (KByte)       16384
> Total PE              7153
> Free PE               0
> Allocated PE          7153
> PV UUID               rDBb94-Kb8v-GMDa-HxIi-???
> 
> Segmentation fault
> 
> 
> As you can see, not all fun and games.
> 
> Now, is there any way to re-initialize the pv data on the failing disk 
> long enough to get it back into the vg and extract the data? The data 
> left in the vg is not critically important (I started backing it up as 
> soon as I noticed the noises), but it would be really nice to get it back.
> 
> I have considered pvcreating it, but I'm not sure how I'd get it into 
> the vg afterwards - would a vgscan/vgchange suffice? Would a reiserfsck 
> --rebuild_tree then prune the fs of anything that was on this disk?
> 
>     / Rickard Olsson,IT-Konsult/
>    / Telefon: +46 70 635 01 42/
>   / http://www.webhackande.se/
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [linux-lvm] LVM1 and a dying disk
  2003-09-03  4:23 ` Heinz J . Mauelshagen
@ 2003-09-03  4:40   ` Rickard Olsson
  2003-09-03  5:53   ` Rickard Olsson
  1 sibling, 0 replies; 6+ messages in thread
From: Rickard Olsson @ 2003-09-03  4:40 UTC (permalink / raw)
  To: linux-lvm

Heinz J . Mauelshagen wrote:

> you say that you don't have a recent LVM metadata backup reflecting
> your VG with /dev/ide/host0/bus1/target0/lun0/disc in it ?

Well, possibly. Many of the files could be rescued off the damaged root 
file system, even if the directory structure was shot. What file(s) 
would I be looking for? There's 18487 numbered directories in lost+found 
with a large number of subdirectories - but if I have a name to go on, I 
may be able to find something useful in there.

    / Rickard Olsson,IT-Konsult/
   / Telefon: +46 70 635 01 42/
  / http://www.webhackande.se/

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [linux-lvm] LVM1 and a dying disk
  2003-09-03  4:23 ` Heinz J . Mauelshagen
  2003-09-03  4:40   ` Rickard Olsson
@ 2003-09-03  5:53   ` Rickard Olsson
  2003-09-03  7:46     ` Heinz J . Mauelshagen
  1 sibling, 1 reply; 6+ messages in thread
From: Rickard Olsson @ 2003-09-03  5:53 UTC (permalink / raw)
  To: linux-lvm

Heinz J . Mauelshagen wrote:

> you say that you don't have a recent LVM metadata backup reflecting
> your VG with /dev/ide/host0/bus1/target0/lun0/disc in it ?

I found a files_vg.conf and put it in /etc/lvmconf, did pvcreate and 
successful vgcfgrestores on all of them. I had to move the disks around 
a bit to make them appear in their old places since they're now in a new 
machine (vgcfgrestore complained otherwise). The bad disk is now at 
/dev/ide/host2/bus1/target1/lun0/disc


majestix root # pvscan
pvscan -- reading all physical volumes (this may take a while...)
pvscan -- ACTIVE   PV "/dev/ide/host4/bus1/target0/lun0/disc" of VG "f2" 
       [
149 GB / 0 free]
pvscan -- inactive PV "/dev/ide/host2/bus1/target0/lun0/disc"  is 
associated to
unknown VG "files_vg" (run vgscan)
pvscan -- inactive PV "/dev/ide/host2/bus1/target1/lun0/disc"  is 
associated to
unknown VG "files_vg" (run vgscan)
pvscan -- inactive PV "/dev/ide/host2/bus0/target0/lun0/disc"  is 
associated to
unknown VG "files_vg" (run vgscan)
pvscan -- inactive PV "/dev/ide/host2/bus0/target1/lun0/disc"  is 
associated to
unknown VG "files_vg" (run vgscan)
pvscan -- inactive PV "/dev/ide/host0/bus1/target0/lun0/disc"  is 
associated to
unknown VG "files_vg" (run vgscan)
pvscan -- inactive PV "/dev/ide/host0/bus1/target1/lun0/disc"  is 
associated to
unknown VG "files_vg" (run vgscan)
pvscan -- total: 7 [819.79 GB] / in use: 7 [819.79 GB] / in no VG: 0 [0]


majestix root # vgscan
vgscan -- reading all physical volumes (this may take a while...)
vgscan -- found active volume group "f2"
vgscan -- ERROR "vg_read_with_pv_and_lv(): current PV" can't get data of 
volume
group "files_vg" from physical volume(s)
vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created
vgscan -- WARNING: This program does not do a VGDA backup of your volume 
groups


majestix root # pvdata /dev/ide/host2/bus1/target1/lun0/disc
--- Physical volume ---
PV Name               /dev/ide/host2/bus1/target1/lun0/disc
VG Name               files_vg
PV Size               93.16 GB [195371568 secs] / NOT usable 16.19 MB 
[LVM: 151
KB]
PV#                   4
PV Status             NOT available
Allocatable           yes (but full)
Cur LV                1
PE Size (KByte)       16384
Total PE              5961
Free PE               0
Allocated PE          5961
PV UUID               KsET9a-qoKI-a2gr-z1GB-jwM1-vFw7-qHgxeZ

--- Volume group ---
VG Name
VG Access             read/write
VG Status             NOT available/resizable
VG #                  0
MAX LV                256
Cur LV                1
Open LV               0
MAX LV Size           1023.97 GB
Max PV                256
Cur PV                6
Act PV                6
VG Size               670.61 GB
PE Size               16 MB
Total PE              42919
Alloc PE / Size       42919 / 670.61 GB
Free  PE / Size       0 / 0
VG UUID               Z9MJ9w-UjoI-BO9X-rAb0-DPtF-pXqO-Haiji7

--- List of logical volumes ---

pvdata -- logical volume "/dev/files_vg/files_lv" at offset   0
pvdata -- logical volume struct at offset   1 is empty
[...]


What did I do wrong (this time)?

    / Rickard Olsson,IT-Konsult/
   / Telefon: +46 70 635 01 42/
  / http://www.webhackande.se/

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [linux-lvm] LVM1 and a dying disk
  2003-09-03  5:53   ` Rickard Olsson
@ 2003-09-03  7:46     ` Heinz J . Mauelshagen
  2003-09-16  6:54       ` [linux-lvm] LVM1 and a dying disk (fixed) Rickard Olsson
  0 siblings, 1 reply; 6+ messages in thread
From: Heinz J . Mauelshagen @ 2003-09-03  7:46 UTC (permalink / raw)
  To: linux-lvm

On Wed, Sep 03, 2003 at 12:52:39PM +0200, Rickard Olsson wrote:
> Heinz J . Mauelshagen wrote:
> 
> > you say that you don't have a recent LVM metadata backup reflecting
> > your VG with /dev/ide/host0/bus1/target0/lun0/disc in it ?
> 
> I found a files_vg.conf and put it in /etc/lvmconf, did pvcreate and 
> successful vgcfgrestores on all of them. I had to move the disks around 
> a bit to make them appear in their old places since they're now in a new 
> machine (vgcfgrestore complained otherwise).

"vgcfgrestore -o ..." does the trick without moving disks around
(which might be completely impossible with certain path changes anyway).
"-o" takes the old path, which has been stored in the metadata backup.

"vgcfgrestore -ll ..." long lists the VG metadata stored in the backup.

> The bad disk is now at 
> /dev/ide/host2/bus1/target1/lun0/disc
> 
> 
> majestix root # pvscan
> pvscan -- reading all physical volumes (this may take a while...)
> pvscan -- ACTIVE   PV "/dev/ide/host4/bus1/target0/lun0/disc" of VG "f2" 
>        [
> 149 GB / 0 free]
> pvscan -- inactive PV "/dev/ide/host2/bus1/target0/lun0/disc"  is 
> associated to
> unknown VG "files_vg" (run vgscan)
> pvscan -- inactive PV "/dev/ide/host2/bus1/target1/lun0/disc"  is 
> associated to
> unknown VG "files_vg" (run vgscan)
> pvscan -- inactive PV "/dev/ide/host2/bus0/target0/lun0/disc"  is 
> associated to
> unknown VG "files_vg" (run vgscan)
> pvscan -- inactive PV "/dev/ide/host2/bus0/target1/lun0/disc"  is 
> associated to
> unknown VG "files_vg" (run vgscan)
> pvscan -- inactive PV "/dev/ide/host0/bus1/target0/lun0/disc"  is 
> associated to
> unknown VG "files_vg" (run vgscan)
> pvscan -- inactive PV "/dev/ide/host0/bus1/target1/lun0/disc"  is 
> associated to
> unknown VG "files_vg" (run vgscan)
> pvscan -- total: 7 [819.79 GB] / in use: 7 [819.79 GB] / in no VG: 0 [0]
> 
> 
> majestix root # vgscan
> vgscan -- reading all physical volumes (this may take a while...)
> vgscan -- found active volume group "f2"
> vgscan -- ERROR "vg_read_with_pv_and_lv(): current PV" can't get data of 
> volume
> group "files_vg" from physical volume(s)
> vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created
> vgscan -- WARNING: This program does not do a VGDA backup of your volume 
> groups
> 
> 
> majestix root # pvdata /dev/ide/host2/bus1/target1/lun0/disc
> --- Physical volume ---
> PV Name               /dev/ide/host2/bus1/target1/lun0/disc
> VG Name               files_vg
> PV Size               93.16 GB [195371568 secs] / NOT usable 16.19 MB 
> [LVM: 151
> KB]
> PV#                   4
> PV Status             NOT available
> Allocatable           yes (but full)
> Cur LV                1
> PE Size (KByte)       16384
> Total PE              5961
> Free PE               0
> Allocated PE          5961
> PV UUID               KsET9a-qoKI-a2gr-z1GB-jwM1-vFw7-qHgxeZ
> 
> --- Volume group ---
> VG Name
> VG Access             read/write
> VG Status             NOT available/resizable
> VG #                  0
> MAX LV                256
> Cur LV                1
> Open LV               0
> MAX LV Size           1023.97 GB
> Max PV                256
> Cur PV                6
> Act PV                6
> VG Size               670.61 GB
> PE Size               16 MB
> Total PE              42919
> Alloc PE / Size       42919 / 670.61 GB
> Free  PE / Size       0 / 0
> VG UUID               Z9MJ9w-UjoI-BO9X-rAb0-DPtF-pXqO-Haiji7
> 
> --- List of logical volumes ---
> 
> pvdata -- logical volume "/dev/files_vg/files_lv" at offset   0
> pvdata -- logical volume struct at offset   1 is empty
> [...]
> 
> 
> What did I do wrong (this time)?

Hmmm, if you pvcreated+vgcfgrestored every PV fine, nothing.

Could you send the output of "pvdata -UPP" for all your PVs of "files_vg"
and files_vg.conf to me in private email, please.


> 
>     / Rickard Olsson,IT-Konsult/
>    / Telefon: +46 70 635 01 42/
>   / http://www.webhackande.se/
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [linux-lvm] LVM1 and a dying disk (fixed)
  2003-09-03  7:46     ` Heinz J . Mauelshagen
@ 2003-09-16  6:54       ` Rickard Olsson
  0 siblings, 0 replies; 6+ messages in thread
From: Rickard Olsson @ 2003-09-16  6:54 UTC (permalink / raw)
  To: linux-lvm

# vgchange -ay
vgchange -- volume group "files_vg" successfully activated


I had a duplicate PV uuid for some reason.

The LVM2 vgcfgrestore didn't like the old LVM1 metadata format and 
LVM2's vgconvert said files_vg doesn't exist. Catch 22. Combining the -u 
switch in LVM2's pvcreate with a regular pvcreate and vgcfgrestore in 
LVM1 finally solved that.

Very, very scary. Now to see if reiserfsck's --rebuild_tree works. :-)

    / Rickard Olsson,IT-Konsult/
   / Telefon: +46 70 635 01 42/
  / http://www.webhackande.se/

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2003-09-16  6:54 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2003-09-03  3:55 [linux-lvm] LVM1 and a dying disk Rickard Olsson
2003-09-03  4:23 ` Heinz J . Mauelshagen
2003-09-03  4:40   ` Rickard Olsson
2003-09-03  5:53   ` Rickard Olsson
2003-09-03  7:46     ` Heinz J . Mauelshagen
2003-09-16  6:54       ` [linux-lvm] LVM1 and a dying disk (fixed) Rickard Olsson

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.