* [linux-lvm] LVM1 and a dying disk
@ 2003-09-03 3:55 Rickard Olsson
2003-09-03 4:23 ` Heinz J . Mauelshagen
0 siblings, 1 reply; 6+ messages in thread
From: Rickard Olsson @ 2003-09-03 3:55 UTC (permalink / raw)
To: linux-lvm
One of the disks in my /dev/files_vg vg have been making weird noises
for a while. This vg is a JBOD with the following disks in it:
majestix root # pvscan
pvscan -- reading all physical volumes (this may take a while...)
pvscan -- ACTIVE PV "/dev/ide/host2/bus1/target0/lun0/disc" of VG "f2"
[149 GB / 0 free]
pvscan -- inactive PV "/dev/ide/host2/bus0/target0/lun0/disc" is in
EXPORTED VG
"files_vg" [149.03 GB / 0 free]
pvscan -- inactive PV "/dev/ide/host2/bus0/target1/lun0/disc" is in
EXPORTED VG
"files_vg" [111.77 GB / 0 free]
pvscan -- inactive PV "/dev/ide/host0/bus1/target1/lun0/disc" is in
EXPORTED VG
"files_vg" [93.14 GB / 0 free]
pvscan -- inactive PV "/dev/ide/host0/bus0/target0/lun0/disc" is in
EXPORTED VG
"files_vg" [111.77 GB / 0 free]
pvscan -- inactive PV "/dev/ide/host0/bus0/target1/lun0/disc" is in
EXPORTED VG
"files_vg" [93.14 GB / 0 free]
pvscan -- total: 6 [708 GB] / in use: 6 [708 GB] / in no VG: 0 [0]
Exported, you say? Yep. The machine that this vg was in suffered a fatal
root filesystem corruption last friday - not related to this disk (I
name it Black Friday, that same day my VW Passat dropped the rear
muffler and my upstream ISP had some major network issues that still
isn't resolved).
The f2 vg is the new vg I'm building to move data over to.
majestix root # vgscan
vgscan -- reading all physical volumes (this may take a while...)
vgscan -- found active volume group "f2"
vgscan -- ERROR "vg_read_with_pv_and_lv(): current PV" can't get data of
volume
group "files_vgPV_EXP" from physical volume(s)
vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created
vgscan -- WARNING: This program does not do a VGDA backup of your volume
groups
And devfs? Gentoo. Devfs is a major and total pain when trying to do a
vgimport, listing 6 of those long paths on the command line is no fun.
And, it doesn't work:
vgimport -- ERROR "pv_read(): PV identifier invalid" reading physical
volume "/d
ev/ide/host0/bus1/target0/lun0/disc"
I did manage to export the vg before the machine died permanently. The
backups I had of the root drive are older then the last major change to
the vg (addition of the Seagate 160GB that shows up as 149.03 GB above).
The troublesome disk is not showing in the pvscan, it's currently on
/dev/ide/host0/bus1/target0/lun0/disc and a pvdata for it reveals:
majestix root # pvdata /dev/ide/host0/bus1/target0/lun0/disc
--- Physical volume ---
PV Name /dev/ide/host0/bus1/target0/lun0/disc
VG Name ÇD?
PV Size 111.79 GB [234441648 secs] / NOT usable 16.19 MB
[LVM: 155 KB]
PV# 0
PV Status NOT available
Allocatable yes (but full)
Cur LV 9306113
PE Size (KByte) 16384
Total PE 7153
Free PE 0
Allocated PE 7153
PV UUID rDBb94-Kb8v-GMDa-HxIi-???
Segmentation fault
As you can see, not all fun and games.
Now, is there any way to re-initialize the pv data on the failing disk
long enough to get it back into the vg and extract the data? The data
left in the vg is not critically important (I started backing it up as
soon as I noticed the noises), but it would be really nice to get it back.
I have considered pvcreating it, but I'm not sure how I'd get it into
the vg afterwards - would a vgscan/vgchange suffice? Would a reiserfsck
--rebuild_tree then prune the fs of anything that was on this disk?
/ Rickard Olsson,IT-Konsult/
/ Telefon: +46 70 635 01 42/
/ http://www.webhackande.se/
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [linux-lvm] LVM1 and a dying disk
2003-09-03 3:55 [linux-lvm] LVM1 and a dying disk Rickard Olsson
@ 2003-09-03 4:23 ` Heinz J . Mauelshagen
2003-09-03 4:40 ` Rickard Olsson
2003-09-03 5:53 ` Rickard Olsson
0 siblings, 2 replies; 6+ messages in thread
From: Heinz J . Mauelshagen @ 2003-09-03 4:23 UTC (permalink / raw)
To: linux-lvm
Rickard,
you say that you don't have a recent LVM metadata backup reflecting
your VG with /dev/ide/host0/bus1/target0/lun0/disc in it ?
If you had, normal procedure would be to
# pvcreate -ff /dev/ide/host0/bus1/target0/lun0/disc
and to
# vgcfgrestore -n YourVGName /dev/ide/host0/bus1/target0/lun0/disc
# vgscan
# vgchange -ay
(Because you exported, you need to pvcreate+vgcfgrestore to all PVs)
If /dev/ide/host0/bus1/target0/lun0/disc took a serious overwrite
(and it looks like it eyeing the pvdata output below), there's almost no
chance to cook up the correct metadata by hand I'm afraid.
People, please take LVM metadata backups!
Regards,
Heinz -- The LVM Guy --
On Wed, Sep 03, 2003 at 10:53:12AM +0200, Rickard Olsson wrote:
> One of the disks in my /dev/files_vg vg have been making weird noises
> for a while. This vg is a JBOD with the following disks in it:
>
>
> majestix root # pvscan
> pvscan -- reading all physical volumes (this may take a while...)
> pvscan -- ACTIVE PV "/dev/ide/host2/bus1/target0/lun0/disc" of VG "f2"
> [149 GB / 0 free]
> pvscan -- inactive PV "/dev/ide/host2/bus0/target0/lun0/disc" is in
> EXPORTED VG
> "files_vg" [149.03 GB / 0 free]
> pvscan -- inactive PV "/dev/ide/host2/bus0/target1/lun0/disc" is in
> EXPORTED VG
> "files_vg" [111.77 GB / 0 free]
> pvscan -- inactive PV "/dev/ide/host0/bus1/target1/lun0/disc" is in
> EXPORTED VG
> "files_vg" [93.14 GB / 0 free]
> pvscan -- inactive PV "/dev/ide/host0/bus0/target0/lun0/disc" is in
> EXPORTED VG
> "files_vg" [111.77 GB / 0 free]
> pvscan -- inactive PV "/dev/ide/host0/bus0/target1/lun0/disc" is in
> EXPORTED VG
> "files_vg" [93.14 GB / 0 free]
> pvscan -- total: 6 [708 GB] / in use: 6 [708 GB] / in no VG: 0 [0]
>
>
> Exported, you say? Yep. The machine that this vg was in suffered a fatal
> root filesystem corruption last friday - not related to this disk (I
> name it Black Friday, that same day my VW Passat dropped the rear
> muffler and my upstream ISP had some major network issues that still
> isn't resolved).
> The f2 vg is the new vg I'm building to move data over to.
>
>
> majestix root # vgscan
> vgscan -- reading all physical volumes (this may take a while...)
> vgscan -- found active volume group "f2"
> vgscan -- ERROR "vg_read_with_pv_and_lv(): current PV" can't get data of
> volume
> group "files_vgPV_EXP" from physical volume(s)
> vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created
> vgscan -- WARNING: This program does not do a VGDA backup of your volume
> groups
>
>
> And devfs? Gentoo. Devfs is a major and total pain when trying to do a
> vgimport, listing 6 of those long paths on the command line is no fun.
> And, it doesn't work:
>
>
> vgimport -- ERROR "pv_read(): PV identifier invalid" reading physical
> volume "/d
> ev/ide/host0/bus1/target0/lun0/disc"
>
>
> I did manage to export the vg before the machine died permanently. The
> backups I had of the root drive are older then the last major change to
> the vg (addition of the Seagate 160GB that shows up as 149.03 GB above).
> The troublesome disk is not showing in the pvscan, it's currently on
> /dev/ide/host0/bus1/target0/lun0/disc and a pvdata for it reveals:
>
>
> majestix root # pvdata /dev/ide/host0/bus1/target0/lun0/disc
> --- Physical volume ---
> PV Name /dev/ide/host0/bus1/target0/lun0/disc
> VG Name ÇD?
> PV Size 111.79 GB [234441648 secs] / NOT usable 16.19 MB
> [LVM: 155 KB]
> PV# 0
> PV Status NOT available
> Allocatable yes (but full)
> Cur LV 9306113
> PE Size (KByte) 16384
> Total PE 7153
> Free PE 0
> Allocated PE 7153
> PV UUID rDBb94-Kb8v-GMDa-HxIi-???
>
> Segmentation fault
>
>
> As you can see, not all fun and games.
>
> Now, is there any way to re-initialize the pv data on the failing disk
> long enough to get it back into the vg and extract the data? The data
> left in the vg is not critically important (I started backing it up as
> soon as I noticed the noises), but it would be really nice to get it back.
>
> I have considered pvcreating it, but I'm not sure how I'd get it into
> the vg afterwards - would a vgscan/vgchange suffice? Would a reiserfsck
> --rebuild_tree then prune the fs of anything that was on this disk?
>
> / Rickard Olsson,IT-Konsult/
> / Telefon: +46 70 635 01 42/
> / http://www.webhackande.se/
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
*** Software bugs are stupid.
Nevertheless it needs not so stupid people to solve them ***
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Heinz Mauelshagen Sistina Software Inc.
Senior Consultant/Developer Am Sonnenhang 11
56242 Marienrachdorf
Germany
Mauelshagen@Sistina.com +49 2626 141200
FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [linux-lvm] LVM1 and a dying disk
2003-09-03 4:23 ` Heinz J . Mauelshagen
@ 2003-09-03 4:40 ` Rickard Olsson
2003-09-03 5:53 ` Rickard Olsson
1 sibling, 0 replies; 6+ messages in thread
From: Rickard Olsson @ 2003-09-03 4:40 UTC (permalink / raw)
To: linux-lvm
Heinz J . Mauelshagen wrote:
> you say that you don't have a recent LVM metadata backup reflecting
> your VG with /dev/ide/host0/bus1/target0/lun0/disc in it ?
Well, possibly. Many of the files could be rescued off the damaged root
file system, even if the directory structure was shot. What file(s)
would I be looking for? There's 18487 numbered directories in lost+found
with a large number of subdirectories - but if I have a name to go on, I
may be able to find something useful in there.
/ Rickard Olsson,IT-Konsult/
/ Telefon: +46 70 635 01 42/
/ http://www.webhackande.se/
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [linux-lvm] LVM1 and a dying disk
2003-09-03 4:23 ` Heinz J . Mauelshagen
2003-09-03 4:40 ` Rickard Olsson
@ 2003-09-03 5:53 ` Rickard Olsson
2003-09-03 7:46 ` Heinz J . Mauelshagen
1 sibling, 1 reply; 6+ messages in thread
From: Rickard Olsson @ 2003-09-03 5:53 UTC (permalink / raw)
To: linux-lvm
Heinz J . Mauelshagen wrote:
> you say that you don't have a recent LVM metadata backup reflecting
> your VG with /dev/ide/host0/bus1/target0/lun0/disc in it ?
I found a files_vg.conf and put it in /etc/lvmconf, did pvcreate and
successful vgcfgrestores on all of them. I had to move the disks around
a bit to make them appear in their old places since they're now in a new
machine (vgcfgrestore complained otherwise). The bad disk is now at
/dev/ide/host2/bus1/target1/lun0/disc
majestix root # pvscan
pvscan -- reading all physical volumes (this may take a while...)
pvscan -- ACTIVE PV "/dev/ide/host4/bus1/target0/lun0/disc" of VG "f2"
[
149 GB / 0 free]
pvscan -- inactive PV "/dev/ide/host2/bus1/target0/lun0/disc" is
associated to
unknown VG "files_vg" (run vgscan)
pvscan -- inactive PV "/dev/ide/host2/bus1/target1/lun0/disc" is
associated to
unknown VG "files_vg" (run vgscan)
pvscan -- inactive PV "/dev/ide/host2/bus0/target0/lun0/disc" is
associated to
unknown VG "files_vg" (run vgscan)
pvscan -- inactive PV "/dev/ide/host2/bus0/target1/lun0/disc" is
associated to
unknown VG "files_vg" (run vgscan)
pvscan -- inactive PV "/dev/ide/host0/bus1/target0/lun0/disc" is
associated to
unknown VG "files_vg" (run vgscan)
pvscan -- inactive PV "/dev/ide/host0/bus1/target1/lun0/disc" is
associated to
unknown VG "files_vg" (run vgscan)
pvscan -- total: 7 [819.79 GB] / in use: 7 [819.79 GB] / in no VG: 0 [0]
majestix root # vgscan
vgscan -- reading all physical volumes (this may take a while...)
vgscan -- found active volume group "f2"
vgscan -- ERROR "vg_read_with_pv_and_lv(): current PV" can't get data of
volume
group "files_vg" from physical volume(s)
vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created
vgscan -- WARNING: This program does not do a VGDA backup of your volume
groups
majestix root # pvdata /dev/ide/host2/bus1/target1/lun0/disc
--- Physical volume ---
PV Name /dev/ide/host2/bus1/target1/lun0/disc
VG Name files_vg
PV Size 93.16 GB [195371568 secs] / NOT usable 16.19 MB
[LVM: 151
KB]
PV# 4
PV Status NOT available
Allocatable yes (but full)
Cur LV 1
PE Size (KByte) 16384
Total PE 5961
Free PE 0
Allocated PE 5961
PV UUID KsET9a-qoKI-a2gr-z1GB-jwM1-vFw7-qHgxeZ
--- Volume group ---
VG Name
VG Access read/write
VG Status NOT available/resizable
VG # 0
MAX LV 256
Cur LV 1
Open LV 0
MAX LV Size 1023.97 GB
Max PV 256
Cur PV 6
Act PV 6
VG Size 670.61 GB
PE Size 16 MB
Total PE 42919
Alloc PE / Size 42919 / 670.61 GB
Free PE / Size 0 / 0
VG UUID Z9MJ9w-UjoI-BO9X-rAb0-DPtF-pXqO-Haiji7
--- List of logical volumes ---
pvdata -- logical volume "/dev/files_vg/files_lv" at offset 0
pvdata -- logical volume struct at offset 1 is empty
[...]
What did I do wrong (this time)?
/ Rickard Olsson,IT-Konsult/
/ Telefon: +46 70 635 01 42/
/ http://www.webhackande.se/
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [linux-lvm] LVM1 and a dying disk
2003-09-03 5:53 ` Rickard Olsson
@ 2003-09-03 7:46 ` Heinz J . Mauelshagen
2003-09-16 6:54 ` [linux-lvm] LVM1 and a dying disk (fixed) Rickard Olsson
0 siblings, 1 reply; 6+ messages in thread
From: Heinz J . Mauelshagen @ 2003-09-03 7:46 UTC (permalink / raw)
To: linux-lvm
On Wed, Sep 03, 2003 at 12:52:39PM +0200, Rickard Olsson wrote:
> Heinz J . Mauelshagen wrote:
>
> > you say that you don't have a recent LVM metadata backup reflecting
> > your VG with /dev/ide/host0/bus1/target0/lun0/disc in it ?
>
> I found a files_vg.conf and put it in /etc/lvmconf, did pvcreate and
> successful vgcfgrestores on all of them. I had to move the disks around
> a bit to make them appear in their old places since they're now in a new
> machine (vgcfgrestore complained otherwise).
"vgcfgrestore -o ..." does the trick without moving disks around
(which might be completely impossible with certain path changes anyway).
"-o" takes the old path, which has been stored in the metadata backup.
"vgcfgrestore -ll ..." long lists the VG metadata stored in the backup.
> The bad disk is now at
> /dev/ide/host2/bus1/target1/lun0/disc
>
>
> majestix root # pvscan
> pvscan -- reading all physical volumes (this may take a while...)
> pvscan -- ACTIVE PV "/dev/ide/host4/bus1/target0/lun0/disc" of VG "f2"
> [
> 149 GB / 0 free]
> pvscan -- inactive PV "/dev/ide/host2/bus1/target0/lun0/disc" is
> associated to
> unknown VG "files_vg" (run vgscan)
> pvscan -- inactive PV "/dev/ide/host2/bus1/target1/lun0/disc" is
> associated to
> unknown VG "files_vg" (run vgscan)
> pvscan -- inactive PV "/dev/ide/host2/bus0/target0/lun0/disc" is
> associated to
> unknown VG "files_vg" (run vgscan)
> pvscan -- inactive PV "/dev/ide/host2/bus0/target1/lun0/disc" is
> associated to
> unknown VG "files_vg" (run vgscan)
> pvscan -- inactive PV "/dev/ide/host0/bus1/target0/lun0/disc" is
> associated to
> unknown VG "files_vg" (run vgscan)
> pvscan -- inactive PV "/dev/ide/host0/bus1/target1/lun0/disc" is
> associated to
> unknown VG "files_vg" (run vgscan)
> pvscan -- total: 7 [819.79 GB] / in use: 7 [819.79 GB] / in no VG: 0 [0]
>
>
> majestix root # vgscan
> vgscan -- reading all physical volumes (this may take a while...)
> vgscan -- found active volume group "f2"
> vgscan -- ERROR "vg_read_with_pv_and_lv(): current PV" can't get data of
> volume
> group "files_vg" from physical volume(s)
> vgscan -- "/etc/lvmtab" and "/etc/lvmtab.d" successfully created
> vgscan -- WARNING: This program does not do a VGDA backup of your volume
> groups
>
>
> majestix root # pvdata /dev/ide/host2/bus1/target1/lun0/disc
> --- Physical volume ---
> PV Name /dev/ide/host2/bus1/target1/lun0/disc
> VG Name files_vg
> PV Size 93.16 GB [195371568 secs] / NOT usable 16.19 MB
> [LVM: 151
> KB]
> PV# 4
> PV Status NOT available
> Allocatable yes (but full)
> Cur LV 1
> PE Size (KByte) 16384
> Total PE 5961
> Free PE 0
> Allocated PE 5961
> PV UUID KsET9a-qoKI-a2gr-z1GB-jwM1-vFw7-qHgxeZ
>
> --- Volume group ---
> VG Name
> VG Access read/write
> VG Status NOT available/resizable
> VG # 0
> MAX LV 256
> Cur LV 1
> Open LV 0
> MAX LV Size 1023.97 GB
> Max PV 256
> Cur PV 6
> Act PV 6
> VG Size 670.61 GB
> PE Size 16 MB
> Total PE 42919
> Alloc PE / Size 42919 / 670.61 GB
> Free PE / Size 0 / 0
> VG UUID Z9MJ9w-UjoI-BO9X-rAb0-DPtF-pXqO-Haiji7
>
> --- List of logical volumes ---
>
> pvdata -- logical volume "/dev/files_vg/files_lv" at offset 0
> pvdata -- logical volume struct at offset 1 is empty
> [...]
>
>
> What did I do wrong (this time)?
Hmmm, if you pvcreated+vgcfgrestored every PV fine, nothing.
Could you send the output of "pvdata -UPP" for all your PVs of "files_vg"
and files_vg.conf to me in private email, please.
>
> / Rickard Olsson,IT-Konsult/
> / Telefon: +46 70 635 01 42/
> / http://www.webhackande.se/
>
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
--
Regards,
Heinz -- The LVM Guy --
*** Software bugs are stupid.
Nevertheless it needs not so stupid people to solve them ***
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Heinz Mauelshagen Sistina Software Inc.
Senior Consultant/Developer Am Sonnenhang 11
56242 Marienrachdorf
Germany
Mauelshagen@Sistina.com +49 2626 141200
FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [linux-lvm] LVM1 and a dying disk (fixed)
2003-09-03 7:46 ` Heinz J . Mauelshagen
@ 2003-09-16 6:54 ` Rickard Olsson
0 siblings, 0 replies; 6+ messages in thread
From: Rickard Olsson @ 2003-09-16 6:54 UTC (permalink / raw)
To: linux-lvm
# vgchange -ay
vgchange -- volume group "files_vg" successfully activated
I had a duplicate PV uuid for some reason.
The LVM2 vgcfgrestore didn't like the old LVM1 metadata format and
LVM2's vgconvert said files_vg doesn't exist. Catch 22. Combining the -u
switch in LVM2's pvcreate with a regular pvcreate and vgcfgrestore in
LVM1 finally solved that.
Very, very scary. Now to see if reiserfsck's --rebuild_tree works. :-)
/ Rickard Olsson,IT-Konsult/
/ Telefon: +46 70 635 01 42/
/ http://www.webhackande.se/
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2003-09-16 6:54 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2003-09-03 3:55 [linux-lvm] LVM1 and a dying disk Rickard Olsson
2003-09-03 4:23 ` Heinz J . Mauelshagen
2003-09-03 4:40 ` Rickard Olsson
2003-09-03 5:53 ` Rickard Olsson
2003-09-03 7:46 ` Heinz J . Mauelshagen
2003-09-16 6:54 ` [linux-lvm] LVM1 and a dying disk (fixed) Rickard Olsson
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.