[linux-lvm] Hosed up arrays and LVM

[linux-lvm] Hosed up arrays and LVM

[Bill Ranck]

Hello folks,
   I have a system where I am using md based multipath arrays with LVM2.
There was some maintenance on the IBM Shark SAN and when I rebooted this
system I got errors when my little startup script tried to do a vgscan.
The mdadm assemble step seems to have worked just fine, but here is a
part of the error messages from vgscan 

[root@strongbad init.d]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "local" using metadata type lvm2
  Couldn't find device with uuid 'CAg9Sn-5Jpw-V9bq-w0zl-Cv2I-tBjP-
qyw7QQ'.
  Couldn't find all physical volumes for volume group HPPR.
  Couldn't find device with uuid 'CAg9Sn-5Jpw-V9bq-w0zl-Cv2I-tBjP-
qyw7QQ'.
  Couldn't find all physical volumes for volume group HPPR.
  Couldn't find device with uuid 'CAg9Sn-5Jpw-V9bq-w0zl-Cv2I-tBjP-
qyw7QQ'.
  Couldn't find all physical volumes for volume group HPPR.
  Couldn't find device with uuid 'CAg9Sn-5Jpw-V9bq-w0zl-Cv2I-tBjP-
qyw7QQ'.
  Couldn't find all physical volumes for volume group HPPR.
  Couldn't find device with uuid 'CAg9Sn-5Jpw-V9bq-w0zl-Cv2I-tBjP-
qyw7QQ'.
  Couldn't find all physical volumes for volume group HPPR.
  Couldn't find device with uuid 'CAg9Sn-5Jpw-V9bq-w0zl-Cv2I-tBjP-
qyw7QQ'.
  Couldn't find all physical volumes for volume group HPPR.
  Couldn't find device with uuid 'CAg9Sn-5Jpw-V9bq-w0zl-Cv2I-tBjP-
qyw7QQ'.
  Couldn't find all physical volumes for volume group HPPR.
  Couldn't find device with uuid 'CAg9Sn-5Jpw-V9bq-w0zl-Cv2I-tBjP-
qyw7QQ'.
  Couldn't find all physical volumes for volume group HPPR.
  Volume group "HPPR" not found
  Found volume group "dbms" using metadata type lvm2

It continues on like that with another volume group and different uuids.
I have /dev/md0 through /dev/md11 defined with md and some of those seem
OK, but md1, md2, md3, md4, md8, md9, md11 are not seen by vgscan.  I
tried pvcreate on those missing arrays, which made them visible, but LVM
does not recognize them as part of the volume groups they belong to
anymore.  

Can anyone suggest a course of action here?

-- 
Bill Ranck
Blacksburg, Va.
540-231-3951

Re: [linux-lvm] Hosed up arrays and LVM

[kelsey hudson]

Bill Ranck wrote:
> It continues on like that with another volume group and different uuids.
> I have /dev/md0 through /dev/md11 defined with md and some of those seem
> OK, but md1, md2, md3, md4, md8, md9, md11 are not seen by vgscan.  I
> tried pvcreate on those missing arrays, which made them visible, but LVM
> does not recognize them as part of the volume groups they belong to
> anymore.  
> 
> Can anyone suggest a course of action here?

ouch, you ran pvcreate on a volume which already had data on it? From 
all I've read, you had better have a good backup laying around, 'cause 
you may not be able to get that data back pvcreate, IIRC, wipes out the 
metadata ...

If you have the old UUIDs for the physical volumes, you may be able to 
write those back to the device using pvcreate -u <old-uuid> on each such 
PV, then try vgscan again.

but, as from the man page:

    You can not recreate (reinitialize)  a  physical  volume  belonging
    to  an existing  volume  group.

so, you may be out of luck. But, good luck anyways ...

-kelsey

Re: [linux-lvm] Hosed up arrays and LVM

[Bill Ranck]

On Sun, 2005-11-20 at 09:38 -0800, kelsey hudson wrote:
> Bill Ranck wrote:
> > It continues on like that with another volume group and different uuids.
> > I have /dev/md0 through /dev/md11 defined with md and some of those seem
> > OK, but md1, md2, md3, md4, md8, md9, md11 are not seen by vgscan.  I
> > tried pvcreate on those missing arrays, which made them visible, but LVM
> > does not recognize them as part of the volume groups they belong to
> > anymore.  
> > 
> > Can anyone suggest a course of action here?
> 
> ouch, you ran pvcreate on a volume which already had data on it? From 
> all I've read, you had better have a good backup laying around, 'cause 
> you may not be able to get that data back pvcreate, IIRC, wipes out the 
> metadata ...
> 
> If you have the old UUIDs for the physical volumes, you may be able to 
> write those back to the device using pvcreate -u <old-uuid> on each such 
> PV, then try vgscan again.

Actually, this worked.  After a night's sleep and some coffee I was able
to see things more clearly.  The original source of the problem was
mdadm.conf not being up to date.  I had added a couple of LUNs to two md
multipath arrays a couple weeks ago specifically to have room for
snapshots.  I forgot to update the mdadm.conf file with the two new LUNs
and when the system rebooted on Sunday after some SAN maintenance the md
assemble step did not create those two arrays.  LVM could not see all
the PVs for the volume group and could not make them active.

I did a pvremove on each of the md arrays that I had pvcreated the day
before.  I then did a new pvcreate using info from the /etc/lvm/backup
files to assign the original uuid to each.  I fixed /etc/mdadm.conf,
assembled the two missing arrays/LUNs, and used pvcreate to assign them
the correct uuids as well.  Then vgcfgrestore restored the rest of the
metadata and I was able to activate the volume groups and mount the
volumes.  All the data survived.  I had to force (-ff) the pvcreate on
the two new LUNs, but since those are just there for snapshots no data
was lost.

I have added some logic to my startup/shutdown script to create a
current version of mdadm.conf file at startup and at shutdown.  It's
saved in / so it does not overwrite the existing one.

So, not truly an LVM problem, but thought I'd point out that since the
disks had never been mounted or assembled into a functioning volume
group, recreating them with the correct uuids and restoring the metadata
saved the day.

-- 
Bill Ranck
Blacksburg, Va.
540-231-3951