[linux-lvm] lvm2 confused about double UUID

[linux-lvm] lvm2 confused about double UUID

[Christian Kujau]

The subject says LVM is confused, but maybe it's me who is confused:

I once had a PV made from a single /dev/sdb and all was well. This sdb 
however was really a 2 disk RAID1 which is now split up, the system now 
sees both disks, sdb and sdc. Of course, sdb and sdc still have the same 
UUID:

# blkid  | egrep 'sd[bc]'
/dev/sdb1: UUID="1O2Tkq-9Jy8-VfuZ-uzFh-1f6v-iVBD-hcK6Vq" TYPE="lvm2pv" 
/dev/sdc1: UUID="1O2Tkq-9Jy8-VfuZ-uzFh-1f6v-iVBD-hcK6Vq" TYPE="lvm2pv"

I decided to continue to use the VG that was set up on the PV, but since 
lvm comlained[0] about finding the same UUID on more than one device, I 
tried to instruct lvm to just use sdb for the VG:

# grep filter /etc/lvm/lvm.conf | grep -v \#
   filter = [ "r|^/dev/sdc1$|", "r|/dev/.*/by-path/.*|", \
              "r|/dev/.*/by-id/.*|", "a/.*/" ]

Now everything looks fine, no more lvm warnings, vg02 seems to use sdb1, 
exactly as I wanted:

# pvs
  PV         VG   Fmt  Attr PSize   PFree  
  /dev/sda2  vg01 lvm2 a-   111.67G   9.27G
  /dev/sdb1  vg02 lvm2 a-   931.46G 137.46G


But when I access vg02, I can clearly see I/O to/from sdc! I'd really like 
to use sdc for something else now that sdb is used for vg02, but I'm afraid to 
do something to sdc, as it still being used - although I don't know why.

Any ideas?

Christian.

# lvm version
  LVM version:     2.02.26 (2007-06-15)
  Library version: 1.02.20 (2007-06-15)
  Driver version:  4.12.0
# uname -r
2.6.24-24-xen                      <-- Ubuntu/8.04

[0] e.g.
   Found duplicate PV 1O2Tkq9Jy8VfuZuzFh1f6viVBDhcK6Vq: using 
   /dev/disk/by-path/pci-fw1.1-scsi-0:0:0:1-part1 not /dev/sdb1
-- 
BOFH excuse #282:

High altitude condensation from U.S.A.F prototype aircraft has contaminated the primary subnet mask. Turn off your computer for 9 days to avoid damaging it.

Re: [linux-lvm] lvm2 confused about double UUID

[Ron Johnson]

On 2009-07-26 19:21, Christian Kujau wrote:
> The subject says LVM is confused, but maybe it's me who is confused:
> 
> I once had a PV made from a single /dev/sdb and all was well. This sdb 
> however was really a 2 disk RAID1 which is now split up, the system now 
> sees both disks, sdb and sdc. Of course, sdb and sdc still have the same 
> UUID:

How did you split the mirrorset?


> # blkid  | egrep 'sd[bc]'
> /dev/sdb1: UUID="1O2Tkq-9Jy8-VfuZ-uzFh-1f6v-iVBD-hcK6Vq" TYPE="lvm2pv" 
> /dev/sdc1: UUID="1O2Tkq-9Jy8-VfuZ-uzFh-1f6v-iVBD-hcK6Vq" TYPE="lvm2pv"
> 
> I decided to continue to use the VG that was set up on the PV, but since 
> lvm comlained[0] about finding the same UUID on more than one device, I 
> tried to instruct lvm to just use sdb for the VG:

Did you reformat sdc before reusing it?  (That should give it a new 
UUID.)

> # grep filter /etc/lvm/lvm.conf | grep -v \#
>    filter = [ "r|^/dev/sdc1$|", "r|/dev/.*/by-path/.*|", \
>               "r|/dev/.*/by-id/.*|", "a/.*/" ]
> 
> Now everything looks fine, no more lvm warnings, vg02 seems to use sdb1, 
> exactly as I wanted:
> 
> # pvs
>   PV         VG   Fmt  Attr PSize   PFree  
>   /dev/sda2  vg01 lvm2 a-   111.67G   9.27G
>   /dev/sdb1  vg02 lvm2 a-   931.46G 137.46G
> 
> 
> But when I access vg02, I can clearly see I/O to/from sdc! I'd really like 
> to use sdc for something else now that sdb is used for vg02, but I'm afraid to 
> do something to sdc, as it still being used - although I don't know why.
> 
> Any ideas?
> 
> Christian.
> 
> # lvm version
>   LVM version:     2.02.26 (2007-06-15)
>   Library version: 1.02.20 (2007-06-15)
>   Driver version:  4.12.0
> # uname -r
> 2.6.24-24-xen                      <-- Ubuntu/8.04

Shouldn't you be asking the Ubuntu forum?

> 
> [0] e.g.
>    Found duplicate PV 1O2Tkq9Jy8VfuZuzFh1f6viVBDhcK6Vq: using 
>    /dev/disk/by-path/pci-fw1.1-scsi-0:0:0:1-part1 not /dev/sdb1


-- 
Scooty Puff, Sr
The Doom-Bringer

Re: [linux-lvm] lvm2 confused about double UUID

[Christian Kujau]

On Sun, 26 Jul 2009 at 19:50, Ron Johnson wrote:
> How did you split the mirrorset?

Sorry, I should've been more specific: the two disk mirror was no "LVM 
mirror", it is an external two-disk enclosure where one can toggle a 
little DIP switch to make it "RAID0" or "RAID1", etc. I split the 
mirrorset via this very switch, now the system sees both external disks 
instead of just one. Hence the disks itself haven't been touch during the 
split, that's why they both have the same UUID.

> Did you reformat sdc before reusing it?  (That should give it a new UUID.)

No. That's what I wanted to do (get a new UUID, use sdc for something 
else), but I'm afraid to do so since it's still somehow being used by LVM 
- and I wonder why pvs(8) says "vg02 uses sdb" but it clearly does not.

I've run "pvs"more verbosely and put the output here:
http://nerdbynature.de/bits/lvm/

>> # uname -r
>> 2.6.24-24-xen                      <-- Ubuntu/8.04
> 
> Shouldn't you be asking the Ubuntu forum?

Sure, if it turns out to be a (Ubuntu-)specific kernel/lvm2 issue, I'll do 
so.

Thanks,
Christian.
-- 
BOFH excuse #303:

fractal radiation jamming the backbone

Re: [linux-lvm] lvm2 confused about double UUID

[Ron Johnson]

On 2009-07-26 20:22, Christian Kujau wrote:
> On Sun, 26 Jul 2009 at 19:50, Ron Johnson wrote:
>> How did you split the mirrorset?
> 
> Sorry, I should've been more specific: the two disk mirror was no "LVM 
> mirror", it is an external two-disk enclosure where one can toggle a 
> little DIP switch to make it "RAID0" or "RAID1", etc. I split the 
> mirrorset via this very switch, now the system sees both external disks 
> instead of just one. Hence the disks itself haven't been touch during the 
> split, that's why they both have the same UUID.
> 
>> Did you reformat sdc before reusing it?  (That should give it a new UUID.)
> 
> No. That's what I wanted to do (get a new UUID, use sdc for something 
> else), but I'm afraid to do so since it's still somehow being used by LVM 
> - and I wonder why pvs(8) says "vg02 uses sdb" but it clearly does not.

So the issue seems really the deeper fact that you've got two 
physical devices with the same UUID.  All your lvm issues tumble out 
from there.

Is that a good summary?

-- 
Scooty Puff, Sr
The Doom-Bringer

Re: [linux-lvm] lvm2 confused about double UUID

[Christian Kujau]

On Sun, 26 Jul 2009 at 18:22, Christian Kujau wrote:
> Sure, if it turns out to be a (Ubuntu-)specific kernel/lvm2 issue, I'll do 
> so.

I've compiled the LVM2 tools from a current CVS checkout and I begin to 
think that the filter= parsing is possibly involved here. I changed the 
filter slightly to be more specific, and tried again:

  filter = [ "a|^/dev/sda.*$|", "a|^/dev/sdb.*$|", "r/.*/" ]

1) with LVM-2.02.26
  PV         VG   Fmt  Attr PSize   PFree   DevSize PV
  /dev/sda2  vg01 lvm2 a-   111.67G   6.56G 111.67G
  /dev/sdb1  vg02 lvm2 a-   931.46G 137.46G 931.46G

2) with LVM-2.02.50, using the same lvm.conf (according to strace)

  Found duplicate PV 1O2Tkq9Jy8VfuZuzFh1f6viVBDhcK6Vq: using /dev/sdc1 not /dev/sdb1
  PV         VG   Fmt  Attr PSize   PFree   DevSize PV
  /dev/sda2  vg01 lvm2 a-   111.67G   6.56G 111.67G
  /dev/sdc1  vg02 lvm2 a-   931.46G 181.46G 931.46G

So, can anybody comment if the filer above makes any sense at all?

Thanks in advance,
Christian.
-- 
BOFH excuse #281:

The co-locator cannot verify the frame-relay gateway to the ISDN server.

Re: [linux-lvm] lvm2 confused about double UUID

[Christian Kujau]

On Sun, 26 Jul 2009 at 21:55, Ron Johnson wrote:
> So the issue seems really the deeper fact that you've got two physical devices
> with the same UUID.  All your lvm issues tumble out from there.
> 
> Is that a good summary?

Yes, that's it. I know, having 2 physical volumes with the same UUID is 
not cool and is certainly not supported, but due to the split that's what 
I have here now and I assumed that I could convince LVM via the filter 
directive to use a certain device for vg02. I mean, vg02 is working fine, 
but I really want to format either sdb or sdc, but I'm afraid of 
destroying something in the LVM layout...

Let me ask another thing: when I assume for a moment that "pvs" is lying 
and really sdc is used for vg02, I'd go ahead and 1) pvcreate sdb, thus 
assigning a new UUID to sdb and 2) remove the filter from lvm.conf. LVM 
should not complain any more (since we have 2 different UUIDs now) and I/O 
still goes to sdc, as it is now apparently. If things go wrong, how can I 
restore the LVM header on sdb? (Would "pvcreate --uuid <old-UUID>" do the 
trick? Or restoring with dd? How many bytes are changed when using 
pvcreate?)

Christian.
-- 
BOFH excuse #281:

The co-locator cannot verify the frame-relay gateway to the ISDN server.

Re: [linux-lvm] lvm2 confused about double UUID

[Ron Johnson]

On 2009-07-26 22:30, Christian Kujau wrote:
> On Sun, 26 Jul 2009 at 21:55, Ron Johnson wrote:
>> So the issue seems really the deeper fact that you've got two physical devices
>> with the same UUID.  All your lvm issues tumble out from there.
>>
>> Is that a good summary?
> 
> Yes, that's it. I know, having 2 physical volumes with the same UUID is 
> not cool and is certainly not supported, but due to the split that's what 
> I have here now and I assumed that I could convince LVM via the filter 
> directive to use a certain device for vg02. I mean, vg02 is working fine, 
> but I really want to format either sdb or sdc, but I'm afraid of 
> destroying something in the LVM layout...
> 
> Let me ask another thing: when I assume for a moment that "pvs" is lying 
> and really sdc is used for vg02, I'd go ahead and 1) pvcreate sdb, thus 
> assigning a new UUID to sdb and 2) remove the filter from lvm.conf. LVM 
> should not complain any more (since we have 2 different UUIDs now) and I/O 
> still goes to sdc, as it is now apparently. If things go wrong, how can I 
> restore the LVM header on sdb? (Would "pvcreate --uuid <old-UUID>" do the 
> trick? Or restoring with dd? How many bytes are changed when using 
> pvcreate?)

Now you're getting beyond my pay grade.

But with uuidgen and tune2fs, you could give sdc1 a new UUID.  That 
would hose lvm, though, and you'd have to do surgery in /etc/lvm.

I'd go the safe/sure route and backup sdc2 if possible (have you 
written anything new to it since breaking the mirror?), and start 
over, dropping it from lvm. refdisking it (is there such a word?), 
etc, etc.

-- 
Scooty Puff, Sr
The Doom-Bringer

Re: [linux-lvm] lvm2 confused about double UUID

[Christian Kujau]

On Mon, 27 Jul 2009 at 01:19, Ron Johnson wrote:
> Now you're getting beyond my pay grade.

Do you accept virtual beer cans? :-)

> But with uuidgen and tune2fs, you could give sdc1 a new UUID.  That would hose
> lvm, though, and you'd have to do surgery in /etc/lvm.

Hm, tune2fs operates on (filesystem-)superblock, IIRC. I'm still tempted 
to do "pvcreate" on one of the discs, but I need to find out how much 
bytes I have to backup so I can restore if something goes wrong.

> I'd go the safe/sure route and backup sdc2 if possible

Well, I've backup up everything on vg02, but te restore would take a few 
days. Not that it's a production setup, but I still want to avoid this 
scenario.

> (have you written anything new to it since breaking the mirror?),

Yes.

> and start over, dropping it from lvm. refdisking it (is there such a 
> word?), etc, etc.

I'm trying to reproduce what happened on a test system, but I think I 
still fail to understand where LVM looks for "possible PVs to use". I 
tried something in a Xen DomU

# mdadm --create /dev/md0 [...] /dev/loop0 /dev/loop1
# pvcreate /dev/md0
# pvs
  PV         VG   Fmt  Attr PSize  PFree
  /dev/md0        lvm2 --   99.94M 99.94M
# blkid | egrep 'md|loop'
/dev/loop0: UUID="xyIqqj-gOCa-nMzi-Dn4e-swIL-7ZFG-fuwhBf" TYPE="LVM2_member"
/dev/loop1: UUID="xyIqqj-gOCa-nMzi-Dn4e-swIL-7ZFG-fuwhBf" TYPE="LVM2_member"
/dev/md0: UUID="xyIqqj-gOCa-nMzi-Dn4e-swIL-7ZFG-fuwhBf" TYPE="LVM2_member"

Of course, "pvs" still lists only /dev/md0 as an available PV, despite 
both loop0 and loop1 are available to the system - unlike sdb and sdc when 
my mirror was still intact.

I think it's only a naming issue: my mirrordevice was presented as "sdb"
to the system, sdc did not exist yet. Now with the mirror split up, both
sdb and sdc show up - and LVM is of course used to sdb. I don't know why
LVM bothers with sdc at all, though. In the test system I do:

# mdadm --stop /dev/md0
# pvs
[nothing]
# ln -s /dev/loop0 /dev/md0
# pvs
  PV         VG   Fmt  Attr PSize  PFree
  /dev/md0        lvm2 --   99.94M 99.94M 

:-)


...anyway, I try to backup the LVM header in the test box and see how it 
goes.

Thanks,
Christian.
-- 
BOFH excuse #188:

..disk or the processor is on fire.

Re: [linux-lvm] lvm2 confused about double UUID

[Alasdair G Kergon]

See vgimportclone.sh in 2.02.46 (should work fine with plenty of
earlier versions too).


NAME
       vgimportclone  -  import  and  rename  duplicated volume group (e.g. a
       hardware snapshot)

SYNOPSIS
       vgimportclone [-n|--basevgname VolumeGroupName]  [-i|--import]  Physi-
       calVolume [PhysicalVolume...]

DESCRIPTION
       vgimportclone  is  used to import a duplicated VG (e.g. hardware snap-
       shot).  Duplicate VG(s) and PV(s) are not able to be used  until  they
       are  made  to  coexist with the origin VG(s) and PV(s).  vgimportclone
       renames the VG associated with the specified  PV(s)  and  changes  the
       associated VG and PV UUIDs.

...

Alasdair

Re: [linux-lvm] lvm2 confused about double UUID

[malahal]

Christian Kujau [lists@nerdbynature.de] wrote:
> The subject says LVM is confused, but maybe it's me who is confused:
> 
> I once had a PV made from a single /dev/sdb and all was well. This sdb 
> however was really a 2 disk RAID1 which is now split up, the system now 
> sees both disks, sdb and sdc. Of course, sdb and sdc still have the same 
> UUID:
> 
> # blkid  | egrep 'sd[bc]'
> /dev/sdb1: UUID="1O2Tkq-9Jy8-VfuZ-uzFh-1f6v-iVBD-hcK6Vq" TYPE="lvm2pv" 
> /dev/sdc1: UUID="1O2Tkq-9Jy8-VfuZ-uzFh-1f6v-iVBD-hcK6Vq" TYPE="lvm2pv"
> 
> I decided to continue to use the VG that was set up on the PV, but since 
> lvm comlained[0] about finding the same UUID on more than one device, I 
> tried to instruct lvm to just use sdb for the VG:
> 
> # grep filter /etc/lvm/lvm.conf | grep -v \#
>    filter = [ "r|^/dev/sdc1$|", "r|/dev/.*/by-path/.*|", \
>               "r|/dev/.*/by-id/.*|", "a/.*/" ]
> 
> Now everything looks fine, no more lvm warnings, vg02 seems to use sdb1, 
> exactly as I wanted:
> 
> # pvs
>   PV         VG   Fmt  Attr PSize   PFree  
>   /dev/sda2  vg01 lvm2 a-   111.67G   9.27G
>   /dev/sdb1  vg02 lvm2 a-   931.46G 137.46G

I believe, "pvs" always rescans and just prints names it finds. It
doesn't really tell what is used. To really know what disk is really
being used, you need to get down to 'device mapper' level. Run 'dmsetup
table' and see what disks are actually used. You can post the output
here.

--Malahal.

Re: [linux-lvm] lvm2 confused about double UUID

[Christian Kujau]

On Mon, 27 Jul 2009 at 06:41, malahal@us.ibm.com wrote:
> I believe, "pvs" always rescans and just prints names it finds. It
> doesn't really tell what is used.

That's a pity then, I thought I could "trust" pvs to show what's really 
used, without going to the devices - which I did, and iostat disagreed 
with LVM. OTOH, this split mirror was confusing LVM, and me as well.

Anyway, I just went for it now: removed the filter= directives, 
ran pvcreate on sdb, now we have different UUID and all LVs survived - all 
but two, but these two were created after the split and must've been 
written to sdb. I think this was done before I installed the filter= in 
lvm.conf, because all the other LVs contain current data.

Lessons learned: deal with double UUIDs right after the mirror split, not 
a weeks later (which I did, because the backup took that long, but that's 
another story).

Thanks for all the comments on this,
Christian.
-- 
BOFH excuse #268:

Neutrino overload on the nameserver