[linux-lvm] shift PV from disk to raid device?

[linux-lvm] shift PV from disk to raid device?

[Kai Schaetzl]

To make a long story short I accidentally put an LVM PV on top of /dev/sda 
instead of /dev/md2 (a Linux RAID 1 partition with the mirror missing) and 
noticed only after I was done with creating logical volumes and filling 
them with content. After adding the second RAID device and syncing I now 
get the duplicate PV uuid warning. The reason is obvious, I have the same 
PV on both drives. Apart from that warning "everything" works, but I would 
like to fix it.

If possible I would like to avoid wiping the complete PV, recreating all 
of it and copying all the content of the logical volumes back to it.
Is there a way to "shift" the PV from /dev/sda to /dev/md2 and leave 
everything else intact? Maybe after breaking the array?

Kai

-- 
Kai Sch�tzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

Re: [linux-lvm] shift PV from disk to raid device?

[Bryn M. Reeves]

Kai Schaetzl wrote:
> If possible I would like to avoid wiping the complete PV, recreating all 
> of it and copying all the content of the logical volumes back to it.
> Is there a way to "shift" the PV from /dev/sda to /dev/md2 and leave 
> everything else intact? Maybe after breaking the array?

man pvmove

Bryn.

Re: [linux-lvm] shift PV from disk to raid device?

[Kai Schaetzl]

Bryn M. Reeves wrote on Tue, 09 Dec 2008 10:30:54 +0000:

I notice that I wrote "/dev/sda" where I should have written "/dev/sda3", 
of course. But I guess this doesn't make a difference.

> pvmove

thank you. I remember this command from reading the LVM how-to some months 
ago, but had forgotten by now. I had checked the pvchange command, but not 
this one. After reading man pvmove I'm not sure if using that would really 
be helpful. If I understand correctly using it will still copy a lot of 
data (all the data on the LVs). Basically it seems to move out all data 
from dev/sdan to a temporary lv, then move them back in to /dev/mdn. From 
my non-expert view this might create a quite "unfavorable" data layout as 
it is all happening within the same "data location" on the disk. And all 
in all it might also take longer than a brute-force approach. I had hoped 
there's some way to just move lvm metadata so that it knows it belongs to 
/dev/mdn and not /dev/sdan anymore.
So, I wonder if it isn't as good or even better if I do the following:

- break the raid array md2 
   (made from /dev/sda3 and /dev/sdb3 with duplicate PVs)
- remove the PV on /dev/sda3
- recreate it and all LVs on /dev/md2
- copy over the data from the LVs on /dev/sdb3

?

Kai

-- 
Kai Sch�tzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

Re: [linux-lvm] shift PV from disk to raid device?

[Bryn M. Reeves]

Kai Schaetzl wrote:
> Bryn M. Reeves wrote on Tue, 09 Dec 2008 10:30:54 +0000:
> 
> I notice that I wrote "/dev/sda" where I should have written "/dev/sda3", 
> of course. But I guess this doesn't make a difference.
> 
>> pvmove
> 
> thank you. I remember this command from reading the LVM how-to some months 
> ago, but had forgotten by now. I had checked the pvchange command, but not 
> this one. After reading man pvmove I'm not sure if using that would really 
> be helpful. If I understand correctly using it will still copy a lot of 
> data (all the data on the LVs). Basically it seems to move out all data 
> from dev/sdan to a temporary lv, then move them back in to /dev/mdn. From 

No, that's not correct. The pvmove command works entirely at the level 
of PVs - no LVs involved and no temporary LVs are created. There is a 
temporary mirror created between the source PV region and destination PV 
region and this is then synced. Once the two regions are in sync, the 
mirror is removed.

This is then repeated for each segment to be moved that is allocated on 
the source PV. The operation is checkpointed after each segment, so if 
there is a crash or sudden power failure the operation will 
automatically re-start from the beginning of segment that was being 
moved at the time of the failure.

> my non-expert view this might create a quite "unfavorable" data layout as 
> it is all happening within the same "data location" on the disk. And all 

It will use the same allocation policies as any LVM command. If you 
don't like the default you can override it via the --alloc option or 
specify explicit lists of destination PV ranges - see the man page.

> in all it might also take longer than a brute-force approach. I had hoped 

Maybe, but it's a lot less likely to wreck your data. In practice, I've 
rarely had performance problems with pvmove.

> there's some way to just move lvm metadata so that it knows it belongs to 
> /dev/mdn and not /dev/sdan anymore.

No need. Once the pvmove has completed you can remove the (now unused) 
source PV using the vgreduce command.

> So, I wonder if it isn't as good or even better if I do the following:
> 
> - break the raid array md2 
>    (made from /dev/sda3 and /dev/sdb3 with duplicate PVs)
> - remove the PV on /dev/sda3
> - recreate it and all LVs on /dev/md2
> - copy over the data from the LVs on /dev/sdb3

Up to you, I have no experience doing it this way, but this is still 
going to copy the same volume of data around. I also don't really 
understand why you want to do all this if the data is already on md2.

Regards,
Bryn.

Re: [linux-lvm] shift PV from disk to raid device?

[Stuart D. Gathman]

On Tue, 9 Dec 2008, Bryn M. Reeves wrote:

> Kai Schaetzl wrote:
> > If possible I would like to avoid wiping the complete PV, recreating all of
> > it and copying all the content of the logical volumes back to it.
> > Is there a way to "shift" the PV from /dev/sda to /dev/md2 and leave
> > everything else intact? Maybe after breaking the array?
> 
> man pvmove

That is not very helpful.

His problem is that sda is one of the mirror of md2.  And he 
accidentally make sda the PV instead of md2.  So pvmove is only part of
solution. 

Here are the steps that I see:

1) remove sda from md2 array
2) use dd or pvremove to clear PV info in md2.  This should not
   affect sda since sda is removed from md2
3) pvcreate md2 as a new PV and add to volume group
4) pvmove sda -> md2.  The data should physically go on sdb, or whatever the
   other mirror of md2 is.
5) check that all LVs are gone from sda, and remove from volume group
6) pvremove sda
7) add sda to md2 array, and synchronize.

Did I miss anything?

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] shift PV from disk to raid device?

[Stuart D. Gathman]

On Tue, 9 Dec 2008, Stuart D. Gathman wrote:

> Here are the steps that I see:
> 
> 1) remove sda from md2 array
> 2) use dd or pvremove to clear PV info in md2.  This should not
>    affect sda since sda is removed from md2
> 3) pvcreate md2 as a new PV and add to volume group
> 4) pvmove sda -> md2.  The data should physically go on sdb, or whatever the
>    other mirror of md2 is.
> 5) check that all LVs are gone from sda, and remove from volume group
> 6) pvremove sda
> 7) add sda to md2 array, and synchronize.
> 
> Did I miss anything?

Oh yeah:

0.5) Backup everything just in case

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] shift PV from disk to raid device?

[Kai Schaetzl]

Bryn M. Reeves wrote on Tue, 09 Dec 2008 14:24:49 +0000:

> I also don't really 
> understand why you want to do all this if the data is already on md2.

Maybe I didn't make the problem clear in my first posting.
I created a single member md RAID1 device /dev/md2 on top of /dev/sda3 
(single member because the other disk /dev/sdb3 was holding the data). 
Then I created a PV on /dev/sda3 and a VG on it. I should have done this 
on /dev/md2, though. Small oversight on my part. (I had to recreate that 
array after a complete lowlevel format of the disk and I had the data all 
fine on /dev/sdb3. So I accidentally created the PV on the "counterpart" 
/dev/sda3 instead of the existing raid device.)
Then I moved all the LVs and data back on that PV, added /dev/sdb3 to the 
md device and let it sync. And now I have two identical PVs on both disks 
instead of one PV on the md device.
So, you think pvmove /dev/sda3 /dev/md2 would just do what I want?
Should I break the array and perform the move on the single member array, 
so that I can keep my data on /dev/sdb3 in case something goes wrong?

Kai

-- 
Kai Sch�tzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

Re: [linux-lvm] shift PV from disk to raid device?

[Bryn M. Reeves]

Stuart D. Gathman wrote:
> His problem is that sda is one of the mirror of md2.  And he 
> accidentally make sda the PV instead of md2.  So pvmove is only part of
> solution. 

I'd misread that. In that case, there should be no need to copy any 
data. Just re-create the PV on the correct device and restore the VG 
metadata, but this set up sounds a little strange to me - I'm not sure 
how you would get into that situation in the first place since you 
cannot (at least with up-to-date tools) run pvcreate on an MD member:

# mdadm -C /dev/md0 -l1 -n2 /dev/loop{0,1}
mdadm: array /dev/md0 started.
# pvcreate /dev/loop0
   Can't open /dev/loop0 exclusively.  Mounted filesystem?

Afaik, that check has been in there for quite some time. You also can't 
just stop the array and run it on a member or it'll trash the MD superblock:

# pvcreate /dev/loop0
   Wiping software RAID md superblock on /dev/loop0
   Physical volume "/dev/loop0" successfully created

Which will break the array. The only way I could make this happen was by 
creating an array, breaking one member of it (i.e. destroying the 
array), creating a PV on it and then re-creating the array, forcing it 
for the fact that one member already had a valid MD superblock:

# mdadm --run /dev/md0
mdadm: failed to run array /dev/md0: Invalid argument
# mdadm -C /dev/md0 -l1 -n2 /dev/loop{0,1}
mdadm: /dev/loop1 appears to be part of a raid array:
     level=1 devices=2 ctime=Tue Dec  9 11:03:16 2008
Continue creating array? y
mdadm: array /dev/md0 started.

That's quite a lot of forcing/overriding and the system no longer 
recognises the MD member (/dev/loop0 in this case) as an LVM2 PV:

# pvs /dev/loop0
   Failed to read physical volume "/dev/loop0"

I can only get the LVM2 tools to recognise this as a PV at this point by 
manually disabling MD component detection 
(devices/md_component_detection = 0), which has been turned on by 
default since 2.00.13.

At this point, when the array is running I can "see" 3 PVs, but the LVM 
tools chose the MD array and do not complain about duplicates:

# pvs -o+uuid
   PV         VG     Fmt  Attr PSize   PFree   PV UUID 

   /dev/md0          lvm2 --   128.00M 128.00M 
kAa32l-uWS3-8y5b-jXQB-e2tI-xaGy-8akTyc

With the array not running, I do get duplicate warnings for the 2nd side 
of the mirror:

# pvs -o+uuid
   Found duplicate PV kAa32luWS38y5bjXQBe2tIxaGy8akTyc: using /dev/loop1 
not /dev/loop0
   PV         VG     Fmt  Attr PSize   PFree   PV UUID 

   /dev/loop1        lvm2 --   128.00M 128.00M 
kAa32l-uWS3-8y5b-jXQB-e2tI-xaGy-8akTyc

The OP didn't mention the versions of the tools in use, so I'm not sure 
what to conclude here. It seems a little odd that he's been able to get 
into such a position.

> Here are the steps that I see:
> 
> 1) remove sda from md2 array
> 2) use dd or pvremove to clear PV info in md2.  This should not
>    affect sda since sda is removed from md2

But the LVM tools still think that md2 is part of the VG and the same PV 
UUID is going to be visible on both members of the array and the MD device..

> 3) pvcreate md2 as a new PV and add to volume group

In my testing, md2 would already be a PV at this point - if it really 
does contain the same data as sda3/sdb3 then it should show up as a PV 
already.

> 4) pvmove sda -> md2.  The data should physically go on sdb, or whatever the
>    other mirror of md2 is.

It seems a bit pointless moving data around if it's already there.

> 5) check that all LVs are gone from sda, and remove from volume group
> 6) pvremove sda
> 7) add sda to md2 array, and synchronize.
> Did I miss anything?

I'd have thought that you could just take a metadata backup 
(vgcfgbackup), wipe the labels from all devices and then run:

   pvcreate --uuid=$the_uuid --restorefile=/path/to/backup /dev/md2
   vgcfgrestore -f /path/to/backup $vgname

And avoid the need to shovel lots of data around.

This works for me, but I'm not completely convinced that I've re-created 
the same situation the original poster was seeing - I had to play a lot 
of tricks with the tools just to get into that state in the first place.

Regards,
Bryn.

Re: [linux-lvm] shift PV from disk to raid device?

[Kai Schaetzl]

Stuart D. Gathman wrote on Tue, 9 Dec 2008 11:54:08 -0500 (EST):

Hallo, thanks for the suggestions. I'll try to follow that logically and with 
possible commands, see below with comments/questions.

> 1) remove sda from md2 array

mdadm /dev/md2 --fail /dev/sda3 --remove /dev/sda3

> 2) use dd or pvremove to clear PV info in md2.  This should not
>    affect sda since sda is removed from md2

md2 shouldn't have PV info. do you mean sdb3? (this would destroy my data on 
sdb3 then), so:

vgchange dom0 -an (deactivate the LVs on sdb3)
pvremove /dev/sdb3 ?
After this point a pvdisplay should show /dev/sda3 as the active PV for dom0.
(At the moment I get: "using /dev/sdb3 not /dev/sda3")

> 3) pvcreate md2 as a new PV and add to volume group

pvcreate /dev/md2
I fear I will get "device busy", not? (just a feeling from recent experience 
with juggling with md and lvm that are on top of each other)
vgextend dom0 /dev/md2
(dom0 then consists of sda3 and md2)

> 4) pvmove sda -> md2.  The data should physically go on sdb, or whatever the
>    other mirror of md2 is.

ok

> 5) check that all LVs are gone from sda, and remove from volume group

vgreduce dom0 /dev/sda

> 6) pvremove sda

ok

> 7) add sda to md2 array, and synchronize.

ok

Did I get it so far correctly? What about (2) (3) where I'm not sure?



Kai

-- 
Kai Sch�tzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

Re: [linux-lvm] shift PV from disk to raid device?

[Bryn M. Reeves]

Kai Schaetzl wrote:
> Stuart D. Gathman wrote on Tue, 9 Dec 2008 11:54:08 -0500 (EST):
> 
> Hallo, thanks for the suggestions. I'll try to follow that logically and with 
> possible commands, see below with comments/questions.
> 
>> 1) remove sda from md2 array
> 
> mdadm /dev/md2 --fail /dev/sda3 --remove /dev/sda3
> 
>> 2) use dd or pvremove to clear PV info in md2.  This should not
>>    affect sda since sda is removed from md2
> 
> md2 shouldn't have PV info. do you mean sdb3? (this would destroy my data on 
> sdb3 then), so:

Are you sure? If the MD array is a straightforward RAID1 mirror of two 
SD devices then the PV data should also be visible on the MD device - 
see my other post.

> vgchange dom0 -an (deactivate the LVs on sdb3)
> pvremove /dev/sdb3 ?
> After this point a pvdisplay should show /dev/sda3 as the active PV for dom0.
> (At the moment I get: "using /dev/sdb3 not /dev/sda3")
> 
>> 3) pvcreate md2 as a new PV and add to volume group
> 
> pvcreate /dev/md2
> I fear I will get "device busy", not? (just a feeling from recent experience 
> with juggling with md and lvm that are on top of each other)
> vgextend dom0 /dev/md2
> (dom0 then consists of sda3 and md2)

If there are still active LVs (or if the LVM2 tools recognise md2 as a 
PV) then you will get an error here - either that the device could not 
be opened exclusively (LVs present) or that you can't initialise the new 
PV without '-ff' as it is already part of a volume group.

Regards,
Bryn.

Re: [linux-lvm] shift PV from disk to raid device?

[Kai Schaetzl]

Thanks for the long reply, Stuart. Still trying to digest it. As a quick 
info: I'm using CentoS 5.2 with mdadm-2.6.4-1.el5.

I created a partition /dev/sda3, created /dev/md2 with it and then 
pvcreated /dev/sda3 mistakenly. I think (!) I remember that if I tried to 
do it the other way around (create PV and LVs on /dev/sda3, then create an 
md device from sda3) it would fail with device busy until I deactivated 
the LVs. Then I could do that and it seemingly worked and didn't destroy 
the data. But there were some error messages when using pv/lvdisplay if I 
remember right. I just did this for testing what I can do with all that as 
I had the chance to do it :-) before recreating the array the correct way 
and filling it with my backup data on sdb3. Unfortunately, then I 
overlooked this tiny bit and created the PV on the wrong device ...

Kai

-- 
Kai Sch�tzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

Re: [linux-lvm] shift PV from disk to raid device?

[Kai Schaetzl]

Bryn M. Reeves wrote on Tue, 09 Dec 2008 18:06:24 +0000:

> Are you sure? If the MD array is a straightforward RAID1 mirror of two 
> SD devices then the PV data should also be visible on the MD device - 
> see my other post.

how could I confirm that? pvs? No, it gives me:

  Found duplicate PV cs0HTBNIjwhdDD2fd6wvHTzAN61x8aWD: using /dev/sdb3 not 
/dev/sda3
  PV         VG   Fmt  Attr PSize   PFree
  /dev/sdb3  dom0 lvm2 a-   456.34G 350.34G

Kai

-- 
Kai Sch�tzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

Re: [linux-lvm] shift PV from disk to raid device?

[Kai Schaetzl]

Bryn M. Reeves wrote on Tue, 09 Dec 2008 17:36:06 +0000:

> I'd have thought that you could just take a metadata backup 
> (vgcfgbackup), wipe the labels from all devices and then run:
> 
>    pvcreate --uuid=$the_uuid --restorefile=/path/to/backup /dev/md2
>    vgcfgrestore -f /path/to/backup $vgname

I had thought about something using vgcfgrestore, but wasn't sure after 
reading the man pages if it would really help. I thought the restore might 
still contain information that it was on /dev/sd.. before and thus just 
create the same situation again.

Possible sequence of commands:

- remove sdb3 from array
- check if PV sda3 and dom0 is now up and ok
- vgcfgbackup dom0 -f somefile
- wipe the labels on sda3 (how?)
- pvcreate --uuid=new uid? --restorefile=somefile /dev/md2
- vgcfgrestore -f somefile dom0

???

Kai

-- 
Kai Sch�tzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

Re: [linux-lvm] shift PV from disk to raid device?

[Stuart D. Gathman]

On Tue, 9 Dec 2008, Kai Schaetzl wrote:

> 
> how could I confirm that? pvs? No, it gives me:
> 
>   Found duplicate PV cs0HTBNIjwhdDD2fd6wvHTzAN61x8aWD: using /dev/sdb3 not 
> /dev/sda3
>   PV         VG   Fmt  Attr PSize   PFree
>   /dev/sdb3  dom0 lvm2 a-   456.34G 350.34G

Yes, there is a problem.  (Keep checking and double checking the procedure
before executing.)  You need to know which device, (sda3, sdb3, md2), LVM is
actually writing to.  

Here is what I think you did:

1) create md2 from sda3 and sdb3
2) pvcreate md2
3) add sda3 to VG - LVM complains about dup and uses sdb3 instead

If LVM is current actually using sdb3, then you need to remove *sdb3* from md2,
and pvmove the data to md2 (which will now have only sda3).

Let's try again:

0.5) backup
1) remove the partition LVM is writing to (sdb3) from md2 (otherwise all
   writes to md2 will scribble on both),
2) pvremove md2 (which hopefully does *not* contain the device LVM is
   putting your data on) to erase duplicate metadata (or use dd - maybe
   pvremove gets confused with dups)
3) pvcreate md2 and add to VG
4) pvmove data from sdb3 to md2
5) reduce sdb3 from VG
6) add sdb3 to md2 and sync

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] shift PV from disk to raid device?

[Kai Schaetzl]

Stuart D. Gathman wrote on Tue, 9 Dec 2008 13:38:46 -0500 (EST):

> 1) create md2 from sda3 and sdb3
> 2) pvcreate md2
> 3) add sda3 to VG - LVM complains about dup and uses sdb3 instead

A bit different. But thanks for this hint. I checked my bash history now. 
Short summary:

mdadm --create --verbose /dev/md3 --level=1 --raid-devices=2 /dev/sda3 missing
mdadm --detail /dev/md3
pvcreate /dev/md3
pvcreate /dev/md3 -ff
vgcreate dom1 /dev/md3 (to avoid name clash with dom0 on sdb3)
.. some lvcreate and other stuff
mdadm --detail /dev/md3
reboot
vgchange dom1 -an

vgrename dom1 dom0


vgchange dom0 -ay
mdadm /dev/md3 -a /dev/sdb3
reboot

So, I did it all alright. I was wrong with my assumption about using the wrong 
device for the PV. However, I only notice now that /dev/md3 is gone for good, 
mdadm cannot find it. I had not been paying attention and thought that md2 was 
the array in question. (Thinking md0=sda1/sdb1, md1=sda2/sdb2, md2=sda3/sdb3, but 
I used md0=sda1/sdb1, md2=sda2/sdb2, md3=sda3/sdb3). So I didn't notice it.

Going further back in the history, it seems I forgot to pvremove the earlier 
existing PV /dev/sda3 (which had the same structure and data) before creating the 
PV on /dev/md3. That's why I had to -ff /dev/md3 I assume. It seems that after 
adding sdb3 to the array and rebooting the array dissolved completely and the 
existing PVs on sda3 and sdb3 came back into existence. And I thought it was an 
effect of the mirror sync.

So, I think I will just start over by removing /dev/sda3 now completely,
recreating the raid array and then recreating the PV on it. I think I should then 
be able to pvmove the LVs from /dev/sdb3 to /dev/md3, right? Problem solved. This 
time.

However, as this is the second time that this or a similar array just vanishes I 
wonder if there's something else going on. e.g. some kind of unwanted interaction 
between LVM and mdraid partitions. The other two raid arrays on the disk are 
absolutely stable and don't use LVM.



Kai

-- 
Kai Sch�tzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.co

Re: [linux-lvm] shift PV from disk to raid device?

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 4288 bytes --]

On Tue, 2008-12-09 at 20:42 +0100, Kai Schaetzl wrote:
> Stuart D. Gathman wrote on Tue, 9 Dec 2008 13:38:46 -0500 (EST):
> 
> > 1) create md2 from sda3 and sdb3
> > 2) pvcreate md2
> > 3) add sda3 to VG - LVM complains about dup and uses sdb3 instead
> 
> A bit different. But thanks for this hint. I checked my bash history now. 
> Short summary:
> 
> mdadm --create --verbose /dev/md3 --level=1 --raid-devices=2 /dev/sda3 missing
> mdadm --detail /dev/md3
> pvcreate /dev/md3
> pvcreate /dev/md3 -ff
> vgcreate dom1 /dev/md3 (to avoid name clash with dom0 on sdb3)
> .. some lvcreate and other stuff
> mdadm --detail /dev/md3
> reboot
> vgchange dom1 -an
> 
> vgrename dom1 dom0
> 
> 
> vgchange dom0 -ay
> mdadm /dev/md3 -a /dev/sdb3
> reboot
> 
> So, I did it all alright. I was wrong with my assumption about using the wrong 
> device for the PV. However, I only notice now that /dev/md3 is gone for good, 
> mdadm cannot find it. I had not been paying attention and thought that md2 was 
> the array in question. (Thinking md0=sda1/sdb1, md1=sda2/sdb2, md2=sda3/sdb3, but 
> I used md0=sda1/sdb1, md2=sda2/sdb2, md3=sda3/sdb3). So I didn't notice it.
> 
> Going further back in the history, it seems I forgot to pvremove the earlier 
> existing PV /dev/sda3 (which had the same structure and data) before creating the 
> PV on /dev/md3. That's why I had to -ff /dev/md3 I assume. It seems that after 
> adding sdb3 to the array and rebooting the array dissolved completely and the 
> existing PVs on sda3 and sdb3 came back into existence. And I thought it was an 
> effect of the mirror sync.
> 
> So, I think I will just start over by removing /dev/sda3 now completely,
> recreating the raid array and then recreating the PV on it. I think I should then 
> be able to pvmove the LVs from /dev/sdb3 to /dev/md3, right? Problem solved. This 
> time.
> 
> However, as this is the second time that this or a similar array just vanishes I 
> wonder if there's something else going on. e.g. some kind of unwanted interaction 
> between LVM and mdraid partitions. The other two raid arrays on the disk are 
> absolutely stable and don't use LVM.

One thing that might be happening here (and I'm not sure since I'm not
an LVM expert) is that when the lvm stack finds the pv on /dev/sdb3
or /dev/sda3, it sees a full size partition (meaning it can access all
the way to the end of the device).  The md subsystem by default uses
version 0.90 superblocks, which are located in the last 64k of the
physical partition.  This means that when you create a pv on a raid
device, the raid device size information returned by the kernel is
somewhat shorter than the physical partition size in order to reserve
the space at the end for the superblock.  However, accessing the bare
device does not reserve that space and it's possible that filesystem
information is overwriting part of the raid superblock and causing it to
fail its checksum check, making the raid device disappear completely.

The solution to this problem is to create the raid device with a
superblock format of 1.1 or 1.2 (aka, -e 1.1 or -e 1.2).  This places
the raid superblock at the *beginning* of the partition instead of the
end, and offsets the data in the partition past the superblock.  By so
doing, if you attempt to read an lvm label (or a filesystem label) off
of the bare scsi device, the label won't be found because it will be in
the wrong place in the partition.  This ensures that you will never
accidentally access the device via the bare scsi drive, instead you will
only be able to access it by bringing the raid device up.

Note: superblock format 1.1 and 1.2 can not be used on bootable
partitions as grub doesn't know how to deal with the offset to the
filesystem, but otherwise can be used anywhere else.

It's also worth noting that adding an internal bitmap to the device,
while it slows down writes, will greatly reduce the time to resync the
device in case of an unexpected/unclean shutdown.

-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [linux-lvm] shift PV from disk to raid device?

[Kai Schaetzl]

Doug Ledford wrote on Tue, 09 Dec 2008 14:56:24 -0500:

> The solution to this problem is to create the raid device with a
> superblock format of 1.1 or 1.2 (aka, -e 1.1 or -e 1.2).

Is this an option to mdraid? mdadm --create --help doesn't show me this 
option. Might mdadm-2.6.4-1.el5 on 5.2 be too old? Or is this an option to 
fdisk/sfdisk?

Kai

-- 
Kai Sch�tzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

Re: [linux-lvm] shift PV from disk to raid device?

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 1060 bytes --]

On Tue, 2008-12-09 at 21:15 +0100, Kai Schaetzl wrote:
> Doug Ledford wrote on Tue, 09 Dec 2008 14:56:24 -0500:
> 
> > The solution to this problem is to create the raid device with a
> > superblock format of 1.1 or 1.2 (aka, -e 1.1 or -e 1.2).
> 
> Is this an option to mdraid? mdadm --create --help doesn't show me this 
> option. Might mdadm-2.6.4-1.el5 on 5.2 be too old? Or is this an option to 
> fdisk/sfdisk?

The various mdadm mode specific help sections (such as --create --help)
only show options specific to that mode.  There are also general options
available from --help-options.  The -e option applies in multiple modes,
so it should be present in the general --help-options section (but it
isn't, even in the latest mdadm version).  It is, however, present and
explained in the man page for mdadm.

-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [linux-lvm] shift PV from disk to raid device?

[Kai Schaetzl]

Doug Ledford wrote on Tue, 09 Dec 2008 14:56:24 -0500:

> superblock

another question: this is not the same as the filesystem superblock, 
right? e.g. "dumpe2fs -h /dev/mdn" shows me "Bad magic number in super-
block" on all raid arrays that have LVM on them. That's to be expected, 
right?

Kai

-- 
Kai Sch�tzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

Re: [linux-lvm] shift PV from disk to raid device?

[Kai Schaetzl]

Doug Ledford wrote on Tue, 09 Dec 2008 14:56:24 -0500:

> One thing that might be happening here (and I'm not sure since I'm not
> an LVM expert) is that when the lvm stack finds the pv on /dev/sdb3
> or /dev/sda3, it sees a full size partition (meaning it can access all
> the way to the end of the device).

But this couldn't normally happen if I just created LVM on top of the md 
device, correct? However, as I created a PV on /dev/sda3, did not remove 
it and then create the md array on it and then the PV on the md device 
there might be something going on ... especially in case the array broke.

> The solution to this problem is to create the raid device with a
> superblock format of 1.1 or 1.2 (aka, -e 1.1 or -e 1.2).

man mdadm gives the difference between 1.1 and 1.2 as 1.2 putting the RAID 
superblock 4k after the beginning. Is this another measure against 
accidental overwrites? Would you recommend 1.2 if I were to use a newer 
superblock version?

Kai

-- 
Kai Sch�tzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com

Re: [linux-lvm] shift PV from disk to raid device?

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 854 bytes --]

On Tue, 2008-12-09 at 21:38 +0100, Kai Schaetzl wrote:
> Doug Ledford wrote on Tue, 09 Dec 2008 14:56:24 -0500:
> 
> > superblock
> 
> another question: this is not the same as the filesystem superblock, 
> right? e.g. "dumpe2fs -h /dev/mdn" shows me "Bad magic number in super-
> block" on all raid arrays that have LVM on them. That's to be expected, 
> right?

Yes.  The md raid stack, the lvm stack, and every file system all have
their own unique superblocks.  You use mdadm to read md raid
superblocks, various lvm tools to read lvm superblocks, and filesystem
specific tools for filesystem superblocks.

-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [linux-lvm] shift PV from disk to raid device?

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 2381 bytes --]

On Tue, 2008-12-09 at 21:49 +0100, Kai Schaetzl wrote:
> Doug Ledford wrote on Tue, 09 Dec 2008 14:56:24 -0500:
> 
> > One thing that might be happening here (and I'm not sure since I'm not
> > an LVM expert) is that when the lvm stack finds the pv on /dev/sdb3
> > or /dev/sda3, it sees a full size partition (meaning it can access all
> > the way to the end of the device).
> 
> But this couldn't normally happen if I just created LVM on top of the md 
> device, correct? However, as I created a PV on /dev/sda3, did not remove 
> it and then create the md array on it and then the PV on the md device 
> there might be something going on ... especially in case the array broke.

Correct.  If you created the PV on sda3, then the lvm stack would think
it could write all the way to the end of the partition, including
overwriting the raid superblock you later created by making the raid
array.  If you create the pv on the raid device itself, then it should
be sized to exactly fit within the limits of the md raid device and
never overwrite the raid superblock.

> > The solution to this problem is to create the raid device with a
> > superblock format of 1.1 or 1.2 (aka, -e 1.1 or -e 1.2).
> 
> man mdadm gives the difference between 1.1 and 1.2 as 1.2 putting the RAID 
> superblock 4k after the beginning. Is this another measure against 
> accidental overwrites? Would you recommend 1.2 if I were to use a newer 
> superblock version?

The only reason for 1.2 to exist is in the case a future boot loader
(like extlinux) is modified to know about filesystem offsets and boot
from these arrays, then you generally will need a version 1.2 superblock
so that the first 512 bytes of the partition can be used by the boot
loader.  With a version 1.1 superblock, the boot loader sector and the
superblock would both want to occupy that first sector.  However, as we
don't ship extlinux yet, and as it doesn't yet support version 1.1 or
1.2 superblocks (although I was told by HPA, the author of extlinux,
that it would be very easy to do), a version 1.1 superblock is just as
good as a version 1.2 superblock.

-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [linux-lvm] shift PV from disk to raid device?

[Stuart D. Gathman]

On Tue, 9 Dec 2008, Kai Schaetzl wrote:

> > The solution to this problem is to create the raid device with a
> > superblock format of 1.1 or 1.2 (aka, -e 1.1 or -e 1.2).
> 
> Is this an option to mdraid? mdadm --create --help doesn't show me this 
> option. Might mdadm-2.6.4-1.el5 on 5.2 be too old? Or is this an option to 
> fdisk/sfdisk?

I don't think changing the raid superblock will help.  I generally
prefer having the superblock at the end - there are times when you
*do* want sda3/sdb3 to alias md3:

 o many versions of grub will not install on md? - you have to unmount /boot,
 break mirror, mount sd? on /boot, install grub, unmount /boot, mount md? on
 /boot again, add back mirror copy of /boot.

 o Also, when recovering a system, it is helpful to be able to mount a
 partition directly, without setting up the raid devices.  I've had customers
 remove a mirror copy and mail me the disk.  I can then mount the filesystems
 directly, while they keep operating.

The key is breaking the aliasing and copying the data to a proper md? PV.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] shift PV from disk to raid device?

[Stuart D. Gathman]

On Tue, 9 Dec 2008, Kai Schaetzl wrote:

> Doug Ledford wrote on Tue, 09 Dec 2008 14:56:24 -0500:
> 
> > superblock
> 
> another question: this is not the same as the filesystem superblock, 
> right? e.g. "dumpe2fs -h /dev/mdn" shows me "Bad magic number in super-
> block" on all raid arrays that have LVM on them. That's to be expected, 
> right?

Yes, because they are PVs, not filesystems.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] shift PV from disk to raid device?

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 2397 bytes --]

On Tue, 2008-12-09 at 16:03 -0500, Stuart D. Gathman wrote:
> On Tue, 9 Dec 2008, Kai Schaetzl wrote:
> 
> > > The solution to this problem is to create the raid device with a
> > > superblock format of 1.1 or 1.2 (aka, -e 1.1 or -e 1.2).
> > 
> > Is this an option to mdraid? mdadm --create --help doesn't show me this 
> > option. Might mdadm-2.6.4-1.el5 on 5.2 be too old? Or is this an option to 
> > fdisk/sfdisk?
> 
> I don't think changing the raid superblock will help.  I generally
> prefer having the superblock at the end - there are times when you
> *do* want sda3/sdb3 to alias md3:

Nope.  This is never true.

>  o many versions of grub will not install on md? - you have to unmount /boot,
>  break mirror, mount sd? on /boot, install grub, unmount /boot, mount md? on
>  /boot again, add back mirror copy of /boot.

Grub will install just fine, however you have to do it manually because
anaconda gets it wrong.  The correct way to install grub on a
raid1 /boot partition is to wait until anaconda gives you the "Done with
install, press the reboot button" prompt, then switch to vt2 where you
have a bash prompt, run grub, then run these commands (assuming that
your /boot raid1 is on partition 1 of disk sda and sdb):

device (hd0) /dev/sda
root (hd0,0)
install --stage2=/grub/stage2 /grub/stage1 (hd0) e2fs_stage1_5
p /grub/stage2 /grub/grub.conf
device (hd0) /dev/sdb
root (hd0,0)
install --stage2=/grub/stage2 /grub/stage1 (hd0) e2fs_stage1_5
p /grub/stage2 /grub/grub.conf
quit

>  o Also, when recovering a system, it is helpful to be able to mount a
>  partition directly, without setting up the raid devices.  I've had customers
>  remove a mirror copy and mail me the disk.  I can then mount the filesystems
>  directly, while they keep operating.

Again, this is never true.  For a raid1 array, you can always bring the
array up in degraded mode and access it that way.  Doing what you are
talking about can lead to silent data corruption if you ever bring the
raid drives back together and reassemble the array.

> The key is breaking the aliasing and copying the data to a proper md? PV.
> 
-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [linux-lvm] shift PV from disk to raid device?

[Kai Schaetzl]

Kai Schaetzl wrote on Tue, 09 Dec 2008 20:42:54 +0100:

> So, I think I will just start over by removing /dev/sda3 now completely,
> recreating the raid array and then recreating the PV on it. I think I should then 
> be able to pvmove the LVs from /dev/sdb3 to /dev/md3, right?

pvmove worked great. Here's what I used to get back to my wanted setup:

boot with only one disk (sda)
vgchange -an dom1
pvremove /dev/sda3
reboot
mdadm --create /dev/md3 --level=1 --raid-devices=2 /dev/sda3 missing
pvcreate /dev/md3
pvextend dom0 /dev/md3
pvmove /dev/sdb3 /dev/md3
pvdisplay (shows /dev/sdb3 as empty now)
pvremove /dev/sdb3
vgreduce dom0 /dev/sdb3
pvremove /dev/sdb3
mdadm /dev/md3 -a /dev/sdb3

Thanks for all the help and information to everyone!

Kai

-- 
Kai Sch�tzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com