[linux-lvm] pvmove root filesystem?

[linux-lvm] pvmove root filesystem?

[Brian J. Murrell]

I have all of my filesystems on LVM, including my root filesystem.  I
also have a separate /boot partition right at the front of my (boot)
IDE disk which is not LVM.  So my boot disk layout currently looks
like this:

   Device Boot    Start       End    Blocks   Id  System
   /dev/hda1            63     19151      9544+  83  Linux
   /dev/hda2         19152  39102335  19541592   8e  Linux LVM

And I have a bunch of scsi disks in the box too which I am going to
commit to LVM as well, shuffling data around until I have _all_ disks
and filesystems under LVM.

I have done some preliminary bonnie++ benchmarks and it appears that
even my 7200 rpm scsi disk is better than my ide disk, so I want to
migrate my 10k rpm scsi disk into being my boot/root disk (my bios
will support booting from a scsi disk despite there being an ide disk
in the system).

My question is, if I put a small boot partition at the front of the
scsi disk I want to be my new boot disk, dedicate the rest of the disk
to LVM, add it to my "rootvol" LVM volume (which currently has root,
usr, home, var, etc. on the IDE disk) and then pvmove the logical
volumes to the scsi disk, can I safely pvmove the root filesystem
while the machine is running?  I think I would move the root
filesystem first in fact.

Once I have copied the /boot from the ide disk to the scsi disk,
and pvmoved the root filesystem (and possibly all of the rest of the
filesystems in the rootvol) to the SCSI disk, I should be able to do
the required "lilo" magic to get the MBR on the SCSI disk active and
then make it the boot drive and reboot no?

Is there anything about pvmoving the root filesystem that I am missing
in my description above that is going to cause disaster?

Thanx,
b.

-- 
Brian J. Murrell

Re: [linux-lvm] pvmove root filesystem?

[Andreas Dilger]

On Nov 01, 2001  23:31 -0500, Brian J. Murrell wrote:
> Once I have copied the /boot from the ide disk to the scsi disk,
> and pvmoved the root filesystem (and possibly all of the rest of the
> filesystems in the rootvol) to the SCSI disk, I should be able to do
> the required "lilo" magic to get the MBR on the SCSI disk active and
> then make it the boot drive and reboot no?
> 
> Is there anything about pvmoving the root filesystem that I am missing
> in my description above that is going to cause disaster?

Brian, the "pvmove" mechanism is not yet 100% safe for mounted filesystems.
If there is little/no activity on the filesystem while it is being moved,
you are probably OK.  There may be a problem with the ext3 journal, it is
hard to say (there was a report on the ext3 mailing list, but I never got
enough details to sort it out.

What I would suggest (for the good of all people reading) is you do this:
1) Make a backup of everything (obviously).
2) Give it a try.
3) Tell us how it worked.  Ensure you run a full (forced) e2fsck on
   everything, preferrably after a reboot (I think if you create the
   file /forcedfsck or so, check your rc.sysinit or equivalent) it
   will do this at boot time.

It is critical that you have a full backup, as I say people have had
problems, and there are known "holes" in the pvmove locking.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: [linux-lvm] pvmove root filesystem?

[Brian J. Murrell]

On Thu, Nov 01, 2001 at 10:15:29PM -0700, Andreas Dilger wrote:
> 
> Brian, the "pvmove" mechanism is not yet 100% safe for mounted filesystems.

Ugh.  I was hoping to not hear something like this.  Has it gotten any
better recently?  What I mean is my LVM identifies itself as:

LVM version LVM 0.9.1_beta7(ish)  by Heinz Mauelshagen  (20 June 2001)
lvm -- Module successfully initialized

which I believe is straight out of 2.4.8-ac12 (my kernel is a
2.4.8-ac12 base from Mandrake and I don't believe they do any LVM
patching -- they use what is in the AC kernels).

Am I going to run into even more problems with this verion of LVM vs.
the latest or has the "live" pvmove stuff not changed much between the
version I am running and current.

> If there is little/no activity on the filesystem while it is being moved,
> you are probably OK.

Not a good sign for root and /var, especially when /var has my mail
spool on it.

> There may be a problem with the ext3 journal, it is
> hard to say (there was a report on the ext3 mailing list, but I never got
> enough details to sort it out.

Hrm.  I am running ext3 on my filesystems as well.

> What I would suggest (for the good of all people reading) is you do this:
> 1) Make a backup of everything (obviously).

I always^Walmost always have a good backup -- daily incremental using
Amanda actually.

> 2) Give it a try.
> 3) Tell us how it worked.  Ensure you run a full (forced) e2fsck on
>    everything, preferrably after a reboot (I think if you create the
>    file /forcedfsck or so, check your rc.sysinit or equivalent) it
>    will do this at boot time.

I just don't want this to turn into a "restore my system from tape"
type of operation, especially if the system is operating (i.e.
processing mail, etc.) while it all goes down the sh*tter.  Maybe I
will contemplate it based on your thoughts of how much worse the
operation could be based on my version of LVM installed.

> It is critical that you have a full backup, as I say people have had
> problems, and there are known "holes" in the pvmove locking.

Of course.

Thanx Andreas,
b.

-- 
Brian J. Murrell

Re: [linux-lvm] pvmove root filesystem?

[Andreas Dilger]

On Nov 02, 2001  00:33 -0500, Brian J. Murrell wrote:
> > Brian, the "pvmove" mechanism is not yet 100% safe for mounted filesystems.
> 
> Ugh.  I was hoping to not hear something like this.  Has it gotten any
> better recently?  What I mean is my LVM identifies itself as:
> 
> LVM version LVM 0.9.1_beta7(ish)  by Heinz Mauelshagen  (20 June 2001)
> lvm -- Module successfully initialized

Well, there have been a lot of changes to the pvmove locking code and
such since then.  Whether it actually makes a difference is hard to
tell.

> Am I going to run into even more problems with this verion of LVM vs.
> the latest or has the "live" pvmove stuff not changed much between the
> version I am running and current.

Nobody has really done any good testing on it.

> > There may be a problem with the ext3 journal, it is
> > hard to say (there was a report on the ext3 mailing list, but I never got
> > enough details to sort it out.
> 
> Hrm.  I am running ext3 on my filesystems as well.

That's why I mentioned it.  You could read the thread in the ext3-users
archive at RedHat.

> I just don't want this to turn into a "restore my system from tape"
> type of operation, especially if the system is operating (i.e.
> processing mail, etc.) while it all goes down the sh*tter.  Maybe I
> will contemplate it based on your thoughts of how much worse the
> operation could be based on my version of LVM installed.

Well, if you are worried about mail, turn it off, back up your mail
spool, and then do the move.

Alternately, if you have some time to kill, do a "safe" test first.
Create a PV/VG/LV on your new disk, get some process writing into it
while it is mounted, and try a pvmove to another disk.  Bonus marks if
you use something that can be verified afterwards (e.g. a tar file).

"Obviously", there is only a danger at the "crossover" point of the
tar extraction and the pvmove (i.e. only while writing to the PE that
is being moved).  In theory, the buffers being written by tar will be
locked when they are being written into the PE that is being moved.
Once the pvmove is done, if e2fsck and "diff -ru" tell you that all
is well, I think you are safe.

If you are paranoid, run something really heavy like bonnie++ during the
pvmove (only the file creation/read/deletion part) and it will complain
if any of the files suddenly go missing or unreadable, but it doesn't
(currently) check the file contents.  That would probably be a good
hack - write synthetic data into each file that can easily be verified
as correct later.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: [linux-lvm] pvmove root filesystem?

[Joe Thornber]

On Thu, Nov 01, 2001 at 11:50:43PM -0700, Andreas Dilger wrote:
> On Nov 02, 2001  00:33 -0500, Brian J. Murrell wrote:
> > > Brian, the "pvmove" mechanism is not yet 100% safe for mounted filesystems.
> > 
> > Ugh.  I was hoping to not hear something like this.  Has it gotten any
> > better recently?  What I mean is my LVM identifies itself as:
> > 
> > LVM version LVM 0.9.1_beta7(ish)  by Heinz Mauelshagen  (20 June 2001)
> > lvm -- Module successfully initialized
> 
> Well, there have been a lot of changes to the pvmove locking code and
> such since then.  Whether it actually makes a difference is hard to
> tell.
> 
> > Am I going to run into even more problems with this verion of LVM vs.
> > the latest or has the "live" pvmove stuff not changed much between the
> > version I am running and current.
> 
> Nobody has really done any good testing on it.

I've done a bit of testing, and have never seen any problems due to
the holes in the locking; as such I am happy to use pvmove on *my*
development system (I don't use raw devices).  I certainly couldn't
recommend it for production systems.

As well as the locking problems there is a potential kernel deadlock,
which I can easily induce under memory pressure (too much deferred io
that cannot complete and no free memory to enable the userland copy to
finish).

- Joe

Re: [linux-lvm] pvmove root filesystem?

[Brent Harding]

What happens, if you just dd one device to another? Is that a good way to
move one file system to another?
When I set up lvm, I used the cpio option that the howto provided. Never
really used cpio before, but I just copied the line in the howto verbadem
and it moved the partition of /dev/hdb3 in to /dev/root/root for the volume
group, currently with only /dev/hdb2 and 4 in it, 3 still has my old setup
in it. Would it've worked just as good to have went
dd if=/dev/hdb3 of=/dev/root/root
to copy over that whole file system to the lvm device without mounting it?
Of course, I'd have pvcreated it, vgcreated, and lvcreated before I'd do
any of it.
At 11:22 AM 11/2/01 +0000, you wrote:
>On Thu, Nov 01, 2001 at 11:50:43PM -0700, Andreas Dilger wrote:
>> On Nov 02, 2001  00:33 -0500, Brian J. Murrell wrote:
>> > > Brian, the "pvmove" mechanism is not yet 100% safe for mounted
filesystems.
>> > 
>> > Ugh.  I was hoping to not hear something like this.  Has it gotten any
>> > better recently?  What I mean is my LVM identifies itself as:
>> > 
>> > LVM version LVM 0.9.1_beta7(ish)  by Heinz Mauelshagen  (20 June 2001)
>> > lvm -- Module successfully initialized
>> 
>> Well, there have been a lot of changes to the pvmove locking code and
>> such since then.  Whether it actually makes a difference is hard to
>> tell.
>> 
>> > Am I going to run into even more problems with this verion of LVM vs.
>> > the latest or has the "live" pvmove stuff not changed much between the
>> > version I am running and current.
>> 
>> Nobody has really done any good testing on it.
>
>I've done a bit of testing, and have never seen any problems due to
>the holes in the locking; as such I am happy to use pvmove on *my*
>development system (I don't use raw devices).  I certainly couldn't
>recommend it for production systems.
>
>As well as the locking problems there is a potential kernel deadlock,
>which I can easily induce under memory pressure (too much deferred io
>that cannot complete and no free memory to enable the userland copy to
>finish).
>
>- Joe
>
>_______________________________________________
>linux-lvm mailing list
>linux-lvm@sistina.com
>http://lists.sistina.com/mailman/listinfo/linux-lvm
>read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
>
>

Re: [linux-lvm] pvmove root filesystem?

[Brent Harding]

Is redhat 7.1 using ext3 already automatically? How does one upgrade from
ext2 to ext3?
At 11:50 PM 11/1/01 -0700, you wrote:
>On Nov 02, 2001  00:33 -0500, Brian J. Murrell wrote:
>> > Brian, the "pvmove" mechanism is not yet 100% safe for mounted
filesystems.
>> 
>> Ugh.  I was hoping to not hear something like this.  Has it gotten any
>> better recently?  What I mean is my LVM identifies itself as:
>> 
>> LVM version LVM 0.9.1_beta7(ish)  by Heinz Mauelshagen  (20 June 2001)
>> lvm -- Module successfully initialized
>
>Well, there have been a lot of changes to the pvmove locking code and
>such since then.  Whether it actually makes a difference is hard to
>tell.
>
>> Am I going to run into even more problems with this verion of LVM vs.
>> the latest or has the "live" pvmove stuff not changed much between the
>> version I am running and current.
>
>Nobody has really done any good testing on it.
>
>> > There may be a problem with the ext3 journal, it is
>> > hard to say (there was a report on the ext3 mailing list, but I never got
>> > enough details to sort it out.
>> 
>> Hrm.  I am running ext3 on my filesystems as well.
>
>That's why I mentioned it.  You could read the thread in the ext3-users
>archive at RedHat.
>
>> I just don't want this to turn into a "restore my system from tape"
>> type of operation, especially if the system is operating (i.e.
>> processing mail, etc.) while it all goes down the sh*tter.  Maybe I
>> will contemplate it based on your thoughts of how much worse the
>> operation could be based on my version of LVM installed.
>
>Well, if you are worried about mail, turn it off, back up your mail
>spool, and then do the move.
>
>Alternately, if you have some time to kill, do a "safe" test first.
>Create a PV/VG/LV on your new disk, get some process writing into it
>while it is mounted, and try a pvmove to another disk.  Bonus marks if
>you use something that can be verified afterwards (e.g. a tar file).
>
>"Obviously", there is only a danger at the "crossover" point of the
>tar extraction and the pvmove (i.e. only while writing to the PE that
>is being moved).  In theory, the buffers being written by tar will be
>locked when they are being written into the PE that is being moved.
>Once the pvmove is done, if e2fsck and "diff -ru" tell you that all
>is well, I think you are safe.
>
>If you are paranoid, run something really heavy like bonnie++ during the
>pvmove (only the file creation/read/deletion part) and it will complain
>if any of the files suddenly go missing or unreadable, but it doesn't
>(currently) check the file contents.  That would probably be a good
>hack - write synthetic data into each file that can easily be verified
>as correct later.
>
>Cheers, Andreas
>--
>Andreas Dilger
>http://sourceforge.net/projects/ext2resize/
>http://www-mddsp.enel.ucalgary.ca/People/adilger/
>
>
>_______________________________________________
>linux-lvm mailing list
>linux-lvm@sistina.com
>http://lists.sistina.com/mailman/listinfo/linux-lvm
>read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
>
>

Re: [linux-lvm] pvmove root filesystem?

[Andreas Dilger]

On Nov 02, 2001  10:44 -0600, Brent Harding wrote:
> Is redhat 7.1 using ext3 already automatically? How does one upgrade from
> ext2 to ext3?

It is an option, AFAIK.  To upgrade, you obviously need a kernel with ext3
support, and then run "tune2fs -h <dev>" to convert the devices.  Also
change /etc/fstab to use ext3 for the filesystem type.  You should have
the latest e2fsprogs installed.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: [linux-lvm] pvmove root filesystem?

[Brent Harding]

Wow, never was an option when I installed, do I do it to my lv that I
already have? I'm using initrd to get lv mounted as root, but still have my
old partition not in the volume group, able to be booted, if I change
/etc/fstab and take initrd out of lilo.
At 10:01 AM 11/2/01 -0700, you wrote:
>On Nov 02, 2001  10:44 -0600, Brent Harding wrote:
>> Is redhat 7.1 using ext3 already automatically? How does one upgrade from
>> ext2 to ext3?
>
>It is an option, AFAIK.  To upgrade, you obviously need a kernel with ext3
>support, and then run "tune2fs -h <dev>" to convert the devices.  Also
>change /etc/fstab to use ext3 for the filesystem type.  You should have
>the latest e2fsprogs installed.
>
>Cheers, Andreas
>--
>Andreas Dilger
>http://sourceforge.net/projects/ext2resize/
>http://www-mddsp.enel.ucalgary.ca/People/adilger/
>
>
>_______________________________________________
>linux-lvm mailing list
>linux-lvm@sistina.com
>http://lists.sistina.com/mailman/listinfo/linux-lvm
>read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html
>
>