[linux-lvm] pvmove errors

[linux-lvm] pvmove errors

[Carey Jung]

Hi,

I'm getting errors from pvmove, similar to a problem reported in December (
http://lists.sistina.com/pipermail/linux-lvm/2002-December/012973.html),
though I'm not sure if it's exactly the same.  Here's the relevant pvmove
output:

# pvmove -v -n /dev/backups2/rje /dev/sdd1 /dev/sda6
...
pvmove -- starting to move linear logical volume "/dev/backups2/rje"
pvmove -- checking for enough free physical extents in "backups2"
pvmove -- /dev/sdd1 [PE 0 [rje [LE 0]] -> /dev/sda6 [PE 2542] [1/6400]
/dev/backups2/group::/dev/backups2/rje: 0831 65920, 0806 166658560
pvmove -- ERROR "Invalid argument" copying extent from "/dev/sdd1"

pvmove -- ERROR "Invalid argument" remapping
pvmove -- ERROR "pv_move(): LE of LV remap" moving physical extents

I believe this is telling me that pvmove is having a problem copying PE 0 on
/dev/sdd1 to PE 2542 on /dev/sda6....  Yet lvdisplay indicates that LE 0 is
already on /dev/sda6, PE 02542 -- if I understand it correctly:

# lvdisplay -v /dev/backups2/rje |head -25
--- Logical volume ---
LV Name                /dev/backups2/rje
VG Name                backups2
LV Write Access        read/write
LV Status              available
LV #                   1
# open                 1
LV Size                200 GB
Current LE             6400
Allocated LE           6400
Allocation             next free
Read ahead sectors     1024
Block device           58:2

   --- Distribution of logical volume on 1 physical volume  ---
   PV Name                  PE on PV     reads      writes
   /dev/sdd1                6400         1849755    26252516

   --- logical volume i/o statistic ---
   1849755 reads  26252516 writes

   --- Logical extents ---
   LE    PV                        PE     reads      writes
   00000 /dev/sda6                 02542  904        527167
   00001 /dev/sda6                 02543  217        36574

pvdisplay confirms that PE 0 on /dev/sdd1 is free, and LE 0 is already on
/dev/sda6, PE 2542:

# pvdisplay -v /dev/sdd1|head -25
--- Physical volume ---
PV Name               /dev/sdd1
VG Name               backups2
PV Size               279.47 GB [586099332 secs] / NOT usable 32.19 MB [LVM:
162 KB]
PV#                   1
PV Status             available
Allocatable           yes
Cur LV                1
PE Size (KByte)       32768
Total PE              8942
Free PE               2568
Allocated PE          6374
PV UUID               joJY49-jRXo-3CYM-rL3Q-qy16-CA6g-0AX9tY

   --- Distribution of physical volume ---
   LV Name                   LE of LV  PE for LV
   /dev/backups2/rje         6400      6374

   --- Physical extents ---
   PE    LV                        LE      Disk sector
   00000 free
   .....
   00025 free
   00026 /dev/backups2/rje         00026   1769856
   00027 /dev/backups2/rje         00027   1835392

# pvdisplay -v /dev/sda6
...
   02542 /dev/backups2/rje         00000   166658560
   02543 /dev/backups2/rje         00001   166724096
   02544 /dev/backups2/rje         00002   166789632
   02545 /dev/backups2/rje         00003   166855168
   02546 /dev/backups2/rje         00004   166920704
...

Am I reading this correctly?  Any ideas what the problem is, or how to fix
it?  I've been moving several LVs from /dev/sdd1 to /dev/sda6; all completed
successfully.  This, the last one, is giving me an error.

Thanks in advance for any help....!

Carey Jung

Re: [linux-lvm] pvmove errors

[Heinz J . Mauelshagen]

Carey,

which LVM/kernel versions are you using ?
The error tells that the change for the mapping in the LVM driver fails.

The mapping on disk (lvdisplay -Dv ...) doesn't get changed in that case.
Must be a flaw in the ioctl error return handling, because it obviously
got changed in the kernel as "lvdisplay -v " shows.

Regards,
Heinz    -- The LVM Guy --


On Wed, Apr 23, 2003 at 07:48:59PM -0500, Carey Jung wrote:
> Hi,
> 
> I'm getting errors from pvmove, similar to a problem reported in December (
> http://lists.sistina.com/pipermail/linux-lvm/2002-December/012973.html),
> though I'm not sure if it's exactly the same.  Here's the relevant pvmove
> output:
> 
> # pvmove -v -n /dev/backups2/rje /dev/sdd1 /dev/sda6
> ...
> pvmove -- starting to move linear logical volume "/dev/backups2/rje"
> pvmove -- checking for enough free physical extents in "backups2"
> pvmove -- /dev/sdd1 [PE 0 [rje [LE 0]] -> /dev/sda6 [PE 2542] [1/6400]
> /dev/backups2/group::/dev/backups2/rje: 0831 65920, 0806 166658560
> pvmove -- ERROR "Invalid argument" copying extent from "/dev/sdd1"
> 
> pvmove -- ERROR "Invalid argument" remapping
> pvmove -- ERROR "pv_move(): LE of LV remap" moving physical extents
> 
> I believe this is telling me that pvmove is having a problem copying PE 0 on
> /dev/sdd1 to PE 2542 on /dev/sda6....  Yet lvdisplay indicates that LE 0 is
> already on /dev/sda6, PE 02542 -- if I understand it correctly:


> 
> # lvdisplay -v /dev/backups2/rje |head -25
> --- Logical volume ---
> LV Name                /dev/backups2/rje
> VG Name                backups2
> LV Write Access        read/write
> LV Status              available
> LV #                   1
> # open                 1
> LV Size                200 GB
> Current LE             6400
> Allocated LE           6400
> Allocation             next free
> Read ahead sectors     1024
> Block device           58:2
> 
>    --- Distribution of logical volume on 1 physical volume  ---
>    PV Name                  PE on PV     reads      writes
>    /dev/sdd1                6400         1849755    26252516
> 
>    --- logical volume i/o statistic ---
>    1849755 reads  26252516 writes
> 
>    --- Logical extents ---
>    LE    PV                        PE     reads      writes
>    00000 /dev/sda6                 02542  904        527167
>    00001 /dev/sda6                 02543  217        36574
> 
> pvdisplay confirms that PE 0 on /dev/sdd1 is free, and LE 0 is already on
> /dev/sda6, PE 2542:
> 
> # pvdisplay -v /dev/sdd1|head -25
> --- Physical volume ---
> PV Name               /dev/sdd1
> VG Name               backups2
> PV Size               279.47 GB [586099332 secs] / NOT usable 32.19 MB [LVM:
> 162 KB]
> PV#                   1
> PV Status             available
> Allocatable           yes
> Cur LV                1
> PE Size (KByte)       32768
> Total PE              8942
> Free PE               2568
> Allocated PE          6374
> PV UUID               joJY49-jRXo-3CYM-rL3Q-qy16-CA6g-0AX9tY
> 
>    --- Distribution of physical volume ---
>    LV Name                   LE of LV  PE for LV
>    /dev/backups2/rje         6400      6374
> 
>    --- Physical extents ---
>    PE    LV                        LE      Disk sector
>    00000 free
>    .....
>    00025 free
>    00026 /dev/backups2/rje         00026   1769856
>    00027 /dev/backups2/rje         00027   1835392
> 
> # pvdisplay -v /dev/sda6
> ...
>    02542 /dev/backups2/rje         00000   166658560
>    02543 /dev/backups2/rje         00001   166724096
>    02544 /dev/backups2/rje         00002   166789632
>    02545 /dev/backups2/rje         00003   166855168
>    02546 /dev/backups2/rje         00004   166920704
> ...
> 
> Am I reading this correctly?  Any ideas what the problem is, or how to fix
> it?  I've been moving several LVs from /dev/sdd1 to /dev/sda6; all completed
> successfully.  This, the last one, is giving me an error.
> 
> Thanks in advance for any help....!
> 
> Carey Jung
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

RE: [linux-lvm] pvmove errors

[Carey Jung]

>
> Carey,
>
> which LVM/kernel versions are you using ?
> The error tells that the change for the mapping in the LVM driver fails.

The latest RH rpm and smp kernel:

# rpm -qa|grep lvm
lvm-1.0.3-4
# cat /proc/version
Linux version 2.4.18-27.7.xsmp (bhcompile@stripples.devel.redhat.com) (gcc
version 2.96 20000731 (Red Hat Linux 7.3 2.96-112)) #1 SMP Fri Mar 14
05:52:30 EST 2003

>
> The mapping on disk (lvdisplay -Dv ...) doesn't get changed in that case.
> Must be a flaw in the ioctl error return handling, because it obviously
> got changed in the kernel as "lvdisplay -v " shows.

That's Greek to me, but something was seriously messed up with the PE
allocation or something, because I ended up trashing the LV after I sent
this message.  I could read the entire partition just fine, so I thought I'd
copy it to another LV and then delete the original LV.  So I created another
LV -- in the same VG as the original -- and started a big rsync copy from
source to destination LV.  Halfway through it, rsync started complaining
that it couldn't find files that were on the source filesystem when it
started.  When I snooped in the original LV, a bunch of directories had
disappeared!  Bear in mind that I was just reading from this filesystem.

My guess is that the PE allocation/mapping to LVs was messed up, so that
some PEs allocated to the new LV were actually mapped to the original LV, so
as I started writing to the new one, the system was overwriting files, etc
in the old LV.  Several directories seemed to have been effectively 'mv'-ed
to the new LV.

I then created yet another LV in a separate VG, rsync'ed both of the 1st two
LVs to it, recovered what was missing from a backup, and then lvremove'd the
first two.  I'm also in the process of copying all the other LVs in that VG
to another VG, just because I'm not sure I can trust its integrity.

By the way, is their some kind of tool one can run to do something analogous
to 'fsck' for VGs?  Something that will walk through LVs and PEs and make
sure the VG is internally consistent?

Thanks,
Carey

Re: [linux-lvm] pvmove errors

[Heinz J . Mauelshagen]

On Fri, Apr 25, 2003 at 08:14:50AM -0500, Carey Jung wrote:
> >
> > Carey,
> >
> > which LVM/kernel versions are you using ?
> > The error tells that the change for the mapping in the LVM driver fails.
> 
> The latest RH rpm and smp kernel:
> 
> # rpm -qa|grep lvm
> lvm-1.0.3-4
> # cat /proc/version
> Linux version 2.4.18-27.7.xsmp (bhcompile@stripples.devel.redhat.com) (gcc
> version 2.96 20000731 (Red Hat Linux 7.3 2.96-112)) #1 SMP Fri Mar 14
> 05:52:30 EST 2003
> 
> >
> > The mapping on disk (lvdisplay -Dv ...) doesn't get changed in that case.
> > Must be a flaw in the ioctl error return handling, because it obviously
> > got changed in the kernel as "lvdisplay -v " shows.
> 
> That's Greek to me,

:)

The kernel hold the mapping table used at runtime and there's one on the disks
to set it up. The one on disk is unlikely to have changed with the error
you reported.

> but something was seriously messed up with the PE
> allocation or something, because I ended up trashing the LV after I sent
> this message.  I could read the entire partition just fine, so I thought I'd
> copy it to another LV and then delete the original LV.  So I created another
> LV -- in the same VG as the original -- and started a big rsync copy from
> source to destination LV.  Halfway through it, rsync started complaining
> that it couldn't find files that were on the source filesystem when it
> started.  When I snooped in the original LV, a bunch of directories had
> disappeared!  Bear in mind that I was just reading from this filesystem.

Well, _if_ the mapping was bogus that will likely cause overwrites.
No way to redo from start then :(

> 
> My guess is that the PE allocation/mapping to LVs was messed up, so that
> some PEs allocated to the new LV were actually mapped to the original LV, so
> as I started writing to the new one, the system was overwriting files, etc
> in the old LV.  Several directories seemed to have been effectively 'mv'-ed
> to the new LV.
> 
> I then created yet another LV in a separate VG, rsync'ed both of the 1st two
> LVs to it, recovered what was missing from a backup, and then lvremove'd the
> first two.  I'm also in the process of copying all the other LVs in that VG
> to another VG, just because I'm not sure I can trust its integrity.
> 
> By the way, is their some kind of tool one can run to do something analogous
> to 'fsck' for VGs?  Something that will walk through LVs and PEs and make
> sure the VG is internally consistent?

vgck(8)

> 
> Thanks,
> Carey
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

RE: [linux-lvm] pvmove errors

[Carey Jung]

> 
> Well, _if_ the mapping was bogus that will likely cause overwrites.
> No way to redo from start then :(
>

Granted.  I think it may have killed (-9) pvmove when it was running.

How should I have proceeded when I saw the problem?

Can vgck find and fix these kinds of errors?

Thanks much,
Carey

Re: [linux-lvm] pvmove errors

[Heinz J . Mauelshagen]

On Fri, Apr 25, 2003 at 10:46:08AM -0500, Carey Jung wrote:
> > 
> > Well, _if_ the mapping was bogus that will likely cause overwrites.
> > No way to redo from start then :(
> >
> 
> Granted.  I think it may have killed (-9) pvmove when it was running.
> 
> How should I have proceeded when I saw the problem?
> 
> Can vgck find and fix these kinds of errors?

vgck is a read only checker. You need to run vgcfgrestore to restore
valid metadata to your PVs. Because you weren't able to change the VG
configuration after the pvmove problem, you should get your most recent 
configuration before the move back (and therefore your data :)

Please upgrade to 1.0.7 (and think about using LVM2.0 which we'll release soon)

> 
> Thanks much,
> Carey
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

[linux-lvm] pvmove errors

[Tom Lanyon]

Hi List,

I need to move all the data off of my external scsi array and remove it
from my volume group.
The external array is /dev/cciss/c1d0p0, so I issued a pvmove with the
following results:

>/sbin/sh-2.05b# pvmove --version
>pvmove: Logical Volume Manager 1.0.8-2
>Heinz Mauelshagen, Red Hat  26/05/2004 (IOP 10)
>
>/sbin/sh-2.05b# pvscan
>pvscan -- reading all physical volumes (this may take a while...)
>pvscan -- ACTIVE    PV  "/dev/cciss/c1d0p1" of VG "Volume0" [99.99 GB / 99.99 GB Free]
>pvscan -- ACTIVE    PV  "/dev/cciss/c0d0p2" of VG "Volume0" [136.61 GB / 68.89 GB Free]
>pvscan -- total: 2 [236.62 GB] / in use: 2 [236.62 GB] / in  no VG: 0 [0]
>
>/sbin/sh-2.05b# pvmove /dev/cciss/c1d0p1
>pvmove -- moving physical extents in active volume group "Volume0"
>pvmove -- WARNING: if you lose power during the move you may need to restore your LVM metadata from backup!
>pvmove -- do you want to continue? [y/n] y
>/dev/Volume0/group::/dev/Volume0/root: 6901 8704, 6802 8832
>pvmove -- ERROR "Invalid argument" copying extent from "/dev/cciss/c1d0p1"
>pvmove -- ERROR "Invalid argument" remapping
>pvmove -- ERROR "pv_move(): LE of LV remap" moving physical extents
>
I've tried a --force but that didn't help.

Any ideas? Need to get this drive out of the VG asap.

Regards,
Tom

-- 
Tom Lanyon
Systems Administrator
NetSpot Pty Ltd
183 Melbourne Street, North Adelaide, 5006
Ph: +618 8361 6800   Fax: +618 8361 6811
Email: tom@netspot.com.au

Re: [linux-lvm] pvmove errors

[Zac Slade]

On Tuesday 31 January 2006 01:17, Tom Lanyon wrote:
> I need to move all the data off of my external scsi array and remove it
> from my volume group.
> The external array is /dev/cciss/c1d0p0, so I issued a pvmove with the
Sounds straightforward.

> following results:
> >/sbin/sh-2.05b# pvscan
> >pvscan -- reading all physical volumes (this may take a while...)
> >pvscan -- ACTIVE    PV  "/dev/cciss/c1d0p1" of VG "Volume0" [99.99 GB /
> > 99.99 GB Free] pvscan -- ACTIVE    PV  "/dev/cciss/c0d0p2" of VG
> > "Volume0" [136.61 GB / 68.89 GB Free] pvscan -- total: 2 [236.62 GB] / in
> > use: 2 [236.62 GB] / in  no VG: 0 [0]
I don't see /dev/cciss/c1d0p0 in the pvscan output.  It does show 
that /dev/cciss/c1d0p1 is unused.  Also /dev/cciss/c0d0p2 is much larger than 
c1d0p1.
> >/sbin/sh-2.05b# pvmove /dev/cciss/c1d0p1
Is this what you mean?  pvmove will remove all extents used on the specified 
PV off of it.
> I've tried a --force but that didn't help.
I'm unclear about what device you are trying to reduce.......  A little more 
clarity please.

--
Zac Slade

Re: [linux-lvm] pvmove errors

[Tom Lanyon]

Zac Slade wrote:

>On Tuesday 31 January 2006 01:17, Tom Lanyon wrote:
>  
>
>>I need to move all the data off of my external scsi array and remove it
>>from my volume group.
>>The external array is /dev/cciss/c1d0p0, so I issued a pvmove with the
>>    
>>
>Sounds straightforward.
>
>  
>
>>following results:
>>    
>>
>>>/sbin/sh-2.05b# pvscan
>>>pvscan -- reading all physical volumes (this may take a while...)
>>>pvscan -- ACTIVE    PV  "/dev/cciss/c1d0p1" of VG "Volume0" [99.99 GB /
>>>99.99 GB Free] pvscan -- ACTIVE    PV  "/dev/cciss/c0d0p2" of VG
>>>"Volume0" [136.61 GB / 68.89 GB Free] pvscan -- total: 2 [236.62 GB] / in
>>>use: 2 [236.62 GB] / in  no VG: 0 [0]
>>>      
>>>
>I don't see /dev/cciss/c1d0p0 in the pvscan output.  It does show 
>that /dev/cciss/c1d0p1 is unused.  Also /dev/cciss/c0d0p2 is much larger than 
>c1d0p1.
>  
>
>>>/sbin/sh-2.05b# pvmove /dev/cciss/c1d0p1
>>>      
>>>
>Is this what you mean?  pvmove will remove all extents used on the specified 
>PV off of it.
>  
>
>>I've tried a --force but that didn't help.
>>    
>>
>I'm unclear about what device you are trying to reduce.......  A little more 
>clarity please.
>
>--
>Zac Slade
>  
>

Heh, Sorry Zac - should've payed more attention to what I was typing...

pvscan -- ACTIVE    PV  "/dev/cciss/c1d0p1" of VG "Volume0" [99.99 GB / 99.99 GB Free]
pvscan -- ACTIVE    PV  "/dev/cciss/c0d0p2" of VG "Volume0" [136.61 GB / 68.89 GB Free] 

I need to remove /dev/cciss/c1d0p1.
pvscan shows that its unused, but a "vgreduce Volume0 /dev/cciss/c1d0p1"
tells me it can't reduce because the PV is used.

Any ideas?

-- 
Tom Lanyon
Systems Administrator
NetSpot Pty Ltd
183 Melbourne Street, North Adelaide, 5006
Ph: +618 8361 6800   Fax: +618 8361 6811
Email: tom@netspot.com.au

Re: [linux-lvm] pvmove errors

[Zac Slade]

On Tuesday 31 January 2006 17:26, Tom Lanyon wrote:
> pvscan -- ACTIVE    PV  "/dev/cciss/c1d0p1" of VG "Volume0" [99.99 GB /
> 99.99 GB Free] pvscan -- ACTIVE    PV  "/dev/cciss/c0d0p2" of VG "Volume0"
> [136.61 GB / 68.89 GB Free]
> I need to remove /dev/cciss/c1d0p1.
Okay this is more clear.  Yes the pv is unused and should be possible to 
cleanly remove the pv from the volumegroup.

> pvscan shows that its unused, but a "vgreduce Volume0 /dev/cciss/c1d0p1"
> tells me it can't reduce because the PV is used.
It's always more helpful to post the output of vgreduce, but I'll take a stab 
here.

> Any ideas?
First you should mark the pv as unallocatable pvchange -x no /dev/cciss/c1d0p1
The next step is to remove the pv from the volume group using vgreduce 
Volume0 /dev/cciss/c1d0p1 or alternatively vgreduce -a Volume0

If those do not work please post the output of those commands.
--
Zac Slade