[linux-lvm] Unable to remove a pvmove LV

[linux-lvm] Unable to remove a pvmove LV

[Delian Krustev]

The problem appeared after I've had a problem with this controller:

0000:00:0e.0 RAID bus controller: Silicon Image, Inc. (formerly CMD Technology Inc) PCI0680 Ultra ATA-133 Host Controller (rev 02)

I've added two more disks to this controller and to the volume group
and messages similar to this:

Oct 26 20:28:21 serv0 kernel: hda: dma_intr: status=0x7f { DriveReady DeviceFault SeekComplete DataRequest CorrectedError Index Error }
Oct 26 20:28:21 serv0 kernel: hda: dma_intr: error=0x00 { }
Oct 26 20:28:21 serv0 kernel: hda: DMA disabled
Oct 26 20:28:21 serv0 kernel: hdb: DMA disabled
Oct 26 20:28:21 serv0 kernel: ide0: reset: success

began to appiear in the logs. I've tried removing the new disks but this LV
became invincible. The machine is running Debian sarge. Details follow:

serv0:/# dpkg -s lvm2 |grep Version
Version: 2.01.04-5
serv0:/# uname -a
Linux serv0 2.6.8-2-k7 #1 Thu May 19 18:03:29 JST 2005 i686 GNU/Linux
serv0:/# lvdisplay
  --- Logical volume ---
  LV Name                /dev/s0_data/data
  VG Name                s0_data
  LV UUID                Wzs9Lu-5IUj-DK88-UVnZ-QJpj-gHuD-3a01jq
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1.02 TB
  Current LE             4164
  Segments               23
  Allocation             inherit
  Read ahead sectors     0
  Block device           254:0

  --- Logical volume ---
  LV Name                /dev/s0_data/pvmove0
  VG Name                s0_data
  LV UUID                H8As4Y-m61V-krSa-lGTd-pNRh-Ef1e-UStXSC
  LV Write Access        read/write
  LV Status              NOT available
  LV Size                46.50 GB
  Current LE             186
  Segments               1
  Allocation             contiguous
  Read ahead sectors     0

serv0:/# lvremove -v -f -d /dev/s0_data/pvmove0
    Using logical volume(s) on command line
  Can't remove locked LV pvmove0
serv0:/# pvmove --abort -v
    Finding all volume groups
    Finding volume group "s0_data"
serv0:/# lvdisplay
  --- Logical volume ---
  LV Name                /dev/s0_data/data
  VG Name                s0_data
  LV UUID                Wzs9Lu-5IUj-DK88-UVnZ-QJpj-gHuD-3a01jq
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1.02 TB
  Current LE             4164
  Segments               23
  Allocation             inherit
  Read ahead sectors     0
  Block device           254:0

  --- Logical volume ---
  LV Name                /dev/s0_data/pvmove0
  VG Name                s0_data
  LV UUID                H8As4Y-m61V-krSa-lGTd-pNRh-Ef1e-UStXSC
  LV Write Access        read/write
  LV Status              NOT available
  LV Size                46.50 GB
  Current LE             186
  Segments               1
  Allocation             contiguous
  Read ahead sectors     0

serv0:/#

Additionally the pvmove process was killed several times by the OOM
killer, since in my opinion it was leaking memory. The PVs are 50 GBs
each, and at some time at over 90 % the pvmove process was eating
all the memory(512 MBs). Surprisingly after trying to resume some minutes
later I've found the move had completed successfully.

Cheers,
Delian

Re: [linux-lvm] Unable to remove a pvmove LV

[Alasdair G Kergon]

On Wed, Nov 02, 2005 at 10:28:20PM +0200, Delian Krustev wrote:
> Surprisingly after trying to resume some minutes
> later I've found the move had completed successfully.
 
The userspace process just manage and monitor the pvmove - the actual
work is done by a kernel thread.

So killing the userspace processes doesn't stop the pvmove.
Only pvmove --abort is capable of telling the kernel to
abandon it.

Alasdair
-- 
agk@redhat.com

Re: [linux-lvm] Unable to remove a pvmove LV

[Randall A. Jones]

Delian Krustev wrote:

> serv0:/# lvdisplay
>   --- Logical volume ---
>   LV Name                /dev/s0_data/data
>   VG Name                s0_data
>   LV UUID                Wzs9Lu-5IUj-DK88-UVnZ-QJpj-gHuD-3a01jq
>   LV Write Access        read/write
>   LV Status              available
>   # open                 1
>   LV Size                1.02 TB
>   Current LE             4164
>   Segments               23
>   Allocation             inherit
>   Read ahead sectors     0
>   Block device           254:0
> 
>   --- Logical volume ---
>   LV Name                /dev/s0_data/pvmove0
>   VG Name                s0_data
>   LV UUID                H8As4Y-m61V-krSa-lGTd-pNRh-Ef1e-UStXSC
>   LV Write Access        read/write
>   LV Status              NOT available
>   LV Size                46.50 GB
>   Current LE             186
>   Segments               1
>   Allocation             contiguous
>   Read ahead sectors     0
> 
> serv0:/# lvremove -v -f -d /dev/s0_data/pvmove0
>     Using logical volume(s) on command line
>   Can't remove locked LV pvmove0
> serv0:/# pvmove --abort -v
>     Finding all volume groups
>     Finding volume group "s0_data"

Another way to remove a misbehaving pvmove logical volume is to restore 
to a metadata config state previous to the pvmove command.  The command 
to do this is vgcfgrestore.  This is only possible if you haven't 
modified the volume group or the logical volume in a way that might 
cause you to lose something important.  For example if you created a new 
LV and put a filesystem on it *after* a failed or uncleaned pvmove, you 
CANNOT restore to the config before the pvmove or you'll lose your new LV.

DISCLAIMER: I don't know what happens if you (accidentally?) try to 
restore metadata that results in removing an active, in use volume group 
or logical volume.  Does anyone have any information about how dangerous 
this is?  Is there a check for active VGs or LVs when restoring previous 
metadata with vgcfgrestore?


For example, assuming a volume group named "vg0", to see the metadata 
config history for vg0 use:

vgcfgrestore -l vg0

Look for the desired config before the pvmove command.  Let's say the 
pvmove happened after the vg0_00009.vg metadata backup was stored, then 
restore that config with:

vgcfgrestore -v --file /etc/lvm/archive/vg0_00009.vg vg0

Be careful what you restore and look at any recent changes carefully 
before using vgcfgrestore.


Regards,
Randy
-- 
..:.::::
Randall Jones     GST      NASA Goddard Space Flight Center
HPC Visualization Support       http://hpcvis.gsfc.nasa.gov
Scientific Visualization Studio    http://svs.gsfc.nasa.gov
rajones@svs.gsfc.nasa.gov      Code 610.3      301-286-2239

Re: [linux-lvm] Unable to remove a pvmove LV

[Delian Krustev]

On Thursday 03 November 2005 02:11, Randall A. Jones wrote:
> Another way to remove a misbehaving pvmove logical volume is to restore
> to a metadata config state previous to the pvmove command.  The command
> to do this is vgcfgrestore.  This is only possible if you haven't
> modified the volume group or the logical volume in a way that might
> cause you to lose something important.  For example if you created a new
> LV and put a filesystem on it *after* a failed or uncleaned pvmove, you
> CANNOT restore to the config before the pvmove or you'll lose your new LV.

Unfortunately I've removed several PVs from it to leave the problematic
cotroller with two HDDs only(which was proven to work).

Cheers,
Delian

Re: [linux-lvm] Unable to remove a pvmove LV

[Delian Krustev]

On Thursday 03 November 2005 01:37, Alasdair G Kergon wrote:
> On Wed, Nov 02, 2005 at 10:28:20PM +0200, Delian Krustev wrote:
> > Surprisingly after trying to resume some minutes
> > later I've found the move had completed successfully.
>
> The userspace process just manage and monitor the pvmove - the actual
> work is done by a kernel thread.
>
> So killing the userspace processes doesn't stop the pvmove.
> Only pvmove --abort is capable of telling the kernel to
> abandon it.

The bug in the userspace is quite nasty though. Has it been reported and/or
fixed in the recent versions ?

Additionally, would you suggest a way to remove the stale LV ? Ofcourse
as a last step I might go creating a new VG and moving the data to it,
but moving a terrabyte is not s.t. I would like to do often.

On the other side, the stale LV is not causing any problems ..

Cheers,
Delian