[linux-lvm] Strange pvmove problem

[linux-lvm] Strange pvmove problem

[Dariush Forouher]

Hi!

I'm trying to move PEs away from two PVs (because I want to remove the
disks from the system), but pvmove always fails:

# pvmove /dev/hdd1 /dev/sda3
    Wiping cache of LVM-capable devices
    Finding volume group "LVM"
    Archiving volume group "LVM" metadata (seqno 51).
    Creating logical volume pvmove0
    Moving 3468 extents of logical volume LVM/data
    Found volume group "LVM"
    Updating volume group metadata
    Creating volume group backup "/etc/lvm/backup/LVM" (seqno 52).
    Found volume group "LVM"
    Found volume group "LVM"
    Suspending LVM-data (253:1)
    Found volume group "LVM"
    Creating LVM-pvmove0
    Loading LVM-pvmove0 table
  device-mapper: reload ioctl failed: Invalid argument
  ABORTING: Temporary mirror activation failed.  Run pvmove --abort.
    Found volume group "LVM"
    Loading LVM-pvmove0 table
  device-mapper: reload ioctl failed: Invalid argument
    Loading LVM-data table
  device-mapper: reload ioctl failed: Invalid argument


After that the system remains in an unstable state. All [vg|pv|lv]*
commands hang and the LV becomes unaccessible too.

After rebooting and excecuting "pvmove" without any arguments the VG
becomes accessible again and I'm back where I started (without any PE
moved). Trying to move data from another PV makes no difference.

The system's running with Debian Testing and Kernel 2.6.16.16.

Can anyone please help me finding out whats causing this?

ciao
Dariush

# pvdisplay
File descriptor 3 left open
File descriptor 4 left open
File descriptor 6 left open
File descriptor 8 left open
  --- Physical volume ---
  PV Name               /dev/hda3
  VG Name               LVM
  PV Size               129.48 GB / not usable 0
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              33147
  Free PE               33147
  Allocated PE          0
  PV UUID               uc4kn4-xMQ7-M4vw-1thl-OcCF-LIV9-aeRTn0

  --- Physical volume ---
  PV Name               /dev/hdd1
  VG Name               LVM
  PV Size               37.27 GB / not usable 0
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              9541
  Free PE               6073
  Allocated PE          3468
  PV UUID               tNzm5d-ClTt-0vJ6-ZQA9-0eYQ-IwKy-tbxV6o

  --- Physical volume ---
  PV Name               /dev/sda3
  VG Name               LVM
  PV Size               214.19 GB / not usable 0
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              54833
  Free PE               54734
  Allocated PE          99
  PV UUID               CfA2kz-5Gt8-Zshc-3Eou-fscj-1Lfn-xOEQSY

  --- Physical volume ---
  PV Name               /dev/sdb1
  VG Name               LVM
  PV Size               232.88 GB / not usable 0
  Allocatable           yes (but full)
  PE Size (KByte)       4096
  Total PE              59618
  Free PE               0
  Allocated PE          59618
  PV UUID               L6S26w-zr8S-t0RN-si42-zWEd-rJA5-Ms0JIb

# vgdisplay
File descriptor 3 left open
File descriptor 4 left open
File descriptor 6 left open
File descriptor 8 left open
  --- Volume group ---
  VG Name               LVM
  System ID
  Format                lvm2
  Metadata Areas        4
  Metadata Sequence No  55
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                4
  Act PV                4
  VG Size               613.82 GB
  PE Size               4.00 MB
  Total PE              157139
  Alloc PE / Size       63185 / 246.82 GB
  Free  PE / Size       93954 / 367.01 GB
  VG UUID               GoVzA8-HPSg-FNhr-6fM3-Fo94-k2IG-462F9E

# lvdisplay
File descriptor 3 left open
File descriptor 4 left open
File descriptor 6 left open
File descriptor 8 left open
  --- Logical volume ---
  LV Name                /dev/LVM/data
  VG Name                LVM
  LV UUID                zscrIa-BwvW-rktv-3L5P-QP67-BkfG-ri3NZL
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                246.82 GB
  Current LE             63185
  Segments               3
  Allocation             inherit
  Read ahead sectors     0
  Block device           253:1

Re: [linux-lvm] Strange pvmove problem

[Dieter Stüken]

Dariush Forouher wrote:
> I'm trying to move PEs away from two PVs (because I want to remove the
> disks from the system), but pvmove always fails:

I had some problems with pvmove some time ago, if the PV to empty
contained parts from several different LVs. But this was due to some 
old bug with a 2.6.12 or 13 kernel. May be it's interesting to have
a look into /etc/lvm/backup/LVM and the associated backups,
to understand whats going on.

To overcome the problem of hanging LVM I shut down my server and
performed a "pvmove --abort" in single user mode while the LVM was
not activated. This restored it to the state before.

Dieter.

Re: [linux-lvm] Strange pvmove problem

[Dariush Forouher]

Dieter St�ken wrote:
> Dariush Forouher wrote:
>> I'm trying to move PEs away from two PVs (because I want to remove the
>> disks from the system), but pvmove always fails:
> 
> I had some problems with pvmove some time ago, if the PV to empty
> contained parts from several different LVs. But this was due to some 
> old bug with a 2.6.12 or 13 kernel. May be it's interesting to have
> a look into /etc/lvm/backup/LVM and the associated backups,
> to understand whats going on.

Uhm, this is the file created right before I started pvmove:

<SNIP>
# Generated by LVM2: Fri Jul 28 10:15:24 2006

contents = "Text Format Volume Group"
version = 1

description = "Created *before* executing 'pvmove -v /dev/hdd1 /dev/hda3'"

creation_host = "palomar"       # Linux palomar 2.6.16.16 #2 PREEMPT Sun
May 14 13:56:04 CEST 2006 i686
creation_time = 1154074524      # Fri Jul 28 10:15:24 2006

LVM {
        id = "GoVzA8-HPSg-FNhr-6fM3-Fo94-k2IG-462F9E"
        seqno = 39
        status = ["RESIZEABLE", "READ", "WRITE"]
        extent_size = 8192              # 4 Megabytes
        max_lv = 0
        max_pv = 0

        physical_volumes {

                pv0 {
                        id = "uc4kn4-xMQ7-M4vw-1thl-OcCF-LIV9-aeRTn0"
                        device = "/dev/hda3"    # Hint only

                        status = ["ALLOCATABLE"]
                        pe_start = 384
                        pe_count = 33147        # 129.48 Gigabytes
                }

                pv1 {
                        id = "tNzm5d-ClTt-0vJ6-ZQA9-0eYQ-IwKy-tbxV6o"
                        device = "/dev/hdd1"    # Hint only

                        status = ["ALLOCATABLE"]
                        pe_start = 384
                        pe_count = 9541 # 37.2695 Gigabytes
                }

                pv2 {
                        id = "CfA2kz-5Gt8-Zshc-3Eou-fscj-1Lfn-xOEQSY"
                        device = "/dev/sda3"    # Hint only

                        status = ["ALLOCATABLE"]
                        pe_start = 384
                        pe_count = 54833        # 214.191 Gigabytes
                }

                pv3 {
                        id = "L6S26w-zr8S-t0RN-si42-zWEd-rJA5-Ms0JIb"
                        device = "/dev/sdb1"    # Hint only

                        status = ["ALLOCATABLE"]
                        pe_start = 384
                        pe_count = 59618        # 232.883 Gigabytes
                }
        }

        logical_volumes {

                data {
                        id = "zscrIa-BwvW-rktv-3L5P-QP67-BkfG-ri3NZL"
                        status = ["READ", "WRITE", "VISIBLE"]
                        segment_count = 3

                        segment1 {
                                start_extent = 0
                                extent_count = 59618    # 232.883 Gigabytes

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = [
                                        "pv3", 0
                                ]
                        }
                        segment2 {
                                start_extent = 59618
                                extent_count = 3468     # 13.5469 Gigabytes

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = [
                                        "pv1", 6073
                                ]
                        }
                        segment3 {
                                start_extent = 63086
                                extent_count = 99       # 396 Megabytes

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = [
                                        "pv2", 54734
                                ]
                        }
                }
        }
}
</SNIP>

I diff'ed it againt more recent backups and against /etc/lvm/backup/LVM.
They are pretty much the same (except the "seqno" attribute).

I'd be already happy if I could free /dev/sda3 (which is _nearly_
unused). Then I could create a new VG and move the files around by hand
(like in the old days, when we had to use partitions). But even thats
not possible since all disks big enough to hold the data are bound by
this (damaged?) VG.

ciao
Dariush

Re: [linux-lvm] Strange pvmove problem

[Dieter Stüken]

Dariush Forouher wrote:
> Uhm, this is the file created right before I started pvmove:
....

seems the pvmove was not even started because of a problems
with the underlaying device-mapper:

>  device-mapper: reload ioctl failed: Invalid argument

sorry, no idea :-(

any  messages from syslog / dmesg?
does it work while LV not active (i.E. boot from rescue CD)?

Dieter.

Re: [linux-lvm] Strange pvmove problem

[Dariush Forouher]

Dieter St�ken wrote:

> any  messages from syslog / dmesg?

Jupp, these:

device-mapper: unrecognised sync argument to mirror log: 2
device-mapper: dm-mirror: Error creating mirror dirty log
device-mapper: error adding target to table
device-mapper: unrecognised sync argument to mirror log: 2
device-mapper: dm-mirror: Error creating mirror dirty log
device-mapper: error adding target to table
device-mapper: device 253:1 too small for target
device-mapper: error adding target to table

> does it work while LV not active (i.E. boot from rescue CD)?

Nope. :(

ciao
Dariush

Re: [linux-lvm] Strange pvmove problem

[Dariush Forouher]

I was able to workaround the problem by creating a new LV (which I
placed exactly on those PVs which I intended to keep) and then moving
all files to the the new LV by hand. This way I didn't have to use pvmove.

Anyway, thanks for your help!

ciao
Dariush

Dieter St�ken wrote:
> Dariush Forouher wrote:
>> Uhm, this is the file created right before I started pvmove:
> ....
> 
> seems the pvmove was not even started because of a problems
> with the underlaying device-mapper:
> 
>>  device-mapper: reload ioctl failed: Invalid argument
> 
> sorry, no idea :-(
> 
> any  messages from syslog / dmesg?
> does it work while LV not active (i.E. boot from rescue CD)?
> 
> Dieter.
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Strange pvmove problem

[Heinz Mauelshagen]

You've got a kernel/userspace version mismatch on device-mapper
and lvm2.

Check your config an upgrade.

Heinz

On Fri, Jul 28, 2006 at 06:21:14PM +0200, Dariush Forouher wrote:
> Dieter St�ken wrote:
> 
> > any  messages from syslog / dmesg?
> 
> Jupp, these:
> 
> device-mapper: unrecognised sync argument to mirror log: 2
> device-mapper: dm-mirror: Error creating mirror dirty log
> device-mapper: error adding target to table
> device-mapper: unrecognised sync argument to mirror log: 2
> device-mapper: dm-mirror: Error creating mirror dirty log
> device-mapper: error adding target to table
> device-mapper: device 253:1 too small for target
> device-mapper: error adding target to table
> 
> > does it work while LV not active (i.E. boot from rescue CD)?
> 
> Nope. :(
> 
> ciao
> Dariush
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Red Hat GmbH
Consulting Development Engineer                   Am Sonnenhang 11
Storage Development                               56242 Marienrachdorf
                                                  Germany
Mauelshagen@RedHat.com                            PHONE +49  171 7803392
                                                  FAX   +49 2626 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] Strange pvmove problem

[Dariush Forouher]

Heinz Mauelshagen wrote:
> You've got a kernel/userspace version mismatch on device-mapper
> and lvm2.
> 
> Check your config an upgrade.

Ah, I thought with using Debian Testing I would have been using bleeding
edge versions allready. Obviously I was wrong.

Thanks for the hint!

ciao
Dariush