Resizing multipath maps: reload ioctl failed: Invalid argument

Resizing multipath maps: reload ioctl failed: Invalid argument

[Tore Anderson]

  Hi.  I'm trying to resize multipath maps, but there appears to be
 something wrong with the ioctl call...  I resized the "www" volume on
 the storage array from 25GB to 45GB, and attempted to reload it on the
 host, which is running 2.6.23-rc3 and multipath-tools 0.4.7.  It
 resulted in the following:

root@atalanta:~# multipath -ll
[...]
www (3600a0b80002984ae0000179b46a687fb)
[size=25 GB][features=1 queue_if_no_path][hwhandler=1 rdac]
\_ round-robin 0 [prio=6][active]
 \_ 3:0:1:0 sdf 8:80  [active][ready]
 \_ 4:0:1:0 sdi 8:128 [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 4:0:0:0 sdc 8:32  [active][ready]
 \_ 3:0:0:0 sdd 8:48  [active][ready]
root@atalanta:~# for d in sdf sdi sdc sdd; do cat /sys/block/$d/size; done
52428800
52428800
52428800
52428800
root@atalanta:~# for d in sdf sdi sdc sdd; do echo 1 > /sys/block/$d/device/rescan; done
root@atalanta:~# for d in sdf sdi sdc sdd; do cat /sys/block/$d/size; done
94371840
94371840
94371840
94371840
root@atalanta:~# multipath -v 2 -d
reload: www (3600a0b80002984ae0000179b46a687fb)
[size=45 GB][features=0][hwhandler=1 rdac]
\_ round-robin 0 [prio=0][undef]
 \_ 4:0:0:0 sdc 8:32  [active][ready]
 \_ 3:0:0:0 sdd 8:48  [active][ready]
\_ round-robin 0 [prio=6][undef]
 \_ 3:0:1:0 sdf 8:80  [active][ready]
 \_ 4:0:1:0 sdi 8:128 [active][ready]
[...]
root@atalanta:~# multipath -v 2
device-mapper: reload ioctl failed: Invalid argument
device-mapper: reload ioctl failed: Invalid argument
device-mapper: reload ioctl failed: Invalid argument
device-mapper: reload ioctl failed: Invalid argument
root@atalanta:~# dmsetup table www | sed 's/52428800/94371840/' | dmsetup reload www
device-mapper: reload ioctl failed: Invalid argument
Command failed

  Ultimately I found no other way to make this work but to umount the
 file system, then run "multipath -f www; multipath", which kind of
 defeated the point of fancy online resizing support in the file system
 and storage array since I need downtime to do it.  Is there any other
 way to make online resizing work?

  By the way, I think the error is happening in the kernel, due to the
 following messages appearing in dmesg after the rescan and multipath
 steps:

sd 3:0:1:0: [sdf] 94371840 512-byte hardware sectors (48318 MB)
sd 3:0:1:0: [sdf] Write Protect is off
sd 3:0:1:0: [sdf] Mode Sense: 77 00 10 08
sd 3:0:1:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA
sd 4:0:1:0: [sdi] 94371840 512-byte hardware sectors (48318 MB)
sd 4:0:1:0: [sdi] Write Protect is off
sd 4:0:1:0: [sdi] Mode Sense: 77 00 10 08
sd 4:0:1:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA
sd 4:0:0:0: [sdc] 94371840 512-byte hardware sectors (48318 MB)
sd 4:0:0:0: [sdc] Write Protect is off
sd 4:0:0:0: [sdc] Mode Sense: 77 00 10 08
sd 4:0:0:0: [sdc] Uses READ/WRITE(6), disabling FUA
sd 4:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 3:0:0:0: [sdd] 94371840 512-byte hardware sectors (48318 MB)
sd 3:0:0:0: [sdd] Write Protect is off
sd 3:0:0:0: [sdd] Mode Sense: 77 00 10 08
sd 3:0:0:0: [sdd] Uses READ/WRITE(6), disabling FUA
sd 3:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
device-mapper: multipath rdac: using RDAC command with timeout 6000
device-mapper: table: device 8:32 too small for target
device-mapper: table: 253:3: multipath: error getting device
device-mapper: ioctl: error adding target to table 
device-mapper: multipath rdac: using RDAC command with timeout 6000
device-mapper: table: device 8:32 too small for target
device-mapper: table: 253:3: multipath: error getting device
device-mapper: ioctl: error adding target to table 
device-mapper: multipath rdac: using RDAC command with timeout 6000
device-mapper: table: device 8:32 too small for target
device-mapper: table: 253:3: multipath: error getting device
device-mapper: ioctl: error adding target to table 
device-mapper: multipath rdac: using RDAC command with timeout 6000
device-mapper: table: device 8:32 too small for target
device-mapper: table: 253:3: multipath: error getting device
device-mapper: ioctl: error adding target to table 

Regards
-- 
Tore Anderson

Re: Resizing multipath maps: reload ioctl failed: Invalid argument

[Jun'ichi Nomura]

Hi,

Tore Anderson wrote:
>   Hi.  I'm trying to resize multipath maps, but there appears to be
>  something wrong with the ioctl call...  I resized the "www" volume on
>  the storage array from 25GB to 45GB, and attempted to reload it on the
>  host, which is running 2.6.23-rc3 and multipath-tools 0.4.7.  It
>  resulted in the following:

It's due to kernel device-mapper restriction.

multipath-tools uses no_flush suspend of device-mapper device
to save I/O errors in all-paths-down case.
(Without the option, device-mapper needs to flush all pending I/Os,
 that will result in I/O errors if there is no working path.)

For no_flush suspend, resizing is disabled because of known dead-lock:
it requires a lock that can be kept hold until the pending I/Os
are flushed.

>   Ultimately I found no other way to make this work but to umount the
>  file system, then run "multipath -f www; multipath", which kind of
>  defeated the point of fancy online resizing support in the file system
>  and storage array since I need downtime to do it.  Is there any other
>  way to make online resizing work?

A workaround for it would be using modified multipath-tools which doesn't
use no_noflush.

Thanks,
-- 
Jun'ichi Nomura, NEC Corporation of America

Re: Resizing multipath maps: reload ioctl failed: Invalid argument

[Tore Anderson]

   Hello,

* Jun'ichi Nomura

> It's due to kernel device-mapper restriction.
> 
> multipath-tools uses no_flush suspend of device-mapper device to save
> I/O errors in all-paths-down case. (Without the option, device-mapper
> needs to flush all pending I/Os, that will result in I/O errors if
> there is no working path.)
> 
> For no_flush suspend, resizing is disabled because of known
> dead-lock: it requires a lock that can be kept hold until the pending
> I/Os are flushed.

   Hmm.  So this deadlock is a kernel bug that (hopefully) will be fixed
  in the future, so resizing can again be enabled, or is this a design
  limitation that it's impossible to do something about?

   Can the no_flush thing be toggled online;  that is, would it be
  possible to alter multipath-tools so that when it attempted to resize a
  map, it would disable no_flush, resize it, and then re-enable it?  I
  think a two-second window of vulnerability to all-paths-down is
  acceptable...

   Similarily, if this no_flush option is relevant only when
  features=queue_if_no_paths (at least that's the impression I get from
  your description of it), would it work to temporarily reload the map
  without this feature, resize the volume, than re-add queue_if_no_path?

> A workaround for it would be using modified multipath-tools which
> doesn't use no_noflush.

   I grepped for noflush and no_flush (and so on) in the multipath-tools
  source code but can't really find where it is set..?

Thanks
-- 
Tore Anderson

Re: Resizing multipath maps: reload ioctl failed: Invalid argument

[Jun'ichi Nomura]

Hi,

Tore Anderson wrote:
>> It's due to kernel device-mapper restriction.
>>
>> multipath-tools uses no_flush suspend of device-mapper device to save
>> I/O errors in all-paths-down case. (Without the option, device-mapper
>> needs to flush all pending I/Os, that will result in I/O errors if
>> there is no working path.)
>>
>> For no_flush suspend, resizing is disabled because of known
>> dead-lock: it requires a lock that can be kept hold until the pending
>> I/Os are flushed.
> 
>   Hmm.  So this deadlock is a kernel bug that (hopefully) will be fixed
>  in the future, so resizing can again be enabled, or is this a design
>  limitation that it's impossible to do something about?

IMO, it's not a design limitation and should be solved in future.

>   Can the no_flush thing be toggled online;  that is, would it be
>  possible to alter multipath-tools so that when it attempted to resize a
>  map, it would disable no_flush, resize it, and then re-enable it?  I
>  think a two-second window of vulnerability to all-paths-down is
>  acceptable...

Everywhen you suspend the device, there is a possibility of
all paths being down. So as far as you don't want to lose data,
you have to use no_flush option.

>   Similarily, if this no_flush option is relevant only when
>  features=queue_if_no_paths (at least that's the impression I get from
>  your description of it), would it work to temporarily reload the map
>  without this feature, resize the volume, than re-add queue_if_no_path?

Current code uses 'no_flush' unconditionally.
I think it's possible to improve it to do that only when
queue_if_no_path is enabled.

>> A workaround for it would be using modified multipath-tools which
>> doesn't use no_noflush.
> 
>   I grepped for noflush and no_flush (and so on) in the multipath-tools
>  source code but can't really find where it is set..?

Hmm, sorry. I misread your first message.
Since the feature is added after 0.4.7, your problem may be caused
by other reason.
The call to dm_task_no_flush() is added after the release of 0.4.7.
It should be in dm_simplecmd() in libmultipath/devmapper.c.

Thanks,
-- 
Jun'ichi Nomura, NEC Corporation of America

Re: Resizing multipath maps: reload ioctl failed: Invalid argument

[Jun'ichi Nomura]

Hi,

Tore Anderson wrote:
>> Current code uses 'no_flush' unconditionally.
>> I think it's possible to improve it to do that only when
>> queue_if_no_path is enabled.
> 
>   It certainly would be an improvement if no_flush and queue_if_no_path
>  could be automatically disabled in the short time it will take to
>  reload the multipath map with an increased size.  For extra safety the
>  tool could also verify that no paths were failed before doing so,
>  maybe.  If everything is working fine it seems rather unlikely that all
>  paths will fail and all HBA driver timeouts will expire in the second
>  it takes to reload the multipath map.

It's not good to override a policy about queue_if_no_path, that will
create a small window of data loss.
Even the small window could result in a disaster.
I meant in the previous mail that there is no need for multipath-tools
to use no_flush if a user explicitly configures the device without
queue_if_no_path.
Assuming that users sensitive to the system availability would set
queue_if_no_path, I'm not sure how useful the improvement is.

>> Since the feature is added after 0.4.7, your problem may be caused
>> by other reason.
>> The call to dm_task_no_flush() is added after the release of 0.4.7.
>> It should be in dm_simplecmd() in libmultipath/devmapper.c.
> 
>   Interesting.  I sent an email titled "dm-rdac not working?" earlier
>  today, where I described another problem with the RDAC hwhandler that
>  also caused I/O errors to propagate up from device-mapper into VFS,
>  even though queue_if_no_path was in use.  Is it possible, you think,
>  that this misbehaviour was due to having loaded the multipath maps
>  using 0.4.7 and therefore without no_flush being set?  (Or in other
>  words:  Is 0.4.8 a requirement for queue_if_no_path to work correctly?)

As far as the table is not suspended, e.g. just failing or reinstating
paths, queue_if_no_path works correctly without no_flush.
When the table is suspended, e.g. a path is added or removed, pending
I/Os will be flushed on suspend even if queue_if_no_path is specified.
That's the situation no_flush solves.

Easiest check is perhaps stracing multipath/multipathd and see what
ioctls they issue. If they do DM_DEV_SUSPEND, using 0.4.8 might be a
solution. If they don't, the cause is in other place.

Thanks,
-- 
Jun'ichi Nomura, NEC Corporation of America

Resizing multipath maps: reload ioctl failed: Invalid argument

[Domenico Viggiani]

Hi,
I'm trying to resize a multipath map (an EMC Clariion LUN) but something
goes wrong with ioctl call. 
After resizing LUN by EMC Navisphere, I rescanned four paths using:
 echo "1" > /sys/bus/scsi/drivers/sd/<h>\:<b>\:<t>\:<l>/block/device/rescan
 ...
I noticed (in /var/log/messages) that kernel "sees" new size only for two
paths, surely the active ones (Clariion is an active/passive array).
Then, I trespassed LUN on the other storage processor and rescanned all
paths a second time: all sizes was updated.
But, re-running 'multipath -v2', I got:
 # multipath -v 2
 device-mapper: reload ioctl failed: Invalid argument
I found no other way to make this work but umounting filesystem,
deactivating LVM and ran 'multipath -f ... && multipath -v2' (and then other
regular LVM stuffs).
OS is Red Hat 4.5.

About this, I found this message in archives:
 http://www.redhat.com/archives/dm-devel/2007-August/msg00188.html

Is it confirmed that online resizing with multipath is impossible?

If yes, in my opinion, this is a serious limitation to a serious enterprise
use of Linux.

Thanks in advance for any help
-- 
Domenico Viggiani

Re: Resizing multipath maps: reload ioctl failed: Invalid argument

[Aaron Bergstralh]

> Hi,
> I'm trying to resize a multipath map (an EMC Clariion LUN) but something
> goes wrong with ioctl call. 
> After resizing LUN by EMC Navisphere, I rescanned four paths using:
>  echo "1" > /sys/bus/scsi/drivers/sd/<h>\:<b>\:<t>\:<l>/block/device/rescan
>  ...
> I noticed (in /var/log/messages) that kernel "sees" new size only for 
> two
> paths, surely the active ones (Clariion is an active/passive array).
> Then, I trespassed LUN on the other storage processor and rescanned all
> paths a second time: all sizes was updated.
> But, re-running 'multipath -v2', I got:
>  # multipath -v 2
>  device-mapper: reload ioctl failed: Invalid argument
> I found no other way to make this work but umounting filesystem,
> deactivating LVM and ran 'multipath -f ... && multipath -v2' (and then 
> other
> regular LVM stuffs).
> OS is Red Hat 4.5.
> 
> About this, I found this message in archives:
>  http://www.redhat.com/archives/dm-devel/2007-August/msg00188.html
> 
> Is it confirmed that online resizing with multipath is impossible?
> 
> If yes, in my opinion, this is a serious limitation to a serious enterprise
> use of Linux.

I have used the following in the past when I needed to resize online. I have only tested this for the round robin selector but a similar method could also work for other selectors.

#Started with the following dm table
dmsetup table m500i_A_lun_4
0 157287936 multipath 0 0 1 1 round-robin 0 2 1 8:80 100 8:160 100

###Used the following in a script to complete the resize
dmsetup suspend m500i_A_lun_4

#Reload with one device removed and old size
dmsetup load m500i_A_lun_4 <<EOF
0 157287936 multipath 0 0 1 1 round-robin 0 1 1 8:80 100
EOF
dmsetup resume m500i_A_lun_4
dmsetup suspend m500i_A_lun_4

#Reload with one device removed for all other devices
dmsetup load m500i_A_lun_4 <<EOF
0 157287936 multipath 0 0 1 1 round-robin 0 1 1 8:160 100
EOF
dmsetup resume m500i_A_lun_4
dmsetup suspend m500i_A_lun_4

#Reload with all devices and new size
dmsetup load m500i_A_lun_4 <<EOF
0 209717248 multipath 0 0 1 1 round-robin 0 2 1 8:80 100 8:160 100
EOF
dmsetup resume m500i_A_lun_4

After the last step everything is able to see the new device size. I know this isn't an ideal solution but it might be useful for some people.

Aaron Bergstralh

Re: Resizing multipath maps: reload ioctl failed: Invalid argument

[Jun'ichi Nomura]

Hi,

Domenico Viggiani wrote:
>  http://www.redhat.com/archives/dm-devel/2007-August/msg00188.html
> 
> Is it confirmed that online resizing with multipath is impossible?
> 
> If yes, in my opinion, this is a serious limitation to a serious enterprise
> use of Linux.

I'm trying to find a kernel fix for this issue.

I'll post patches for 2.6.23 kernel.
(But it should be easily adaptible to other versions.)

I have tested the patches mainly on modified LVM2 (i.e. resizing
mirror LV).
Since I don't have resizable multipath device,
if you are interested in testing, it would be much appreciated.

Alternative is to use dmsetup directly, as Aaron suggested.
But there is a corner case that if all paths are down before
'dmsetup suspend', I/O errors will occur regardless of
queue_if_no_path setting.

Thanks,
-- 
Jun'ichi Nomura, NEC Corporation of America