4.4-rc4 multipath not working?

4.4-rc4 multipath not working?

[E V]

Just tried booting 4.4-rc4 on a system with using the mptsas driver:
LSISAS3008: FWVersion(03.00.08.00), ChipRevision(0x02), BiosVersion(05.07.02.00)

And multipath doesn't seem to be working, dmesg has lots of these:

device-mapper: table: 254:2: multipath: unknown path selector type
device-mapper: ioctl: error adding target to table

Is it expected to work? and any suggestions on getting it to work,

Thanks,
-Eli

Re: 4.4-rc4 multipath not working?

[Mike Snitzer]

On Fri, Dec 11 2015 at 12:23pm -0500,
E V <eliventer@gmail.com> wrote:

> Just tried booting 4.4-rc4 on a system with using the mptsas driver:
> LSISAS3008: FWVersion(03.00.08.00), ChipRevision(0x02), BiosVersion(05.07.02.00)
> 
> And multipath doesn't seem to be working, dmesg has lots of these:
> 
> device-mapper: table: 254:2: multipath: unknown path selector type
> device-mapper: ioctl: error adding target to table

You compile the path selector module you're attempting to use?
 
> Is it expected to work? and any suggestions on getting it to work,

Yes, no reason dm-multipath shouldn't work (other than user error).

Figure out what the multipath table load line that failed to load
(verbose logging with multipathd?) and make sure you have the
corresponding path selector modules enabled in your .config

Re: 4.4-rc4 multipath not working?

[E V]

On Fri, Dec 11, 2015 at 4:00 PM, Mike Snitzer <snitzer@redhat.com> wrote:
> On Fri, Dec 11 2015 at 12:23pm -0500,
> E V <eliventer@gmail.com> wrote:
>
>> Just tried booting 4.4-rc4 on a system with using the mptsas driver:
>> LSISAS3008: FWVersion(03.00.08.00), ChipRevision(0x02), BiosVersion(05.07.02.00)
>>
>> And multipath doesn't seem to be working, dmesg has lots of these:
>>
>> device-mapper: table: 254:2: multipath: unknown path selector type
>> device-mapper: ioctl: error adding target to table
>
> You compile the path selector module you're attempting to use?
Yes as far as I can tell, here are the _DM_ config options in my config:
CONFIG_BLK_DEV_DM_BUILTIN=y
# CONFIG_DM_MQ_DEFAULT is not set
# CONFIG_DM_DEBUG is not set
# CONFIG_DM_CRYPT is not set
# CONFIG_DM_SNAPSHOT is not set
# CONFIG_DM_THIN_PROVISIONING is not set
# CONFIG_DM_CACHE is not set
# CONFIG_DM_ERA is not set
# CONFIG_DM_MIRROR is not set
CONFIG_DM_RAID=m
# CONFIG_DM_ZERO is not set
CONFIG_DM_MULTIPATH=y
CONFIG_DM_MULTIPATH_QL=y
# CONFIG_DM_MULTIPATH_ST is not set
# CONFIG_DM_DELAY is not set
CONFIG_DM_UEVENT=y
# CONFIG_DM_FLAKEY is not set
# CONFIG_DM_VERITY is not set
# CONFIG_DM_SWITCH is not set
# CONFIG_DM_LOG_WRITES is not set


>> Is it expected to work? and any suggestions on getting it to work,
>
> Yes, no reason dm-multipath shouldn't work (other than user error).
Entirely possible, this is my first multipath setup.

> Figure out what the multipath table load line that failed to load
> (verbose logging with multipathd?) and make sure you have the
> corresponding path selector modules enabled in your .config

Running multipathd with -v3 shows this in it's output, trimming 1st
few columns for the 1st mpath device in the output. Doesn't mean much
to me, other then I see the queue_length selector is set as expected:
read /etc/multipath.conf
multipath.conf line 8, duplicate keyword: defaults
loading /lib/multipath/libcheckdirectio.so checker
loading /lib/multipath/libprioconst.so prioritizer
set open fds limit to 1048576/1048576
sdab: ownership set to mpathc
sdab: not found in pathvec
sdab: mask = 0xc
sdab: path state = running
sdab: get_state
directio: starting new request
directio: io finished 4096/0
sdab: state = up
sdab: const prio = 1
mpathc: pgfailback = -1 (config file default)
mpathc: pgpolicy = failover (internal default)
mpathc: selector = queue_length (config file default)
mpathc: features = 0 (config file default)
mpathc: hwhandler = 0 (internal default)
mpathc: rr_weight = 1 (internal default)
mpathc: minio = 1 rq (config file default)
mpathc: no_path_retry = 0 (internal default)
mpathc: fast_io_fail_tmo = 5 (config file default)
mpathc: retain_attached_hw_handler = 1 (config file default)
sdd: No SAS end device for 'end_device-1:0'
sdab: No SAS end device for 'end_device-1:1'
mpathc: remove queue_if_no_path from '0'
mpathc: assembled map [0 0 2 1 queue_length 1 1 8:48 1 queue_length 1
1 65:176 1]
mpathc: set ACT_CREATE (map does not exist)
mpathc: domap (0) failure for create/reload map
mpathc: ignoring map
sdd: orphan path, map flushed
const prioritizer refcount 46
directio checker refcount 46
sdab: orphan path, map flushed

then after it goes through all the devices it says this:
dm-5: remove map (uevent)
dm-5: No DM_NAME in uevent, ignoring
uevent 'add' from '/devices/virtual/block/dm-5'
uevent 'remove' from '/devices/virtual/block/dm-5'

Re: 4.4-rc4 multipath not working?

[Mike Snitzer]

On Mon, Dec 14 2015 at 11:55am -0500,
E V <eliventer@gmail.com> wrote:

> On Fri, Dec 11, 2015 at 4:00 PM, Mike Snitzer <snitzer@redhat.com> wrote:
> > On Fri, Dec 11 2015 at 12:23pm -0500,
> > E V <eliventer@gmail.com> wrote:
> >
> >> Just tried booting 4.4-rc4 on a system with using the mptsas driver:
> >> LSISAS3008: FWVersion(03.00.08.00), ChipRevision(0x02), BiosVersion(05.07.02.00)
> >>
> >> And multipath doesn't seem to be working, dmesg has lots of these:
> >>
> >> device-mapper: table: 254:2: multipath: unknown path selector type
> >> device-mapper: ioctl: error adding target to table
> >
> > You compile the path selector module you're attempting to use?
> Yes as far as I can tell, here are the _DM_ config options in my config:
> CONFIG_BLK_DEV_DM_BUILTIN=y
> # CONFIG_DM_MQ_DEFAULT is not set
> # CONFIG_DM_DEBUG is not set
> # CONFIG_DM_CRYPT is not set
> # CONFIG_DM_SNAPSHOT is not set
> # CONFIG_DM_THIN_PROVISIONING is not set
> # CONFIG_DM_CACHE is not set
> # CONFIG_DM_ERA is not set
> # CONFIG_DM_MIRROR is not set
> CONFIG_DM_RAID=m
> # CONFIG_DM_ZERO is not set
> CONFIG_DM_MULTIPATH=y
> CONFIG_DM_MULTIPATH_QL=y
> # CONFIG_DM_MULTIPATH_ST is not set
> # CONFIG_DM_DELAY is not set
> CONFIG_DM_UEVENT=y
> # CONFIG_DM_FLAKEY is not set
> # CONFIG_DM_VERITY is not set
> # CONFIG_DM_SWITCH is not set
> # CONFIG_DM_LOG_WRITES is not set
> 
> 
> >> Is it expected to work? and any suggestions on getting it to work,
> >
> > Yes, no reason dm-multipath shouldn't work (other than user error).
> Entirely possible, this is my first multipath setup.
> 
> > Figure out what the multipath table load line that failed to load
> > (verbose logging with multipathd?) and make sure you have the
> > corresponding path selector modules enabled in your .config
> 
> Running multipathd with -v3 shows this in it's output, trimming 1st
> few columns for the 1st mpath device in the output. Doesn't mean much
> to me, other then I see the queue_length selector is set as expected:
> read /etc/multipath.conf
> multipath.conf line 8, duplicate keyword: defaults
> loading /lib/multipath/libcheckdirectio.so checker
> loading /lib/multipath/libprioconst.so prioritizer
> set open fds limit to 1048576/1048576
> sdab: ownership set to mpathc
> sdab: not found in pathvec
> sdab: mask = 0xc
> sdab: path state = running
> sdab: get_state
> directio: starting new request
> directio: io finished 4096/0
> sdab: state = up
> sdab: const prio = 1
> mpathc: pgfailback = -1 (config file default)
> mpathc: pgpolicy = failover (internal default)
> mpathc: selector = queue_length (config file default)
> mpathc: features = 0 (config file default)
> mpathc: hwhandler = 0 (internal default)
> mpathc: rr_weight = 1 (internal default)
> mpathc: minio = 1 rq (config file default)
> mpathc: no_path_retry = 0 (internal default)
> mpathc: fast_io_fail_tmo = 5 (config file default)
> mpathc: retain_attached_hw_handler = 1 (config file default)
> sdd: No SAS end device for 'end_device-1:0'
> sdab: No SAS end device for 'end_device-1:1'
> mpathc: remove queue_if_no_path from '0'
> mpathc: assembled map [0 0 2 1 queue_length 1 1 8:48 1 queue_length 1 1 65:176 1]

This table looks malformed.

Expected format is (from drivers/md/dm-mpath.c):

/*-----------------------------------------------------------------
 * Constructor/argument parsing:
 * <#multipath feature args> [<arg>]*
 * <#hw_handler args> [hw_handler [<arg>]*]
 * <#priority groups>
 * <initial priority group>
 *     [<selector> <#selector args> [<arg>]*
 *      <#paths> <#per-path selector args>
 *         [<path> [<arg>]* ]+ ]+
 *---------------------------------------------------------------*/

So in your case, breaking the map into logical sections, it is:

0 0 2 1

  <#multipath feature args> = 0 
  <#hw_handler args> = 0
  <#priority groups> = 2
  <initial priority group> = 1

queue_length 1 1 8:48 1

     <selector> = queue_length
     <#selector args> = 1 <BUT NO SELECTOR ARG!?>
     <#paths> = 1 <path> = 8:48 <arg> = 1

queue_length 1 1 65:176 1

     <selector> = queue_length <#selector args> = 1 <BUT NO SELECTOR ARG!?>
     <#paths> = 1 <path> = 65:176 <arg> = 1

What do you have in your multipath.conf for 'path_selector'?

AFAIK, it should be: path_selector queue-length 0

(cc'ing Ben in case you need more multipath.conf help)

Re: 4.4-rc4 multipath not working?

[Benjamin Marzinski]

Mike is correct about the malformed table. looking at your ouput, I see

> > mpathc: selector = queue_length (config file default)

This means that you likely have

path_selector "queue_length"

in the defaults section of /etc/multipath.conf

This should be

path_selector "queue-length 0"

With both a dash instead of an underscore, and a zero to let the kernel
know that this selector takes no arguments.

multipath does some sanity checking on the config values, but it
definitely could be better.  For instance, on the values like this, that
it simply passes to the kernel as a string, it doesn't do any. 

-Ben

On Mon, Dec 14, 2015 at 01:51:18PM -0500, Mike Snitzer wrote:
> On Mon, Dec 14 2015 at 11:55am -0500,
> E V <eliventer@gmail.com> wrote:
> 
> > On Fri, Dec 11, 2015 at 4:00 PM, Mike Snitzer <snitzer@redhat.com> wrote:
> > > On Fri, Dec 11 2015 at 12:23pm -0500,
> > > E V <eliventer@gmail.com> wrote:
> > >
> > >> Just tried booting 4.4-rc4 on a system with using the mptsas driver:
> > >> LSISAS3008: FWVersion(03.00.08.00), ChipRevision(0x02), BiosVersion(05.07.02.00)
> > >>
> > >> And multipath doesn't seem to be working, dmesg has lots of these:
> > >>
> > >> device-mapper: table: 254:2: multipath: unknown path selector type
> > >> device-mapper: ioctl: error adding target to table
> > >
> > > You compile the path selector module you're attempting to use?
> > Yes as far as I can tell, here are the _DM_ config options in my config:
> > CONFIG_BLK_DEV_DM_BUILTIN=y
> > # CONFIG_DM_MQ_DEFAULT is not set
> > # CONFIG_DM_DEBUG is not set
> > # CONFIG_DM_CRYPT is not set
> > # CONFIG_DM_SNAPSHOT is not set
> > # CONFIG_DM_THIN_PROVISIONING is not set
> > # CONFIG_DM_CACHE is not set
> > # CONFIG_DM_ERA is not set
> > # CONFIG_DM_MIRROR is not set
> > CONFIG_DM_RAID=m
> > # CONFIG_DM_ZERO is not set
> > CONFIG_DM_MULTIPATH=y
> > CONFIG_DM_MULTIPATH_QL=y
> > # CONFIG_DM_MULTIPATH_ST is not set
> > # CONFIG_DM_DELAY is not set
> > CONFIG_DM_UEVENT=y
> > # CONFIG_DM_FLAKEY is not set
> > # CONFIG_DM_VERITY is not set
> > # CONFIG_DM_SWITCH is not set
> > # CONFIG_DM_LOG_WRITES is not set
> > 
> > 
> > >> Is it expected to work? and any suggestions on getting it to work,
> > >
> > > Yes, no reason dm-multipath shouldn't work (other than user error).
> > Entirely possible, this is my first multipath setup.
> > 
> > > Figure out what the multipath table load line that failed to load
> > > (verbose logging with multipathd?) and make sure you have the
> > > corresponding path selector modules enabled in your .config
> > 
> > Running multipathd with -v3 shows this in it's output, trimming 1st
> > few columns for the 1st mpath device in the output. Doesn't mean much
> > to me, other then I see the queue_length selector is set as expected:
> > read /etc/multipath.conf
> > multipath.conf line 8, duplicate keyword: defaults
> > loading /lib/multipath/libcheckdirectio.so checker
> > loading /lib/multipath/libprioconst.so prioritizer
> > set open fds limit to 1048576/1048576
> > sdab: ownership set to mpathc
> > sdab: not found in pathvec
> > sdab: mask = 0xc
> > sdab: path state = running
> > sdab: get_state
> > directio: starting new request
> > directio: io finished 4096/0
> > sdab: state = up
> > sdab: const prio = 1
> > mpathc: pgfailback = -1 (config file default)
> > mpathc: pgpolicy = failover (internal default)
> > mpathc: selector = queue_length (config file default)
> > mpathc: features = 0 (config file default)
> > mpathc: hwhandler = 0 (internal default)
> > mpathc: rr_weight = 1 (internal default)
> > mpathc: minio = 1 rq (config file default)
> > mpathc: no_path_retry = 0 (internal default)
> > mpathc: fast_io_fail_tmo = 5 (config file default)
> > mpathc: retain_attached_hw_handler = 1 (config file default)
> > sdd: No SAS end device for 'end_device-1:0'
> > sdab: No SAS end device for 'end_device-1:1'
> > mpathc: remove queue_if_no_path from '0'
> > mpathc: assembled map [0 0 2 1 queue_length 1 1 8:48 1 queue_length 1 1 65:176 1]
> 
> This table looks malformed.
> 
> Expected format is (from drivers/md/dm-mpath.c):
> 
> /*-----------------------------------------------------------------
>  * Constructor/argument parsing:
>  * <#multipath feature args> [<arg>]*
>  * <#hw_handler args> [hw_handler [<arg>]*]
>  * <#priority groups>
>  * <initial priority group>
>  *     [<selector> <#selector args> [<arg>]*
>  *      <#paths> <#per-path selector args>
>  *         [<path> [<arg>]* ]+ ]+
>  *---------------------------------------------------------------*/
> 
> So in your case, breaking the map into logical sections, it is:
> 
> 0 0 2 1
> 
>   <#multipath feature args> = 0 
>   <#hw_handler args> = 0
>   <#priority groups> = 2
>   <initial priority group> = 1
> 
> queue_length 1 1 8:48 1
> 
>      <selector> = queue_length
>      <#selector args> = 1 <BUT NO SELECTOR ARG!?>
>      <#paths> = 1 <path> = 8:48 <arg> = 1
> 
> queue_length 1 1 65:176 1
> 
>      <selector> = queue_length <#selector args> = 1 <BUT NO SELECTOR ARG!?>
>      <#paths> = 1 <path> = 65:176 <arg> = 1
> 
> What do you have in your multipath.conf for 'path_selector'?
> 
> AFAIK, it should be: path_selector queue-length 0
> 
> (cc'ing Ben in case you need more multipath.conf help)

Re: 4.4-rc4 multipath not working?

[E V]

Ok stop and start of multipathd now with path_selector "queue-length
0" in defaults changes the map, but things still fail. Would a reboot
be in order, or just the start stop of multipath should clear it up?

mpathc: selector = queue-length 0 (config file default)
mpathc: features = 0 (config file default)
mpathc: hwhandler = 0 (internal default)
mpathc: rr_weight = 1 (internal default)
mpathc: minio = 1 rq (config file default)
mpathc: no_path_retry = 0 (internal default)
mpathc: fast_io_fail_tmo = 5 (config file default)
mpathc: retain_attached_hw_handler = 1 (config file default)
sdd: No SAS end device for 'end_device-1:0'
sdab: No SAS end device for 'end_device-1:1'
mpathc: remove queue_if_no_path from '0'
mpathc: assembled map [0 0 2 1 queue-length 0 1 1 8:48 1 queue-length
0 1 1 65:176 1]
mpathc: set ACT_CREATE (map does not exist)
mpathc: domap (0) failure for create/reload map
mpathc: ignoring map

On Mon, Dec 14, 2015 at 2:54 PM, Benjamin Marzinski <bmarzins@redhat.com> wrote:
> Mike is correct about the malformed table. looking at your ouput, I see
>
>> > mpathc: selector = queue_length (config file default)
>
> This means that you likely have
>
> path_selector "queue_length"
>
> in the defaults section of /etc/multipath.conf
>
> This should be
>
> path_selector "queue-length 0"
>
> With both a dash instead of an underscore, and a zero to let the kernel
> know that this selector takes no arguments.
>
> multipath does some sanity checking on the config values, but it
> definitely could be better.  For instance, on the values like this, that
> it simply passes to the kernel as a string, it doesn't do any.
>
> -Ben
>
> On Mon, Dec 14, 2015 at 01:51:18PM -0500, Mike Snitzer wrote:
>> On Mon, Dec 14 2015 at 11:55am -0500,
>> E V <eliventer@gmail.com> wrote:
>>
>> > On Fri, Dec 11, 2015 at 4:00 PM, Mike Snitzer <snitzer@redhat.com> wrote:
>> > > On Fri, Dec 11 2015 at 12:23pm -0500,
>> > > E V <eliventer@gmail.com> wrote:
>> > >
>> > >> Just tried booting 4.4-rc4 on a system with using the mptsas driver:
>> > >> LSISAS3008: FWVersion(03.00.08.00), ChipRevision(0x02), BiosVersion(05.07.02.00)
>> > >>
>> > >> And multipath doesn't seem to be working, dmesg has lots of these:
>> > >>
>> > >> device-mapper: table: 254:2: multipath: unknown path selector type
>> > >> device-mapper: ioctl: error adding target to table
>> > >
>> > > You compile the path selector module you're attempting to use?
>> > Yes as far as I can tell, here are the _DM_ config options in my config:
>> > CONFIG_BLK_DEV_DM_BUILTIN=y
>> > # CONFIG_DM_MQ_DEFAULT is not set
>> > # CONFIG_DM_DEBUG is not set
>> > # CONFIG_DM_CRYPT is not set
>> > # CONFIG_DM_SNAPSHOT is not set
>> > # CONFIG_DM_THIN_PROVISIONING is not set
>> > # CONFIG_DM_CACHE is not set
>> > # CONFIG_DM_ERA is not set
>> > # CONFIG_DM_MIRROR is not set
>> > CONFIG_DM_RAID=m
>> > # CONFIG_DM_ZERO is not set
>> > CONFIG_DM_MULTIPATH=y
>> > CONFIG_DM_MULTIPATH_QL=y
>> > # CONFIG_DM_MULTIPATH_ST is not set
>> > # CONFIG_DM_DELAY is not set
>> > CONFIG_DM_UEVENT=y
>> > # CONFIG_DM_FLAKEY is not set
>> > # CONFIG_DM_VERITY is not set
>> > # CONFIG_DM_SWITCH is not set
>> > # CONFIG_DM_LOG_WRITES is not set
>> >
>> >
>> > >> Is it expected to work? and any suggestions on getting it to work,
>> > >
>> > > Yes, no reason dm-multipath shouldn't work (other than user error).
>> > Entirely possible, this is my first multipath setup.
>> >
>> > > Figure out what the multipath table load line that failed to load
>> > > (verbose logging with multipathd?) and make sure you have the
>> > > corresponding path selector modules enabled in your .config
>> >
>> > Running multipathd with -v3 shows this in it's output, trimming 1st
>> > few columns for the 1st mpath device in the output. Doesn't mean much
>> > to me, other then I see the queue_length selector is set as expected:
>> > read /etc/multipath.conf
>> > multipath.conf line 8, duplicate keyword: defaults
>> > loading /lib/multipath/libcheckdirectio.so checker
>> > loading /lib/multipath/libprioconst.so prioritizer
>> > set open fds limit to 1048576/1048576
>> > sdab: ownership set to mpathc
>> > sdab: not found in pathvec
>> > sdab: mask = 0xc
>> > sdab: path state = running
>> > sdab: get_state
>> > directio: starting new request
>> > directio: io finished 4096/0
>> > sdab: state = up
>> > sdab: const prio = 1
>> > mpathc: pgfailback = -1 (config file default)
>> > mpathc: pgpolicy = failover (internal default)
>> > mpathc: selector = queue_length (config file default)
>> > mpathc: features = 0 (config file default)
>> > mpathc: hwhandler = 0 (internal default)
>> > mpathc: rr_weight = 1 (internal default)
>> > mpathc: minio = 1 rq (config file default)
>> > mpathc: no_path_retry = 0 (internal default)
>> > mpathc: fast_io_fail_tmo = 5 (config file default)
>> > mpathc: retain_attached_hw_handler = 1 (config file default)
>> > sdd: No SAS end device for 'end_device-1:0'
>> > sdab: No SAS end device for 'end_device-1:1'
>> > mpathc: remove queue_if_no_path from '0'
>> > mpathc: assembled map [0 0 2 1 queue_length 1 1 8:48 1 queue_length 1 1 65:176 1]
>>
>> This table looks malformed.
>>
>> Expected format is (from drivers/md/dm-mpath.c):
>>
>> /*-----------------------------------------------------------------
>>  * Constructor/argument parsing:
>>  * <#multipath feature args> [<arg>]*
>>  * <#hw_handler args> [hw_handler [<arg>]*]
>>  * <#priority groups>
>>  * <initial priority group>
>>  *     [<selector> <#selector args> [<arg>]*
>>  *      <#paths> <#per-path selector args>
>>  *         [<path> [<arg>]* ]+ ]+
>>  *---------------------------------------------------------------*/
>>
>> So in your case, breaking the map into logical sections, it is:
>>
>> 0 0 2 1
>>
>>   <#multipath feature args> = 0
>>   <#hw_handler args> = 0
>>   <#priority groups> = 2
>>   <initial priority group> = 1
>>
>> queue_length 1 1 8:48 1
>>
>>      <selector> = queue_length
>>      <#selector args> = 1 <BUT NO SELECTOR ARG!?>
>>      <#paths> = 1 <path> = 8:48 <arg> = 1
>>
>> queue_length 1 1 65:176 1
>>
>>      <selector> = queue_length <#selector args> = 1 <BUT NO SELECTOR ARG!?>
>>      <#paths> = 1 <path> = 65:176 <arg> = 1
>>
>> What do you have in your multipath.conf for 'path_selector'?
>>
>> AFAIK, it should be: path_selector queue-length 0
>>
>> (cc'ing Ben in case you need more multipath.conf help)

Re: 4.4-rc4 multipath not working?

[Benjamin Marzinski]

On Mon, Dec 14, 2015 at 03:09:50PM -0500, E V wrote:
> Ok stop and start of multipathd now with path_selector "queue-length
> 0" in defaults changes the map, but things still fail. Would a reboot
> be in order, or just the start stop of multipath should clear it up?

You don't need to reboot. You can see from the output that the change
has already taken effect.  Could you post the output of 

# tail /var/log/messages

right after you run the multipath command.  Most likely you will see
messages like

Dec 14 08:49:56 ask-07 kernel: device-mapper: table: 253:7: multipath:
error getting device
Dec 14 08:49:56 ask-07 kernel: device-mapper: ioctl: error adding target
to table

This usually means that one of the paths devices (In your case 8:48 or
65:176) or a partition of one of those devices is already in use.  It
could be a partition in use by another device-mapper device, or possibly
a filesystem is already mounted on the device. At any rate, if you free
up the device from whatever has exclusively grabbed it, multipath should
be able to set up on it.

-Ben

> 
> mpathc: selector = queue-length 0 (config file default)
> mpathc: features = 0 (config file default)
> mpathc: hwhandler = 0 (internal default)
> mpathc: rr_weight = 1 (internal default)
> mpathc: minio = 1 rq (config file default)
> mpathc: no_path_retry = 0 (internal default)
> mpathc: fast_io_fail_tmo = 5 (config file default)
> mpathc: retain_attached_hw_handler = 1 (config file default)
> sdd: No SAS end device for 'end_device-1:0'
> sdab: No SAS end device for 'end_device-1:1'
> mpathc: remove queue_if_no_path from '0'
> mpathc: assembled map [0 0 2 1 queue-length 0 1 1 8:48 1 queue-length
> 0 1 1 65:176 1]
> mpathc: set ACT_CREATE (map does not exist)
> mpathc: domap (0) failure for create/reload map
> mpathc: ignoring map
> 
> On Mon, Dec 14, 2015 at 2:54 PM, Benjamin Marzinski <bmarzins@redhat.com> wrote:
> > Mike is correct about the malformed table. looking at your ouput, I see
> >
> >> > mpathc: selector = queue_length (config file default)
> >
> > This means that you likely have
> >
> > path_selector "queue_length"
> >
> > in the defaults section of /etc/multipath.conf
> >
> > This should be
> >
> > path_selector "queue-length 0"
> >
> > With both a dash instead of an underscore, and a zero to let the kernel
> > know that this selector takes no arguments.
> >
> > multipath does some sanity checking on the config values, but it
> > definitely could be better.  For instance, on the values like this, that
> > it simply passes to the kernel as a string, it doesn't do any.
> >
> > -Ben
> >
> > On Mon, Dec 14, 2015 at 01:51:18PM -0500, Mike Snitzer wrote:
> >> On Mon, Dec 14 2015 at 11:55am -0500,
> >> E V <eliventer@gmail.com> wrote:
> >>
> >> > On Fri, Dec 11, 2015 at 4:00 PM, Mike Snitzer <snitzer@redhat.com> wrote:
> >> > > On Fri, Dec 11 2015 at 12:23pm -0500,
> >> > > E V <eliventer@gmail.com> wrote:
> >> > >
> >> > >> Just tried booting 4.4-rc4 on a system with using the mptsas driver:
> >> > >> LSISAS3008: FWVersion(03.00.08.00), ChipRevision(0x02), BiosVersion(05.07.02.00)
> >> > >>
> >> > >> And multipath doesn't seem to be working, dmesg has lots of these:
> >> > >>
> >> > >> device-mapper: table: 254:2: multipath: unknown path selector type
> >> > >> device-mapper: ioctl: error adding target to table
> >> > >
> >> > > You compile the path selector module you're attempting to use?
> >> > Yes as far as I can tell, here are the _DM_ config options in my config:
> >> > CONFIG_BLK_DEV_DM_BUILTIN=y
> >> > # CONFIG_DM_MQ_DEFAULT is not set
> >> > # CONFIG_DM_DEBUG is not set
> >> > # CONFIG_DM_CRYPT is not set
> >> > # CONFIG_DM_SNAPSHOT is not set
> >> > # CONFIG_DM_THIN_PROVISIONING is not set
> >> > # CONFIG_DM_CACHE is not set
> >> > # CONFIG_DM_ERA is not set
> >> > # CONFIG_DM_MIRROR is not set
> >> > CONFIG_DM_RAID=m
> >> > # CONFIG_DM_ZERO is not set
> >> > CONFIG_DM_MULTIPATH=y
> >> > CONFIG_DM_MULTIPATH_QL=y
> >> > # CONFIG_DM_MULTIPATH_ST is not set
> >> > # CONFIG_DM_DELAY is not set
> >> > CONFIG_DM_UEVENT=y
> >> > # CONFIG_DM_FLAKEY is not set
> >> > # CONFIG_DM_VERITY is not set
> >> > # CONFIG_DM_SWITCH is not set
> >> > # CONFIG_DM_LOG_WRITES is not set
> >> >
> >> >
> >> > >> Is it expected to work? and any suggestions on getting it to work,
> >> > >
> >> > > Yes, no reason dm-multipath shouldn't work (other than user error).
> >> > Entirely possible, this is my first multipath setup.
> >> >
> >> > > Figure out what the multipath table load line that failed to load
> >> > > (verbose logging with multipathd?) and make sure you have the
> >> > > corresponding path selector modules enabled in your .config
> >> >
> >> > Running multipathd with -v3 shows this in it's output, trimming 1st
> >> > few columns for the 1st mpath device in the output. Doesn't mean much
> >> > to me, other then I see the queue_length selector is set as expected:
> >> > read /etc/multipath.conf
> >> > multipath.conf line 8, duplicate keyword: defaults
> >> > loading /lib/multipath/libcheckdirectio.so checker
> >> > loading /lib/multipath/libprioconst.so prioritizer
> >> > set open fds limit to 1048576/1048576
> >> > sdab: ownership set to mpathc
> >> > sdab: not found in pathvec
> >> > sdab: mask = 0xc
> >> > sdab: path state = running
> >> > sdab: get_state
> >> > directio: starting new request
> >> > directio: io finished 4096/0
> >> > sdab: state = up
> >> > sdab: const prio = 1
> >> > mpathc: pgfailback = -1 (config file default)
> >> > mpathc: pgpolicy = failover (internal default)
> >> > mpathc: selector = queue_length (config file default)
> >> > mpathc: features = 0 (config file default)
> >> > mpathc: hwhandler = 0 (internal default)
> >> > mpathc: rr_weight = 1 (internal default)
> >> > mpathc: minio = 1 rq (config file default)
> >> > mpathc: no_path_retry = 0 (internal default)
> >> > mpathc: fast_io_fail_tmo = 5 (config file default)
> >> > mpathc: retain_attached_hw_handler = 1 (config file default)
> >> > sdd: No SAS end device for 'end_device-1:0'
> >> > sdab: No SAS end device for 'end_device-1:1'
> >> > mpathc: remove queue_if_no_path from '0'
> >> > mpathc: assembled map [0 0 2 1 queue_length 1 1 8:48 1 queue_length 1 1 65:176 1]
> >>
> >> This table looks malformed.
> >>
> >> Expected format is (from drivers/md/dm-mpath.c):
> >>
> >> /*-----------------------------------------------------------------
> >>  * Constructor/argument parsing:
> >>  * <#multipath feature args> [<arg>]*
> >>  * <#hw_handler args> [hw_handler [<arg>]*]
> >>  * <#priority groups>
> >>  * <initial priority group>
> >>  *     [<selector> <#selector args> [<arg>]*
> >>  *      <#paths> <#per-path selector args>
> >>  *         [<path> [<arg>]* ]+ ]+
> >>  *---------------------------------------------------------------*/
> >>
> >> So in your case, breaking the map into logical sections, it is:
> >>
> >> 0 0 2 1
> >>
> >>   <#multipath feature args> = 0
> >>   <#hw_handler args> = 0
> >>   <#priority groups> = 2
> >>   <initial priority group> = 1
> >>
> >> queue_length 1 1 8:48 1
> >>
> >>      <selector> = queue_length
> >>      <#selector args> = 1 <BUT NO SELECTOR ARG!?>
> >>      <#paths> = 1 <path> = 8:48 <arg> = 1
> >>
> >> queue_length 1 1 65:176 1
> >>
> >>      <selector> = queue_length <#selector args> = 1 <BUT NO SELECTOR ARG!?>
> >>      <#paths> = 1 <path> = 65:176 <arg> = 1
> >>
> >> What do you have in your multipath.conf for 'path_selector'?
> >>
> >> AFAIK, it should be: path_selector queue-length 0
> >>
> >> (cc'ing Ben in case you need more multipath.conf help)

Re: 4.4-rc4 multipath not working?

[E V]

Ok, makes sense. Looks like mdadm auto assembled my devices on boot
without the dm-? devices. So mdadm -S freed up the devices and then
restarting multipath tools, and mdadm -A /dev/md/0 /dev/dm-5+...
worked perfect. Guess I just need to figure out how to make the mdadm
assemble take place after multipathd is up on boot. Thanks a lot.

On Mon, Dec 14, 2015 at 3:36 PM, Benjamin Marzinski <bmarzins@redhat.com> wrote:
> On Mon, Dec 14, 2015 at 03:09:50PM -0500, E V wrote:
>> Ok stop and start of multipathd now with path_selector "queue-length
>> 0" in defaults changes the map, but things still fail. Would a reboot
>> be in order, or just the start stop of multipath should clear it up?
>
> You don't need to reboot. You can see from the output that the change
> has already taken effect.  Could you post the output of
>
> # tail /var/log/messages
>
> right after you run the multipath command.  Most likely you will see
> messages like
>
> Dec 14 08:49:56 ask-07 kernel: device-mapper: table: 253:7: multipath:
> error getting device
> Dec 14 08:49:56 ask-07 kernel: device-mapper: ioctl: error adding target
> to table
>
> This usually means that one of the paths devices (In your case 8:48 or
> 65:176) or a partition of one of those devices is already in use.  It
> could be a partition in use by another device-mapper device, or possibly
> a filesystem is already mounted on the device. At any rate, if you free
> up the device from whatever has exclusively grabbed it, multipath should
> be able to set up on it.
>
> -Ben
>
>>
>> mpathc: selector = queue-length 0 (config file default)
>> mpathc: features = 0 (config file default)
>> mpathc: hwhandler = 0 (internal default)
>> mpathc: rr_weight = 1 (internal default)
>> mpathc: minio = 1 rq (config file default)
>> mpathc: no_path_retry = 0 (internal default)
>> mpathc: fast_io_fail_tmo = 5 (config file default)
>> mpathc: retain_attached_hw_handler = 1 (config file default)
>> sdd: No SAS end device for 'end_device-1:0'
>> sdab: No SAS end device for 'end_device-1:1'
>> mpathc: remove queue_if_no_path from '0'
>> mpathc: assembled map [0 0 2 1 queue-length 0 1 1 8:48 1 queue-length
>> 0 1 1 65:176 1]
>> mpathc: set ACT_CREATE (map does not exist)
>> mpathc: domap (0) failure for create/reload map
>> mpathc: ignoring map
>>
>> On Mon, Dec 14, 2015 at 2:54 PM, Benjamin Marzinski <bmarzins@redhat.com> wrote:
>> > Mike is correct about the malformed table. looking at your ouput, I see
>> >
>> >> > mpathc: selector = queue_length (config file default)
>> >
>> > This means that you likely have
>> >
>> > path_selector "queue_length"
>> >
>> > in the defaults section of /etc/multipath.conf
>> >
>> > This should be
>> >
>> > path_selector "queue-length 0"
>> >
>> > With both a dash instead of an underscore, and a zero to let the kernel
>> > know that this selector takes no arguments.
>> >
>> > multipath does some sanity checking on the config values, but it
>> > definitely could be better.  For instance, on the values like this, that
>> > it simply passes to the kernel as a string, it doesn't do any.
>> >
>> > -Ben
>> >
>> > On Mon, Dec 14, 2015 at 01:51:18PM -0500, Mike Snitzer wrote:
>> >> On Mon, Dec 14 2015 at 11:55am -0500,
>> >> E V <eliventer@gmail.com> wrote:
>> >>
>> >> > On Fri, Dec 11, 2015 at 4:00 PM, Mike Snitzer <snitzer@redhat.com> wrote:
>> >> > > On Fri, Dec 11 2015 at 12:23pm -0500,
>> >> > > E V <eliventer@gmail.com> wrote:
>> >> > >
>> >> > >> Just tried booting 4.4-rc4 on a system with using the mptsas driver:
>> >> > >> LSISAS3008: FWVersion(03.00.08.00), ChipRevision(0x02), BiosVersion(05.07.02.00)
>> >> > >>
>> >> > >> And multipath doesn't seem to be working, dmesg has lots of these:
>> >> > >>
>> >> > >> device-mapper: table: 254:2: multipath: unknown path selector type
>> >> > >> device-mapper: ioctl: error adding target to table
>> >> > >
>> >> > > You compile the path selector module you're attempting to use?
>> >> > Yes as far as I can tell, here are the _DM_ config options in my config:
>> >> > CONFIG_BLK_DEV_DM_BUILTIN=y
>> >> > # CONFIG_DM_MQ_DEFAULT is not set
>> >> > # CONFIG_DM_DEBUG is not set
>> >> > # CONFIG_DM_CRYPT is not set
>> >> > # CONFIG_DM_SNAPSHOT is not set
>> >> > # CONFIG_DM_THIN_PROVISIONING is not set
>> >> > # CONFIG_DM_CACHE is not set
>> >> > # CONFIG_DM_ERA is not set
>> >> > # CONFIG_DM_MIRROR is not set
>> >> > CONFIG_DM_RAID=m
>> >> > # CONFIG_DM_ZERO is not set
>> >> > CONFIG_DM_MULTIPATH=y
>> >> > CONFIG_DM_MULTIPATH_QL=y
>> >> > # CONFIG_DM_MULTIPATH_ST is not set
>> >> > # CONFIG_DM_DELAY is not set
>> >> > CONFIG_DM_UEVENT=y
>> >> > # CONFIG_DM_FLAKEY is not set
>> >> > # CONFIG_DM_VERITY is not set
>> >> > # CONFIG_DM_SWITCH is not set
>> >> > # CONFIG_DM_LOG_WRITES is not set
>> >> >
>> >> >
>> >> > >> Is it expected to work? and any suggestions on getting it to work,
>> >> > >
>> >> > > Yes, no reason dm-multipath shouldn't work (other than user error).
>> >> > Entirely possible, this is my first multipath setup.
>> >> >
>> >> > > Figure out what the multipath table load line that failed to load
>> >> > > (verbose logging with multipathd?) and make sure you have the
>> >> > > corresponding path selector modules enabled in your .config
>> >> >
>> >> > Running multipathd with -v3 shows this in it's output, trimming 1st
>> >> > few columns for the 1st mpath device in the output. Doesn't mean much
>> >> > to me, other then I see the queue_length selector is set as expected:
>> >> > read /etc/multipath.conf
>> >> > multipath.conf line 8, duplicate keyword: defaults
>> >> > loading /lib/multipath/libcheckdirectio.so checker
>> >> > loading /lib/multipath/libprioconst.so prioritizer
>> >> > set open fds limit to 1048576/1048576
>> >> > sdab: ownership set to mpathc
>> >> > sdab: not found in pathvec
>> >> > sdab: mask = 0xc
>> >> > sdab: path state = running
>> >> > sdab: get_state
>> >> > directio: starting new request
>> >> > directio: io finished 4096/0
>> >> > sdab: state = up
>> >> > sdab: const prio = 1
>> >> > mpathc: pgfailback = -1 (config file default)
>> >> > mpathc: pgpolicy = failover (internal default)
>> >> > mpathc: selector = queue_length (config file default)
>> >> > mpathc: features = 0 (config file default)
>> >> > mpathc: hwhandler = 0 (internal default)
>> >> > mpathc: rr_weight = 1 (internal default)
>> >> > mpathc: minio = 1 rq (config file default)
>> >> > mpathc: no_path_retry = 0 (internal default)
>> >> > mpathc: fast_io_fail_tmo = 5 (config file default)
>> >> > mpathc: retain_attached_hw_handler = 1 (config file default)
>> >> > sdd: No SAS end device for 'end_device-1:0'
>> >> > sdab: No SAS end device for 'end_device-1:1'
>> >> > mpathc: remove queue_if_no_path from '0'
>> >> > mpathc: assembled map [0 0 2 1 queue_length 1 1 8:48 1 queue_length 1 1 65:176 1]
>> >>
>> >> This table looks malformed.
>> >>
>> >> Expected format is (from drivers/md/dm-mpath.c):
>> >>
>> >> /*-----------------------------------------------------------------
>> >>  * Constructor/argument parsing:
>> >>  * <#multipath feature args> [<arg>]*
>> >>  * <#hw_handler args> [hw_handler [<arg>]*]
>> >>  * <#priority groups>
>> >>  * <initial priority group>
>> >>  *     [<selector> <#selector args> [<arg>]*
>> >>  *      <#paths> <#per-path selector args>
>> >>  *         [<path> [<arg>]* ]+ ]+
>> >>  *---------------------------------------------------------------*/
>> >>
>> >> So in your case, breaking the map into logical sections, it is:
>> >>
>> >> 0 0 2 1
>> >>
>> >>   <#multipath feature args> = 0
>> >>   <#hw_handler args> = 0
>> >>   <#priority groups> = 2
>> >>   <initial priority group> = 1
>> >>
>> >> queue_length 1 1 8:48 1
>> >>
>> >>      <selector> = queue_length
>> >>      <#selector args> = 1 <BUT NO SELECTOR ARG!?>
>> >>      <#paths> = 1 <path> = 8:48 <arg> = 1
>> >>
>> >> queue_length 1 1 65:176 1
>> >>
>> >>      <selector> = queue_length <#selector args> = 1 <BUT NO SELECTOR ARG!?>
>> >>      <#paths> = 1 <path> = 65:176 <arg> = 1
>> >>
>> >> What do you have in your multipath.conf for 'path_selector'?
>> >>
>> >> AFAIK, it should be: path_selector queue-length 0
>> >>
>> >> (cc'ing Ben in case you need more multipath.conf help)

Re: 4.4-rc4 multipath not working?

[Benjamin Marzinski]

On Mon, Dec 14, 2015 at 03:57:11PM -0500, E V wrote:
> Ok, makes sense. Looks like mdadm auto assembled my devices on boot
> without the dm-? devices. So mdadm -S freed up the devices and then
> restarting multipath tools, and mdadm -A /dev/md/0 /dev/dm-5+...
> worked perfect. Guess I just need to figure out how to make the mdadm
> assemble take place after multipathd is up on boot. Thanks a lot.

Once multipath has sucessfully grabbed a device, it will add it's wwid
to /etc/multipath/wwids.  On future boots, as soon as a scsi device is
discovered, udev rules will check if the device belongs to multipath. If
the wwid is in /etc/multipath/wwids a flag will be set in the udev
database that should cause dm and md to ignore the path devices, and use
the multipath device itself (at least on RHEL, things work slightly
differently on different distros).

One thing to note is that if md is assembling the device in the
initramfs (I'm not sure that it ever does this for non-boot devices, but
it could) or if it is grabbing the device before multipathd starts back
up in late boot, you may need to remake your initramfs to update
/etc/multipath/wwids there as well as in the regular filesystem. Again,
this is referring to RHEL. On another distro YMMV.

-Ben

> 
> On Mon, Dec 14, 2015 at 3:36 PM, Benjamin Marzinski <bmarzins@redhat.com> wrote:
> > On Mon, Dec 14, 2015 at 03:09:50PM -0500, E V wrote:
> >> Ok stop and start of multipathd now with path_selector "queue-length
> >> 0" in defaults changes the map, but things still fail. Would a reboot
> >> be in order, or just the start stop of multipath should clear it up?
> >
> > You don't need to reboot. You can see from the output that the change
> > has already taken effect.  Could you post the output of
> >
> > # tail /var/log/messages
> >
> > right after you run the multipath command.  Most likely you will see
> > messages like
> >
> > Dec 14 08:49:56 ask-07 kernel: device-mapper: table: 253:7: multipath:
> > error getting device
> > Dec 14 08:49:56 ask-07 kernel: device-mapper: ioctl: error adding target
> > to table
> >
> > This usually means that one of the paths devices (In your case 8:48 or
> > 65:176) or a partition of one of those devices is already in use.  It
> > could be a partition in use by another device-mapper device, or possibly
> > a filesystem is already mounted on the device. At any rate, if you free
> > up the device from whatever has exclusively grabbed it, multipath should
> > be able to set up on it.
> >
> > -Ben
> >
> >>
> >> mpathc: selector = queue-length 0 (config file default)
> >> mpathc: features = 0 (config file default)
> >> mpathc: hwhandler = 0 (internal default)
> >> mpathc: rr_weight = 1 (internal default)
> >> mpathc: minio = 1 rq (config file default)
> >> mpathc: no_path_retry = 0 (internal default)
> >> mpathc: fast_io_fail_tmo = 5 (config file default)
> >> mpathc: retain_attached_hw_handler = 1 (config file default)
> >> sdd: No SAS end device for 'end_device-1:0'
> >> sdab: No SAS end device for 'end_device-1:1'
> >> mpathc: remove queue_if_no_path from '0'
> >> mpathc: assembled map [0 0 2 1 queue-length 0 1 1 8:48 1 queue-length
> >> 0 1 1 65:176 1]
> >> mpathc: set ACT_CREATE (map does not exist)
> >> mpathc: domap (0) failure for create/reload map
> >> mpathc: ignoring map
> >>
> >> On Mon, Dec 14, 2015 at 2:54 PM, Benjamin Marzinski <bmarzins@redhat.com> wrote:
> >> > Mike is correct about the malformed table. looking at your ouput, I see
> >> >
> >> >> > mpathc: selector = queue_length (config file default)
> >> >
> >> > This means that you likely have
> >> >
> >> > path_selector "queue_length"
> >> >
> >> > in the defaults section of /etc/multipath.conf
> >> >
> >> > This should be
> >> >
> >> > path_selector "queue-length 0"
> >> >
> >> > With both a dash instead of an underscore, and a zero to let the kernel
> >> > know that this selector takes no arguments.
> >> >
> >> > multipath does some sanity checking on the config values, but it
> >> > definitely could be better.  For instance, on the values like this, that
> >> > it simply passes to the kernel as a string, it doesn't do any.
> >> >
> >> > -Ben
> >> >
> >> > On Mon, Dec 14, 2015 at 01:51:18PM -0500, Mike Snitzer wrote:
> >> >> On Mon, Dec 14 2015 at 11:55am -0500,
> >> >> E V <eliventer@gmail.com> wrote:
> >> >>
> >> >> > On Fri, Dec 11, 2015 at 4:00 PM, Mike Snitzer <snitzer@redhat.com> wrote:
> >> >> > > On Fri, Dec 11 2015 at 12:23pm -0500,
> >> >> > > E V <eliventer@gmail.com> wrote:
> >> >> > >
> >> >> > >> Just tried booting 4.4-rc4 on a system with using the mptsas driver:
> >> >> > >> LSISAS3008: FWVersion(03.00.08.00), ChipRevision(0x02), BiosVersion(05.07.02.00)
> >> >> > >>
> >> >> > >> And multipath doesn't seem to be working, dmesg has lots of these:
> >> >> > >>
> >> >> > >> device-mapper: table: 254:2: multipath: unknown path selector type
> >> >> > >> device-mapper: ioctl: error adding target to table
> >> >> > >
> >> >> > > You compile the path selector module you're attempting to use?
> >> >> > Yes as far as I can tell, here are the _DM_ config options in my config:
> >> >> > CONFIG_BLK_DEV_DM_BUILTIN=y
> >> >> > # CONFIG_DM_MQ_DEFAULT is not set
> >> >> > # CONFIG_DM_DEBUG is not set
> >> >> > # CONFIG_DM_CRYPT is not set
> >> >> > # CONFIG_DM_SNAPSHOT is not set
> >> >> > # CONFIG_DM_THIN_PROVISIONING is not set
> >> >> > # CONFIG_DM_CACHE is not set
> >> >> > # CONFIG_DM_ERA is not set
> >> >> > # CONFIG_DM_MIRROR is not set
> >> >> > CONFIG_DM_RAID=m
> >> >> > # CONFIG_DM_ZERO is not set
> >> >> > CONFIG_DM_MULTIPATH=y
> >> >> > CONFIG_DM_MULTIPATH_QL=y
> >> >> > # CONFIG_DM_MULTIPATH_ST is not set
> >> >> > # CONFIG_DM_DELAY is not set
> >> >> > CONFIG_DM_UEVENT=y
> >> >> > # CONFIG_DM_FLAKEY is not set
> >> >> > # CONFIG_DM_VERITY is not set
> >> >> > # CONFIG_DM_SWITCH is not set
> >> >> > # CONFIG_DM_LOG_WRITES is not set
> >> >> >
> >> >> >
> >> >> > >> Is it expected to work? and any suggestions on getting it to work,
> >> >> > >
> >> >> > > Yes, no reason dm-multipath shouldn't work (other than user error).
> >> >> > Entirely possible, this is my first multipath setup.
> >> >> >
> >> >> > > Figure out what the multipath table load line that failed to load
> >> >> > > (verbose logging with multipathd?) and make sure you have the
> >> >> > > corresponding path selector modules enabled in your .config
> >> >> >
> >> >> > Running multipathd with -v3 shows this in it's output, trimming 1st
> >> >> > few columns for the 1st mpath device in the output. Doesn't mean much
> >> >> > to me, other then I see the queue_length selector is set as expected:
> >> >> > read /etc/multipath.conf
> >> >> > multipath.conf line 8, duplicate keyword: defaults
> >> >> > loading /lib/multipath/libcheckdirectio.so checker
> >> >> > loading /lib/multipath/libprioconst.so prioritizer
> >> >> > set open fds limit to 1048576/1048576
> >> >> > sdab: ownership set to mpathc
> >> >> > sdab: not found in pathvec
> >> >> > sdab: mask = 0xc
> >> >> > sdab: path state = running
> >> >> > sdab: get_state
> >> >> > directio: starting new request
> >> >> > directio: io finished 4096/0
> >> >> > sdab: state = up
> >> >> > sdab: const prio = 1
> >> >> > mpathc: pgfailback = -1 (config file default)
> >> >> > mpathc: pgpolicy = failover (internal default)
> >> >> > mpathc: selector = queue_length (config file default)
> >> >> > mpathc: features = 0 (config file default)
> >> >> > mpathc: hwhandler = 0 (internal default)
> >> >> > mpathc: rr_weight = 1 (internal default)
> >> >> > mpathc: minio = 1 rq (config file default)
> >> >> > mpathc: no_path_retry = 0 (internal default)
> >> >> > mpathc: fast_io_fail_tmo = 5 (config file default)
> >> >> > mpathc: retain_attached_hw_handler = 1 (config file default)
> >> >> > sdd: No SAS end device for 'end_device-1:0'
> >> >> > sdab: No SAS end device for 'end_device-1:1'
> >> >> > mpathc: remove queue_if_no_path from '0'
> >> >> > mpathc: assembled map [0 0 2 1 queue_length 1 1 8:48 1 queue_length 1 1 65:176 1]
> >> >>
> >> >> This table looks malformed.
> >> >>
> >> >> Expected format is (from drivers/md/dm-mpath.c):
> >> >>
> >> >> /*-----------------------------------------------------------------
> >> >>  * Constructor/argument parsing:
> >> >>  * <#multipath feature args> [<arg>]*
> >> >>  * <#hw_handler args> [hw_handler [<arg>]*]
> >> >>  * <#priority groups>
> >> >>  * <initial priority group>
> >> >>  *     [<selector> <#selector args> [<arg>]*
> >> >>  *      <#paths> <#per-path selector args>
> >> >>  *         [<path> [<arg>]* ]+ ]+
> >> >>  *---------------------------------------------------------------*/
> >> >>
> >> >> So in your case, breaking the map into logical sections, it is:
> >> >>
> >> >> 0 0 2 1
> >> >>
> >> >>   <#multipath feature args> = 0
> >> >>   <#hw_handler args> = 0
> >> >>   <#priority groups> = 2
> >> >>   <initial priority group> = 1
> >> >>
> >> >> queue_length 1 1 8:48 1
> >> >>
> >> >>      <selector> = queue_length
> >> >>      <#selector args> = 1 <BUT NO SELECTOR ARG!?>
> >> >>      <#paths> = 1 <path> = 8:48 <arg> = 1
> >> >>
> >> >> queue_length 1 1 65:176 1
> >> >>
> >> >>      <selector> = queue_length <#selector args> = 1 <BUT NO SELECTOR ARG!?>
> >> >>      <#paths> = 1 <path> = 65:176 <arg> = 1
> >> >>
> >> >> What do you have in your multipath.conf for 'path_selector'?
> >> >>
> >> >> AFAIK, it should be: path_selector queue-length 0
> >> >>
> >> >> (cc'ing Ben in case you need more multipath.conf help)