User specific priorities on for iscsi paths doesn't seem to work

User specific priorities on for iscsi paths doesn't seem to work

[Akshay Lal]

I'm having a few issues with path priorities. It seems that the choice 
of path to use during I/O is independent of the user defined priorities 
for each path.

I am setting the priorities by executing writing a script that is used 
by prio_callout. This seems to work when I execute multipath -ll since 
all the specified priorities show up correctly. (the 
path_grouping_policy being used is  failover)

multipath -ll:
--------------
mpath2 (244534e3833623961) dm-1 DSNET,Dispersed Store
[size=47G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=10][enabled]
 \_ 85:0:0:0 sdc 8:32  [active][ready]
\_ round-robin 0 [prio=5][enabled]
 \_ 87:0:0:0 sde 8:64  [active][ready]
mpath1 (244534e3266616134) dm-0 DSNET,Dispersed Store
[size=47G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=5][enabled]
 \_ 84:0:0:0 sdb 8:16  [active][ready]
\_ round-robin 0 [prio=10][enabled]
 \_ 86:0:0:0 sdd 8:48  [active][ready]

At this point when I dd to /dev/mapper/mpath1 or /dev/mapper/mpath2 the 
priorities don't come into play at all. Data starts going to /dev/sdc 
for /dev/mapper/mpath2 & /dev/sdb for /dev/mapper/mpath1, which totally 
by-passes the set priorities. Below is a list of all the conf files and 
scripts I'm using.



/etc/multipath.conf
--------------------
defaults {
        udev_dir                /dev
        polling_interval        1
        selector                "round-robin 0"
        path_grouping_policy    multibus
        getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
        prio_callout            "/bin/bash 
/root/MultipathScripts/mpath_prio_alt %n"
        path_checker            tur
        rr_min_io               128
        max_fds                 8192
        rr_weight               priorities
        failback                immediate
        no_path_retry           queue
        user_friendly_names     yes
}


/root/MultipathScripts/mpath_prio_alt :
---------------------------------------
#!/bin/bash
# if not passed any device name, return a priority of 0
if [ -z "${1}" ];then
        echo 0
        exit
fi

ENTRY="$(ls -l1 /dev/disk/by-path | grep /${1}'$')"
DEV="${ENTRY##* ip-}"
if [ "$DEV" = "${ENTRY}" ];then #This is not an iSCSI device
        echo 0
        exit
fi

# stripping $ENTRY to obtain the IP:Port-iscsi-iqn-lunNumber
DEV="${ENTRY##* ip-}"
#DEV="${DEV%% ->*}" # the pattern changed in CentOS 5.3
#DEV="$(echo ${DEV} | sed 's/-lun-[0-9][0-9]* ->.*//')"
DEV="${DEV%%-lun-[0-9]* ->*}"

echo $(grep ${DEV} /root/MultipathScripts/iscsi.list | cut -f2)


/root/MultipathScripts/iscsi.list:
----------------------------------
192.xxx.xxx.103:3260-iscsi-iqn.2008-07.com.company:stoarge-2    1
192.xxx.xxx.106:3260-iscsi-iqn.2008-07.com.company:stoarge-2    0
192.xxx.xxx.103:3260-iscsi-iqn.2008-07.com.company:stoarge-1    0
192.xxx.xxx.106:3260-iscsi-iqn.2008-07.com.company:stoarge-1    1


Is there something I'm doing wrong? I would like to be able to define 
the priorities per device, and ensure that data only traverses on the 
lower priority path when
a) a failure to the first path (path with a higher priority) occures
b) no other path with a higher priority exists

--
Akshay Lal

Re: User specific priorities on for iscsi paths doesn't seem to work

[Mike Snitzer]

On Fri, Aug 07 2009 at  4:25pm -0400,
Akshay Lal <alal@cleversafe.com> wrote:

> I'm having a few issues with path priorities. It seems that the choice  
> of path to use during I/O is independent of the user defined priorities  
> for each path.
>
> I am setting the priorities by executing writing a script that is used  
> by prio_callout. This seems to work when I execute multipath -ll since  
> all the specified priorities show up correctly. (the  
> path_grouping_policy being used is  failover)

...

> Is there something I'm doing wrong? I would like to be able to define  
> the priorities per device, and ensure that data only traverses on the  
> lower priority path when
> a) a failure to the first path (path with a higher priority) occures
> b) no other path with a higher priority exists

Do things behave as you'd like if you change path_grouping_policy to
'group_by_prio'?

Mike

Re: User specific priorities on for iscsi paths doesn't seem to work

[Akshay Lal]

Mike:

It seems that if I were to set the path_grouping_policy to 
"group_by_prio", then it seems to be working similar to a multibus 
configuration. What I would like is for within a single multipath group, 
(say mpath1), specify a primary path and an alternate/failover path. If 
I can make this configurable via user land tool that'd be great. In this 
vein, I had considered priorities with hope that if I can set the 
priority of a certain path within a group then the path with the highest 
priority will always be chosen and the other path (with the lower 
priority) will only come into play when the primary goes down.

Below is the output of the multipath -ll & conf file when setting the 
path_grouping_policy to group_by_prio.


multipath -ll:
--------------
mpath2 (244534e3833623961) dm-1 DSNET,Dispersed Store
[size=47G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=15][enabled]
 \_ 85:0:0:0 sdc 8:32  [active][ready]
 \_ 87:0:0:0 sde 8:64  [active][ready]
mpath1 (244534e3266616134) dm-0 DSNET,Dispersed Store
[size=47G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=15][active]
 \_ 84:0:0:0 sdb 8:16  [active][ready]
 \_ 86:0:0:0 sdd 8:48  [active][ready]


/etc/multipath.conf:
--------------------
defaults {
        udev_dir                /dev
        polling_interval        1
        selector                "round-robin 0"
        path_grouping_policy    group_by_prio
        getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
        prio_callout            "/bin/bash 
/root/MultipathScripts/mpath_prio_alt %n"
        path_checker            tur
        rr_min_io               128
        max_fds                 8192
        rr_weight               priorities
        failback                immediate
        no_path_retry           queue
        user_friendly_names     yes
}


Mike Snitzer wrote:
> On Fri, Aug 07 2009 at  4:25pm -0400,
> Akshay Lal <alal@cleversafe.com> wrote:
>
>   
>> I'm having a few issues with path priorities. It seems that the choice  
>> of path to use during I/O is independent of the user defined priorities  
>> for each path.
>>
>> I am setting the priorities by executing writing a script that is used  
>> by prio_callout. This seems to work when I execute multipath -ll since  
>> all the specified priorities show up correctly. (the  
>> path_grouping_policy being used is  failover)
>>     
>
> ...
>
>   
>> Is there something I'm doing wrong? I would like to be able to define  
>> the priorities per device, and ensure that data only traverses on the  
>> lower priority path when
>> a) a failure to the first path (path with a higher priority) occures
>> b) no other path with a higher priority exists
>>     
>
> Do things behave as you'd like if you change path_grouping_policy to
> 'group_by_prio'?
>
> Mike
>   

Re: User specific priorities on for iscsi paths doesn't seem to work

[Mike Snitzer]

On Fri, Aug 07 2009 at  5:07pm -0400,
Akshay Lal <alal@cleversafe.com> wrote:

> Mike Snitzer wrote:
>> On Fri, Aug 07 2009 at  4:25pm -0400,
>> Akshay Lal <alal@cleversafe.com> wrote:
>>
>>   
>>> I'm having a few issues with path priorities. It seems that the 
>>> choice  of path to use during I/O is independent of the user defined 
>>> priorities  for each path.
>>>
>>> I am setting the priorities by executing writing a script that is 
>>> used  by prio_callout. This seems to work when I execute multipath 
>>> -ll since  all the specified priorities show up correctly. (the   
>>> path_grouping_policy being used is  failover)
>>>     
>>
>> ...
>>
>>   
>>> Is there something I'm doing wrong? I would like to be able to define 
>>>  the priorities per device, and ensure that data only traverses on 
>>> the  lower priority path when
>>> a) a failure to the first path (path with a higher priority) occures
>>> b) no other path with a higher priority exists
>>>     
>>
>> Do things behave as you'd like if you change path_grouping_policy to
>> 'group_by_prio'?
>>
>> Mike
>>   


> Mike:
>
> It seems that if I were to set the path_grouping_policy to  
> "group_by_prio", then it seems to be working similar to a multibus  
> configuration. What I would like is for within a single multipath group,  
> (say mpath1), specify a primary path and an alternate/failover path. If  
> I can make this configurable via user land tool that'd be great. In this  
> vein, I had considered priorities with hope that if I can set the  
> priority of a certain path within a group then the path with the highest  
> priority will always be chosen and the other path (with the lower  
> priority) will only come into play when the primary goes down.
>
> Below is the output of the multipath -ll & conf file when setting the  
> path_grouping_policy to group_by_prio.
>
>
> multipath -ll:
> --------------
> mpath2 (244534e3833623961) dm-1 DSNET,Dispersed Store
> [size=47G][features=1 queue_if_no_path][hwhandler=0][rw]
> \_ round-robin 0 [prio=15][enabled]
> \_ 85:0:0:0 sdc 8:32  [active][ready]
> \_ 87:0:0:0 sde 8:64  [active][ready]
> mpath1 (244534e3266616134) dm-0 DSNET,Dispersed Store
> [size=47G][features=1 queue_if_no_path][hwhandler=0][rw]
> \_ round-robin 0 [prio=15][active]
> \_ 84:0:0:0 sdb 8:16  [active][ready]
> \_ 86:0:0:0 sdd 8:48  [active][ready]
>
>
> /etc/multipath.conf:
> --------------------
> defaults {
>        udev_dir                /dev
>        polling_interval        1
>        selector                "round-robin 0"
>        path_grouping_policy    group_by_prio
>        getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
>        prio_callout            "/bin/bash  
> /root/MultipathScripts/mpath_prio_alt %n"
>        path_checker            tur
>        rr_min_io               128
>        max_fds                 8192
>        rr_weight               priorities
>        failback                immediate
>        no_path_retry           queue
>        user_friendly_names     yes
> }


Please don't top-post.

I'm pretty sure John meant to say "group_by_prio" rather than "failover"
in his initial reply to this thread.  John originally got this insight
(dummy device section et. al. applies to RHEL 5.3) back in April:

https://www.redhat.com/archives/dm-devel/2009-April/msg00157.html

Which multipath/distro are you using?

Mike

Re: User specific priorities on for iscsi paths doesn't seem to work

[Akshay Lal]

Mike Snitzer wrote:
> On Fri, Aug 07 2009 at  5:07pm -0400,
> Akshay Lal <alal@cleversafe.com> wrote:
>
>   
>> Mike Snitzer wrote:
>>     
>>> On Fri, Aug 07 2009 at  4:25pm -0400,
>>> Akshay Lal <alal@cleversafe.com> wrote:
>>>
>>>   
>>>       
>>>> I'm having a few issues with path priorities. It seems that the 
>>>> choice  of path to use during I/O is independent of the user defined 
>>>> priorities  for each path.
>>>>
>>>> I am setting the priorities by executing writing a script that is 
>>>> used  by prio_callout. This seems to work when I execute multipath 
>>>> -ll since  all the specified priorities show up correctly. (the   
>>>> path_grouping_policy being used is  failover)
>>>>     
>>>>         
>>> ...
>>>
>>>   
>>>       
>>>> Is there something I'm doing wrong? I would like to be able to define 
>>>>  the priorities per device, and ensure that data only traverses on 
>>>> the  lower priority path when
>>>> a) a failure to the first path (path with a higher priority) occures
>>>> b) no other path with a higher priority exists
>>>>     
>>>>         
>>> Do things behave as you'd like if you change path_grouping_policy to
>>> 'group_by_prio'?
>>>
>>> Mike
>>>   
>>>       
>
>
>   
>> Mike:
>>
>> It seems that if I were to set the path_grouping_policy to  
>> "group_by_prio", then it seems to be working similar to a multibus  
>> configuration. What I would like is for within a single multipath group,  
>> (say mpath1), specify a primary path and an alternate/failover path. If  
>> I can make this configurable via user land tool that'd be great. In this  
>> vein, I had considered priorities with hope that if I can set the  
>> priority of a certain path within a group then the path with the highest  
>> priority will always be chosen and the other path (with the lower  
>> priority) will only come into play when the primary goes down.
>>
>> Below is the output of the multipath -ll & conf file when setting the  
>> path_grouping_policy to group_by_prio.
>>
>>
>> multipath -ll:
>> --------------
>> mpath2 (244534e3833623961) dm-1 DSNET,Dispersed Store
>> [size=47G][features=1 queue_if_no_path][hwhandler=0][rw]
>> \_ round-robin 0 [prio=15][enabled]
>> \_ 85:0:0:0 sdc 8:32  [active][ready]
>> \_ 87:0:0:0 sde 8:64  [active][ready]
>> mpath1 (244534e3266616134) dm-0 DSNET,Dispersed Store
>> [size=47G][features=1 queue_if_no_path][hwhandler=0][rw]
>> \_ round-robin 0 [prio=15][active]
>> \_ 84:0:0:0 sdb 8:16  [active][ready]
>> \_ 86:0:0:0 sdd 8:48  [active][ready]
>>
>>
>> /etc/multipath.conf:
>> --------------------
>> defaults {
>>        udev_dir                /dev
>>        polling_interval        1
>>        selector                "round-robin 0"
>>        path_grouping_policy    group_by_prio
>>        getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
>>        prio_callout            "/bin/bash  
>> /root/MultipathScripts/mpath_prio_alt %n"
>>        path_checker            tur
>>        rr_min_io               128
>>        max_fds                 8192
>>        rr_weight               priorities
>>        failback                immediate
>>        no_path_retry           queue
>>        user_friendly_names     yes
>> }
>>     
>
>
> Please don't top-post.
>
> I'm pretty sure John meant to say "group_by_prio" rather than "failover"
> in his initial reply to this thread.  John originally got this insight
> (dummy device section et. al. applies to RHEL 5.3) back in April:
>
> https://www.redhat.com/archives/dm-devel/2009-April/msg00157.html
>
> Which multipath/distro are you using?
>
> Mike
>   

I tried out John's approach as well and it still seems to be giving me 
the similar results to what I mentioned before, i.e., replicating multibus


Priority list:
--------------
192.168.7.103:3260-iscsi-iqn.2008-07.com.cleversafe:vault-2    5
192.168.7.106:3260-iscsi-iqn.2008-07.com.cleversafe:vault-2    10
192.168.7.103:3260-iscsi-iqn.2008-07.com.cleversafe:vault-1    10
192.168.7.106:3260-iscsi-iqn.2008-07.com.cleversafe:vault-1    5

iSCSI disk associated with iSCSI sessions:
------------------------------------------
Target: iqn.2008-07.com.cleversafe:vault-1
    Current Portal: 192.168.7.106:3260,1
    Persistent Portal: 192.168.7.106:3260,1
            Attached scsi disk sdb        State: running
    Current Portal: 192.168.7.103:3260,1
    Persistent Portal: 192.168.7.103:3260,1
            Attached scsi disk sdd        State: running

Target: iqn.2008-07.com.cleversafe:vault-2
    Current Portal: 192.168.7.106:3260,1
    Persistent Portal: 192.168.7.106:3260,1
            Attached scsi disk sdc        State: running
    Current Portal: 192.168.7.103:3260,1
    Persistent Portal: 192.168.7.103:3260,1
            Attached scsi disk sde        State: running


multipath -ll:
---------------
mpath2 (244534e3833623961) dm-1 DSNET,Dispersed Store
[size=47G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=15][enabled]
 \_ 85:0:0:0 sdc 8:32  [active][ready]
 \_ 87:0:0:0 sde 8:64  [active][ready]
mpath1 (244534e3266616134) dm-0 DSNET,Dispersed Store
[size=47G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=15][enabled]
 \_ 84:0:0:0 sdb 8:16  [active][ready]
 \_ 86:0:0:0 sdd 8:48  [active][ready]



Versions being used:
dm-multipath: device-mapper-multipath-0.4.7-23.el5_3.4
kernel: 2.6.29.6

Re: User specific priorities on for iscsi paths doesn't seem to work

[Akshay Lal]

Akshay Lal wrote:
> Mike Snitzer wrote:
>> On Fri, Aug 07 2009 at  5:07pm -0400,
>> Akshay Lal <alal@cleversafe.com> wrote:
>>
>>  
>>> Mike Snitzer wrote:
>>>    
>>>> On Fri, Aug 07 2009 at  4:25pm -0400,
>>>> Akshay Lal <alal@cleversafe.com> wrote:
>>>>
>>>>        
>>>>> I'm having a few issues with path priorities. It seems that the 
>>>>> choice  of path to use during I/O is independent of the user 
>>>>> defined priorities  for each path.
>>>>>
>>>>> I am setting the priorities by executing writing a script that is 
>>>>> used  by prio_callout. This seems to work when I execute multipath 
>>>>> -ll since  all the specified priorities show up correctly. (the   
>>>>> path_grouping_policy being used is  failover)
>>>>>             
>>>> ...
>>>>
>>>>        
>>>>> Is there something I'm doing wrong? I would like to be able to 
>>>>> define  the priorities per device, and ensure that data only 
>>>>> traverses on the  lower priority path when
>>>>> a) a failure to the first path (path with a higher priority) occures
>>>>> b) no other path with a higher priority exists
>>>>>             
>>>> Do things behave as you'd like if you change path_grouping_policy to
>>>> 'group_by_prio'?
>>>>
>>>> Mike
>>>>         
>>
>>
>>  
>>> Mike:
>>>
>>> It seems that if I were to set the path_grouping_policy to  
>>> "group_by_prio", then it seems to be working similar to a multibus  
>>> configuration. What I would like is for within a single multipath 
>>> group,  (say mpath1), specify a primary path and an 
>>> alternate/failover path. If  I can make this configurable via user 
>>> land tool that'd be great. In this  vein, I had considered 
>>> priorities with hope that if I can set the  priority of a certain 
>>> path within a group then the path with the highest  priority will 
>>> always be chosen and the other path (with the lower  priority) will 
>>> only come into play when the primary goes down.
>>>
>>> Below is the output of the multipath -ll & conf file when setting 
>>> the  path_grouping_policy to group_by_prio.
>>>
>>>
>>> multipath -ll:
>>> --------------
>>> mpath2 (244534e3833623961) dm-1 DSNET,Dispersed Store
>>> [size=47G][features=1 queue_if_no_path][hwhandler=0][rw]
>>> \_ round-robin 0 [prio=15][enabled]
>>> \_ 85:0:0:0 sdc 8:32  [active][ready]
>>> \_ 87:0:0:0 sde 8:64  [active][ready]
>>> mpath1 (244534e3266616134) dm-0 DSNET,Dispersed Store
>>> [size=47G][features=1 queue_if_no_path][hwhandler=0][rw]
>>> \_ round-robin 0 [prio=15][active]
>>> \_ 84:0:0:0 sdb 8:16  [active][ready]
>>> \_ 86:0:0:0 sdd 8:48  [active][ready]
>>>
>>>
>>> /etc/multipath.conf:
>>> --------------------
>>> defaults {
>>>        udev_dir                /dev
>>>        polling_interval        1
>>>        selector                "round-robin 0"
>>>        path_grouping_policy    group_by_prio
>>>        getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
>>>        prio_callout            "/bin/bash  
>>> /root/MultipathScripts/mpath_prio_alt %n"
>>>        path_checker            tur
>>>        rr_min_io               128
>>>        max_fds                 8192
>>>        rr_weight               priorities
>>>        failback                immediate
>>>        no_path_retry           queue
>>>        user_friendly_names     yes
>>> }
>>>     
>>
>>
>> Please don't top-post.
>>
>> I'm pretty sure John meant to say "group_by_prio" rather than "failover"
>> in his initial reply to this thread.  John originally got this insight
>> (dummy device section et. al. applies to RHEL 5.3) back in April:
>>
>> https://www.redhat.com/archives/dm-devel/2009-April/msg00157.html
>>
>> Which multipath/distro are you using?
>>
>> Mike
>>   
>
> I tried out John's approach as well and it still seems to be giving me 
> the similar results to what I mentioned before, i.e., replicating 
> multibus
>
>
> Priority list:
> --------------
> 192.168.7.103:3260-iscsi-iqn.2008-07.com.cleversafe:vault-2    5
> 192.168.7.106:3260-iscsi-iqn.2008-07.com.cleversafe:vault-2    10
> 192.168.7.103:3260-iscsi-iqn.2008-07.com.cleversafe:vault-1    10
> 192.168.7.106:3260-iscsi-iqn.2008-07.com.cleversafe:vault-1    5
>
> iSCSI disk associated with iSCSI sessions:
> ------------------------------------------
> Target: iqn.2008-07.com.cleversafe:vault-1
>    Current Portal: 192.168.7.106:3260,1
>    Persistent Portal: 192.168.7.106:3260,1
>            Attached scsi disk sdb        State: running
>    Current Portal: 192.168.7.103:3260,1
>    Persistent Portal: 192.168.7.103:3260,1
>            Attached scsi disk sdd        State: running
>
> Target: iqn.2008-07.com.cleversafe:vault-2
>    Current Portal: 192.168.7.106:3260,1
>    Persistent Portal: 192.168.7.106:3260,1
>            Attached scsi disk sdc        State: running
>    Current Portal: 192.168.7.103:3260,1
>    Persistent Portal: 192.168.7.103:3260,1
>            Attached scsi disk sde        State: running
>
>
> multipath -ll:
> ---------------
> mpath2 (244534e3833623961) dm-1 DSNET,Dispersed Store
> [size=47G][features=1 queue_if_no_path][hwhandler=0][rw]
> \_ round-robin 0 [prio=15][enabled]
> \_ 85:0:0:0 sdc 8:32  [active][ready]
> \_ 87:0:0:0 sde 8:64  [active][ready]
> mpath1 (244534e3266616134) dm-0 DSNET,Dispersed Store
> [size=47G][features=1 queue_if_no_path][hwhandler=0][rw]
> \_ round-robin 0 [prio=15][enabled]
> \_ 84:0:0:0 sdb 8:16  [active][ready]
> \_ 86:0:0:0 sdd 8:48  [active][ready]
>
>
>
> Versions being used:
> dm-multipath: device-mapper-multipath-0.4.7-23.el5_3.4
> kernel: 2.6.29.6
>
>
Sorry I forgot to add the multipath conf file

/etc/multipath.conf :
---------------------
# Blacklist all devices by default. Remove this to enable multipathing
# on the default devices.
#blacklist {
#        devnode "*"
#}

##
## Defaults for the multipath daemon
##
#
defaults {
        udev_dir                /dev
        polling_interval        1
        selector                "round-robin 0"
        path_grouping_policy    group_by_prio
        getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
        prio_callout            "/bin/bash /sbin/mpath_prio_alt %n"
        path_checker            tur
#        rr_min_io               128
        max_fds                 8192
        rr_weight               uniform
        failback                immediate
        no_path_retry           queue
        user_friendly_names     yes
}
devices {
        device {
                vendor          "dummy"
                product         "dummy"
            prio_callout    "/sbin/mpath_prio_alt %n"
        }
}



--
Akshay Lal

Re: Re: User specific priorities on for iscsi paths doesn't seem to work

[John A. Sullivan III]

On Fri, 2009-08-07 at 17:21 -0400, Mike Snitzer wrote:
> On Fri, Aug 07 2009 at  5:07pm -0400,
> Akshay Lal <alal@cleversafe.com> wrote:
> 
> > Mike Snitzer wrote:
> >> On Fri, Aug 07 2009 at  4:25pm -0400,
> >> Akshay Lal <alal@cleversafe.com> wrote:
> >>
> >>   
> >>> I'm having a few issues with path priorities. It seems that the 
> >>> choice  of path to use during I/O is independent of the user defined 
> >>> priorities  for each path.
> >>>
> >>> I am setting the priorities by executing writing a script that is 
> >>> used  by prio_callout. This seems to work when I execute multipath 
> >>> -ll since  all the specified priorities show up correctly. (the   
> >>> path_grouping_policy being used is  failover)
> >>>     
> >>
> >> ...
> >>
> >>   
> >>> Is there something I'm doing wrong? I would like to be able to define 
> >>>  the priorities per device, and ensure that data only traverses on 
> >>> the  lower priority path when
> >>> a) a failure to the first path (path with a higher priority) occures
> >>> b) no other path with a higher priority exists
> >>>     
> >>
> >> Do things behave as you'd like if you change path_grouping_policy to
> >> 'group_by_prio'?
> >>
> >> Mike
> >>   
> 
> 
> > Mike:
> >
> > It seems that if I were to set the path_grouping_policy to  
> > "group_by_prio", then it seems to be working similar to a multibus  
> > configuration. What I would like is for within a single multipath group,  
> > (say mpath1), specify a primary path and an alternate/failover path. If  
> > I can make this configurable via user land tool that'd be great. In this  
> > vein, I had considered priorities with hope that if I can set the  
> > priority of a certain path within a group then the path with the highest  
> > priority will always be chosen and the other path (with the lower  
> > priority) will only come into play when the primary goes down.
> >
> > Below is the output of the multipath -ll & conf file when setting the  
> > path_grouping_policy to group_by_prio.
> >
> >
> > multipath -ll:
> > --------------
> > mpath2 (244534e3833623961) dm-1 DSNET,Dispersed Store
> > [size=47G][features=1 queue_if_no_path][hwhandler=0][rw]
> > \_ round-robin 0 [prio=15][enabled]
> > \_ 85:0:0:0 sdc 8:32  [active][ready]
> > \_ 87:0:0:0 sde 8:64  [active][ready]
> > mpath1 (244534e3266616134) dm-0 DSNET,Dispersed Store
> > [size=47G][features=1 queue_if_no_path][hwhandler=0][rw]
> > \_ round-robin 0 [prio=15][active]
> > \_ 84:0:0:0 sdb 8:16  [active][ready]
> > \_ 86:0:0:0 sdd 8:48  [active][ready]
> >
> >
> > /etc/multipath.conf:
> > --------------------
> > defaults {
> >        udev_dir                /dev
> >        polling_interval        1
> >        selector                "round-robin 0"
> >        path_grouping_policy    group_by_prio
> >        getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
> >        prio_callout            "/bin/bash  
> > /root/MultipathScripts/mpath_prio_alt %n"
> >        path_checker            tur
> >        rr_min_io               128
> >        max_fds                 8192
> >        rr_weight               priorities
> >        failback                immediate
> >        no_path_retry           queue
> >        user_friendly_names     yes
> > }
> 
> 
> Please don't top-post.
> 
> I'm pretty sure John meant to say "group_by_prio" rather than "failover"
> in his initial reply to this thread.  John originally got this insight
> (dummy device section et. al. applies to RHEL 5.3) back in April:
> 
> https://www.redhat.com/archives/dm-devel/2009-April/msg00157.html
> 
> Which multipath/distro are you using?
> 
> Mike
<snip>
Actually, we are using failover in our environment.  Perhaps I have
missed something but it is working well for us as far as I can tell.  We
have a single path unless it fails in which case we go to the next path
in priority order.  We are handling load balancing across paths in a
different way as we found the performance of multibus was less than we
could achieve otherwise.  Then again, this is not an area of expertise
for me.  Thanks - John
-- 
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

http://www.spiritualoutreach.com
Making Christianity intelligible to secular society

Re: User specific priorities on for iscsi paths doesn't seem to work

[John A. Sullivan III]

On Fri, 2009-08-07 at 15:25 -0500, Akshay Lal wrote:
> I'm having a few issues with path priorities. It seems that the choice 
> of path to use during I/O is independent of the user defined priorities 
> for each path.
> 
> I am setting the priorities by executing writing a script that is used 
> by prio_callout. This seems to work when I execute multipath -ll since 
> all the specified priorities show up correctly. (the 
> path_grouping_policy being used is  failover)
> 
> multipath -ll:
> --------------
> mpath2 (244534e3833623961) dm-1 DSNET,Dispersed Store
> [size=47G][features=1 queue_if_no_path][hwhandler=0][rw]
> \_ round-robin 0 [prio=10][enabled]
>  \_ 85:0:0:0 sdc 8:32  [active][ready]
> \_ round-robin 0 [prio=5][enabled]
>  \_ 87:0:0:0 sde 8:64  [active][ready]
> mpath1 (244534e3266616134) dm-0 DSNET,Dispersed Store
> [size=47G][features=1 queue_if_no_path][hwhandler=0][rw]
> \_ round-robin 0 [prio=5][enabled]
>  \_ 84:0:0:0 sdb 8:16  [active][ready]
> \_ round-robin 0 [prio=10][enabled]
>  \_ 86:0:0:0 sdd 8:48  [active][ready]
> 
> At this point when I dd to /dev/mapper/mpath1 or /dev/mapper/mpath2 the 
> priorities don't come into play at all. Data starts going to /dev/sdc 
> for /dev/mapper/mpath2 & /dev/sdb for /dev/mapper/mpath1, which totally 
> by-passes the set priorities. Below is a list of all the conf files and 
> scripts I'm using.
> 
> 
> 
> /etc/multipath.conf
> --------------------
> defaults {
>         udev_dir                /dev
>         polling_interval        1
>         selector                "round-robin 0"
>         path_grouping_policy    multibus
>         getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
>         prio_callout            "/bin/bash 
> /root/MultipathScripts/mpath_prio_alt %n"
>         path_checker            tur
>         rr_min_io               128
>         max_fds                 8192
>         rr_weight               priorities
>         failback                immediate
>         no_path_retry           queue
>         user_friendly_names     yes
> }
> 
> 
> /root/MultipathScripts/mpath_prio_alt :
> ---------------------------------------
> #!/bin/bash
> # if not passed any device name, return a priority of 0
> if [ -z "${1}" ];then
>         echo 0
>         exit
> fi
> 
> ENTRY="$(ls -l1 /dev/disk/by-path | grep /${1}'$')"
> DEV="${ENTRY##* ip-}"
> if [ "$DEV" = "${ENTRY}" ];then #This is not an iSCSI device
>         echo 0
>         exit
> fi
> 
> # stripping $ENTRY to obtain the IP:Port-iscsi-iqn-lunNumber
> DEV="${ENTRY##* ip-}"
> #DEV="${DEV%% ->*}" # the pattern changed in CentOS 5.3
> #DEV="$(echo ${DEV} | sed 's/-lun-[0-9][0-9]* ->.*//')"
> DEV="${DEV%%-lun-[0-9]* ->*}"
> 
> echo $(grep ${DEV} /root/MultipathScripts/iscsi.list | cut -f2)
> 
> 
> /root/MultipathScripts/iscsi.list:
> ----------------------------------
> 192.xxx.xxx.103:3260-iscsi-iqn.2008-07.com.company:stoarge-2    1
> 192.xxx.xxx.106:3260-iscsi-iqn.2008-07.com.company:stoarge-2    0
> 192.xxx.xxx.103:3260-iscsi-iqn.2008-07.com.company:stoarge-1    0
> 192.xxx.xxx.106:3260-iscsi-iqn.2008-07.com.company:stoarge-1    1
> 
> 
> Is there something I'm doing wrong? I would like to be able to define 
> the priorities per device, and ensure that data only traverses on the 
> lower priority path when
> a) a failure to the first path (path with a higher priority) occures
> b) no other path with a higher priority exists
<snip>
I believe to do what you want you will need two changes.  First, set
path_grouping_policy to failover.

Second, multipathd needs to have everything in memory so not only do you
need /bin/bash loaded into memory (via your prio-callout line in
defaults) but you must also load your script into memory.  We did this
with a dummy device. Here is our devices section of multipath.conf:

devices {
       device {
               vendor                  "NEXENTA"
               product                 "COMSTAR"
               getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
               features                "0"
               hardware_handler        "0"
#               path_grouping_policy    failover
               rr_weight               uniform
#               rr_min_io               1000
               path_checker            tur
       }
        device {
                vendor          "dummy"
                product         "Used_to_load_mpath_prio_ssi_into_memory"
                prio_callout            "/sbin/mpath_prio_ssi %n"
        }
}

Hope this helps - John
-- 
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

http://www.spiritualoutreach.com
Making Christianity intelligible to secular society