multipathd ignoring dev_loss_tmo setting

multipathd ignoring dev_loss_tmo setting

[Martins, Bruno O]

Hello guys,

I am trying to modify /etc/multipath.conf on my system so that the
parameter 'dev_loss_tmo' is changed from the default value.

My multipath.conf file contains the following:

defaults {
        verbosity 2
        polling_interval 5
        max_polling_interval 10
        multipath_dir "/lib64/multipath"
        path_selector "round-robin 0"
        path_grouping_policy "failover"
        uid_attribute "ID_SERIAL"
        prio "const"
        prio_args ""
        features "0"
        path_checker "directio"
        alias_prefix "mpath"
        failback "manual"
        rr_min_io 1000
        rr_min_io_rq 1
        max_fds "max"
        rr_weight "uniform"
        no_path_retry "fail"
        queue_without_daemon "no"
        checker_timeout 15
        flush_on_last_del "no"
        user_friendly_names "yes"
        fast_io_fail_tmo 5
        dev_loss_tmo 10
        bindings_file "/etc/multipath/bindings"
        wwids_file /etc/multipath/wwids
        log_checker_err always
        retain_attached_hw_handler no
        detect_prio no
}

However, when checking the value currently in use I am getting the
wrong value (which is '30') for some of the remote ports:

for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do
d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f);
done

rport-3:0-0:0x5742b0f00007c500:10
rport-3:0-1:0x5742b0f00007c500:10
rport-3:0-2:0x5742b0f00007c500:10
rport-3:0-3:0x5000097408369800:30
rport-3:0-4:0x500009757804cbff:30
rport-4:0-0:0x5742b0f00007c500:10
rport-4:0-1:0x5742b0f00007c500:10
rport-4:0-2:0x5000097408369800:30
rport-4:0-3:0x5742b0f00007c500:10
rport-4:0-4:0x500009757804cbff:30
rport-5:0-0:0x5742b0f00007c500:10
rport-5:0-1:0x5742b0f00007c500:10
rport-5:0-2:0x5742b0f00007c500:10
rport-5:0-3:0x5000097408369800:30
rport-5:0-4:0x500009757804cbff:30
rport-6:0-0:0x5742b0f00007c500:10
rport-6:0-1:0x5742b0f00007c500:10
rport-6:0-2:0x5000097408369800:30
rport-6:0-3:0x5742b0f00007c500:10
rport-6:0-4:0x500009757804cbff:30

systool is giving me the same information:

systool -c fc_remote_ports -v | grep dev_loss_tmo

    dev_loss_tmo        = "10"
    dev_loss_tmo        = "10"
    dev_loss_tmo        = "10"
    dev_loss_tmo        = "10"
    dev_loss_tmo        = "10"
    dev_loss_tmo        = "10"
    dev_loss_tmo        = "10"
    dev_loss_tmo        = "10"
    dev_loss_tmo        = "10"
    dev_loss_tmo        = "30"
    dev_loss_tmo        = "10"
    dev_loss_tmo        = "30"
    dev_loss_tmo        = "30"
    dev_loss_tmo        = "10"
    dev_loss_tmo        = "30"
    dev_loss_tmo        = "10"
    dev_loss_tmo        = "30"
    dev_loss_tmo        = "30"
    dev_loss_tmo        = "30"
    dev_loss_tmo        = "30"

Where is this value coming from? May this be a bug? I couldn't find
anything useful on the Internet regarding this.

I am using the following versions:

rpm -qa multipath-tools
multipath-tools-0.4.9-109.1

uname -a
Linux mysystem 3.0.101-63-default #1 SMP Tue Jun 23 16:02:31 UTC 2015
(4b89d0c) x86_64 x86_64 x86_64 GNU/Linux

Thanks for your help!

Kind regards,

Bruno

Re: multipathd ignoring dev_loss_tmo setting

[Steffen Maier]

On 02/28/2019 12:38 PM, Martins, Bruno O wrote:
> I am trying to modify /etc/multipath.conf on my system so that the
> parameter 'dev_loss_tmo' is changed from the default value.
> 
> My multipath.conf file contains the following:
> 
> defaults {

>          no_path_retry "fail"
>          queue_without_daemon "no"
>          checker_timeout 15
>          flush_on_last_del "no"
>          user_friendly_names "yes"
>          fast_io_fail_tmo 5
>          dev_loss_tmo 10

> }

If I'm not mistaken, the default settings could be overridden by (built-in) 
device section settings.
However, in your case, where you see some rports having your desired setting 
but others not, it looks like it's due to multipath-tools only setting the 
_tmo's for those rports where it has paths it manages in path groups.
So if you don't have multipath managed paths behind e.g. 
rport-3:0-3:0x5000097408369800 it remains to use the rport's default tmo coming 
from scsi_transport_fc in the kernel.
If so, we typically don't care for those "unmanaged" ports.

> However, when checking the value currently in use I am getting the
> wrong value (which is '30') for some of the remote ports:
> 
> for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do
> d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f);
> done
> 
> rport-3:0-0:0x5742b0f00007c500:10
> rport-3:0-1:0x5742b0f00007c500:10
> rport-3:0-2:0x5742b0f00007c500:10
> rport-3:0-3:0x5000097408369800:30
> rport-3:0-4:0x500009757804cbff:30

> systool is giving me the same information:
> 
> systool -c fc_remote_ports -v | grep dev_loss_tmo

>      dev_loss_tmo        = "10"
>      dev_loss_tmo        = "30"
>      dev_loss_tmo        = "10"
>      dev_loss_tmo        = "30"
>      dev_loss_tmo        = "30"
>      dev_loss_tmo        = "30"
>      dev_loss_tmo        = "30"
> 
> Where is this value coming from? May this be a bug? I couldn't find
> anything useful on the Internet regarding this.
> 
> I am using the following versions:
> 
> rpm -qa multipath-tools
> multipath-tools-0.4.9-109.1
> 
> uname -a
> Linux mysystem 3.0.101-63-default #1 SMP Tue Jun 23 16:02:31 UTC 2015
> (4b89d0c) x86_64 x86_64 x86_64 GNU/Linux



-- 
Mit freundlichen Gruessen / Kind regards
Steffen Maier

Linux on IBM Z Development

https://www.ibm.com/privacy/us/en/
IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Matthias Hartmann
Geschaeftsfuehrung: Dirk Wittkopp
Sitz der Gesellschaft: Boeblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294

Re: multipathd ignoring dev_loss_tmo setting

[Benjamin Marzinski]

On Thu, Feb 28, 2019 at 11:38:22AM +0000, Martins, Bruno O wrote:
> Hello guys,
> 
> I am trying to modify /etc/multipath.conf on my system so that the
> parameter 'dev_loss_tmo' is changed from the default value.
> 
> However, when checking the value currently in use I am getting the
> wrong value (which is '30') for some of the remote ports:
>

Are you sure those rports are used by multipath devices? multipath only
changes dev_loss_tmo for rports associated with a multipath path device.

-Ben
 
> for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do
> d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f);
> done
> 
> rport-3:0-0:0x5742b0f00007c500:10
> rport-3:0-1:0x5742b0f00007c500:10
> rport-3:0-2:0x5742b0f00007c500:10
> rport-3:0-3:0x5000097408369800:30
> rport-3:0-4:0x500009757804cbff:30
> rport-4:0-0:0x5742b0f00007c500:10
> rport-4:0-1:0x5742b0f00007c500:10
> rport-4:0-2:0x5000097408369800:30
> rport-4:0-3:0x5742b0f00007c500:10
> rport-4:0-4:0x500009757804cbff:30
> rport-5:0-0:0x5742b0f00007c500:10
> rport-5:0-1:0x5742b0f00007c500:10
> rport-5:0-2:0x5742b0f00007c500:10
> rport-5:0-3:0x5000097408369800:30
> rport-5:0-4:0x500009757804cbff:30
> rport-6:0-0:0x5742b0f00007c500:10
> rport-6:0-1:0x5742b0f00007c500:10
> rport-6:0-2:0x5000097408369800:30
> rport-6:0-3:0x5742b0f00007c500:10
> rport-6:0-4:0x500009757804cbff:30
> 
> systool is giving me the same information:
> 
> systool -c fc_remote_ports -v | grep dev_loss_tmo
> 
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "30"
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "30"
>     dev_loss_tmo        = "30"
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "30"
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "30"
>     dev_loss_tmo        = "30"
>     dev_loss_tmo        = "30"
>     dev_loss_tmo        = "30"
> 
> Where is this value coming from? May this be a bug? I couldn't find
> anything useful on the Internet regarding this.
> 
> I am using the following versions:
> 
> rpm -qa multipath-tools
> multipath-tools-0.4.9-109.1
> 
> uname -a
> Linux mysystem 3.0.101-63-default #1 SMP Tue Jun 23 16:02:31 UTC 2015
> (4b89d0c) x86_64 x86_64 x86_64 GNU/Linux
> 
> Thanks for your help!
> 
> Kind regards,
> 
> Bruno
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: multipathd ignoring dev_loss_tmo setting

[Martins, Bruno O]

On Thu, 2019-02-28 at 13:53 -0600, Benjamin Marzinski wrote:
> On Thu, Feb 28, 2019 at 11:38:22AM +0000, Martins, Bruno O wrote:
> > Hello guys,
> > 
> > I am trying to modify /etc/multipath.conf on my system so that the
> > parameter 'dev_loss_tmo' is changed from the default value.
> > 
> > However, when checking the value currently in use I am getting the
> > wrong value (which is '30') for some of the remote ports:
> > 
> 
> Are you sure those rports are used by multipath devices? multipath
> only
> changes dev_loss_tmo for rports associated with a multipath path
> device.
> 
> -Ben
>  
> > for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do
> > d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f);
> > done
> > 
> > rport-3:0-0:0x5742b0f00007c500:10
> > rport-3:0-1:0x5742b0f00007c500:10
> > rport-3:0-2:0x5742b0f00007c500:10
> > rport-3:0-3:0x5000097408369800:30
> > rport-3:0-4:0x500009757804cbff:30
> > rport-4:0-0:0x5742b0f00007c500:10
> > rport-4:0-1:0x5742b0f00007c500:10
> > rport-4:0-2:0x5000097408369800:30
> > rport-4:0-3:0x5742b0f00007c500:10
> > rport-4:0-4:0x500009757804cbff:30
> > rport-5:0-0:0x5742b0f00007c500:10
> > rport-5:0-1:0x5742b0f00007c500:10
> > rport-5:0-2:0x5742b0f00007c500:10
> > rport-5:0-3:0x5000097408369800:30
> > rport-5:0-4:0x500009757804cbff:30
> > rport-6:0-0:0x5742b0f00007c500:10
> > rport-6:0-1:0x5742b0f00007c500:10
> > rport-6:0-2:0x5000097408369800:30
> > rport-6:0-3:0x5742b0f00007c500:10
> > rport-6:0-4:0x500009757804cbff:30
> > 
> > systool is giving me the same information:
> > 
> > systool -c fc_remote_ports -v | grep dev_loss_tmo
> > 
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "30"
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "30"
> >     dev_loss_tmo        = "30"
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "30"
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "30"
> >     dev_loss_tmo        = "30"
> >     dev_loss_tmo        = "30"
> >     dev_loss_tmo        = "30"
> > 
> > Where is this value coming from? May this be a bug? I couldn't find
> > anything useful on the Internet regarding this.
> > 
> > I am using the following versions:
> > 
> > rpm -qa multipath-tools
> > multipath-tools-0.4.9-109.1
> > 
> > uname -a
> > Linux mysystem 3.0.101-63-default #1 SMP Tue Jun 23 16:02:31 UTC
> > 2015
> > (4b89d0c) x86_64 x86_64 x86_64 GNU/Linux
> > 
> > Thanks for your help!
> > 
> > Kind regards,
> > 
> > Bruno
> > 
> > --
> > dm-devel mailing list
> > dm-devel@redhat.com
> > 
> > https://www.redhat.com/mailman/listinfo/dm-devel
> > 

Hi Benjamin,

Thanks for your reply!

I believe they are:

[ 10:02:45 ] root@myhost:~# multipath -ll | grep 3:0:3
  |- 3:0:3:12  sdblc 128:1568 active ready running
  |- 3:0:3:13  sdbnu 132:1664 active ready running
  |- 3:0:3:18  sdbre 66:1792  active ready running
  |- 3:0:3:1   sdbkg 70:1728  active ready running
  |- 3:0:3:2   sdbnv 132:1680 active ready running
  |- 3:0:3:20  sdbrg 66:1824  active ready running
  |- 3:0:3:17  sdbpg 134:1760 active ready running
  |- 3:0:3:16  sdbpf 134:1744 active ready running
  |- 3:0:3:11  sdbkf 70:1712  active ready running
  |- 3:0:3:19  sdbrf 66:1808  active ready running
  |- 3:0:3:14  sdbnw 132:1696 active ready running
  |- 3:0:3:15  sdbpe 134:1728 active ready running

Is this the best way to check that information?

BR,

Re: multipathd ignoring dev_loss_tmo setting

[Benjamin Marzinski]

On Fri, Mar 01, 2019 at 10:04:52AM +0000, Martins, Bruno O wrote:
> On Thu, 2019-02-28 at 13:53 -0600, Benjamin Marzinski wrote:
> > On Thu, Feb 28, 2019 at 11:38:22AM +0000, Martins, Bruno O wrote:
> > > Hello guys,
> > > 
> > > I am trying to modify /etc/multipath.conf on my system so that the
> > > parameter 'dev_loss_tmo' is changed from the default value.
> > > 
> > > However, when checking the value currently in use I am getting the
> > > wrong value (which is '30') for some of the remote ports:
> > > 
> > 
> > Are you sure those rports are used by multipath devices? multipath
> > only
> > changes dev_loss_tmo for rports associated with a multipath path
> > device.
> > 
> > -Ben
> >  
> 
> Hi Benjamin,
> 
> Thanks for your reply!
> 
> I believe they are:
> 
> [ 10:02:45 ] root@myhost:~# multipath -ll | grep 3:0:3
>   |- 3:0:3:12  sdblc 128:1568 active ready running
>   |- 3:0:3:13  sdbnu 132:1664 active ready running
>   |- 3:0:3:18  sdbre 66:1792  active ready running
>   |- 3:0:3:1   sdbkg 70:1728  active ready running
>   |- 3:0:3:2   sdbnv 132:1680 active ready running
>   |- 3:0:3:20  sdbrg 66:1824  active ready running
>   |- 3:0:3:17  sdbpg 134:1760 active ready running
>   |- 3:0:3:16  sdbpf 134:1744 active ready running
>   |- 3:0:3:11  sdbkf 70:1712  active ready running
>   |- 3:0:3:19  sdbrf 66:1808  active ready running
>   |- 3:0:3:14  sdbnw 132:1696 active ready running
>   |- 3:0:3:15  sdbpe 134:1728 active ready running
> 
> Is this the best way to check that information?

The scsi HBTL isn't guaranteed to line up with the rport id.  To find this
out you can either run

# ls -l /sys/block

and then check the rport for your path devices from the link
destination.

For instance

[root@ask-07 block]# ls -l /sys/block
<snip>
lrwxrwxrwx. 1 root root 0 Feb 21 02:45 sdb -> ../devices/pci0000:00/0000:00:0a.0/0000:06:00.0/host16/rport-16:0-0/target16:0:0/16:0:0:0/block/sdb
lrwxrwxrwx. 1 root root 0 Feb 21 02:45 sdc -> ../devices/pci0000:00/0000:00:0a.0/0000:06:00.0/host16/rport-16:0-0/target16:0:0/16:0:0:1/block/sdc
lrwxrwxrwx. 1 root root 0 Feb 21 02:45 sdd -> ../devices/pci0000:00/0000:00:0a.0/0000:06:00.1/host17/rport-17:0-0/target17:0:0/17:0:0:0/block/sdd
lrwxrwxrwx. 1 root root 0 Feb 21 02:45 sde -> ../devices/pci0000:00/0000:00:0a.0/0000:06:00.1/host17/rport-17:0-0/target17:0:0/17:0:0:1/block/sde
<snip>
 
Here, sdb and sdc are using rport-16:0-0, and sdd and sde are using
rport-17:0-0

Otherwise, you can pick an rport with the wrong dev_loss_tmo, and check

/sys/class/fc_remote_ports/<rport>/device/<target>/

In there there will be a number of scsi HBTL identifiers, for example

# ls /sys/class/fc_remote_ports/rport-16\:0-0/device/target16\:0\:0/
16:0:0:0  16:0:0:1  fc_transport  power  subsystem  uevent

If these HBTL ids (16:0:0:0 and 16:0:0:1) are for multipath path
devices, then multipath should be updating dev_loss_tmo for them. In my
example above, the scsi HBTL does correspond to the rport id, but this
isn't always the case.

-Ben

> BR,

Re: multipathd ignoring dev_loss_tmo setting

[Martins, Bruno O]

On Fri, 2019-03-01 at 11:35 -0600, Benjamin Marzinski wrote:
> On Fri, Mar 01, 2019 at 10:04:52AM +0000, Martins, Bruno O wrote:
> > On Thu, 2019-02-28 at 13:53 -0600, Benjamin Marzinski wrote:
> > > On Thu, Feb 28, 2019 at 11:38:22AM +0000, Martins, Bruno O wrote:
> > > > Hello guys,
> > > > 
> > > > I am trying to modify /etc/multipath.conf on my system so that
> > > > the
> > > > parameter 'dev_loss_tmo' is changed from the default value.
> > > > 
> > > > However, when checking the value currently in use I am getting
> > > > the
> > > > wrong value (which is '30') for some of the remote ports:
> > > > 
> > > 
> > > Are you sure those rports are used by multipath devices?
> > > multipath
> > > only
> > > changes dev_loss_tmo for rports associated with a multipath path
> > > device.
> > > 
> > > -Ben
> > >  
> > 
> > Hi Benjamin,
> > 
> > Thanks for your reply!
> > 
> > I believe they are:
> > 
> > [ 10:02:45 ] root@myhost:~# multipath -ll | grep 3:0:3
> >   |- 3:0:3:12  sdblc 128:1568 active ready running
> >   |- 3:0:3:13  sdbnu 132:1664 active ready running
> >   |- 3:0:3:18  sdbre 66:1792  active ready running
> >   |- 3:0:3:1   sdbkg 70:1728  active ready running
> >   |- 3:0:3:2   sdbnv 132:1680 active ready running
> >   |- 3:0:3:20  sdbrg 66:1824  active ready running
> >   |- 3:0:3:17  sdbpg 134:1760 active ready running
> >   |- 3:0:3:16  sdbpf 134:1744 active ready running
> >   |- 3:0:3:11  sdbkf 70:1712  active ready running
> >   |- 3:0:3:19  sdbrf 66:1808  active ready running
> >   |- 3:0:3:14  sdbnw 132:1696 active ready running
> >   |- 3:0:3:15  sdbpe 134:1728 active ready running
> > 
> > Is this the best way to check that information?
> 
> The scsi HBTL isn't guaranteed to line up with the rport id.  To find
> this
> out you can either run
> 
> # ls -l /sys/block
> 
> and then check the rport for your path devices from the link
> destination.
> 
> For instance
> 
> [root@ask-07 block]# ls -l /sys/block
> <snip>
> lrwxrwxrwx. 1 root root 0 Feb 21 02:45 sdb ->
> ../devices/pci0000:00/0000:00:0a.0/0000:06:00.0/host16/rport-16:0-
> 0/target16:0:0/16:0:0:0/block/sdb
> lrwxrwxrwx. 1 root root 0 Feb 21 02:45 sdc ->
> ../devices/pci0000:00/0000:00:0a.0/0000:06:00.0/host16/rport-16:0-
> 0/target16:0:0/16:0:0:1/block/sdc
> lrwxrwxrwx. 1 root root 0 Feb 21 02:45 sdd ->
> ../devices/pci0000:00/0000:00:0a.0/0000:06:00.1/host17/rport-17:0-
> 0/target17:0:0/17:0:0:0/block/sdd
> lrwxrwxrwx. 1 root root 0 Feb 21 02:45 sde ->
> ../devices/pci0000:00/0000:00:0a.0/0000:06:00.1/host17/rport-17:0-
> 0/target17:0:0/17:0:0:1/block/sde
> <snip>
>  
> Here, sdb and sdc are using rport-16:0-0, and sdd and sde are using
> rport-17:0-0
> 
> Otherwise, you can pick an rport with the wrong dev_loss_tmo, and
> check
> 
> /sys/class/fc_remote_ports/<rport>/device/<target>/
> 
> In there there will be a number of scsi HBTL identifiers, for example
> 
> # ls /sys/class/fc_remote_ports/rport-16\:0-0/device/target16\:0\:0/
> 16:0:0:0  16:0:0:1  fc_transport  power  subsystem  uevent
> 
> If these HBTL ids (16:0:0:0 and 16:0:0:1) are for multipath path
> devices, then multipath should be updating dev_loss_tmo for them. In
> my
> example above, the scsi HBTL does correspond to the rport id, but
> this
> isn't always the case.
> 
> -Ben
> 
> > BR,
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> 
> https://www.redhat.com/mailman/listinfo/dm-devel
> 
> 


Hello Ben,

Then I believe this rport's with wrong dev_loss_tmo are part of
multipathd. Is this correct?

[ 09:46:21 ] root@mysystem:~# ls -l /sys/block | grep rport-3:0-0
lrwxrwxrwx 1 root root 0 Feb  6 17:50 sdaa ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:26/block/sdaa
lrwxrwxrwx 1 root root 0 Feb  6 17:50 sdab ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:27/block/sdab
lrwxrwxrwx 1 root root 0 Feb  6 17:50 sdac ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:28/block/sdac
lrwxrwxrwx 1 root root 0 Feb  6 17:50 sdad ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:29/block/sdad
lrwxrwxrwx 1 root root 0 Feb  6 17:50 sdae ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:30/block/sdae
lrwxrwxrwx 1 root root 0 Feb  6 17:50 sdaf ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:31/block/sdaf
lrwxrwxrwx 1 root root 0 Feb  6 17:50 sdag ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:32/block/sdag
lrwxrwxrwx 1 root root 0 Feb  6 17:50 sdah ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:33/block/sdah
lrwxrwxrwx 1 root root 0 Feb  6 17:50 sdai ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:34/block/sdai
lrwxrwxrwx 1 root root 0 Feb  6 17:50 sdaj ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:35/block/sdaj
lrwxrwxrwx 1 root root 0 Feb  6 17:50 sdak ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:36/block/sdak
lrwxrwxrwx 1 root root 0 Feb  6 17:50 sdal ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:37/block/sdal
lrwxrwxrwx 1 root root 0 Nov 27 15:34 sdaug ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:0/block/sdaug
lrwxrwxrwx 1 root root 0 Nov 27 15:34 sdauh ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:1/block/sdauh
lrwxrwxrwx 1 root root 0 Nov 27 15:34 sdauj ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:3/block/sdauj
lrwxrwxrwx 1 root root 0 Nov 27 15:34 sdauk ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:4/block/sdauk
lrwxrwxrwx 1 root root 0 Nov 27 15:34 sdaul ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:5/block/sdaul
lrwxrwxrwx 1 root root 0 Nov 27 15:34 sdaum ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:6/block/sdaum
lrwxrwxrwx 1 root root 0 Nov 27 15:34 sdaun ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:7/block/sdaun
lrwxrwxrwx 1 root root 0 Nov 27 15:34 sdauo ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:8/block/sdauo
lrwxrwxrwx 1 root root 0 Nov 27 15:34 sdaup ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:9/block/sdaup
lrwxrwxrwx 1 root root 0 Nov 27 15:34 sdauq ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:10/block/sdauq
(...)

Best regards,

Bruno

Re: multipathd ignoring dev_loss_tmo setting

[Benjamin Marzinski]

On Mon, Mar 04, 2019 at 09:48:59AM +0000, Martins, Bruno O wrote:
> On Fri, 2019-03-01 at 11:35 -0600, Benjamin Marzinski wrote:
> > On Fri, Mar 01, 2019 at 10:04:52AM +0000, Martins, Bruno O wrote:
> > > On Thu, 2019-02-28 at 13:53 -0600, Benjamin Marzinski wrote:
> > > > On Thu, Feb 28, 2019 at 11:38:22AM +0000, Martins, Bruno O wrote:
> > > > > Hello guys,
> > > > > 
> > > > > I am trying to modify /etc/multipath.conf on my system so that
> > > > > the
> > > > > parameter 'dev_loss_tmo' is changed from the default value.
> > > > > 
> > > > > However, when checking the value currently in use I am getting
> > > > > the
> > > > > wrong value (which is '30') for some of the remote ports:
> > > > > 
> > > > 
> > > > Are you sure those rports are used by multipath devices?
> > > > multipath
> > > > only
> > > > changes dev_loss_tmo for rports associated with a multipath path
> > > > device.
> > > > 
> > > > -Ben
> > > >  
> 
> Hello Ben,
> 
> Then I believe this rport's with wrong dev_loss_tmo are part of
> multipathd. Is this correct?

yeah, that looks correct. If you run

# multipath -r

does that correct the issue?  If not, can you attach the result of
running

# multipath -rv 3

-Ben

> 
> Best regards,
> 
> Bruno

Re: multipathd ignoring dev_loss_tmo setting

[Martin Wilck]

On Thu, 2019-02-28 at 11:38 +0000,  Martins, Bruno O wrote:
> Hello guys,
> 
> I am trying to modify /etc/multipath.conf on my system so that the
> parameter 'dev_loss_tmo' is changed from the default value.
> 
> My multipath.conf file contains the following:
> 
> defaults {
>         verbosity 2
>         polling_interval 5
>         max_polling_interval 10
>         multipath_dir "/lib64/multipath"
>         path_selector "round-robin 0"
>         path_grouping_policy "failover"
>         uid_attribute "ID_SERIAL"
>         prio "const"
>         prio_args ""
>         features "0"
>         path_checker "directio"
>         alias_prefix "mpath"
>         failback "manual"
>         rr_min_io 1000
>         rr_min_io_rq 1
>         max_fds "max"
>         rr_weight "uniform"
>         no_path_retry "fail"
>         queue_without_daemon "no"
>         checker_timeout 15
>         flush_on_last_del "no"
>         user_friendly_names "yes"
>         fast_io_fail_tmo 5
>         dev_loss_tmo 10
>         bindings_file "/etc/multipath/bindings"
>         wwids_file /etc/multipath/wwids
>         log_checker_err always
>         retain_attached_hw_handler no
>         detect_prio no
> }
> 
> However, when checking the value currently in use I am getting the
> wrong value (which is '30') for some of the remote ports:
> 
> for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do
> d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f);
> done
> 
> rport-3:0-0:0x5742b0f00007c500:10
> rport-3:0-1:0x5742b0f00007c500:10
> rport-3:0-2:0x5742b0f00007c500:10
> rport-3:0-3:0x5000097408369800:30
> rport-3:0-4:0x500009757804cbff:30
> rport-4:0-0:0x5742b0f00007c500:10
> rport-4:0-1:0x5742b0f00007c500:10
> rport-4:0-2:0x5000097408369800:30
> rport-4:0-3:0x5742b0f00007c500:10
> rport-4:0-4:0x500009757804cbff:30
> rport-5:0-0:0x5742b0f00007c500:10
> rport-5:0-1:0x5742b0f00007c500:10
> rport-5:0-2:0x5742b0f00007c500:10
> rport-5:0-3:0x5000097408369800:30
> rport-5:0-4:0x500009757804cbff:30
> rport-6:0-0:0x5742b0f00007c500:10
> rport-6:0-1:0x5742b0f00007c500:10
> rport-6:0-2:0x5000097408369800:30
> rport-6:0-3:0x5742b0f00007c500:10
> rport-6:0-4:0x500009757804cbff:30
> 
> systool is giving me the same information:
> 
> systool -c fc_remote_ports -v | grep dev_loss_tmo
> 
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "10"
> 
> 
> > 
> > I am using the following versions:
> > 
> > rpm -qa multipath-tools
> > multipath-tools-0.4.9-109.1
> > 
> > uname -a
> > Linux mysystem 3.0.101-63-default #1 SMP Tue Jun 23 16:02:31 UTC
> 2015
> > (4b89d0c) x86_64 x86_64 x86_64 GNU/Linux
> > 
> > Thanks for your help!
> > 
> > Kind regards,
> > 
> > Bruno
> > 
> > --
> > dm-devel mailing list
> > dm-devel@redhat.com
> > https://www.redhat.com/mailman/listinfo/dm-devel
> > 
> 
> 
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "30"
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "30"
>     dev_loss_tmo        = "30"
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "30"
>     dev_loss_tmo        = "10"
>     dev_loss_tmo        = "30"
>     dev_loss_tmo        = "30"
>     dev_loss_tmo        = "30"
>     dev_loss_tmo        = "30"
> 
> Where is this value coming from? May this be a bug? I couldn't find
> anything useful on the Internet regarding this.

It'd be very helpful if you could upload "multipath -v3" (or multipathd
with verbosity 3) logs somewhere.

It looks as if you're using some SLE11 variant, so maybe you want to
open a support case?

Another question would be why you want such a low dev_loss_tmo. It's
not generally recommended, because on the kernel side, removing and re-
adding a device is a lot more complex than disabling and re-enabling
it. The fast_io_fail_tmo should provide you with quick path failover
already. My recommendation is to set dev_loss_tmo to a value which
would, in the given data center, indicate that the device loss is
really not due to a temporary outage but due to a permantly removed
device (e.g. permanent storage configuration change). So basically, the
dev_loss_tmo shouldn't be shorter than the admin's lunch break.

Martin

Re: multipathd ignoring dev_loss_tmo setting

[Martins, Bruno O]

On Mon, 2019-03-04 at 13:09 +0100, Martin Wilck wrote:
> On Thu, 2019-02-28 at 11:38 +0000,  Martins, Bruno O wrote:
> > Hello guys,
> > 
> > I am trying to modify /etc/multipath.conf on my system so that the
> > parameter 'dev_loss_tmo' is changed from the default value.
> > 
> > My multipath.conf file contains the following:
> > 
> > defaults {
> >         verbosity 2
> >         polling_interval 5
> >         max_polling_interval 10
> >         multipath_dir "/lib64/multipath"
> >         path_selector "round-robin 0"
> >         path_grouping_policy "failover"
> >         uid_attribute "ID_SERIAL"
> >         prio "const"
> >         prio_args ""
> >         features "0"
> >         path_checker "directio"
> >         alias_prefix "mpath"
> >         failback "manual"
> >         rr_min_io 1000
> >         rr_min_io_rq 1
> >         max_fds "max"
> >         rr_weight "uniform"
> >         no_path_retry "fail"
> >         queue_without_daemon "no"
> >         checker_timeout 15
> >         flush_on_last_del "no"
> >         user_friendly_names "yes"
> >         fast_io_fail_tmo 5
> >         dev_loss_tmo 10
> >         bindings_file "/etc/multipath/bindings"
> >         wwids_file /etc/multipath/wwids
> >         log_checker_err always
> >         retain_attached_hw_handler no
> >         detect_prio no
> > }
> > 
> > However, when checking the value currently in use I am getting the
> > wrong value (which is '30') for some of the remote ports:
> > 
> > for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do
> > d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f);
> > done
> > 
> > rport-3:0-0:0x5742b0f00007c500:10
> > rport-3:0-1:0x5742b0f00007c500:10
> > rport-3:0-2:0x5742b0f00007c500:10
> > rport-3:0-3:0x5000097408369800:30
> > rport-3:0-4:0x500009757804cbff:30
> > rport-4:0-0:0x5742b0f00007c500:10
> > rport-4:0-1:0x5742b0f00007c500:10
> > rport-4:0-2:0x5000097408369800:30
> > rport-4:0-3:0x5742b0f00007c500:10
> > rport-4:0-4:0x500009757804cbff:30
> > rport-5:0-0:0x5742b0f00007c500:10
> > rport-5:0-1:0x5742b0f00007c500:10
> > rport-5:0-2:0x5742b0f00007c500:10
> > rport-5:0-3:0x5000097408369800:30
> > rport-5:0-4:0x500009757804cbff:30
> > rport-6:0-0:0x5742b0f00007c500:10
> > rport-6:0-1:0x5742b0f00007c500:10
> > rport-6:0-2:0x5000097408369800:30
> > rport-6:0-3:0x5742b0f00007c500:10
> > rport-6:0-4:0x500009757804cbff:30
> > 
> > systool is giving me the same information:
> > 
> > systool -c fc_remote_ports -v | grep dev_loss_tmo
> > 
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "10"
> > 
> > 
> > > I am using the following versions:
> > > 
> > > rpm -qa multipath-tools
> > > multipath-tools-0.4.9-109.1
> > > 
> > > uname -a
> > > Linux mysystem 3.0.101-63-default #1 SMP Tue Jun 23 16:02:31 UTC
> > 
> > 2015
> > > (4b89d0c) x86_64 x86_64 x86_64 GNU/Linux
> > > 
> > > Thanks for your help!
> > > 
> > > Kind regards,
> > > 
> > > Bruno
> > > 
> > > --
> > > dm-devel mailing list
> > > dm-devel@redhat.com
> > > 
> > > https://www.redhat.com/mailman/listinfo/dm-devel
> > > 
> > > 
> > 
> > 
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "30"
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "30"
> >     dev_loss_tmo        = "30"
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "30"
> >     dev_loss_tmo        = "10"
> >     dev_loss_tmo        = "30"
> >     dev_loss_tmo        = "30"
> >     dev_loss_tmo        = "30"
> >     dev_loss_tmo        = "30"
> > 
> > Where is this value coming from? May this be a bug? I couldn't find
> > anything useful on the Internet regarding this.
> 
> It'd be very helpful if you could upload "multipath -v3" (or
> multipathd
> with verbosity 3) logs somewhere.
> 
> It looks as if you're using some SLE11 variant, so maybe you want to
> open a support case?
> 
> Another question would be why you want such a low dev_loss_tmo. It's
> not generally recommended, because on the kernel side, removing and
> re-
> adding a device is a lot more complex than disabling and re-enabling
> it. The fast_io_fail_tmo should provide you with quick path failover
> already. My recommendation is to set dev_loss_tmo to a value which
> would, in the given data center, indicate that the device loss is
> really not due to a temporary outage but due to a permantly removed
> device (e.g. permanent storage configuration change). So basically,
> the
> dev_loss_tmo shouldn't be shorter than the admin's lunch break.
> 
> Martin
> 
> 
> 
> 

Hello Martin,

Yes, I'm using SuSE:

[ 14:01:44 ] root@mysystem:/tmp# cat /etc/SuSE-release 
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 4

The thing here is that my applications are crashing due to multipath
issues on my Oracle DB cluster, with errors like these:

[ 13:59:27 ] root@mysystem:~# cat /var/log/messages | grep multipath |
head -n 20
Mar  2 23:00:36 mysystem multipathd: sdayi: failed to set rport to
'Blocked', error 2
Mar  2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk1: sdayi -
tur checker timed out
Mar  2 23:00:36 mysystem multipathd: checker failed path 67:1376 in map
BPM1ADB1REDO1DG-hdisk1
Mar  2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk1: remaining
active paths: 3
Mar  2 23:00:36 mysystem multipathd: sdayj: failed to set rport to
'Blocked', error 2
Mar  2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk2: sdayj -
tur checker timed out
Mar  2 23:00:36 mysystem multipathd: checker failed path 67:1392 in map
BPM1ADB1REDO1DG-hdisk2
Mar  2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk2: remaining
active paths: 3
Mar  2 23:00:36 mysystem multipathd: sdayk: failed to set rport to
'Blocked', error 2
Mar  2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk3: sdayk -
tur checker timed out
Mar  2 23:00:36 mysystem multipathd: checker failed path 67:1408 in map
BPM1ADB1REDO1DG-hdisk3
Mar  2 23:00:36 mysystem kernel: [9249542.734463] device-mapper:
multipath: Failing path 67:1376.
Mar  2 23:00:48 mysystem kernel: [9249542.734701] device-mapper:
multipath: Failing path 67:1392.
Mar  2 23:00:48 mysystem kernel: [9249542.734925] device-mapper:
multipath: Failing path 67:1408.
Mar  2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk3: remaining
active paths: 3
Mar  2 23:00:48 mysystem multipathd: sdayo: failed to set rport to
'Blocked', error 2
Mar  2 23:00:48 mysystem multipathd: BPM1ADB1REDO2DG-hdisk2: sdayo -
tur checker timed out
Mar  2 23:00:48 mysystem multipathd: checker failed path 67:1472 in map
BPM1ADB1REDO2DG-hdisk2
Mar  2 23:00:48 mysystem multipathd: BPM1ADB1REDO2DG-hdisk2: remaining
active paths: 3
Mar  2 23:00:48 mysystem multipathd: sdayp: failed to set rport to
'Blocked', error 2

Output of 'multipath -v3' is available here:
https://paste.gnome.org/pojggla8w

Thanks for your cooperation!

Best regards,

Bruno

Re: multipathd ignoring dev_loss_tmo setting

[Martin Wilck]

On Mon, 2019-03-04 at 14:09 +0000, Martins, Bruno O wrote:
> On Mon, 2019-03-04 at 13:09 +0100, Martin Wilck wrote:
> > On Thu, 2019-02-28 at 11:38 +0000,  Martins, Bruno O wrote:
> > > Hello guys,
> > > 
> > > I am trying to modify /etc/multipath.conf on my system so that
> > > the
> > > parameter 'dev_loss_tmo' is changed from the default value.
> > > 
> > > My multipath.conf file contains the following:
> > > 
> > > defaults {
> > >         verbosity 2
> > >         polling_interval 5
> > >         max_polling_interval 10
> > >         multipath_dir "/lib64/multipath"
> > >         path_selector "round-robin 0"
> > >         path_grouping_policy "failover"
> > >         uid_attribute "ID_SERIAL"
> > >         prio "const"
> > >         prio_args ""
> > >         features "0"
> > >         path_checker "directio"
> > >         alias_prefix "mpath"
> > >         failback "manual"
> > >         rr_min_io 1000
> > >         rr_min_io_rq 1
> > >         max_fds "max"
> > >         rr_weight "uniform"
> > >         no_path_retry "fail"
> > >         queue_without_daemon "no"
> > >         checker_timeout 15
> > >         flush_on_last_del "no"
> > >         user_friendly_names "yes"
> > >         fast_io_fail_tmo 5
> > >         dev_loss_tmo 10
> > >         bindings_file "/etc/multipath/bindings"
> > >         wwids_file /etc/multipath/wwids
> > >         log_checker_err always
> > >         retain_attached_hw_handler no
> > >         detect_prio no
> > > }
> > > 
> > > However, when checking the value currently in use I am getting
> > > the
> > > wrong value (which is '30') for some of the remote ports:
> > > 
> > > for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do
> > > d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat
> > > $f);
> > > done
> > > 
> > > rport-3:0-0:0x5742b0f00007c500:10
> > > rport-3:0-1:0x5742b0f00007c500:10
> > > rport-3:0-2:0x5742b0f00007c500:10
> > > rport-3:0-3:0x5000097408369800:30
> > > rport-3:0-4:0x500009757804cbff:30
> > > rport-4:0-0:0x5742b0f00007c500:10
> > > rport-4:0-1:0x5742b0f00007c500:10
> > > rport-4:0-2:0x5000097408369800:30
> > > rport-4:0-3:0x5742b0f00007c500:10
> > > rport-4:0-4:0x500009757804cbff:30
> > > rport-5:0-0:0x5742b0f00007c500:10
> > > rport-5:0-1:0x5742b0f00007c500:10
> > > rport-5:0-2:0x5742b0f00007c500:10
> > > rport-5:0-3:0x5000097408369800:30
> > > rport-5:0-4:0x500009757804cbff:30
> > > rport-6:0-0:0x5742b0f00007c500:10
> > > rport-6:0-1:0x5742b0f00007c500:10
> > > rport-6:0-2:0x5000097408369800:30
> > > rport-6:0-3:0x5742b0f00007c500:10
> > > rport-6:0-4:0x500009757804cbff:30
> > > 
> > > systool is giving me the same information:
> > > 
> > > systool -c fc_remote_ports -v | grep dev_loss_tmo
> > > 
> > >     dev_loss_tmo        = "10"
> > >     dev_loss_tmo        = "10"
> > >     dev_loss_tmo        = "10"
> > >     dev_loss_tmo        = "10"
> > > 
> > > 
> > > > I am using the following versions:
> > > > 
> > > > rpm -qa multipath-tools
> > > > multipath-tools-0.4.9-109.1
> > > > 
> > > > uname -a
> > > > Linux mysystem 3.0.101-63-default #1 SMP Tue Jun 23 16:02:31
> > > > UTC
> > > 
> > > 2015
> > > > (4b89d0c) x86_64 x86_64 x86_64 GNU/Linux
> > > > 
> > > > Thanks for your help!
> > > > 
> > > > Kind regards,
> > > > 
> > > > Bruno
> > > > 
> > > > --
> > > > dm-devel mailing list
> > > > dm-devel@redhat.com
> > > > 
> > > > https://www.redhat.com/mailman/listinfo/dm-devel
> > > > 
> > > > 
> > > 
> > >     dev_loss_tmo        = "10"
> > >     dev_loss_tmo        = "10"
> > >     dev_loss_tmo        = "10"
> > >     dev_loss_tmo        = "10"
> > >     dev_loss_tmo        = "10"
> > >     dev_loss_tmo        = "30"
> > >     dev_loss_tmo        = "10"
> > >     dev_loss_tmo        = "30"
> > >     dev_loss_tmo        = "30"
> > >     dev_loss_tmo        = "10"
> > >     dev_loss_tmo        = "30"
> > >     dev_loss_tmo        = "10"
> > >     dev_loss_tmo        = "30"
> > >     dev_loss_tmo        = "30"
> > >     dev_loss_tmo        = "30"
> > >     dev_loss_tmo        = "30"
> > > 
> > > Where is this value coming from? May this be a bug? I couldn't
> > > find
> > > anything useful on the Internet regarding this.
> > 
> > It'd be very helpful if you could upload "multipath -v3" (or
> > multipathd
> > with verbosity 3) logs somewhere.
> > 
> > It looks as if you're using some SLE11 variant, so maybe you want
> > to
> > open a support case?
> > 
> > Another question would be why you want such a low dev_loss_tmo.
> > It's
> > not generally recommended, because on the kernel side, removing and
> > re-
> > adding a device is a lot more complex than disabling and re-
> > enabling
> > it. The fast_io_fail_tmo should provide you with quick path
> > failover
> > already. My recommendation is to set dev_loss_tmo to a value which
> > would, in the given data center, indicate that the device loss is
> > really not due to a temporary outage but due to a permantly removed
> > device (e.g. permanent storage configuration change). So basically,
> > the
> > dev_loss_tmo shouldn't be shorter than the admin's lunch break.
> > 
> > Martin
> > 
> > 
> > 
> > 
> 
> Hello Martin,
> 
> Yes, I'm using SuSE:
> 
> [ 14:01:44 ] root@mysystem:/tmp# cat /etc/SuSE-release 
> SUSE Linux Enterprise Server 11 (x86_64)
> VERSION = 11
> PATCHLEVEL = 4
> 
> The thing here is that my applications are crashing due to multipath
> issues on my Oracle DB cluster, with errors like these:
> 
> [ 13:59:27 ] root@mysystem:~# cat /var/log/messages | grep multipath
> |
> head -n 20
> Mar  2 23:00:36 mysystem multipathd: sdayi: failed to set rport to
> 'Blocked', error 2
> Mar  2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk1: sdayi -
> tur checker timed out
> Mar  2 23:00:36 mysystem multipathd: checker failed path 67:1376 in
> map
> BPM1ADB1REDO1DG-hdisk1From git://anongit.freedesktop.org/xorg/xserver
> 
> Mar  2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk1:
> remaining
> active paths: 3
> Mar  2 23:00:36 mysystem multipathd: sdayj: failed to set rport to
> 'Blocked', error 2
> Mar  2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk2: sdayj -
> tur checker timed out
> Mar  2 23:00:36 mysystem multipathd: checker failed path 67:1392 in
> map
> BPM1ADB1REDO1DG-hdisk2
> Mar  2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk2:
> remaining
> active paths: 3
> Mar  2 23:00:36 mysystem multipathd: sdayk: failed to set rport to
> 'Blocked', error 2
> Mar  2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk3: sdayk -
> tur checker timed out
> Mar  2 23:00:36 mysystem multipathd: checker failed path 67:1408 in
> map
> BPM1ADB1REDO1DG-hdisk3
> Mar  2 23:00:36 mysystem kernel: [9249542.734463] device-mapper:
> multipath: Failing path 67:1376.
> Mar  2 23:00:48 mysystem kernel: [9249542.734701] device-mapper:
> multipath: Failing path 67:1392.
> Mar  2 23:00:48 mysystem kernel: [9249542.734925] device-mapper:
> multipath: Failing path 67:1408.
> Mar  2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk3:
> remaining
> active paths: 3
> Mar  2 23:00:48 mysystem multipathd: sdayo: failed to set rport to
> 'Blocked', error 2
> Mar  2 23:00:48 mysystem multipathd: BPM1ADB1REDO2DG-hdisk2: sdayo -
> tur checker timed oute
> Mar  2 23:00:48 mysystem multipathd: checker failed path 67:1472 in
> map
> BPM1ADB1REDO2DG-hdisk2
> Mar  2 23:00:48 mysystem multipathd: BPM1ADB1REDO2DG-hdisk2:
> remaining
> active paths: 3
> Mar  2 23:00:48 mysystem multipathd: sdayp: failed to set rport to
> 'Blocked', error 2
> 
> Output of 'multipath -v3' is available here:
> https://paste.gnome.org/pojggla8w
> 

Your logs show that the SYMMETRIX LUNs have a dev_loss_tmo of 30 and
the NFINIDAT LUNs have 10. That makes sense because the old multipath
version you are using is lacking a hwtable entry for NFINIDAT,
therefore the defaults are being used. But the SYMMETRIX has a hwtable
entry:

	/*
	 * EMC / Clariion controller family
	 *
	 * Maintainer : Edward Goggin, EMC
	 * Mail : egoggin@emc.com
	 */
	{
		.vendor        = "EMC",
		.product       = "SYMMETRIX",
		.features      = DEFAULT_FEATURES,
		.hwhandler     = DEFAULT_HWHANDLER,
		.pgpolicy      = MULTIBUS,
		.pgfailback    = FAILBACK_UNDEF,
		.rr_weight     = RR_WEIGHT_NONE,
		.no_path_retry = 6,
		.checker_name  = TUR,
		.prio_name     = DEFAULT_PRIO,
		.prio_args     = NULL,
	},

The point is here ".no_path_retry = 6". This overrides your
"no_path_retry" setting from the defaults section. multipath sets the
dev_loss_tmo such that it is at least (no_path_retry *
polling_interval), which is 30 in your case.

If you want to override this, you need to create a device entry in
multipath.conf:

devices {
	device {
		vendor EMC
		product SYMMETRIX
		no_path_retry fail
		}
	}
}

Wrt the failures you are reporting, it seems to me that you're not
using the latest updates SUSE has released for SLE-11. I am pretty
certain that "fixing" the dev_loss_tmo to 10s for SYMMETRIX would not
fix that issue.

It seems to me that you should really contact your SUSE support team.
I'll be happy to support you further through the regular channels.

Regards,
Martin