* multipathd ignoring dev_loss_tmo setting
@ 2019-02-28 11:38 Martins, Bruno O
2019-02-28 12:08 ` Steffen Maier
` (2 more replies)
0 siblings, 3 replies; 10+ messages in thread
From: Martins, Bruno O @ 2019-02-28 11:38 UTC (permalink / raw)
To: dm-devel@redhat.com
Hello guys,
I am trying to modify /etc/multipath.conf on my system so that the
parameter 'dev_loss_tmo' is changed from the default value.
My multipath.conf file contains the following:
defaults {
verbosity 2
polling_interval 5
max_polling_interval 10
multipath_dir "/lib64/multipath"
path_selector "round-robin 0"
path_grouping_policy "failover"
uid_attribute "ID_SERIAL"
prio "const"
prio_args ""
features "0"
path_checker "directio"
alias_prefix "mpath"
failback "manual"
rr_min_io 1000
rr_min_io_rq 1
max_fds "max"
rr_weight "uniform"
no_path_retry "fail"
queue_without_daemon "no"
checker_timeout 15
flush_on_last_del "no"
user_friendly_names "yes"
fast_io_fail_tmo 5
dev_loss_tmo 10
bindings_file "/etc/multipath/bindings"
wwids_file /etc/multipath/wwids
log_checker_err always
retain_attached_hw_handler no
detect_prio no
}
However, when checking the value currently in use I am getting the
wrong value (which is '30') for some of the remote ports:
for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do
d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f);
done
rport-3:0-0:0x5742b0f00007c500:10
rport-3:0-1:0x5742b0f00007c500:10
rport-3:0-2:0x5742b0f00007c500:10
rport-3:0-3:0x5000097408369800:30
rport-3:0-4:0x500009757804cbff:30
rport-4:0-0:0x5742b0f00007c500:10
rport-4:0-1:0x5742b0f00007c500:10
rport-4:0-2:0x5000097408369800:30
rport-4:0-3:0x5742b0f00007c500:10
rport-4:0-4:0x500009757804cbff:30
rport-5:0-0:0x5742b0f00007c500:10
rport-5:0-1:0x5742b0f00007c500:10
rport-5:0-2:0x5742b0f00007c500:10
rport-5:0-3:0x5000097408369800:30
rport-5:0-4:0x500009757804cbff:30
rport-6:0-0:0x5742b0f00007c500:10
rport-6:0-1:0x5742b0f00007c500:10
rport-6:0-2:0x5000097408369800:30
rport-6:0-3:0x5742b0f00007c500:10
rport-6:0-4:0x500009757804cbff:30
systool is giving me the same information:
systool -c fc_remote_ports -v | grep dev_loss_tmo
dev_loss_tmo = "10"
dev_loss_tmo = "10"
dev_loss_tmo = "10"
dev_loss_tmo = "10"
dev_loss_tmo = "10"
dev_loss_tmo = "10"
dev_loss_tmo = "10"
dev_loss_tmo = "10"
dev_loss_tmo = "10"
dev_loss_tmo = "30"
dev_loss_tmo = "10"
dev_loss_tmo = "30"
dev_loss_tmo = "30"
dev_loss_tmo = "10"
dev_loss_tmo = "30"
dev_loss_tmo = "10"
dev_loss_tmo = "30"
dev_loss_tmo = "30"
dev_loss_tmo = "30"
dev_loss_tmo = "30"
Where is this value coming from? May this be a bug? I couldn't find
anything useful on the Internet regarding this.
I am using the following versions:
rpm -qa multipath-tools
multipath-tools-0.4.9-109.1
uname -a
Linux mysystem 3.0.101-63-default #1 SMP Tue Jun 23 16:02:31 UTC 2015
(4b89d0c) x86_64 x86_64 x86_64 GNU/Linux
Thanks for your help!
Kind regards,
Bruno
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: multipathd ignoring dev_loss_tmo setting
2019-02-28 11:38 multipathd ignoring dev_loss_tmo setting Martins, Bruno O
@ 2019-02-28 12:08 ` Steffen Maier
2019-02-28 19:53 ` Benjamin Marzinski
2019-03-04 12:09 ` Martin Wilck
2 siblings, 0 replies; 10+ messages in thread
From: Steffen Maier @ 2019-02-28 12:08 UTC (permalink / raw)
To: dm-devel
On 02/28/2019 12:38 PM, Martins, Bruno O wrote:
> I am trying to modify /etc/multipath.conf on my system so that the
> parameter 'dev_loss_tmo' is changed from the default value.
>
> My multipath.conf file contains the following:
>
> defaults {
> no_path_retry "fail"
> queue_without_daemon "no"
> checker_timeout 15
> flush_on_last_del "no"
> user_friendly_names "yes"
> fast_io_fail_tmo 5
> dev_loss_tmo 10
> }
If I'm not mistaken, the default settings could be overridden by (built-in)
device section settings.
However, in your case, where you see some rports having your desired setting
but others not, it looks like it's due to multipath-tools only setting the
_tmo's for those rports where it has paths it manages in path groups.
So if you don't have multipath managed paths behind e.g.
rport-3:0-3:0x5000097408369800 it remains to use the rport's default tmo coming
from scsi_transport_fc in the kernel.
If so, we typically don't care for those "unmanaged" ports.
> However, when checking the value currently in use I am getting the
> wrong value (which is '30') for some of the remote ports:
>
> for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do
> d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f);
> done
>
> rport-3:0-0:0x5742b0f00007c500:10
> rport-3:0-1:0x5742b0f00007c500:10
> rport-3:0-2:0x5742b0f00007c500:10
> rport-3:0-3:0x5000097408369800:30
> rport-3:0-4:0x500009757804cbff:30
> systool is giving me the same information:
>
> systool -c fc_remote_ports -v | grep dev_loss_tmo
> dev_loss_tmo = "10"
> dev_loss_tmo = "30"
> dev_loss_tmo = "10"
> dev_loss_tmo = "30"
> dev_loss_tmo = "30"
> dev_loss_tmo = "30"
> dev_loss_tmo = "30"
>
> Where is this value coming from? May this be a bug? I couldn't find
> anything useful on the Internet regarding this.
>
> I am using the following versions:
>
> rpm -qa multipath-tools
> multipath-tools-0.4.9-109.1
>
> uname -a
> Linux mysystem 3.0.101-63-default #1 SMP Tue Jun 23 16:02:31 UTC 2015
> (4b89d0c) x86_64 x86_64 x86_64 GNU/Linux
--
Mit freundlichen Gruessen / Kind regards
Steffen Maier
Linux on IBM Z Development
https://www.ibm.com/privacy/us/en/
IBM Deutschland Research & Development GmbH
Vorsitzender des Aufsichtsrats: Matthias Hartmann
Geschaeftsfuehrung: Dirk Wittkopp
Sitz der Gesellschaft: Boeblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: multipathd ignoring dev_loss_tmo setting
2019-02-28 11:38 multipathd ignoring dev_loss_tmo setting Martins, Bruno O
2019-02-28 12:08 ` Steffen Maier
@ 2019-02-28 19:53 ` Benjamin Marzinski
2019-03-01 10:04 ` Martins, Bruno O
2019-03-04 12:09 ` Martin Wilck
2 siblings, 1 reply; 10+ messages in thread
From: Benjamin Marzinski @ 2019-02-28 19:53 UTC (permalink / raw)
To: Martins, Bruno O; +Cc: dm-devel@redhat.com
On Thu, Feb 28, 2019 at 11:38:22AM +0000, Martins, Bruno O wrote:
> Hello guys,
>
> I am trying to modify /etc/multipath.conf on my system so that the
> parameter 'dev_loss_tmo' is changed from the default value.
>
> However, when checking the value currently in use I am getting the
> wrong value (which is '30') for some of the remote ports:
>
Are you sure those rports are used by multipath devices? multipath only
changes dev_loss_tmo for rports associated with a multipath path device.
-Ben
> for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do
> d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f);
> done
>
> rport-3:0-0:0x5742b0f00007c500:10
> rport-3:0-1:0x5742b0f00007c500:10
> rport-3:0-2:0x5742b0f00007c500:10
> rport-3:0-3:0x5000097408369800:30
> rport-3:0-4:0x500009757804cbff:30
> rport-4:0-0:0x5742b0f00007c500:10
> rport-4:0-1:0x5742b0f00007c500:10
> rport-4:0-2:0x5000097408369800:30
> rport-4:0-3:0x5742b0f00007c500:10
> rport-4:0-4:0x500009757804cbff:30
> rport-5:0-0:0x5742b0f00007c500:10
> rport-5:0-1:0x5742b0f00007c500:10
> rport-5:0-2:0x5742b0f00007c500:10
> rport-5:0-3:0x5000097408369800:30
> rport-5:0-4:0x500009757804cbff:30
> rport-6:0-0:0x5742b0f00007c500:10
> rport-6:0-1:0x5742b0f00007c500:10
> rport-6:0-2:0x5000097408369800:30
> rport-6:0-3:0x5742b0f00007c500:10
> rport-6:0-4:0x500009757804cbff:30
>
> systool is giving me the same information:
>
> systool -c fc_remote_ports -v | grep dev_loss_tmo
>
> dev_loss_tmo = "10"
> dev_loss_tmo = "10"
> dev_loss_tmo = "10"
> dev_loss_tmo = "10"
> dev_loss_tmo = "10"
> dev_loss_tmo = "10"
> dev_loss_tmo = "10"
> dev_loss_tmo = "10"
> dev_loss_tmo = "10"
> dev_loss_tmo = "30"
> dev_loss_tmo = "10"
> dev_loss_tmo = "30"
> dev_loss_tmo = "30"
> dev_loss_tmo = "10"
> dev_loss_tmo = "30"
> dev_loss_tmo = "10"
> dev_loss_tmo = "30"
> dev_loss_tmo = "30"
> dev_loss_tmo = "30"
> dev_loss_tmo = "30"
>
> Where is this value coming from? May this be a bug? I couldn't find
> anything useful on the Internet regarding this.
>
> I am using the following versions:
>
> rpm -qa multipath-tools
> multipath-tools-0.4.9-109.1
>
> uname -a
> Linux mysystem 3.0.101-63-default #1 SMP Tue Jun 23 16:02:31 UTC 2015
> (4b89d0c) x86_64 x86_64 x86_64 GNU/Linux
>
> Thanks for your help!
>
> Kind regards,
>
> Bruno
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: multipathd ignoring dev_loss_tmo setting
2019-02-28 19:53 ` Benjamin Marzinski
@ 2019-03-01 10:04 ` Martins, Bruno O
2019-03-01 17:35 ` Benjamin Marzinski
0 siblings, 1 reply; 10+ messages in thread
From: Martins, Bruno O @ 2019-03-01 10:04 UTC (permalink / raw)
To: bmarzins@redhat.com; +Cc: dm-devel@redhat.com
On Thu, 2019-02-28 at 13:53 -0600, Benjamin Marzinski wrote:
> On Thu, Feb 28, 2019 at 11:38:22AM +0000, Martins, Bruno O wrote:
> > Hello guys,
> >
> > I am trying to modify /etc/multipath.conf on my system so that the
> > parameter 'dev_loss_tmo' is changed from the default value.
> >
> > However, when checking the value currently in use I am getting the
> > wrong value (which is '30') for some of the remote ports:
> >
>
> Are you sure those rports are used by multipath devices? multipath
> only
> changes dev_loss_tmo for rports associated with a multipath path
> device.
>
> -Ben
>
> > for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do
> > d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f);
> > done
> >
> > rport-3:0-0:0x5742b0f00007c500:10
> > rport-3:0-1:0x5742b0f00007c500:10
> > rport-3:0-2:0x5742b0f00007c500:10
> > rport-3:0-3:0x5000097408369800:30
> > rport-3:0-4:0x500009757804cbff:30
> > rport-4:0-0:0x5742b0f00007c500:10
> > rport-4:0-1:0x5742b0f00007c500:10
> > rport-4:0-2:0x5000097408369800:30
> > rport-4:0-3:0x5742b0f00007c500:10
> > rport-4:0-4:0x500009757804cbff:30
> > rport-5:0-0:0x5742b0f00007c500:10
> > rport-5:0-1:0x5742b0f00007c500:10
> > rport-5:0-2:0x5742b0f00007c500:10
> > rport-5:0-3:0x5000097408369800:30
> > rport-5:0-4:0x500009757804cbff:30
> > rport-6:0-0:0x5742b0f00007c500:10
> > rport-6:0-1:0x5742b0f00007c500:10
> > rport-6:0-2:0x5000097408369800:30
> > rport-6:0-3:0x5742b0f00007c500:10
> > rport-6:0-4:0x500009757804cbff:30
> >
> > systool is giving me the same information:
> >
> > systool -c fc_remote_ports -v | grep dev_loss_tmo
> >
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "30"
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "30"
> > dev_loss_tmo = "30"
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "30"
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "30"
> > dev_loss_tmo = "30"
> > dev_loss_tmo = "30"
> > dev_loss_tmo = "30"
> >
> > Where is this value coming from? May this be a bug? I couldn't find
> > anything useful on the Internet regarding this.
> >
> > I am using the following versions:
> >
> > rpm -qa multipath-tools
> > multipath-tools-0.4.9-109.1
> >
> > uname -a
> > Linux mysystem 3.0.101-63-default #1 SMP Tue Jun 23 16:02:31 UTC
> > 2015
> > (4b89d0c) x86_64 x86_64 x86_64 GNU/Linux
> >
> > Thanks for your help!
> >
> > Kind regards,
> >
> > Bruno
> >
> > --
> > dm-devel mailing list
> > dm-devel@redhat.com
> >
> > https://www.redhat.com/mailman/listinfo/dm-devel
> >
Hi Benjamin,
Thanks for your reply!
I believe they are:
[ 10:02:45 ] root@myhost:~# multipath -ll | grep 3:0:3
|- 3:0:3:12 sdblc 128:1568 active ready running
|- 3:0:3:13 sdbnu 132:1664 active ready running
|- 3:0:3:18 sdbre 66:1792 active ready running
|- 3:0:3:1 sdbkg 70:1728 active ready running
|- 3:0:3:2 sdbnv 132:1680 active ready running
|- 3:0:3:20 sdbrg 66:1824 active ready running
|- 3:0:3:17 sdbpg 134:1760 active ready running
|- 3:0:3:16 sdbpf 134:1744 active ready running
|- 3:0:3:11 sdbkf 70:1712 active ready running
|- 3:0:3:19 sdbrf 66:1808 active ready running
|- 3:0:3:14 sdbnw 132:1696 active ready running
|- 3:0:3:15 sdbpe 134:1728 active ready running
Is this the best way to check that information?
BR,
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: multipathd ignoring dev_loss_tmo setting
2019-03-01 10:04 ` Martins, Bruno O
@ 2019-03-01 17:35 ` Benjamin Marzinski
2019-03-04 9:48 ` Martins, Bruno O
0 siblings, 1 reply; 10+ messages in thread
From: Benjamin Marzinski @ 2019-03-01 17:35 UTC (permalink / raw)
To: Martins, Bruno O; +Cc: dm-devel@redhat.com
On Fri, Mar 01, 2019 at 10:04:52AM +0000, Martins, Bruno O wrote:
> On Thu, 2019-02-28 at 13:53 -0600, Benjamin Marzinski wrote:
> > On Thu, Feb 28, 2019 at 11:38:22AM +0000, Martins, Bruno O wrote:
> > > Hello guys,
> > >
> > > I am trying to modify /etc/multipath.conf on my system so that the
> > > parameter 'dev_loss_tmo' is changed from the default value.
> > >
> > > However, when checking the value currently in use I am getting the
> > > wrong value (which is '30') for some of the remote ports:
> > >
> >
> > Are you sure those rports are used by multipath devices? multipath
> > only
> > changes dev_loss_tmo for rports associated with a multipath path
> > device.
> >
> > -Ben
> >
>
> Hi Benjamin,
>
> Thanks for your reply!
>
> I believe they are:
>
> [ 10:02:45 ] root@myhost:~# multipath -ll | grep 3:0:3
> |- 3:0:3:12 sdblc 128:1568 active ready running
> |- 3:0:3:13 sdbnu 132:1664 active ready running
> |- 3:0:3:18 sdbre 66:1792 active ready running
> |- 3:0:3:1 sdbkg 70:1728 active ready running
> |- 3:0:3:2 sdbnv 132:1680 active ready running
> |- 3:0:3:20 sdbrg 66:1824 active ready running
> |- 3:0:3:17 sdbpg 134:1760 active ready running
> |- 3:0:3:16 sdbpf 134:1744 active ready running
> |- 3:0:3:11 sdbkf 70:1712 active ready running
> |- 3:0:3:19 sdbrf 66:1808 active ready running
> |- 3:0:3:14 sdbnw 132:1696 active ready running
> |- 3:0:3:15 sdbpe 134:1728 active ready running
>
> Is this the best way to check that information?
The scsi HBTL isn't guaranteed to line up with the rport id. To find this
out you can either run
# ls -l /sys/block
and then check the rport for your path devices from the link
destination.
For instance
[root@ask-07 block]# ls -l /sys/block
<snip>
lrwxrwxrwx. 1 root root 0 Feb 21 02:45 sdb -> ../devices/pci0000:00/0000:00:0a.0/0000:06:00.0/host16/rport-16:0-0/target16:0:0/16:0:0:0/block/sdb
lrwxrwxrwx. 1 root root 0 Feb 21 02:45 sdc -> ../devices/pci0000:00/0000:00:0a.0/0000:06:00.0/host16/rport-16:0-0/target16:0:0/16:0:0:1/block/sdc
lrwxrwxrwx. 1 root root 0 Feb 21 02:45 sdd -> ../devices/pci0000:00/0000:00:0a.0/0000:06:00.1/host17/rport-17:0-0/target17:0:0/17:0:0:0/block/sdd
lrwxrwxrwx. 1 root root 0 Feb 21 02:45 sde -> ../devices/pci0000:00/0000:00:0a.0/0000:06:00.1/host17/rport-17:0-0/target17:0:0/17:0:0:1/block/sde
<snip>
Here, sdb and sdc are using rport-16:0-0, and sdd and sde are using
rport-17:0-0
Otherwise, you can pick an rport with the wrong dev_loss_tmo, and check
/sys/class/fc_remote_ports/<rport>/device/<target>/
In there there will be a number of scsi HBTL identifiers, for example
# ls /sys/class/fc_remote_ports/rport-16\:0-0/device/target16\:0\:0/
16:0:0:0 16:0:0:1 fc_transport power subsystem uevent
If these HBTL ids (16:0:0:0 and 16:0:0:1) are for multipath path
devices, then multipath should be updating dev_loss_tmo for them. In my
example above, the scsi HBTL does correspond to the rport id, but this
isn't always the case.
-Ben
> BR,
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: multipathd ignoring dev_loss_tmo setting
2019-03-01 17:35 ` Benjamin Marzinski
@ 2019-03-04 9:48 ` Martins, Bruno O
2019-03-12 22:32 ` Benjamin Marzinski
0 siblings, 1 reply; 10+ messages in thread
From: Martins, Bruno O @ 2019-03-04 9:48 UTC (permalink / raw)
To: bmarzins@redhat.com; +Cc: dm-devel@redhat.com
On Fri, 2019-03-01 at 11:35 -0600, Benjamin Marzinski wrote:
> On Fri, Mar 01, 2019 at 10:04:52AM +0000, Martins, Bruno O wrote:
> > On Thu, 2019-02-28 at 13:53 -0600, Benjamin Marzinski wrote:
> > > On Thu, Feb 28, 2019 at 11:38:22AM +0000, Martins, Bruno O wrote:
> > > > Hello guys,
> > > >
> > > > I am trying to modify /etc/multipath.conf on my system so that
> > > > the
> > > > parameter 'dev_loss_tmo' is changed from the default value.
> > > >
> > > > However, when checking the value currently in use I am getting
> > > > the
> > > > wrong value (which is '30') for some of the remote ports:
> > > >
> > >
> > > Are you sure those rports are used by multipath devices?
> > > multipath
> > > only
> > > changes dev_loss_tmo for rports associated with a multipath path
> > > device.
> > >
> > > -Ben
> > >
> >
> > Hi Benjamin,
> >
> > Thanks for your reply!
> >
> > I believe they are:
> >
> > [ 10:02:45 ] root@myhost:~# multipath -ll | grep 3:0:3
> > |- 3:0:3:12 sdblc 128:1568 active ready running
> > |- 3:0:3:13 sdbnu 132:1664 active ready running
> > |- 3:0:3:18 sdbre 66:1792 active ready running
> > |- 3:0:3:1 sdbkg 70:1728 active ready running
> > |- 3:0:3:2 sdbnv 132:1680 active ready running
> > |- 3:0:3:20 sdbrg 66:1824 active ready running
> > |- 3:0:3:17 sdbpg 134:1760 active ready running
> > |- 3:0:3:16 sdbpf 134:1744 active ready running
> > |- 3:0:3:11 sdbkf 70:1712 active ready running
> > |- 3:0:3:19 sdbrf 66:1808 active ready running
> > |- 3:0:3:14 sdbnw 132:1696 active ready running
> > |- 3:0:3:15 sdbpe 134:1728 active ready running
> >
> > Is this the best way to check that information?
>
> The scsi HBTL isn't guaranteed to line up with the rport id. To find
> this
> out you can either run
>
> # ls -l /sys/block
>
> and then check the rport for your path devices from the link
> destination.
>
> For instance
>
> [root@ask-07 block]# ls -l /sys/block
> <snip>
> lrwxrwxrwx. 1 root root 0 Feb 21 02:45 sdb ->
> ../devices/pci0000:00/0000:00:0a.0/0000:06:00.0/host16/rport-16:0-
> 0/target16:0:0/16:0:0:0/block/sdb
> lrwxrwxrwx. 1 root root 0 Feb 21 02:45 sdc ->
> ../devices/pci0000:00/0000:00:0a.0/0000:06:00.0/host16/rport-16:0-
> 0/target16:0:0/16:0:0:1/block/sdc
> lrwxrwxrwx. 1 root root 0 Feb 21 02:45 sdd ->
> ../devices/pci0000:00/0000:00:0a.0/0000:06:00.1/host17/rport-17:0-
> 0/target17:0:0/17:0:0:0/block/sdd
> lrwxrwxrwx. 1 root root 0 Feb 21 02:45 sde ->
> ../devices/pci0000:00/0000:00:0a.0/0000:06:00.1/host17/rport-17:0-
> 0/target17:0:0/17:0:0:1/block/sde
> <snip>
>
> Here, sdb and sdc are using rport-16:0-0, and sdd and sde are using
> rport-17:0-0
>
> Otherwise, you can pick an rport with the wrong dev_loss_tmo, and
> check
>
> /sys/class/fc_remote_ports/<rport>/device/<target>/
>
> In there there will be a number of scsi HBTL identifiers, for example
>
> # ls /sys/class/fc_remote_ports/rport-16\:0-0/device/target16\:0\:0/
> 16:0:0:0 16:0:0:1 fc_transport power subsystem uevent
>
> If these HBTL ids (16:0:0:0 and 16:0:0:1) are for multipath path
> devices, then multipath should be updating dev_loss_tmo for them. In
> my
> example above, the scsi HBTL does correspond to the rport id, but
> this
> isn't always the case.
>
> -Ben
>
> > BR,
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
>
> https://www.redhat.com/mailman/listinfo/dm-devel
>
>
Hello Ben,
Then I believe this rport's with wrong dev_loss_tmo are part of
multipathd. Is this correct?
[ 09:46:21 ] root@mysystem:~# ls -l /sys/block | grep rport-3:0-0
lrwxrwxrwx 1 root root 0 Feb 6 17:50 sdaa ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:26/block/sdaa
lrwxrwxrwx 1 root root 0 Feb 6 17:50 sdab ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:27/block/sdab
lrwxrwxrwx 1 root root 0 Feb 6 17:50 sdac ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:28/block/sdac
lrwxrwxrwx 1 root root 0 Feb 6 17:50 sdad ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:29/block/sdad
lrwxrwxrwx 1 root root 0 Feb 6 17:50 sdae ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:30/block/sdae
lrwxrwxrwx 1 root root 0 Feb 6 17:50 sdaf ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:31/block/sdaf
lrwxrwxrwx 1 root root 0 Feb 6 17:50 sdag ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:32/block/sdag
lrwxrwxrwx 1 root root 0 Feb 6 17:50 sdah ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:33/block/sdah
lrwxrwxrwx 1 root root 0 Feb 6 17:50 sdai ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:34/block/sdai
lrwxrwxrwx 1 root root 0 Feb 6 17:50 sdaj ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:35/block/sdaj
lrwxrwxrwx 1 root root 0 Feb 6 17:50 sdak ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:36/block/sdak
lrwxrwxrwx 1 root root 0 Feb 6 17:50 sdal ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:37/block/sdal
lrwxrwxrwx 1 root root 0 Nov 27 15:34 sdaug ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:0/block/sdaug
lrwxrwxrwx 1 root root 0 Nov 27 15:34 sdauh ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:1/block/sdauh
lrwxrwxrwx 1 root root 0 Nov 27 15:34 sdauj ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:3/block/sdauj
lrwxrwxrwx 1 root root 0 Nov 27 15:34 sdauk ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:4/block/sdauk
lrwxrwxrwx 1 root root 0 Nov 27 15:34 sdaul ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:5/block/sdaul
lrwxrwxrwx 1 root root 0 Nov 27 15:34 sdaum ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:6/block/sdaum
lrwxrwxrwx 1 root root 0 Nov 27 15:34 sdaun ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:7/block/sdaun
lrwxrwxrwx 1 root root 0 Nov 27 15:34 sdauo ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:8/block/sdauo
lrwxrwxrwx 1 root root 0 Nov 27 15:34 sdaup ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:9/block/sdaup
lrwxrwxrwx 1 root root 0 Nov 27 15:34 sdauq ->
../devices/pci0000:00/0000:00:01.0/0000:07:00.0/host3/rport-3:0-
0/target3:0:0/3:0:0:10/block/sdauq
(...)
Best regards,
Bruno
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: multipathd ignoring dev_loss_tmo setting
2019-02-28 11:38 multipathd ignoring dev_loss_tmo setting Martins, Bruno O
2019-02-28 12:08 ` Steffen Maier
2019-02-28 19:53 ` Benjamin Marzinski
@ 2019-03-04 12:09 ` Martin Wilck
2019-03-04 14:09 ` Martins, Bruno O
2 siblings, 1 reply; 10+ messages in thread
From: Martin Wilck @ 2019-03-04 12:09 UTC (permalink / raw)
To: Martins, Bruno O, dm-devel@redhat.com
On Thu, 2019-02-28 at 11:38 +0000, Martins, Bruno O wrote:
> Hello guys,
>
> I am trying to modify /etc/multipath.conf on my system so that the
> parameter 'dev_loss_tmo' is changed from the default value.
>
> My multipath.conf file contains the following:
>
> defaults {
> verbosity 2
> polling_interval 5
> max_polling_interval 10
> multipath_dir "/lib64/multipath"
> path_selector "round-robin 0"
> path_grouping_policy "failover"
> uid_attribute "ID_SERIAL"
> prio "const"
> prio_args ""
> features "0"
> path_checker "directio"
> alias_prefix "mpath"
> failback "manual"
> rr_min_io 1000
> rr_min_io_rq 1
> max_fds "max"
> rr_weight "uniform"
> no_path_retry "fail"
> queue_without_daemon "no"
> checker_timeout 15
> flush_on_last_del "no"
> user_friendly_names "yes"
> fast_io_fail_tmo 5
> dev_loss_tmo 10
> bindings_file "/etc/multipath/bindings"
> wwids_file /etc/multipath/wwids
> log_checker_err always
> retain_attached_hw_handler no
> detect_prio no
> }
>
> However, when checking the value currently in use I am getting the
> wrong value (which is '30') for some of the remote ports:
>
> for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do
> d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f);
> done
>
> rport-3:0-0:0x5742b0f00007c500:10
> rport-3:0-1:0x5742b0f00007c500:10
> rport-3:0-2:0x5742b0f00007c500:10
> rport-3:0-3:0x5000097408369800:30
> rport-3:0-4:0x500009757804cbff:30
> rport-4:0-0:0x5742b0f00007c500:10
> rport-4:0-1:0x5742b0f00007c500:10
> rport-4:0-2:0x5000097408369800:30
> rport-4:0-3:0x5742b0f00007c500:10
> rport-4:0-4:0x500009757804cbff:30
> rport-5:0-0:0x5742b0f00007c500:10
> rport-5:0-1:0x5742b0f00007c500:10
> rport-5:0-2:0x5742b0f00007c500:10
> rport-5:0-3:0x5000097408369800:30
> rport-5:0-4:0x500009757804cbff:30
> rport-6:0-0:0x5742b0f00007c500:10
> rport-6:0-1:0x5742b0f00007c500:10
> rport-6:0-2:0x5000097408369800:30
> rport-6:0-3:0x5742b0f00007c500:10
> rport-6:0-4:0x500009757804cbff:30
>
> systool is giving me the same information:
>
> systool -c fc_remote_ports -v | grep dev_loss_tmo
>
> dev_loss_tmo = "10"
> dev_loss_tmo = "10"
> dev_loss_tmo = "10"
> dev_loss_tmo = "10"
>
>
> >
> > I am using the following versions:
> >
> > rpm -qa multipath-tools
> > multipath-tools-0.4.9-109.1
> >
> > uname -a
> > Linux mysystem 3.0.101-63-default #1 SMP Tue Jun 23 16:02:31 UTC
> 2015
> > (4b89d0c) x86_64 x86_64 x86_64 GNU/Linux
> >
> > Thanks for your help!
> >
> > Kind regards,
> >
> > Bruno
> >
> > --
> > dm-devel mailing list
> > dm-devel@redhat.com
> > https://www.redhat.com/mailman/listinfo/dm-devel
> >
>
>
> dev_loss_tmo = "10"
> dev_loss_tmo = "10"
> dev_loss_tmo = "10"
> dev_loss_tmo = "10"
> dev_loss_tmo = "10"
> dev_loss_tmo = "30"
> dev_loss_tmo = "10"
> dev_loss_tmo = "30"
> dev_loss_tmo = "30"
> dev_loss_tmo = "10"
> dev_loss_tmo = "30"
> dev_loss_tmo = "10"
> dev_loss_tmo = "30"
> dev_loss_tmo = "30"
> dev_loss_tmo = "30"
> dev_loss_tmo = "30"
>
> Where is this value coming from? May this be a bug? I couldn't find
> anything useful on the Internet regarding this.
It'd be very helpful if you could upload "multipath -v3" (or multipathd
with verbosity 3) logs somewhere.
It looks as if you're using some SLE11 variant, so maybe you want to
open a support case?
Another question would be why you want such a low dev_loss_tmo. It's
not generally recommended, because on the kernel side, removing and re-
adding a device is a lot more complex than disabling and re-enabling
it. The fast_io_fail_tmo should provide you with quick path failover
already. My recommendation is to set dev_loss_tmo to a value which
would, in the given data center, indicate that the device loss is
really not due to a temporary outage but due to a permantly removed
device (e.g. permanent storage configuration change). So basically, the
dev_loss_tmo shouldn't be shorter than the admin's lunch break.
Martin
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: multipathd ignoring dev_loss_tmo setting
2019-03-04 12:09 ` Martin Wilck
@ 2019-03-04 14:09 ` Martins, Bruno O
2019-03-04 14:40 ` Martin Wilck
0 siblings, 1 reply; 10+ messages in thread
From: Martins, Bruno O @ 2019-03-04 14:09 UTC (permalink / raw)
To: dm-devel@redhat.com, mwilck@suse.de
On Mon, 2019-03-04 at 13:09 +0100, Martin Wilck wrote:
> On Thu, 2019-02-28 at 11:38 +0000, Martins, Bruno O wrote:
> > Hello guys,
> >
> > I am trying to modify /etc/multipath.conf on my system so that the
> > parameter 'dev_loss_tmo' is changed from the default value.
> >
> > My multipath.conf file contains the following:
> >
> > defaults {
> > verbosity 2
> > polling_interval 5
> > max_polling_interval 10
> > multipath_dir "/lib64/multipath"
> > path_selector "round-robin 0"
> > path_grouping_policy "failover"
> > uid_attribute "ID_SERIAL"
> > prio "const"
> > prio_args ""
> > features "0"
> > path_checker "directio"
> > alias_prefix "mpath"
> > failback "manual"
> > rr_min_io 1000
> > rr_min_io_rq 1
> > max_fds "max"
> > rr_weight "uniform"
> > no_path_retry "fail"
> > queue_without_daemon "no"
> > checker_timeout 15
> > flush_on_last_del "no"
> > user_friendly_names "yes"
> > fast_io_fail_tmo 5
> > dev_loss_tmo 10
> > bindings_file "/etc/multipath/bindings"
> > wwids_file /etc/multipath/wwids
> > log_checker_err always
> > retain_attached_hw_handler no
> > detect_prio no
> > }
> >
> > However, when checking the value currently in use I am getting the
> > wrong value (which is '30') for some of the remote ports:
> >
> > for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do
> > d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f);
> > done
> >
> > rport-3:0-0:0x5742b0f00007c500:10
> > rport-3:0-1:0x5742b0f00007c500:10
> > rport-3:0-2:0x5742b0f00007c500:10
> > rport-3:0-3:0x5000097408369800:30
> > rport-3:0-4:0x500009757804cbff:30
> > rport-4:0-0:0x5742b0f00007c500:10
> > rport-4:0-1:0x5742b0f00007c500:10
> > rport-4:0-2:0x5000097408369800:30
> > rport-4:0-3:0x5742b0f00007c500:10
> > rport-4:0-4:0x500009757804cbff:30
> > rport-5:0-0:0x5742b0f00007c500:10
> > rport-5:0-1:0x5742b0f00007c500:10
> > rport-5:0-2:0x5742b0f00007c500:10
> > rport-5:0-3:0x5000097408369800:30
> > rport-5:0-4:0x500009757804cbff:30
> > rport-6:0-0:0x5742b0f00007c500:10
> > rport-6:0-1:0x5742b0f00007c500:10
> > rport-6:0-2:0x5000097408369800:30
> > rport-6:0-3:0x5742b0f00007c500:10
> > rport-6:0-4:0x500009757804cbff:30
> >
> > systool is giving me the same information:
> >
> > systool -c fc_remote_ports -v | grep dev_loss_tmo
> >
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "10"
> >
> >
> > > I am using the following versions:
> > >
> > > rpm -qa multipath-tools
> > > multipath-tools-0.4.9-109.1
> > >
> > > uname -a
> > > Linux mysystem 3.0.101-63-default #1 SMP Tue Jun 23 16:02:31 UTC
> >
> > 2015
> > > (4b89d0c) x86_64 x86_64 x86_64 GNU/Linux
> > >
> > > Thanks for your help!
> > >
> > > Kind regards,
> > >
> > > Bruno
> > >
> > > --
> > > dm-devel mailing list
> > > dm-devel@redhat.com
> > >
> > > https://www.redhat.com/mailman/listinfo/dm-devel
> > >
> > >
> >
> >
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "30"
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "30"
> > dev_loss_tmo = "30"
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "30"
> > dev_loss_tmo = "10"
> > dev_loss_tmo = "30"
> > dev_loss_tmo = "30"
> > dev_loss_tmo = "30"
> > dev_loss_tmo = "30"
> >
> > Where is this value coming from? May this be a bug? I couldn't find
> > anything useful on the Internet regarding this.
>
> It'd be very helpful if you could upload "multipath -v3" (or
> multipathd
> with verbosity 3) logs somewhere.
>
> It looks as if you're using some SLE11 variant, so maybe you want to
> open a support case?
>
> Another question would be why you want such a low dev_loss_tmo. It's
> not generally recommended, because on the kernel side, removing and
> re-
> adding a device is a lot more complex than disabling and re-enabling
> it. The fast_io_fail_tmo should provide you with quick path failover
> already. My recommendation is to set dev_loss_tmo to a value which
> would, in the given data center, indicate that the device loss is
> really not due to a temporary outage but due to a permantly removed
> device (e.g. permanent storage configuration change). So basically,
> the
> dev_loss_tmo shouldn't be shorter than the admin's lunch break.
>
> Martin
>
>
>
>
Hello Martin,
Yes, I'm using SuSE:
[ 14:01:44 ] root@mysystem:/tmp# cat /etc/SuSE-release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 4
The thing here is that my applications are crashing due to multipath
issues on my Oracle DB cluster, with errors like these:
[ 13:59:27 ] root@mysystem:~# cat /var/log/messages | grep multipath |
head -n 20
Mar 2 23:00:36 mysystem multipathd: sdayi: failed to set rport to
'Blocked', error 2
Mar 2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk1: sdayi -
tur checker timed out
Mar 2 23:00:36 mysystem multipathd: checker failed path 67:1376 in map
BPM1ADB1REDO1DG-hdisk1
Mar 2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk1: remaining
active paths: 3
Mar 2 23:00:36 mysystem multipathd: sdayj: failed to set rport to
'Blocked', error 2
Mar 2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk2: sdayj -
tur checker timed out
Mar 2 23:00:36 mysystem multipathd: checker failed path 67:1392 in map
BPM1ADB1REDO1DG-hdisk2
Mar 2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk2: remaining
active paths: 3
Mar 2 23:00:36 mysystem multipathd: sdayk: failed to set rport to
'Blocked', error 2
Mar 2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk3: sdayk -
tur checker timed out
Mar 2 23:00:36 mysystem multipathd: checker failed path 67:1408 in map
BPM1ADB1REDO1DG-hdisk3
Mar 2 23:00:36 mysystem kernel: [9249542.734463] device-mapper:
multipath: Failing path 67:1376.
Mar 2 23:00:48 mysystem kernel: [9249542.734701] device-mapper:
multipath: Failing path 67:1392.
Mar 2 23:00:48 mysystem kernel: [9249542.734925] device-mapper:
multipath: Failing path 67:1408.
Mar 2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk3: remaining
active paths: 3
Mar 2 23:00:48 mysystem multipathd: sdayo: failed to set rport to
'Blocked', error 2
Mar 2 23:00:48 mysystem multipathd: BPM1ADB1REDO2DG-hdisk2: sdayo -
tur checker timed out
Mar 2 23:00:48 mysystem multipathd: checker failed path 67:1472 in map
BPM1ADB1REDO2DG-hdisk2
Mar 2 23:00:48 mysystem multipathd: BPM1ADB1REDO2DG-hdisk2: remaining
active paths: 3
Mar 2 23:00:48 mysystem multipathd: sdayp: failed to set rport to
'Blocked', error 2
Output of 'multipath -v3' is available here:
https://paste.gnome.org/pojggla8w
Thanks for your cooperation!
Best regards,
Bruno
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: multipathd ignoring dev_loss_tmo setting
2019-03-04 14:09 ` Martins, Bruno O
@ 2019-03-04 14:40 ` Martin Wilck
0 siblings, 0 replies; 10+ messages in thread
From: Martin Wilck @ 2019-03-04 14:40 UTC (permalink / raw)
To: Martins, Bruno O, dm-devel@redhat.com
On Mon, 2019-03-04 at 14:09 +0000, Martins, Bruno O wrote:
> On Mon, 2019-03-04 at 13:09 +0100, Martin Wilck wrote:
> > On Thu, 2019-02-28 at 11:38 +0000, Martins, Bruno O wrote:
> > > Hello guys,
> > >
> > > I am trying to modify /etc/multipath.conf on my system so that
> > > the
> > > parameter 'dev_loss_tmo' is changed from the default value.
> > >
> > > My multipath.conf file contains the following:
> > >
> > > defaults {
> > > verbosity 2
> > > polling_interval 5
> > > max_polling_interval 10
> > > multipath_dir "/lib64/multipath"
> > > path_selector "round-robin 0"
> > > path_grouping_policy "failover"
> > > uid_attribute "ID_SERIAL"
> > > prio "const"
> > > prio_args ""
> > > features "0"
> > > path_checker "directio"
> > > alias_prefix "mpath"
> > > failback "manual"
> > > rr_min_io 1000
> > > rr_min_io_rq 1
> > > max_fds "max"
> > > rr_weight "uniform"
> > > no_path_retry "fail"
> > > queue_without_daemon "no"
> > > checker_timeout 15
> > > flush_on_last_del "no"
> > > user_friendly_names "yes"
> > > fast_io_fail_tmo 5
> > > dev_loss_tmo 10
> > > bindings_file "/etc/multipath/bindings"
> > > wwids_file /etc/multipath/wwids
> > > log_checker_err always
> > > retain_attached_hw_handler no
> > > detect_prio no
> > > }
> > >
> > > However, when checking the value currently in use I am getting
> > > the
> > > wrong value (which is '30') for some of the remote ports:
> > >
> > > for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do
> > > d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat
> > > $f);
> > > done
> > >
> > > rport-3:0-0:0x5742b0f00007c500:10
> > > rport-3:0-1:0x5742b0f00007c500:10
> > > rport-3:0-2:0x5742b0f00007c500:10
> > > rport-3:0-3:0x5000097408369800:30
> > > rport-3:0-4:0x500009757804cbff:30
> > > rport-4:0-0:0x5742b0f00007c500:10
> > > rport-4:0-1:0x5742b0f00007c500:10
> > > rport-4:0-2:0x5000097408369800:30
> > > rport-4:0-3:0x5742b0f00007c500:10
> > > rport-4:0-4:0x500009757804cbff:30
> > > rport-5:0-0:0x5742b0f00007c500:10
> > > rport-5:0-1:0x5742b0f00007c500:10
> > > rport-5:0-2:0x5742b0f00007c500:10
> > > rport-5:0-3:0x5000097408369800:30
> > > rport-5:0-4:0x500009757804cbff:30
> > > rport-6:0-0:0x5742b0f00007c500:10
> > > rport-6:0-1:0x5742b0f00007c500:10
> > > rport-6:0-2:0x5000097408369800:30
> > > rport-6:0-3:0x5742b0f00007c500:10
> > > rport-6:0-4:0x500009757804cbff:30
> > >
> > > systool is giving me the same information:
> > >
> > > systool -c fc_remote_ports -v | grep dev_loss_tmo
> > >
> > > dev_loss_tmo = "10"
> > > dev_loss_tmo = "10"
> > > dev_loss_tmo = "10"
> > > dev_loss_tmo = "10"
> > >
> > >
> > > > I am using the following versions:
> > > >
> > > > rpm -qa multipath-tools
> > > > multipath-tools-0.4.9-109.1
> > > >
> > > > uname -a
> > > > Linux mysystem 3.0.101-63-default #1 SMP Tue Jun 23 16:02:31
> > > > UTC
> > >
> > > 2015
> > > > (4b89d0c) x86_64 x86_64 x86_64 GNU/Linux
> > > >
> > > > Thanks for your help!
> > > >
> > > > Kind regards,
> > > >
> > > > Bruno
> > > >
> > > > --
> > > > dm-devel mailing list
> > > > dm-devel@redhat.com
> > > >
> > > > https://www.redhat.com/mailman/listinfo/dm-devel
> > > >
> > > >
> > >
> > > dev_loss_tmo = "10"
> > > dev_loss_tmo = "10"
> > > dev_loss_tmo = "10"
> > > dev_loss_tmo = "10"
> > > dev_loss_tmo = "10"
> > > dev_loss_tmo = "30"
> > > dev_loss_tmo = "10"
> > > dev_loss_tmo = "30"
> > > dev_loss_tmo = "30"
> > > dev_loss_tmo = "10"
> > > dev_loss_tmo = "30"
> > > dev_loss_tmo = "10"
> > > dev_loss_tmo = "30"
> > > dev_loss_tmo = "30"
> > > dev_loss_tmo = "30"
> > > dev_loss_tmo = "30"
> > >
> > > Where is this value coming from? May this be a bug? I couldn't
> > > find
> > > anything useful on the Internet regarding this.
> >
> > It'd be very helpful if you could upload "multipath -v3" (or
> > multipathd
> > with verbosity 3) logs somewhere.
> >
> > It looks as if you're using some SLE11 variant, so maybe you want
> > to
> > open a support case?
> >
> > Another question would be why you want such a low dev_loss_tmo.
> > It's
> > not generally recommended, because on the kernel side, removing and
> > re-
> > adding a device is a lot more complex than disabling and re-
> > enabling
> > it. The fast_io_fail_tmo should provide you with quick path
> > failover
> > already. My recommendation is to set dev_loss_tmo to a value which
> > would, in the given data center, indicate that the device loss is
> > really not due to a temporary outage but due to a permantly removed
> > device (e.g. permanent storage configuration change). So basically,
> > the
> > dev_loss_tmo shouldn't be shorter than the admin's lunch break.
> >
> > Martin
> >
> >
> >
> >
>
> Hello Martin,
>
> Yes, I'm using SuSE:
>
> [ 14:01:44 ] root@mysystem:/tmp# cat /etc/SuSE-release
> SUSE Linux Enterprise Server 11 (x86_64)
> VERSION = 11
> PATCHLEVEL = 4
>
> The thing here is that my applications are crashing due to multipath
> issues on my Oracle DB cluster, with errors like these:
>
> [ 13:59:27 ] root@mysystem:~# cat /var/log/messages | grep multipath
> |
> head -n 20
> Mar 2 23:00:36 mysystem multipathd: sdayi: failed to set rport to
> 'Blocked', error 2
> Mar 2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk1: sdayi -
> tur checker timed out
> Mar 2 23:00:36 mysystem multipathd: checker failed path 67:1376 in
> map
> BPM1ADB1REDO1DG-hdisk1From git://anongit.freedesktop.org/xorg/xserver
>
> Mar 2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk1:
> remaining
> active paths: 3
> Mar 2 23:00:36 mysystem multipathd: sdayj: failed to set rport to
> 'Blocked', error 2
> Mar 2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk2: sdayj -
> tur checker timed out
> Mar 2 23:00:36 mysystem multipathd: checker failed path 67:1392 in
> map
> BPM1ADB1REDO1DG-hdisk2
> Mar 2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk2:
> remaining
> active paths: 3
> Mar 2 23:00:36 mysystem multipathd: sdayk: failed to set rport to
> 'Blocked', error 2
> Mar 2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk3: sdayk -
> tur checker timed out
> Mar 2 23:00:36 mysystem multipathd: checker failed path 67:1408 in
> map
> BPM1ADB1REDO1DG-hdisk3
> Mar 2 23:00:36 mysystem kernel: [9249542.734463] device-mapper:
> multipath: Failing path 67:1376.
> Mar 2 23:00:48 mysystem kernel: [9249542.734701] device-mapper:
> multipath: Failing path 67:1392.
> Mar 2 23:00:48 mysystem kernel: [9249542.734925] device-mapper:
> multipath: Failing path 67:1408.
> Mar 2 23:00:36 mysystem multipathd: BPM1ADB1REDO1DG-hdisk3:
> remaining
> active paths: 3
> Mar 2 23:00:48 mysystem multipathd: sdayo: failed to set rport to
> 'Blocked', error 2
> Mar 2 23:00:48 mysystem multipathd: BPM1ADB1REDO2DG-hdisk2: sdayo -
> tur checker timed oute
> Mar 2 23:00:48 mysystem multipathd: checker failed path 67:1472 in
> map
> BPM1ADB1REDO2DG-hdisk2
> Mar 2 23:00:48 mysystem multipathd: BPM1ADB1REDO2DG-hdisk2:
> remaining
> active paths: 3
> Mar 2 23:00:48 mysystem multipathd: sdayp: failed to set rport to
> 'Blocked', error 2
>
> Output of 'multipath -v3' is available here:
> https://paste.gnome.org/pojggla8w
>
Your logs show that the SYMMETRIX LUNs have a dev_loss_tmo of 30 and
the NFINIDAT LUNs have 10. That makes sense because the old multipath
version you are using is lacking a hwtable entry for NFINIDAT,
therefore the defaults are being used. But the SYMMETRIX has a hwtable
entry:
/*
* EMC / Clariion controller family
*
* Maintainer : Edward Goggin, EMC
* Mail : egoggin@emc.com
*/
{
.vendor = "EMC",
.product = "SYMMETRIX",
.features = DEFAULT_FEATURES,
.hwhandler = DEFAULT_HWHANDLER,
.pgpolicy = MULTIBUS,
.pgfailback = FAILBACK_UNDEF,
.rr_weight = RR_WEIGHT_NONE,
.no_path_retry = 6,
.checker_name = TUR,
.prio_name = DEFAULT_PRIO,
.prio_args = NULL,
},
The point is here ".no_path_retry = 6". This overrides your
"no_path_retry" setting from the defaults section. multipath sets the
dev_loss_tmo such that it is at least (no_path_retry *
polling_interval), which is 30 in your case.
If you want to override this, you need to create a device entry in
multipath.conf:
devices {
device {
vendor EMC
product SYMMETRIX
no_path_retry fail
}
}
}
Wrt the failures you are reporting, it seems to me that you're not
using the latest updates SUSE has released for SLE-11. I am pretty
certain that "fixing" the dev_loss_tmo to 10s for SYMMETRIX would not
fix that issue.
It seems to me that you should really contact your SUSE support team.
I'll be happy to support you further through the regular channels.
Regards,
Martin
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: multipathd ignoring dev_loss_tmo setting
2019-03-04 9:48 ` Martins, Bruno O
@ 2019-03-12 22:32 ` Benjamin Marzinski
0 siblings, 0 replies; 10+ messages in thread
From: Benjamin Marzinski @ 2019-03-12 22:32 UTC (permalink / raw)
To: Martins, Bruno O; +Cc: dm-devel@redhat.com
On Mon, Mar 04, 2019 at 09:48:59AM +0000, Martins, Bruno O wrote:
> On Fri, 2019-03-01 at 11:35 -0600, Benjamin Marzinski wrote:
> > On Fri, Mar 01, 2019 at 10:04:52AM +0000, Martins, Bruno O wrote:
> > > On Thu, 2019-02-28 at 13:53 -0600, Benjamin Marzinski wrote:
> > > > On Thu, Feb 28, 2019 at 11:38:22AM +0000, Martins, Bruno O wrote:
> > > > > Hello guys,
> > > > >
> > > > > I am trying to modify /etc/multipath.conf on my system so that
> > > > > the
> > > > > parameter 'dev_loss_tmo' is changed from the default value.
> > > > >
> > > > > However, when checking the value currently in use I am getting
> > > > > the
> > > > > wrong value (which is '30') for some of the remote ports:
> > > > >
> > > >
> > > > Are you sure those rports are used by multipath devices?
> > > > multipath
> > > > only
> > > > changes dev_loss_tmo for rports associated with a multipath path
> > > > device.
> > > >
> > > > -Ben
> > > >
>
> Hello Ben,
>
> Then I believe this rport's with wrong dev_loss_tmo are part of
> multipathd. Is this correct?
yeah, that looks correct. If you run
# multipath -r
does that correct the issue? If not, can you attach the result of
running
# multipath -rv 3
-Ben
>
> Best regards,
>
> Bruno
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2019-03-12 22:32 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-02-28 11:38 multipathd ignoring dev_loss_tmo setting Martins, Bruno O
2019-02-28 12:08 ` Steffen Maier
2019-02-28 19:53 ` Benjamin Marzinski
2019-03-01 10:04 ` Martins, Bruno O
2019-03-01 17:35 ` Benjamin Marzinski
2019-03-04 9:48 ` Martins, Bruno O
2019-03-12 22:32 ` Benjamin Marzinski
2019-03-04 12:09 ` Martin Wilck
2019-03-04 14:09 ` Martins, Bruno O
2019-03-04 14:40 ` Martin Wilck
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox