From: Laurence Oberman <loberman@redhat.com>
To: Bart Van Assche <bart.vanassche@sandisk.com>
Cc: Mike Snitzer <snitzer@redhat.com>,
dm-devel@redhat.com, linux-scsi@vger.kernel.org
Subject: Re: [dm-devel] dm-mq and end_clone_request()
Date: Wed, 3 Aug 2016 14:03:05 -0400 (EDT) [thread overview]
Message-ID: <1165785378.8091030.1470247385715.JavaMail.zimbra@redhat.com> (raw)
In-Reply-To: <4b015567-0fdb-dbf5-96e2-627e76b48a03@sandisk.com>
----- Original Message -----
> From: "Bart Van Assche" <bart.vanassche@sandisk.com>
> To: "Laurence Oberman" <loberman@redhat.com>, "Mike Snitzer" <snitzer@redhat.com>
> Cc: dm-devel@redhat.com, linux-scsi@vger.kernel.org
> Sent: Wednesday, August 3, 2016 12:06:17 PM
> Subject: Re: [dm-devel] dm-mq and end_clone_request()
>
> On 08/02/2016 06:33 PM, Laurence Oberman wrote:
> > #!/bin/bash
> > for i in /sys/class/srp_remote_ports/*
> > do
> > echo "Deleting host $i, it will re-connect via srp_daemon"
> > echo 1 > $i/delete
> > sleep 10
> > done
>
> Hello Laurence,
>
> Sorry but the above looks wrong to me. There should be a second loop
> around this loop and the sleep statement should be moved from the inner
> loop to the outer loop. The above code logs out one (initiator, target)
> port pair at a time instead of logging out all paths at once.
>
> Bart.
>
Hi Bart
Latest tests are still good on our side.
I am now taking both paths out at the same time but still we seem stable here.
First test removed sleep and we still had a delay, second test add a background so they ran as close as possible to the same time.
Both tests passed.
I will email messages log just to you.
With no sleep we still have a gap when we delete paths of 9s and we are good.
Aug 3 13:41:21 jumpclient multipathd: 360001ff0b035d000000000008d700001: remaining active paths: 1
Aug 3 13:41:22 jumpclient multipathd: 360001ff0b035d000000000028d720003: remaining active paths: 1
Aug 3 13:41:22 jumpclient multipathd: 360001ff0b035d000000000048d740005: remaining active paths: 1
Aug 3 13:41:22 jumpclient multipathd: 360001ff0b035d000000000068d760007: remaining active paths: 1
Aug 3 13:41:23 jumpclient multipathd: 360001ff0b035d0000000000b8d7b000c: remaining active paths: 1
Aug 3 13:41:23 jumpclient multipathd: 360001ff0b035d0000000000d8d7d000e: remaining active paths: 1
Aug 3 13:41:23 jumpclient multipathd: 360001ff0b035d000000000118d810012: remaining active paths: 1
Aug 3 13:41:24 jumpclient multipathd: 360001ff0b035d000000000138d830014: remaining active paths: 1
Aug 3 13:41:24 jumpclient multipathd: 360001ff0b035d000000000158d850016: remaining active paths: 1
Aug 3 13:41:25 jumpclient multipathd: 360001ff0b035d000000000178d870018: remaining active paths: 1
Aug 3 13:41:25 jumpclient multipathd: 360001ff0b035d000000000198d89001a: remaining active paths: 1
Aug 3 13:41:25 jumpclient multipathd: 360001ff0b035d0000000001a8d8a001b: remaining active paths: 1
Aug 3 13:41:25 jumpclient multipathd: 360001ff0b035d0000000001c8d8c001d: remaining active paths: 1
Aug 3 13:41:26 jumpclient multipathd: 360001ff0b035d0000000001e8d8e001f: remaining active paths: 1
Aug 3 13:41:26 jumpclient multipathd: 360001ff0b035d0000000001f8d8f0020: remaining active paths: 1
Aug 3 13:41:26 jumpclient multipathd: 360001ff0b035d000000000208d900021: remaining active paths: 1
Aug 3 13:41:26 jumpclient multipathd: 360001ff0b035d000000000228d920023: remaining active paths: 1
Aug 3 13:41:28 jumpclient multipathd: 360001ff0b035d000000000248d940025: remaining active paths: 1
Aug 3 13:41:29 jumpclient multipathd: 360001ff0b035d000000000268d960027: remaining active paths: 1
Aug 3 13:41:29 jumpclient multipathd: 360001ff0b035d000000000278d970028: remaining active paths: 1
Aug 3 13:41:30 jumpclient multipathd: 360001ff0b035d000000000288d980029: remaining active paths: 1
Aug 3 13:41:35 jumpclient multipathd: 360001ff0b035d000000000008d700001: remaining active paths: 0
Aug 3 13:41:36 jumpclient multipathd: 360001ff0b035d000000000028d720003: remaining active paths: 0
Aug 3 13:41:37 jumpclient multipathd: 360001ff0b035d000000000048d740005: remaining active paths: 0
Aug 3 13:41:37 jumpclient multipathd: 360001ff0b035d000000000068d760007: remaining active paths: 0
Aug 3 13:41:38 jumpclient multipathd: 360001ff0b035d0000000000b8d7b000c: remaining active paths: 0
Aug 3 13:41:38 jumpclient multipathd: 360001ff0b035d0000000000d8d7d000e: remaining active paths: 0
Aug 3 13:41:38 jumpclient multipathd: 360001ff0b035d000000000108d800011: remaining active paths: 0
Aug 3 13:41:38 jumpclient multipathd: 360001ff0b035d000000000118d810012: remaining active paths: 0
Aug 3 13:41:38 jumpclient multipathd: 360001ff0b035d000000000138d830014: remaining active paths: 0
Aug 3 13:41:39 jumpclient multipathd: 360001ff0b035d000000000158d850016: remaining active paths: 0
Aug 3 13:41:39 jumpclient multipathd: 360001ff0b035d000000000178d870018: remaining active paths: 0
Aug 3 13:41:39 jumpclient multipathd: 360001ff0b035d000000000198d89001a: remaining active paths: 0
Aug 3 13:41:39 jumpclient multipathd: 360001ff0b035d0000000001a8d8a001b: remaining active paths: 0
Aug 3 13:41:39 jumpclient multipathd: 360001ff0b035d0000000001c8d8c001d: remaining active paths: 0
Aug 3 13:41:39 jumpclient multipathd: 360001ff0b035d0000000001e8d8e001f: remaining active paths: 0
Aug 3 13:41:41 jumpclient multipathd: 360001ff0b035d0000000001f8d8f0020: remaining active paths: 0
Aug 3 13:41:41 jumpclient multipathd: 360001ff0b035d000000000208d900021: remaining active paths: 0
Aug 3 13:41:43 jumpclient multipathd: 360001ff0b035d000000000248d940025: remaining active paths: 0
Aug 3 13:41:43 jumpclient multipathd: 360001ff0b035d000000000268d960027: remaining active paths: 0
Aug 3 13:41:44 jumpclient multipathd: 360001ff0b035d000000000288d980029: remaining active paths: 0
Aug 3 13:42:44 jumpclient multipathd: 360001ff0b035d000000000138d830014: remaining active paths: 2
Aug 3 13:42:44 jumpclient multipathd: 360001ff0b035d000000000158d850016: remaining active paths: 2
These are the only errors and they are expected.
Aug 3 13:41:22 jumpclient kernel: blk_update_request: I/O error, dev sdd, sector 31141264880
Aug 3 13:41:22 jumpclient kernel: blk_update_request: I/O error, dev sdd, sector 79928
Aug 3 13:41:22 jumpclient kernel: blk_update_request: I/O error, dev sdd, sector 65264
Aug 3 13:41:22 jumpclient kernel: blk_update_request: I/O error, dev sdd, sector 55232
Aug 3 13:41:22 jumpclient kernel: blk_update_request: I/O error, dev sdd, sector 14152
Aug 3 13:41:22 jumpclient kernel: blk_update_request: I/O error, dev sdd, sector 168880
Aug 3 13:41:22 jumpclient kernel: blk_update_request: I/O error, dev sdd, sector 269392
Aug 3 13:41:22 jumpclient kernel: blk_update_request: I/O error, dev sdd, sector 309200
Aug 3 13:41:22 jumpclient kernel: blk_update_request: I/O error, dev sdd, sector 87520
Aug 3 13:41:22 jumpclient kernel: blk_update_request: I/O error, dev sdd, sector 7744
Aug 3 13:41:28 jumpclient kernel: blk_update_request: I/O error, dev sdca, sector 119984
Aug 3 13:41:29 jumpclient kernel: blk_update_request: I/O error, dev sdcd, sector 31139908984
Aug 3 13:41:29 jumpclient kernel: blk_update_request: I/O error, dev sdcd, sector 131136
Aug 3 13:41:29 jumpclient kernel: blk_update_request: I/O error, dev sdcd, sector 97536
Aug 3 13:41:29 jumpclient kernel: blk_update_request: I/O error, dev sdcd, sector 123264
Aug 3 13:41:29 jumpclient kernel: blk_update_request: I/O error, dev sdcd, sector 110336
Aug 3 13:41:29 jumpclient kernel: blk_update_request: I/O error, dev sdcd, sector 158136
Aug 3 13:41:29 jumpclient kernel: blk_update_request: I/O error, dev sdcd, sector 156136
Aug 3 13:41:29 jumpclient kernel: blk_update_request: I/O error, dev sdcd, sector 173072
Aug 3 13:41:29 jumpclient kernel: blk_update_request: I/O error, dev sdcd, sector 6984
Aug 3 13:41:35 jumpclient kernel: blk_update_request: I/O error, dev sdc, sector 130224
Aug 3 13:41:35 jumpclient kernel: blk_update_request: I/O error, dev sdc, sector 225816
Aug 3 13:41:36 jumpclient kernel: blk_update_request: I/O error, dev sdc, sector 248120
Aug 3 13:41:36 jumpclient kernel: blk_update_request: I/O error, dev sdc, sector 242528
Aug 3 13:41:36 jumpclient kernel: blk_update_request: I/O error, dev sdk, sector 251248
Aug 3 13:41:36 jumpclient kernel: blk_update_request: I/O error, dev sdk, sector 242032
Aug 3 13:41:36 jumpclient kernel: blk_update_request: I/O error, dev sdk, sector 203736
Aug 3 13:41:36 jumpclient kernel: blk_update_request: I/O error, dev sdk, sector 31141107808
Aug 3 13:41:36 jumpclient kernel: blk_update_request: I/O error, dev sdk, sector 233336
Aug 3 13:41:36 jumpclient kernel: blk_update_request: I/O error, dev sdk, sector 187944
Aug 3 13:41:41 jumpclient kernel: blk_update_request: I/O error, dev sdbl, sector 85800
Aug 3 13:41:41 jumpclient kernel: blk_update_request: I/O error, dev sdbl, sector 74120
Aug 3 13:41:41 jumpclient kernel: blk_update_request: I/O error, dev sdbl, sector 78216
Aug 3 13:41:41 jumpclient kernel: blk_update_request: I/O error, dev sdbl, sector 79976
Aug 3 13:41:41 jumpclient kernel: blk_update_request: I/O error, dev sdbl, sector 79552
Aug 3 13:41:41 jumpclient kernel: blk_update_request: I/O error, dev sdbl, sector 87888
Aug 3 13:41:43 jumpclient kernel: blk_update_request: I/O error, dev sdbt, sector 274368
Aug 3 13:41:43 jumpclient kernel: blk_update_request: I/O error, dev sdbt, sector 31139814080
Aug 3 13:41:43 jumpclient kernel: blk_update_request: I/O error, dev sdbx, sector 6776
Aug 3 13:41:43 jumpclient kernel: blk_update_request: I/O error, dev sdbx, sector 302152
Changing script to add & we take both away at the same time but still we seem to survive here.
This is my configuration:
360001ff0b035d000000000078d770008 dm-9 DDN ,SFA14K
size=29T features='3 queue_if_no_path queue_mode mq' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=90 status=active
| `- 1:0:0:7 sday 67:32 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
`- 2:0:0:7 sdj 8:144 active ready running
device {
vendor "DDN"
product "SFA14K"
path_grouping_policy group_by_prio
prio alua
path_selector "round-robin 0"
path_checker tur
failback 2
rr_weight uniform
no_path_retry 12
dev_loss_tmo 10
fast_io_fail_tmo 5
features "1 queue_if_no_path"
}
With &
#!/bin/bash
for i in /sys/class/srp_remote_ports/*
do
echo "Deleting host $i, it will re-connect via srp_daemon"
echo 1 > $i/delete &
#sleep 3
done
Here we lose both paths at the same time.
[root@jumpclient bart_tests]# grep remaining messages
Aug 3 13:49:38 jumpclient multipathd: 360001ff0b035d000000000008d700001: remaining active paths: 0
Aug 3 13:49:38 jumpclient multipathd: 360001ff0b035d000000000028d720003: remaining active paths: 0
Aug 3 13:49:38 jumpclient multipathd: 360001ff0b035d000000000048d740005: remaining active paths: 0
Aug 3 13:49:41 jumpclient multipathd: 360001ff0b035d000000000068d760007: remaining active paths: 0
Aug 3 13:49:42 jumpclient multipathd: 360001ff0b035d0000000000d8d7d000e: remaining active paths: 0
Aug 3 13:49:45 jumpclient multipathd: 360001ff0b035d000000000118d810012: remaining active paths: 0
Aug 3 13:49:45 jumpclient multipathd: 360001ff0b035d000000000108d800011: remaining active paths: 0
Aug 3 13:49:47 jumpclient multipathd: 360001ff0b035d000000000158d850016: remaining active paths: 0
Aug 3 13:49:48 jumpclient multipathd: 360001ff0b035d000000000178d870018: remaining active paths: 0
Aug 3 13:49:48 jumpclient multipathd: 360001ff0b035d000000000198d89001a: remaining active paths: 0
Aug 3 13:49:48 jumpclient multipathd: 360001ff0b035d0000000001a8d8a001b: remaining active paths: 0
Aug 3 13:49:55 jumpclient multipathd: 360001ff0b035d0000000001e8d8e001f: remaining active paths: 0
Aug 3 13:49:55 jumpclient multipathd: 360001ff0b035d0000000001f8d8f0020: remaining active paths: 0
Aug 3 13:49:58 jumpclient multipathd: 360001ff0b035d000000000248d940025: remaining active paths: 0
Aug 3 13:49:59 jumpclient multipathd: 360001ff0b035d000000000268d960027: remaining active paths: 0
Aug 3 13:50:00 jumpclient multipathd: 360001ff0b035d000000000288d980029: remaining active paths: 0
Aug 3 13:51:17 jumpclient multipathd: 360001ff0b035d000000000038d730004: remaining active paths: 2
Aug 3 13:51:17 jumpclient multipathd: 360001ff0b035d000000000028d720003: remaining active paths: 2
Aug 3 13:51:19 jumpclient multipathd: 360001ff0b035d000000000078d770008: remaining active paths: 2
Aug 3 13:51:20 jumpclient multipathd: 360001ff0b035d0000000000a8d7a000b: remaining active paths: 2
Aug 3 13:51:23 jumpclient multipathd: 360001ff0b035d0000000000d8d7d000e: remaining active paths: 2
Aug 3 13:51:24 jumpclient multipathd: 360001ff0b035d000000000108d800011: remaining active paths: 2
Aug 3 13:51:25 jumpclient multipathd: 360001ff0b035d0000000000f8d7f0010: remaining active paths: 2
Aug 3 13:51:26 jumpclient multipathd: 360001ff0b035d000000000128d820013: remaining active paths: 2
Aug 3 13:51:29 jumpclient multipathd: 360001ff0b035d0000000001c8d8c001d: remaining active paths: 2
Aug 3 13:51:33 jumpclient multipathd: 360001ff0b035d000000000228d920023: remaining active paths: 2
Aug 3 13:51:34 jumpclient multipathd: 360001ff0b035d000000000238d930024: remaining active paths: 2
We still survive.
[root@jumpclient bart_tests]# grep -i error messages
Aug 3 13:49:38 jumpclient kernel: blk_update_request: I/O error, dev sdc, sector 98288
Aug 3 13:49:38 jumpclient kernel: blk_update_request: I/O error, dev sdc, sector 98320
Aug 3 13:49:38 jumpclient kernel: blk_update_request: I/O error, dev sdc, sector 46976
Aug 3 13:49:38 jumpclient kernel: blk_update_request: I/O error, dev sde, sector 216720
Aug 3 13:49:38 jumpclient kernel: blk_update_request: I/O error, dev sdg, sector 130672
Aug 3 13:49:41 jumpclient kernel: blk_update_request: I/O error, dev sdi, sector 56984
Aug 3 13:49:41 jumpclient kernel: blk_update_request: I/O error, dev sdi, sector 56120
Aug 3 13:49:41 jumpclient kernel: blk_update_request: I/O error, dev sdi, sector 62112
Aug 3 13:49:42 jumpclient kernel: blk_update_request: I/O error, dev sdp, sector 156944
Aug 3 13:49:42 jumpclient kernel: blk_update_request: I/O error, dev sdp, sector 31140975496
Aug 3 13:49:44 jumpclient kernel: blk_update_request: I/O error, dev sdt, sector 207392
Aug 3 13:49:44 jumpclient kernel: blk_update_request: I/O error, dev sdt, sector 200568
Aug 3 13:49:44 jumpclient kernel: blk_update_request: I/O error, dev sdt, sector 251048
Aug 3 13:49:44 jumpclient kernel: blk_update_request: I/O error, dev sdt, sector 247616
Aug 3 13:49:44 jumpclient kernel: blk_update_request: I/O error, dev sdt, sector 210592
Aug 3 13:49:44 jumpclient kernel: blk_update_request: I/O error, dev sdt, sector 200120
Aug 3 13:49:44 jumpclient kernel: blk_update_request: I/O error, dev sdt, sector 203000
Aug 3 13:49:44 jumpclient kernel: blk_update_request: I/O error, dev sdt, sector 248640
Aug 3 13:49:47 jumpclient kernel: blk_update_request: I/O error, dev sdx, sector 48232
Aug 3 13:49:48 jumpclient kernel: blk_update_request: I/O error, dev sdz, sector 9984
Aug 3 13:49:55 jumpclient kernel: blk_update_request: I/O error, dev sdag, sector 130512
Aug 3 13:49:58 jumpclient kernel: blk_update_request: I/O error, dev sdai, sector 39040
Aug 3 13:49:58 jumpclient kernel: blk_update_request: I/O error, dev sdam, sector 31140570528
Aug 3 13:49:59 jumpclient kernel: blk_update_request: I/O error, dev sdao, sector 204552
Aug 3 13:50:00 jumpclient kernel: blk_update_request: I/O error, dev sdaq, sector 31142052904
next prev parent reply other threads:[~2016-08-03 18:03 UTC|newest]
Thread overview: 66+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <4ed669ed-beae-76a8-b806-a284565b327a@sandisk.com>
2016-07-20 14:08 ` dm-mq and end_clone_request() Mike Snitzer
2016-07-20 14:27 ` Mike Snitzer
2016-07-20 17:37 ` Bart Van Assche
2016-07-20 18:33 ` Mike Snitzer
2016-07-21 20:58 ` [dm-devel] " Bart Van Assche
2016-07-25 17:53 ` Mike Snitzer
2016-07-25 21:23 ` Mike Snitzer
2016-07-25 22:00 ` Bart Van Assche
2016-07-26 1:16 ` Mike Snitzer
2016-07-26 22:51 ` Bart Van Assche
2016-07-27 14:08 ` Mike Snitzer
2016-07-27 15:52 ` [dm-devel] " Benjamin Marzinski
2016-07-27 19:06 ` Bart Van Assche
2016-07-27 19:54 ` Mike Snitzer
2016-07-27 20:09 ` Mike Snitzer
2016-07-27 23:05 ` Bart Van Assche
2016-07-28 13:33 ` Mike Snitzer
2016-07-28 15:23 ` Bart Van Assche
2016-07-28 15:40 ` Mike Snitzer
2016-07-29 6:28 ` [dm-devel] " Hannes Reinecke
[not found] ` <317679447.7168375.1469729769593.JavaMail.zimbra@redhat.com>
[not found] ` <6880321d-e14f-169b-d100-6e460dd9bd09@sandisk.com>
[not found] ` <1110327939.7305916.1469819453678.JavaMail.zimbra@redhat.com>
[not found] ` <a5c1a149-b1a2-b5a4-2207-bdaf32db3cbd@sandisk.com>
[not found] ` <757522831.7667712.1470059860543.JavaMail.zimbra@redhat.com>
[not found] ` <536022978.7668211.1470060125271.JavaMail.zimbra@redhat.com>
[not found] ` <931235537.7668834.1470060339483.JavaMail.zimbra@redhat.com>
[not found] ` <1264951811.7684268.1470065187014.JavaMail.zimbra@redhat.com>
[not found] ` <17da3ab0-233a-2cec-f921-bfd42c953ccc@sandisk.com>
2016-08-01 17:59 ` Mike Snitzer
2016-08-01 18:55 ` Bart Van Assche
2016-08-01 19:15 ` Mike Snitzer
2016-08-01 20:46 ` Mike Snitzer
2016-08-01 22:41 ` Bart Van Assche
2016-08-02 17:45 ` Mike Snitzer
2016-08-03 0:19 ` Bart Van Assche
2016-08-03 0:40 ` Mike Snitzer
2016-08-03 1:33 ` Laurence Oberman
2016-08-03 2:10 ` Mike Snitzer
2016-08-03 2:18 ` Laurence Oberman
2016-08-03 2:55 ` Laurence Oberman
2016-08-03 15:10 ` Laurence Oberman
2016-08-03 16:06 ` Bart Van Assche
2016-08-03 17:25 ` Laurence Oberman
2016-08-03 18:03 ` Laurence Oberman [this message]
2016-08-03 16:55 ` Bart Van Assche
2016-08-04 16:10 ` Mike Snitzer
2016-08-04 17:42 ` Bart Van Assche
2016-08-04 23:58 ` Mike Snitzer
2016-08-05 1:07 ` Laurence Oberman
2016-08-05 11:43 ` Laurence Oberman
2016-08-05 15:39 ` Laurence Oberman
2016-08-05 15:43 ` Bart Van Assche
2016-08-05 18:42 ` [dm-devel] " Bart Van Assche
2016-08-06 14:47 ` Laurence Oberman
2016-08-07 22:31 ` [dm-devel] " Bart Van Assche
2016-08-08 12:45 ` Laurence Oberman
2016-08-08 13:44 ` Johannes Thumshirn
2016-08-08 14:32 ` Laurence Oberman
2016-08-08 14:54 ` Bart Van Assche
2016-08-08 15:11 ` Bart Van Assche
2016-08-08 15:26 ` Laurence Oberman
2016-08-08 15:28 ` Bart Van Assche
2016-08-08 22:39 ` Bart Van Assche
2016-08-08 22:52 ` Laurence Oberman
2016-08-09 0:09 ` Laurence Oberman
2016-08-09 15:51 ` Bart Van Assche
2016-08-09 17:12 ` [dm-devel] " Laurence Oberman
2016-08-09 17:16 ` Bart Van Assche
2016-08-09 17:21 ` Laurence Oberman
2016-08-10 21:38 ` Laurence Oberman
2016-08-11 16:51 ` Laurence Oberman
2016-08-05 18:40 ` Bart Van Assche
2016-07-21 20:32 ` Mike Snitzer
2016-07-21 20:40 ` [dm-devel] " Bart Van Assche
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1165785378.8091030.1470247385715.JavaMail.zimbra@redhat.com \
--to=loberman@redhat.com \
--cc=bart.vanassche@sandisk.com \
--cc=dm-devel@redhat.com \
--cc=linux-scsi@vger.kernel.org \
--cc=snitzer@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).