v3.15 dm-mpath regression: cable pull test causes I/O hang

v3.15 dm-mpath regression: cable pull test causes I/O hang

[Bart Van Assche]

Hello,

While running a cable pull simulation test with dm_multipath on top of
the SRP initiator driver I noticed that after a few iterations I/O locks
up instead of dm_multipath processing the path failure properly (see also
below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are
vulnerable. This issue does not occur with kernel 3.14. I have tried to
bisect this but gave up when I noticed that I/O locked up completely with
a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec
(dm mpath: push back requests instead of queueing). But with the bisect I
have been able to narrow down this issue to one of the patches in "Merge
tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/
device-mapper/linux-dm". Does anyone have a suggestion how to analyze this
further or how to fix this ?

Thanks,

Bart.

systemd-udevd   D ffff880831b58000     0  9926    356 0x00000006
 ffff8807f8bb79b8 0000000000000002 ffff880831b58000 ffff8807f8bb7fd8
 00000000000131c0 00000000000131c0 ffff88083b490000 ffff88085fc53ad0
 ffff88085ff6bf38 ffff8807f8bb7a40 0000000000000002 ffffffff81135bd0
Call Trace:
 [<ffffffff814bba8d>] io_schedule+0x9d/0x130
 [<ffffffff81135bde>] sleep_on_page+0xe/0x20
 [<ffffffff814bc0d8>] __wait_on_bit_lock+0x48/0xb0
 [<ffffffff81135cea>] __lock_page+0x6a/0x70
 [<ffffffff811471df>] truncate_inode_pages_range+0x3ff/0x690
 [<ffffffff81147485>] truncate_inode_pages+0x15/0x20
 [<ffffffff811d2f85>] kill_bdev+0x35/0x40
 [<ffffffff811d4509>] __blkdev_put+0x69/0x1b0
 [<ffffffff811d4fb0>] blkdev_put+0x50/0x160
 [<ffffffff811d5175>] blkdev_close+0x25/0x30
 [<ffffffff81199eda>] __fput+0xea/0x1f0
 [<ffffffff8119a02e>] ____fput+0xe/0x10
 [<ffffffff81074d9c>] task_work_run+0xac/0xe0
 [<ffffffff8104ff37>] do_exit+0x2c7/0xc60
 [<ffffffff81051c7c>] do_group_exit+0x4c/0xc0
 [<ffffffff81064261>] get_signal_to_deliver+0x2e1/0x940
 [<ffffffff81002528>] do_signal+0x48/0x630
 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0
 [<ffffffff814c1918>] int_signal+0x12/0x17

Re: v3.15 dm-mpath regression: cable pull test causes I/O hang

[Mike Snitzer]

On Fri, Jun 27 2014 at  9:02am -0400,
Bart Van Assche <bvanassche@acm.org> wrote:

> Hello,
> 
> While running a cable pull simulation test with dm_multipath on top of
> the SRP initiator driver I noticed that after a few iterations I/O locks
> up instead of dm_multipath processing the path failure properly (see also
> below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are
> vulnerable. This issue does not occur with kernel 3.14. I have tried to
> bisect this but gave up when I noticed that I/O locked up completely with
> a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec
> (dm mpath: push back requests instead of queueing). But with the bisect I
> have been able to narrow down this issue to one of the patches in "Merge
> tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/
> device-mapper/linux-dm". Does anyone have a suggestion how to analyze this
> further or how to fix this ?
> 
> Thanks,
> 
> Bart.
> 
> systemd-udevd   D ffff880831b58000     0  9926    356 0x00000006
>  ffff8807f8bb79b8 0000000000000002 ffff880831b58000 ffff8807f8bb7fd8
>  00000000000131c0 00000000000131c0 ffff88083b490000 ffff88085fc53ad0
>  ffff88085ff6bf38 ffff8807f8bb7a40 0000000000000002 ffffffff81135bd0
> Call Trace:
>  [<ffffffff814bba8d>] io_schedule+0x9d/0x130
>  [<ffffffff81135bde>] sleep_on_page+0xe/0x20
>  [<ffffffff814bc0d8>] __wait_on_bit_lock+0x48/0xb0
>  [<ffffffff81135cea>] __lock_page+0x6a/0x70
>  [<ffffffff811471df>] truncate_inode_pages_range+0x3ff/0x690
>  [<ffffffff81147485>] truncate_inode_pages+0x15/0x20
>  [<ffffffff811d2f85>] kill_bdev+0x35/0x40
>  [<ffffffff811d4509>] __blkdev_put+0x69/0x1b0
>  [<ffffffff811d4fb0>] blkdev_put+0x50/0x160
>  [<ffffffff811d5175>] blkdev_close+0x25/0x30
>  [<ffffffff81199eda>] __fput+0xea/0x1f0
>  [<ffffffff8119a02e>] ____fput+0xe/0x10
>  [<ffffffff81074d9c>] task_work_run+0xac/0xe0
>  [<ffffffff8104ff37>] do_exit+0x2c7/0xc60
>  [<ffffffff81051c7c>] do_group_exit+0x4c/0xc0
>  [<ffffffff81064261>] get_signal_to_deliver+0x2e1/0x940
>  [<ffffffff81002528>] do_signal+0x48/0x630
>  [<ffffffff81002b81>] do_notify_resume+0x71/0xc0
>  [<ffffffff814c1918>] int_signal+0x12/0x17

(we've seen sync on last close cause problems when the block device
isn't reachable).

Any other threads that look suspect in output from?:
 echo t > /proc/sysrq-trigger

Can you provide your dmsetup table output for the relevant mpath device?
Are you using queue_if_no_path?  Also, AFAIK you don't use
multipath-tools, but if by some chance you do please provide your
multipath.conf.  I'll attempt to reproduce.

But I'm almost tempted to just revert _all_ of 3.15's dm-mpath changes,
and only reintroduce them once they can pass your testing.  I'd like to
avoid that, so Hannes and/or Jun'ichi, it is time for looking at this
seriously.. any help would be very appreciated.  For starters, have you
guys done cable pull tests with > 3.15 ?

Re: v3.15 dm-mpath regression: cable pull test causes I/O hang

[Bart Van Assche]

[-- Attachment #1: Type: text/plain, Size: 891 bytes --]

On 06/27/14 15:33, Mike Snitzer wrote:
> (we've seen sync on last close cause problems when the block device
> isn't reachable).
> 
> Any other threads that look suspect in output from?:
>  echo t > /proc/sysrq-trigger

I have attached the echo w > /proc/sysrq-trigger output since that's the
only output I had saved. I have been analyzing the SysRq-t output but I
haven't found any additional clue in that output.

> Can you provide your dmsetup table output for the relevant mpath device?
> Are you using queue_if_no_path?  Also, AFAIK you don't use
> multipath-tools, but if by some chance you do please provide your
> multipath.conf.  I'll attempt to reproduce.

The test I ran was a fio data verification test on top of a dm device.
multipathd was running during this test. Is multipath.conf sufficient ?
I have attached my multipath configuration file to this e-mail.

Thanks,

Bart.


[-- Attachment #2: multipath-lockup-v3.16-rc2.txt.xz --]
[-- Type: application/x-xz, Size: 19228 bytes --]

[-- Attachment #3: multipath.conf --]
[-- Type: text/plain, Size: 470 bytes --]

defaults {
	polling_interval	1
	queue_without_daemon	no
	features		"3 queue_if_no_path pg_init_retries 50"
	path_grouping_policy	group_by_prio
}
blacklist {
	device {
		vendor	"ATA"
		product	"*"
	}
}
devices {
	device {
		vendor			FUSIONIO
		product			"*"
		features		"3 queue_if_no_path pg_init_retries 50"
		#hardware_handler	"1 alua"
		#prio			alua
		#failback		followover
		path_checker		tur
		path_selector		"queue-length 0"
		#path_selector		"round-robin 0"
	}
}

[-- Attachment #4: Type: text/plain, Size: 0 bytes --]

Re: v3.15 dm-mpath regression: cable pull test causes I/O hang

[Mike Snitzer]

On Fri, Jun 27 2014 at  9:33am -0400,
Mike Snitzer <snitzer@redhat.com> wrote:

> On Fri, Jun 27 2014 at  9:02am -0400,
> Bart Van Assche <bvanassche@acm.org> wrote:
> 
> > Hello,
> > 
> > While running a cable pull simulation test with dm_multipath on top of
> > the SRP initiator driver I noticed that after a few iterations I/O locks
> > up instead of dm_multipath processing the path failure properly (see also
> > below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are
> > vulnerable. This issue does not occur with kernel 3.14. I have tried to
> > bisect this but gave up when I noticed that I/O locked up completely with
> > a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec
> > (dm mpath: push back requests instead of queueing). But with the bisect I
> > have been able to narrow down this issue to one of the patches in "Merge
> > tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/
> > device-mapper/linux-dm". Does anyone have a suggestion how to analyze this
> > further or how to fix this ?

I still don't have a _known_ fix for your issue but I reviewed commit
e809917735ebf1b9a56c24e877ce0d320baee2ec closer and identified what
looks to be a regression in logic for multipath_busy, it now calls
!pg_ready() instead of directly checking pg_init_in_progress.  I think
this is needed (Hannes, what do you think?):

diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
index 3f6fd9d..561ead6 100644
--- a/drivers/md/dm-mpath.c
+++ b/drivers/md/dm-mpath.c
@@ -373,7 +373,7 @@ static int __must_push_back(struct multipath *m)
 		 dm_noflush_suspending(m->ti)));
 }
 
-#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required)
+#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required && !(m)->pg_init_in_progress)
 
 /*
  * Map cloned requests

Re: v3.15 dm-mpath regression: cable pull test causes I/O hang

[Hannes Reinecke]

On 07/03/2014 12:02 AM, Mike Snitzer wrote:
> On Fri, Jun 27 2014 at  9:33am -0400,
> Mike Snitzer <snitzer@redhat.com> wrote:
>
>> On Fri, Jun 27 2014 at  9:02am -0400,
>> Bart Van Assche <bvanassche@acm.org> wrote:
>>
>>> Hello,
>>>
>>> While running a cable pull simulation test with dm_multipath on top of
>>> the SRP initiator driver I noticed that after a few iterations I/O locks
>>> up instead of dm_multipath processing the path failure properly (see also
>>> below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are
>>> vulnerable. This issue does not occur with kernel 3.14. I have tried to
>>> bisect this but gave up when I noticed that I/O locked up completely with
>>> a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec
>>> (dm mpath: push back requests instead of queueing). But with the bisect I
>>> have been able to narrow down this issue to one of the patches in "Merge
>>> tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/
>>> device-mapper/linux-dm". Does anyone have a suggestion how to analyze this
>>> further or how to fix this ?
>
> I still don't have a _known_ fix for your issue but I reviewed commit
> e809917735ebf1b9a56c24e877ce0d320baee2ec closer and identified what
> looks to be a regression in logic for multipath_busy, it now calls
> !pg_ready() instead of directly checking pg_init_in_progress.  I think
> this is needed (Hannes, what do you think?):
>
> diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
> index 3f6fd9d..561ead6 100644
> --- a/drivers/md/dm-mpath.c
> +++ b/drivers/md/dm-mpath.c
> @@ -373,7 +373,7 @@ static int __must_push_back(struct multipath *m)
>   		 dm_noflush_suspending(m->ti)));
>   }
>
> -#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required)
> +#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required && !(m)->pg_init_in_progress)
>
>   /*
>    * Map cloned requests
>
D'oh. You're correct. Can you try with this patch?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: v3.15 dm-mpath regression: cable pull test causes I/O hang

[Bart Van Assche]

On 07/03/14 00:02, Mike Snitzer wrote:
> On Fri, Jun 27 2014 at  9:33am -0400,
> Mike Snitzer <snitzer@redhat.com> wrote:
> 
>> On Fri, Jun 27 2014 at  9:02am -0400,
>> Bart Van Assche <bvanassche@acm.org> wrote:
>>
>>> Hello,
>>>
>>> While running a cable pull simulation test with dm_multipath on top of
>>> the SRP initiator driver I noticed that after a few iterations I/O locks
>>> up instead of dm_multipath processing the path failure properly (see also
>>> below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are
>>> vulnerable. This issue does not occur with kernel 3.14. I have tried to
>>> bisect this but gave up when I noticed that I/O locked up completely with
>>> a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec
>>> (dm mpath: push back requests instead of queueing). But with the bisect I
>>> have been able to narrow down this issue to one of the patches in "Merge
>>> tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/
>>> device-mapper/linux-dm". Does anyone have a suggestion how to analyze this
>>> further or how to fix this ?
> 
> I still don't have a _known_ fix for your issue but I reviewed commit
> e809917735ebf1b9a56c24e877ce0d320baee2ec closer and identified what
> looks to be a regression in logic for multipath_busy, it now calls
> !pg_ready() instead of directly checking pg_init_in_progress.  I think
> this is needed (Hannes, what do you think?):
> 
> diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
> index 3f6fd9d..561ead6 100644
> --- a/drivers/md/dm-mpath.c
> +++ b/drivers/md/dm-mpath.c
> @@ -373,7 +373,7 @@ static int __must_push_back(struct multipath *m)
>  		 dm_noflush_suspending(m->ti)));
>  }
>  
> -#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required)
> +#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required && !(m)->pg_init_in_progress)
>  
>  /*
>   * Map cloned requests

Hello Mike,

Sorry but even with this patch applied and additionally with commit IDs
86d56134f1b6 ("kobject: Make support for uevent_helper optional") and
bcccff93af35 ("kobject: don't block for each kobject_uevent") reverted
my multipath test still hangs after a few iterations. I also reran the
same test with kernel 3.14.3 and it is still running after 30 iterations.

Bart.

Re: v3.15 dm-mpath regression: cable pull test causes I/O hang

[Hannes Reinecke]

On 07/03/2014 03:56 PM, Bart Van Assche wrote:
> On 07/03/14 00:02, Mike Snitzer wrote:
>> On Fri, Jun 27 2014 at  9:33am -0400,
>> Mike Snitzer <snitzer@redhat.com> wrote:
>>
>>> On Fri, Jun 27 2014 at  9:02am -0400,
>>> Bart Van Assche <bvanassche@acm.org> wrote:
>>>
>>>> Hello,
>>>>
>>>> While running a cable pull simulation test with dm_multipath on top of
>>>> the SRP initiator driver I noticed that after a few iterations I/O locks
>>>> up instead of dm_multipath processing the path failure properly (see also
>>>> below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are
>>>> vulnerable. This issue does not occur with kernel 3.14. I have tried to
>>>> bisect this but gave up when I noticed that I/O locked up completely with
>>>> a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec
>>>> (dm mpath: push back requests instead of queueing). But with the bisect I
>>>> have been able to narrow down this issue to one of the patches in "Merge
>>>> tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/
>>>> device-mapper/linux-dm". Does anyone have a suggestion how to analyze this
>>>> further or how to fix this ?
>>
>> I still don't have a _known_ fix for your issue but I reviewed commit
>> e809917735ebf1b9a56c24e877ce0d320baee2ec closer and identified what
>> looks to be a regression in logic for multipath_busy, it now calls
>> !pg_ready() instead of directly checking pg_init_in_progress.  I think
>> this is needed (Hannes, what do you think?):
>>
>> diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
>> index 3f6fd9d..561ead6 100644
>> --- a/drivers/md/dm-mpath.c
>> +++ b/drivers/md/dm-mpath.c
>> @@ -373,7 +373,7 @@ static int __must_push_back(struct multipath *m)
>>   		 dm_noflush_suspending(m->ti)));
>>   }
>>
>> -#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required)
>> +#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required && !(m)->pg_init_in_progress)
>>
>>   /*
>>    * Map cloned requests
>
> Hello Mike,
>
> Sorry but even with this patch applied and additionally with commit IDs
> 86d56134f1b6 ("kobject: Make support for uevent_helper optional") and
> bcccff93af35 ("kobject: don't block for each kobject_uevent") reverted
> my multipath test still hangs after a few iterations. I also reran the
> same test with kernel 3.14.3 and it is still running after 30 iterations.
>
Hmm. Would've been too easy.
Sigh.

Cheers,

Hannes

-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: v3.15 dm-mpath regression: cable pull test causes I/O hang

[Mike Snitzer]

On Thu, Jul 03 2014 at  9:56am -0400,
Bart Van Assche <bvanassche@acm.org> wrote:

> On 07/03/14 00:02, Mike Snitzer wrote:
> > On Fri, Jun 27 2014 at  9:33am -0400,
> > Mike Snitzer <snitzer@redhat.com> wrote:
> > 
> >> On Fri, Jun 27 2014 at  9:02am -0400,
> >> Bart Van Assche <bvanassche@acm.org> wrote:
> >>
> >>> Hello,
> >>>
> >>> While running a cable pull simulation test with dm_multipath on top of
> >>> the SRP initiator driver I noticed that after a few iterations I/O locks
> >>> up instead of dm_multipath processing the path failure properly (see also
> >>> below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are
> >>> vulnerable. This issue does not occur with kernel 3.14. I have tried to
> >>> bisect this but gave up when I noticed that I/O locked up completely with
> >>> a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec
> >>> (dm mpath: push back requests instead of queueing). But with the bisect I
> >>> have been able to narrow down this issue to one of the patches in "Merge
> >>> tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/
> >>> device-mapper/linux-dm". Does anyone have a suggestion how to analyze this
> >>> further or how to fix this ?
> > 
> > I still don't have a _known_ fix for your issue but I reviewed commit
> > e809917735ebf1b9a56c24e877ce0d320baee2ec closer and identified what
> > looks to be a regression in logic for multipath_busy, it now calls
> > !pg_ready() instead of directly checking pg_init_in_progress.  I think
> > this is needed (Hannes, what do you think?):
> > 
> > diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
> > index 3f6fd9d..561ead6 100644
> > --- a/drivers/md/dm-mpath.c
> > +++ b/drivers/md/dm-mpath.c
> > @@ -373,7 +373,7 @@ static int __must_push_back(struct multipath *m)
> >  		 dm_noflush_suspending(m->ti)));
> >  }
> >  
> > -#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required)
> > +#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required && !(m)->pg_init_in_progress)
> >  
> >  /*
> >   * Map cloned requests
> 
> Hello Mike,
> 
> Sorry but even with this patch applied and additionally with commit IDs
> 86d56134f1b6 ("kobject: Make support for uevent_helper optional") and
> bcccff93af35 ("kobject: don't block for each kobject_uevent") reverted
> my multipath test still hangs after a few iterations. I also reran the
> same test with kernel 3.14.3 and it is still running after 30 iterations.

OK, thanks for testing though!  I still think the patch is needed.

You are using queue_if_no_path, do you see hangs due to paths not being
restored after the "cable" is restored?  Any errors in the multipathd
userspace logging?  Or abnormal errors in kernel?  Basically I'm looking
for some other clue besides the hung task timeout spew.

How easy would it be to replicate your testbed?  Is it uniquely FIO hw
dependent?  How are you simulating the cable pull tests?

I'd love to setup a testbed that would enable me to chase this more
interactively rather than punting to you for testing.

Hannes, do you have a testbed for heavy cable pull testing?  Are you
able to replicate these hangs?

Re: v3.15 dm-mpath regression: cable pull test causes I/O hang

[Hannes Reinecke]

On 07/03/2014 04:05 PM, Mike Snitzer wrote:
> On Thu, Jul 03 2014 at  9:56am -0400,
> Bart Van Assche <bvanassche@acm.org> wrote:
>
>> On 07/03/14 00:02, Mike Snitzer wrote:
>>> On Fri, Jun 27 2014 at  9:33am -0400,
>>> Mike Snitzer <snitzer@redhat.com> wrote:
>>>
>>>> On Fri, Jun 27 2014 at  9:02am -0400,
>>>> Bart Van Assche <bvanassche@acm.org> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> While running a cable pull simulation test with dm_multipath on top of
>>>>> the SRP initiator driver I noticed that after a few iterations I/O locks
>>>>> up instead of dm_multipath processing the path failure properly (see also
>>>>> below for a call trace). At least kernel versions 3.15 and 3.16-rc2 are
>>>>> vulnerable. This issue does not occur with kernel 3.14. I have tried to
>>>>> bisect this but gave up when I noticed that I/O locked up completely with
>>>>> a kernel built from git commit ID e809917735ebf1b9a56c24e877ce0d320baee2ec
>>>>> (dm mpath: push back requests instead of queueing). But with the bisect I
>>>>> have been able to narrow down this issue to one of the patches in "Merge
>>>>> tag 'dm-3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/
>>>>> device-mapper/linux-dm". Does anyone have a suggestion how to analyze this
>>>>> further or how to fix this ?
>>>
>>> I still don't have a _known_ fix for your issue but I reviewed commit
>>> e809917735ebf1b9a56c24e877ce0d320baee2ec closer and identified what
>>> looks to be a regression in logic for multipath_busy, it now calls
>>> !pg_ready() instead of directly checking pg_init_in_progress.  I think
>>> this is needed (Hannes, what do you think?):
>>>
>>> diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
>>> index 3f6fd9d..561ead6 100644
>>> --- a/drivers/md/dm-mpath.c
>>> +++ b/drivers/md/dm-mpath.c
>>> @@ -373,7 +373,7 @@ static int __must_push_back(struct multipath *m)
>>>   		 dm_noflush_suspending(m->ti)));
>>>   }
>>>
>>> -#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required)
>>> +#define pg_ready(m) (!(m)->queue_io && !(m)->pg_init_required && !(m)->pg_init_in_progress)
>>>
>>>   /*
>>>    * Map cloned requests
>>
>> Hello Mike,
>>
>> Sorry but even with this patch applied and additionally with commit IDs
>> 86d56134f1b6 ("kobject: Make support for uevent_helper optional") and
>> bcccff93af35 ("kobject: don't block for each kobject_uevent") reverted
>> my multipath test still hangs after a few iterations. I also reran the
>> same test with kernel 3.14.3 and it is still running after 30 iterations.
>
> OK, thanks for testing though!  I still think the patch is needed.
>
> You are using queue_if_no_path, do you see hangs due to paths not being
> restored after the "cable" is restored?  Any errors in the multipathd
> userspace logging?  Or abnormal errors in kernel?  Basically I'm looking
> for some other clue besides the hung task timeout spew.
>
> How easy would it be to replicate your testbed?  Is it uniquely FIO hw
> dependent?  How are you simulating the cable pull tests?
>
> I'd love to setup a testbed that would enable me to chase this more
> interactively rather than punting to you for testing.
>
> Hannes, do you have a testbed for heavy cable pull testing?  Are you
> able to replicate these hangs?
>
Yes, I do. But sadly I've been tied up with polishing up SLES12 
(release deadline is looming nearer) and for some inexplicable 
reason management seems to find releasing a product more important 
than working on mainline issue ...
But I hope to find some time soonish (ie start of next week) to work 
on this; it's the very next thing on my to-do list.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)

Re: v3.15 dm-mpath regression: cable pull test causes I/O hang

[Mike Snitzer]

On Thu, Jul 03 2014 at 10:15am -0400,
Hannes Reinecke <hare@suse.de> wrote:

> On 07/03/2014 04:05 PM, Mike Snitzer wrote:
> >
> >Hannes, do you have a testbed for heavy cable pull testing?  Are you
> >able to replicate these hangs?
> >
> Yes, I do. But sadly I've been tied up with polishing up SLES12
> (release deadline is looming nearer) and for some inexplicable
> reason management seems to find releasing a product more important
> than working on mainline issue ...

OK, so long as you didn't include these dm-mpath changes.. but if you
did then you could kill 2 birds with one stone ;)

> But I hope to find some time soonish (ie start of next week) to work
> on this; it's the very next thing on my to-do list.

OK, I really appreciate it, thanks.

Re: v3.15 dm-mpath regression: cable pull test causes I/O hang

[Bart Van Assche]

On 07/03/14 16:05, Mike Snitzer wrote:
> How easy would it be to replicate your testbed?  Is it uniquely FIO hw
> dependent?  How are you simulating the cable pull tests?
> 
> I'd love to setup a testbed that would enable me to chase this more
> interactively rather than punting to you for testing.

Hello Mike,

The only nonstandard hardware that is required to run my test is a pair
of InfiniBand HCA's and an IB cable to connect these back-to-back. The
test I ran is as follows:
* Let an SRP initiator log in to an SRP target system.
* Start multipathd and srpd.
* Start a fio data integrity test on the initiator system on top of
  /dev/dm-0.
* From the target system simulate a cable pull by disabling IB traffic
  via the ibportstate command.
* After a random delay, unload and reload SCST and the IB stack. This
  makes the IB ports operational again.
* After a random delay, repeat the previous two steps.

If you want I can send you the scripts I use to run this test and also
the instructions that are necessary to build and install the SCST SRP
target driver.

Bart.

Re: v3.15 dm-mpath regression: cable pull test causes I/O hang

[Mike Snitzer]

On Thu, Jul 03 2014 at 10:34am -0400,
Bart Van Assche <bvanassche@acm.org> wrote:

> On 07/03/14 16:05, Mike Snitzer wrote:
> > How easy would it be to replicate your testbed?  Is it uniquely FIO hw
> > dependent?  How are you simulating the cable pull tests?
> > 
> > I'd love to setup a testbed that would enable me to chase this more
> > interactively rather than punting to you for testing.
> 
> Hello Mike,
> 
> The only nonstandard hardware that is required to run my test is a pair
> of InfiniBand HCA's and an IB cable to connect these back-to-back. The
> test I ran is as follows:
> * Let an SRP initiator log in to an SRP target system.
> * Start multipathd and srpd.
> * Start a fio data integrity test on the initiator system on top of
>   /dev/dm-0.
> * From the target system simulate a cable pull by disabling IB traffic
>   via the ibportstate command.
> * After a random delay, unload and reload SCST and the IB stack. This
>   makes the IB ports operational again.
> * After a random delay, repeat the previous two steps.

I'll work on getting some IB cards.  But I _should_ be able to achieve
the same using iSCSI right?

> If you want I can send you the scripts I use to run this test and also
> the instructions that are necessary to build and install the SCST SRP
> target driver.

Please do, thanks!

Also, Red Hat has a rather extensive battery of dm-mpath FC cable pull
tests for RHEL but these dm-mpath changes haven't been included in any
RHEL yet.. I _could_ port these upstream changes to a test RHEL7 kernel
just to leverage the RHEL-based testing.

Re: v3.15 dm-mpath regression: cable pull test causes I/O hang

[Bart Van Assche]

[-- Attachment #1: Type: text/plain, Size: 2763 bytes --]

On 07/03/14 17:00, Mike Snitzer wrote:
> On Thu, Jul 03 2014 at 10:34am -0400,
> Bart Van Assche <bvanassche@acm.org> wrote:
> 
>> On 07/03/14 16:05, Mike Snitzer wrote:
>>> How easy would it be to replicate your testbed?  Is it uniquely FIO hw
>>> dependent?  How are you simulating the cable pull tests?
>>>
>>> I'd love to setup a testbed that would enable me to chase this more
>>> interactively rather than punting to you for testing.
>>
>> Hello Mike,
>>
>> The only nonstandard hardware that is required to run my test is a pair
>> of InfiniBand HCA's and an IB cable to connect these back-to-back. The
>> test I ran is as follows:
>> * Let an SRP initiator log in to an SRP target system.
>> * Start multipathd and srpd.
>> * Start a fio data integrity test on the initiator system on top of
>>   /dev/dm-0.
>> * From the target system simulate a cable pull by disabling IB traffic
>>   via the ibportstate command.
>> * After a random delay, unload and reload SCST and the IB stack. This
>>   makes the IB ports operational again.
>> * After a random delay, repeat the previous two steps.
> 
> I'll work on getting some IB cards.  But I _should_ be able to achieve
> the same using iSCSI right?

I'm not sure. There are differences between the SRP and iSCSI initiator
that could matter here, e.g. that the SRP initiator triggers
scsi_remove_host() some time after a path failure occurred but the iSCSI
initiator not. So far I have not yet been able to trigger this issue
with the iSCSI initiator with replacement_timeout = 1 and by using the
following loop to simulate path failures: while true; do iptables -A
INPUT -p tcp --destination-port 3260 -j DROP; sleep 10; iptables -D
INPUT -p tcp --destination-port 3260 -j DROP; sleep 10; done

>> If you want I can send you the scripts I use to run this test and also
>> the instructions that are necessary to build and install the SCST SRP
>> target driver.
> 
> Please do, thanks!

The test I run at the initiator side is as follows:

# modprobe ib_srp
# systemctl restart srpd
# systemctl start multipathd
# mkfs.ext4 -FO ^has_journal /dev/dm-0
# umount /mnt; fsck /dev/dm-0 && mount /dev/dm-0 /mnt && rm -f
/mnt/test* && fio --verify=md5 --rw=randwrite --size=10M --bs=4K
--iodepth=64 --sync=1 --direct=1 --ioengine=libaio --directory=/mnt
--name=test --thread --numjobs=1 --loops=$((10**9))

The script I run at the target side is as follows (should also be
possible with the upstream SRP target driver instead of SCST):
* Download, build and install SCST.
* Create a configuration file (/etc/scst.conf) in which /dev/ram0 is
exported via the vdisk_blockio driver.
* Start SCST.
* Run the attached toggle-ib-port-loop script e.g. as follows:
initiator=${initiator_host_name} toggle-ib-port-loop

Bart.

[-- Attachment #2: toggle-ib-port-loop --]
[-- Type: text/plain, Size: 1473 bytes --]

#!/bin/bash

# How to start this test.
# On the initiator system, run:
# ~bart/bin/reload-srp-initiator
# /etc/init.d/srpd start
# mkfs.ext4 -O ^has_journal /dev/sdb
# /etc/init.d/multipathd start
# umount /mnt; mount /dev/dm-0 /mnt && rm -f /mnt/test* && ~bart/bin/fio-stress-test-6 /mnt 16
# On the target system, run:
# initiator=antec ~bart/software/tools/toggle-ib-port-loop

function port_guid() {
    local gid guid

    gid="$(</sys/class/infiniband/mlx4_0/ports/$1/gids/0)" || return $?
    guid="${gid#fe80:0000:0000:0000}"
    echo "0x${guid//:/}"
}

if [ -z "${initiator}" ]; then
    echo "Error: variable \${initiator} has not been set"
    exit 1
fi

guid1="$(port_guid 1)"
guid2="$(port_guid 2)"

set -x

/etc/init.d/srpd stop
while true; do
    ssh ${initiator} ibportstate -G "$guid1" 1 disable
    ssh ${initiator} ibportstate -G "$guid2" 2 disable
    sleep $((RANDOM*150/32767))
    /etc/init.d/scst stop
    /etc/init.d/opensmd stop
    /etc/init.d/openibd stop
    for m in mlx4_en mlx4_ib mlx4_core; do
	modprobe -r $m
    done
    /etc/init.d/openibd start
    /etc/init.d/opensmd start
    umount /dev/sr1
    ibstat |
        sed -n 's/^[[:blank:]]*Port GUID: 0x\(..\)\(..\)\(..\)....\(..\)\(..\)\(..\)/00:\2:\3:\4:\5:\6/p' |
        while read a; do
	    p="$(cd /sys/class/net && grep -lw $a */address)"
	    if [ -n "$p" ]; then
	        ifup "$(dirname $p)"
	    fi
	done
    /etc/init.d/scst restart
    sleep $((30 + RANDOM*30/32767))
done

[-- Attachment #3: Type: text/plain, Size: 0 bytes --]

Re: v3.15 dm-mpath regression: cable pull test causes I/O hang

[Junichi Nomura]

On 07/03/14 23:34, Bart Van Assche wrote:
> On 07/03/14 16:05, Mike Snitzer wrote:
> The only nonstandard hardware that is required to run my test is a pair
> of InfiniBand HCA's and an IB cable to connect these back-to-back. The
> test I ran is as follows:
> * Let an SRP initiator log in to an SRP target system.
> * Start multipathd and srpd.
> * Start a fio data integrity test on the initiator system on top of
>   /dev/dm-0.
> * From the target system simulate a cable pull by disabling IB traffic
>   via the ibportstate command.
> * After a random delay, unload and reload SCST and the IB stack. This
>   makes the IB ports operational again.
> * After a random delay, repeat the previous two steps.
> 
> If you want I can send you the scripts I use to run this test and also
> the instructions that are necessary to build and install the SCST SRP
> target driver.

Hi Bart,

could you collect output of 'dmsetup table' on the test machine?
My particular interest is whether pg_init is involved or not
since Hannes's patch made major surgery on pg_init processing.

Also, output of the followings while the problem is happening might help:
  dmsetup status
  dmsetup info -c
  cat /proc/diskstat
  multipathd -k"show paths"
  multipathd -k"show maps status"
  multipathd -k"show maps stats"

-- 
Jun'ichi Nomura, NEC Corporation

Re: v3.15 dm-mpath regression: cable pull test causes I/O hang

[Bart Van Assche]

[-- Attachment #1: Type: text/plain, Size: 654 bytes --]

On 07/04/14 05:10, Junichi Nomura wrote:
> could you collect output of 'dmsetup table' on the test machine?
> My particular interest is whether pg_init is involved or not
> since Hannes's patch made major surgery on pg_init processing.
> 
> Also, output of the followings while the problem is happening might help:
>   dmsetup status
>   dmsetup info -c
>   cat /proc/diskstat
>   multipathd -k"show paths"
>   multipathd -k"show maps status"
>   multipathd -k"show maps stats"

Hello Junichi,

Thanks for looking into this issue. I have attached the requested
information to this e-mail for a test run with kernel 3.16-rc4 at the
initiator side.

Bart.

[-- Attachment #2: multipath-status.txt --]
[-- Type: text/plain, Size: 34727 bytes --]

# dmsetup table
26133373139386639: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:16 1000 1 65:192 1000 1 
26230643330303865: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:48 1000 1 65:208 1000 1 
23834333461383137: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:208 1000 1 65:128 1000 1 
26361626563396362: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:80 1000 1 65:224 1000 1 
26363633765623538: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:32 1000 1 8:128 1000 1 
23832346438613834: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:96 1000 1 65:0 1000 1 
26638323034336331: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:80 1000 1 8:224 1000 1 
23937393633303630: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:240 1000 1 65:160 1000 1 
23931656631326633: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:112 1000 1 65:32 1000 1 
26665353936313532: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:176 1000 1 66:16 1000 1 
26135303862613661: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:144 1000 1 65:64 1000 1 
26439316335316263: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:112 1000 1 65:240 1000 1 
26564666266393235: 0 5368709120 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 65:144 1000 1 66:0 1000 1 
26562383264626236: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:64 1000 1 8:192 1000 1 
26236616132323164: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:176 1000 1 65:96 1000 1 
26466363537333266: 0 256000 multipath 3 queue_if_no_path pg_init_retries 50 0 1 1 service-time 0 2 2 8:48 1000 1 8:160 1000 1 
# dmsetup status
26133373139386639: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 65:16 A 0 0 1 65:192 A 0 0 1 
26230643330303865: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 65:48 A 0 0 1 65:208 A 0 0 1 
23834333461383137: 0 5368709120 multipath 2 1 0 0 1 1 E 0 2 2 8:208 A 0 0 1 65:128 A 0 0 1 
26361626563396362: 0 5368709120 multipath 2 1 0 0 1 1 E 0 2 2 65:80 A 0 0 1 65:224 A 0 0 1 
26363633765623538: 0 256000 multipath 2 0 0 0 1 1 A 0 2 2 8:32 A 0 0 1 8:128 A 0 0 1 
23832346438613834: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:96 A 0 0 1 65:0 A 0 0 1 
26638323034336331: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:80 A 0 0 1 8:224 A 0 0 1 
23937393633303630: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 8:240 A 0 0 1 65:160 A 0 0 1 
23931656631326633: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:112 A 0 0 1 65:32 A 0 0 1 
26665353936313532: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 65:176 A 0 0 1 66:16 A 0 0 1 
26135303862613661: 0 256000 multipath 2 0 0 0 1 1 A 0 2 2 8:144 A 0 0 1 65:64 A 0 0 1 
26439316335316263: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 65:112 A 0 0 1 65:240 A 0 0 1 
26564666266393235: 0 5368709120 multipath 2 0 0 0 1 1 A 0 2 2 65:144 A 0 0 1 66:0 A 0 0 1 
26562383264626236: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:64 A 0 0 1 8:192 A 0 0 1 
26236616132323164: 0 256000 multipath 2 0 0 0 1 1 A 0 2 2 8:176 A 0 0 1 65:96 A 0 0 1 
26466363537333266: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:48 A 0 0 1 8:160 A 0 0 1
# dmsetup info -c
Name              Maj Min Stat Open Targ Event  UUID                            
26133373139386639 253  10 L--w    0    1      0 mpath-26133373139386639         
26230643330303865 253  11 L--w    0    1      0 mpath-26230643330303865         
23834333461383137 253   9 L--w    1    1      0 mpath-23834333461383137         
26361626563396362 253  13 L--w    1    1      0 mpath-26361626563396362         
26363633765623538 253   0 L--w    1    1      5 mpath-26363633765623538         
23832346438613834 253   4 L--w    1    1      0 mpath-23832346438613834         
26638323034336331 253   3 L--w    1    1      0 mpath-26638323034336331         
23937393633303630 253   8 L--w    0    1      0 mpath-23937393633303630         
23931656631326633 253   5 L--w    1    1      0 mpath-23931656631326633         
26665353936313532 253  15 L--w    0    1      0 mpath-26665353936313532         
26135303862613661 253   6 L--w    0    1      0 mpath-26135303862613661         
26439316335316263 253  12 L--w    0    1      0 mpath-26439316335316263         
26564666266393235 253  14 L--w    0    1      0 mpath-26564666266393235         
26562383264626236 253   2 L--w    1    1      0 mpath-26562383264626236         
26236616132323164 253   7 L--w    0    1      0 mpath-26236616132323164         
26466363537333266 253   1 L--w    1    1      0 mpath-26466363537333266 
# cat /proc/diskstats
   8       0 sda 63494 687 2082482 45080 11530 6225 362768 690500 0 23530 734810
   8       1 sda1 170 0 1360 190 0 0 0 0 0 190 190
   8       2 sda2 63143 687 2079674 44620 11509 6225 362768 690490 0 23500 734330
   8      16 sdb 374 6 3040 460 0 0 0 0 0 460 460
   8      17 sdb1 180 0 1440 190 0 0 0 0 0 190 190
  11       0 sr0 0 0 0 0 0 0 0 0 0 0 0
 253       0 dm-0 283565 1406 2268002 22758620 295490 134 2560844 532030 0 671930 23290480
   8      32 sdc 1489 0 11912 30130 1168 0 9320 1020 0 1640 31150
   8      48 sdd 362 0 2896 1310 0 0 0 0 0 1310 1310
   8      64 sde 347 0 2776 1180 0 0 0 0 0 1170 1180
   8      80 sdf 346 0 2768 910 0 0 0 0 0 900 910
   8      96 sdg 389 0 3112 1020 0 0 0 0 0 1020 1020
   8     112 sdh 417 0 3336 1080 0 0 0 0 0 1070 1080
   8     128 sdi 234 0 1872 450 1132 0 9050 1590 0 2030 2040
   8     144 sdj 422 0 3400 1050 0 0 0 0 0 1040 1050
   8     160 sdk 234 0 1872 520 0 0 0 0 0 520 520
   8     176 sdl 421 0 3392 1030 0 0 0 0 0 1020 1030
   8     192 sdm 234 0 1872 510 0 0 0 0 0 510 510
   8     208 sdn 369 0 2952 880 0 0 0 0 0 880 880
   8     224 sdo 234 0 1872 500 0 0 0 0 0 500 500
 253       1 dm-1 127 0 1016 1070 0 0 0 0 1 278360 278820
   8     240 sdp 425 0 3424 890 0 0 0 0 0 880 890
  65       0 sdq 234 0 1872 500 0 0 0 0 0 500 500
  65      16 sdr 424 0 3416 910 0 0 0 0 0 910 910
 253       2 dm-2 112 0 896 1050 0 0 0 0 1 278240 278710
  65      32 sds 233 0 1864 590 0 0 0 0 0 590 590
  65      48 sdt 426 0 3432 860 0 0 0 0 0 850 860
  65      64 sdu 421 0 3392 830 0 0 0 0 0 830 830
 253       3 dm-3 111 0 888 1120 0 0 0 0 1 278170 278700
  65      96 sdw 422 0 3400 910 0 0 0 0 0 910 910
  65      80 sdv 404 0 3232 900 0 0 0 0 0 900 900
 253       4 dm-4 154 0 1232 1310 0 0 0 0 1 278110 278720
  65     112 sdx 425 0 3424 960 0 0 0 0 0 960 960
  65     128 sdy 235 0 1880 570 0 0 0 0 0 570 570
 253       5 dm-5 183 0 1464 1410 0 0 0 0 1 278050 278700
  65     144 sdz 425 0 3424 820 0 0 0 0 0 820 820
  65     160 sdaa 425 0 3424 830 0 0 0 0 0 830 830
 253       6 dm-6 374 6 3040 1750 0 0 0 0 0 940 1750
  65     176 sdab 426 0 3432 820 0 0 0 0 0 820 820
  65     192 sdac 428 0 3448 830 0 0 0 0 0 830 830
  65     208 sdad 425 0 3424 740 0 0 0 0 0 740 740
 253       7 dm-7 374 6 3040 1620 0 0 0 0 0 880 1620
  65     224 sdae 236 0 1888 500 0 0 0 0 0 500 500
  65     240 sdaf 427 0 3440 640 0 0 0 0 0 640 640
  66       0 sdag 427 0 3440 530 0 0 0 0 0 530 530
  66      16 sdah 426 0 3432 540 0 0 0 0 0 540 540
 253       8 dm-8 378 6 3072 1190 0 0 0 0 0 640 1190
 253       9 dm-9 134 0 1072 520 0 0 0 0 1 277420 277640
 253      10 dm-10 378 6 3072 1180 0 0 0 0 0 660 1180
 253      11 dm-11 378 6 3072 1000 0 0 0 0 0 530 1000
 253      12 dm-12 378 6 3072 990 0 0 0 0 0 560 990
 253      13 dm-13 168 0 1344 500 0 0 0 0 1 276920 277150
 253      14 dm-14 378 6 3072 910 0 0 0 0 0 480 910
 253      15 dm-15 378 6 3072 950 0 0 0 0 0 500 950
# multipathd -k"show paths"
hcil      dev  dev_t  pri dm_st  chk_st dev_st  next_check    
15:0:0:0  sdc  8:32   1   active ready  running XX........ 1/4
15:0:0:1  sdd  8:48   1   active ready  running XX........ 1/4
15:0:0:2  sde  8:64   1   active ready  running XX........ 1/4
15:0:0:3  sdf  8:80   1   active ready  running XX........ 1/4
15:0:0:4  sdg  8:96   1   active ready  running XX........ 1/4
15:0:0:5  sdh  8:112  1   active ready  running XX........ 1/4
15:0:0:6  sdj  8:144  1   active ready  running XX........ 1/4
16:0:0:0  sdi  8:128  1   active ready  running XX........ 1/4
15:0:0:7  sdl  8:176  1   active ready  running XX........ 1/4
16:0:0:1  sdk  8:160  1   active ready  running XX........ 1/4
16:0:0:2  sdm  8:192  1   active ready  running XX........ 1/4
16:0:0:3  sdo  8:224  1   active ready  running XX........ 1/4
15:0:0:11 sdp  8:240  1   active ready  running XX........ 1/4
15:0:0:10 sdn  8:208  1   active ready  running XX........ 1/4
16:0:0:4  sdq  65:0   1   active ready  running XX........ 1/4
15:0:0:12 sdr  65:16  1   active ready  running XX........ 1/4
16:0:0:5  sds  65:32  1   active ready  running XX........ 1/4
16:0:0:6  sdu  65:64  1   active ready  running XX........ 1/4
15:0:0:13 sdt  65:48  1   active ready  running XX........ 1/4
16:0:0:10 sdy  65:128 1   active ready  running XX........ 1/4
16:0:0:7  sdw  65:96  1   active ready  running XX........ 1/4
15:0:0:15 sdx  65:112 1   active ready  running XX........ 1/4
15:0:0:14 sdv  65:80  1   active ready  running XX........ 1/4
15:0:0:16 sdz  65:144 1   active ready  running XX........ 1/4
15:0:0:17 sdab 65:176 1   active ready  running XX........ 1/4
16:0:0:12 sdac 65:192 1   active ready  running XX........ 1/4
16:0:0:11 sdaa 65:160 1   active ready  running XX........ 1/4
16:0:0:13 sdad 65:208 1   active ready  running XX........ 1/4
16:0:0:14 sdae 65:224 1   active ready  running XX........ 1/4
16:0:0:15 sdaf 65:240 1   active ready  running XX........ 1/4
16:0:0:17 sdah 66:16  1   active ready  running XX........ 1/4
16:0:0:16 sdag 66:0   1   active ready  running XX........ 1/4
# multipathd -k"show maps status"
name              failback queueing paths dm-st  write_prot
26363633765623538 -        on       2     active rw        
26466363537333266 -        on       2     active rw        
26562383264626236 -        on       2     active rw        
26638323034336331 -        on       2     active rw        
23832346438613834 -        on       2     active rw        
23931656631326633 -        on       2     active rw        
26135303862613661 -        on       2     active rw        
26236616132323164 -        on       2     active rw        
23937393633303630 -        on       2     active rw        
23834333461383137 -        on       2     active rw        
26133373139386639 -        on       2     active rw        
26230643330303865 -        on       2     active rw        
26439316335316263 -        on       2     active rw        
26361626563396362 -        on       2     active rw        
26564666266393235 -        on       2     active rw        
26665353936313532 -        on       2     active rw
# multipathd -k"show maps stats"
name              path_faults switch_grp map_loads total_q_time q_timeouts
26363633765623538 5           0          9         0            0         
26466363537333266 0           0          2         0            0         
26562383264626236 0           0          2         0            0         
26638323034336331 0           0          2         0            0         
23832346438613834 0           0          2         0            0         
23931656631326633 0           0          2         0            0         
26135303862613661 0           0          2         0            0         
26236616132323164 0           0          2         0            0         
23937393633303630 0           0          2         0            0         
23834333461383137 0           0          2         0            0         
26133373139386639 0           0          2         0            0         
26230643330303865 0           0          2         0            0         
26439316335316263 0           0          2         0            0         
26361626563396362 0           0          2         0            0         
26564666266393235 0           0          2         0            0         
26665353936313532 0           0          2         0            0         





SysRq : Show Blocked State
  task                        PC stack   pid father
systemd-udevd   D ffff88083b6f8000     0  8283    358 0x00000000
 ffff8807c69f3b30 0000000000000002 ffff88083b6f8000 ffff8807c69f3fd8
 00000000000131c0 00000000000131c0 ffffffff81a154c0 ffff8807d7578698
 ffff8807d75786a0 0000000000000246 ffff88083b6f8000 0000000000000000
Call Trace:
 [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80
 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0
 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0
 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff811d53db>] blkdev_open+0x5b/0x80
 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320
 [<ffffffff81196210>] finish_open+0x30/0x40
 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150
 [<ffffffff811a4725>] ? path_init+0x5d5/0x750
 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0
 [<ffffffff811a8a58>] ? path_openat+0x78/0x620
 [<ffffffff811a8a97>] path_openat+0xb7/0x620
 [<ffffffff811a967a>] do_filp_open+0x3a/0x90
 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0
 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210
 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200
 [<ffffffff81197b5e>] SyS_open+0x1e/0x20
 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5
systemd-udevd   D ffff88083b322230     0  8317    358 0x00000000
 ffff8807cfa27b30 0000000000000002 ffff88083b322230 ffff8807cfa27fd8
 00000000000131c0 00000000000131c0 ffff88083b490000 ffff88083ec1a098
 ffff88083ec1a0a0 0000000000000246 ffff88083b322230 0000000000000000
Call Trace:
 [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80
 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0
 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0
 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff811d53db>] blkdev_open+0x5b/0x80
 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320
 [<ffffffff81196210>] finish_open+0x30/0x40
 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150
 [<ffffffff811a4725>] ? path_init+0x5d5/0x750
 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0
 [<ffffffff811a8a58>] ? path_openat+0x78/0x620
 [<ffffffff811a8a97>] path_openat+0xb7/0x620
 [<ffffffff811a967a>] do_filp_open+0x3a/0x90
 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0
 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210
 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200
 [<ffffffff81197b5e>] SyS_open+0x1e/0x20
 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5
systemd-udevd   D ffff88083b00a230     0  8329    358 0x00000000
 ffff8807ee1b3b30 0000000000000002 ffff88083b00a230 ffff8807ee1b3fd8
 00000000000131c0 00000000000131c0 ffff88083baea230 ffff88083ec1b418
 ffff88083ec1b420 0000000000000246 ffff88083b00a230 0000000000000000
Call Trace:
 [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80
 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0
 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0
 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff811d53db>] blkdev_open+0x5b/0x80
 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320
 [<ffffffff81196210>] finish_open+0x30/0x40
 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150
 [<ffffffff811a4725>] ? path_init+0x5d5/0x750
 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0
 [<ffffffff811a8a58>] ? path_openat+0x78/0x620
 [<ffffffff811a8a97>] path_openat+0xb7/0x620
 [<ffffffff811a967a>] do_filp_open+0x3a/0x90
 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0
 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210
 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200
 [<ffffffff81197b5e>] SyS_open+0x1e/0x20
 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5
systemd-udevd   D ffff8807d1290000     0  8330    358 0x00000000
 ffff8807cee23b30 0000000000000002 ffff8807d1290000 ffff8807cee23fd8
 00000000000131c0 00000000000131c0 ffff88083b492230 ffff8807d77a6e98
 ffff8807d77a6ea0 0000000000000246 ffff8807d1290000 0000000000000000
Call Trace:
 [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80
 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0
 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0
 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff811d53db>] blkdev_open+0x5b/0x80
 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320
 [<ffffffff81196210>] finish_open+0x30/0x40
 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150
 [<ffffffff811a4725>] ? path_init+0x5d5/0x750
 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0
 [<ffffffff811a8a58>] ? path_openat+0x78/0x620
 [<ffffffff811a8a97>] path_openat+0xb7/0x620
 [<ffffffff811a967a>] do_filp_open+0x3a/0x90
 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0
 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210
 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200
 [<ffffffff81197b5e>] SyS_open+0x1e/0x20
 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5
systemd-udevd   D ffff8807d168c460     0  8334    358 0x00000000
 ffff8807d13bfb30 0000000000000002 ffff8807d168c460 ffff8807d13bffd8
 00000000000131c0 00000000000131c0 ffff88083b422230 ffff8807d77a5498
 ffff8807d77a54a0 0000000000000246 ffff8807d168c460 0000000000000000
Call Trace:
 [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80
 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0
 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0
 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff811d53db>] blkdev_open+0x5b/0x80
 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320
 [<ffffffff81196210>] finish_open+0x30/0x40
 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150
 [<ffffffff811a4725>] ? path_init+0x5d5/0x750
 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0
 [<ffffffff811a8a58>] ? path_openat+0x78/0x620
 [<ffffffff811a8a97>] path_openat+0xb7/0x620
 [<ffffffff811a967a>] do_filp_open+0x3a/0x90
 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0
 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210
 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200
 [<ffffffff81197b5e>] SyS_open+0x1e/0x20
 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5
systemd-udevd   D ffff8807d1688000     0  8335    358 0x00000000
 ffff8807cf5f3b30 0000000000000002 ffff8807d1688000 ffff8807cf5f3fd8
 00000000000131c0 00000000000131c0 ffff88083b494460 ffff88083ec1a718
 ffff88083ec1a720 0000000000000246 ffff8807d1688000 0000000000000000
Call Trace:
 [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80
 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0
 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0
 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff811d53db>] blkdev_open+0x5b/0x80
 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320
 [<ffffffff81196210>] finish_open+0x30/0x40
 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150
 [<ffffffff811a4725>] ? path_init+0x5d5/0x750
 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0
 [<ffffffff811a8a58>] ? path_openat+0x78/0x620
 [<ffffffff811a8a97>] path_openat+0xb7/0x620
 [<ffffffff811a967a>] do_filp_open+0x3a/0x90
 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0
 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210
 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200
 [<ffffffff81197b5e>] SyS_open+0x1e/0x20
 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5
systemd-udevd   D ffff88082df38000     0  8341    358 0x00000000
 ffff8807d1763b30 0000000000000002 ffff88082df38000 ffff8807d1763fd8
 00000000000131c0 00000000000131c0 ffff88083b422230 ffff8807d757e818
 ffff8807d757e820 0000000000000246 ffff88082df38000 0000000000000000
Call Trace:
 [<ffffffff814bc3a3>] schedule_preempt_disabled+0x33/0x80
 [<ffffffff814bd3dc>] mutex_lock_nested+0x15c/0x3d0
 [<ffffffff811d4b56>] ? __blkdev_get+0x66/0x4c0
 [<ffffffff811d4b56>] __blkdev_get+0x66/0x4c0
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff811d5195>] blkdev_get+0x1e5/0x380
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
 [<ffffffff811d5380>] ? blkdev_get_by_dev+0x50/0x50
 [<ffffffff811d53db>] blkdev_open+0x5b/0x80
 [<ffffffff81195fc2>] do_dentry_open.isra.15+0x1e2/0x320
 [<ffffffff81196210>] finish_open+0x30/0x40
 [<ffffffff811a82cd>] do_last.isra.61+0xa3d/0x1150
 [<ffffffff811a4725>] ? path_init+0x5d5/0x750
 [<ffffffff810a6900>] ? trace_hardirqs_on_caller+0xc0/0x1c0
 [<ffffffff811a8a58>] ? path_openat+0x78/0x620
 [<ffffffff811a8a97>] path_openat+0xb7/0x620
 [<ffffffff811a967a>] do_filp_open+0x3a/0x90
 [<ffffffff814c1091>] ? _raw_spin_unlock+0x31/0x50
 [<ffffffff811b8a1c>] ? __alloc_fd+0xac/0x1b0
 [<ffffffff81197a5e>] do_sys_open+0x12e/0x210
 [<ffffffff8101209d>] ? syscall_trace_enter+0xfd/0x200
 [<ffffffff81197b5e>] SyS_open+0x1e/0x20
 [<ffffffff814c1ea2>] tracesys+0xd0/0xd5
blkid           D ffff8807d3110000     0  8422      1 0x00000006
 ffff8807d59879b8 0000000000000002 ffff8807d3110000 ffff8807d5987fd8
 00000000000131c0 00000000000131c0 ffff88083b4c4460 ffff88085fd33ad0
 ffff88085ff83038 ffff8807d5987a40 0000000000000002 ffffffff81135db0
Call Trace:
 [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60
 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130
 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20
 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0
 [<ffffffff81135eca>] __lock_page+0x6a/0x70
 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40
 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690
 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20
 [<ffffffff811d3425>] kill_bdev+0x35/0x40
 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0
 [<ffffffff811d5450>] blkdev_put+0x50/0x160
 [<ffffffff811d5615>] blkdev_close+0x25/0x30
 [<ffffffff8119a37a>] __fput+0xea/0x1f0
 [<ffffffff8119a4ce>] ____fput+0xe/0x10
 [<ffffffff81074ddc>] task_work_run+0xac/0xe0
 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60
 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940
 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60
 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0
 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940
 [<ffffffff81002528>] do_signal+0x48/0x630
 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40
 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0
 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20
 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0
 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0
 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0
 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0
 [<ffffffff814c1f98>] int_signal+0x12/0x17
blkid           D ffff8807d3632230     0  8458      1 0x00000006
 ffff8807d360f9b8 0000000000000002 ffff8807d3632230 ffff8807d360ffd8
 00000000000131c0 00000000000131c0 ffff88083b492230 ffff88085fc93ad0
 ffff88085ff7fa38 ffff8807d360fa40 0000000000000002 ffffffff81135db0
Call Trace:
 [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60
 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130
 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20
 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0
 [<ffffffff81135eca>] __lock_page+0x6a/0x70
 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40
 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690
 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20
 [<ffffffff811d3425>] kill_bdev+0x35/0x40
 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0
 [<ffffffff811d5450>] blkdev_put+0x50/0x160
 [<ffffffff811d5615>] blkdev_close+0x25/0x30
 [<ffffffff8119a37a>] __fput+0xea/0x1f0
 [<ffffffff8119a4ce>] ____fput+0xe/0x10
 [<ffffffff81074ddc>] task_work_run+0xac/0xe0
 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60
 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940
 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60
 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0
 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940
 [<ffffffff81002528>] do_signal+0x48/0x630
 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40
 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0
 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20
 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0
 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0
 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0
 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0
 [<ffffffff814c1f98>] int_signal+0x12/0x17
blkid           D ffff8807d4432230     0  8467      1 0x00000006
 ffff8807d4daf9b8 0000000000000002 ffff8807d4432230 ffff8807d4daffd8
 00000000000131c0 00000000000131c0 ffff88083b4c2230 ffff88085fd53ad0
 ffff88085ffa4338 ffff8807d4dafa40 0000000000000002 ffffffff81135db0
Call Trace:
 [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60
 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130
 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20
 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0
 [<ffffffff81135eca>] __lock_page+0x6a/0x70
 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40
 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690
 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20
 [<ffffffff811d3425>] kill_bdev+0x35/0x40
 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0
 [<ffffffff811d5450>] blkdev_put+0x50/0x160
 [<ffffffff811d5615>] blkdev_close+0x25/0x30
 [<ffffffff8119a37a>] __fput+0xea/0x1f0
 [<ffffffff8119a4ce>] ____fput+0xe/0x10
 [<ffffffff81074ddc>] task_work_run+0xac/0xe0
 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60
 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940
 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60
 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0
 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940
 [<ffffffff81002528>] do_signal+0x48/0x630
 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40
 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0
 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20
 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0
 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0
 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0
 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0
 [<ffffffff814c1f98>] int_signal+0x12/0x17
blkid           D ffff8807d275a230     0  8472      1 0x00000006
 ffff8807d27bb9b8 0000000000000002 ffff8807d275a230 ffff8807d27bbfd8
 00000000000131c0 00000000000131c0 ffff8807d1290000 ffff88085fc93ad0
 ffff88085ff87838 ffff8807d27bba40 0000000000000002 ffffffff81135db0
Call Trace:
 [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60
 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130
 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20
 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0
 [<ffffffff81135eca>] __lock_page+0x6a/0x70
 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40
 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690
 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20
 [<ffffffff811d3425>] kill_bdev+0x35/0x40
 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0
 [<ffffffff811d5450>] blkdev_put+0x50/0x160
 [<ffffffff811d5615>] blkdev_close+0x25/0x30
 [<ffffffff8119a37a>] __fput+0xea/0x1f0
 [<ffffffff8119a4ce>] ____fput+0xe/0x10
 [<ffffffff81074ddc>] task_work_run+0xac/0xe0
 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60
 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940
 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60
 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0
 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940
 [<ffffffff81002528>] do_signal+0x48/0x630
 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40
 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0
 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20
 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0
 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0
 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0
 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0
 [<ffffffff814c1f98>] int_signal+0x12/0x17
blkid           D ffff8807c6b10000     0  8482      1 0x00000006
 ffff8807d4dbf9b8 0000000000000002 ffff8807c6b10000 ffff8807d4dbffd8
 00000000000131c0 00000000000131c0 ffff88082df38000 ffff88085fd73ad0
 ffff88085ff74638 ffff8807d4dbfa40 0000000000000002 ffffffff81135db0
Call Trace:
 [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60
 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130
 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20
 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0
 [<ffffffff81135eca>] __lock_page+0x6a/0x70
 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40
 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690
 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20
 [<ffffffff811d3425>] kill_bdev+0x35/0x40
 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0
 [<ffffffff811d5450>] blkdev_put+0x50/0x160
 [<ffffffff811d5615>] blkdev_close+0x25/0x30
 [<ffffffff8119a37a>] __fput+0xea/0x1f0
 [<ffffffff8119a4ce>] ____fput+0xe/0x10
 [<ffffffff81074ddc>] task_work_run+0xac/0xe0
 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60
 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940
 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60
 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0
 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940
 [<ffffffff81002528>] do_signal+0x48/0x630
 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40
 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0
 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20
 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0
 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0
 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0
 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0
 [<ffffffff814c1f98>] int_signal+0x12/0x17
blkid           D ffff8807c6b88000     0  8549      1 0x00000006
 ffff8807c6bb39b8 0000000000000002 ffff8807c6b88000 ffff8807c6bb3fd8
 00000000000131c0 00000000000131c0 ffff8807d168c460 ffff88085fc33ad0
 ffff88085ff88a38 ffff8807c6bb3a40 0000000000000002 ffffffff81135db0
Call Trace:
 [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60
 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130
 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20
 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0
 [<ffffffff81135eca>] __lock_page+0x6a/0x70
 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40
 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690
 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20
 [<ffffffff811d3425>] kill_bdev+0x35/0x40
 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0
 [<ffffffff811d5450>] blkdev_put+0x50/0x160
 [<ffffffff811d5615>] blkdev_close+0x25/0x30
 [<ffffffff8119a37a>] __fput+0xea/0x1f0
 [<ffffffff8119a4ce>] ____fput+0xe/0x10
 [<ffffffff81074ddc>] task_work_run+0xac/0xe0
 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60
 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940
 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60
 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0
 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940
 [<ffffffff81002528>] do_signal+0x48/0x630
 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40
 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0
 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20
 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0
 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0
 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0
 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0
 [<ffffffff814c1f98>] int_signal+0x12/0x17
blkid           D ffff8807c7268000     0  8615      1 0x00000006
 ffff8807d43b39b8 0000000000000002 ffff8807c7268000 ffff8807d43b3fd8
 00000000000131c0 00000000000131c0 ffff88083b4a8000 ffff88085fcb3ad0
 ffff88085ffb0038 ffff8807d43b3a40 0000000000000002 ffffffff81135db0
Call Trace:
 [<ffffffff81135db0>] ? wait_on_page_read+0x60/0x60
 [<ffffffff814bc0ed>] io_schedule+0x9d/0x130
 [<ffffffff81135dbe>] sleep_on_page+0xe/0x20
 [<ffffffff814bc738>] __wait_on_bit_lock+0x48/0xb0
 [<ffffffff81135eca>] __lock_page+0x6a/0x70
 [<ffffffff8109c8e0>] ? autoremove_wake_function+0x40/0x40
 [<ffffffff811474cf>] truncate_inode_pages_range+0x3ff/0x690
 [<ffffffff81147775>] truncate_inode_pages+0x15/0x20
 [<ffffffff811d3425>] kill_bdev+0x35/0x40
 [<ffffffff811d49a9>] __blkdev_put+0x69/0x1b0
 [<ffffffff811d5450>] blkdev_put+0x50/0x160
 [<ffffffff811d5615>] blkdev_close+0x25/0x30
 [<ffffffff8119a37a>] __fput+0xea/0x1f0
 [<ffffffff8119a4ce>] ____fput+0xe/0x10
 [<ffffffff81074ddc>] task_work_run+0xac/0xe0
 [<ffffffff8104ff67>] do_exit+0x2c7/0xc60
 [<ffffffff81064091>] ? get_signal_to_deliver+0xd1/0x940
 [<ffffffff814c118c>] ? _raw_spin_unlock_irq+0x2c/0x60
 [<ffffffff81051cac>] do_group_exit+0x4c/0xc0
 [<ffffffff810642a1>] get_signal_to_deliver+0x2e1/0x940
 [<ffffffff81002528>] do_signal+0x48/0x630
 [<ffffffff811d3f77>] ? blkdev_read_iter+0x37/0x40
 [<ffffffff8119809e>] ? new_sync_read+0x7e/0xb0
 [<ffffffff8127f0d7>] ? debug_smp_processor_id+0x17/0x20
 [<ffffffff810ba5d6>] ? rcu_eqs_exit_common.isra.43+0x26/0xe0
 [<ffffffff8113539a>] ? context_tracking_user_exit+0x5a/0x1a0
 [<ffffffff810a693d>] ? trace_hardirqs_on_caller+0xfd/0x1c0
 [<ffffffff81002b81>] do_notify_resume+0x71/0xc0
 [<ffffffff814c1f98>] int_signal+0x12/0x17

[-- Attachment #3: Type: text/plain, Size: 0 bytes --]

Re: v3.15 dm-mpath regression: cable pull test causes I/O hang

[Junichi Nomura]

On 07/07/14 22:40, Bart Van Assche wrote:
> Thanks for looking into this issue. I have attached the requested
> information to this e-mail for a test run with kernel 3.16-rc4 at the
> initiator side.

Thank you, Bart. The information is helpful.

From "dmsetup table" output, hardware handler is not used
in your setup. So pg_init is not involved.

> # cat /proc/diskstats
..
>  253       1 dm-1 127 0 1016 1070 0 0 0 0 1 278360 278820

In-flight IO remains.

> # dmsetup info -c
> Name              Maj Min Stat Open Targ Event  UUID                            
...
> 26466363537333266 253   1 L--w    1    1      0 mpath-26466363537333266 

Typical reason of remainig IO is device staying as suspended.
But the device state is ok here.

> # dmsetup status
...
> 26466363537333266: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:48 A 0 0 1 8:160 A 0 0 1

Single path group with both paths being active on dm-1.
But the path group is not active.

I suspect what's happening here is nobody clears m->queue_io:
multipath_busy() returns busy when queue_io=1 while clearing queue_io
needs __choose_pgpath(), which won't be called if multipath_busy() is true.

I think if you run 'sg_inq /dev/dm-1' for example in this case,
the device will start working again.
Since ioctl is not affected by multipath_busy(), somehow the problem
was hidden in many cases by udev activities, for example.

Attached patch should fix the problem.
Could you give it a try?

-
Jun'ichi Nomura, NEC Corporation


pg_ready() checks the current state of the multipath and may return
false even if a new IO is needed to change the state.

OTOH, if multipath_busy() returns busy, a new IO will not be sent
to multipath target and the state change won't happen. That results
in lock up.

The intent of multipath_busy() is to avoid unnecessary cycles of
dequeue + request_fn + requeue if it is known that multipath device
will requeue.

Such situation would be:
  - path group is being activated
  - there is no path and the multipath is setup to requeue if no path

This patch should fix the problem introduced as a part of this commit:
  commit e809917735ebf1b9a56c24e877ce0d320baee2ec
  dm mpath: push back requests instead of queueing

diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
index ebfa411..d58343e 100644
--- a/drivers/md/dm-mpath.c
+++ b/drivers/md/dm-mpath.c
@@ -1620,8 +1620,9 @@ static int multipath_busy(struct dm_target *ti)
 
 	spin_lock_irqsave(&m->lock, flags);
 
-	/* pg_init in progress, requeue until done */
-	if (!pg_ready(m)) {
+	/* pg_init in progress or no paths available */
+	if (m->pg_init_in_progress ||
+	    (!m->nr_valid_paths && m->queue_if_no_path)) {
 		busy = 1;
 		goto out;
 	}

Re: v3.15 dm-mpath regression: cable pull test causes I/O hang

[Bart Van Assche]

On 07/08/14 02:55, Junichi Nomura wrote:
> pg_ready() checks the current state of the multipath and may return
> false even if a new IO is needed to change the state.
> 
> OTOH, if multipath_busy() returns busy, a new IO will not be sent
> to multipath target and the state change won't happen. That results
> in lock up.
> 
> The intent of multipath_busy() is to avoid unnecessary cycles of
> dequeue + request_fn + requeue if it is known that multipath device
> will requeue.
> 
> Such situation would be:
>   - path group is being activated
>   - there is no path and the multipath is setup to requeue if no path
> 
> This patch should fix the problem introduced as a part of this commit:
>   commit e809917735ebf1b9a56c24e877ce0d320baee2ec
>   dm mpath: push back requests instead of queueing
> 
> diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
> index ebfa411..d58343e 100644
> --- a/drivers/md/dm-mpath.c
> +++ b/drivers/md/dm-mpath.c
> @@ -1620,8 +1620,9 @@ static int multipath_busy(struct dm_target *ti)
>  
>  	spin_lock_irqsave(&m->lock, flags);
>  
> -	/* pg_init in progress, requeue until done */
> -	if (!pg_ready(m)) {
> +	/* pg_init in progress or no paths available */
> +	if (m->pg_init_in_progress ||
> +	    (!m->nr_valid_paths && m->queue_if_no_path)) {
>  		busy = 1;
>  		goto out;
>  	}
> 

This patch seems to fix the issue reported at the start of this thread -
with this patch applied my test passes.

Thanks !

Bart.

Re: v3.15 dm-mpath regression: cable pull test causes I/O hang

[Mike Snitzer]

On Mon, Jul 07 2014 at  8:55pm -0400,
Junichi Nomura <j-nomura@ce.jp.nec.com> wrote:

> On 07/07/14 22:40, Bart Van Assche wrote:
> > Thanks for looking into this issue. I have attached the requested
> > information to this e-mail for a test run with kernel 3.16-rc4 at the
> > initiator side.
> 
> Thank you, Bart. The information is helpful.
> 
> >From "dmsetup table" output, hardware handler is not used
> in your setup. So pg_init is not involved.
> 
> > # cat /proc/diskstats
> ..
> >  253       1 dm-1 127 0 1016 1070 0 0 0 0 1 278360 278820
> 
> In-flight IO remains.
> 
> > # dmsetup info -c
> > Name              Maj Min Stat Open Targ Event  UUID                            
> ...
> > 26466363537333266 253   1 L--w    1    1      0 mpath-26466363537333266 
> 
> Typical reason of remainig IO is device staying as suspended.
> But the device state is ok here.
> 
> > # dmsetup status
> ...
> > 26466363537333266: 0 256000 multipath 2 1 0 0 1 1 E 0 2 2 8:48 A 0 0 1 8:160 A 0 0 1
> 
> Single path group with both paths being active on dm-1.
> But the path group is not active.
> 
> I suspect what's happening here is nobody clears m->queue_io:
> multipath_busy() returns busy when queue_io=1 while clearing queue_io
> needs __choose_pgpath(), which won't be called if multipath_busy() is true.
> 
> I think if you run 'sg_inq /dev/dm-1' for example in this case,
> the device will start working again.
> Since ioctl is not affected by multipath_busy(), somehow the problem
> was hidden in many cases by udev activities, for example.
> 
> Attached patch should fix the problem.
> Could you give it a try?
> 
> -
> Jun'ichi Nomura, NEC Corporation
> 
> 
> pg_ready() checks the current state of the multipath and may return
> false even if a new IO is needed to change the state.
> 
> OTOH, if multipath_busy() returns busy, a new IO will not be sent
> to multipath target and the state change won't happen. That results
> in lock up.
> 
> The intent of multipath_busy() is to avoid unnecessary cycles of
> dequeue + request_fn + requeue if it is known that multipath device
> will requeue.
> 
> Such situation would be:
>   - path group is being activated
>   - there is no path and the multipath is setup to requeue if no path
> 
> This patch should fix the problem introduced as a part of this commit:
>   commit e809917735ebf1b9a56c24e877ce0d320baee2ec
>   dm mpath: push back requests instead of queueing


Thanks Jun'ichi!  I knew multipath_busy() wasn't quite right ;)

I've merged your fix with a revised header, if you see anything wrong
with the header please feel free to let me know!  See:
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=75c76c45b76e53b7c2f025d30e7e308bfe331004

This will likely go to Linus for 3.16-rc5 by this Friday.

BTW, I also staged a related cleanup for 3.17, see:
https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=2122e57c913cd842e5061137a7093bd06e728677

Thanks again,
Mike

Re: v3.15 dm-mpath regression: cable pull test causes I/O hang

[Junichi Nomura]

On 07/09/14 01:33, Mike Snitzer wrote:
> I've merged your fix with a revised header, if you see anything wrong
> with the header please feel free to let me know!  See:
> https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=75c76c45b76e53b7c2f025d30e7e308bfe331004
> 
> This will likely go to Linus for 3.16-rc5 by this Friday.

Thank you, Mike. I have no problem with the header.

> BTW, I also staged a related cleanup for 3.17, see:
> https://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=2122e57c913cd842e5061137a7093bd06e728677

Yeah, I'm ok with that.

-- 
Jun'ichi Nomura, NEC Corporation