threads scsi_eh_ and scsi_tmf_ don't die when ib_srp reconnect

threads scsi_eh_ and scsi_tmf_ don't die when ib_srp reconnect

[Konstantin Krotov]

Hello list!


I was faced with the following problem:

Im exported to compute-node scsi disks through ib_srp from storages. And 
if reconnect occurs for any reason, eg

"[1201117.626498] scsi host62: ib_srp: REJ received
[1201117.626507] scsi host62: ib_srp: SRP LOGIN from 
fe80:0000:0000:0000:0002:c903:0009:1796 to 
fe80:0000:0000:0000:0002:c903:0009:c1a0 REJECTED, reason 0x00010001
[1201117.626656] scsi host62: ib_srp: Connection 0/8 failed"

i find threads in output ps:
"root     28047  0.0  0.0      0     0 ?        S    18:04   0:00 
[scsi_eh_62]
root     28048  0.0  0.0      0     0 ?        S<   18:04   0:00 
[scsi_tmf_62]"

Each reconnect generates two thread, which do not die.

kernel version 3.19.1

modinfo ib_srp
filename:       /lib/modules/3.19-3-amd64/updates/dkms/ib_srp.ko
release_date:   Apr 09 2015
version:        2.0.32#in-tree

-- 
WBR
Konstantin V. Krotov

Re: threads scsi_eh_ and scsi_tmf_ don't die when ib_srp reconnect

[Bart Van Assche]

On 08/10/2015 08:18 AM, Konstantin Krotov wrote:
> I was faced with the following problem:
>
> Im exported to compute-node scsi disks through ib_srp from storages. And
> if reconnect occurs for any reason, eg
>
> "[1201117.626498] scsi host62: ib_srp: REJ received
> [1201117.626507] scsi host62: ib_srp: SRP LOGIN from
> fe80:0000:0000:0000:0002:c903:0009:1796 to
> fe80:0000:0000:0000:0002:c903:0009:c1a0 REJECTED, reason 0x00010001
> [1201117.626656] scsi host62: ib_srp: Connection 0/8 failed"
>
> i find threads in output ps:
> "root     28047  0.0  0.0      0     0 ?        S    18:04   0:00
> [scsi_eh_62]
> root     28048  0.0  0.0      0     0 ?        S<   18:04   0:00
> [scsi_tmf_62]"
>
> Each reconnect generates two thread, which do not die.
>
> kernel version 3.19.1
>
> modinfo ib_srp
> filename:       /lib/modules/3.19-3-amd64/updates/dkms/ib_srp.ko
> release_date:   Apr 09 2015
> version:        2.0.32#in-tree

Hello Konstantin,

Does the above output mean that you are using the ib_srp-backport driver 
? If so, please upgrade to version v2.0.34. That version includes all 
SRP initiator fixes that have been posted on the linux-rdma and 
linux-scsi mailing lists for inclusion in Linux kernel v4.3.

Bart.

Re: threads scsi_eh_ and scsi_tmf_ don't die when ib_srp reconnect

[Konstantin Krotov]

10.08.2015 18:30, Bart Van Assche пишет:
> On 08/10/2015 08:18 AM, Konstantin Krotov wrote:
>> I was faced with the following problem:
>>
>> Im exported to compute-node scsi disks through ib_srp from storages. And
>> if reconnect occurs for any reason, eg
>>
>> "[1201117.626498] scsi host62: ib_srp: REJ received
>> [1201117.626507] scsi host62: ib_srp: SRP LOGIN from
>> fe80:0000:0000:0000:0002:c903:0009:1796 to
>> fe80:0000:0000:0000:0002:c903:0009:c1a0 REJECTED, reason 0x00010001
>> [1201117.626656] scsi host62: ib_srp: Connection 0/8 failed"
>>
>> i find threads in output ps:
>> "root     28047  0.0  0.0      0     0 ?        S    18:04   0:00
>> [scsi_eh_62]
>> root     28048  0.0  0.0      0     0 ?        S<   18:04   0:00
>> [scsi_tmf_62]"
>>
>> Each reconnect generates two thread, which do not die.
>>
>> kernel version 3.19.1
>>
>> modinfo ib_srp
>> filename:       /lib/modules/3.19-3-amd64/updates/dkms/ib_srp.ko
>> release_date:   Apr 09 2015
>> version:        2.0.32#in-tree
>
> Hello Konstantin,
>
> Does the above output mean that you are using the ib_srp-backport driver
> ? If so, please upgrade to version v2.0.34. That version includes all
> SRP initiator fixes that have been posted on the linux-rdma and
> linux-scsi mailing lists for inclusion in Linux kernel v4.3.
>
> Bart.
>
Hello, thanks for reply.

I build a package version 2.0.34, but problem is reproduced:
[  260.266374] scsi host15: ib_srp: REJ received
[  260.266383] scsi host15: ib_srp: SRP LOGIN from 
fe80:0000:0000:0000:0002:c903:0008:c884 to 
fe80:0000:0000:0000:0002:c903:0009:c1a0 REJECTED, reason 0x00010001
[  260.266588] scsi host15: ib_srp: Connection 0/8 failed
[  260.266700] scsi host15: ib_srp: Sending CM DREQ failed

and increases the number of threads scsi_tmf_ and scsi_eh_ at every 
reconnect..

modinfo ib_srp
filename:       /lib/modules/3.19-3-amd64/updates/dkms/ib_srp.ko
release_date:   Jul 27 2015
version:        2.0.34#in-tree
license:        Dual BSD/GPL
description:    InfiniBand SCSI RDMA Protocol initiator
author:         Roland Dreier
srcversion:     75206A3E8C5F05BED61C20B
depends: 
ib_core,scsi_transport_srp,ib_sa,scsi_mod,ib_cm,rdma_cm,scsi_transport_srp
vermagic:       3.19-3-amd64 SMP mod_unload modversions

modinfo scsi_transport_srp
filename:       /lib/modules/3.19-3-amd64/updates/dkms/scsi_transport_srp.ko
license:        GPL
description:    SRP Transport Attributes
author:         FUJITA Tomonori
depends:        scsi_mod
vermagic:       3.19-3-amd64 SMP mod_unload modversions

-- 
WBR
Konstantin V. Krotov


--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: threads scsi_eh_ and scsi_tmf_ don't die when ib_srp reconnect

[Bart Van Assche]

On 08/11/2015 06:41 AM, Konstantin Krotov wrote:
> Hello, thanks for reply.
>
> I build a package version 2.0.34, but problem is reproduced:
> [  260.266374] scsi host15: ib_srp: REJ received
> [  260.266383] scsi host15: ib_srp: SRP LOGIN from
> fe80:0000:0000:0000:0002:c903:0008:c884 to
> fe80:0000:0000:0000:0002:c903:0009:c1a0 REJECTED, reason 0x00010001
> [  260.266588] scsi host15: ib_srp: Connection 0/8 failed
> [  260.266700] scsi host15: ib_srp: Sending CM DREQ failed
>
> and increases the number of threads scsi_tmf_ and scsi_eh_ at every
> reconnect..

Hello Konstantin,

I have tried to reproduce this but without success so far. Whether I 
install upstream kernel v4.2-rc6 or another kernel version + 
ib_srp-backport on the initiator system, the result is the same when I 
block SRP login on the target system: the ib_srp driver keeps trying to 
log in without creating new SCSI hosts. Are you using a kernel from Greg 
KH's stable repository or another kernel ? And to which values were the 
ib_srp kernel module parameters set (cat /etc/modprobe.d/ib_srp.conf) ?

Bart.

Re: threads scsi_eh_ and scsi_tmf_ don't die when ib_srp reconnect

[Konstantin Krotov]

11.08.2015 23:28, Bart Van Assche пишет:
> On 08/11/2015 06:41 AM, Konstantin Krotov wrote:
>> Hello, thanks for reply.
>>
>> I build a package version 2.0.34, but problem is reproduced:
>> [  260.266374] scsi host15: ib_srp: REJ received
>> [  260.266383] scsi host15: ib_srp: SRP LOGIN from
>> fe80:0000:0000:0000:0002:c903:0008:c884 to
>> fe80:0000:0000:0000:0002:c903:0009:c1a0 REJECTED, reason 0x00010001
>> [  260.266588] scsi host15: ib_srp: Connection 0/8 failed
>> [  260.266700] scsi host15: ib_srp: Sending CM DREQ failed
>>
>> and increases the number of threads scsi_tmf_ and scsi_eh_ at every
>> reconnect..
>
> Hello Konstantin,
>
> I have tried to reproduce this but without success so far. Whether I
> install upstream kernel v4.2-rc6 or another kernel version +
> ib_srp-backport on the initiator system, the result is the same when I
> block SRP login on the target system: the ib_srp driver keeps trying to
> log in without creating new SCSI hosts.

> Are you using a kernel from Greg KH's stable repository or another kernel ? And to which values were the
yes, sources from upstream, 3.19.1

> ib_srp kernel module parameters set (cat /etc/modprobe.d/ib_srp.conf) ?
srp.konf missing, i load modules with the default settings:
allow_ext_sg:N
ch_count:0
cmd_sg_entries:12
dev_loss_tmo:600
enable_imm_data:Y
fast_io_fail_tmo:15
indirect_sg_entries:12
prefer_fr:N
reconnect_delay:20
register_always:N
srp_sg_tablesize:0
topspin_workarounds:1

>
> Bart.

-- 
WBR
Konstantin V. Krotov

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: threads scsi_eh_ and scsi_tmf_ don't die when ib_srp reconnect

[Bart Van Assche]

On 08/11/2015 10:15 PM, Konstantin Krotov wrote:
> 11.08.2015 23:28, Bart Van Assche пишет:
>> Are you using a kernel from Greg KH's stable repository or another kernel ? And to which values were the
> yes, sources from upstream, 3.19.1

Hello Konstantin,

Can you test the patch below ? That patch not only applies to the v4.2-rc6
kernel but also to the ib_srp-backport driver.

Thanks,

Bart.

Subject: [PATCH] IB/srp: Stop the scsi_eh_<n> and scsi_tmf_<n> threads if login fails

scsi_host_alloc() not only allocates memory for a SCSI host but also
creates the scsi_eh_<n> kernel thread and the scsi_tmf_<n> workqueue.
Stop these threads if login fails by calling scsi_host_put().

Reported-by: Konstantin Krotov <kkv@clodo.ru>
Signed-off-by: Bart Van Assche <bart.vanassche@sandisk.com>
Cc: stable <stable@vger.kernel.org>
---
 drivers/infiniband/ulp/srp/ib_srp.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index f962267..b5f9456 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -2765,6 +2765,13 @@ static int srp_sdev_count(struct Scsi_Host *host)
 	return c;
 }
 
+/*
+ * Return values:
+ * < 0 upon failure. Caller is responsible for SRP target port cleanup.
+ * 0 and target->state == SRP_TARGET_REMOVED if the target port will be removed
+ *    asynchronously.
+ * 0 and target->state != SRP_TARGET_REMOVED upon success.
+ */
 static int srp_add_target(struct srp_host *host, struct srp_target_port *target)
 {
 	struct srp_rport_identifiers ids;
@@ -3316,6 +3323,7 @@ err_disconnect:
 	}
 
 	kfree(target->ch);
+	scsi_host_put(target->scsi_host);
 	goto out;
 }
 
-- 
2.1.4


--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: threads scsi_eh_ and scsi_tmf_ don't die when ib_srp reconnect

[Konstantin Krotov]

12.08.2015 19:50, Bart Van Assche пишет:
> On 08/11/2015 10:15 PM, Konstantin Krotov wrote:
>> 11.08.2015 23:28, Bart Van Assche пишет:
>>> Are you using a kernel from Greg KH's stable repository or another kernel ? And to which values were the
>> yes, sources from upstream, 3.19.1
>
> Hello Konstantin,
>
> Can you test the patch below ? That patch not only applies to the v4.2-rc6
> kernel but also to the ib_srp-backport driver.
>
> Thanks,
>
> Bart.
>
> Subject: [PATCH] IB/srp: Stop the scsi_eh_<n> and scsi_tmf_<n> threads if login fails
>
> scsi_host_alloc() not only allocates memory for a SCSI host but also
> creates the scsi_eh_<n> kernel thread and the scsi_tmf_<n> workqueue.
> Stop these threads if login fails by calling scsi_host_put().
>
> Reported-by: Konstantin Krotov <kkv@clodo.ru>
> Signed-off-by: Bart Van Assche <bart.vanassche@sandisk.com>
> Cc: stable <stable@vger.kernel.org>
> ---
>   drivers/infiniband/ulp/srp/ib_srp.c | 8 ++++++++
>   1 file changed, 8 insertions(+)
>
> diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
> index f962267..b5f9456 100644
> --- a/drivers/infiniband/ulp/srp/ib_srp.c
> +++ b/drivers/infiniband/ulp/srp/ib_srp.c
> @@ -2765,6 +2765,13 @@ static int srp_sdev_count(struct Scsi_Host *host)
>   	return c;
>   }
>
> +/*
> + * Return values:
> + * < 0 upon failure. Caller is responsible for SRP target port cleanup.
> + * 0 and target->state == SRP_TARGET_REMOVED if the target port will be removed
> + *    asynchronously.
> + * 0 and target->state != SRP_TARGET_REMOVED upon success.
> + */
>   static int srp_add_target(struct srp_host *host, struct srp_target_port *target)
>   {
>   	struct srp_rport_identifiers ids;
> @@ -3316,6 +3323,7 @@ err_disconnect:
>   	}
>
>   	kfree(target->ch);
> +	scsi_host_put(target->scsi_host);
>   	goto out;
>   }
>
>
Hello Bart

I tested the patch, the problem is not reproduced.

Thanks.

-- 
WBR
Konstantin V. Krotov

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: threads scsi_eh_ and scsi_tmf_ don't die when ib_srp reconnect

[Bart Van Assche]

On 08/14/2015 02:06 AM, Konstantin Krotov wrote:
> I tested the patch, the problem is not reproduced.

Thanks for confirming that the patch fixes this issue. I will send a 
slightly modified version of this patch soon to the Linux RDMA maintainer.

Bart.