From: Max Gurtovoy <mgurtovoy@nvidia.com>
To: Sagi Grimberg <sagi@grimberg.me>, Yi Zhang <yi.zhang@redhat.com>
Cc: Haakon Bugge <haakon.bugge@oracle.com>,
Max Gurtovoy <maxg@mellanox.com>,
"open list:NVM EXPRESS DRIVER" <linux-nvme@lists.infradead.org>,
"OFED mailing list" <linux-rdma@vger.kernel.org>
Subject: Re: [bug report] NVMe/IB: reset_controller need more than 1min
Date: Mon, 14 Feb 2022 14:11:56 +0200 [thread overview]
Message-ID: <bdb675e4-1de8-400b-9d43-9f0c18147dc4@nvidia.com> (raw)
In-Reply-To: <a2fabeb3-5f19-11b6-8c00-a479e3759182@grimberg.me>
On 2/14/2022 1:32 PM, Sagi Grimberg wrote:
>
>> Hi Sagi/Max
>> Here are more findings with the bisect:
>>
>> The time for reset operation changed from 3s[1] to 12s[2] after
>> commit[3], and after commit[4], the reset operation timeout at the
>> second reset[5], let me know if you need any testing for it, thanks.
>
> Does this at least eliminate the timeout?
> --
> diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
> index a162f6c6da6e..60e415078893 100644
> --- a/drivers/nvme/host/nvme.h
> +++ b/drivers/nvme/host/nvme.h
> @@ -25,7 +25,7 @@ extern unsigned int nvme_io_timeout;
> extern unsigned int admin_timeout;
> #define NVME_ADMIN_TIMEOUT (admin_timeout * HZ)
>
> -#define NVME_DEFAULT_KATO 5
> +#define NVME_DEFAULT_KATO 10
>
> #ifdef CONFIG_ARCH_NO_SG_CHAIN
> #define NVME_INLINE_SG_CNT 0
> --
>
or for the initial test you can use --keep-alive-tmo=<10 or 15> flag in
the connect command
>>
>> [1]
>> # time nvme reset /dev/nvme0
>>
>> real 0m3.049s
>> user 0m0.000s
>> sys 0m0.006s
>> [2]
>> # time nvme reset /dev/nvme0
>>
>> real 0m12.498s
>> user 0m0.000s
>> sys 0m0.006s
>> [3]
>> commit 5ec5d3bddc6b912b7de9e3eb6c1f2397faeca2bc (HEAD)
>> Author: Max Gurtovoy <maxg@mellanox.com>
>> Date: Tue May 19 17:05:56 2020 +0300
>>
>> nvme-rdma: add metadata/T10-PI support
>>
>> [4]
>> commit a70b81bd4d9d2d6c05cfe6ef2a10bccc2e04357a (HEAD)
>> Author: Hannes Reinecke <hare@suse.de>
>> Date: Fri Apr 16 13:46:20 2021 +0200
>>
>> nvme: sanitize KATO setting-
>
> This change effectively changed the keep-alive timeout
> from 15 to 5 and modified the host to send keepalives every
> 2.5 seconds instead of 5.
>
> I guess that in combination that now it takes longer to
> create and delete rdma resources (either qps or mrs)
> it starts to timeout in setups where there are a lot of
> queues.
next prev parent reply other threads:[~2022-02-14 12:50 UTC|newest]
Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-05-21 16:12 [bug report] NVMe/IB: reset_controller need more than 1min Yi Zhang
2021-05-21 18:00 ` Sagi Grimberg
2021-05-22 4:27 ` Yi Zhang
2021-06-23 10:01 ` Yi Zhang
2021-06-23 21:32 ` Sagi Grimberg
2021-06-24 16:14 ` Yi Zhang
2021-12-11 3:01 ` Yi Zhang
2021-12-12 9:45 ` Sagi Grimberg
2021-12-13 6:12 ` Yi Zhang
2021-12-13 9:04 ` Sagi Grimberg
2021-12-13 17:05 ` Yi Zhang
2021-12-14 10:39 ` Sagi Grimberg
2021-12-14 12:00 ` Max Gurtovoy
2021-12-15 1:15 ` Yi Zhang
2021-12-15 12:10 ` Max Gurtovoy
2021-12-16 2:18 ` Yi Zhang
2021-12-16 13:21 ` Max Gurtovoy
2021-12-16 16:32 ` Yi Zhang
2021-12-16 17:33 ` Haakon Bugge
2021-12-17 7:03 ` Yi Zhang
2021-12-17 11:19 ` Haakon Bugge
2022-02-14 9:47 ` Yi Zhang
2022-02-14 11:00 ` Chaitanya Kulkarni
2022-02-14 11:32 ` Sagi Grimberg
2022-02-14 12:11 ` Max Gurtovoy [this message]
2022-02-15 13:52 ` Yi Zhang
2022-02-15 14:30 ` Max Gurtovoy
2022-02-21 10:00 ` Yi Zhang
2022-02-23 10:04 ` Max Gurtovoy
2022-02-23 10:30 ` Sagi Grimberg
2022-02-23 11:20 ` Max Gurtovoy
2022-03-01 0:06 ` Yi Zhang
2022-03-16 15:16 ` Sagi Grimberg
2022-03-19 7:29 ` Yi Zhang
2022-03-20 10:50 ` Max Gurtovoy
2022-03-20 13:03 ` Sagi Grimberg
2022-03-20 15:11 ` Max Gurtovoy
2022-03-21 9:28 ` Sagi Grimberg
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=bdb675e4-1de8-400b-9d43-9f0c18147dc4@nvidia.com \
--to=mgurtovoy@nvidia.com \
--cc=haakon.bugge@oracle.com \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-rdma@vger.kernel.org \
--cc=maxg@mellanox.com \
--cc=sagi@grimberg.me \
--cc=yi.zhang@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox