From: Casey Chen <cachen@purestorage.com>
To: linux-nvme@lists.infradead.org
Cc: leon@kernel.org, sagi@grimberg.me, kbusch@kernel.org, hch@lst.de,
axboe@kernel.dk, linux-rdma@vger.kernel.org,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH v3] nvme-rdma: fix ib_device removal race that hangs PCI unbind
Date: Tue, 1 Sep 2026 16:38:09 -0600 [thread overview]
Message-ID: <20260901223809.2346699-1-cachen@purestorage.com> (raw)
In-Reply-To: <20260831221355.1715088-1-cachen@purestorage.com>
On 31/08/2026 16:13, Casey Chen wrote:
> The ->add path is new in this version and has not run on the setup that
> reproduced the hang, so this wants a fresh soak before it is applied.
That soak is done now, so the caveat above no longer applies.
The test bounces both mlx5 interfaces carrying the NVMe-oF RDMA
connections, one at a time, with IO running throughout and a userspace
daemon reconnecting the controllers:
ethtool -i <iface> | grep bus-info
echo 1 | sudo tee /sys/bus/pci/devices/<bdf>/remove
sleep 30
echo 1 | sudo tee /sys/bus/pci/rescan
sleep 30
30 iterations over the two interfaces, so 60 remove/rescan cycles, with
60s of settle time between iterations - roughly 90 minutes.
No hang with v3 applied. Every write to remove returned, the interface
came back on the following rescan and the controllers reconnected. The
->add path is covered by this, since each rescan re-probes the HCA as a
new ib_device and so goes through nvme_rdma_add_one().
For contrast, on the same setup without the fix the write to remove
eventually never returns and the task is left in D state with the trace
in the commit message. It is intermittent - most removals complete
normally, and only one that lands while a connect is in flight strands a
controller.
prev parent reply other threads:[~2026-09-01 22:38 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-08-06 21:18 [PATCH] nvme-rdma: fix ib_device removal race that hangs PCI unbind Casey Chen
2026-08-13 10:07 ` Leon Romanovsky
2026-08-23 0:37 ` Sagi Grimberg
2026-08-28 23:24 ` [PATCH v2] " Casey Chen
2026-08-30 21:10 ` Sagi Grimberg
2026-08-30 21:15 ` Sagi Grimberg
2026-08-31 22:07 ` Casey Chen
2026-08-31 22:13 ` [PATCH v3] " Casey Chen
2026-09-01 22:38 ` Casey Chen [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260901223809.2346699-1-cachen@purestorage.com \
--to=cachen@purestorage.com \
--cc=axboe@kernel.dk \
--cc=hch@lst.de \
--cc=kbusch@kernel.org \
--cc=leon@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-rdma@vger.kernel.org \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.