All of lore.kernel.org
 help / color / mirror / Atom feed
From: Casey Chen <cachen@purestorage.com>
To: linux-nvme@lists.infradead.org
Cc: leon@kernel.org, sagi@grimberg.me, kbusch@kernel.org, hch@lst.de,
	axboe@kernel.dk, linux-rdma@vger.kernel.org,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH v3] nvme-rdma: fix ib_device removal race that hangs PCI unbind
Date: Tue,  1 Sep 2026 16:38:09 -0600	[thread overview]
Message-ID: <20260901223809.2346699-1-cachen@purestorage.com> (raw)
In-Reply-To: <20260831221355.1715088-1-cachen@purestorage.com>

On 31/08/2026 16:13, Casey Chen wrote:
> The ->add path is new in this version and has not run on the setup that
> reproduced the hang, so this wants a fresh soak before it is applied.

That soak is done now, so the caveat above no longer applies.

The test bounces both mlx5 interfaces carrying the NVMe-oF RDMA
connections, one at a time, with IO running throughout and a userspace
daemon reconnecting the controllers:

  ethtool -i <iface> | grep bus-info
  echo 1 | sudo tee /sys/bus/pci/devices/<bdf>/remove
  sleep 30
  echo 1 | sudo tee /sys/bus/pci/rescan
  sleep 30

30 iterations over the two interfaces, so 60 remove/rescan cycles, with
60s of settle time between iterations - roughly 90 minutes.

No hang with v3 applied. Every write to remove returned, the interface
came back on the following rescan and the controllers reconnected. The
->add path is covered by this, since each rescan re-probes the HCA as a
new ib_device and so goes through nvme_rdma_add_one().

For contrast, on the same setup without the fix the write to remove
eventually never returns and the task is left in D state with the trace
in the commit message. It is intermittent - most removals complete
normally, and only one that lands while a connect is in flight strands a
controller.

      reply	other threads:[~2026-09-01 22:38 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-08-06 21:18 [PATCH] nvme-rdma: fix ib_device removal race that hangs PCI unbind Casey Chen
2026-08-13 10:07 ` Leon Romanovsky
2026-08-23  0:37 ` Sagi Grimberg
2026-08-28 23:24 ` [PATCH v2] " Casey Chen
2026-08-30 21:10   ` Sagi Grimberg
2026-08-30 21:15     ` Sagi Grimberg
2026-08-31 22:07       ` Casey Chen
2026-08-31 22:13   ` [PATCH v3] " Casey Chen
2026-09-01 22:38     ` Casey Chen [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20260901223809.2346699-1-cachen@purestorage.com \
    --to=cachen@purestorage.com \
    --cc=axboe@kernel.dk \
    --cc=hch@lst.de \
    --cc=kbusch@kernel.org \
    --cc=leon@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-nvme@lists.infradead.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=sagi@grimberg.me \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.