linux-nvme.lists.infradead.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v3 0/2] nvme: fabrics and target fixes
@ 2026-11-16  8:12 Guixin Liu
  2026-11-16  8:12 ` [PATCH v3 1/2] nvme-fc: fix use-after-free in nvme_fc_match_disconn_ls() Guixin Liu
  2026-11-16  8:12 ` [PATCH v3 2/2] nvme-auth: fix data race on ctrl->transaction Guixin Liu
  0 siblings, 2 replies; 6+ messages in thread
From: Guixin Liu @ 2026-11-16  8:12 UTC (permalink / raw)
  To: Keith Busch, Jens Axboe, Christoph Hellwig, Sagi Grimberg,
	Nilay Shroff, Daniel Wagner, Hannes Reinecke
  Cc: linux-nvme

Two remaining fixes from the previous rounds of this series:

 - a use-after-free in nvme_fc_match_disconn_ls(): no iterator is
   sound across the rport lock drop inside the loop, so the loop is
   reworked to run entirely under the rport lock and to take a
   reference only for the match; reproduced under KASAN through fcloop
 - a data race on the shared authentication transaction counter,
   reworked per review to a per-queue counter, dropping the shared
   ctrl->transaction field entirely

The other three patches from v2 are already resolved: the GFP_KERNEL
persistent reservation change and the tbkas configfs locking are in
nvme-7.4 now, and the reconnect_delay fix was superseded by an earlier
identical patch posted by someone else.

Both patches are verified on an initiator/target VM pair: the fc loop
is compile tested and exercised under KASAN through fcloop with six
controllers on one remote port across tbkas force-reconnect cycles
and concurrent disconnects with no reports; the
DH-HMAC-CHAP authenticated session including I/O covers the auth path.

Changes since v2:

 - rework the nvme-auth fix to a per-queue counter (Keith Busch),
   drop the now unused shared field, and add the Fixes tag (sashiko);
   the Reviewed-by on v2 covered the atomic_t variant and is dropped
 - resend the unchanged nvme-fc patch, still awaiting review
 - drop the reconnect_delay patch (superseded), the GFP_KERNEL patch
   (applied) and the tbkas patch (applied)

Link: https://lore.kernel.org/all/20261109022646.290792-1-kanie@linux.alibaba.com/

Guixin Liu (2):
  nvme-fc: fix use-after-free in nvme_fc_match_disconn_ls()
  nvme-auth: fix data race on ctrl->transaction

 drivers/nvme/host/auth.c |  3 +--
 drivers/nvme/host/fc.c   | 22 ++++++++++++----------
 drivers/nvme/host/nvme.h |  1 -
 3 files changed, 13 insertions(+), 13 deletions(-)

-- 
2.43.7



^ permalink raw reply	[flat|nested] 6+ messages in thread

* [PATCH v3 1/2] nvme-fc: fix use-after-free in nvme_fc_match_disconn_ls()
  2026-11-16  8:12 [PATCH v3 0/2] nvme: fabrics and target fixes Guixin Liu
@ 2026-11-16  8:12 ` Guixin Liu
  2026-11-16  8:12 ` [PATCH v3 2/2] nvme-auth: fix data race on ctrl->transaction Guixin Liu
  1 sibling, 0 replies; 6+ messages in thread
From: Guixin Liu @ 2026-11-16  8:12 UTC (permalink / raw)
  To: Keith Busch, Jens Axboe, Christoph Hellwig, Sagi Grimberg,
	Nilay Shroff, Daniel Wagner, Hannes Reinecke
  Cc: linux-nvme

Commit b71cbcf7d170e ("nvme-fc: don't hold rport lock when putting ctrl")
dropped the rport lock inside the loop to put the reference of a
non-matching controller, because nvme_fc_ctrl_free() takes the rport
lock to unlink the controller. But no iterator is sound across that
drop: with list_for_each_entry_safe() the prefetched next entry can be
freed in the window, and with a plain iterator the loop's own
nvme_fc_ctrl_put() can be the final reference, freeing the current
entry before the iterator reads its ->next pointer.

The reference dance is unnecessary in the first place. An entry cannot
leave ctrl_list while rport->lock is held, as the final put unlinks it
under that very lock, so association_id can be inspected under
ctrl->lock without holding a reference. Take a reference only for the
matching controller, still under the rport lock, and hand it to the
caller as before. The loop then no longer needs to drop the lock at
all, which also removes the reason the lock was dropped for in the
first place.

Fixes: b71cbcf7d170e ("nvme-fc: don't hold rport lock when putting ctrl")
Cc: stable@vger.kernel.org
Signed-off-by: Guixin Liu <kanie@linux.alibaba.com>
---

Reproduced under KASAN in a VM through fcloop with several controllers
on one remote port and a tbkas force-reconnect racing concurrent
controller disconnects; the lock-drop window was widened with a
test-only mdelay(200) to make the microsecond window deterministic:

  BUG: KASAN: slab-use-after-free in nvme_fc_match_disconn_ls
  Read of size 8 by task kworker
  Workqueue: events nvme_fc_handle_ls_rqst_work
  Call Trace:
   nvme_fc_match_disconn_ls
   nvme_fc_ls_disconnect_assoc
   nvme_fc_handle_ls_rqst_work
   process_one_work

The freed object was the prefetched next entry, allocated by
nvme_fc_alloc_ctrl() on connect and freed by nvme_free_ctrl() on the
concurrent teardown. The loop increment dereferences it before
nvme_fc_ctrl_get() gets a chance to reject it, which is why the
get-unless-zero check in the loop body does not help. The reworked
loop does not drop the lock, so no such window is left to widen.
---
 drivers/nvme/host/fc.c | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)

diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 2cbd744ffd22..ce4598dd227f 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -1469,29 +1469,31 @@ nvme_fc_match_disconn_ls(struct nvme_fc_rport *rport,
 {
 	struct fcnvme_ls_disconnect_assoc_rqst *rqst =
 					&lsop->rqstbuf->rq_dis_assoc;
-	struct nvme_fc_ctrl *ctrl, *tmp, *ret = NULL;
+	struct nvme_fc_ctrl *ctrl, *ret = NULL;
 	struct nvmefc_ls_rcv_op *oldls = NULL;
 	u64 association_id = be64_to_cpu(rqst->associd.association_id);
 	unsigned long flags;
 
 	spin_lock_irqsave(&rport->lock, flags);
 
-	list_for_each_entry_safe(ctrl, tmp, &rport->ctrl_list, ctrl_list) {
-		if (!nvme_fc_ctrl_get(ctrl))
-			continue;
+	list_for_each_entry(ctrl, &rport->ctrl_list, ctrl_list) {
 		spin_lock(&ctrl->lock);
 		if (association_id == ctrl->association_id) {
-			oldls = ctrl->rcv_disconn;
-			ctrl->rcv_disconn = lsop;
-			ret = ctrl;
+			/*
+			 * Entries cannot leave the list while rport->lock is
+			 * held, so they can be inspected without a reference.
+			 * Only the match needs one, for the caller.
+			 */
+			if (nvme_fc_ctrl_get(ctrl)) {
+				oldls = ctrl->rcv_disconn;
+				ctrl->rcv_disconn = lsop;
+				ret = ctrl;
+			}
 		}
 		spin_unlock(&ctrl->lock);
 		if (ret)
 			/* leave the ctrl get reference */
 			break;
-		spin_unlock_irqrestore(&rport->lock, flags);
-		nvme_fc_ctrl_put(ctrl);
-		spin_lock_irqsave(&rport->lock, flags);
 	}
 
 	spin_unlock_irqrestore(&rport->lock, flags);
-- 
2.43.7



^ permalink raw reply related	[flat|nested] 6+ messages in thread

* [PATCH v3 2/2] nvme-auth: fix data race on ctrl->transaction
  2026-11-16  8:12 [PATCH v3 0/2] nvme: fabrics and target fixes Guixin Liu
  2026-11-16  8:12 ` [PATCH v3 1/2] nvme-fc: fix use-after-free in nvme_fc_match_disconn_ls() Guixin Liu
@ 2026-11-16  8:12 ` Guixin Liu
  2026-11-16 13:22   ` Hannes Reinecke
  1 sibling, 1 reply; 6+ messages in thread
From: Guixin Liu @ 2026-11-16  8:12 UTC (permalink / raw)
  To: Keith Busch, Jens Axboe, Christoph Hellwig, Sagi Grimberg,
	Nilay Shroff, Daniel Wagner, Hannes Reinecke
  Cc: linux-nvme

nvme_ctrl_auth_work() queues the DH-HMAC-CHAP exchange of every queue
on nvme-auth-wq without waiting for them, so multiple
nvme_queue_auth_work() instances run concurrently and all increment
ctrl->transaction in nvme_queue_auth_work(). The plain increment is a
data race, and the lost updates hand the same transaction id to
concurrent queues.

The counter does not need to be shared at all: the controller only
echoes the transaction id within a queue, so a per-queue counter is
sufficient. Increment chap->transaction instead, drop the reset in
nvme_auth_reset_dhchap() so the id stays monotonic across
re-authentication, and remove the now unused ctrl->transaction field.

Fixes: f50fff73d620 ("nvme: implement In-Band authentication")
Cc: stable@vger.kernel.org
Signed-off-by: Guixin Liu <kanie@linux.alibaba.com>
---

Changes since v2: reworked per review to use a per-queue counter and
drop the shared field entirely (Keith Busch), and the Fixes tag was
added (sashiko).  The Reviewed-by on v2 covered the atomic_t version
of this change and is dropped.
---
 drivers/nvme/host/auth.c | 3 +--
 drivers/nvme/host/nvme.h | 1 -
 2 files changed, 1 insertion(+), 3 deletions(-)

diff --git a/drivers/nvme/host/auth.c b/drivers/nvme/host/auth.c
index 9a3da74dec5a..54f3354ecae9 100644
--- a/drivers/nvme/host/auth.c
+++ b/drivers/nvme/host/auth.c
@@ -644,7 +644,6 @@ static void nvme_auth_reset_dhchap(struct nvme_dhchap_queue_context *chap)
 	chap->s1 = 0;
 	chap->s2 = 0;
 	chap->bi_directional = false;
-	chap->transaction = 0;
 	memset(chap->c1, 0, sizeof(chap->c1));
 	memset(chap->c2, 0, sizeof(chap->c2));
 	mempool_free(chap->buf, nvme_chap_buf_pool);
@@ -764,7 +763,7 @@ static void nvme_queue_auth_work(struct work_struct *work)
 		return;
 	}
 
-	chap->transaction = ctrl->transaction++;
+	chap->transaction++;
 
 	/* DH-HMAC-CHAP Step 1: send negotiate */
 	dev_dbg(ctrl->device, "%s: qid %d send negotiate\n",
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 2543e52d6358..e015d3a3e29c 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -463,7 +463,6 @@ struct nvme_ctrl {
 	struct nvme_dhchap_queue_context *dhchap_ctxs;
 	struct nvme_dhchap_key *host_key;
 	struct nvme_dhchap_key *ctrl_key;
-	u16 transaction;
 #endif
 	key_serial_t tls_pskid;
 
-- 
2.43.7



^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: [PATCH v3 2/2] nvme-auth: fix data race on ctrl->transaction
  2026-11-16  8:12 ` [PATCH v3 2/2] nvme-auth: fix data race on ctrl->transaction Guixin Liu
@ 2026-11-16 13:22   ` Hannes Reinecke
  2026-11-16 15:37     ` Keith Busch
  2026-11-17  2:16     ` Guixin Liu
  0 siblings, 2 replies; 6+ messages in thread
From: Hannes Reinecke @ 2026-11-16 13:22 UTC (permalink / raw)
  To: Guixin Liu, Keith Busch, Jens Axboe, Christoph Hellwig,
	Sagi Grimberg, Nilay Shroff, Daniel Wagner
  Cc: linux-nvme

On 11/16/26 9:12 AM, Guixin Liu wrote:
> nvme_ctrl_auth_work() queues the DH-HMAC-CHAP exchange of every queue
> on nvme-auth-wq without waiting for them, so multiple
> nvme_queue_auth_work() instances run concurrently and all increment
> ctrl->transaction in nvme_queue_auth_work(). The plain increment is a
> data race, and the lost updates hand the same transaction id to
> concurrent queues.
> 
> The counter does not need to be shared at all: the controller only
> echoes the transaction id within a queue, so a per-queue counter is
> sufficient. Increment chap->transaction instead, drop the reset in
> nvme_auth_reset_dhchap() so the id stays monotonic across
> re-authentication, and remove the now unused ctrl->transaction field.
> 
I disagree. The authentication transaction ID identifies the
transaction itself, and is used to differentiate between different
transactions. As such it needs to be unique within each queue.
But as it carries controller-specific data we should also ensure
the ID is unique per controller, ensuring we catch any possible
misdirected authentication packet.

It should be possible to move the controller value to an atomic
value, and then we could do

chap->transaction = atomic_inc_return(&ctrl->transaction);

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH v3 2/2] nvme-auth: fix data race on ctrl->transaction
  2026-11-16 13:22   ` Hannes Reinecke
@ 2026-11-16 15:37     ` Keith Busch
  2026-11-17  2:16     ` Guixin Liu
  1 sibling, 0 replies; 6+ messages in thread
From: Keith Busch @ 2026-11-16 15:37 UTC (permalink / raw)
  To: Hannes Reinecke
  Cc: Guixin Liu, Jens Axboe, Christoph Hellwig, Sagi Grimberg,
	Nilay Shroff, Daniel Wagner, linux-nvme

On Mon, Nov 16, 2026 at 02:22:51PM +0100, Hannes Reinecke wrote:
> On 11/16/26 9:12 AM, Guixin Liu wrote:
> > nvme_ctrl_auth_work() queues the DH-HMAC-CHAP exchange of every queue
> > on nvme-auth-wq without waiting for them, so multiple
> > nvme_queue_auth_work() instances run concurrently and all increment
> > ctrl->transaction in nvme_queue_auth_work(). The plain increment is a
> > data race, and the lost updates hand the same transaction id to
> > concurrent queues.
> > 
> > The counter does not need to be shared at all: the controller only
> > echoes the transaction id within a queue, so a per-queue counter is
> > sufficient. Increment chap->transaction instead, drop the reset in
> > nvme_auth_reset_dhchap() so the id stays monotonic across
> > re-authentication, and remove the now unused ctrl->transaction field.
> > 
> I disagree. The authentication transaction ID identifies the
> transaction itself, and is used to differentiate between different
> transactions. As such it needs to be unique within each queue.

But the proposal *is* unique within the queue. It's just not unique at
the controller level, and it sounds like the protocol doesn't need to
be.

> But as it carries controller-specific data we should also ensure
> the ID is unique per controller, ensuring we catch any possible
> misdirected authentication packet.

I don't understand. The qid:T_ID tuple already identifies the
transaction. If a controller mixes up transactions across queues, it's
already broken in ways we can't fix, so we shouldn't design around it.
 
> It should be possible to move the controller value to an atomic
> value, and then we could do

If there's no protocol reason to require this type of uniqueness, then
I'm not seeing why the driver is putting this constraint on itself.


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH v3 2/2] nvme-auth: fix data race on ctrl->transaction
  2026-11-16 13:22   ` Hannes Reinecke
  2026-11-16 15:37     ` Keith Busch
@ 2026-11-17  2:16     ` Guixin Liu
  1 sibling, 0 replies; 6+ messages in thread
From: Guixin Liu @ 2026-11-17  2:16 UTC (permalink / raw)
  To: Hannes Reinecke, Keith Busch, Jens Axboe, Christoph Hellwig,
	Sagi Grimberg, Nilay Shroff, Daniel Wagner
  Cc: linux-nvme



在 2026/11/16 21:22, Hannes Reinecke 写道:
> On 11/16/26 9:12 AM, Guixin Liu wrote:
>> nvme_ctrl_auth_work() queues the DH-HMAC-CHAP exchange of every queue
>> on nvme-auth-wq without waiting for them, so multiple
>> nvme_queue_auth_work() instances run concurrently and all increment
>> ctrl->transaction in nvme_queue_auth_work(). The plain increment is a
>> data race, and the lost updates hand the same transaction id to
>> concurrent queues.
>>
>> The counter does not need to be shared at all: the controller only
>> echoes the transaction id within a queue, so a per-queue counter is
>> sufficient. Increment chap->transaction instead, drop the reset in
>> nvme_auth_reset_dhchap() so the id stays monotonic across
>> re-authentication, and remove the now unused ctrl->transaction field.
>>
> I disagree. The authentication transaction ID identifies the
> transaction itself, and is used to differentiate between different
> transactions. As such it needs to be unique within each queue.
> But as it carries controller-specific data we should also ensure
> the ID is unique per controller, ensuring we catch any possible
> misdirected authentication packet.
A misdirected packet is caught regardless of the transaction id.
Each queue's exchange carries its own random challenge:
nvmet_auth_challenge() fills req->sq->dhcahp_c1 with get_random_bytes()
per exchange, and the response digest covers that challenge.
A response computed for queue A fails the digest check on queue B even
when the transaction id and the step happen to match, because the
expected digest is different.

So controller-level uniqueness would only detect a case the crypto
layer already rejects, while within a queue the id is unique either
way.

Best Regards,
Guixin Liu
>
> It should be possible to move the controller value to an atomic
> value, and then we could do
>
> chap->transaction = atomic_inc_return(&ctrl->transaction);
>
> Cheers,
>
> Hannes



^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2026-11-17  2:16 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-16  8:12 [PATCH v3 0/2] nvme: fabrics and target fixes Guixin Liu
2026-11-16  8:12 ` [PATCH v3 1/2] nvme-fc: fix use-after-free in nvme_fc_match_disconn_ls() Guixin Liu
2026-11-16  8:12 ` [PATCH v3 2/2] nvme-auth: fix data race on ctrl->transaction Guixin Liu
2026-11-16 13:22   ` Hannes Reinecke
2026-11-16 15:37     ` Keith Busch
2026-11-17  2:16     ` Guixin Liu

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).