[PATCHSET 0/98] Various block related fixes

[PATCHSET 0/98] Various block related fixes

[Jens Axboe]

Hi,

Got a giant pile of stuff dumped on me, and went through and did some
vetting and, particularly, commit message rewriting to get it to a
somewhat comprehensible shape. If you're CC'ed on a patch, please
review it. Feel free to queue it up, or reply back why the particular
patch is garbage, if it is garbage.

Thanks!

-- 
Jens Axboe

[PATCH 01/98] blk-cgroup: keep the old blkg while reassociating a bio

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

bio_associate_blkg_from_css() drops the bio's old blkg before it looks up
the new one through the css. The css is pinned by the blkg, not by the
bio, so if the cgroup is gone the bio may hold the last reference and the
blkcg can be freed under blkg_tryget_closest(). The rcu_read_lock()
around the whole thing used to cover that, until commit 457d3c4f0fdd
dropped it.

Put the old blkg after the new one has been looked up.

Fixes: 457d3c4f0fdd ("blk-cgroup: don't nest queue_lock under rcu in bio_associate_blkg()")
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-cgroup.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 82b66147d778..e79177ac2dc2 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -2139,8 +2139,7 @@ static inline struct blkcg_gq *blkg_tryget_closest(struct bio *bio,
 void bio_associate_blkg_from_css(struct bio *bio,
 				 struct cgroup_subsys_state *css)
 {
-	if (bio->bi_blkg)
-		blkg_put(bio->bi_blkg);
+	struct blkcg_gq *old_blkg = bio->bi_blkg;
 
 	if (css && css->parent) {
 		bio->bi_blkg = blkg_tryget_closest(bio, css);
@@ -2148,6 +2147,10 @@ void bio_associate_blkg_from_css(struct bio *bio,
 		blkg_get(bdev_get_queue(bio->bi_bdev)->root_blkg);
 		bio->bi_blkg = bdev_get_queue(bio->bi_bdev)->root_blkg;
 	}
+
+	/* @old_blkg may be the only thing which pins @css */
+	if (old_blkg)
+		blkg_put(old_blkg);
 }
 EXPORT_SYMBOL_GPL(bio_associate_blkg_from_css);
 
-- 
2.56.0

Re: [PATCH 01/98] blk-cgroup: keep the old blkg while reassociating a bio

[Tejun Heo]

On Wed, Oct 28, 2026 at 10:41:04AM -0600, Jens Axboe wrote:
> bio_associate_blkg_from_css() drops the bio's old blkg before it looks up
> the new one through the css. The css is pinned by the blkg, not by the
> bio, so if the cgroup is gone the bio may hold the last reference and the
> blkcg can be freed under blkg_tryget_closest(). The rcu_read_lock()
> around the whole thing used to cover that, until commit 457d3c4f0fdd
> dropped it.
>
> Put the old blkg after the new one has been looked up.
>
> Fixes: 457d3c4f0fdd ("blk-cgroup: don't nest queue_lock under rcu in bio_associate_blkg()")
> Cc: Tejun Heo <tj@kernel.org>
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> Signed-off-by: Jens Axboe <axboe@kernel.dk>

Acked-by: Tejun Heo <tj@kernel.org>

Thanks.

--
tejun

[PATCH 02/98] blk-iocost: free struct ioc after a grace period

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

ioc_qos_prfill() and ioc_cost_model_prfill() take ioc->lock from
blkcg_print_blkgs(), which since commit 56cc24f59c14 runs under RCU only.
That covers the policy data, which is freed after a grace period, but
not the ioc it points to: ioc_rqos_exit() frees that right away, and a
concurrent read of io.cost.qos locks freed memory.

Free the ioc with kfree_rcu_mightsleep().

Fixes: 56cc24f59c14 ("blk-cgroup: don't nest queue_lock under rcu in blkcg_print_blkgs()")
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-iocost.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index a81488c755ca..7b5d865d8c14 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -2920,7 +2920,8 @@ static void ioc_rqos_exit(struct rq_qos *rqos)
 
 	timer_shutdown_sync(&ioc->timer);
 	free_percpu(ioc->pcpu_stat);
-	kfree(ioc);
+	/* blkcg_print_blkgs() gets here from a pd with only RCU held */
+	kfree_rcu_mightsleep(ioc);
 }
 
 static const struct rq_qos_ops ioc_rqos_ops = {
-- 
2.56.0

Re: [PATCH 02/98] blk-iocost: free struct ioc after a grace period

[Tejun Heo]

On Wed, Oct 28, 2026 at 10:41:05AM -0600, Jens Axboe wrote:
> ioc_qos_prfill() and ioc_cost_model_prfill() take ioc->lock from
> blkcg_print_blkgs(), which since commit 56cc24f59c14 runs under RCU only.
> That covers the policy data, which is freed after a grace period, but
> not the ioc it points to: ioc_rqos_exit() frees that right away, and a
> concurrent read of io.cost.qos locks freed memory.
>
> Free the ioc with kfree_rcu_mightsleep().
>
> Fixes: 56cc24f59c14 ("blk-cgroup: don't nest queue_lock under rcu in blkcg_print_blkgs()")
> Cc: Tejun Heo <tj@kernel.org>
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> Signed-off-by: Jens Axboe <axboe@kernel.dk>

Acked-by: Tejun Heo <tj@kernel.org>

Thanks.

--
tejun

[PATCH 03/98] blk-throttle: don't truncate the cgroup v1 iops limits

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

tg_set_conf() parses the limit into a u64 and stores the iops ones
through an unsigned int, so anything above UINT_MAX is truncated and a
multiple of 2^32 becomes a limit of 0. The next IO of the group then
divides by it in tg_within_iops_limit().

Clamp to UINT_MAX, which means no limit, like tg_set_limit() does on the
default hierarchy.

Fixes: af133ceb2610 ("blkcg: move blkio_group_conf->iops and ->bps to blk-throttle")
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-throttle.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index ffc3b70065d4..4f10fe05744b 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -1386,7 +1386,8 @@ static ssize_t tg_set_conf(struct kernfs_open_file *of,
 	if (is_u64)
 		*(u64 *)((void *)tg + of_cft(of)->private) = v;
 	else
-		*(unsigned int *)((void *)tg + of_cft(of)->private) = v;
+		*(unsigned int *)((void *)tg + of_cft(of)->private) =
+			min_t(u64, v, UINT_MAX);
 
 	tg_conf_updated(tg, false);
 	ret = 0;
-- 
2.56.0

Re: [PATCH 03/98] blk-throttle: don't truncate the cgroup v1 iops limits

[Tejun Heo]

On Wed, Oct 28, 2026 at 10:41:06AM -0600, Jens Axboe wrote:
> tg_set_conf() parses the limit into a u64 and stores the iops ones
> through an unsigned int, so anything above UINT_MAX is truncated and a
> multiple of 2^32 becomes a limit of 0. The next IO of the group then
> divides by it in tg_within_iops_limit().
>
> Clamp to UINT_MAX, which means no limit, like tg_set_limit() does on the
> default hierarchy.
>
> Fixes: af133ceb2610 ("blkcg: move blkio_group_conf->iops and ->bps to blk-throttle")
> Cc: Tejun Heo <tj@kernel.org>
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> Signed-off-by: Jens Axboe <axboe@kernel.dk>

Acked-by: Tejun Heo <tj@kernel.org>

Thanks.

--
tejun

[PATCH 04/98] block: fix use-after-free of the disk in sync_bdevs()

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

sync_bdevs() holds a reference to the bdev inode while it takes
bd_disk->open_mutex. For a partition the inode doesn't keep the gendisk
alive, the device reference does: part_release() drops the disk as soon
as the last reference to the partition device is gone. A sync(2) racing
with the last close of a partition of an already deleted disk locks a
freed open_mutex.

Take a device reference before touching the disk, like
blkdev_get_no_open() does, and skip the bdev if it's on its way out.

Fixes: a8698707a183 ("block: move bd_mutex to struct gendisk")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/bdev.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/block/bdev.c b/block/bdev.c
index fac74319e9fb..cc35126e483d 100644
--- a/block/bdev.c
+++ b/block/bdev.c
@@ -1366,6 +1366,15 @@ void sync_bdevs(bool wait)
 		old_inode = inode;
 		bdev = I_BDEV(inode);
 
+		/*
+		 * The inode does not keep the gendisk of a partition alive, the
+		 * device does.
+		 */
+		if (!kobject_get_unless_zero(&bdev->bd_device.kobj)) {
+			spin_lock(&blockdev_superblock->s_inode_list_lock);
+			continue;
+		}
+
 		mutex_lock(&bdev->bd_disk->open_mutex);
 		if (!atomic_read(&bdev->bd_openers)) {
 			; /* skip */
@@ -1381,6 +1390,7 @@ void sync_bdevs(bool wait)
 			filemap_fdatawrite(inode->i_mapping);
 		}
 		mutex_unlock(&bdev->bd_disk->open_mutex);
+		put_device(&bdev->bd_device);
 
 		spin_lock(&blockdev_superblock->s_inode_list_lock);
 	}
-- 
2.56.0

[PATCH 05/98] block: don't lose the status of a failed flush request

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

flush_end_io() stores the status in fq->rq_status for whoever drops the
last reference to finish the flush sequence. blk_mq_put_rq_ref() calls
it with BLK_STS_OK from tag iterators and the timeout handler and
overwrites an error the real completion stored. With two extra
references, reading /sys/block/<disk>/inflight is enough for that, a
failed flush completes everything waiting on it with success and fsync()
lies to the application.

Only store a status that is an error.

Fixes: 2e315dc07df0 ("blk-mq: grab rq->refcount before calling ->fn in blk_mq_tagset_busy_iter")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-flush.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/block/blk-flush.c b/block/blk-flush.c
index 403a46c86411..38bd4ae5d9e7 100644
--- a/block/blk-flush.c
+++ b/block/blk-flush.c
@@ -212,7 +212,13 @@ static enum rq_end_io_ret flush_end_io(struct request *flush_rq,
 	spin_lock_irqsave(&fq->mq_flush_lock, flags);
 
 	if (!req_ref_put_and_test(flush_rq)) {
-		fq->rq_status = error;
+		/*
+		 * Remember the status of the flush for whoever drops the last
+		 * reference. blk_mq_put_rq_ref() gets here with BLK_STS_OK and
+		 * must not wipe it out.
+		 */
+		if (error != BLK_STS_OK)
+			fq->rq_status = error;
 		spin_unlock_irqrestore(&fq->mq_flush_lock, flags);
 		return RQ_END_IO_NONE;
 	}
-- 
2.56.0

[PATCH 06/98] blk-mq: don't batch allocate tags from an inactive hctx

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

blk_mq_get_tag() checks BLK_MQ_S_INACTIVE after getting a tag and gives
it back if the flag is set, so the caller retries on the hctx of the CPU
it runs on now. The batched allocation for plugged submitters doesn't,
so a task migrated off a dying CPU can fill plug->cached_rqs from a
drained hctx, and those requests never complete.

Check the flag in blk_mq_get_tags() too and release the tags if it's
set. The caller falls back to blk_mq_get_tag(), which fails the same way
and starts over.

Fixes: 47c122e35d7e ("block: pre-allocate requests if plug is started and is a batch")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-mq-tag.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c
index 2b751bfa50e1..aaea2e2c2f23 100644
--- a/block/blk-mq-tag.c
+++ b/block/blk-mq-tag.c
@@ -139,6 +139,18 @@ unsigned long blk_mq_get_tags(struct blk_mq_alloc_data *data, int nr_tags,
 		return 0;
 	ret = __sbitmap_queue_get_batch(bt, nr_tags, offset);
 	*offset += tags->nr_reserved_tags;
+
+	/*
+	 * Give up this allocation if the hctx is inactive, like blk_mq_get_tag()
+	 * does. The caller falls back to it and will retry on an active hctx.
+	 */
+	if (ret && unlikely(test_bit(BLK_MQ_S_INACTIVE, &data->hctx->state))) {
+		unsigned int i;
+
+		for_each_set_bit(i, &ret, BITS_PER_LONG)
+			blk_mq_put_tag(tags, data->ctx, *offset + i);
+		return 0;
+	}
 	return ret;
 }
 
-- 
2.56.0

[PATCH 07/98] block: keep zone write plugs with plugged BIOs alive

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

The issuer of a plugged zone write hears about the completion in
bio_endio(), before blk_zone_write_plug_finish_request() unplugs the
zone, and may reset the zone and write it again in between. Since commit
836efd35c472 that revives the plug the reset marked dead, with the new
BIO on the list and wp_offset still 0. The unplug of the first write
then sees an empty zone and marks the plug dead again, BIO and all. Once
that BIO is done the plug is freed and the write pointer offset is lost.

Only remove the plug of an empty zone when no BIO is left on it.

Fixes: 836efd35c472 ("block: fix handling of dead zone write plugs")
Cc: Damien Le Moal <dlemoal@kernel.org>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-zoned.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/block/blk-zoned.c b/block/blk-zoned.c
index 4c5889296191..18c05e2a1746 100644
--- a/block/blk-zoned.c
+++ b/block/blk-zoned.c
@@ -1812,7 +1812,14 @@ static void disk_zone_wplug_unplug_bio(struct gendisk *disk,
 	else if (!bio_list_empty(&zwplug->bio_list))
 		disk_zone_wplug_schedule_work(disk, zwplug);
 
-	if (!zwplug->wp_offset || disk_zone_wplug_is_full(disk, zwplug))
+	/*
+	 * The issuer of the write was told about its completion before we got
+	 * here. It may already have reset the zone and started to write it
+	 * again, in which case the zone is empty but has BIOs plugged. The plug
+	 * must stay alive to track them.
+	 */
+	if ((!zwplug->wp_offset && bio_list_empty(&zwplug->bio_list)) ||
+	    disk_zone_wplug_is_full(disk, zwplug))
 		disk_mark_zone_wplug_dead(zwplug);
 
 	spin_unlock_irqrestore(&zwplug->lock, flags);
-- 
2.56.0

[PATCH 08/98] block: keep the zone write plug of a failed write

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

When a write fails, blk_zone_write_plug_bio_endio() sets
BLK_ZONE_WPLUG_NEED_WP_UPDATE as the write pointer offset is now wrong.
disk_zone_wplug_unplug_bio() doesn't look at that flag, so a failed
write that would have filled the zone makes it look full, the plug is
freed and "full" is written to zones_cond[] for a partially written
zone. Reports then show it as full and zone append emulation fails with
unaligned writes, even after the zone report an error requires.

Leave the plug alone while BLK_ZONE_WPLUG_NEED_WP_UPDATE is set.

Fixes: fe0418eb9bd6 ("block: Prevent potential deadlocks in zone write plug error recovery")
Cc: Damien Le Moal <dlemoal@kernel.org>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-zoned.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/block/blk-zoned.c b/block/blk-zoned.c
index 18c05e2a1746..882d07a7c80a 100644
--- a/block/blk-zoned.c
+++ b/block/blk-zoned.c
@@ -1817,9 +1817,13 @@ static void disk_zone_wplug_unplug_bio(struct gendisk *disk,
 	 * here. It may already have reset the zone and started to write it
 	 * again, in which case the zone is empty but has BIOs plugged. The plug
 	 * must stay alive to track them.
+	 *
+	 * If the write failed, the write pointer offset says nothing about the
+	 * zone. Keep the plug until the offset was updated.
 	 */
-	if ((!zwplug->wp_offset && bio_list_empty(&zwplug->bio_list)) ||
-	    disk_zone_wplug_is_full(disk, zwplug))
+	if (!(zwplug->flags & BLK_ZONE_WPLUG_NEED_WP_UPDATE) &&
+	    ((!zwplug->wp_offset && bio_list_empty(&zwplug->bio_list)) ||
+	     disk_zone_wplug_is_full(disk, zwplug)))
 		disk_mark_zone_wplug_dead(zwplug);
 
 	spin_unlock_irqrestore(&zwplug->lock, flags);
-- 
2.56.0

[PATCH 09/98] blk-mq: end the request if getting a keyslot fails

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

When blk_crypto_rq_get_keyslot() fails, blk_mq_submit_bio() ends the bio
by hand and frees the request. By then blk_mq_bio_to_request() has done
the IO accounting, which blk_mq_free_request() doesn't undo, so the
in_flight counter never drops and iostat shows a device that is 100%
busy. If the bio went through zone write plugging the zone stays plugged
as well, and every later write to it waits forever.

Use blk_mq_end_request() like the dispatch path does for a request the
driver rejected, after marking it for zone write plugging.

Fixes: a892c8d52c02 ("block: Inline encryption support for blk-mq")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-mq.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 7400f678258e..c91306f8156f 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -3195,8 +3195,9 @@ void blk_mq_submit_bio(struct bio *bio)
 
 	ret = blk_crypto_rq_get_keyslot(rq);
 	if (ret != BLK_STS_OK) {
-		bio_endio_status(bio, ret);
-		blk_mq_free_request(rq);
+		if (bio_zone_write_plugging(bio))
+			blk_zone_write_plug_init_request(rq);
+		blk_mq_end_request(rq, ret);
 		return;
 	}
 
-- 
2.56.0

[PATCH 10/98] block: don't unplug a zone under a failed write

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

Since commit 552c1149af7a disk_zone_wplug_abort() clears
BLK_ZONE_WPLUG_PLUGGED. For a failed write that happens in bio_endio(),
before the request reaches blk_zone_write_plug_finish_request(). The
issuer can reset the zone and submit a new write in between, which is
issued unplugged, and when the old request is finished it clears the
flag under that one, so the next write goes out alongside it.

Preserve the flag across the abort, the unplug of the failed write
clears it. The exception is QUEUE_FLAG_ZONED_QD1_WRITES, where the abort
takes the plug off the worker's list and nobody is left to clear it.

Fixes: 552c1149af7a ("block: Clear BLK_ZONE_WPLUG_PLUGGED when aborting plugged BIOs")
Cc: Damien Le Moal <dlemoal@kernel.org>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-zoned.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/block/blk-zoned.c b/block/blk-zoned.c
index 882d07a7c80a..7f405dce2fcb 100644
--- a/block/blk-zoned.c
+++ b/block/blk-zoned.c
@@ -1871,9 +1871,20 @@ void blk_zone_write_plug_bio_endio(struct bio *bio)
 	 * needing a write pointer update.
 	 */
 	if (bio->bi_status != BLK_STS_OK) {
+		unsigned int plugged = 0;
+
 		spin_lock_irqsave(&zwplug->lock, flags);
+		/*
+		 * The abort unplugs the zone. But the flag still belongs to a
+		 * write in flight (this one, until its request is finished) or
+		 * to the BIO work, and they clear it when they are done. The
+		 * zone write plugs worker does not rely on the flag to issue
+		 * one write at a time and forgets the plug with the abort.
+		 */
+		if (!blk_queue_zoned_qd1_writes(disk->queue))
+			plugged = zwplug->flags & BLK_ZONE_WPLUG_PLUGGED;
 		disk_zone_wplug_abort(zwplug);
-		zwplug->flags |= BLK_ZONE_WPLUG_NEED_WP_UPDATE;
+		zwplug->flags |= BLK_ZONE_WPLUG_NEED_WP_UPDATE | plugged;
 		spin_unlock_irqrestore(&zwplug->lock, flags);
 	}
 
-- 
2.56.0

[PATCH 11/98] block: check dma alignment of bvec iters in async dio

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

__blkdev_direct_IO_async() uses bio_iov_iter_set() for an ITER_BVEC
iterator and never gets to the alignment check in
bio_iov_iter_get_pages(), which is the one that exists for io_uring
registered buffers in the first place. A bio based driver has nobody
else to enforce its DMA alignment: a misaligned READ_FIXED from brd
returns the wrong data, and dm-verity with panic_on_error panics.

Move bio_iov_bvec_aligned() to blk.h and check in the async path too,
failing with -EINVAL like the other direct IO paths.

Fixes: 5ff3f74e145a ("block: simplify direct io validity check")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/bio.c  | 33 ---------------------------------
 block/blk.h  | 33 +++++++++++++++++++++++++++++++++
 block/fops.c | 10 ++++++++--
 3 files changed, 41 insertions(+), 35 deletions(-)

diff --git a/block/bio.c b/block/bio.c
index 06f805d3866a..57d84ca7170f 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -1272,39 +1272,6 @@ static int bio_iov_iter_align_down(struct bio *bio, struct iov_iter *iter,
 	return 0;
 }
 
-#ifdef CONFIG_DEBUG_KERNEL
-static inline bool bio_iov_bvec_aligned(const struct bio *bio,
-					unsigned mem_align_mask)
-{
-	struct bvec_iter iter;
-	struct bio_vec bv;
-
-	/*
-	 * Correct callers never break the alignment requirements, so this
-	 * exhaustive check is only paid for in debug builds.
-	 */
-	for_each_mp_bvec(bv, bio->bi_io_vec, iter, bio->bi_iter)
-		if ((bv.bv_offset | bv.bv_len) & mem_align_mask)
-			return false;
-	return true;
-}
-#else
-static inline bool bio_iov_bvec_aligned(const struct bio *bio,
-					unsigned mem_align_mask)
-{
-	/*
-	 * We forward the bio_vec as-is, so ITER_BVEC callers must provide
-	 * segments already aligned to the device's DMA alignment. The only
-	 * unchecked user-controllable offset that reaches here is an io_uring
-	 * registered buffer where just the first segment can be unaligned
-	 * (the rest is virtually contiguous), so checking only that one is
-	 * sufficient to know if the entire vector is valid.
-	 */
-	return !(mp_bvec_iter_offset(bio->bi_io_vec, bio->bi_iter) &
-							mem_align_mask);
-}
-#endif
-
 /**
  * bio_iov_iter_get_pages - add user or kernel pages to a bio
  * @bio: bio to add pages to
diff --git a/block/blk.h b/block/blk.h
index 2cc03aa54c53..a48665e10994 100644
--- a/block/blk.h
+++ b/block/blk.h
@@ -202,6 +202,39 @@ static inline bool rq_mergeable(struct request *rq)
 	return true;
 }
 
+#ifdef CONFIG_DEBUG_KERNEL
+static inline bool bio_iov_bvec_aligned(const struct bio *bio,
+					unsigned int mem_align_mask)
+{
+	struct bvec_iter iter;
+	struct bio_vec bv;
+
+	/*
+	 * Correct callers never break the alignment requirements, so this
+	 * exhaustive check is only paid for in debug builds.
+	 */
+	for_each_mp_bvec(bv, bio->bi_io_vec, iter, bio->bi_iter)
+		if ((bv.bv_offset | bv.bv_len) & mem_align_mask)
+			return false;
+	return true;
+}
+#else
+static inline bool bio_iov_bvec_aligned(const struct bio *bio,
+					unsigned int mem_align_mask)
+{
+	/*
+	 * We forward the bio_vec as-is, so ITER_BVEC callers must provide
+	 * segments already aligned to the device's DMA alignment. The only
+	 * unchecked user-controllable offset that reaches here is an io_uring
+	 * registered buffer where just the first segment can be unaligned
+	 * (the rest is virtually contiguous), so checking only that one is
+	 * sufficient to know if the entire vector is valid.
+	 */
+	return !(mp_bvec_iter_offset(bio->bi_io_vec, bio->bi_iter) &
+							mem_align_mask);
+}
+#endif
+
 /*
  * There are two different ways to handle DISCARD merges:
  *  1) If max_discard_segments > 1, the driver treats every bio as a range and
diff --git a/block/fops.c b/block/fops.c
index 375ec2c15926..dfeac910f126 100644
--- a/block/fops.c
+++ b/block/fops.c
@@ -365,9 +365,15 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb,
 	 * Users don't rely on the iterator being in any particular
 	 * state for async I/O returning -EIOCBQUEUED, hence we can
 	 * avoid expensive iov_iter_advance(). Bypass
-	 * bio_iov_iter_get_pages() and set the bvec directly.
+	 * bio_iov_iter_get_pages() and set the bvec directly, which leaves
+	 * its alignment check to us.
 	 */
-	if (!bio_iov_iter_set(bio, iter)) {
+	if (bio_iov_iter_set(bio, iter)) {
+		if (!bio_iov_bvec_aligned(bio, bdev_dma_alignment(bdev))) {
+			ret = -EINVAL;
+			goto out_bio_put;
+		}
+	} else {
 		ret = blkdev_iov_iter_get_pages(bio, iter, bdev);
 		if (unlikely(ret))
 			goto out_bio_put;
-- 
2.56.0

[PATCH 12/98] drbd: stop the start_resync timer of a dying device

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

start_resync_timer gets armed for up to a second and nothing ever
deletes it, so drbd_destroy_device() can free the device with the timer
pending. Shut it down where request_timer is shut down.

Fixes: 370a43e7982d ("drbd: Work on the Ahead -> SyncSource transition")
Cc: Philipp Reisner <philipp.reisner@linbit.com>
Cc: Lars Ellenberg <lars.ellenberg@linbit.com>
Cc: "Christoph Böhmwalder" <christoph.boehmwalder@linbit.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/block/drbd/drbd_main.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c
index a2a841c89201..099dff1ea256 100644
--- a/drivers/block/drbd/drbd_main.c
+++ b/drivers/block/drbd/drbd_main.c
@@ -2163,6 +2163,7 @@ void drbd_destroy_device(struct kref *kref)
 	struct drbd_peer_device *peer_device, *tmp_peer_device;
 
 	timer_shutdown_sync(&device->request_timer);
+	timer_shutdown_sync(&device->start_resync_timer);
 
 	/* paranoia asserts */
 	D_ASSERT(device, device->open_cnt == 0);
-- 
2.56.0

[PATCH 13/98] blk-cgroup: fix root io.stat byte counts on 32-bit

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

blkcg_fill_root_iostats() shifts the unsigned long sector counts by 9
before adding them to the u64 byte counts, so on 32-bit the contribution
of a CPU wraps every 4GB. Widen before the shift.

Fixes: ef45fe470e1e ("blk-cgroup: show global disk stats in root cgroup io.stat")
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-cgroup.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index e79177ac2dc2..c9187d95a54b 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -1118,11 +1118,11 @@ static void blkcg_fill_root_iostats(void)
 				cpu_dkstats->ios[STAT_DISCARD];
 			// convert sectors to bytes
 			tmp.bytes[BLKG_IOSTAT_READ] +=
-				cpu_dkstats->sectors[STAT_READ] << 9;
+				(u64)cpu_dkstats->sectors[STAT_READ] << 9;
 			tmp.bytes[BLKG_IOSTAT_WRITE] +=
-				cpu_dkstats->sectors[STAT_WRITE] << 9;
+				(u64)cpu_dkstats->sectors[STAT_WRITE] << 9;
 			tmp.bytes[BLKG_IOSTAT_DISCARD] +=
-				cpu_dkstats->sectors[STAT_DISCARD] << 9;
+				(u64)cpu_dkstats->sectors[STAT_DISCARD] << 9;
 		}
 
 		flags = u64_stats_update_begin_irqsave(&blkg->iostat.sync);
-- 
2.56.0

Re: [PATCH 13/98] blk-cgroup: fix root io.stat byte counts on 32-bit

[Tejun Heo]

On Wed, Oct 28, 2026 at 10:41:16AM -0600, Jens Axboe wrote:
> blkcg_fill_root_iostats() shifts the unsigned long sector counts by 9
> before adding them to the u64 byte counts, so on 32-bit the contribution
> of a CPU wraps every 4GB. Widen before the shift.
>
> Fixes: ef45fe470e1e ("blk-cgroup: show global disk stats in root cgroup io.stat")
> Cc: Tejun Heo <tj@kernel.org>
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> Signed-off-by: Jens Axboe <axboe@kernel.dk>

Acked-by: Tejun Heo <tj@kernel.org>

Thanks.

--
tejun

[PATCH 14/98] drbd: take req_lock to skip the initial sync as peer

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

_drbd_set_state() must be called with req_lock held. drbd_adm_new_c_uuid()
does, receive_uuids() on the peer doesn't, so a concurrent state change
gets lost: the device is Secondary again, or its failed disk UpToDate,
while the work for the lost change is already queued.

Fixes: b411b3637fa7 ("The DRBD driver")
Cc: Philipp Reisner <philipp.reisner@linbit.com>
Cc: Lars Ellenberg <lars.ellenberg@linbit.com>
Cc: "Christoph Böhmwalder" <christoph.boehmwalder@linbit.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/block/drbd/drbd_receiver.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/block/drbd/drbd_receiver.c b/drivers/block/drbd/drbd_receiver.c
index 2135c14354a8..a405ccdf919e 100644
--- a/drivers/block/drbd/drbd_receiver.c
+++ b/drivers/block/drbd/drbd_receiver.c
@@ -4096,8 +4096,10 @@ static int receive_uuids(struct drbd_connection *connection, struct packet_info
 					BM_LOCKED_TEST_ALLOWED, NULL);
 			_drbd_uuid_set(device, UI_CURRENT, p_uuid[UI_CURRENT]);
 			_drbd_uuid_set(device, UI_BITMAP, 0);
+			spin_lock_irq(&device->resource->req_lock);
 			_drbd_set_state(_NS2(device, disk, D_UP_TO_DATE, pdsk, D_UP_TO_DATE),
 					CS_VERBOSE, NULL);
+			spin_unlock_irq(&device->resource->req_lock);
 			drbd_md_sync(device);
 			updated_uuids = 1;
 		}
-- 
2.56.0

[PATCH 15/98] zloop: refuse zone files without read/write_iter

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

zloop_do_rw() calls ->read_iter() and ->write_iter() on the zone files
without checking that they exist. A base_dir on hugetlbfs, or zone files
that are links into procfs, create a device just fine and the first IO
jumps to NULL. Check for both when a zone file is opened, like loop does
for its backing file.

Fixes: eb0570c7df23 ("block: new zoned loop block device driver")
Cc: Damien Le Moal <dlemoal@kernel.org>
Cc: Christoph Hellwig <hch@lst.de>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/block/zloop.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/drivers/block/zloop.c b/drivers/block/zloop.c
index 9145e0f98d56..b4076940ae7f 100644
--- a/drivers/block/zloop.c
+++ b/drivers/block/zloop.c
@@ -1056,6 +1056,16 @@ static struct file *zloop_filp_open_fmt(int oflags, umode_t mode,
 		return ERR_PTR(-ENOMEM);
 	file = filp_open(p, oflags, mode);
 	kfree(p);
+	if (IS_ERR(file))
+		return file;
+
+	/* Zone files are accessed with ->read_iter() and ->write_iter(). */
+	if ((oflags & O_ACCMODE) == O_RDWR &&
+	    (!file->f_op->read_iter || !file->f_op->write_iter)) {
+		fput(file);
+		return ERR_PTR(-EINVAL);
+	}
+
 	return file;
 }
 
-- 
2.56.0

Re: [PATCH 15/98] zloop: refuse zone files without read/write_iter

[Johannes Thumshirn]

On 10/28/26 5:41 PM, Jens Axboe wrote:
> From: Josef Bacik <josef@toxicpanda.com>
>
> zloop_do_rw() calls ->read_iter() and ->write_iter() on the zone files
> without checking that they exist. A base_dir on hugetlbfs, or zone files
> that are links into procfs, create a device just fine and the first IO
> jumps to NULL. Check for both when a zone file is opened, like loop does
> for its backing file.

Not particularly against the patch, but I don't think you can have a zloop

zone file be a link into procfs. Well you can if you create the zloop, 
tear it

down again, then rm a zone file and re-create it as a link into proc. So 
I guess:

Reviewed-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>


> Fixes: eb0570c7df23 ("block: new zoned loop block device driver")
> Cc: Damien Le Moal <dlemoal@kernel.org>
> Cc: Christoph Hellwig <hch@lst.de>
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> Signed-off-by: Jens Axboe <axboe@kernel.dk>
> ---
>   drivers/block/zloop.c | 10 ++++++++++
>   1 file changed, 10 insertions(+)
>
> diff --git a/drivers/block/zloop.c b/drivers/block/zloop.c
> index 9145e0f98d56..b4076940ae7f 100644
> --- a/drivers/block/zloop.c
> +++ b/drivers/block/zloop.c
> @@ -1056,6 +1056,16 @@ static struct file *zloop_filp_open_fmt(int oflags, umode_t mode,
>   		return ERR_PTR(-ENOMEM);
>   	file = filp_open(p, oflags, mode);
>   	kfree(p);
> +	if (IS_ERR(file))
> +		return file;
> +
> +	/* Zone files are accessed with ->read_iter() and ->write_iter(). */
> +	if ((oflags & O_ACCMODE) == O_RDWR &&

I think you can drop the O_RDWR check, the zone files are always opened 
O_RDWR,

only zloop_dev_exists() passes O_RDONLY to zloop_filp_open_fmt() but 
that is just to

check if the files exist and they're fput() right after it.


> +	    (!file->f_op->read_iter || !file->f_op->write_iter)) {
> +		fput(file);
> +		return ERR_PTR(-EINVAL);
> +	}
> +
>   	return file;
>   }
>   

[PATCH 16/98] block/rnbd-srv: don't free the device on a sysfs error

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

rnbd_srv_create_dev_sysfs() drops the only kobject reference on its
error paths, and the release function frees the rnbd_srv_dev.
process_msg_open() keeps using it and ends in destroy_device() with a
list_del() and a second kfree(). This doesn't need memory pressure: a
client opening a device while its last mapping goes away gets -EEXIST
from kobject_add() against the old directory.

Only undo kobject_add() on failure and leave the reference to
destroy_device(). Initialise the kobject once, the next open retries.

Fixes: 8cee532f469b ("block/rnbd: server: sysfs interface functions")
Fixes: d3a95ccaaf4d ("block/rnbd: call kobject_put in the failure path")
Cc: "Md. Haris Iqbal" <haris.iqbal@ionos.com>
Cc: Jack Wang <jinpu.wang@ionos.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/block/rnbd/rnbd-srv-sysfs.c | 16 ++++++++++------
 drivers/block/rnbd/rnbd-srv.c       |  3 +++
 2 files changed, 13 insertions(+), 6 deletions(-)

diff --git a/drivers/block/rnbd/rnbd-srv-sysfs.c b/drivers/block/rnbd/rnbd-srv-sysfs.c
index 64780094442c..33d5d1738be5 100644
--- a/drivers/block/rnbd/rnbd-srv-sysfs.c
+++ b/drivers/block/rnbd/rnbd-srv-sysfs.c
@@ -44,12 +44,17 @@ int rnbd_srv_create_dev_sysfs(struct rnbd_srv_dev *dev,
 	struct kobject *bdev_kobj;
 	int ret;
 
-	ret = kobject_init_and_add(&dev->dev_kobj, &dev_ktype,
-				   rnbd_devs_kobj, "%pg", bdev);
-	if (ret) {
-		kobject_put(&dev->dev_kobj);
+	/*
+	 * The release function frees the device, which the caller goes on
+	 * using and which others may share: never drop the reference of the
+	 * kobject here. That is for destroy_device().
+	 */
+	if (!dev->dev_kobj.state_initialized)
+		kobject_init(&dev->dev_kobj, &dev_ktype);
+
+	ret = kobject_add(&dev->dev_kobj, rnbd_devs_kobj, "%pg", bdev);
+	if (ret)
 		return ret;
-	}
 
 	dev->dev_sessions_kobj = kobject_create_and_add("sessions",
 							&dev->dev_kobj);
@@ -69,7 +74,6 @@ int rnbd_srv_create_dev_sysfs(struct rnbd_srv_dev *dev,
 	kobject_put(dev->dev_sessions_kobj);
 free_dev_kobj:
 	kobject_del(&dev->dev_kobj);
-	kobject_put(&dev->dev_kobj);
 	return ret;
 }
 
diff --git a/drivers/block/rnbd/rnbd-srv.c b/drivers/block/rnbd/rnbd-srv.c
index 79c9a5fb418f..2b0619389fc9 100644
--- a/drivers/block/rnbd/rnbd-srv.c
+++ b/drivers/block/rnbd/rnbd-srv.c
@@ -208,6 +208,9 @@ static void destroy_device(struct kref *kref)
 		 * Destroy kobj only if it was really created.
 		 */
 		rnbd_srv_destroy_dev_sysfs(dev);
+	else if (dev->dev_kobj.state_initialized)
+		/* The files could not be created. Frees the name, too. */
+		kobject_put(&dev->dev_kobj);
 	else
 		kfree(dev);
 }
-- 
2.56.0

[PATCH 17/98] block/rnbd-srv: drop the sess_dev kobject unlocked

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

rnbd_srv_create_dev_session_sysfs() drops the kobject reference when it
fails. The release function is rnbd_destroy_sess_dev(), which takes
srv_dev->lock, and the caller holds that lock: the worker deadlocks on
itself. A client opening one device as "ram0" and "./ram0" gets there
through -EEXIST from kobject_add().

Have process_msg_open() drop the reference after releasing the lock, and
leave the rest of the cleanup to the release function.

Fixes: 8cee532f469b ("block/rnbd: server: sysfs interface functions")
Fixes: d3a95ccaaf4d ("block/rnbd: call kobject_put in the failure path")
Cc: "Md. Haris Iqbal" <haris.iqbal@ionos.com>
Cc: Jack Wang <jinpu.wang@ionos.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/block/rnbd/rnbd-srv-sysfs.c | 12 ++++++------
 drivers/block/rnbd/rnbd-srv.c       |  8 +++++++-
 2 files changed, 13 insertions(+), 7 deletions(-)

diff --git a/drivers/block/rnbd/rnbd-srv-sysfs.c b/drivers/block/rnbd/rnbd-srv-sysfs.c
index 33d5d1738be5..9d00c42f2624 100644
--- a/drivers/block/rnbd/rnbd-srv-sysfs.c
+++ b/drivers/block/rnbd/rnbd-srv-sysfs.c
@@ -193,6 +193,10 @@ static const struct kobj_type rnbd_srv_sess_dev_ktype = {
 	.release	= rnbd_srv_sess_dev_release,
 };
 
+/*
+ * On failure the caller has to drop the reference of the kobject. The release
+ * function takes the lock of the device, which the caller holds here.
+ */
 int rnbd_srv_create_dev_session_sysfs(struct rnbd_srv_sess_dev *sess_dev)
 {
 	int ret;
@@ -200,17 +204,13 @@ int rnbd_srv_create_dev_session_sysfs(struct rnbd_srv_sess_dev *sess_dev)
 	ret = kobject_init_and_add(&sess_dev->kobj, &rnbd_srv_sess_dev_ktype,
 				   sess_dev->dev->dev_sessions_kobj, "%s",
 				   sess_dev->sess->sessname);
-	if (ret) {
-		kobject_put(&sess_dev->kobj);
+	if (ret)
 		return ret;
-	}
 
 	ret = sysfs_create_group(&sess_dev->kobj,
 				 &rnbd_srv_default_dev_session_attr_group);
-	if (ret) {
+	if (ret)
 		kobject_del(&sess_dev->kobj);
-		kobject_put(&sess_dev->kobj);
-	}
 
 	return ret;
 }
diff --git a/drivers/block/rnbd/rnbd-srv.c b/drivers/block/rnbd/rnbd-srv.c
index 2b0619389fc9..1e3de82fd70b 100644
--- a/drivers/block/rnbd/rnbd-srv.c
+++ b/drivers/block/rnbd/rnbd-srv.c
@@ -590,6 +590,7 @@ rnbd_srv_create_set_sess_dev(struct rnbd_srv_session *srv_sess,
 		return sdev;
 
 	kref_init(&sdev->kref);
+	INIT_LIST_HEAD(&sdev->dev_list);
 
 	strscpy(sdev->pathname, open_msg->dev_name, sizeof(sdev->pathname));
 
@@ -789,7 +790,12 @@ static int process_msg_open(struct rnbd_srv_session *srv_sess,
 		rnbd_srv_err(srv_sess_dev,
 			      "Opening device failed, failed to create dev client sysfs files, err: %d\n",
 			      ret);
-		goto free_srv_sess_dev;
+		/*
+		 * rnbd_destroy_sess_dev() undoes everything down to the open
+		 * of the block device.
+		 */
+		kobject_put(&srv_sess_dev->kobj);
+		goto free_path;
 	}
 
 	list_add(&srv_sess_dev->dev_list, &srv_dev->sess_dev_list);
-- 
2.56.0

[PATCH 18/98] aoe: don't free a device with its work queued

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

flush() picks the devices without pending work and frees them, but since
aoedev_downdev() may sleep, the check and the DEVFL_TKILL marking are no
longer one step under the lock. A reply to an ATA IDENTIFY in between
queues d->work, and the device is freed with it queued.

Set DEVFL_TKILL under d->lock, don't queue the work for a device that
has it, and cancel the work in freedev().

Fixes: 430380b4637a ("block: aoe: Fix kernel crash due to atomic sleep when exiting")
Cc: Justin Sanders <justin@coraid.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/block/aoe/aoecmd.c | 2 +-
 drivers/block/aoe/aoedev.c | 4 ++++
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/drivers/block/aoe/aoecmd.c b/drivers/block/aoe/aoecmd.c
index a4744a30a8af..c5b102947998 100644
--- a/drivers/block/aoe/aoecmd.c
+++ b/drivers/block/aoe/aoecmd.c
@@ -974,7 +974,7 @@ ataid_complete(struct aoedev *d, struct aoetgt *t, unsigned char *id)
 			d->fw_ver, (long long)ssize);
 	d->ssize = ssize;
 	d->geo.start = 0;
-	if (d->flags & (DEVFL_GDALLOC|DEVFL_NEWSIZE))
+	if (d->flags & (DEVFL_GDALLOC | DEVFL_NEWSIZE | DEVFL_TKILL))
 		return;
 	if (d->gd != NULL)
 		d->flags |= DEVFL_NEWSIZE;
diff --git a/drivers/block/aoe/aoedev.c b/drivers/block/aoe/aoedev.c
index 6e8b3807c1c8..5b443b3b224e 100644
--- a/drivers/block/aoe/aoedev.c
+++ b/drivers/block/aoe/aoedev.c
@@ -286,6 +286,8 @@ freedev(struct aoedev *d)
 		return;
 
 	timer_delete_sync(&d->timer);
+	/* With DEVFL_TKILL set only the work itself still queues it */
+	cancel_work_sync(&d->work);
 	if (d->gd) {
 		aoedisk_rm_debugfs(d);
 		del_gendisk(d->gd);
@@ -356,7 +358,9 @@ flush(const char __user *str, size_t cnt, int exiting)
 		spin_unlock(&d->lock);
 		spin_unlock_irqrestore(&devlist_lock, flags);
 		aoedev_downdev(d);
+		spin_lock_irqsave(&d->lock, flags);
 		d->flags |= DEVFL_TKILL;
+		spin_unlock_irqrestore(&d->lock, flags);
 		goto restart1;
 cont:
 		spin_unlock(&d->lock);
-- 
2.56.0

Re: [PATCH 18/98] aoe: don't free a device with its work queued

[Bart Van Assche]

On 10/28/26 09:41, Jens Axboe wrote:
> @@ -356,7 +358,9 @@ flush(const char __user *str, size_t cnt, int exiting)
>  		spin_unlock(&d->lock);
>  		spin_unlock_irqrestore(&devlist_lock, flags);
>  		aoedev_downdev(d);
> +		spin_lock_irqsave(&d->lock, flags);
>  		d->flags |= DEVFL_TKILL;
> +		spin_unlock_irqrestore(&d->lock, flags);
>  		goto restart1;

Setting DEVFL_TKILL only after aoedev_downdev(d) returns leaves a race
window open while aoedev_downdev(d) drops d->lock and sleeps in
blk_mq_freeze_queue(d->blkq):

1. An incoming ATA IDENTIFY reply processed by ataid_complete() during
   that window still sees !(d->flags & DEVFL_TKILL), sets DEVFL_NEWSIZE
   (or DEVFL_GDALLOC), and queues d->work.
2. aoecmd_sleepwork() does not check DEVFL_TKILL either. If it runs
   before freedev() reaches cancel_work_sync(&d->work), it calls
   aoeblk_gdalloc() or sets d->flags |= DEVFL_UP again after
   aoedev_downdev() cleared DEVFL_UP, allowing new requests to be queued
   onto d and hanging del_gendisk() in freedev().

Should d->flags |= DEVFL_TKILL be set under d->lock before unlocking and
calling aoedev_downdev(d) (which also prevents two concurrent
aoedev_flush() callers from running aoedev_downdev() on the same
device), and should aoecmd_sleepwork() bail out if DEVFL_TKILL is set in
case d->work was queued between flush_workqueue(aoe_wq) and acquiring
d->lock?

Re: [PATCH 18/98] aoe: don't free a device with its work queued

[Bart Van Assche]

On 10/28/26 09:41, Jens Axboe wrote:
> @@ -356,7 +358,9 @@ flush(const char __user *str, size_t cnt, int exiting)
>  		spin_unlock(&d->lock);
>  		spin_unlock_irqrestore(&devlist_lock, flags);
>  		aoedev_downdev(d);
> +		spin_lock_irqsave(&d->lock, flags);
>  		d->flags |= DEVFL_TKILL;
> +		spin_unlock_irqrestore(&d->lock, flags);
>  		goto restart1;

Setting DEVFL_TKILL only after aoedev_downdev(d) returns leaves a race
window open while aoedev_downdev(d) drops d->lock and sleeps in
blk_mq_freeze_queue(d->blkq):

1. ataid_complete(), rexmit_timer(), or aoeblk_gdalloc()'s error path
   can still see !(d->flags & DEVFL_TKILL) and queue d->work during that
   window.
2. aoecmd_sleepwork() does not check DEVFL_TKILL either. If it runs
   before freedev() reaches cancel_work_sync(&d->work), it can call
   aoeblk_gdalloc() (hitting WARN_ON(d->flags & DEVFL_TKILL) if
   DEVFL_TKILL was set in the meantime) or set d->flags |= DEVFL_UP
   again after aoedev_downdev() cleared DEVFL_UP, letting new I/O enter
   and hanging del_gendisk() in freedev().

Instead of setting DEVFL_TKILL after aoedev_downdev(d), how about:

- Serializing flush() with a static flush_mutex so a second
  aoedev_flush() caller cannot enter pass 2 (freedev()) and pass 3
  (kfree()) on a device while the first caller is still in pass 1.
- Setting d->flags |= DEVFL_TKILL under d->lock before unlocking, then
  running cancel_work_sync(&d->work) before aoedev_downdev(d) (and
  keeping cancel_work_sync(&d->work) after timer_delete_sync(&d->timer)
  in freedev()).
- Checking DEVFL_TKILL at all three queue_work(aoe_wq, &d->work) sites
  (ataid_complete(), rexmit_timer(), and aoeblk_gdalloc()), as well as
  in aoecmd_sleepwork() and aoeblk_gdalloc().

Here is a replacement patch against the same baseline (checked with
checkpatch.pl and syntax-checked with Clang; not runtime-tested):

---
diff --git a/drivers/block/aoe/aoeblk.c b/drivers/block/aoe/aoeblk.c
index 34ead75e7e02..69586dc6d379 100644
--- a/drivers/block/aoe/aoeblk.c
+++ b/drivers/block/aoe/aoeblk.c
@@ -385,7 +385,10 @@ aoeblk_gdalloc(void *vp)
 	spin_lock_irqsave(&d->lock, flags);
 	WARN_ON(!(d->flags & DEVFL_GD_NOW));
 	WARN_ON(!(d->flags & DEVFL_GDALLOC));
-	WARN_ON(d->flags & DEVFL_TKILL);
+	if (d->flags & DEVFL_TKILL) {
+		spin_unlock_irqrestore(&d->lock, flags);
+		goto out_disk_cleanup;
+	}
 	WARN_ON(d->gd);
 	WARN_ON(d->flags & DEVFL_UP);
 	d->bufpool = mp;
@@ -427,7 +430,8 @@ err_mempool:
 err:
 	spin_lock_irqsave(&d->lock, flags);
 	d->flags &= ~DEVFL_GD_NOW;
-	queue_work(aoe_wq, &d->work);
+	if (!(d->flags & DEVFL_TKILL))
+		queue_work(aoe_wq, &d->work);
 	spin_unlock_irqrestore(&d->lock, flags);
 }
 
diff --git a/drivers/block/aoe/aoecmd.c b/drivers/block/aoe/aoecmd.c
index a4744a30a8af..c7f762180b9b 100644
--- a/drivers/block/aoe/aoecmd.c
+++ b/drivers/block/aoe/aoecmd.c
@@ -787,7 +787,8 @@ rexmit_timer(struct timer_list *timer)
 			 */
 			list_splice(&flist, &d->factive[0]);
 			d->flags |= DEVFL_DEAD;
-			queue_work(aoe_wq, &d->work);
+			if (!(d->flags & DEVFL_TKILL))
+				queue_work(aoe_wq, &d->work);
 			goto out;
 		}
 
@@ -899,6 +900,13 @@ aoecmd_sleepwork(struct work_struct *work)
 {
 	struct aoedev *d = container_of(work, struct aoedev, work);
 
+	spin_lock_irq(&d->lock);
+	if (d->flags & DEVFL_TKILL) {
+		spin_unlock_irq(&d->lock);
+		return;
+	}
+	spin_unlock_irq(&d->lock);
+
 	if (d->flags & DEVFL_DEAD)
 		aoedev_downdev(d);
 
@@ -909,7 +917,8 @@ aoecmd_sleepwork(struct work_struct *work)
 		set_capacity_and_notify(d->gd, d->ssize);
 
 		spin_lock_irq(&d->lock);
-		d->flags |= DEVFL_UP;
+		if (!(d->flags & DEVFL_TKILL))
+			d->flags |= DEVFL_UP;
 		d->flags &= ~DEVFL_NEWSIZE;
 		spin_unlock_irq(&d->lock);
 	}
@@ -974,7 +983,7 @@ ataid_complete(struct aoedev *d, struct aoetgt *t, unsigned char *id)
 			d->fw_ver, (long long)ssize);
 	d->ssize = ssize;
 	d->geo.start = 0;
-	if (d->flags & (DEVFL_GDALLOC|DEVFL_NEWSIZE))
+	if (d->flags & (DEVFL_GDALLOC | DEVFL_NEWSIZE | DEVFL_TKILL))
 		return;
 	if (d->gd != NULL)
 		d->flags |= DEVFL_NEWSIZE;
diff --git a/drivers/block/aoe/aoedev.c b/drivers/block/aoe/aoedev.c
index 6e8b3807c1c8..4eb2e68439db 100644
--- a/drivers/block/aoe/aoedev.c
+++ b/drivers/block/aoe/aoedev.c
@@ -286,6 +286,7 @@ freedev(struct aoedev *d)
 		return;
 
 	timer_delete_sync(&d->timer);
+	cancel_work_sync(&d->work);
 	if (d->gd) {
 		aoedisk_rm_debugfs(d);
 		del_gendisk(d->gd);
@@ -311,6 +312,8 @@ enum flush_parms {
 	EXITING = 1,
 };
 
+static DEFINE_MUTEX(flush_mutex);
+
 static int
 flush(const char __user *str, size_t cnt, int exiting)
 {
@@ -333,6 +336,7 @@ flush(const char __user *str, size_t cnt, int exiting)
 			specified = 1;
 	}
 
+	mutex_lock(&flush_mutex);
 	flush_workqueue(aoe_wq);
 	/* pass one: do aoedev_downdev, which might sleep */
 restart1:
@@ -353,10 +357,11 @@ restart1:
 		|| d->ref)
 			goto cont;
 
+		d->flags |= DEVFL_TKILL;
 		spin_unlock(&d->lock);
 		spin_unlock_irqrestore(&devlist_lock, flags);
+		cancel_work_sync(&d->work);
 		aoedev_downdev(d);
-		d->flags |= DEVFL_TKILL;
 		goto restart1;
 cont:
 		spin_unlock(&d->lock);
@@ -397,6 +402,7 @@ restart2:
 		kfree(doomed);
 	}
 	spin_unlock_irqrestore(&devlist_lock, flags);
+	mutex_unlock(&flush_mutex);
 
 	return 0;
 }

[PATCH 19/98] blk-cgroup: don't free a blkg with its punt work queued

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

blkg_async_bio_workfn() relies on the bios on the list pinning the blkg.
But blkcg_punt_bio_submit() can call queue_work() after a running
instance already took and completed the bio, and queue_work() succeeds
on an empty list for the same reason. Nothing holds the blkg then, and
free_work shares its memory with async_bio_work on top of that.

Hold a reference across blkcg_punt_bio_submit(), give the two works
memory of their own, and cancel async_bio_work in blkg_free_workfn().

Fixes: d3f77dfdc718 ("blkcg: implement REQ_CGROUP_PUNT")
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-cgroup.c | 13 ++++++++++++-
 block/blk-cgroup.h |  6 ++----
 2 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index c9187d95a54b..00da55c6ec43 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -121,6 +121,11 @@ static void blkg_free_workfn(struct work_struct *work)
 	struct request_queue *q = blkg->q;
 	int i;
 
+#ifdef CONFIG_BLK_CGROUP_PUNT_BIO
+	/* It may have been queued for bios which an earlier run has taken */
+	cancel_work_sync(&blkg->async_bio_work);
+#endif
+
 	/*
 	 * pd_free_fn() can also be called from blkcg_deactivate_policy(),
 	 * in order to make sure pd_free_fn() is called in order, the deletion
@@ -218,7 +223,7 @@ static void blkg_async_bio_workfn(struct work_struct *work)
 	struct blk_plug plug;
 	bool need_plug = false;
 
-	/* as long as there are pending bios, @blkg can't go away */
+	/* blkg_free_workfn() waits for us */
 	spin_lock(&blkg->async_bio_lock);
 	bio_list_merge_init(&bios, &blkg->async_bios);
 	spin_unlock(&blkg->async_bio_lock);
@@ -245,10 +250,16 @@ void blkcg_punt_bio_submit(struct bio *bio)
 	struct blkcg_gq *blkg = bio->bi_blkg;
 
 	if (blkg->parent) {
+		/*
+		 * Once it is on the list, a run which is under way can submit
+		 * @bio, and its reference goes when it completes.
+		 */
+		blkg_get(blkg);
 		spin_lock(&blkg->async_bio_lock);
 		bio_list_add(&blkg->async_bios, bio);
 		spin_unlock(&blkg->async_bio_lock);
 		queue_work(blkcg_punt_bio_wq, &blkg->async_bio_work);
+		blkg_put(blkg);
 	} else {
 		/* never bounce for the root cgroup */
 		submit_bio(bio);
diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h
index e67c69839129..1fd60cb4157c 100644
--- a/block/blk-cgroup.h
+++ b/block/blk-cgroup.h
@@ -76,11 +76,9 @@ struct blkcg_gq {
 #ifdef CONFIG_BLK_CGROUP_PUNT_BIO
 	spinlock_t			async_bio_lock;
 	struct bio_list			async_bios;
+	struct work_struct		async_bio_work;
 #endif
-	union {
-		struct work_struct	async_bio_work;
-		struct work_struct	free_work;
-	};
+	struct work_struct		free_work;
 
 	atomic_t			use_delay;
 	atomic64_t			delay_nsec;
-- 
2.56.0

Re: [PATCH 19/98] blk-cgroup: don't free a blkg with its punt work queued

[Tejun Heo]

On Wed, Oct 28, 2026 at 10:41:22AM -0600, Jens Axboe wrote:
> blkg_async_bio_workfn() relies on the bios on the list pinning the blkg.
> But blkcg_punt_bio_submit() can call queue_work() after a running
> instance already took and completed the bio, and queue_work() succeeds
> on an empty list for the same reason. Nothing holds the blkg then, and
> free_work shares its memory with async_bio_work on top of that.
>
> Hold a reference across blkcg_punt_bio_submit(), give the two works
> memory of their own, and cancel async_bio_work in blkg_free_workfn().
>
> Fixes: d3f77dfdc718 ("blkcg: implement REQ_CGROUP_PUNT")
> Cc: Tejun Heo <tj@kernel.org>
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> Signed-off-by: Jens Axboe <axboe@kernel.dk>

Acked-by: Tejun Heo <tj@kernel.org>

Thanks.

--
tejun

[PATCH 20/98] mtip32xx: don't mark the disk dead in the irq handler

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

mtip_check_surprise_removal() calls blk_mark_disk_dead() from the
interrupt handler, and that takes bd_holder_lock and calls into the
holder. Only note the removal there and let the service thread tell the
block layer.

Fixes: f55e017c6420 ("block: add a mark_dead holder operation")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/block/mtip32xx/mtip32xx.c | 35 +++++++++++++++++++++++++------
 drivers/block/mtip32xx/mtip32xx.h |  3 +++
 2 files changed, 32 insertions(+), 6 deletions(-)

diff --git a/drivers/block/mtip32xx/mtip32xx.c b/drivers/block/mtip32xx/mtip32xx.c
index 113bdb868c46..58dc0d63df32 100644
--- a/drivers/block/mtip32xx/mtip32xx.c
+++ b/drivers/block/mtip32xx/mtip32xx.c
@@ -130,7 +130,7 @@ struct mtip_compat_ide_task_request_s {
  * return value
  *	 true if device removed, else false
  */
-static bool mtip_check_surprise_removal(struct driver_data *dd)
+static bool mtip_device_removed(struct driver_data *dd)
 {
 	u16 vendor_id = 0;
 
@@ -141,14 +141,34 @@ static bool mtip_check_surprise_removal(struct driver_data *dd)
 	pci_read_config_word(dd->pdev, 0x00, &vendor_id);
 	if (vendor_id == 0xFFFF) {
 		dd->sr = true;
-		if (dd->disk)
-			blk_mark_disk_dead(dd->disk);
 		return true; /* device removed */
 	}
 
 	return false; /* device present */
 }
 
+/* As above, and tell the block layer. May sleep. */
+static bool mtip_check_surprise_removal(struct driver_data *dd)
+{
+	if (!mtip_device_removed(dd))
+		return false;
+
+	if (dd->disk && !test_and_set_bit(MTIP_DDF_DISK_DEAD_BIT, &dd->dd_flag))
+		blk_mark_disk_dead(dd->disk);
+	return true;
+}
+
+/* For the interrupt handler: the service thread tells the block layer. */
+static bool mtip_check_surprise_removal_irq(struct driver_data *dd)
+{
+	if (!mtip_device_removed(dd))
+		return false;
+
+	set_bit(MTIP_PF_SR_BIT, &dd->port->flags);
+	wake_up_interruptible(&dd->port->svc_wait);
+	return true;
+}
+
 static struct mtip_cmd *mtip_cmd_from_tag(struct driver_data *dd,
 					  unsigned int tag)
 {
@@ -715,7 +735,7 @@ static inline void mtip_process_errors(struct driver_data *dd, u32 port_stat)
 		dev_warn(&dd->pdev->dev,
 			"Port stat errors %x unhandled\n",
 			(port_stat & ~PORT_IRQ_HANDLED));
-		if (mtip_check_surprise_removal(dd))
+		if (mtip_check_surprise_removal_irq(dd))
 			return;
 	}
 	if (likely(port_stat & (PORT_IRQ_TF_ERR | PORT_IRQ_IF_ERR))) {
@@ -740,7 +760,7 @@ static inline irqreturn_t mtip_handle_irq(struct driver_data *data)
 		/* Acknowledge the interrupt status on the port.*/
 		port_stat = readl(port->mmio + PORT_IRQ_STAT);
 		if (unlikely(port_stat == 0xFFFFFFFF)) {
-			mtip_check_surprise_removal(dd);
+			mtip_check_surprise_removal_irq(dd);
 			return IRQ_HANDLED;
 		}
 		writel(port_stat, port->mmio + PORT_IRQ_STAT);
@@ -784,7 +804,7 @@ static inline irqreturn_t mtip_handle_irq(struct driver_data *data)
 		}
 
 		if (unlikely(port_stat & PORT_IRQ_ERR)) {
-			if (unlikely(mtip_check_surprise_removal(dd))) {
+			if (unlikely(mtip_check_surprise_removal_irq(dd))) {
 				/* don't proceed further */
 				return IRQ_HANDLED;
 			}
@@ -2492,6 +2512,9 @@ static int mtip_service_thread(void *data)
 
 		set_bit(MTIP_PF_SVC_THD_ACTIVE_BIT, &port->flags);
 
+		if (test_and_clear_bit(MTIP_PF_SR_BIT, &port->flags))
+			mtip_check_surprise_removal(dd);
+
 restart_eh:
 		/* Demux bits: start with error handling */
 		if (test_bit(MTIP_PF_EH_ACTIVE_BIT, &port->flags)) {
diff --git a/drivers/block/mtip32xx/mtip32xx.h b/drivers/block/mtip32xx/mtip32xx.h
index 0963c07b5845..1af2c75414a5 100644
--- a/drivers/block/mtip32xx/mtip32xx.h
+++ b/drivers/block/mtip32xx/mtip32xx.h
@@ -134,11 +134,13 @@ enum {
 	MTIP_PF_ISSUE_CMDS_BIT      = 5,
 	MTIP_PF_REBUILD_BIT         = 6,
 	MTIP_PF_SVC_THD_STOP_BIT    = 8,
+	MTIP_PF_SR_BIT              = 11, /* surprise removal seen in irq */
 
 	MTIP_PF_SVC_THD_WORK	= ((1 << MTIP_PF_EH_ACTIVE_BIT) |
 				  (1 << MTIP_PF_ISSUE_CMDS_BIT) |
 				  (1 << MTIP_PF_REBUILD_BIT) |
 				  (1 << MTIP_PF_SVC_THD_STOP_BIT) |
+				  (1 << MTIP_PF_SR_BIT) |
 				  (1 << MTIP_PF_TO_ACTIVE_BIT)),
 
 	/* below are bit numbers in 'dd_flag' defined in driver_data */
@@ -150,6 +152,7 @@ enum {
 	MTIP_DDF_RESUME_BIT         = 6,
 	MTIP_DDF_INIT_DONE_BIT      = 7,
 	MTIP_DDF_REBUILD_FAILED_BIT = 8,
+	MTIP_DDF_DISK_DEAD_BIT      = 9,
 
 	MTIP_DDF_STOP_IO      = ((1 << MTIP_DDF_REMOVE_PENDING_BIT) |
 				(1 << MTIP_DDF_SEC_LOCK_BIT) |
-- 
2.56.0

Re: [PATCH 20/98] mtip32xx: don't mark the disk dead in the irq handler

[Bart Van Assche]

On 10/28/26 09:41, Jens Axboe wrote:
> @@ -2492,6 +2512,9 @@ static int mtip_service_thread(void *data)
>  
>  		set_bit(MTIP_PF_SVC_THD_ACTIVE_BIT, &port->flags);
>  
> +		if (test_and_clear_bit(MTIP_PF_SR_BIT, &port->flags))
> +			mtip_check_surprise_removal(dd);
> +
>  restart_eh:

Deferring blk_mark_disk_dead() out of the hardirq handler is needed, but
calling mtip_check_surprise_removal(dd) -> blk_mark_disk_dead(dd->disk)
in mtip_service_thread() before aborting in-flight NCQ commands can
deadlock when a filesystem is mounted on the device:

1. blk_mark_disk_dead() invokes fs_bdev_mark_dead(), which locks
   sb->s_umount via bdev_super_lock() and calls sync_filesystem(). If
   another task already holds s_umount while waiting for in-flight I/O
   on the dead PCIe device, mtip_service_thread() blocks on s_umount.
2. In patch 22/98, mtip_pci_remove() then calls
   mtip_stop_service_thread(dd) (kthread_stop()) before calling
   blk_mq_tagset_busy_iter(&dd->tags, mtip_abort_cmd, dd), so
   mtip_pci_remove() blocks waiting for mtip_service_thread() before it
   can abort the in-flight commands holding up s_umount.

Quiescing dd->queue and aborting in-flight commands via
blk_mq_tagset_busy_iter(&dd->tags, mtip_abort_cmd, dd) before calling
blk_mark_disk_dead(dd->disk) on surprise removal would avoid that
deadlock.

Re: [PATCH 20/98] mtip32xx: don't mark the disk dead in the irq handler

[Bart Van Assche]

On 10/28/26 09:41, Jens Axboe wrote:
> @@ -2492,6 +2512,9 @@ static int mtip_service_thread(void *data)
>  
>  		set_bit(MTIP_PF_SVC_THD_ACTIVE_BIT, &port->flags);
>  
> +		if (test_and_clear_bit(MTIP_PF_SR_BIT, &port->flags))
> +			mtip_check_surprise_removal(dd);
> +
>  restart_eh:

Deferring blk_mark_disk_dead() out of the hardirq handler is needed, but
calling blk_mark_disk_dead(dd->disk) in mtip_service_thread() before
aborting in-flight NCQ commands can deadlock when a filesystem is
mounted on the device:

1. blk_mark_disk_dead() invokes fs_bdev_mark_dead(), which takes
   sb->s_umount via bdev_super_lock() and calls sync_filesystem(). If
   another task already holds s_umount while waiting for in-flight I/O
   on the dead PCIe device, mtip_service_thread() blocks on s_umount.
2. In patch 22/98, mtip_pci_remove() calls mtip_stop_service_thread(dd)
   (kthread_stop()) before aborting in-flight commands, so
   mtip_pci_remove() blocks waiting for mtip_service_thread() before it
   can abort the commands holding up s_umount.

Also, mtip_abort_cmd() currently calls mtip_softirq_done_fn(req)
directly without transitioning the request through
blk_mq_complete_request().

How about keeping mtip_check_surprise_removal() non-sleeping everywhere
(setting dd->sr, a new MTIP_DDF_DEAD_BIT to fail new normal and reserved
requests immediately with BLK_STS_IOERR, and waking the service thread
with MTIP_PF_SR_BIT), and having a helper mtip_handle_surprise_removal()
quiesce dd->queue and abort in-flight requests via
blk_mq_tagset_busy_iter() + blk_mq_complete_request() before calling
blk_mark_disk_dead(dd->disk)?

Patch 21/98 still applies cleanly on top of this replacement, and my
reply to patch 22/98 has the matching follow-up change. Checked with
checkpatch.pl and syntax-checked with Clang; not runtime-tested on
mtip32xx hardware:

---
diff --git a/drivers/block/mtip32xx/mtip32xx.c b/drivers/block/mtip32xx/mtip32xx.c
index 113bdb868c46..35568d0e74cf 100644
--- a/drivers/block/mtip32xx/mtip32xx.c
+++ b/drivers/block/mtip32xx/mtip32xx.c
@@ -141,8 +141,11 @@ static bool mtip_check_surprise_removal(struct driver_data *dd)
 	pci_read_config_word(dd->pdev, 0x00, &vendor_id);
 	if (vendor_id == 0xFFFF) {
 		dd->sr = true;
-		if (dd->disk)
-			blk_mark_disk_dead(dd->disk);
+		set_bit(MTIP_DDF_DEAD_BIT, &dd->dd_flag);
+		if (dd->port) {
+			set_bit(MTIP_PF_SR_BIT, &dd->port->flags);
+			wake_up_interruptible(&dd->port->svc_wait);
+		}
 		return true; /* device removed */
 	}
 
@@ -2438,9 +2441,12 @@ static bool mtip_abort_cmd(struct request *req, void *data)
 
 	dbg_printk(MTIP_DRV_NAME " Aborting request, tag = %d\n", req->tag);
 
+	if (blk_mq_request_completed(req))
+		return true;
+
 	clear_bit(req->tag, dd->port->cmds_to_issue);
 	cmd->status = BLK_STS_IOERR;
-	mtip_softirq_done_fn(req);
+	blk_mq_complete_request(req);
 	return true;
 }
 
@@ -2462,6 +2468,18 @@ static bool mtip_queue_cmd(struct request *req, void *data)
  *	0
  */
 
+static void mtip_handle_surprise_removal(struct driver_data *dd)
+{
+	if (dd->queue) {
+		blk_mq_quiesce_queue(dd->queue);
+		blk_mq_tagset_busy_iter(&dd->tags, mtip_abort_cmd, dd);
+		blk_mq_unquiesce_queue(dd->queue);
+	}
+	if (test_bit(MTIP_DDF_INIT_DONE_BIT, &dd->dd_flag) &&
+	    !test_and_set_bit(MTIP_DDF_DISK_DEAD_BIT, &dd->dd_flag))
+		blk_mark_disk_dead(dd->disk);
+}
+
 static int mtip_service_thread(void *data)
 {
 	struct driver_data *dd = (struct driver_data *)data;
@@ -2492,6 +2510,15 @@ static int mtip_service_thread(void *data)
 
 		set_bit(MTIP_PF_SVC_THD_ACTIVE_BIT, &port->flags);
 
+		if (test_and_clear_bit(MTIP_PF_SR_BIT, &port->flags)) {
+			mtip_handle_surprise_removal(dd);
+			clear_bit(MTIP_PF_EH_ACTIVE_BIT, &port->flags);
+			clear_bit(MTIP_PF_TO_ACTIVE_BIT, &port->flags);
+			clear_bit(MTIP_PF_ISSUE_CMDS_BIT, &port->flags);
+			clear_bit(MTIP_PF_REBUILD_BIT, &port->flags);
+			continue;
+		}
+
 restart_eh:
 		/* Demux bits: start with error handling */
 		if (test_bit(MTIP_PF_EH_ACTIVE_BIT, &port->flags)) {
@@ -3241,6 +3268,8 @@ static inline bool is_stopped(struct driver_data *dd, struct request *rq)
 		return true;
 	if (test_bit(MTIP_DDF_REBUILD_FAILED_BIT, &dd->dd_flag))
 		return true;
+	if (test_bit(MTIP_DDF_DEAD_BIT, &dd->dd_flag))
+		return true;
 
 	return false;
 }
@@ -3279,6 +3308,9 @@ static blk_status_t mtip_issue_reserved_cmd(struct blk_mq_hw_ctx *hctx,
 		dd->port->command_list + sizeof(struct mtip_cmd_hdr) * rq->tag;
 	struct mtip_cmd_sg *command_sg;
 
+	if (test_bit(MTIP_DDF_DEAD_BIT, &dd->dd_flag))
+		return BLK_STS_IOERR;
+
 	if (mtip_commands_active(dd->port))
 		return BLK_STS_DEV_RESOURCE;
 
diff --git a/drivers/block/mtip32xx/mtip32xx.h b/drivers/block/mtip32xx/mtip32xx.h
index 0963c07b5845..af0249e3c550 100644
--- a/drivers/block/mtip32xx/mtip32xx.h
+++ b/drivers/block/mtip32xx/mtip32xx.h
@@ -134,11 +134,13 @@ enum {
 	MTIP_PF_ISSUE_CMDS_BIT      = 5,
 	MTIP_PF_REBUILD_BIT         = 6,
 	MTIP_PF_SVC_THD_STOP_BIT    = 8,
+	MTIP_PF_SR_BIT              = 11, /* surprise removal seen */
 
 	MTIP_PF_SVC_THD_WORK	= ((1 << MTIP_PF_EH_ACTIVE_BIT) |
 				  (1 << MTIP_PF_ISSUE_CMDS_BIT) |
 				  (1 << MTIP_PF_REBUILD_BIT) |
 				  (1 << MTIP_PF_SVC_THD_STOP_BIT) |
+				  (1 << MTIP_PF_SR_BIT) |
 				  (1 << MTIP_PF_TO_ACTIVE_BIT)),
 
 	/* below are bit numbers in 'dd_flag' defined in driver_data */
@@ -150,12 +152,15 @@ enum {
 	MTIP_DDF_RESUME_BIT         = 6,
 	MTIP_DDF_INIT_DONE_BIT      = 7,
 	MTIP_DDF_REBUILD_FAILED_BIT = 8,
+	MTIP_DDF_DEAD_BIT           = 9,
+	MTIP_DDF_DISK_DEAD_BIT      = 10,
 
 	MTIP_DDF_STOP_IO      = ((1 << MTIP_DDF_REMOVE_PENDING_BIT) |
 				(1 << MTIP_DDF_SEC_LOCK_BIT) |
 				(1 << MTIP_DDF_OVER_TEMP_BIT) |
 				(1 << MTIP_DDF_WRITE_PROTECT_BIT) |
-				(1 << MTIP_DDF_REBUILD_FAILED_BIT)),
+				(1 << MTIP_DDF_REBUILD_FAILED_BIT) |
+				(1 << MTIP_DDF_DEAD_BIT)),
 
 };
 

[PATCH 21/98] mtip32xx: abort timed out commands of a removed device

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

mtip_device_reset() returns 0 for a removed device, so the timeout
handler takes that for a successful reset and reissues the commands to
registers that are no longer there, forever. Return -ENODEV so they get
aborted instead.

Fixes: abb0ccd185c9 ("mtip32xx: Implement timeout handler")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/block/mtip32xx/mtip32xx.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/block/mtip32xx/mtip32xx.c b/drivers/block/mtip32xx/mtip32xx.c
index 58dc0d63df32..ee5b170e767c 100644
--- a/drivers/block/mtip32xx/mtip32xx.c
+++ b/drivers/block/mtip32xx/mtip32xx.c
@@ -456,7 +456,7 @@ static int mtip_device_reset(struct driver_data *dd)
 	int rv = 0;
 
 	if (mtip_check_surprise_removal(dd))
-		return 0;
+		return -ENODEV;
 
 	if (mtip_hba_reset(dd) < 0)
 		rv = -EFAULT;
-- 
2.56.0

[PATCH 22/98] mtip32xx: abort commands in flight on surprise removal

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

mtip_pci_remove() relies on blk_mark_disk_dead(), which only stops new
IO. del_gendisk() waits for the requests the driver still owns, and
nothing completes them for a removed device: the timeout handler leaves
them to the service thread, which exits as soon as it sees
MTIP_DDF_REMOVE_PENDING_BIT. The removal hangs.

Stop the service thread and abort the commands in flight before
del_gendisk() if the device is gone.

Fixes: e8b58ef09e84 ("mtip32xx: fix device removal")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/block/mtip32xx/mtip32xx.c | 28 +++++++++++++++++++++++-----
 1 file changed, 23 insertions(+), 5 deletions(-)

diff --git a/drivers/block/mtip32xx/mtip32xx.c b/drivers/block/mtip32xx/mtip32xx.c
index ee5b170e767c..301c3db21ca8 100644
--- a/drivers/block/mtip32xx/mtip32xx.c
+++ b/drivers/block/mtip32xx/mtip32xx.c
@@ -3887,6 +3887,17 @@ static int mtip_pci_probe(struct pci_dev *pdev,
 	return rv;
 }
 
+static void mtip_stop_service_thread(struct driver_data *dd)
+{
+	if (!dd->mtip_svc_handler)
+		return;
+
+	set_bit(MTIP_PF_SVC_THD_STOP_BIT, &dd->port->flags);
+	wake_up_interruptible(&dd->port->svc_wait);
+	kthread_stop(dd->mtip_svc_handler);
+	dd->mtip_svc_handler = NULL;
+}
+
 /*
  * Called for each probed device when the device is removed or the
  * driver is unloaded.
@@ -3917,16 +3928,23 @@ static void mtip_pci_remove(struct pci_dev *pdev)
 	set_bit(MTIP_DDF_REMOVE_PENDING_BIT, &dd->dd_flag);
 	mutex_lock(&dd->ioctl_mutex);
 
+	/*
+	 * Nobody will complete the commands a removed device has been given,
+	 * and del_gendisk() waits for them.
+	 */
+	if (dd->sr) {
+		mtip_stop_service_thread(dd);
+		blk_mq_quiesce_queue(dd->queue);
+		blk_mq_tagset_busy_iter(&dd->tags, mtip_abort_cmd, dd);
+		blk_mq_unquiesce_queue(dd->queue);
+	}
+
 	if (test_bit(MTIP_DDF_INIT_DONE_BIT, &dd->dd_flag))
 		del_gendisk(dd->disk);
 
 	mtip_hw_debugfs_exit(dd);
 
-	if (dd->mtip_svc_handler) {
-		set_bit(MTIP_PF_SVC_THD_STOP_BIT, &dd->port->flags);
-		wake_up_interruptible(&dd->port->svc_wait);
-		kthread_stop(dd->mtip_svc_handler);
-	}
+	mtip_stop_service_thread(dd);
 
 	if (!dd->sr) {
 		/*
-- 
2.56.0

Re: [PATCH 22/98] mtip32xx: abort commands in flight on surprise removal

[Bart Van Assche]

On 10/28/26 09:41, Jens Axboe wrote:
> +	/*
> +	 * Nobody will complete the commands a removed device has been given,
> +	 * and del_gendisk() waits for them.
> +	 */
> +	if (dd->sr) {
> +		mtip_stop_service_thread(dd);
> +		blk_mq_quiesce_queue(dd->queue);
> +		blk_mq_tagset_busy_iter(&dd->tags, mtip_abort_cmd, dd);
> +		blk_mq_unquiesce_queue(dd->queue);
> +	}
> +
>  	if (test_bit(MTIP_DDF_INIT_DONE_BIT, &dd->dd_flag))
>  		del_gendisk(dd->disk);

Two issues with this ordering:

1. Calling mtip_stop_service_thread(dd) (kthread_stop()) before aborting
   in-flight commands can deadlock with patch 20/98 if
   mtip_service_thread() is blocked inside blk_mark_disk_dead() ->
   fs_bdev_mark_dead() waiting for sb->s_umount while the s_umount
   holder is waiting on those uncompleted NCQ commands.
2. If PCIe link loss occurs after mtip_pci_remove() has already checked
   if (dd->sr) (false) and entered del_gendisk(dd->disk),
   MTIP_DDF_REMOVE_PENDING_BIT is already set. mtip_service_thread()
   then exits immediately upon seeing MTIP_DDF_REMOVE_PENDING_BIT
   without handling MTIP_PF_SR_BIT or MTIP_PF_TO_ACTIVE_BIT, leaving
   in-flight commands uncompleted until the request timer fires.

If mtip_service_thread() aborts in-flight commands when handling
MTIP_PF_SR_BIT (before blk_mark_disk_dead()) and does not exit early on
MTIP_DDF_REMOVE_PENDING_BIT while surprise-removal/timeout work is
pending, mtip_stop_service_thread() can stay after del_gendisk() and
both races are closed.

Re: [PATCH 22/98] mtip32xx: abort commands in flight on surprise removal

[Bart Van Assche]

On 10/28/26 09:41, Jens Axboe wrote:
> +	/*
> +	 * Nobody will complete the commands a removed device has been given,
> +	 * and del_gendisk() waits for them.
> +	 */
> +	if (dd->sr) {
> +		mtip_stop_service_thread(dd);
> +		blk_mq_quiesce_queue(dd->queue);
> +		blk_mq_tagset_busy_iter(&dd->tags, mtip_abort_cmd, dd);
> +		blk_mq_unquiesce_queue(dd->queue);
> +	}
> +
>  	if (test_bit(MTIP_DDF_INIT_DONE_BIT, &dd->dd_flag))
>  		del_gendisk(dd->disk);

Two issues with this ordering:

1. Calling mtip_stop_service_thread(dd) (kthread_stop()) before aborting
   in-flight commands deadlocks with patch 20/98 if
   mtip_service_thread() is blocked inside blk_mark_disk_dead() ->
   fs_bdev_mark_dead() waiting for sb->s_umount while the s_umount
   holder is waiting on those uncompleted NCQ commands.
2. If PCIe link loss happens after mtip_pci_remove() has already checked
   dd->sr (false) and entered del_gendisk(dd->disk),
   MTIP_DDF_REMOVE_PENDING_BIT is already set. mtip_service_thread()
   exits immediately upon seeing MTIP_DDF_REMOVE_PENDING_BIT without
   handling MTIP_PF_SR_BIT or MTIP_PF_TO_ACTIVE_BIT, leaving in-flight
   commands uncompleted forever in del_gendisk().

With the replacement proposed for patch 20/98 (where
mtip_handle_surprise_removal() aborts in-flight commands before calling
blk_mark_disk_dead()), we can instead keep mtip_service_thread() running
until kthread_stop() after del_gendisk(dd->disk) by removing its early
exit on MTIP_DDF_REMOVE_PENDING_BIT (and clearing MTIP_PF_REBUILD_BIT if
remove is pending or mtip_ftl_rebuild_poll(dd) <= 0, since rebuild poll
returns -EFAULT once MTIP_DDF_REMOVE_PENDING_BIT is set). That handles
surprise removal both before and during del_gendisk() from the service
thread without racing two tagset iterations.

Below is the replacement for 22/98 on top of the 20/98 replacement +
unmodified 21/98 (checked with checkpatch.pl and syntax-checked with
Clang; not runtime-tested on mtip32xx hardware):

---
diff --git a/drivers/block/mtip32xx/mtip32xx.c b/drivers/block/mtip32xx/mtip32xx.c
index 757f9b74871e..b9dfa7112c95 100644
--- a/drivers/block/mtip32xx/mtip32xx.c
+++ b/drivers/block/mtip32xx/mtip32xx.c
@@ -2504,10 +2504,6 @@ static int mtip_service_thread(void *data)
 			test_bit(MTIP_PF_SVC_THD_STOP_BIT, &port->flags))
 			goto st_out;
 
-		if (unlikely(test_bit(MTIP_DDF_REMOVE_PENDING_BIT,
-				&dd->dd_flag)))
-			goto st_out;
-
 		set_bit(MTIP_PF_SVC_THD_ACTIVE_BIT, &port->flags);
 
 		if (test_and_clear_bit(MTIP_PF_SR_BIT, &port->flags)) {
@@ -2588,7 +2584,9 @@ restart_eh:
 		}
 
 		if (test_bit(MTIP_PF_REBUILD_BIT, &port->flags)) {
-			if (mtip_ftl_rebuild_poll(dd) == 0)
+			if (test_bit(MTIP_DDF_REMOVE_PENDING_BIT,
+				     &dd->dd_flag) ||
+			    mtip_ftl_rebuild_poll(dd) <= 0)
 				clear_bit(MTIP_PF_REBUILD_BIT, &port->flags);
 		}
 	}
@@ -3926,6 +3924,9 @@ static void mtip_pci_remove(struct pci_dev *pdev)
 	set_bit(MTIP_DDF_REMOVE_PENDING_BIT, &dd->dd_flag);
 	mutex_lock(&dd->ioctl_mutex);
 
+	if (dd->sr && !dd->mtip_svc_handler)
+		mtip_handle_surprise_removal(dd);
+
 	if (test_bit(MTIP_DDF_INIT_DONE_BIT, &dd->dd_flag))
 		del_gendisk(dd->disk);
 

[PATCH 23/98] blk-iolatency: flush enable_work after going offline

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

blkcg_iolatency_exit() flushes enable_work before it deactivates the
policy, but iolatency_pd_offline() schedules it again for the last group
with a target, so the blk_iolatency is freed with the work queued. Flush
it after the deactivation.

Fixes: 8a177a36da6c ("blk-iolatency: Fix inflight count imbalances and IO hangs on offline")
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-iolatency.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c
index 2caa79a008ad..c15b1aba606c 100644
--- a/block/blk-iolatency.c
+++ b/block/blk-iolatency.c
@@ -637,8 +637,9 @@ static void blkcg_iolatency_exit(struct rq_qos *rqos)
 	struct blk_iolatency *blkiolat = BLKIOLATENCY(rqos);
 
 	timer_shutdown_sync(&blkiolat->timer);
-	flush_work(&blkiolat->enable_work);
 	blkcg_deactivate_policy(rqos->disk, &blkcg_policy_iolatency);
+	/* iolatency_pd_offline() schedules it for the last group with a target */
+	flush_work(&blkiolat->enable_work);
 	kfree(blkiolat);
 }
 
-- 
2.56.0

Re: [PATCH 23/98] blk-iolatency: flush enable_work after going offline

[Tejun Heo]

On Wed, Oct 28, 2026 at 10:41:26AM -0600, Jens Axboe wrote:
> blkcg_iolatency_exit() flushes enable_work before it deactivates the
> policy, but iolatency_pd_offline() schedules it again for the last group
> with a target, so the blk_iolatency is freed with the work queued. Flush
> it after the deactivation.
>
> Fixes: 8a177a36da6c ("blk-iolatency: Fix inflight count imbalances and IO hangs on offline")
> Cc: Tejun Heo <tj@kernel.org>
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> Signed-off-by: Jens Axboe <axboe@kernel.dk>

Acked-by: Tejun Heo <tj@kernel.org>

Thanks.

--
tejun

[PATCH 24/98] zram: accept lz4 dictionaries of more than 64 KiB

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

LZ4_loadDict() and LZ4_loadDictHC() only use the last 64KiB of a
dictionary and return how much they used, which the zram backends take
for an error. Expect that instead.

Fixes: fb4f644ee8da ("zram: add dictionary support to lz4")
Fixes: 1e673c8cf7f9 ("zram: add dictionary support to lz4hc")
Cc: Minchan Kim <minchan@kernel.org>
Cc: Sergey Senozhatsky <senozhatsky@chromium.org>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/block/zram/backend_lz4.c   | 4 +++-
 drivers/block/zram/backend_lz4hc.c | 4 +++-
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/block/zram/backend_lz4.c b/drivers/block/zram/backend_lz4.c
index 1e4ad31d39a6..15db3292e19e 100644
--- a/drivers/block/zram/backend_lz4.c
+++ b/drivers/block/zram/backend_lz4.c
@@ -4,6 +4,7 @@
 
 #include <linux/kernel.h>
 #include <linux/lz4.h>
+#include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 
@@ -48,7 +49,8 @@ static int lz4_setup_params(struct zcomp_params *params)
 
 	ret = LZ4_loadDict(dict_stream,
 			   params->dict, params->dict_sz);
-	if (ret != params->dict_sz) {
+	/* Only the last 64 KiB of a dictionary are of use */
+	if (ret != min_t(size_t, params->dict_sz, SZ_64K)) {
 		kfree(dict_stream);
 		return -EINVAL;
 	}
diff --git a/drivers/block/zram/backend_lz4hc.c b/drivers/block/zram/backend_lz4hc.c
index d8aa01bb258f..d163ff0ae8ae 100644
--- a/drivers/block/zram/backend_lz4hc.c
+++ b/drivers/block/zram/backend_lz4hc.c
@@ -4,6 +4,7 @@
 
 #include <linux/kernel.h>
 #include <linux/lz4.h>
+#include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/vmalloc.h>
 
@@ -96,7 +97,8 @@ static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 		LZ4_resetStreamHC(zctx->cstrm, params->level);
 		ret = LZ4_loadDictHC(zctx->cstrm, params->dict,
 				     params->dict_sz);
-		if (ret != params->dict_sz)
+		/* Only the last 64 KiB of a dictionary are of use */
+		if (ret != min_t(size_t, params->dict_sz, SZ_64K))
 			return -EINVAL;
 		ret = LZ4_compress_HC_continue(zctx->cstrm, req->src, req->dst,
 					       req->src_len, req->dst_len);
-- 
2.56.0

Re: [PATCH 24/98] zram: accept lz4 dictionaries of more than 64 KiB

[Sergey Senozhatsky]

On (26/10/28 10:41), Jens Axboe wrote:
[..]
>  #include <linux/kernel.h>
>  #include <linux/lz4.h>
> +#include <linux/sizes.h>
>  #include <linux/slab.h>
>  #include <linux/vmalloc.h>
>  
> @@ -48,7 +49,8 @@ static int lz4_setup_params(struct zcomp_params *params)
>  
>  	ret = LZ4_loadDict(dict_stream,
>  			   params->dict, params->dict_sz);
> -	if (ret != params->dict_sz) {
> +	/* Only the last 64 KiB of a dictionary are of use */
> +	if (ret != min_t(size_t, params->dict_sz, SZ_64K)) {
>  		kfree(dict_stream);
>  		return -EINVAL;
>  	}
> diff --git a/drivers/block/zram/backend_lz4hc.c b/drivers/block/zram/backend_lz4hc.c
> index d8aa01bb258f..d163ff0ae8ae 100644
> --- a/drivers/block/zram/backend_lz4hc.c
> +++ b/drivers/block/zram/backend_lz4hc.c
> @@ -4,6 +4,7 @@
>  
>  #include <linux/kernel.h>
>  #include <linux/lz4.h>
> +#include <linux/sizes.h>
>  #include <linux/slab.h>
>  #include <linux/vmalloc.h>
>  
> @@ -96,7 +97,8 @@ static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,
>  		LZ4_resetStreamHC(zctx->cstrm, params->level);
>  		ret = LZ4_loadDictHC(zctx->cstrm, params->dict,
>  				     params->dict_sz);
> -		if (ret != params->dict_sz)
> +		/* Only the last 64 KiB of a dictionary are of use */
> +		if (ret != min_t(size_t, params->dict_sz, SZ_64K))
>  			return -EINVAL;

I think this should patch lz4hc_setup_params() instead.
Similarly to lz4 hunk.

Re: [PATCH 24/98] zram: accept lz4 dictionaries of more than 64 KiB

[Jens Axboe]

On 10/28/26 7:20 PM, Sergey Senozhatsky wrote:
> On (26/10/28 10:41), Jens Axboe wrote:
> [..]
>>  #include <linux/kernel.h>
>>  #include <linux/lz4.h>
>> +#include <linux/sizes.h>
>>  #include <linux/slab.h>
>>  #include <linux/vmalloc.h>
>>  
>> @@ -48,7 +49,8 @@ static int lz4_setup_params(struct zcomp_params *params)
>>  
>>  	ret = LZ4_loadDict(dict_stream,
>>  			   params->dict, params->dict_sz);
>> -	if (ret != params->dict_sz) {
>> +	/* Only the last 64 KiB of a dictionary are of use */
>> +	if (ret != min_t(size_t, params->dict_sz, SZ_64K)) {
>>  		kfree(dict_stream);
>>  		return -EINVAL;
>>  	}
>> diff --git a/drivers/block/zram/backend_lz4hc.c b/drivers/block/zram/backend_lz4hc.c
>> index d8aa01bb258f..d163ff0ae8ae 100644
>> --- a/drivers/block/zram/backend_lz4hc.c
>> +++ b/drivers/block/zram/backend_lz4hc.c
>> @@ -4,6 +4,7 @@
>>  
>>  #include <linux/kernel.h>
>>  #include <linux/lz4.h>
>> +#include <linux/sizes.h>
>>  #include <linux/slab.h>
>>  #include <linux/vmalloc.h>
>>  
>> @@ -96,7 +97,8 @@ static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,
>>  		LZ4_resetStreamHC(zctx->cstrm, params->level);
>>  		ret = LZ4_loadDictHC(zctx->cstrm, params->dict,
>>  				     params->dict_sz);
>> -		if (ret != params->dict_sz)
>> +		/* Only the last 64 KiB of a dictionary are of use */
>> +		if (ret != min_t(size_t, params->dict_sz, SZ_64K))
>>  			return -EINVAL;
> 
> I think this should patch lz4hc_setup_params() instead.
> Similarly to lz4 hunk.

You're the maintainer, so queue up a patch like that. Like I told
Bart, I'm not sending this out to get review. Either ack it if it's
good, and if not, as the maintainer of this driver, just fix it and
I'll happily drop this one.

-- 
Jens Axboe

Re: [PATCH 24/98] zram: accept lz4 dictionaries of more than 64 KiB

[Sergey Senozhatsky]

On (26/10/28 19:54), Jens Axboe wrote:
> On 10/28/26 7:20 PM, Sergey Senozhatsky wrote:
> > On (26/10/28 10:41), Jens Axboe wrote:
> > [..]
> >>  #include <linux/kernel.h>
> >>  #include <linux/lz4.h>
> >> +#include <linux/sizes.h>
> >>  #include <linux/slab.h>
> >>  #include <linux/vmalloc.h>
> >>  
> >> @@ -48,7 +49,8 @@ static int lz4_setup_params(struct zcomp_params *params)
> >>  
> >>  	ret = LZ4_loadDict(dict_stream,
> >>  			   params->dict, params->dict_sz);
> >> -	if (ret != params->dict_sz) {
> >> +	/* Only the last 64 KiB of a dictionary are of use */
> >> +	if (ret != min_t(size_t, params->dict_sz, SZ_64K)) {
> >>  		kfree(dict_stream);
> >>  		return -EINVAL;
> >>  	}
> >> diff --git a/drivers/block/zram/backend_lz4hc.c b/drivers/block/zram/backend_lz4hc.c
> >> index d8aa01bb258f..d163ff0ae8ae 100644
> >> --- a/drivers/block/zram/backend_lz4hc.c
> >> +++ b/drivers/block/zram/backend_lz4hc.c
> >> @@ -4,6 +4,7 @@
> >>  
> >>  #include <linux/kernel.h>
> >>  #include <linux/lz4.h>
> >> +#include <linux/sizes.h>
> >>  #include <linux/slab.h>
> >>  #include <linux/vmalloc.h>
> >>  
> >> @@ -96,7 +97,8 @@ static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,
> >>  		LZ4_resetStreamHC(zctx->cstrm, params->level);
> >>  		ret = LZ4_loadDictHC(zctx->cstrm, params->dict,
> >>  				     params->dict_sz);
> >> -		if (ret != params->dict_sz)
> >> +		/* Only the last 64 KiB of a dictionary are of use */
> >> +		if (ret != min_t(size_t, params->dict_sz, SZ_64K))
> >>  			return -EINVAL;
> > 
> > I think this should patch lz4hc_setup_params() instead.
> > Similarly to lz4 hunk.
> 
> You're the maintainer, so queue up a patch like that. Like I told
> Bart, I'm not sending this out to get review. Either ack it if it's
> good, and if not, as the maintainer of this driver, just fix it and
> I'll happily drop this one.

Works for me, I'll queue up a fix (will give you credit).

Re: [PATCH 24/98] zram: accept lz4 dictionaries of more than 64 KiB

[Jens Axboe]

On 10/28/26 7:55 PM, Sergey Senozhatsky wrote:
> On (26/10/28 19:54), Jens Axboe wrote:
>> On 10/28/26 7:20 PM, Sergey Senozhatsky wrote:
>>> On (26/10/28 10:41), Jens Axboe wrote:
>>> [..]
>>>>  #include <linux/kernel.h>
>>>>  #include <linux/lz4.h>
>>>> +#include <linux/sizes.h>
>>>>  #include <linux/slab.h>
>>>>  #include <linux/vmalloc.h>
>>>>  
>>>> @@ -48,7 +49,8 @@ static int lz4_setup_params(struct zcomp_params *params)
>>>>  
>>>>  	ret = LZ4_loadDict(dict_stream,
>>>>  			   params->dict, params->dict_sz);
>>>> -	if (ret != params->dict_sz) {
>>>> +	/* Only the last 64 KiB of a dictionary are of use */
>>>> +	if (ret != min_t(size_t, params->dict_sz, SZ_64K)) {
>>>>  		kfree(dict_stream);
>>>>  		return -EINVAL;
>>>>  	}
>>>> diff --git a/drivers/block/zram/backend_lz4hc.c b/drivers/block/zram/backend_lz4hc.c
>>>> index d8aa01bb258f..d163ff0ae8ae 100644
>>>> --- a/drivers/block/zram/backend_lz4hc.c
>>>> +++ b/drivers/block/zram/backend_lz4hc.c
>>>> @@ -4,6 +4,7 @@
>>>>  
>>>>  #include <linux/kernel.h>
>>>>  #include <linux/lz4.h>
>>>> +#include <linux/sizes.h>
>>>>  #include <linux/slab.h>
>>>>  #include <linux/vmalloc.h>
>>>>  
>>>> @@ -96,7 +97,8 @@ static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,
>>>>  		LZ4_resetStreamHC(zctx->cstrm, params->level);
>>>>  		ret = LZ4_loadDictHC(zctx->cstrm, params->dict,
>>>>  				     params->dict_sz);
>>>> -		if (ret != params->dict_sz)
>>>> +		/* Only the last 64 KiB of a dictionary are of use */
>>>> +		if (ret != min_t(size_t, params->dict_sz, SZ_64K))
>>>>  			return -EINVAL;
>>>
>>> I think this should patch lz4hc_setup_params() instead.
>>> Similarly to lz4 hunk.
>>
>> You're the maintainer, so queue up a patch like that. Like I told
>> Bart, I'm not sending this out to get review. Either ack it if it's
>> good, and if not, as the maintainer of this driver, just fix it and
>> I'll happily drop this one.
> 
> Works for me, I'll queue up a fix (will give you credit).

Perfect, thank you! I'll drop this one.

-- 
Jens Axboe

[PATCH 25/98] blk-throttle: pass the I/O counts of a dead group on

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

The recursive io_service_bytes and io_serviced files sum up the online
groups plus what dead descendants left in aux_cnt. That worked while the
counters lived in the blkg. Since commit 7ca464383aec they're in the
throtl_grp and nobody passes them up, so the files go backwards when a
child is removed. Add them to the parent in throtl_pd_offline().

Fixes: 7ca464383aec ("blk-throtl: stop using blkg->stat_bytes and ->stat_ios")
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-throttle.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index 4f10fe05744b..80a5e4b2872f 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -1670,7 +1670,18 @@ static void tg_flush_bios(struct throtl_grp *tg)
 
 static void throtl_pd_offline(struct blkg_policy_data *pd)
 {
-	tg_flush_bios(pd_to_tg(pd));
+	struct throtl_grp *tg = pd_to_tg(pd);
+	struct throtl_grp *parent = blkg_to_tg(tg_to_blkg(tg)->parent);
+
+	tg_flush_bios(tg);
+
+	/* the recursive files of the ancestors go on counting what it did */
+	if (parent) {
+		blkg_rwstat_add_aux(&parent->stat_bytes, &tg->stat_bytes);
+		blkg_rwstat_add_aux(&parent->stat_ios, &tg->stat_ios);
+		blkg_rwstat_reset(&tg->stat_bytes);
+		blkg_rwstat_reset(&tg->stat_ios);
+	}
 }
 
 struct blkcg_policy blkcg_policy_throtl = {
-- 
2.56.0

Re: [PATCH 25/98] blk-throttle: pass the I/O counts of a dead group on

[Tejun Heo]

On Wed, Oct 28, 2026 at 10:41:28AM -0600, Jens Axboe wrote:
> The recursive io_service_bytes and io_serviced files sum up the online
> groups plus what dead descendants left in aux_cnt. That worked while the
> counters lived in the blkg. Since commit 7ca464383aec they're in the
> throtl_grp and nobody passes them up, so the files go backwards when a
> child is removed. Add them to the parent in throtl_pd_offline().
>
> Fixes: 7ca464383aec ("blk-throtl: stop using blkg->stat_bytes and ->stat_ios")
> Cc: Tejun Heo <tj@kernel.org>
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> Signed-off-by: Jens Axboe <axboe@kernel.dk>

Acked-by: Tejun Heo <tj@kernel.org>

Thanks.

--
tejun

[PATCH 26/98] blk-throttle: make blkio.reset_stats reset I/O counts

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

blkio.throttle.io_service_bytes and io_serviced were reset by
blkcg_reset_stats() while the counters were part of the blkg. Now that
they live in the throtl_grp, add a pd_reset_stats_fn for them.

Fixes: 7ca464383aec ("blk-throtl: stop using blkg->stat_bytes and ->stat_ios")
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-throttle.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index 80a5e4b2872f..c46807a0faab 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -1684,6 +1684,14 @@ static void throtl_pd_offline(struct blkg_policy_data *pd)
 	}
 }
 
+static void throtl_pd_reset_stats(struct blkg_policy_data *pd)
+{
+	struct throtl_grp *tg = pd_to_tg(pd);
+
+	blkg_rwstat_reset(&tg->stat_bytes);
+	blkg_rwstat_reset(&tg->stat_ios);
+}
+
 struct blkcg_policy blkcg_policy_throtl = {
 	.dfl_cftypes		= throtl_files,
 	.legacy_cftypes		= throtl_legacy_files,
@@ -1693,6 +1701,7 @@ struct blkcg_policy blkcg_policy_throtl = {
 	.pd_online_fn		= throtl_pd_online,
 	.pd_offline_fn		= throtl_pd_offline,
 	.pd_free_fn		= throtl_pd_free,
+	.pd_reset_stats_fn	= throtl_pd_reset_stats,
 };
 
 static void tg_cancel_writeback_bios(struct throtl_grp *tg,
-- 
2.56.0

Re: [PATCH 26/98] blk-throttle: make blkio.reset_stats reset I/O counts

[Tejun Heo]

On Wed, Oct 28, 2026 at 10:41:29AM -0600, Jens Axboe wrote:
> blkio.throttle.io_service_bytes and io_serviced were reset by
> blkcg_reset_stats() while the counters were part of the blkg. Now that
> they live in the throtl_grp, add a pd_reset_stats_fn for them.
>
> Fixes: 7ca464383aec ("blk-throtl: stop using blkg->stat_bytes and ->stat_ios")
> Cc: Tejun Heo <tj@kernel.org>
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> Signed-off-by: Jens Axboe <axboe@kernel.dk>

Acked-by: Tejun Heo <tj@kernel.org>

Thanks.

--
tejun

[PATCH 27/98] blk-mq: don't trust the cookie in blk_mq_poll()

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

blk_mq_poll() indexes queue_hw_ctx[] with bio->bi_cookie unchecked. Zone
write plugging keeps the segment count of a plugged write in
__bi_nr_segments, which shares a union with the cookie, and clears
REQ_POLLED, but bio_poll() only looks at the cookie. A plugged polled
write with more segments than there are hardware queues reads past the
array. The bio is also SLAB_TYPESAFE_BY_RCU, so a cookie from a bio
reused for another device can meet the wrong array as well.

Ignore a cookie that is out of range or doesn't name a poll queue.

Fixes: 3e08773c3841 ("block: switch polling to be bio based")
Fixes: dd291d77cc90 ("block: Introduce zone write plugging")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-mq.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index c91306f8156f..5ebe2e23df5c 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -5245,9 +5245,22 @@ static int blk_hctx_poll(struct request_queue *q, struct blk_mq_hw_ctx *hctx,
 int blk_mq_poll(struct request_queue *q, blk_qc_t cookie,
 		struct io_comp_batch *iob, unsigned int flags)
 {
+	struct blk_mq_hw_ctx *hctx;
+
 	if (!blk_mq_can_poll(q))
 		return 0;
-	return blk_hctx_poll(q, q->queue_hw_ctx[cookie], iob, flags);
+
+	/*
+	 * The cookie is no more than a hint. iocb_bio_iopoll() may have read it
+	 * from a bio that was reused for another queue right afterwards, and
+	 * zone write plugging keeps the number of segments in the same field.
+	 */
+	if (cookie >= q->nr_hw_queues)
+		return 0;
+	hctx = q->queue_hw_ctx[cookie];
+	if (hctx->type != HCTX_TYPE_POLL)
+		return 0;
+	return blk_hctx_poll(q, hctx, iob, flags);
 }
 
 int blk_rq_poll(struct request *rq, struct io_comp_batch *iob,
-- 
2.56.0

[PATCH 28/98] blk-iocost: check again that the disk is alive

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

blkg_conf_open_bdev() checks that the disk is alive under rq_qos_mutex,
but ioc_qos_write() drops the mutex to freeze the queue first. If
del_gendisk() runs in between, the tag set may be gone by the time
blk_mq_quiesce_queue() looks at it. Check disk_live() again after
retaking the mutex.

Fixes: 9730763f4756 ("block: correct locking order for protecting blk-wbt parameters")
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-iocost.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index 7b5d865d8c14..bfb0dd320c4a 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -3322,6 +3322,11 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
 
 	body = ctx.body;
 	disk = ctx.bdev->bd_disk;
+	/* It may have gone while the mutex was not held */
+	if (!disk_live(disk)) {
+		ret = -ENODEV;
+		goto close_bdev;
+	}
 	if (!queue_is_mq(disk->queue)) {
 		ret = -EOPNOTSUPP;
 		goto close_bdev;
-- 
2.56.0

Re: [PATCH 28/98] blk-iocost: check again that the disk is alive

[Bart Van Assche]

On 10/28/26 09:41, Jens Axboe wrote:
> @@ -3322,6 +3322,11 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
>  
>  	body = ctx.body;
>  	disk = ctx.bdev->bd_disk;
> +	/* It may have gone while the mutex was not held */
> +	if (!disk_live(disk)) {
> +		ret = -ENODEV;
> +		goto close_bdev;
> +	}

Look at the close_bdev unwind path in ioc_qos_write():

close_bdev:
	q = ctx.bdev->bd_queue;
	blkg_conf_close_bdev(&ctx);
	blk_mq_unfreeze_queue(q, memflags);

blkg_conf_close_bdev(&ctx) unlocks q->rq_qos_mutex and calls
blkdev_put_no_open(ctx.bdev). When !disk_live(disk) is true because
del_gendisk() (and potentially put_disk()) ran while rq_qos_mutex was
dropped, ctx.bdev may hold the final reference to the bdev/gendisk.
blkdev_put_no_open(ctx.bdev) will then run disk_release() ->
blk_put_queue(q) and free q while it is still frozen, before the next
line calls blk_mq_unfreeze_queue(q, memflags) on the freed queue.

q->rq_qos_mutex needs to be unlocked and blk_mq_unfreeze_queue(q,
memflags) called before blkdev_put_no_open(ctx.bdev) drops the bdev
reference.

Re: [PATCH 28/98] blk-iocost: check again that the disk is alive

[Bart Van Assche]

On 10/28/26 09:41, Jens Axboe wrote:
> @@ -3322,6 +3322,11 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
>  
>  	body = ctx.body;
>  	disk = ctx.bdev->bd_disk;
> +	/* It may have gone while the mutex was not held */
> +	if (!disk_live(disk)) {
> +		ret = -ENODEV;
> +		goto close_bdev;
> +	}

Look at the close_bdev unwind path in ioc_qos_write():

close_bdev:
	q = ctx.bdev->bd_queue;
	blkg_conf_close_bdev(&ctx);
	blk_mq_unfreeze_queue(q, memflags);

blkg_conf_close_bdev(&ctx) unlocks q->rq_qos_mutex and calls
blkdev_put_no_open(ctx.bdev). When !disk_live(disk) is true because
del_gendisk() and put_disk() ran while rq_qos_mutex was dropped,
ctx.bdev can hold the last reference to the bdev/gendisk. Calling
blkg_conf_close_bdev(&ctx) before blk_mq_unfreeze_queue(q, memflags)
will then run disk_release() -> blk_put_queue(q) and free q while it is
still frozen, followed by a use-after-free in blk_mq_unfreeze_queue().

Since __blk_mq_unfreeze_queue() acquires q->mq_freeze_lock (which nests
outside q->rq_qos_mutex), close_bdev should unlock q->rq_qos_mutex, call
blk_mq_unfreeze_queue(q, memflags) while ctx.bdev still pins the disk
and queue, and only then call blkdev_put_no_open(ctx.bdev).

Here is a replacement patch (checked with checkpatch.pl and
syntax-checked with Clang; not runtime-tested):

---
diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index 7b5d865d8c14..df713bbc790b 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -3322,6 +3322,11 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
 
 	body = ctx.body;
 	disk = ctx.bdev->bd_disk;
+	/* It may have gone while the mutex was not held */
+	if (!disk_live(disk)) {
+		ret = -ENODEV;
+		goto close_bdev;
+	}
 	if (!queue_is_mq(disk->queue)) {
 		ret = -EOPNOTSUPP;
 		goto close_bdev;
@@ -3437,8 +3442,9 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
 
 close_bdev:
 	q = ctx.bdev->bd_queue;
-	blkg_conf_close_bdev(&ctx);
+	mutex_unlock(&q->rq_qos_mutex);
 	blk_mq_unfreeze_queue(q, memflags);
+	blkdev_put_no_open(ctx.bdev);
 	return ret ?: nbytes;
 
 einval:

[PATCH 29/98] blk-mq: keep sharing tags fairly after a queue update

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

A hctx only gets BLK_MQ_F_TAG_QUEUE_SHARED from
blk_mq_add_queue_tag_set(). The hctxs blk_mq_update_nr_hw_queues()
allocates for queues that joined long ago never get it, so they aren't
counted in active_queues and hctx_may_queue() lets them take all the
tags. One NVMe namespace can starve the others after the first update.

Set the flag on the new hctxs.

Fixes: 868f2f0b7206 ("blk-mq: dynamic h/w context count")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-mq.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 5ebe2e23df5c..6d3c60b4ea2b 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -5184,6 +5184,9 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
 			goto fallback;
 		}
 		blk_mq_map_swqueue(q);
+		/* The new hctxs don't know yet that they share their tags */
+		if (set->flags & BLK_MQ_F_TAG_QUEUE_SHARED)
+			queue_set_hctx_shared(q, true);
 	}
 switch_back:
 	/* The blk_mq_elv_switch_back unfreezes queue for us. */
-- 
2.56.0

[PATCH 30/98] blk-mq: don't bring back flags of a completed request

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

The timeout handler sets and clears RQF_TIMED_OUT in a request it only
holds a reference to, with a plain read-modify-write, while the
completion clears flags in rq_flags the same way. If the two interleave,
blk_mq_finish_request() runs twice with the flags back: the zone write
plug reference is dropped twice and the scheduler's ->finish_request()
is called again.

Change RQF_TIMED_OUT with cmpxchg, so the handler never writes back a
flag that changed under it.

Fixes: e5c0ca13659e ("blk-mq: release scheduler resource when request completes")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-mq.c      |  2 +-
 block/blk-timeout.c |  3 ++-
 block/blk.h         | 14 ++++++++++++++
 3 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 6d3c60b4ea2b..0491e9821322 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -1639,7 +1639,7 @@ EXPORT_SYMBOL_GPL(blk_mq_queue_inflight);
 
 static void blk_mq_rq_timed_out(struct request *req)
 {
-	req->rq_flags |= RQF_TIMED_OUT;
+	blk_rq_change_timed_out(req, true);
 	if (req->q->mq_ops->timeout) {
 		enum blk_eh_timer_return ret;
 
diff --git a/block/blk-timeout.c b/block/blk-timeout.c
index 1b8de0417fc1..0590720e38c7 100644
--- a/block/blk-timeout.c
+++ b/block/blk-timeout.c
@@ -137,7 +137,8 @@ void blk_add_timer(struct request *req)
 	if (!req->timeout)
 		req->timeout = q->rq_timeout;
 
-	req->rq_flags &= ~RQF_TIMED_OUT;
+	if (req->rq_flags & RQF_TIMED_OUT)
+		blk_rq_change_timed_out(req, false);
 
 	expiry = jiffies + req->timeout;
 	WRITE_ONCE(req->deadline, expiry);
diff --git a/block/blk.h b/block/blk.h
index a48665e10994..de3a1633403b 100644
--- a/block/blk.h
+++ b/block/blk.h
@@ -365,6 +365,20 @@ static inline void bio_integrity_free(struct bio *bio)
 unsigned long blk_rq_timeout(unsigned long timeout);
 void blk_add_timer(struct request *req);
 
+/*
+ * The timeout handler changes RQF_TIMED_OUT in a request which it does not
+ * own: whoever completes it changes rq_flags at the same time, with plain
+ * loads and stores. Never write back a flag that has changed under us.
+ */
+static inline void blk_rq_change_timed_out(struct request *rq, bool set)
+{
+	req_flags_t old = READ_ONCE(rq->rq_flags), new;
+
+	do {
+		new = set ? old | RQF_TIMED_OUT : old & ~RQF_TIMED_OUT;
+	} while (!try_cmpxchg(&rq->rq_flags, &old, new));
+}
+
 enum bio_merge_status {
 	BIO_MERGE_OK,
 	BIO_MERGE_NONE,
-- 
2.56.0

[PATCH 31/98] blk-mq: don't forget a timer the driver has restarted

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

If ->timeout() returns BLK_EH_RESET_TIMER, blk_add_timer() gives the
request a new deadline that blk_mq_timeout_work() doesn't know about
when it picks the next expiry. A 2s nbd read next to a 60s one expires
once and then not again for a minute. Let the new deadline take part.

Fixes: 320ae51feed5 ("blk-mq: new multi-queue block IO queueing mechanism")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-mq.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 0491e9821322..6d3ea97f9a00 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -1710,8 +1710,11 @@ static bool blk_mq_handle_expired(struct request *rq, void *priv)
 {
 	struct blk_expired_data *expired = priv;
 
-	if (blk_mq_req_expired(rq, expired))
+	if (blk_mq_req_expired(rq, expired)) {
 		blk_mq_rq_timed_out(rq);
+		/* If its timer has been started again, count the new deadline */
+		blk_mq_req_expired(rq, expired);
+	}
 	return true;
 }
 
-- 
2.56.0

[PATCH 32/98] blk-mq-dma: initialize the dma state for P2P, too

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

blk_rq_dma_map_iter_start() promises that the caller need not initialize
the dma_iova_state, but the P2P bus address path returns before it does
so. The state then holds what the last request with that tag left, and
blk_rq_dma_map_coalesce() describes a transfer as one IOVA range that it
isn't. Clear it on that path too.

Fixes: 858299dc6160 ("block: add scatterlist-less DMA mapping helpers")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-mq-dma.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/block/blk-mq-dma.c b/block/blk-mq-dma.c
index 88fd9cbc951f..e0e7a18790b5 100644
--- a/block/blk-mq-dma.c
+++ b/block/blk-mq-dma.c
@@ -181,6 +181,8 @@ static bool blk_dma_map_iter_start(struct request *req, struct device *dma_dev,
 	switch (pci_p2pdma_state(&iter->p2pdma, dma_dev,
 				 phys_to_page(vec.paddr))) {
 	case PCI_P2PDMA_MAP_BUS_ADDR:
+		/* don't leave the state of an earlier mapping to the caller */
+		memset(state, 0, sizeof(*state));
 		return blk_dma_map_bus(iter, &vec);
 	case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE:
 		/*
-- 
2.56.0

[PATCH 33/98] blk-mq-dma: map the metadata a passthrough request has

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

blk_rq_integrity_dma_map_iter_start() sizes the metadata from the sector
count and the device format. A passthrough request has whatever
userspace gave it, so too much is mapped past the allocated IOVA range
and too little hands the device an unmapped length. If the data needs a
second bio, the iterator moves on to it and dereferences a NULL
integrity payload.

Take the size from the payload and stop at a bio without one.

Fixes: fec9b16dc555 ("blk-mq-dma: add scatter-less integrity data DMA mapping")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-mq-dma.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/block/blk-mq-dma.c b/block/blk-mq-dma.c
index e0e7a18790b5..5df704bbd98a 100644
--- a/block/blk-mq-dma.c
+++ b/block/blk-mq-dma.c
@@ -13,6 +13,10 @@ static bool __blk_map_iter_next(struct blk_map_iter *iter)
 	if (!iter->bio || !iter->bio->bi_next)
 		return false;
 
+	/* All the metadata of a passthrough request is in its first bio */
+	if (iter->is_integrity && !bio_integrity(iter->bio->bi_next))
+		return false;
+
 	iter->bio = iter->bio->bi_next;
 	if (iter->is_integrity) {
 		iter->iter = bio_integrity(iter->bio)->bip_iter;
@@ -348,6 +352,10 @@ bool blk_rq_integrity_dma_map_iter_start(struct request *req,
 					   blk_rq_sectors(req));
 	struct bio *bio = req->bio;
 
+	/* User space decides how much metadata a passthrough request has */
+	if (blk_rq_is_passthrough(req))
+		len = bio_integrity(bio)->bip_iter.bi_size;
+
 	iter->iter = (struct blk_map_iter) {
 		.bio = bio,
 		.iter = bio_integrity(bio)->bip_iter,
-- 
2.56.0

[PATCH 34/98] block: don't enter reclaim under rq_qos_mutex

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

ioc_qos_write() and the wbt_lat_usec writers take rq_qos_mutex with the
queue frozen, blkcg_deactivate_policy() does the same with blkcg_mutex,
so nothing holding either may wait for IO. blkg_conf_prep() got
GFP_NOIO for the blkg in commit 5d726c4dbeed, but the
radix_tree_preload() right after it, the policy init functions and the
GFP_KERNEL fallback in blkcg_activate_policy() can still enter reclaim
there. lockdep reports the cycle on the first write to io.max.

Use GFP_NOIO for all of them.

Fixes: 9730763f4756 ("block: correct locking order for protecting blk-wbt parameters")
Fixes: 5d726c4dbeed ("blk-cgroup: fix possible deadlock while configuring policy")
Cc: Tejun Heo <tj@kernel.org>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-cgroup.c    | 13 +++++++++----
 block/blk-iocost.c    |  5 +++--
 block/blk-iolatency.c |  3 ++-
 block/blk-throttle.c  |  3 ++-
 4 files changed, 16 insertions(+), 8 deletions(-)

diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 00da55c6ec43..5b7a32293dc6 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -884,7 +884,11 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,
 			parent = blkcg_parent(parent);
 		}
 
-		/* Drop locks to do new blkg allocation with GFP_KERNEL. */
+		/*
+		 * Drop the spinlock to allocate the new blkg. rq_qos_mutex and
+		 * blkcg_mutex stay held, and both are taken with the queue
+		 * frozen elsewhere: GFP_NOIO.
+		 */
 		spin_unlock_irq(&q->queue_lock);
 
 		new_blkg = blkg_alloc(pos, disk, GFP_NOIO);
@@ -893,7 +897,7 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,
 			goto fail_exit;
 		}
 
-		if (radix_tree_preload(GFP_KERNEL)) {
+		if (radix_tree_preload(GFP_NOIO)) {
 			blkg_free(new_blkg);
 			ret = -ENOMEM;
 			goto fail_exit;
@@ -1648,7 +1652,8 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)
 		if (!pd) {
 			/*
 			 * GFP_NOWAIT failed.  Free the existing one and
-			 * prealloc for @blkg w/ GFP_KERNEL.
+			 * prealloc for @blkg w/ GFP_NOIO: blkcg_mutex is held,
+			 * and a bio based queue has not been frozen above.
 			 */
 			if (pinned_blkg)
 				blkg_put(pinned_blkg);
@@ -1660,7 +1665,7 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol)
 			if (pd_prealloc)
 				pol->pd_free_fn(pd_prealloc);
 			pd_prealloc = pol->pd_alloc_fn(disk, blkg->blkcg,
-						       GFP_KERNEL);
+						       GFP_NOIO);
 			if (pd_prealloc)
 				goto retry;
 			else
diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index bfb0dd320c4a..99fe89f687ae 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -2938,11 +2938,12 @@ static int blk_iocost_init(struct gendisk *disk)
 	struct ioc *ioc;
 	int i, cpu, ret;
 
-	ioc = kzalloc_obj(*ioc);
+	/* The caller holds rq_qos_mutex, which is taken with the queue frozen */
+	ioc = kzalloc_obj(*ioc, GFP_NOIO);
 	if (!ioc)
 		return -ENOMEM;
 
-	ioc->pcpu_stat = alloc_percpu(struct ioc_pcpu_stat);
+	ioc->pcpu_stat = alloc_percpu_gfp(struct ioc_pcpu_stat, GFP_NOIO);
 	if (!ioc->pcpu_stat) {
 		kfree(ioc);
 		return -ENOMEM;
diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c
index c15b1aba606c..a3bd153c67be 100644
--- a/block/blk-iolatency.c
+++ b/block/blk-iolatency.c
@@ -761,7 +761,8 @@ static int blk_iolatency_init(struct gendisk *disk)
 	struct blk_iolatency *blkiolat;
 	int ret;
 
-	blkiolat = kzalloc_obj(*blkiolat);
+	/* The caller holds rq_qos_mutex, which is taken with the queue frozen */
+	blkiolat = kzalloc_obj(*blkiolat, GFP_NOIO);
 	if (!blkiolat)
 		return -ENOMEM;
 
diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index c46807a0faab..c5e26551956a 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -1322,7 +1322,8 @@ static int blk_throtl_init(struct gendisk *disk)
 	unsigned int memflags;
 	int ret;
 
-	td = kzalloc_node(sizeof(*td), GFP_KERNEL, q->node);
+	/* The caller holds rq_qos_mutex, which is taken with the queue frozen */
+	td = kzalloc_node(sizeof(*td), GFP_NOIO, q->node);
 	if (!td)
 		return -ENOMEM;
 
-- 
2.56.0

Re: [PATCH 34/98] block: don't enter reclaim under rq_qos_mutex

[Tejun Heo]

On Wed, Oct 28, 2026 at 10:41:37AM -0600, Jens Axboe wrote:
> ioc_qos_write() and the wbt_lat_usec writers take rq_qos_mutex with the
> queue frozen, blkcg_deactivate_policy() does the same with blkcg_mutex,
> so nothing holding either may wait for IO. blkg_conf_prep() got
> GFP_NOIO for the blkg in commit 5d726c4dbeed, but the
> radix_tree_preload() right after it, the policy init functions and the
> GFP_KERNEL fallback in blkcg_activate_policy() can still enter reclaim
> there. lockdep reports the cycle on the first write to io.max.
>
> Use GFP_NOIO for all of them.
>
> Fixes: 9730763f4756 ("block: correct locking order for protecting blk-wbt parameters")
> Fixes: 5d726c4dbeed ("blk-cgroup: fix possible deadlock while configuring policy")
> Cc: Tejun Heo <tj@kernel.org>
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> Signed-off-by: Jens Axboe <axboe@kernel.dk>

Acked-by: Tejun Heo <tj@kernel.org>

Thanks.

--
tejun

[PATCH 35/98] blktrace: remove the debugfs files before the channel

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

blk_trace_free() closes the relay channel, which frees it, and removes
the dropped and msg files afterwards, while both of them read bt->rchan.
Remove the files first, which waits for their readers, then close the
channel.

Fixes: 7f2173894f7b ("blktrace: use rbuf->stats.full as a drop indicator in relayfs")
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 kernel/trace/blktrace.c | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c
index 8cd2520b4c99..a37857a2fbd3 100644
--- a/kernel/trace/blktrace.c
+++ b/kernel/trace/blktrace.c
@@ -468,18 +468,17 @@ static void __blk_add_trace(struct blk_trace *bt, sector_t sector, int bytes,
 
 static void blk_trace_free(struct request_queue *q, struct blk_trace *bt)
 {
-	relay_close(bt->rchan);
-
 	/*
 	 * If 'bt->dir' is not set, then both 'dropped' and 'msg' are created
-	 * under 'q->debugfs_dir', thus lookup and remove them.
+	 * under 'q->debugfs_dir', thus lookup and remove them. They use the
+	 * relay channel: wait for their users before it goes away.
 	 */
-	if (!bt->dir) {
-		debugfs_lookup_and_remove("dropped", q->debugfs_dir);
-		debugfs_lookup_and_remove("msg", q->debugfs_dir);
-	} else {
-		debugfs_remove(bt->dir);
-	}
+	debugfs_lookup_and_remove("dropped", bt->dir ?: q->debugfs_dir);
+	debugfs_lookup_and_remove("msg", bt->dir ?: q->debugfs_dir);
+
+	relay_close(bt->rchan);
+	debugfs_remove(bt->dir);
+
 	free_percpu(bt->sequence);
 	free_percpu(bt->msg_data);
 	kfree(bt);
-- 
2.56.0

[PATCH 36/98] blk-mq: quiesce a queue that joins a quiesced tag set

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

blk_mq_quiesce_tagset() only quiesces the queues on the list at that
moment. A queue added before blk_mq_unquiesce_tagset() runs unquiesced,
and then gets unquiesced without ever having been quiesced, which trips
the warning in blk_mq_unquiesce_queue(). NVMe gets there when a
controller fails while its namespaces are being scanned.

Count the quiesce calls in the tag set and quiesce a joining queue that
many times, under a spinlock that also serializes the two functions. It
can't be tag_list_lock, see commit 59e25ef2b413.

Fixes: 414dd48e882c ("blk-mq: add tagset quiesce interface")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-mq.c         | 22 ++++++++++++++++++++++
 include/linux/blk-mq.h |  6 ++++++
 2 files changed, 28 insertions(+)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 6d3ea97f9a00..0e8449424c9a 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -336,12 +336,15 @@ void blk_mq_quiesce_tagset(struct blk_mq_tag_set *set)
 {
 	struct request_queue *q;
 
+	spin_lock(&set->quiesce_lock);
+	set->quiesce_depth++;
 	rcu_read_lock();
 	list_for_each_entry_rcu(q, &set->tag_list, tag_set_list) {
 		if (!blk_queue_skip_tagset_quiesce(q))
 			blk_mq_quiesce_queue_nowait(q);
 	}
 	rcu_read_unlock();
+	spin_unlock(&set->quiesce_lock);
 
 	blk_mq_wait_quiesce_done(set);
 }
@@ -351,12 +354,16 @@ void blk_mq_unquiesce_tagset(struct blk_mq_tag_set *set)
 {
 	struct request_queue *q;
 
+	spin_lock(&set->quiesce_lock);
+	if (set->quiesce_depth)
+		set->quiesce_depth--;
 	rcu_read_lock();
 	list_for_each_entry_rcu(q, &set->tag_list, tag_set_list) {
 		if (!blk_queue_skip_tagset_quiesce(q))
 			blk_mq_unquiesce_queue(q);
 	}
 	rcu_read_unlock();
+	spin_unlock(&set->quiesce_lock);
 }
 EXPORT_SYMBOL_GPL(blk_mq_unquiesce_tagset);
 
@@ -4346,6 +4353,8 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q)
 static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,
 				     struct request_queue *q)
 {
+	unsigned int i;
+
 	mutex_lock(&set->tag_list_lock);
 
 	/*
@@ -4359,7 +4368,18 @@ static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,
 	}
 	if (set->flags & BLK_MQ_F_TAG_QUEUE_SHARED)
 		queue_set_hctx_shared(q, true);
+
+	/*
+	 * A queue that joins a quiesced tag set is quiesced like the others,
+	 * and blk_mq_unquiesce_tagset() finds what it is going to undo.
+	 */
+	spin_lock(&set->quiesce_lock);
 	list_add_tail_rcu(&q->tag_set_list, &set->tag_list);
+	if (!blk_queue_skip_tagset_quiesce(q)) {
+		for (i = 0; i < set->quiesce_depth; i++)
+			blk_mq_quiesce_queue_nowait(q);
+	}
+	spin_unlock(&set->quiesce_lock);
 
 	mutex_unlock(&set->tag_list_lock);
 }
@@ -4928,6 +4948,8 @@ int blk_mq_alloc_tag_set(struct blk_mq_tag_set *set)
 
 	mutex_init(&set->tag_list_lock);
 	INIT_LIST_HEAD(&set->tag_list);
+	spin_lock_init(&set->quiesce_lock);
+	set->quiesce_depth = 0;
 
 	return 0;
 
diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
index 5e130b282b42..aa3b35405edc 100644
--- a/include/linux/blk-mq.h
+++ b/include/linux/blk-mq.h
@@ -523,6 +523,10 @@ enum hctx_type {
  * @tag_list_lock: Serializes tag_list accesses.
  * @tag_list:	   List of the request queues that use this tag set. See also
  *		   request_queue.tag_set_list.
+ * @quiesce_lock:  Serializes blk_mq_[un]quiesce_tagset() against each other
+ *		   and against a request queue being added to @tag_list.
+ * @quiesce_depth: How often blk_mq_quiesce_tagset() has been called without
+ *		   blk_mq_unquiesce_tagset().
  * @srcu:	   Use as lock when type of the request queue is blocking
  *		   (BLK_MQ_F_BLOCKING).
  * @update_nr_hwq_lock:
@@ -548,6 +552,8 @@ struct blk_mq_tag_set {
 
 	struct mutex		tag_list_lock;
 	struct list_head	tag_list;
+	spinlock_t		quiesce_lock;
+	unsigned int		quiesce_depth;
 	struct srcu_struct	*srcu;
 
 	struct rw_semaphore	update_nr_hwq_lock;
-- 
2.56.0

[PATCH 37/98] blk-mq: exit the hctxs of a queue under tag_list_lock

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

blk_mq_update_nr_hw_queues() exits and replaces the hctxs of every queue
on set->tag_list under tag_list_lock. blk_mq_exit_queue() exits its
hctxs without the lock and only then takes the queue off the list, so an
update running at the same time exits them again and sets up new ones
that nobody exits. del_gendisk() is kept away by update_nr_hwq_lock,
disk_release() of a never added disk and blk_mq_destroy_queue() are not,
and nvme_alloc_ns() drops such a disk when it finds the queues frozen by
a reset.

Hold tag_list_lock for both steps.

Fixes: 630ef623ed26 ("blk-mq: Swap two calls in blk_mq_exit_queue()")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-mq.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 0e8449424c9a..37c8f68e23c4 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -4339,7 +4339,8 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q)
 {
 	struct blk_mq_tag_set *set = q->tag_set;
 
-	mutex_lock(&set->tag_list_lock);
+	lockdep_assert_held(&set->tag_list_lock);
+
 	list_del_rcu(&q->tag_set_list);
 	if (list_is_singular(&set->tag_list)) {
 		/* just transitioned to unshared */
@@ -4347,7 +4348,6 @@ static void blk_mq_del_queue_tag_set(struct request_queue *q)
 		/* update existing queue */
 		blk_mq_update_tag_set_shared(set, false);
 	}
-	mutex_unlock(&set->tag_list_lock);
 }
 
 static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,
@@ -4711,10 +4711,16 @@ void blk_mq_exit_queue(struct request_queue *q)
 {
 	struct blk_mq_tag_set *set = q->tag_set;
 
+	/*
+	 * As long as the queue is on set->tag_list, its hctxs belong to
+	 * blk_mq_update_nr_hw_queues() as well, which exits and replaces them.
+	 */
+	mutex_lock(&set->tag_list_lock);
 	/* Checks hctx->flags & BLK_MQ_F_TAG_QUEUE_SHARED. */
 	blk_mq_exit_hw_queues(q, set, set->nr_hw_queues);
 	/* May clear BLK_MQ_F_TAG_QUEUE_SHARED in hctx->flags. */
 	blk_mq_del_queue_tag_set(q);
+	mutex_unlock(&set->tag_list_lock);
 }
 
 static int __blk_mq_alloc_rq_maps(struct blk_mq_tag_set *set)
-- 
2.56.0

[PATCH 38/98] blk-mq: set up a queue under tag_list_lock

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

blk_mq_init_allocated_queue() only takes tag_list_lock at the end, when
it adds the queue to the set. Before that it sizes the queue by
set->nr_hw_queues, hands out set->tags[], maps by set->map[], and
blk_mq_map_swqueue() frees tags of the set. An update clearing the maps
at that time makes the new queue free tags that all the other queues
share. An NVMe controller reset during a namespace scan does exactly
that.

Take the lock before the first look at the set. An update can be part of
error handling, so the allocations under the lock must not wait for IO:
memalloc_noio_save(), as in add_disk_fwnode().

Fixes: 868f2f0b7206 ("blk-mq: dynamic h/w context count")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-mq.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/block/blk-mq.c b/block/blk-mq.c
index 37c8f68e23c4..458ef0970527 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -4355,7 +4355,7 @@ static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,
 {
 	unsigned int i;
 
-	mutex_lock(&set->tag_list_lock);
+	lockdep_assert_held(&set->tag_list_lock);
 
 	/*
 	 * Check to see if we're transitioning to shared (from 1 to 2 queues).
@@ -4380,8 +4380,6 @@ static void blk_mq_add_queue_tag_set(struct blk_mq_tag_set *set,
 			blk_mq_quiesce_queue_nowait(q);
 	}
 	spin_unlock(&set->quiesce_lock);
-
-	mutex_unlock(&set->tag_list_lock);
 }
 
 /* All allocations will be freed in release handler of q->mq_kobj */
@@ -4658,6 +4656,8 @@ static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
 int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
 		struct request_queue *q)
 {
+	unsigned int memflags;
+
 	/* mark the queue as mq asap */
 	q->mq_ops = set->ops;
 
@@ -4676,6 +4676,16 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
 	INIT_LIST_HEAD(&q->unused_hctx_list);
 	spin_lock_init(&q->unused_hctx_lock);
 
+	/*
+	 * Keep blk_mq_update_nr_hw_queues() away until the queue is on
+	 * set->tag_list: it changes the number of hardware queues, the tags and
+	 * the maps, and blk_mq_map_swqueue() allocates and frees tags of the set.
+	 * An update can be part of the error handling of a driver, so it must
+	 * not have to wait for somebody who waits for I/O.
+	 */
+	memflags = memalloc_noio_save();
+	mutex_lock(&set->tag_list_lock);
+
 	blk_mq_realloc_hw_ctxs(set, q);
 	if (!q->nr_hw_queues)
 		goto err_hctxs;
@@ -4696,9 +4706,14 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
 	blk_mq_init_cpu_queues(q, set->nr_hw_queues);
 	blk_mq_map_swqueue(q);
 	blk_mq_add_queue_tag_set(set, q);
+
+	mutex_unlock(&set->tag_list_lock);
+	memalloc_noio_restore(memflags);
 	return 0;
 
 err_hctxs:
+	mutex_unlock(&set->tag_list_lock);
+	memalloc_noio_restore(memflags);
 	blk_mq_release(q);
 err_exit:
 	q->mq_ops = NULL;
-- 
2.56.0

[PATCH 39/98] rnbd-srv: fix use-after-free on two CLOSE messages

[Jens Axboe]

From: Chris Mason <mason@kernel.org>

process_msg_close() looks the sess_dev up with a kref, drops it right
away, and only then takes the session mutex for the teardown. A session
has several connections handled by different workers, so two CLOSE
messages for the same ID both finish the lookup before either one tears
the sess_dev down, and the second one works on freed memory. Any host
that can open a device on the server can do that.

Take the mutex first and look the ID up with xa_load() under it. The
second CLOSE then finds nothing.

Fixes: 2de6c8de192b ("block/rnbd: server: main functionality")
Cc: "Md. Haris Iqbal" <haris.iqbal@ionos.com>
Cc: Jack Wang <jinpu.wang@ionos.com>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/block/rnbd/rnbd-srv.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/drivers/block/rnbd/rnbd-srv.c b/drivers/block/rnbd/rnbd-srv.c
index 1e3de82fd70b..1cfa2c580759 100644
--- a/drivers/block/rnbd/rnbd-srv.c
+++ b/drivers/block/rnbd/rnbd-srv.c
@@ -360,14 +360,15 @@ static void process_msg_close(struct rnbd_srv_session *srv_sess,
 
 	trace_process_msg_close(srv_sess, close_msg);
 
-	sess_dev = rnbd_get_sess_dev(le32_to_cpu(close_msg->device_id),
-				      srv_sess);
-	if (IS_ERR(sess_dev))
-		return;
-
-	rnbd_put_sess_dev(sess_dev);
+	/*
+	 * Each teardown of a sess_dev runs under srv_sess->lock and erases the
+	 * ID, so an entry that is found under the lock is alive.
+	 */
 	mutex_lock(&srv_sess->lock);
-	rnbd_srv_destroy_dev_session_sysfs(sess_dev);
+	sess_dev = xa_load(&srv_sess->index_idr,
+			   le32_to_cpu(close_msg->device_id));
+	if (sess_dev)
+		rnbd_srv_destroy_dev_session_sysfs(sess_dev);
 	mutex_unlock(&srv_sess->lock);
 }
 
-- 
2.56.0

[PATCH 40/98] rnbd-srv: publish the sess_dev at the end of OPEN

[Jens Axboe]

From: Chris Mason <mason@kernel.org>

rnbd_sess_dev_alloc() stores the empty sess_dev in the XArray as its
first step. An IO message with the new ID from another connection can
find it before bdev_file is set, or hold a reference while a failing
OPEN frees it after a mere RCU grace period. Both are reachable from any
host that can open a device.

Reserve the ID with a NULL entry and store the sess_dev once nothing can
fail anymore. The error path still erases the ID, and no longer needs
synchronize_rcu().

Fixes: 2de6c8de192b ("block/rnbd: server: main functionality")
Cc: "Md. Haris Iqbal" <haris.iqbal@ionos.com>
Cc: Jack Wang <jinpu.wang@ionos.com>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/block/rnbd/rnbd-srv.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/drivers/block/rnbd/rnbd-srv.c b/drivers/block/rnbd/rnbd-srv.c
index 1cfa2c580759..1c80e8fe0267 100644
--- a/drivers/block/rnbd/rnbd-srv.c
+++ b/drivers/block/rnbd/rnbd-srv.c
@@ -431,7 +431,7 @@ static struct rnbd_srv_sess_dev
 	if (!sess_dev)
 		return ERR_PTR(-ENOMEM);
 
-	error = xa_alloc(&srv_sess->index_idr, &sess_dev->device_id, sess_dev,
+	error = xa_alloc(&srv_sess->index_idr, &sess_dev->device_id, NULL,
 			 xa_limit_32b, GFP_NOWAIT);
 	if (error < 0) {
 		pr_warn("Allocating idr failed, err: %d\n", error);
@@ -802,6 +802,13 @@ static int process_msg_open(struct rnbd_srv_session *srv_sess,
 	list_add(&srv_sess_dev->dev_list, &srv_dev->sess_dev_list);
 	mutex_unlock(&srv_dev->lock);
 
+	/*
+	 * Nothing fails from here on, so let I/O messages find the sess_dev.
+	 * The ID is reserved, so this store allocates nothing.
+	 */
+	xa_store(&srv_sess->index_idr, srv_sess_dev->device_id, srv_sess_dev,
+		 GFP_KERNEL);
+
 	rnbd_srv_info(srv_sess_dev, "Opened device '%s'\n", srv_dev->name);
 
 	kfree(full_path);
@@ -813,7 +820,6 @@ static int process_msg_open(struct rnbd_srv_session *srv_sess,
 
 free_srv_sess_dev:
 	xa_erase(&srv_sess->index_idr, srv_sess_dev->device_id);
-	synchronize_rcu();
 	kfree(srv_sess_dev);
 srv_dev_put:
 	if (open_msg->access_mode != RNBD_ACCESS_RO) {
-- 
2.56.0

[PATCH 41/98] zram: check a bio's range against the device size

[Jens Axboe]

From: Chris Mason <mason@kernel.org>

bio_check_eod() counts whole sectors, so a bio shorter than a sector, or
one whose tail is shorter than a sector, passes it even at or past the
end of the device. zram takes such a piece to the slot of its sector and
reads, locks and modifies an entry outside the table, or dereferences a
NULL table on a device without a size. valid_io_request() refused these
until commit 9fe95babc742 removed it, and any in-kernel builder of bios
that takes size and sector from a peer can feed them in.

Check the range at the top of zram_submit_bio(), with the size rounded
up to sectors, and fail with an IO error like bio_check_eod() does.

Fixes: 9fe95babc742 ("zram: remove valid_io_request")
Cc: Minchan Kim <minchan@kernel.org>
Cc: Sergey Senozhatsky <senozhatsky@chromium.org>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/block/zram/zram_drv.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index 024402438bd1..2555057a48b5 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -2794,6 +2794,23 @@ static void zram_bio_write(struct zram *zram, struct bio *bio)
 	bio_endio(bio);
 }
 
+/*
+ * bio_check_eod() counts whole sectors: it does not look at a bio that is
+ * shorter than a sector, nor at a tail that is shorter than one. zram takes
+ * such a piece to the slot of its sector, so round the size up. An empty bio
+ * has to start inside the device as well: the loops for reads and writes run
+ * once before their test.
+ */
+static bool zram_bio_in_range(struct zram *zram, struct bio *bio)
+{
+	sector_t nr_sectors = get_capacity(zram->disk);
+	sector_t start = bio->bi_iter.bi_sector;
+	sector_t len = ((sector_t)bio->bi_iter.bi_size + SECTOR_SIZE - 1) >>
+			SECTOR_SHIFT;
+
+	return start < nr_sectors && len <= nr_sectors - start;
+}
+
 /*
  * Handler function for all zram I/O requests.
  */
@@ -2801,6 +2818,11 @@ static void zram_submit_bio(struct bio *bio)
 {
 	struct zram *zram = bio->bi_bdev->bd_disk->private_data;
 
+	if (unlikely(!zram_bio_in_range(zram, bio))) {
+		bio_io_error(bio);
+		return;
+	}
+
 	switch (bio_op(bio)) {
 	case REQ_OP_READ:
 		zram_bio_read(zram, bio);
-- 
2.56.0

Re: [PATCH 41/98] zram: check a bio's range against the device size

[Bart Van Assche]

On 10/28/26 09:41, Jens Axboe wrote:
> +/*
> + * bio_check_eod() counts whole sectors: it does not look at a bio that is
> + * shorter than a sector, nor at a tail that is shorter than one. zram takes
> + * such a piece to the slot of its sector, so round the size up. An empty bio
> + * has to start inside the device as well: the loops for reads and writes run
> + * once before their test.
> + */
> +static bool zram_bio_in_range(struct zram *zram, struct bio *bio)
> +{
> +	sector_t nr_sectors = get_capacity(zram->disk);
> +	sector_t start = bio->bi_iter.bi_sector;
> +	sector_t len = ((sector_t)bio->bi_iter.bi_size + SECTOR_SIZE - 1) >>
> +			SECTOR_SHIFT;
> +
> +	return start < nr_sectors && len <= nr_sectors - start;
> +}

There are two problems with zram_bio_in_range():

1. It explicitly allows an empty REQ_OP_READ or REQ_OP_WRITE bio
   (bi_size == 0) as long as start < nr_sectors. Both zram_bio_read()
   and zram_bio_write() use a do { ... } while (iter.bi_size) loop whose
   first statement is bio_iter_iovec(bio, iter), which immediately
   dereferences NULL bio->bi_io_vec on an empty bio.

2. It allows sub-4K / unaligned bios as long as they fall within
   nr_sectors. However, ZRAM_LOGICAL_BLOCK_SIZE is 4096, and on
   PAGE_SIZE == 4096 kernels is_partial_io() is hardcoded to return
   false:
   - zram_bvec_read() calls zram_read_page(zram, bvec->bv_page, index,
     bio), ignoring bv_offset and bv_len and overwriting all 4096 bytes
     of bv_page.
   - zram_bvec_write() calls zram_write_page(zram, bvec->bv_page,
     index), clobbering the rest of the 4 KiB zram slot and compressing
     up to 4095 bytes of adjacent kernel page memory into the slot where
     a subsequent read will return it.

Shouldn't this reinstate valid_io_request()'s ZRAM_LOGICAL_BLOCK_SIZE
alignment checks on bi_sector and bi_size (as well as rejecting
bi_size == 0)?

Re: [PATCH 41/98] zram: check a bio's range against the device size

[Bart Van Assche]

On 10/28/26 09:41, Jens Axboe wrote:
> +static bool zram_bio_in_range(struct zram *zram, struct bio *bio)
> +{
> +	sector_t nr_sectors = get_capacity(zram->disk);
> +	sector_t start = bio->bi_iter.bi_sector;
> +	sector_t len = ((sector_t)bio->bi_iter.bi_size + SECTOR_SIZE - 1) >>
> +			SECTOR_SHIFT;
> +
> +	return start < nr_sectors && len <= nr_sectors - start;
> +}

There are two problems with zram_bio_in_range():

1. It allows an empty REQ_OP_READ or REQ_OP_WRITE bio (bi_size == 0) as
   long as start < nr_sectors. Both zram_bio_read() and zram_bio_write()
   use a do { ... } while (iter.bi_size) loop whose first statement is
   bio_iter_iovec(bio, iter), dereferencing NULL bio->bi_io_vec on an
   empty bio. (Empty REQ_OP_FLUSH bios are already completed in
   submit_bio_noacct() because zram does not set a write cache.)

2. It allows sub-4K / unaligned bios as long as they fall within
   nr_sectors. ZRAM_LOGICAL_BLOCK_SIZE is 4096, and on PAGE_SIZE == 4096
   kernels is_partial_io() is hardcoded to return false:
   - zram_bvec_read() calls zram_read_page(zram, bvec->bv_page, index,
     bio), ignoring bv_offset and bv_len and overwriting all 4096 bytes
     of bv_page.
   - zram_bvec_write() calls zram_write_page(zram, bvec->bv_page,
     index), clobbering the rest of the 4 KiB zram slot and compressing
     up to 4095 bytes of adjacent kernel page memory into the slot where
     a subsequent read can read it back.

Reinstating valid_io_request()'s ZRAM_LOGICAL_BLOCK_SIZE alignment
checks on bi_sector and bi_size alongside rejecting bi_size == 0 and
out-of-range bios closes all of these cases.

Here is a replacement patch (checked with checkpatch.pl and
syntax-checked with Clang; not runtime-tested):

---
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index ab064c84b092..8d4e3791b424 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -2796,6 +2796,23 @@ static void zram_bio_write(struct zram *zram, struct bio *bio)
 	bio_endio(bio);
 }
 
+static bool zram_valid_io_request(struct zram *zram, struct bio *bio)
+{
+	sector_t nr_sectors = get_capacity(zram->disk);
+	sector_t start = bio->bi_iter.bi_sector;
+	unsigned int size = bio->bi_iter.bi_size;
+
+	if (!size)
+		return false;
+	if (start & (ZRAM_SECTOR_PER_LOGICAL_BLOCK - 1))
+		return false;
+	if (size & (ZRAM_LOGICAL_BLOCK_SIZE - 1))
+		return false;
+
+	return start < nr_sectors &&
+	       (size >> SECTOR_SHIFT) <= nr_sectors - start;
+}
+
 /*
  * Handler function for all zram I/O requests.
  */
@@ -2803,6 +2820,11 @@ static void zram_submit_bio(struct bio *bio)
 {
 	struct zram *zram = bio->bi_bdev->bd_disk->private_data;
 
+	if (unlikely(!zram_valid_io_request(zram, bio))) {
+		bio_io_error(bio);
+		return;
+	}
+
 	switch (bio_op(bio)) {
 	case REQ_OP_READ:
 		zram_bio_read(zram, bio);

[PATCH 42/98] block: partitions/mac: compute the sectors in 64 bits

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

start_block and block_count are converted to 512 byte sectors in 32-bit
arithmetic, although put_partition() takes a sector_t. With a block size
above 512 bytes anything beyond 2TiB wraps and the partition lands on
top of another one.

Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/partitions/mac.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/block/partitions/mac.c b/block/partitions/mac.c
index df03ca428e15..f372392af95b 100644
--- a/block/partitions/mac.c
+++ b/block/partitions/mac.c
@@ -97,8 +97,8 @@ int mac_partition(struct parsed_partitions *state)
 		if (be16_to_cpu(part->signature) != MAC_PARTITION_MAGIC)
 			break;
 		put_partition(state, slot,
-			be32_to_cpu(part->start_block) * (secsize/512),
-			be32_to_cpu(part->block_count) * (secsize/512));
+			(sector_t)be32_to_cpu(part->start_block) * (secsize/512),
+			(sector_t)be32_to_cpu(part->block_count) * (secsize/512));
 
 		if (!strncasecmp(part->type, "Linux_RAID", 10))
 			state->parts[slot].flags = ADDPART_FLAG_RAID;
-- 
2.56.0

[PATCH 43/98] blk-mq: free the bitmaps of tags in the SRCU callback

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

Commit ad0d05dbddc1 keeps struct blk_mq_tags and the requests until the
iterators under tags_srcu are gone, but the two bitmaps they walk are
still freed right away. When the number of hardware queues shrinks,
blk_mq_tagset_busy_iter() may be about to walk tags that
__blk_mq_update_nr_hw_queues() frees. sbitmap_free() clears the map
pointer, so that's a NULL dereference or a use-after-free, depending on
the timing.

Free the bitmaps with the rest in the callback.

Fixes: ad0d05dbddc1 ("blk-mq: Defer freeing of tags page_list to SRCU callback")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 block/blk-mq-tag.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c
index aaea2e2c2f23..e3edf18697af 100644
--- a/block/blk-mq-tag.c
+++ b/block/blk-mq-tag.c
@@ -623,17 +623,17 @@ static void blk_mq_free_tags_callback(struct rcu_head *head)
 		kmemleak_free(page_address(page));
 		__free_pages(page, page->private);
 	}
+	/* An iterator may walk the bitmaps for as long as it may see the tags */
+	sbitmap_queue_free(&tags->bitmap_tags);
+	sbitmap_queue_free(&tags->breserved_tags);
 	kfree(tags);
 }
 
 void blk_mq_free_tags(struct blk_mq_tags *tags)
 {
-	sbitmap_queue_free(&tags->bitmap_tags);
-	sbitmap_queue_free(&tags->breserved_tags);
-
 	/* if tags pages is not allocated yet, free tags directly */
 	if (list_empty(&tags->page_list)) {
-		kfree(tags);
+		blk_mq_free_tags_callback(&tags->rcu_head);
 		return;
 	}
 
-- 
2.56.0

Re: [PATCH 43/98] blk-mq: free the bitmaps of tags in the SRCU callback

[Hannes Reinecke]

On 10/28/26 5:41 PM, Jens Axboe wrote:
> From: Josef Bacik <josef@toxicpanda.com>
> 
> Commit ad0d05dbddc1 keeps struct blk_mq_tags and the requests until the
> iterators under tags_srcu are gone, but the two bitmaps they walk are
> still freed right away. When the number of hardware queues shrinks,
> blk_mq_tagset_busy_iter() may be about to walk tags that
> __blk_mq_update_nr_hw_queues() frees. sbitmap_free() clears the map
> pointer, so that's a NULL dereference or a use-after-free, depending on
> the timing.
> 
> Free the bitmaps with the rest in the callback.
> 
> Fixes: ad0d05dbddc1 ("blk-mq: Defer freeing of tags page_list to SRCU callback")
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> Signed-off-by: Jens Axboe <axboe@kernel.dk>
> ---
>   block/blk-mq-tag.c | 8 ++++----
>   1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c
> index aaea2e2c2f23..e3edf18697af 100644
> --- a/block/blk-mq-tag.c
> +++ b/block/blk-mq-tag.c
> @@ -623,17 +623,17 @@ static void blk_mq_free_tags_callback(struct rcu_head *head)
>   		kmemleak_free(page_address(page));
>   		__free_pages(page, page->private);
>   	}
> +	/* An iterator may walk the bitmaps for as long as it may see the tags */
> +	sbitmap_queue_free(&tags->bitmap_tags);
> +	sbitmap_queue_free(&tags->breserved_tags);
>   	kfree(tags);
>   }
>   
>   void blk_mq_free_tags(struct blk_mq_tags *tags)
>   {
> -	sbitmap_queue_free(&tags->bitmap_tags);
> -	sbitmap_queue_free(&tags->breserved_tags);
> -
>   	/* if tags pages is not allocated yet, free tags directly */
>   	if (list_empty(&tags->page_list)) {
> -		kfree(tags);
> +		blk_mq_free_tags_callback(&tags->rcu_head);
>   		return;
>   	}
>   
Reviewed-by: Hannes Reinecke <hare@kernel.org>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

[PATCH 44/98] block: apply the segment size to the integrity buffer

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

The integrity buffer is a single segment and must not exceed
max_segment_size. max_integrity_io_size() applies that limit to the data
size instead, which is off by a factor of 64 for 8 bytes of PI per 512
bytes of data. A SCSI host that doesn't set a maximum segment size gets
64KiB, lpfc and qla2xxx among them, so a DIX disk behind one of those
does no IO above 64KiB.

Limit the buffer and derive the data size from that, which makes it 4MiB
for that case.

Fixes: ec7f31b2a2d3 ("block: make bio auto-integrity deadlock safe")
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 include/linux/blkdev.h | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index d003a9d2d1f6..1a965fb5ae6f 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -1819,11 +1819,12 @@ static inline int bio_split_rw_at(struct bio *bio,
  */
 static inline unsigned int max_integrity_io_size(struct queue_limits *lim)
 {
-	u64 max_intervals;
+	unsigned int buf_size = min_t(unsigned int, lim->max_segment_size,
+				      BLK_INTEGRITY_MAX_SIZE);
 
-	max_intervals = BLK_INTEGRITY_MAX_SIZE / lim->integrity.metadata_size;
-	return min_t(u64, lim->max_segment_size,
-		max_intervals << lim->integrity.interval_exp);
+	return min_t(u64, UINT_MAX,
+		(u64)(buf_size / lim->integrity.metadata_size) <<
+			lim->integrity.interval_exp);
 }
 
 #define DEFINE_IO_COMP_BATCH(name)	struct io_comp_batch name = { }
-- 
2.56.0

[PATCH 45/98] nvmet: don't truncate the length of big discards

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

nvmet_bdev_execute_discard() shifts the 32-bit nlb of a DSM range into
sectors before widening it to sector_t, so on a 4k namespace the top
bits are lost. A 3 TiB range discards 1 TiB, a 2 TiB range discards
nothing, and both complete with success. The target announces no DMRSL,
and hosts other than Linux (and nvme dsm passthrough) do send ranges
that big.

Widen before the shift.

Fixes: a07b4970f464 ("nvmet: add a generic NVMe target")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/io-cmd-bdev.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c
index f2d9e8901df4..ed2e2c89e3e9 100644
--- a/drivers/nvme/target/io-cmd-bdev.c
+++ b/drivers/nvme/target/io-cmd-bdev.c
@@ -389,7 +389,8 @@ static void nvmet_bdev_execute_discard(struct nvmet_req *req)
 		if (status)
 			break;
 
-		nr_sects = le32_to_cpu(range.nlb) << (ns->blksize_shift - 9);
+		nr_sects = (sector_t)le32_to_cpu(range.nlb) <<
+			   (ns->blksize_shift - 9);
 		__blkdev_issue_discard(ns->bdev,
 				nvmet_lba_to_sect(ns, range.slba), nr_sects,
 				GFP_KERNEL, &bio);
-- 
2.56.0

Re: [PATCH 45/98] nvmet: don't truncate the length of big discards

[Hannes Reinecke]

On 10/28/26 5:41 PM, Jens Axboe wrote:
> From: Josef Bacik <josef@toxicpanda.com>
> 
> nvmet_bdev_execute_discard() shifts the 32-bit nlb of a DSM range into
> sectors before widening it to sector_t, so on a 4k namespace the top
> bits are lost. A 3 TiB range discards 1 TiB, a 2 TiB range discards
> nothing, and both complete with success. The target announces no DMRSL,
> and hosts other than Linux (and nvme dsm passthrough) do send ranges
> that big.
> 
> Widen before the shift.
> 
> Fixes: a07b4970f464 ("nvmet: add a generic NVMe target")
> Cc: Christoph Hellwig <hch@lst.de>
> Cc: Sagi Grimberg <sagi@grimberg.me>
> Cc: Chaitanya Kulkarni <kch@nvidia.com>
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> Signed-off-by: Jens Axboe <axboe@kernel.dk>
> ---
>   drivers/nvme/target/io-cmd-bdev.c | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c
> index f2d9e8901df4..ed2e2c89e3e9 100644
> --- a/drivers/nvme/target/io-cmd-bdev.c
> +++ b/drivers/nvme/target/io-cmd-bdev.c
> @@ -389,7 +389,8 @@ static void nvmet_bdev_execute_discard(struct nvmet_req *req)
>   		if (status)
>   			break;
>   
> -		nr_sects = le32_to_cpu(range.nlb) << (ns->blksize_shift - 9);
> +		nr_sects = (sector_t)le32_to_cpu(range.nlb) <<
> +			   (ns->blksize_shift - 9);
>   		__blkdev_issue_discard(ns->bdev,
>   				nvmet_lba_to_sect(ns, range.slba), nr_sects,
>   				GFP_KERNEL, &bio);

Reviewed-by: Hannes Reinecke <hare@kernel.org>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

[PATCH 46/98] nvme-loop: delete the controller if a reset fails

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

If the reconnect of a reset fails, nvme_loop_reset_ctrl_work() tears the
controller down by hand with nvme_uninit_ctrl(). Since that was split
from nvme_stop_ctrl() it no longer removes the namespaces, which hold
references to the controller. A reset while the target no longer accepts
the host (allow_any_host 0) is enough to get there: a sysfs reset hangs
in kernfs_drain(), an ioctl reset leaves a controller stuck in
CONNECTING with its block devices, the cntlid can't be reused, and
module unload deletes the controller a second time and hits a
use-after-free.

Do what the other fabrics drivers do and call nvme_delete_ctrl(). The
delete path quiesces the admin queue and only unquiesces it if the queue
is live, which it isn't after a failed reset, so unquiesce it
unconditionally before destroying it so a stuck request can't block the
freeze.

Fixes: d09f2b45f346 ("nvme: split nvme_uninit_ctrl into stop and uninit")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/loop.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index fcb1f8186fdd..93ba6357a850 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -422,6 +422,8 @@ static void nvme_loop_shutdown_ctrl(struct nvme_loop_ctrl *ctrl)
 static void nvme_loop_delete_ctrl_host(struct nvme_ctrl *ctrl)
 {
 	nvme_loop_shutdown_ctrl(to_loop_ctrl(ctrl));
+	/* Left quiesced if a failed reset has left no admin queue behind */
+	nvme_unquiesce_admin_queue(ctrl);
 	nvme_remove_admin_tag_set(ctrl);
 }
 
@@ -485,9 +487,8 @@ static void nvme_loop_reset_ctrl_work(struct work_struct *work)
 	nvme_cancel_admin_tagset(&ctrl->ctrl);
 	nvme_loop_destroy_admin_queue(ctrl);
 out_disable:
-	nvme_remove_admin_tag_set(&ctrl->ctrl);
 	dev_warn(ctrl->ctrl.device, "Removing after reset failure\n");
-	nvme_uninit_ctrl(&ctrl->ctrl);
+	nvme_delete_ctrl(&ctrl->ctrl);
 }
 
 static const struct nvme_ctrl_ops nvme_loop_ctrl_ops = {
-- 
2.56.0

[PATCH 47/98] nvme: refuse metadata sizes above 255 bytes

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

The metadata size of an LBA format is a 16-bit field, but
blk_integrity->metadata_size is an unsigned char. nvme_init_integrity()
truncates: a 264-byte format tells the block layer 8, so the integrity
buffer is 8 bytes per block while the controller transfers 264, and a
read overflows it. 256 bytes becomes 0 and every request fails in
nvme_setup_rw().

Treat such a format like the other unsupported ones: capacity 0,
passthrough only.

Fixes: a48f041d91bf ("block: Reduce the size of struct blk_integrity")
Cc: Keith Busch <kbusch@kernel.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/host/core.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index af71d02b8476..69d3ba98d50b 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1867,6 +1867,10 @@ static bool nvme_init_integrity(struct nvme_ns_head *head,
 	    !(head->features & NVME_NS_METADATA_SUPPORTED))
 		return nvme_ns_has_pi(head);
 
+	/* struct blk_integrity has eight bits for the metadata size. */
+	if (head->ms > U8_MAX)
+		return false;
+
 	switch (head->pi_type) {
 	case NVME_NS_DPS_PI_TYPE3:
 		switch (head->guard_type) {
-- 
2.56.0

Re: [PATCH 47/98] nvme: refuse metadata sizes above 255 bytes

[Keith Busch]

On Wed, Oct 28, 2026 at 10:41:50AM -0600, Jens Axboe wrote:
> +	/* struct blk_integrity has eight bits for the metadata size. */
> +	if (head->ms > U8_MAX)
> +		return false;

I've never seen a device with 256 or greater metadata, but if the specs
allow it, should the block layer accomodate? It's just the queue_limits,
so widening the type isn't a big deal; it fits in the existing padding
anyway so the struct doesn't even have to grow.

There's potential issues with logical_block_metadata_cap, but that's
just user space reporting and doesn't affect the kernel behavior. Seems
better than throttling the capacity to 0 for an unsupported format that
we can totally support.

[PATCH 48/98] nvmet-tcp: rewind the iterator after non-data records

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

A TLS record that isn't application data (an Alert, say) lands in
whatever buffer was waiting for a PDU header, command data or a digest.
nvmet_tcp_tls_record_ok() fails the receive and the bytes aren't
counted, but the iterator has already advanced. tls_alert_recv() then
expects the iterator at the start of the alert, and if the alert filled
the kvec it has stepped off the end of it:

  BUG: KASAN: stack-out-of-bounds in tls_alert_recv+0x33/0x170
   nvmet_tcp_tls_record_ok+0x139/0x1a0
   nvmet_tcp_try_recv_pdu+0x37f/0x12a0

Six bytes of a header followed by an alert from the host is all it
takes. For command data the following bytes are stored behind the
dropped record and the command runs before its data is complete.

Rewind the iterator by what was received.

Fixes: a1c5dd8355b1 ("nvmet-tcp: control messages for recvmsg()")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/tcp.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index c38ea3501ea5..a0d942c34d3a 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -1174,7 +1174,7 @@ static inline bool nvmet_tcp_pdu_valid(u8 type)
 }
 
 static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,
-		struct msghdr *msg, char *cbuf)
+		struct msghdr *msg, char *cbuf, int len)
 {
 	struct cmsghdr *cmsg = (struct cmsghdr *)cbuf;
 	u8 ctype, level, description;
@@ -1187,6 +1187,7 @@ static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,
 	case TLS_RECORD_TYPE_DATA:
 		break;
 	case TLS_RECORD_TYPE_ALERT:
+		iov_iter_revert(&msg->msg_iter, len);
 		tls_alert_recv(queue->sock->sk, msg, &level, &description);
 		if (level == TLS_ALERT_LEVEL_FATAL) {
 			pr_err("queue %d: TLS Alert desc %u\n",
@@ -1200,6 +1201,7 @@ static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,
 		break;
 	default:
 		/* discard this record type */
+		iov_iter_revert(&msg->msg_iter, len);
 		pr_err("queue %d: TLS record %d unhandled\n",
 		       queue->idx, ctype);
 		ret = -EAGAIN;
@@ -1228,7 +1230,7 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)
 	if (unlikely(len < 0))
 		return len;
 	if (queue->tls_pskid) {
-		ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf);
+		ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf, len);
 		if (ret < 0)
 			return ret;
 	}
@@ -1290,7 +1292,7 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)
 			return len;
 		if (queue->tls_pskid) {
 			ret = nvmet_tcp_tls_record_ok(cmd->queue,
-					&cmd->recv_msg, cmd->recv_cbuf);
+					&cmd->recv_msg, cmd->recv_cbuf, len);
 			if (ret < 0)
 				return ret;
 		}
@@ -1331,7 +1333,7 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue)
 	if (unlikely(len < 0))
 		return len;
 	if (queue->tls_pskid) {
-		ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf);
+		ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf, len);
 		if (ret < 0)
 			return ret;
 	}
@@ -1803,7 +1805,7 @@ static int nvmet_tcp_try_peek_pdu(struct nvmet_tcp_queue *queue)
 		return len;
 	}
 
-	ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf);
+	ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf, len);
 	if (ret < 0)
 		return ret;
 
-- 
2.56.0

Re: [PATCH 48/98] nvmet-tcp: rewind the iterator after non-data records

[Bart Van Assche]

On 10/28/26 09:41, Jens Axboe wrote:
>  	case TLS_RECORD_TYPE_ALERT:
> +		iov_iter_revert(&msg->msg_iter, len);
>  		tls_alert_recv(queue->sock->sk, msg, &level, &description);
...
>  		if (queue->tls_pskid) {
>  			ret = nvmet_tcp_tls_record_ok(cmd->queue,
> -					&cmd->recv_msg, cmd->recv_cbuf);
> +					&cmd->recv_msg, cmd->recv_cbuf, len);

Rewinding msg_iter fixes the iterator advance, but TLS_RECORD_TYPE_ALERT
is still broken in two ways here:

1. In nvmet_tcp_try_recv_data(), cmd->recv_msg.msg_iter is an ITER_BVEC
   initialized with iov_iter_bvec(), whereas tls_alert_recv()
   (net/handshake/alert.c) unconditionally reads union member
   msg->msg_iter.kvec->iov_base. For an ITER_BVEC, iov->iov_base aliases
   bvec->bv_page (struct page *), so tls_alert_recv() reads the first
   two bytes of struct page instead of the received alert payload and
   silently ignores fatal alerts when byte 0 != TLS_ALERT_LEVEL_FATAL.
   (Also, up to 2 bytes of the command's SG page were overwritten before
   iov_iter_revert().)

2. If queue->left == 1 when a 2-byte TLS alert arrives, sock_recvmsg()
   returns len == 1, and tls_alert_recv() reads data[0] and data[1]
   without checking len >= 2.

tls_alert_recv() either needs to handle non-kvec iterators and len < 2
properly, or nvmet-tcp should treat any TLS alert record during live
PDU/data transfer as fatal without calling tls_alert_recv() on
cmd->recv_msg.

Re: [PATCH 48/98] nvmet-tcp: rewind the iterator after non-data records

[Hannes Reinecke]

On 10/28/26 8:57 PM, Bart Van Assche wrote:
> On 10/28/26 09:41, Jens Axboe wrote:
>>   	case TLS_RECORD_TYPE_ALERT:
>> +		iov_iter_revert(&msg->msg_iter, len);
>>   		tls_alert_recv(queue->sock->sk, msg, &level, &description);
> ...
>>   		if (queue->tls_pskid) {
>>   			ret = nvmet_tcp_tls_record_ok(cmd->queue,
>> -					&cmd->recv_msg, cmd->recv_cbuf);
>> +					&cmd->recv_msg, cmd->recv_cbuf, len);
> 
> Rewinding msg_iter fixes the iterator advance, but TLS_RECORD_TYPE_ALERT
> is still broken in two ways here:
> 
> 1. In nvmet_tcp_try_recv_data(), cmd->recv_msg.msg_iter is an ITER_BVEC
>     initialized with iov_iter_bvec(), whereas tls_alert_recv()
>     (net/handshake/alert.c) unconditionally reads union member
>     msg->msg_iter.kvec->iov_base. For an ITER_BVEC, iov->iov_base aliases
>     bvec->bv_page (struct page *), so tls_alert_recv() reads the first
>     two bytes of struct page instead of the received alert payload and
>     silently ignores fatal alerts when byte 0 != TLS_ALERT_LEVEL_FATAL.
>     (Also, up to 2 bytes of the command's SG page were overwritten before
>     iov_iter_revert().)
> 
> 2. If queue->left == 1 when a 2-byte TLS alert arrives, sock_recvmsg()
>     returns len == 1, and tls_alert_recv() reads data[0] and data[1]
>     without checking len >= 2.
> 
> tls_alert_recv() either needs to handle non-kvec iterators and len < 2
> properly, or nvmet-tcp should treat any TLS alert record during live
> PDU/data transfer as fatal without calling tls_alert_recv() on
> cmd->recv_msg.
> 
There is a larger rework under way for handling TLS alerts, so we
should defer this until the rework is merged.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

Re: [PATCH 48/98] nvmet-tcp: rewind the iterator after non-data records

[Bart Van Assche]

On 10/28/26 09:41, Jens Axboe wrote:
>  	case TLS_RECORD_TYPE_ALERT:
> +		iov_iter_revert(&msg->msg_iter, len);
>  		tls_alert_recv(queue->sock->sk, msg, &level, &description);
...
>  		if (queue->tls_pskid) {
>  			ret = nvmet_tcp_tls_record_ok(cmd->queue,
> -					&cmd->recv_msg, cmd->recv_cbuf);
> +					&cmd->recv_msg, cmd->recv_cbuf, len);

Rewinding msg_iter fixes the iterator advance, but TLS_RECORD_TYPE_ALERT
is still broken in two ways here:

1. In nvmet_tcp_try_recv_data(), cmd->recv_msg.msg_iter is an ITER_BVEC
   initialized with iov_iter_bvec(), whereas tls_alert_recv() in
   net/handshake/alert.c unconditionally reads union member
   msg->msg_iter.kvec->iov_base. For an ITER_BVEC, iov->iov_base aliases
   bvec->bv_page (struct page *), so tls_alert_recv() reads the first
   two bytes of struct page instead of the received alert payload and
   ignores fatal alerts whenever byte 0 != TLS_ALERT_LEVEL_FATAL.
2. If queue->left == 1 when a 2-byte TLS alert arrives (or a peer sends
   a 1-byte malformed alert record), sock_recvmsg() returns len == 1,
   and tls_alert_recv() reads data[0] and data[1] without checking that
   2 bytes were received.

To fix this inside nvmet-tcp without cross-tree changes to
net/handshake/alert.c, nvmet_tcp_tls_record_ok() can extract the alert
bytes from a local ITER_SOURCE copy of the rewound msg->msg_iter via
copy_from_iter_full(), pull the second byte from queue->sock when
len == 1 and !(msg->msg_flags & (MSG_EOR | MSG_PEEK)), reject < 2-byte
alerts with -EPROTO, and pass a local 2-byte ITER_KVEC msghdr to
tls_alert_recv() so its tracepoint still works.

Note: patch 51/98 will need a trivial context adjustment around the
nvmet_tcp_tls_alert_recv() call in nvmet_tcp_tls_record_ok() (and
propagating MSG_EOR from rem_msg lets 51/98's nvmet_tcp_tls_drop_rest()
check work unchanged).

Here is a replacement patch (checked with checkpatch.pl and
syntax-checked with Clang; not runtime-tested):

---
diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index c38ea3501ea5..e6eadfc35d01 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -1173,8 +1173,54 @@ static inline bool nvmet_tcp_pdu_valid(u8 type)
 	return false;
 }
 
+static int nvmet_tcp_tls_alert_recv(struct nvmet_tcp_queue *queue,
+		struct msghdr *msg, int len, u8 *level, u8 *description)
+{
+	struct iov_iter iter = msg->msg_iter;
+	size_t n = min_t(size_t, len, 2);
+	u8 alert[2];
+	struct kvec kvec = {
+		.iov_base = alert,
+		.iov_len = sizeof(alert),
+	};
+	struct msghdr alert_msg = {};
+
+	iter.data_source = ITER_SOURCE;
+	if (!copy_from_iter_full(alert, n, &iter))
+		return -EPROTO;
+
+	if (n < sizeof(alert)) {
+		char cbuf[CMSG_LEN(sizeof(char))] = {};
+		struct kvec rem = {
+			.iov_base = alert + n,
+			.iov_len = sizeof(alert) - n,
+		};
+		struct msghdr rem_msg = {
+			.msg_control = cbuf,
+			.msg_controllen = sizeof(cbuf),
+			.msg_flags = MSG_DONTWAIT,
+		};
+		int ret;
+
+		if (msg->msg_flags & (MSG_EOR | MSG_PEEK))
+			return -EPROTO;
+		ret = kernel_recvmsg(queue->sock, &rem_msg, &rem, 1,
+				     rem.iov_len, rem_msg.msg_flags);
+		if (ret != rem.iov_len ||
+		    tls_get_record_type(queue->sock->sk,
+					(struct cmsghdr *)cbuf) !=
+		    TLS_RECORD_TYPE_ALERT)
+			return -EPROTO;
+		msg->msg_flags |= rem_msg.msg_flags & MSG_EOR;
+	}
+
+	iov_iter_kvec(&alert_msg.msg_iter, ITER_DEST, &kvec, 1, sizeof(alert));
+	tls_alert_recv(queue->sock->sk, &alert_msg, level, description);
+	return 0;
+}
+
 static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,
-		struct msghdr *msg, char *cbuf)
+		struct msghdr *msg, char *cbuf, int len)
 {
 	struct cmsghdr *cmsg = (struct cmsghdr *)cbuf;
 	u8 ctype, level, description;
@@ -1187,7 +1233,11 @@ static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,
 	case TLS_RECORD_TYPE_DATA:
 		break;
 	case TLS_RECORD_TYPE_ALERT:
-		tls_alert_recv(queue->sock->sk, msg, &level, &description);
+		iov_iter_revert(&msg->msg_iter, len);
+		ret = nvmet_tcp_tls_alert_recv(queue, msg, len, &level,
+					       &description);
+		if (ret)
+			break;
 		if (level == TLS_ALERT_LEVEL_FATAL) {
 			pr_err("queue %d: TLS Alert desc %u\n",
 			       queue->idx, description);
@@ -1200,6 +1250,7 @@ static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,
 		break;
 	default:
 		/* discard this record type */
+		iov_iter_revert(&msg->msg_iter, len);
 		pr_err("queue %d: TLS record %d unhandled\n",
 		       queue->idx, ctype);
 		ret = -EAGAIN;
@@ -1228,7 +1279,7 @@ recv:
 	if (unlikely(len < 0))
 		return len;
 	if (queue->tls_pskid) {
-		ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf);
+		ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf, len);
 		if (ret < 0)
 			return ret;
 	}
@@ -1290,7 +1341,7 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)
 			return len;
 		if (queue->tls_pskid) {
 			ret = nvmet_tcp_tls_record_ok(cmd->queue,
-					&cmd->recv_msg, cmd->recv_cbuf);
+					&cmd->recv_msg, cmd->recv_cbuf, len);
 			if (ret < 0)
 				return ret;
 		}
@@ -1331,7 +1382,7 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue)
 	if (unlikely(len < 0))
 		return len;
 	if (queue->tls_pskid) {
-		ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf);
+		ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf, len);
 		if (ret < 0)
 			return ret;
 	}
@@ -1803,7 +1854,7 @@ static int nvmet_tcp_try_peek_pdu(struct nvmet_tcp_queue *queue)
 		return len;
 	}
 
-	ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf);
+	ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf, len);
 	if (ret < 0)
 		return ret;
 

[PATCH 49/98] nvmet-tcp: ask for the TLS record type on every receive

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

nvmet_tcp_alloc_cmd() points recv_msg.msg_control at recv_cbuf once.
The first receive that stores a record type there advances msg_control
and leaves msg_controllen at 0 for the lifetime of the command. From
then on nvmet_tcp_try_recv_data() sees the record type of that first
receive: if it was an alert, all later data is dropped as alerts and the
queue stalls, if it was data, a later alert can't be reported and
tls_sw_recvmsg() fails with -EIO.

Set up the control buffer before each receive, like the other receive
paths do.

Fixes: a1c5dd8355b1 ("nvmet-tcp: control messages for recvmsg()")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/tcp.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index a0d942c34d3a..261672ab5c1f 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -1286,6 +1286,11 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)
 	int len, ret;
 
 	while (msg_data_left(&cmd->recv_msg)) {
+		if (queue->tls_pskid) {
+			/* The last receive has used it up */
+			cmd->recv_msg.msg_control = cmd->recv_cbuf;
+			cmd->recv_msg.msg_controllen = sizeof(cmd->recv_cbuf);
+		}
 		len = sock_recvmsg(cmd->queue->sock, &cmd->recv_msg,
 			cmd->recv_msg.msg_flags);
 		if (len <= 0)
-- 
2.56.0

Re: [PATCH 49/98] nvmet-tcp: ask for the TLS record type on every receive

[Hannes Reinecke]

On 10/28/26 5:41 PM, Jens Axboe wrote:
> From: Josef Bacik <josef@toxicpanda.com>
> 
> nvmet_tcp_alloc_cmd() points recv_msg.msg_control at recv_cbuf once.
> The first receive that stores a record type there advances msg_control
> and leaves msg_controllen at 0 for the lifetime of the command. From
> then on nvmet_tcp_try_recv_data() sees the record type of that first
> receive: if it was an alert, all later data is dropped as alerts and the
> queue stalls, if it was data, a later alert can't be reported and
> tls_sw_recvmsg() fails with -EIO.
> 
> Set up the control buffer before each receive, like the other receive
> paths do.
> 
> Fixes: a1c5dd8355b1 ("nvmet-tcp: control messages for recvmsg()")
> Cc: Christoph Hellwig <hch@lst.de>
> Cc: Sagi Grimberg <sagi@grimberg.me>
> Cc: Chaitanya Kulkarni <kch@nvidia.com>
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> Signed-off-by: Jens Axboe <axboe@kernel.dk>
> ---
>   drivers/nvme/target/tcp.c | 5 +++++
>   1 file changed, 5 insertions(+)
> 
> diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
> index a0d942c34d3a..261672ab5c1f 100644
> --- a/drivers/nvme/target/tcp.c
> +++ b/drivers/nvme/target/tcp.c
> @@ -1286,6 +1286,11 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)
>   	int len, ret;
>   
>   	while (msg_data_left(&cmd->recv_msg)) {
> +		if (queue->tls_pskid) {
> +			/* The last receive has used it up */
> +			cmd->recv_msg.msg_control = cmd->recv_cbuf;
> +			cmd->recv_msg.msg_controllen = sizeof(cmd->recv_cbuf);
> +		}
>   		len = sock_recvmsg(cmd->queue->sock, &cmd->recv_msg,
>   			cmd->recv_msg.msg_flags);
>   		if (len <= 0)

Reviewed-by: Hannes Reinecke <hare@kernel.org>

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

[PATCH 50/98] nvme-tcp: wait for an in-flight TLS handshake callback

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

If the wait for the TLS handshake times out or is interrupted,
nvme_tcp_start_tls() cancels the request and returns.
tls_handshake_cancel() returns false if the handshake already completed,
in which case nvme_tcp_tls_done() may still be running while
nvme_tcp_alloc_queue() tears the queue down. On the first connect that
frees the controller, including the queue array the callback writes to.

If the cancel lost, wait for the callback to signal completion before
returning, like the SUNRPC callers do.

Fixes: be8e82caa685 ("nvme-tcp: enable TLS handshake upcall")
Cc: Keith Busch <kbusch@kernel.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/host/tcp.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 7190eb10c248..d735994ee517 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1827,7 +1827,12 @@ static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl,
 		dev_err(nctrl->device,
 			"queue %d: TLS handshake failed, error %d\n",
 			qid, ret);
-		tls_handshake_cancel(queue->sock->sk);
+		/*
+		 * If the cancellation lost to handshake_complete(),
+		 * nvme_tcp_tls_done() is in flight and still uses the queue.
+		 */
+		if (!tls_handshake_cancel(queue->sock->sk))
+			wait_for_completion(&queue->tls_complete);
 	} else {
 		if (queue->tls_err) {
 			dev_err(nctrl->device,
-- 
2.56.0

Re: [PATCH 50/98] nvme-tcp: wait for an in-flight TLS handshake callback

[Hannes Reinecke]

On 10/28/26 5:41 PM, Jens Axboe wrote:
> From: Josef Bacik <josef@toxicpanda.com>
> 
> If the wait for the TLS handshake times out or is interrupted,
> nvme_tcp_start_tls() cancels the request and returns.
> tls_handshake_cancel() returns false if the handshake already completed,
> in which case nvme_tcp_tls_done() may still be running while
> nvme_tcp_alloc_queue() tears the queue down. On the first connect that
> frees the controller, including the queue array the callback writes to.
> 
> If the cancel lost, wait for the callback to signal completion before
> returning, like the SUNRPC callers do.
> 
> Fixes: be8e82caa685 ("nvme-tcp: enable TLS handshake upcall")
> Cc: Keith Busch <kbusch@kernel.org>
> Cc: Christoph Hellwig <hch@lst.de>
> Cc: Sagi Grimberg <sagi@grimberg.me>
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> Signed-off-by: Jens Axboe <axboe@kernel.dk>
> ---
>   drivers/nvme/host/tcp.c | 7 ++++++-
>   1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
> index 7190eb10c248..d735994ee517 100644
> --- a/drivers/nvme/host/tcp.c
> +++ b/drivers/nvme/host/tcp.c
> @@ -1827,7 +1827,12 @@ static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl,
>   		dev_err(nctrl->device,
>   			"queue %d: TLS handshake failed, error %d\n",
>   			qid, ret);
> -		tls_handshake_cancel(queue->sock->sk);
> +		/*
> +		 * If the cancellation lost to handshake_complete(),
> +		 * nvme_tcp_tls_done() is in flight and still uses the queue.
> +		 */
> +		if (!tls_handshake_cancel(queue->sock->sk))
> +			wait_for_completion(&queue->tls_complete);
>   	} else {
>   		if (queue->tls_err) {
>   			dev_err(nctrl->device,

This has already been reported to the linux-nvme list
(https://lore.kernel.org/linux-nvme/20261022071934.3818224-1-thnkslprpt@gmail.com/)
Please drop.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

[PATCH 51/98] nvmet-tcp: look again after a short read with TLS

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

The receive helpers return -EAGAIN both for a short read and for a
dropped non-data TLS record, and nvmet_tcp_try_recv() takes that as
"socket empty" and waits for the next data_ready. On plain TCP that's
right. tls_sw_recvmsg() stops where the record type changes though, so
what follows an alert or a handshake record may already be queued and
nobody looks at it until the host sends something else.

Return 1 in those cases so nvmet_tcp_try_recv() counts it against the
budget and tries again until the socket itself says -EAGAIN. While at
it, drain the rest of a dropped record using MSG_EOR instead of pulling
it in PDU-sized pieces, and ratelimit the messages since the host
controls how many there are.

Fixes: a1c5dd8355b1 ("nvmet-tcp: control messages for recvmsg()")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/tcp.c | 54 +++++++++++++++++++++++++++++++--------
 1 file changed, 43 insertions(+), 11 deletions(-)

diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index 261672ab5c1f..a2a5f39d2bb7 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -1173,6 +1173,24 @@ static inline bool nvmet_tcp_pdu_valid(u8 type)
 	return false;
 }
 
+/*
+ * The buffer took as much of the record as the PDU still wants, which may be
+ * one byte.  The rest comes with the type of the record, never with that
+ * which follows: throw it away at once.
+ */
+static void nvmet_tcp_tls_drop_rest(struct nvmet_tcp_queue *queue)
+{
+	char cbuf[CMSG_LEN(sizeof(char))] = {};
+	struct msghdr msg = {
+		.msg_control = cbuf,
+		.msg_controllen = sizeof(cbuf),
+		.msg_flags = MSG_DONTWAIT,
+	};
+
+	iov_iter_discard(&msg.msg_iter, ITER_DEST, TLS_MAX_PAYLOAD_SIZE);
+	sock_recvmsg(queue->sock, &msg, msg.msg_flags);
+}
+
 static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,
 		struct msghdr *msg, char *cbuf, int len)
 {
@@ -1190,26 +1208,38 @@ static int nvmet_tcp_tls_record_ok(struct nvmet_tcp_queue *queue,
 		iov_iter_revert(&msg->msg_iter, len);
 		tls_alert_recv(queue->sock->sk, msg, &level, &description);
 		if (level == TLS_ALERT_LEVEL_FATAL) {
-			pr_err("queue %d: TLS Alert desc %u\n",
-			       queue->idx, description);
+			pr_err_ratelimited("queue %d: TLS Alert desc %u\n",
+					   queue->idx, description);
 			ret = -ENOTCONN;
 		} else {
-			pr_warn("queue %d: TLS Alert desc %u\n",
-			       queue->idx, description);
+			pr_warn_ratelimited("queue %d: TLS Alert desc %u\n",
+					    queue->idx, description);
 			ret = -EAGAIN;
 		}
 		break;
 	default:
 		/* discard this record type */
 		iov_iter_revert(&msg->msg_iter, len);
-		pr_err("queue %d: TLS record %d unhandled\n",
-		       queue->idx, ctype);
+		pr_err_ratelimited("queue %d: TLS record %d unhandled\n",
+				   queue->idx, ctype);
 		ret = -EAGAIN;
 		break;
 	}
+	if (ret == -EAGAIN && !(msg->msg_flags & MSG_EOR))
+		nvmet_tcp_tls_drop_rest(queue);
 	return ret;
 }
 
+/*
+ * A receive from a TLS socket ends with the record, or where the type of the
+ * records changes.  There may be more behind a record that has been dropped,
+ * and nobody will tell: have nvmet_tcp_try_recv() look again.
+ */
+static int nvmet_tcp_tls_again(int ret)
+{
+	return ret == -EAGAIN ? 1 : ret;
+}
+
 static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)
 {
 	struct nvme_tcp_hdr *hdr = &queue->pdu.cmd.hdr;
@@ -1224,6 +1254,7 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)
 	if (queue->tls_pskid) {
 		msg.msg_control = cbuf;
 		msg.msg_controllen = sizeof(cbuf);
+		msg.msg_flags &= ~MSG_EOR;
 	}
 	len = kernel_recvmsg(queue->sock, &msg, &iov, 1,
 			iov.iov_len, msg.msg_flags);
@@ -1232,13 +1263,13 @@ static int nvmet_tcp_try_recv_pdu(struct nvmet_tcp_queue *queue)
 	if (queue->tls_pskid) {
 		ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf, len);
 		if (ret < 0)
-			return ret;
+			return nvmet_tcp_tls_again(ret);
 	}
 
 	queue->offset += len;
 	queue->left -= len;
 	if (queue->left)
-		return -EAGAIN;
+		return queue->tls_pskid && len ? 1 : -EAGAIN;
 
 	if (queue->offset == sizeof(struct nvme_tcp_hdr)) {
 		u8 hdgst = nvmet_tcp_hdgst_len(queue);
@@ -1290,6 +1321,7 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)
 			/* The last receive has used it up */
 			cmd->recv_msg.msg_control = cmd->recv_cbuf;
 			cmd->recv_msg.msg_controllen = sizeof(cmd->recv_cbuf);
+			cmd->recv_msg.msg_flags &= ~MSG_EOR;
 		}
 		len = sock_recvmsg(cmd->queue->sock, &cmd->recv_msg,
 			cmd->recv_msg.msg_flags);
@@ -1299,7 +1331,7 @@ static int nvmet_tcp_try_recv_data(struct nvmet_tcp_queue *queue)
 			ret = nvmet_tcp_tls_record_ok(cmd->queue,
 					&cmd->recv_msg, cmd->recv_cbuf, len);
 			if (ret < 0)
-				return ret;
+				return nvmet_tcp_tls_again(ret);
 		}
 
 		cmd->pdu_recv += len;
@@ -1340,13 +1372,13 @@ static int nvmet_tcp_try_recv_ddgst(struct nvmet_tcp_queue *queue)
 	if (queue->tls_pskid) {
 		ret = nvmet_tcp_tls_record_ok(queue, &msg, cbuf, len);
 		if (ret < 0)
-			return ret;
+			return nvmet_tcp_tls_again(ret);
 	}
 
 	queue->offset += len;
 	queue->left -= len;
 	if (queue->left)
-		return -EAGAIN;
+		return queue->tls_pskid && len ? 1 : -EAGAIN;
 
 	if (queue->data_digest && cmd->exp_ddgst != cmd->recv_ddgst) {
 		pr_err("queue %d: cmd %d pdu (%d) data digest error: recv %#x expected %#x\n",
-- 
2.56.0

[PATCH 52/98] nvme-tcp: keep the socket while the queue is polled

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

Quiescing keeps ->queue_rq() out, not ->poll(): bio_poll() enters even
a frozen queue. So nvme_tcp_poll() can run while error recovery or a
reset stops and frees the queues, and keeps running afterwards for the
requeued requests whose bios still carry a cookie. The NVME_TCP_Q_LIVE
check doesn't help, queue->sock->sk is read before it and
nvme_tcp_free_queue() has set queue->sock to NULL:

  BUG: KASAN: null-ptr-deref in nvme_tcp_poll+0x42/0x1c0

The poller dies holding a queue reference and the controller never
reconnects.

nvme_tcp_stop_queue() clears the bit under queue_lock. Trylock it in
nvme_tcp_poll() too and check the bit and the socket under it; whoever
holds it is either polling or stopping this queue. That means the lock
can't live with the queue, so initialize it with the controller.

Fixes: 1a9460cef571 ("nvme-tcp: support simple polling")
Cc: Keith Busch <kbusch@kernel.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/host/tcp.c | 29 ++++++++++++++++++++++-------
 1 file changed, 22 insertions(+), 7 deletions(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index d735994ee517..98c55fba53c6 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1530,7 +1530,6 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
 
 	kfree(queue->pdu);
 	mutex_destroy(&queue->send_mutex);
-	mutex_destroy(&queue->queue_lock);
 	mutex_destroy(&queue->pf_cache_lock);
 }
 
@@ -1855,7 +1854,6 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
 	int ret, rcv_pdu_size;
 	struct file *sock_file;
 
-	mutex_init(&queue->queue_lock);
 	queue->ctrl = ctrl;
 	init_llist_head(&queue->req_list);
 	INIT_LIST_HEAD(&queue->send_list);
@@ -1993,7 +1991,6 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
 	queue->sock = NULL;
 err_destroy_mutex:
 	mutex_destroy(&queue->send_mutex);
-	mutex_destroy(&queue->queue_lock);
 	mutex_destroy(&queue->pf_cache_lock);
 	return ret;
 }
@@ -2922,18 +2919,33 @@ static void nvme_tcp_map_queues(struct blk_mq_tag_set *set)
 static int nvme_tcp_poll(struct blk_mq_hw_ctx *hctx, struct io_comp_batch *iob)
 {
 	struct nvme_tcp_queue *queue = hctx->driver_data;
-	struct sock *sk = queue->sock->sk;
-	int ret;
+	struct sock *sk;
+	int ret = 0;
 
 	if (!test_bit(NVME_TCP_Q_LIVE, &queue->flags))
 		return 0;
 
+	/*
+	 * Nothing in the block layer keeps a poller out while the queue is
+	 * torn down.  The lock keeps the socket until we are through; if
+	 * somebody else has it, he polls, or the queue is being stopped.
+	 */
+	if (!mutex_trylock(&queue->queue_lock))
+		return 0;
+	if (!test_bit(NVME_TCP_Q_LIVE, &queue->flags))
+		goto unlock;
+
+	sk = queue->sock->sk;
 	set_bit(NVME_TCP_Q_POLLING, &queue->flags);
 	if (sk_can_busy_loop(sk) && skb_queue_empty_lockless(&sk->sk_receive_queue))
 		sk_busy_loop(sk, true);
 	ret = nvme_tcp_try_recv(queue);
 	clear_bit(NVME_TCP_Q_POLLING, &queue->flags);
-	return ret < 0 ? ret : queue->nr_cqe;
+	if (ret >= 0)
+		ret = queue->nr_cqe;
+unlock:
+	mutex_unlock(&queue->queue_lock);
+	return ret;
 }
 
 static int nvme_tcp_get_address(struct nvme_ctrl *ctrl, char *buf, int size)
@@ -3020,7 +3032,7 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev,
 		struct nvmf_ctrl_options *opts)
 {
 	struct nvme_tcp_ctrl *ctrl;
-	int ret;
+	int ret, i;
 
 	ctrl = kzalloc_obj(*ctrl);
 	if (!ctrl)
@@ -3089,6 +3101,9 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev,
 		ret = -ENOMEM;
 		goto out_free_ctrl;
 	}
+	/* A poller may try the lock of a queue that has been freed */
+	for (i = 0; i < ctrl->ctrl.queue_count; i++)
+		mutex_init(&ctrl->queues[i].queue_lock);
 
 	ctrl->net = get_net(current->nsproxy->net_ns);
 
-- 
2.56.0

[PATCH 53/98] nvme-tcp: don't get stuck on a TLS control record

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

The host receives with read_sock(), and tls_sw_read_sock() won't hand
out a non-data record: it puts it back and returns -EINVAL.
nvme_tcp_io_work() bails on the error and finds the same record again
next time, so nothing behind it is ever read. One warning alert or
NewSessionTicket after the ICResp kills the queue: commands time out and
the controller resets. The Linux target sends none, but others may.

Fetch such a record with recvmsg() instead. A fatal alert starts error
recovery, anything else is dropped whole, with a log line, and the queue
carries on. io_work and a poller may both see the record, so a bit makes
sure only one of them deals with it, and that one retries read_sock()
afterwards.

Fixes: be8e82caa685 ("nvme-tcp: enable TLS handshake upcall")
Cc: Keith Busch <kbusch@kernel.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/host/tcp.c | 87 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 86 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 98c55fba53c6..87ecee0665c7 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -95,6 +95,7 @@ enum nvme_tcp_queue_flags {
 	NVME_TCP_Q_LIVE		= 1,
 	NVME_TCP_Q_POLLING	= 2,
 	NVME_TCP_Q_IO_CPU_SET	= 3,
+	NVME_TCP_Q_TLS_CTRL	= 4,
 };
 
 enum nvme_tcp_recv_state {
@@ -1418,7 +1419,7 @@ static int nvme_tcp_try_send(struct nvme_tcp_queue *queue)
 	return ret;
 }
 
-static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue)
+static int nvme_tcp_read_sock(struct nvme_tcp_queue *queue)
 {
 	struct socket *sock = queue->sock;
 	struct sock *sk = sock->sk;
@@ -1431,6 +1432,90 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue)
 	queue->nr_cqe = 0;
 	consumed = sock->ops->read_sock(sk, &rd_desc, nvme_tcp_recv_skb);
 	release_sock(sk);
+	return consumed;
+}
+
+/*
+ * read_sock() leaves a TLS record that is no data where it is, in front of
+ * all that follows.  Take it out of the way.
+ */
+static int nvme_tcp_recv_tls_ctrl(struct nvme_tcp_queue *queue)
+{
+	struct nvme_ctrl *ctrl = &queue->ctrl->ctrl;
+	char cbuf[CMSG_LEN(sizeof(char))] = {};
+	struct msghdr msg = {
+		.msg_control = cbuf,
+		.msg_controllen = sizeof(cbuf),
+		.msg_flags = MSG_DONTWAIT,
+	};
+	u8 ctype, level, description, data[2] = {};
+	struct kvec iov = { .iov_base = data, .iov_len = sizeof(data) };
+	int len;
+
+	/*
+	 * io_work and a poller may both have met the record, and recvmsg()
+	 * would take data as well.  Only one of them goes on, and he looks
+	 * again: nobody else removes such a record from now on.
+	 */
+	if (test_and_set_bit_lock(NVME_TCP_Q_TLS_CTRL, &queue->flags))
+		return 0;
+	len = nvme_tcp_read_sock(queue);
+	if (len != -EINVAL || !queue->rd_enabled)
+		goto out;
+
+	len = kernel_recvmsg(queue->sock, &msg, &iov, 1, iov.iov_len,
+			     msg.msg_flags);
+	if (len < 0)
+		goto out;
+
+	ctype = tls_get_record_type(queue->sock->sk, (struct cmsghdr *)cbuf);
+	switch (ctype) {
+	case 0:
+	case TLS_RECORD_TYPE_DATA:
+		goto fail;
+	case TLS_RECORD_TYPE_ALERT:
+		iov_iter_revert(&msg.msg_iter, len);
+		tls_alert_recv(queue->sock->sk, &msg, &level, &description);
+		dev_warn_ratelimited(ctrl->device,
+				     "queue %d: TLS Alert level %u desc %u\n",
+				     nvme_tcp_queue_id(queue), level,
+				     description);
+		if (level == TLS_ALERT_LEVEL_FATAL)
+			goto fail;
+		break;
+	default:
+		dev_warn_ratelimited(ctrl->device,
+				     "queue %d: TLS record %u dropped\n",
+				     nvme_tcp_queue_id(queue), ctype);
+		break;
+	}
+
+	if (!(msg.msg_flags & MSG_EOR)) {
+		msg.msg_control = cbuf;
+		msg.msg_controllen = sizeof(cbuf);
+		iov_iter_discard(&msg.msg_iter, ITER_DEST,
+				 TLS_MAX_PAYLOAD_SIZE);
+		sock_recvmsg(queue->sock, &msg, MSG_DONTWAIT);
+	}
+	len = 1;
+out:
+	clear_bit_unlock(NVME_TCP_Q_TLS_CTRL, &queue->flags);
+	return len;
+
+fail:
+	queue->rd_enabled = false;
+	nvme_tcp_error_recovery(ctrl);
+	len = -ENOTCONN;
+	goto out;
+}
+
+static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue)
+{
+	int consumed = nvme_tcp_read_sock(queue);
+
+	if (consumed == -EINVAL && nvme_tcp_queue_tls(queue) &&
+	    queue->rd_enabled)
+		consumed = nvme_tcp_recv_tls_ctrl(queue);
 	return consumed == -EAGAIN ? 0 : consumed;
 }
 
-- 
2.56.0

Re: [PATCH 53/98] nvme-tcp: don't get stuck on a TLS control record

[Hannes Reinecke]

On 10/28/26 5:41 PM, Jens Axboe wrote:
> From: Josef Bacik <josef@toxicpanda.com>
> 
> The host receives with read_sock(), and tls_sw_read_sock() won't hand
> out a non-data record: it puts it back and returns -EINVAL.
> nvme_tcp_io_work() bails on the error and finds the same record again
> next time, so nothing behind it is ever read. One warning alert or
> NewSessionTicket after the ICResp kills the queue: commands time out and
> the controller resets. The Linux target sends none, but others may.
> 
> Fetch such a record with recvmsg() instead. A fatal alert starts error
> recovery, anything else is dropped whole, with a log line, and the queue
> carries on. io_work and a poller may both see the record, so a bit makes
> sure only one of them deals with it, and that one retries read_sock()
> afterwards.
> 
> Fixes: be8e82caa685 ("nvme-tcp: enable TLS handshake upcall")
> Cc: Keith Busch <kbusch@kernel.org>
> Cc: Christoph Hellwig <hch@lst.de>
> Cc: Sagi Grimberg <sagi@grimberg.me>
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> Signed-off-by: Jens Axboe <axboe@kernel.dk>
> ---
>   drivers/nvme/host/tcp.c | 87 ++++++++++++++++++++++++++++++++++++++++-
>   1 file changed, 86 insertions(+), 1 deletion(-)
> 
TLS record handling is currently being reworked, so please drop.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

[PATCH 54/98] nvme-tcp: start error recovery after a TLS KeyUpdate

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

Once the controller sends a KeyUpdate, kTLS fails every receive with
-EKEYEXPIRED until a new key is installed, and nobody on the host side
does that. nvme_tcp_io_work() returns on the error and the queue is dead
until a command or the keep-alive times out.

The connection is unusable at that point, say so and start error
recovery right away.

Fixes: 0471b1093e3a ("tls: block decryption when a rekey is pending")
Cc: Keith Busch <kbusch@kernel.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/host/tcp.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 87ecee0665c7..64f563686938 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1516,6 +1516,14 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue)
 	if (consumed == -EINVAL && nvme_tcp_queue_tls(queue) &&
 	    queue->rd_enabled)
 		consumed = nvme_tcp_recv_tls_ctrl(queue);
+	if (consumed == -EKEYEXPIRED && queue->rd_enabled) {
+		/* Nobody is going to install the new key */
+		dev_err(queue->ctrl->ctrl.device,
+			"queue %d: TLS KeyUpdate is not supported\n",
+			nvme_tcp_queue_id(queue));
+		queue->rd_enabled = false;
+		nvme_tcp_error_recovery(&queue->ctrl->ctrl);
+	}
 	return consumed == -EAGAIN ? 0 : consumed;
 }
 
-- 
2.56.0

Re: [PATCH 54/98] nvme-tcp: start error recovery after a TLS KeyUpdate

[Hannes Reinecke]

On 10/28/26 5:41 PM, Jens Axboe wrote:
> From: Josef Bacik <josef@toxicpanda.com>
> 
> Once the controller sends a KeyUpdate, kTLS fails every receive with
> -EKEYEXPIRED until a new key is installed, and nobody on the host side
> does that. nvme_tcp_io_work() returns on the error and the queue is dead
> until a command or the keep-alive times out.
> 
> The connection is unusable at that point, say so and start error
> recovery right away.
> 
> Fixes: 0471b1093e3a ("tls: block decryption when a rekey is pending")
> Cc: Keith Busch <kbusch@kernel.org>
> Cc: Christoph Hellwig <hch@lst.de>
> Cc: Sagi Grimberg <sagi@grimberg.me>
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> Signed-off-by: Jens Axboe <axboe@kernel.dk>
> ---
>   drivers/nvme/host/tcp.c | 8 ++++++++
>   1 file changed, 8 insertions(+)
> 
> diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
> index 87ecee0665c7..64f563686938 100644
> --- a/drivers/nvme/host/tcp.c
> +++ b/drivers/nvme/host/tcp.c
> @@ -1516,6 +1516,14 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue)
>   	if (consumed == -EINVAL && nvme_tcp_queue_tls(queue) &&
>   	    queue->rd_enabled)
>   		consumed = nvme_tcp_recv_tls_ctrl(queue);
> +	if (consumed == -EKEYEXPIRED && queue->rd_enabled) {
> +		/* Nobody is going to install the new key */
> +		dev_err(queue->ctrl->ctrl.device,
> +			"queue %d: TLS KeyUpdate is not supported\n",
> +			nvme_tcp_queue_id(queue));
> +		queue->rd_enabled = false;
> +		nvme_tcp_error_recovery(&queue->ctrl->ctrl);
> +	}
>   	return consumed == -EAGAIN ? 0 : consumed;
>   }
>   

Already been reported 
(https://lore.kernel.org/linux-nvme/20261027-nvme-tcp-ctrlrec-fixes-v2-0-9656395ebcf2@gmail.com/).
Please drop.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

[PATCH 55/98] nvme-tcp: don't act on PDUs for a command being sent

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

A request belongs to the send path from the moment it's put on req_list
until its last byte is handed to the socket, but the receive path acts
on whatever the controller sends for it. A response completes the
request while the send side is still reading from its bio:

  BUG: KASAN: slab-use-after-free in nvme_tcp_try_send_data+0x175/0x6b0
   nvme_tcp_try_send+0x4af/0x600
   nvme_tcp_io_work+0x249/0x380

and an R2T resets and requeues it, since the queued check only looks at
the two lists and a request being sent is on neither, so it ends up
completed twice. An honest controller doesn't do that, but its answer
can legitimately overtake a sender that has handed over the last byte
and not yet cleared queue->request.

Make such a PDU wait: don't consume the bytes that complete its header,
so it's seen again by the next receive. For that nvme_tcp_req_busy()
has to find the request in one of the three places at all times, so
nvme_tcp_fetch_request() sets queue->request before taking it off the
lists. io_work reruns when the sender was io_work or the socket was
full, waits for send_mutex when the submitter was sending, and a poller
just polls again.

Fixes: 3f2304f8c6d6 ("nvme-tcp: add NVMe over TCP host driver")
Cc: Keith Busch <kbusch@kernel.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/host/tcp.c | 87 ++++++++++++++++++++++++++++++++++-------
 1 file changed, 73 insertions(+), 14 deletions(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 64f563686938..06c205703cf5 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -123,6 +123,7 @@ struct nvme_tcp_queue {
 	size_t			data_remaining;
 	size_t			ddgst_remaining;
 	unsigned int		nr_cqe;
+	bool			rcv_parked;
 
 	/* send state */
 	struct nvme_tcp_request *request;
@@ -465,11 +466,29 @@ nvme_tcp_fetch_request(struct nvme_tcp_queue *queue)
 			return NULL;
 	}
 
+	WRITE_ONCE(queue->request, req);
+	/* nvme_tcp_req_busy() must find the request in one place at any time */
+	smp_wmb();
 	list_del_init(&req->entry);
 	init_llist_node(&req->lentry);
 	return req;
 }
 
+/*
+ * Does the send path still hold @req? As long as it does the request must
+ * neither be completed nor be queued once more, whatever the controller says.
+ */
+static bool nvme_tcp_req_busy(struct nvme_tcp_queue *queue,
+			      struct nvme_tcp_request *req)
+{
+	if (llist_on_list(&req->lentry) || !list_empty(&req->entry))
+		return true;
+	/* pairs with the barrier in nvme_tcp_fetch_request() */
+	smp_rmb();
+	/* pairs with the release in nvme_tcp_done_send_req() */
+	return smp_load_acquire(&queue->request) == req;
+}
+
 #define NVME_TCP_CRC_SEED (~0)
 
 static inline void nvme_tcp_ddgst_update(u32 *crcp,
@@ -672,6 +691,9 @@ static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,
 	}
 
 	req = blk_mq_rq_to_pdu(rq);
+	if (unlikely(nvme_tcp_req_busy(queue, req)))
+		return -EBUSY;
+
 	if (req->status == cpu_to_le16(NVME_SC_SUCCESS))
 		req->status = cqe->status;
 
@@ -707,6 +729,9 @@ static int nvme_tcp_handle_c2h_data(struct nvme_tcp_queue *queue,
 	}
 
 	req = blk_mq_rq_to_pdu(rq);
+	if (unlikely(nvme_tcp_req_busy(queue, req)))
+		return -EBUSY;
+
 	if (!blk_rq_payload_bytes(rq) || !req->curr_bio || !req->data_len) {
 		dev_err(queue->ctrl->ctrl.device,
 			"queue %d tag %#x unexpected data\n",
@@ -800,6 +825,8 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue,
 		return -ENOENT;
 	}
 	req = blk_mq_rq_to_pdu(rq);
+	if (unlikely(nvme_tcp_req_busy(queue, req)))
+		return -EBUSY;
 
 	if (unlikely(rq_data_dir(rq) != WRITE)) {
 		dev_err(queue->ctrl->ctrl.device,
@@ -829,14 +856,6 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue,
 		return -EPROTO;
 	}
 
-	if (llist_on_list(&req->lentry) ||
-	    !list_empty(&req->entry)) {
-		dev_err(queue->ctrl->ctrl.device,
-			"req %d unexpected r2t while processing request\n",
-			rq->tag);
-		return -EPROTO;
-	}
-
 	req->pdu_len = 0;
 	req->h2cdata_left = r2t_length;
 	req->h2cdata_offset = r2t_offset;
@@ -939,17 +958,32 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,
 
 	switch (hdr->type) {
 	case nvme_tcp_c2h_data:
-		return nvme_tcp_handle_c2h_data(queue, (void *)queue->pdu);
+		ret = nvme_tcp_handle_c2h_data(queue, (void *)queue->pdu);
+		break;
 	case nvme_tcp_rsp:
-		nvme_tcp_init_recv_ctx(queue);
-		return nvme_tcp_handle_comp(queue, (void *)queue->pdu);
+		ret = nvme_tcp_handle_comp(queue, (void *)queue->pdu);
+		break;
 	case nvme_tcp_r2t:
-		nvme_tcp_init_recv_ctx(queue);
-		return nvme_tcp_handle_r2t(queue, (void *)queue->pdu);
+		ret = nvme_tcp_handle_r2t(queue, (void *)queue->pdu);
+		break;
 	default:
 		goto unsupported_pdu;
 	}
 
+	if (unlikely(ret == -EBUSY)) {
+		/*
+		 * The command is still being sent. Leave the end of the header
+		 * in the socket and look at it again when that is done.
+		 */
+		queue->pdu_remaining += rcv_len;
+		queue->pdu_offset -= rcv_len;
+		*offset -= rcv_len;
+		*len += rcv_len;
+	} else if (hdr->type != nvme_tcp_c2h_data) {
+		nvme_tcp_init_recv_ctx(queue);
+	}
+	return ret;
+
 unsupported_pdu:
 	dev_err(queue->ctrl->ctrl.device,
 		"unsupported pdu type (%d)\n", hdr->type);
@@ -1110,6 +1144,10 @@ static int nvme_tcp_recv_skb(read_descriptor_t *desc, struct sk_buff *skb,
 		default:
 			result = -EFAULT;
 		}
+		if (unlikely(result == -EBUSY)) {
+			queue->rcv_parked = true;
+			return consumed - len;
+		}
 		if (result) {
 			dev_err(queue->ctrl->ctrl.device,
 				"receive failed:  %d\n", result);
@@ -1182,7 +1220,8 @@ static void nvme_tcp_state_change(struct sock *sk)
 
 static inline void nvme_tcp_done_send_req(struct nvme_tcp_queue *queue)
 {
-	queue->request = NULL;
+	/* pairs with nvme_tcp_req_busy() */
+	smp_store_release(&queue->request, NULL);
 }
 
 static void nvme_tcp_fail_request(struct nvme_tcp_request *req)
@@ -1430,6 +1469,7 @@ static int nvme_tcp_read_sock(struct nvme_tcp_queue *queue)
 	rd_desc.count = 1;
 	lock_sock(sk);
 	queue->nr_cqe = 0;
+	queue->rcv_parked = false;
 	consumed = sock->ops->read_sock(sk, &rd_desc, nvme_tcp_recv_skb);
 	release_sock(sk);
 	return consumed;
@@ -1527,6 +1567,22 @@ static int nvme_tcp_try_recv(struct nvme_tcp_queue *queue)
 	return consumed == -EAGAIN ? 0 : consumed;
 }
 
+/*
+ * A PDU has been left in the socket because its command is still being sent.
+ * A sender in another context does not wake us when it is done: wait for it.
+ * If there is something left to send then, either the socket is full and tells
+ * us when it has room, or io_work has been queued.
+ */
+static bool nvme_tcp_recv_again(struct nvme_tcp_queue *queue)
+{
+	bool again;
+
+	mutex_lock(&queue->send_mutex);
+	again = !queue->request && !nvme_tcp_queue_has_pending(queue);
+	mutex_unlock(&queue->send_mutex);
+	return again;
+}
+
 static void nvme_tcp_io_work(struct work_struct *w)
 {
 	struct nvme_tcp_queue *queue =
@@ -1552,6 +1608,9 @@ static void nvme_tcp_io_work(struct work_struct *w)
 		else if (unlikely(result < 0))
 			return;
 
+		if (unlikely(queue->rcv_parked) && nvme_tcp_recv_again(queue))
+			pending = true;
+
 		/* did we get some space after spending time in recv? */
 		if (nvme_tcp_queue_has_pending(queue) &&
 		    sk_stream_is_writeable(queue->sock->sk))
-- 
2.56.0

[PATCH 56/98] nvme-tcp: don't queue the AER while it is being sent

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

Every AER completion runs nvme_async_event_work(), which has
nvme_tcp_submit_async_event() set up and queue ctrl->async_req again. A
controller that completes the AER twice can do that while the request is
still on req_list or send_list, or being sent. Added to req_list twice
the list becomes a ring and nvme_tcp_process_req_list() never ends:

  WARNING: lib/list_debug.c:35 at __list_add_valid_or_report+0xd8/0xf0
   nvme_tcp_try_send+0x51b/0x600

Leave the AER alone while the send path holds it, there's one on its way
already. Initialize the list nodes at allocation so the check is valid
from the start; nvme_tcp_fetch_request() leaves them that way.

Fixes: 3f2304f8c6d6 ("nvme-tcp: add NVMe over TCP host driver")
Cc: Keith Busch <kbusch@kernel.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/host/tcp.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 06c205703cf5..be3938c3cb6f 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -766,11 +766,14 @@ static int nvme_tcp_handle_comp(struct nvme_tcp_queue *queue,
 	 * for them but rather special case them here.
 	 */
 	if (unlikely(nvme_is_aen_req(nvme_tcp_queue_id(queue),
-				     cqe->command_id)))
+				     cqe->command_id))) {
+		if (nvme_tcp_req_busy(queue, &queue->ctrl->async_req))
+			return -EBUSY;
 		nvme_complete_async_event(&queue->ctrl->ctrl, cqe->status,
 				&cqe->result);
-	else
+	} else {
 		ret = nvme_tcp_process_nvme_cqe(queue, cqe);
+	}
 
 	return ret;
 }
@@ -1646,6 +1649,8 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl)
 		return -ENOMEM;
 
 	async->queue = &ctrl->queues[0];
+	init_llist_node(&async->lentry);
+	INIT_LIST_HEAD(&async->entry);
 	return 0;
 }
 
@@ -2884,6 +2889,10 @@ static void nvme_tcp_submit_async_event(struct nvme_ctrl *arg)
 	struct nvme_command *cmd = &pdu->cmd;
 	u8 hdgst = nvme_tcp_hdgst_len(queue);
 
+	/* A controller which completes it twice makes us come here twice */
+	if (nvme_tcp_req_busy(queue, &ctrl->async_req))
+		return;
+
 	memset(pdu, 0, sizeof(*pdu));
 	pdu->hdr.type = nvme_tcp_cmd;
 	if (queue->hdr_digest)
@@ -2900,8 +2909,6 @@ static void nvme_tcp_submit_async_event(struct nvme_ctrl *arg)
 	ctrl->async_req.offset = 0;
 	ctrl->async_req.curr_bio = NULL;
 	ctrl->async_req.data_len = 0;
-	init_llist_node(&ctrl->async_req.lentry);
-	INIT_LIST_HEAD(&ctrl->async_req.entry);
 
 	nvme_tcp_queue_request(&ctrl->async_req, true);
 }
-- 
2.56.0

[PATCH 57/98] nvme-tcp: don't send SIGPIPE to whoever submits I/O

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

nvme_tcp_queue_request() tries to send inline, so the first PDU of a
command goes out in the context of the submitting task, and none of the
sends pass MSG_NOSIGNAL. Once a connection is down and error recovery
hasn't stopped the queue yet, tcp_sendmsg() returns -EPIPE and
sk_stream_error() sends SIGPIPE to current. A lost connection that the
driver would recover from transparently instead kills whatever is doing
I/O on the namespace: with 16 dd processes and a connection cut every
other second, a few die within a minute.

Pass MSG_NOSIGNAL, like nbd, drbd, ceph, cifs and sunrpc do.

Fixes: db5ad6b7f8cd ("nvme-tcp: try to send request in queue_rq context")
Cc: Keith Busch <kbusch@kernel.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/host/tcp.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index be3938c3cb6f..c9531e653a2c 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1249,7 +1249,8 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_request *req)
 	while (true) {
 		struct bio_vec bvec;
 		struct msghdr msg = {
-			.msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES,
+			.msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL |
+				     MSG_SPLICE_PAGES,
 		};
 		struct page *page = nvme_tcp_req_cur_page(req);
 		size_t offset = nvme_tcp_req_cur_offset(req);
@@ -1308,7 +1309,9 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req)
 	struct nvme_tcp_queue *queue = req->queue;
 	struct nvme_tcp_cmd_pdu *pdu = nvme_tcp_req_cmd_pdu(req);
 	struct bio_vec bvec;
-	struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, };
+	struct msghdr msg = {
+		.msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SPLICE_PAGES,
+	};
 	bool inline_data = nvme_tcp_has_inline_data(req);
 	u8 hdgst = nvme_tcp_hdgst_len(queue);
 	int len = sizeof(*pdu) + hdgst - req->offset;
@@ -1349,7 +1352,9 @@ static int nvme_tcp_try_send_data_pdu(struct nvme_tcp_request *req)
 	struct nvme_tcp_queue *queue = req->queue;
 	struct nvme_tcp_data_pdu *pdu = nvme_tcp_req_data_pdu(req);
 	struct bio_vec bvec;
-	struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_MORE, };
+	struct msghdr msg = {
+		.msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL | MSG_MORE,
+	};
 	u8 hdgst = nvme_tcp_hdgst_len(queue);
 	int len = sizeof(*pdu) - req->offset + hdgst;
 	int ret;
@@ -1384,7 +1389,7 @@ static int nvme_tcp_try_send_ddgst(struct nvme_tcp_request *req)
 	size_t offset = req->offset;
 	u32 h2cdata_left = req->h2cdata_left;
 	int ret;
-	struct msghdr msg = { .msg_flags = MSG_DONTWAIT };
+	struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_NOSIGNAL };
 	struct kvec iov = {
 		.iov_base = (u8 *)&req->ddgst + req->offset,
 		.iov_len = NVME_TCP_DIGEST_LENGTH - req->offset
@@ -1696,7 +1701,7 @@ static int nvme_tcp_init_connection(struct nvme_tcp_queue *queue)
 	struct nvme_tcp_icresp_pdu *icresp;
 	char cbuf[CMSG_LEN(sizeof(char))] = {};
 	u8 ctype;
-	struct msghdr msg = {};
+	struct msghdr msg = { .msg_flags = MSG_NOSIGNAL };
 	struct kvec iov;
 	bool ctrl_hdgst, ctrl_ddgst;
 	u32 maxh2cdata;
-- 
2.56.0

[PATCH 58/98] nvme-fabrics: reconnect if the association is gone

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

A reconnect connects the admin queue, gets a controller ID, and connects
the I/O queues with it. If the controller goes away in between, the
Connect of an I/O queue fails with Connect Invalid Parameters pointing
at the cntlid, with DNR set:

  nvme nvme0: Connect Invalid Data Parameter, cntlid: 1
  nvme nvme0: Failed reconnect attempt 1/600
  nvme nvme0: Removing controller (16770)...

nvmf_should_reconnect() honours DNR and the host deletes the controller
with 599 attempts to go. DNR is right in that resending that command is
pointless, but the next attempt would create a new association with a
new cntlid. This is not a verdict on the host like a failed admin
Connect is.

Treat it as a lost association and return -ENOTCONN, so the retries are
bounded by ctrl_loss_tmo like after any other loss.

Fixes: adfde7ed0b30 ("nvme-fabrics: short-circuit reconnect retries")
Cc: Keith Busch <kbusch@kernel.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/host/fabrics.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index 59f823dfbbcc..96d5ae6e1c39 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -542,8 +542,19 @@ int nvmf_connect_io_queue(struct nvme_ctrl *ctrl, u16 qid)
 			NVME_SUBMIT_RESERVED |
 			NVME_SUBMIT_NOWAIT);
 	if (ret) {
-		nvmf_log_connect_error(ctrl, ret, le32_to_cpu(res.u32),
-				       &cmd, data);
+		result = le32_to_cpu(res.u32);
+		nvmf_log_connect_error(ctrl, ret, result, &cmd, data);
+		/*
+		 * The controller ID we got from the admin queue's Connect is
+		 * not known any more: the association has gone since. DNR is
+		 * right about this command and says nothing about the next
+		 * association. Don't let nvmf_should_reconnect() see it.
+		 */
+		if (ret > 0 &&
+		    (ret & NVME_SCT_SC_MASK) == NVME_SC_CONNECT_INVALID_PARAM &&
+		    result >> 16 && (result & 0xffff) ==
+				offsetof(struct nvmf_connect_data, cntlid))
+			ret = -ENOTCONN;
 		goto out_free_data;
 	}
 	result = le32_to_cpu(res.u32);
-- 
2.56.0

[PATCH 59/98] nvmet-auth: serialize the commands of a controller

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

The host and controller keys, the DH context and the hash of DH-HMAC-CHAP
live in the controller, and the authentication commands of all its
queues use them without a lock. A host can send AUTH_Negotiate on the
admin queue at any time, and nvmet_setup_auth() then frees and reloads
the keys while another queue computes a response from them:

  BUG: KASAN: slab-use-after-free in nvme_auth_transform_key+0xa3/0x250
   nvmet_auth_host_hash+0x10e/0x530
   nvmet_execute_auth_send+0xa4c/0x1a00
  Freed by task 194:
   nvmet_setup_auth+0x166/0x3a0
   nvmet_execute_auth_send+0x3e7/0x1a00

Neither queue needs to have authenticated for that.

Add a controller-wide mutex and hold it for the duration of an
authentication command, dropping it before the request is completed.
nvmet_alloc_ctrl() takes it around nvmet_setup_auth() too, as the
controller is already on the subsystem list by then and an I/O queue
can connect.

[axboe: upstream already has the per-queue auth_lock and expiry handling
 from 53f6efe00cea, so only the controller lock is added here. It nests
 inside sq->auth_lock.]

Fixes: db1312dd9548 ("nvmet: implement basic In-Band Authentication")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Cc: Hannes Reinecke <hare@suse.de>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/core.c             | 4 ++++
 drivers/nvme/target/fabrics-cmd-auth.c | 6 ++++++
 drivers/nvme/target/nvmet.h            | 1 +
 3 files changed, 11 insertions(+)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 8353950f346f..5bd78bb83cf9 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -1642,6 +1642,7 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
 	if (!ctrl)
 		goto out_put_subsystem;
 	mutex_init(&ctrl->lock);
+	mutex_init(&ctrl->auth_lock);
 
 	ctrl->port = args->port;
 	ctrl->ops = args->ops;
@@ -1728,7 +1729,10 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
 
 	nvmet_start_keep_alive_timer(ctrl);
 
+	/* An I/O queue may connect as soon as the controller is on the list. */
+	mutex_lock(&ctrl->auth_lock);
 	dhchap_status = nvmet_setup_auth(ctrl, args->sq, false);
+	mutex_unlock(&ctrl->auth_lock);
 	if (dhchap_status) {
 		pr_err("Failed to setup authentication, dhchap status %u\n",
 		       dhchap_status);
diff --git a/drivers/nvme/target/fabrics-cmd-auth.c b/drivers/nvme/target/fabrics-cmd-auth.c
index 1fe21f2672f1..9779e664b65f 100644
--- a/drivers/nvme/target/fabrics-cmd-auth.c
+++ b/drivers/nvme/target/fabrics-cmd-auth.c
@@ -269,6 +269,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
 	u8 dhchap_status;
 
 	mutex_lock(&req->sq->auth_lock);
+	mutex_lock(&ctrl->auth_lock);
 
 	if (req->cmd->auth_send.secp != NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER) {
 		status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
@@ -297,6 +298,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
 	}
 	if (unlikely(tl != req->transfer_len)) {
 		pr_debug("%s: transfer length mismatch (%u)\n", __func__, tl);
+		mutex_unlock(&ctrl->auth_lock);
 		mutex_unlock(&req->sq->auth_lock);
 		nvmet_check_transfer_len(req, tl);
 		return;
@@ -442,6 +444,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
 		nvmet_ctrl_fatal_error(ctrl);
 
 complete:
+	mutex_unlock(&ctrl->auth_lock);
 	mutex_unlock(&req->sq->auth_lock);
 	nvmet_req_complete(req, status);
 }
@@ -562,6 +565,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)
 	u16 status = 0;
 
 	mutex_lock(&req->sq->auth_lock);
+	mutex_lock(&ctrl->auth_lock);
 
 	if (req->cmd->auth_receive.secp != NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER) {
 		status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
@@ -590,6 +594,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)
 	}
 	if (unlikely(al != req->transfer_len)) {
 		pr_debug("%s: transfer length mismatch (%u)\n", __func__, al);
+		mutex_unlock(&ctrl->auth_lock);
 		mutex_unlock(&req->sq->auth_lock);
 		nvmet_check_transfer_len(req, al);
 		return;
@@ -650,6 +655,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)
 		nvmet_auth_sq_free(req->sq);
 		nvmet_ctrl_fatal_error(ctrl);
 	}
+	mutex_unlock(&ctrl->auth_lock);
 	mutex_unlock(&req->sq->auth_lock);
 	nvmet_req_complete(req, status);
 }
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 9cec19e6f485..f6a9f2e1c0ed 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -307,6 +307,7 @@ struct nvmet_ctrl {
 	struct nvme_error_slot	slots[NVMET_ERROR_LOG_SLOTS];
 	bool			pi_support;
 	bool			concat;
+	struct mutex		auth_lock; /* one AUTH command at a time */
 #ifdef CONFIG_NVME_TARGET_AUTH
 	struct nvme_dhchap_key	*host_key;
 	struct nvme_dhchap_key	*ctrl_key;
-- 
2.56.0

Re: [PATCH 59/98] nvmet-auth: serialize the commands of a controller

[Hannes Reinecke]

On 10/28/26 5:42 PM, Jens Axboe wrote:
> From: Josef Bacik <josef@toxicpanda.com>
> 
> The host and controller keys, the DH context and the hash of DH-HMAC-CHAP
> live in the controller, and the authentication commands of all its
> queues use them without a lock. A host can send AUTH_Negotiate on the
> admin queue at any time, and nvmet_setup_auth() then frees and reloads
> the keys while another queue computes a response from them:
> 
>    BUG: KASAN: slab-use-after-free in nvme_auth_transform_key+0xa3/0x250
>     nvmet_auth_host_hash+0x10e/0x530
>     nvmet_execute_auth_send+0xa4c/0x1a00
>    Freed by task 194:
>     nvmet_setup_auth+0x166/0x3a0
>     nvmet_execute_auth_send+0x3e7/0x1a00
> 
> Neither queue needs to have authenticated for that.
> 
> Add a controller-wide mutex and hold it for the duration of an
> authentication command, dropping it before the request is completed.
> nvmet_alloc_ctrl() takes it around nvmet_setup_auth() too, as the
> controller is already on the subsystem list by then and an I/O queue
> can connect.
> 
> [axboe: upstream already has the per-queue auth_lock and expiry handling
>   from 53f6efe00cea, so only the controller lock is added here. It nests
>   inside sq->auth_lock.]
> 
> Fixes: db1312dd9548 ("nvmet: implement basic In-Band Authentication")
> Cc: Christoph Hellwig <hch@lst.de>
> Cc: Sagi Grimberg <sagi@grimberg.me>
> Cc: Chaitanya Kulkarni <kch@nvidia.com>
> Cc: Hannes Reinecke <hare@suse.de>
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> Signed-off-by: Jens Axboe <axboe@kernel.dk>
> ---
>   drivers/nvme/target/core.c             | 4 ++++
>   drivers/nvme/target/fabrics-cmd-auth.c | 6 ++++++
>   drivers/nvme/target/nvmet.h            | 1 +
>   3 files changed, 11 insertions(+)
> 
> diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
> index 8353950f346f..5bd78bb83cf9 100644
> --- a/drivers/nvme/target/core.c
> +++ b/drivers/nvme/target/core.c
> @@ -1642,6 +1642,7 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
>   	if (!ctrl)
>   		goto out_put_subsystem;
>   	mutex_init(&ctrl->lock);
> +	mutex_init(&ctrl->auth_lock);
>   
>   	ctrl->port = args->port;
>   	ctrl->ops = args->ops;
> @@ -1728,7 +1729,10 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
>   
>   	nvmet_start_keep_alive_timer(ctrl);
>   
> +	/* An I/O queue may connect as soon as the controller is on the list. */
> +	mutex_lock(&ctrl->auth_lock);
>   	dhchap_status = nvmet_setup_auth(ctrl, args->sq, false);
> +	mutex_unlock(&ctrl->auth_lock);
>   	if (dhchap_status) {
>   		pr_err("Failed to setup authentication, dhchap status %u\n",
>   		       dhchap_status);
> diff --git a/drivers/nvme/target/fabrics-cmd-auth.c b/drivers/nvme/target/fabrics-cmd-auth.c
> index 1fe21f2672f1..9779e664b65f 100644
> --- a/drivers/nvme/target/fabrics-cmd-auth.c
> +++ b/drivers/nvme/target/fabrics-cmd-auth.c
> @@ -269,6 +269,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
>   	u8 dhchap_status;
>   
>   	mutex_lock(&req->sq->auth_lock);
> +	mutex_lock(&ctrl->auth_lock);
>   
>   	if (req->cmd->auth_send.secp != NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER) {
>   		status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
> @@ -297,6 +298,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
>   	}
>   	if (unlikely(tl != req->transfer_len)) {
>   		pr_debug("%s: transfer length mismatch (%u)\n", __func__, tl);
> +		mutex_unlock(&ctrl->auth_lock);
>   		mutex_unlock(&req->sq->auth_lock);
>   		nvmet_check_transfer_len(req, tl);
>   		return;
> @@ -442,6 +444,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
>   		nvmet_ctrl_fatal_error(ctrl);
>   
>   complete:
> +	mutex_unlock(&ctrl->auth_lock);
>   	mutex_unlock(&req->sq->auth_lock);
>   	nvmet_req_complete(req, status);
>   }
> @@ -562,6 +565,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)
>   	u16 status = 0;
>   
>   	mutex_lock(&req->sq->auth_lock);
> +	mutex_lock(&ctrl->auth_lock);
>   
>   	if (req->cmd->auth_receive.secp != NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER) {
>   		status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
> @@ -590,6 +594,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)
>   	}
>   	if (unlikely(al != req->transfer_len)) {
>   		pr_debug("%s: transfer length mismatch (%u)\n", __func__, al);
> +		mutex_unlock(&ctrl->auth_lock);
>   		mutex_unlock(&req->sq->auth_lock);
>   		nvmet_check_transfer_len(req, al);
>   		return;
> @@ -650,6 +655,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)
>   		nvmet_auth_sq_free(req->sq);
>   		nvmet_ctrl_fatal_error(ctrl);
>   	}
> +	mutex_unlock(&ctrl->auth_lock);
>   	mutex_unlock(&req->sq->auth_lock);
>   	nvmet_req_complete(req, status);
>   }
> diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
> index 9cec19e6f485..f6a9f2e1c0ed 100644
> --- a/drivers/nvme/target/nvmet.h
> +++ b/drivers/nvme/target/nvmet.h
> @@ -307,6 +307,7 @@ struct nvmet_ctrl {
>   	struct nvme_error_slot	slots[NVMET_ERROR_LOG_SLOTS];
>   	bool			pi_support;
>   	bool			concat;
> +	struct mutex		auth_lock; /* one AUTH command at a time */
>   #ifdef CONFIG_NVME_TARGET_AUTH
>   	struct nvme_dhchap_key	*host_key;
>   	struct nvme_dhchap_key	*ctrl_key;

Hmm. Not particularly happy with this.
In the end, it renders sq->auth_lock pretty much pointless (the only
place where it remains on its own is in nvmet_auth_expiry_work() ).
And it would trigger if a authentication reset while another 
authentication (eg on the I/O queues) is still ongoing.
With this patch we would switch the authentication variables
in the middle of the authentication protocol run for the I/O
queue, most likely causing that authentication to fail.

We should rather fail ongoing authentications whenever we
call nvmet_auth_setup().

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

[PATCH 60/98] nvmet-auth: keep the negotiated hash per queue

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

Every queue runs its own DH-HMAC-CHAP transaction, but the hash that
AUTH_Negotiate settles on is stored in the controller. A host that
doesn't offer the configured hash gets a fallback, and that changes
ctrl->shash_id under the transactions of the other queues, whose
challenges have the length of the old hash:

  qid 1: AUTH_Negotiate (SHA-256), 32 byte challenge
  qid 0: AUTH_Negotiate (SHA-512 only)
  qid 1: DH-HMAC-CHAP_Reply with a hash length of 64

  BUG: KASAN: slab-out-of-bounds in nvmet_auth_host_hash+0x2c9/0x530

No secret is needed for that.

Store the negotiated hash in the queue and use it for the rest of the
transaction. ctrl->shash_id stays the configured preference.

Fixes: db1312dd9548 ("nvmet: implement basic In-Band Authentication")
Cc: Hannes Reinecke <hare@suse.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/auth.c             | 35 +++++++++++++-------------
 drivers/nvme/target/fabrics-cmd-auth.c | 12 ++++-----
 drivers/nvme/target/nvmet.h            |  1 +
 3 files changed, 25 insertions(+), 23 deletions(-)

diff --git a/drivers/nvme/target/auth.c b/drivers/nvme/target/auth.c
index 1680e5e6839c..0f64fd4ade59 100644
--- a/drivers/nvme/target/auth.c
+++ b/drivers/nvme/target/auth.c
@@ -299,14 +299,14 @@ int nvmet_auth_host_hash(struct nvmet_req *req, u8 *response,
 	if (IS_ERR(transformed_key))
 		return PTR_ERR(transformed_key);
 
-	ret = nvme_auth_hmac_init(&hmac, ctrl->shash_id, transformed_key->key,
-				  transformed_key->len);
+	ret = nvme_auth_hmac_init(&hmac, req->sq->shash_id,
+				  transformed_key->key, transformed_key->len);
 	if (ret)
 		goto out_free_response;
 
-	if (shash_len != nvme_auth_hmac_hash_len(ctrl->shash_id)) {
+	if (shash_len != nvme_auth_hmac_hash_len(req->sq->shash_id)) {
 		pr_err("%s: hash len mismatch (len %u digest %zu)\n", __func__,
-		       shash_len, nvme_auth_hmac_hash_len(ctrl->shash_id));
+		       shash_len, nvme_auth_hmac_hash_len(req->sq->shash_id));
 		ret = -EINVAL;
 		goto out_free_response;
 	}
@@ -317,7 +317,7 @@ int nvmet_auth_host_hash(struct nvmet_req *req, u8 *response,
 			ret = -ENOMEM;
 			goto out_free_response;
 		}
-		ret = nvme_auth_augmented_challenge(ctrl->shash_id,
+		ret = nvme_auth_augmented_challenge(req->sq->shash_id,
 						    req->sq->dhchap_skey,
 						    req->sq->dhchap_skey_len,
 						    req->sq->dhchap_c1,
@@ -371,14 +371,14 @@ int nvmet_auth_ctrl_hash(struct nvmet_req *req, u8 *response,
 	if (IS_ERR(transformed_key))
 		return PTR_ERR(transformed_key);
 
-	ret = nvme_auth_hmac_init(&hmac, ctrl->shash_id, transformed_key->key,
-				  transformed_key->len);
+	ret = nvme_auth_hmac_init(&hmac, req->sq->shash_id,
+				  transformed_key->key, transformed_key->len);
 	if (ret)
 		goto out_free_response;
 
-	if (shash_len != nvme_auth_hmac_hash_len(ctrl->shash_id)) {
+	if (shash_len != nvme_auth_hmac_hash_len(req->sq->shash_id)) {
 		pr_err("%s: hash len mismatch (len %u digest %zu)\n", __func__,
-		       shash_len, nvme_auth_hmac_hash_len(ctrl->shash_id));
+		       shash_len, nvme_auth_hmac_hash_len(req->sq->shash_id));
 		ret = -EINVAL;
 		goto out_free_response;
 	}
@@ -389,7 +389,7 @@ int nvmet_auth_ctrl_hash(struct nvmet_req *req, u8 *response,
 			ret = -ENOMEM;
 			goto out_free_response;
 		}
-		ret = nvme_auth_augmented_challenge(ctrl->shash_id,
+		ret = nvme_auth_augmented_challenge(req->sq->shash_id,
 						    req->sq->dhchap_skey,
 						    req->sq->dhchap_skey_len,
 						    req->sq->dhchap_c2,
@@ -456,7 +456,8 @@ int nvmet_auth_ctrl_sesskey(struct nvmet_req *req,
 	struct nvmet_ctrl *ctrl = req->sq->ctrl;
 	int ret;
 
-	req->sq->dhchap_skey_len = nvme_auth_hmac_hash_len(ctrl->shash_id);
+	req->sq->dhchap_skey_len =
+		nvme_auth_hmac_hash_len(req->sq->shash_id);
 	req->sq->dhchap_skey = kzalloc(req->sq->dhchap_skey_len, GFP_KERNEL);
 	if (!req->sq->dhchap_skey)
 		return -ENOMEM;
@@ -464,7 +465,7 @@ int nvmet_auth_ctrl_sesskey(struct nvmet_req *req,
 					pkey, pkey_size,
 					req->sq->dhchap_skey,
 					req->sq->dhchap_skey_len,
-					ctrl->shash_id);
+					req->sq->shash_id);
 	if (ret)
 		pr_debug("failed to compute session key, err %d\n", ret);
 #ifdef CONFIG_NVME_TARGET_AUTH_DEBUG
@@ -478,7 +479,7 @@ int nvmet_auth_ctrl_sesskey(struct nvmet_req *req,
 
 void nvmet_auth_insert_psk(struct nvmet_sq *sq)
 {
-	int hash_len = nvme_auth_hmac_hash_len(sq->ctrl->shash_id);
+	int hash_len = nvme_auth_hmac_hash_len(sq->shash_id);
 	u8 *psk, *tls_psk;
 	char *digest;
 	size_t psk_len;
@@ -487,7 +488,7 @@ void nvmet_auth_insert_psk(struct nvmet_sq *sq)
 	struct key *tls_key = NULL;
 #endif
 
-	ret = nvme_auth_generate_psk(sq->ctrl->shash_id,
+	ret = nvme_auth_generate_psk(sq->shash_id,
 				     sq->dhchap_skey,
 				     sq->dhchap_skey_len,
 				     sq->dhchap_c1, sq->dhchap_c2,
@@ -497,7 +498,7 @@ void nvmet_auth_insert_psk(struct nvmet_sq *sq)
 			__func__, sq->ctrl->cntlid, sq->qid, ret);
 		return;
 	}
-	ret = nvme_auth_generate_digest(sq->ctrl->shash_id, psk, psk_len,
+	ret = nvme_auth_generate_digest(sq->shash_id, psk, psk_len,
 					sq->ctrl->subsys->subsysnqn,
 					sq->ctrl->hostnqn, &digest);
 	if (ret) {
@@ -505,7 +506,7 @@ void nvmet_auth_insert_psk(struct nvmet_sq *sq)
 			__func__, sq->ctrl->cntlid, sq->qid, ret);
 		goto out_free_psk;
 	}
-	ret = nvme_auth_derive_tls_psk(sq->ctrl->shash_id, psk, psk_len,
+	ret = nvme_auth_derive_tls_psk(sq->shash_id, psk, psk_len,
 				       digest, &tls_psk);
 	if (ret) {
 		pr_warn("%s: ctrl %d qid %d failed to derive TLS PSK, error %d\n",
@@ -515,7 +516,7 @@ void nvmet_auth_insert_psk(struct nvmet_sq *sq)
 #ifdef CONFIG_NVME_TARGET_TCP_TLS
 	tls_key = nvme_tls_psk_refresh(NULL, sq->ctrl->hostnqn,
 				       sq->ctrl->subsys->subsysnqn,
-				       sq->ctrl->shash_id, tls_psk, psk_len,
+				       sq->shash_id, tls_psk, psk_len,
 				       digest);
 	if (IS_ERR(tls_key)) {
 		pr_warn("%s: ctrl %d qid %d failed to refresh key, error %ld\n",
diff --git a/drivers/nvme/target/fabrics-cmd-auth.c b/drivers/nvme/target/fabrics-cmd-auth.c
index 9779e664b65f..841fd2683702 100644
--- a/drivers/nvme/target/fabrics-cmd-auth.c
+++ b/drivers/nvme/target/fabrics-cmd-auth.c
@@ -111,8 +111,9 @@ static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d, u32 tl)
 		pr_debug("%s: ctrl %d qid %d: no usable hash found, falling back to %s\n",
 			 __func__, ctrl->cntlid, req->sq->qid,
 			 nvme_auth_hmac_name(fallback_hash_id));
-		ctrl->shash_id = fallback_hash_id;
+		hash_id = fallback_hash_id;
 	}
+	req->sq->shash_id = hash_id;
 
 	dhgid = -1;
 	fallback_dhgid = -1;
@@ -454,7 +455,7 @@ static int nvmet_auth_challenge(struct nvmet_req *req, void *d, int al)
 	struct nvmf_auth_dhchap_challenge_data *data = d;
 	struct nvmet_ctrl *ctrl = req->sq->ctrl;
 	int ret = 0;
-	int hash_len = nvme_auth_hmac_hash_len(ctrl->shash_id);
+	int hash_len = nvme_auth_hmac_hash_len(req->sq->shash_id);
 	int data_size = sizeof(*data) + hash_len;
 
 	if (ctrl->dh_tfm)
@@ -469,7 +470,7 @@ static int nvmet_auth_challenge(struct nvmet_req *req, void *d, int al)
 	data->auth_type = NVME_AUTH_DHCHAP_MESSAGES;
 	data->auth_id = NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE;
 	data->t_id = cpu_to_le16(req->sq->dhchap_tid);
-	data->hashid = ctrl->shash_id;
+	data->hashid = req->sq->shash_id;
 	data->hl = hash_len;
 	data->seqnum = cpu_to_le32(req->sq->dhchap_s1);
 	req->sq->dhchap_c1 = kmalloc(data->hl, GFP_KERNEL);
@@ -493,7 +494,7 @@ static int nvmet_auth_success1(struct nvmet_req *req, void *d, int al)
 {
 	struct nvmf_auth_dhchap_success1_data *data = d;
 	struct nvmet_ctrl *ctrl = req->sq->ctrl;
-	int hash_len = nvme_auth_hmac_hash_len(ctrl->shash_id);
+	int hash_len = nvme_auth_hmac_hash_len(req->sq->shash_id);
 
 	WARN_ON(al < sizeof(*data));
 	memset(data, 0, sizeof(*data));
@@ -530,7 +531,6 @@ static void nvmet_auth_failure1(struct nvmet_req *req, void *d, int al)
 
 u32 nvmet_auth_receive_data_len(struct nvmet_req *req)
 {
-	struct nvmet_ctrl *ctrl = req->sq->ctrl;
 	u32 al = le32_to_cpu(req->cmd->auth_receive.al);
 	u32 min_len;
 
@@ -544,7 +544,7 @@ u32 nvmet_auth_receive_data_len(struct nvmet_req *req)
 	case NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1:
 		min_len = sizeof(struct nvmf_auth_dhchap_success1_data);
 		if (req->sq->dhchap_c2)
-			min_len += nvme_auth_hmac_hash_len(ctrl->shash_id);
+			min_len += nvme_auth_hmac_hash_len(req->sq->shash_id);
 		break;
 	default:
 		min_len = sizeof(struct nvmf_auth_dhchap_failure_data);
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index f6a9f2e1c0ed..46501d0757c0 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -167,6 +167,7 @@ struct nvmet_sq {
 	bool			auth_expiry_active;
 	u16			dhchap_tid;
 	u8			sc_c;
+	u8			shash_id;
 	u8			dhchap_status;
 	u8			dhchap_step;
 	u8			*dhchap_c1;
-- 
2.56.0

Re: [PATCH 60/98] nvmet-auth: keep the negotiated hash per queue

[Hannes Reinecke]

On 10/28/26 5:42 PM, Jens Axboe wrote:
> From: Josef Bacik <josef@toxicpanda.com>
> 
> Every queue runs its own DH-HMAC-CHAP transaction, but the hash that
> AUTH_Negotiate settles on is stored in the controller. A host that
> doesn't offer the configured hash gets a fallback, and that changes
> ctrl->shash_id under the transactions of the other queues, whose
> challenges have the length of the old hash:
> 
>    qid 1: AUTH_Negotiate (SHA-256), 32 byte challenge
>    qid 0: AUTH_Negotiate (SHA-512 only)
>    qid 1: DH-HMAC-CHAP_Reply with a hash length of 64
> 
>    BUG: KASAN: slab-out-of-bounds in nvmet_auth_host_hash+0x2c9/0x530
> 
> No secret is needed for that.
> 
> Store the negotiated hash in the queue and use it for the rest of the
> transaction. ctrl->shash_id stays the configured preference.
> 
> Fixes: db1312dd9548 ("nvmet: implement basic In-Band Authentication")
> Cc: Hannes Reinecke <hare@suse.de>
> Cc: Christoph Hellwig <hch@lst.de>
> Cc: Sagi Grimberg <sagi@grimberg.me>
> Cc: Chaitanya Kulkarni <kch@nvidia.com>
> Signed-off-by: Josef Bacik <josef@toxicpanda.com>
> Signed-off-by: Jens Axboe <axboe@kernel.dk>
> ---
>   drivers/nvme/target/auth.c             | 35 +++++++++++++-------------
>   drivers/nvme/target/fabrics-cmd-auth.c | 12 ++++-----
>   drivers/nvme/target/nvmet.h            |  1 +
>   3 files changed, 25 insertions(+), 23 deletions(-)
> 
Weird. But spec seems to be okay with that, I guess.
But then we don't really need to the shash_id from the controller,
so it should be removed from there.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich

[PATCH 61/98] nvmet-auth: authenticate the I/O queues again

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

Since commit c91a20129185 the target neither asks a host to authenticate
an I/O queue nor checks that it did: nvmet_check_auth_status() is true
for everything but the admin queue. Nothing else ties an I/O queue to
the admin queue's authentication. The Connect of an I/O queue names its
controller by subsystem NQN, host NQN and cntlid, so anyone who can
reach the port and knows the NQNs can attach a queue to an authenticated
controller and do I/O. Property Set isn't subject to authentication
either, so an unauthenticated host can enable its own controller and
connect an I/O queue to that.

Ask for authentication in the Connect response of an I/O queue and
refuse its commands until it has passed, as before that commit. The
Linux host authenticates every queue that is asked to. A queue with a
TLS session is exempt, its peer held a PSK the port accepts, and since
commit ecf4d2d88351 such a controller does get a host key when the admin
queue re-authenticates.

Fixes: c91a20129185 ("nvmet-auth: authenticate on admin queue only")
Cc: Hannes Reinecke <hare@suse.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/auth.c        | 14 +++++++-------
 drivers/nvme/target/fabrics-cmd.c |  4 ----
 2 files changed, 7 insertions(+), 11 deletions(-)

diff --git a/drivers/nvme/target/auth.c b/drivers/nvme/target/auth.c
index 0f64fd4ade59..9ae8293cc734 100644
--- a/drivers/nvme/target/auth.c
+++ b/drivers/nvme/target/auth.c
@@ -275,13 +275,13 @@ void nvmet_destroy_auth(struct nvmet_ctrl *ctrl)
 
 bool nvmet_check_auth_status(struct nvmet_req *req)
 {
-	if (req->sq->ctrl->host_key) {
-		if (req->sq->qid > 0)
-			return true;
-		if (!req->sq->authenticated)
-			return false;
-	}
-	return true;
+	struct nvmet_sq *sq = req->sq;
+
+	if (!sq->ctrl->host_key || sq->authenticated)
+		return true;
+
+	/* Whoever set up the TLS session of an I/O queue knew the PSK. */
+	return sq->qid && nvmet_queue_tls_keyid(sq);
 }
 
 int nvmet_auth_host_hash(struct nvmet_req *req, u8 *response,
diff --git a/drivers/nvme/target/fabrics-cmd.c b/drivers/nvme/target/fabrics-cmd.c
index b451029ab8f4..38311187d27a 100644
--- a/drivers/nvme/target/fabrics-cmd.c
+++ b/drivers/nvme/target/fabrics-cmd.c
@@ -251,10 +251,6 @@ static u32 nvmet_connect_result(struct nvmet_ctrl *ctrl, struct nvmet_sq *sq)
 	bool needs_auth = nvmet_has_auth(ctrl, sq);
 	key_serial_t keyid = nvmet_queue_tls_keyid(sq);
 
-	/* Do not authenticate I/O queues */
-	if (sq->qid)
-		needs_auth = false;
-
 	if (keyid)
 		pr_debug("%s: ctrl %d qid %d should %sauthenticate, tls psk %08x\n",
 			 __func__, ctrl->cntlid, sq->qid,
-- 
2.56.0

[PATCH 62/98] nvmet: don't mix queues with and without TLS

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

Since commit fa2e0f8bbc68 a controller whose admin queue uses TLS gets
no host key, the PSK stands in for DH-HMAC-CHAP. But a port with TREQ
"not required" takes plain connections too, as does any other port the
subsystem is linked to, and the Connect of an I/O queue names its
controller by NQNs and cntlid only. So anyone who reaches such a port
can add a plain queue to a TLS controller, and with no host key nothing
is asked of it. The other way round, a TLS queue skips DH-HMAC-CHAP on
a controller whose other queues had to pass it.

A host uses TLS for all queues or none. Refuse an I/O queue Connect that
differs from the admin queue in that.

Fixes: fa2e0f8bbc68 ("nvmet-tcp: support secure channel concatenation")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/core.c        | 1 +
 drivers/nvme/target/fabrics-cmd.c | 9 +++++++++
 drivers/nvme/target/nvmet.h       | 1 +
 3 files changed, 11 insertions(+)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 5bd78bb83cf9..333739f14c2b 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -1646,6 +1646,7 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
 
 	ctrl->port = args->port;
 	ctrl->ops = args->ops;
+	ctrl->tls = args->sq && nvmet_queue_tls_keyid(args->sq);
 
 #ifdef CONFIG_NVME_TARGET_PASSTHRU
 	/* By default, set loop targets to clear IDS by default */
diff --git a/drivers/nvme/target/fabrics-cmd.c b/drivers/nvme/target/fabrics-cmd.c
index 38311187d27a..ddb732f62981 100644
--- a/drivers/nvme/target/fabrics-cmd.c
+++ b/drivers/nvme/target/fabrics-cmd.c
@@ -377,6 +377,15 @@ static void nvmet_execute_io_connect(struct nvmet_req *req)
 		goto out_ctrl_put;
 	}
 
+	/* All queues of a controller use TLS, or none of them does. */
+	if (ctrl->tls != !!nvmet_queue_tls_keyid(req->sq)) {
+		pr_warn("ctrl %d: queue %d with%s TLS\n", ctrl->cntlid, qid,
+			ctrl->tls ? "out" : "");
+		status = NVME_SC_CONNECT_INVALID_PARAM | NVME_STATUS_DNR;
+		req->cqe->result.u32 = IPO_IATTR_CONNECT_DATA(cntlid);
+		goto out_ctrl_put;
+	}
+
 	status = nvmet_install_queue(ctrl, req);
 	if (status)
 		goto out_ctrl_put;
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 46501d0757c0..cc7fface77de 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -308,6 +308,7 @@ struct nvmet_ctrl {
 	struct nvme_error_slot	slots[NVMET_ERROR_LOG_SLOTS];
 	bool			pi_support;
 	bool			concat;
+	bool			tls;
 	struct mutex		auth_lock; /* one AUTH command at a time */
 #ifdef CONFIG_NVME_TARGET_AUTH
 	struct nvme_dhchap_key	*host_key;
-- 
2.56.0

[PATCH 63/98] nvmet-auth: refuse a reply without a DH value

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

With a DH group configured, nvmet_auth_host_hash() augments the
challenge with the queue's session key, but nvmet_auth_reply() only
computes that key if the DH-HMAC-CHAP_Reply carries a DH value. Without
one, whatever the queue holds is used: nothing on the first transaction,
so the DH exchange is skipped, and after nvmet_auth_sq_free() a NULL
pointer with the hash length still set:

  BUG: KASAN: null-ptr-deref in __hmac_sha256_preparekey+0xa5/0x250
   nvme_auth_hmac+0xb7/0x160
   nvmet_auth_host_hash+0x2a5/0x530

Refuse such a reply, as the host refuses a challenge without a DH value,
and clear the length along with the pointer.

Fixes: 7a277c37d352 ("nvmet-auth: Diffie-Hellman key exchange support")
Cc: Hannes Reinecke <hare@suse.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/auth.c             | 1 +
 drivers/nvme/target/fabrics-cmd-auth.c | 3 +++
 2 files changed, 4 insertions(+)

diff --git a/drivers/nvme/target/auth.c b/drivers/nvme/target/auth.c
index 9ae8293cc734..d1af4a921e7a 100644
--- a/drivers/nvme/target/auth.c
+++ b/drivers/nvme/target/auth.c
@@ -237,6 +237,7 @@ void nvmet_auth_sq_free(struct nvmet_sq *sq)
 	sq->dhchap_c2 = NULL;
 	kfree(sq->dhchap_skey);
 	sq->dhchap_skey = NULL;
+	sq->dhchap_skey_len = 0;
 }
 
 void nvmet_auth_sq_destroy(struct nvmet_sq *sq)
diff --git a/drivers/nvme/target/fabrics-cmd-auth.c b/drivers/nvme/target/fabrics-cmd-auth.c
index 841fd2683702..a559fec3aa65 100644
--- a/drivers/nvme/target/fabrics-cmd-auth.c
+++ b/drivers/nvme/target/fabrics-cmd-auth.c
@@ -180,6 +180,9 @@ static u8 nvmet_auth_reply(struct nvmet_req *req, void *d, u32 tl)
 		if (nvmet_auth_ctrl_sesskey(req, data->rval + 2 * data->hl,
 					    dhvlen) < 0)
 			return NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+	} else if (ctrl->dh_gid != NVME_AUTH_DHGROUP_NULL) {
+		/* No DH value, no session key for the augmented challenge. */
+		return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
 	}
 
 	response = kmalloc(data->hl, GFP_KERNEL);
-- 
2.56.0

[PATCH 64/98] nvmet-auth: keep the old keys if the new ones fail

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

AUTH_Negotiate on the admin queue has nvmet_setup_auth() reload the
host's keys, and it frees the old host key first. If decoding the new
secret fails, or an allocation does, the controller is left without a
host key, which means no authentication at all: nvmet_check_auth_status()
passes everything and I/O queue Connects no longer ask for it. configfs
accepts any string starting with "DHHC-1:" and a digit, so one typo
while changing a secret opens the namespaces to any connected admin
queue.

Decode the new keys first and only then replace the old ones.

Fixes: db1312dd9548 ("nvmet: implement basic In-Band Authentication")
Cc: Hannes Reinecke <hare@suse.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/auth.c | 69 ++++++++++++++++----------------------
 1 file changed, 28 insertions(+), 41 deletions(-)

diff --git a/drivers/nvme/target/auth.c b/drivers/nvme/target/auth.c
index d1af4a921e7a..5809ce32a2e4 100644
--- a/drivers/nvme/target/auth.c
+++ b/drivers/nvme/target/auth.c
@@ -135,6 +135,7 @@ u8 nvmet_setup_auth(struct nvmet_ctrl *ctrl, struct nvmet_sq *sq, bool reset)
 	int ret = 0;
 	struct nvmet_host_link *p;
 	struct nvmet_host *host = NULL;
+	struct nvme_dhchap_key *host_key, *ctrl_key = NULL;
 
 	down_read(&nvmet_config_sem);
 	if (nvmet_is_disc_subsys(ctrl->subsys))
@@ -172,55 +173,41 @@ u8 nvmet_setup_auth(struct nvmet_ctrl *ctrl, struct nvmet_sq *sq, bool reset)
 		goto out_unlock;
 	}
 
-	if (host->dhchap_hash_id == ctrl->shash_id) {
-		pr_debug("Re-use existing hash ID %d\n",
-			 ctrl->shash_id);
-	} else {
-		ctrl->shash_id = host->dhchap_hash_id;
-	}
-
-	/* Skip the 'DHHC-1:XX:' prefix */
-	nvme_auth_free_key(ctrl->host_key);
-	ctrl->host_key = nvme_auth_extract_key(host->dhchap_secret + 10,
-					       host->dhchap_key_hash);
-	if (IS_ERR(ctrl->host_key)) {
+	/*
+	 * Skip the 'DHHC-1:XX:' prefix. Keep the old keys until the new ones
+	 * are there: a controller without a host key needs no authentication.
+	 */
+	host_key = nvme_auth_extract_key(host->dhchap_secret + 10,
+					 host->dhchap_key_hash);
+	if (IS_ERR(host_key)) {
 		ret = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
-		ctrl->host_key = NULL;
-		goto out_free_hash;
+		goto out_unlock;
 	}
+	if (host->dhchap_ctrl_secret) {
+		ctrl_key = nvme_auth_extract_key(host->dhchap_ctrl_secret + 10,
+						 host->dhchap_ctrl_key_hash);
+		if (IS_ERR(ctrl_key)) {
+			nvme_auth_free_key(host_key);
+			ret = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
+			goto out_unlock;
+		}
+	}
+	nvme_auth_free_key(ctrl->host_key);
+	ctrl->host_key = host_key;
+	nvme_auth_free_key(ctrl->ctrl_key);
+	ctrl->ctrl_key = ctrl_key;
+	ctrl->shash_id = host->dhchap_hash_id;
 #ifdef CONFIG_NVME_TARGET_AUTH_DEBUG
 	pr_debug("%s: using hash %s key %*ph\n", __func__,
 		 ctrl->host_key->hash > 0 ?
 		 nvme_auth_hmac_name(ctrl->host_key->hash) : "none",
 		 (int)ctrl->host_key->len, ctrl->host_key->key);
+	if (ctrl->ctrl_key)
+		pr_debug("%s: using ctrl hash %s key %*ph\n", __func__,
+			 ctrl->ctrl_key->hash > 0 ?
+			 nvme_auth_hmac_name(ctrl->ctrl_key->hash) : "none",
+			 (int)ctrl->ctrl_key->len, ctrl->ctrl_key->key);
 #endif
-	nvme_auth_free_key(ctrl->ctrl_key);
-	if (!host->dhchap_ctrl_secret) {
-		ctrl->ctrl_key = NULL;
-		goto out_unlock;
-	}
-
-	ctrl->ctrl_key = nvme_auth_extract_key(host->dhchap_ctrl_secret + 10,
-					       host->dhchap_ctrl_key_hash);
-	if (IS_ERR(ctrl->ctrl_key)) {
-		ret = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
-		ctrl->ctrl_key = NULL;
-		goto out_free_hash;
-	}
-#ifdef CONFIG_NVME_TARGET_AUTH_DEBUG
-	pr_debug("%s: using ctrl hash %s key %*ph\n", __func__,
-		 ctrl->ctrl_key->hash > 0 ?
-		 nvme_auth_hmac_name(ctrl->ctrl_key->hash) : "none",
-		 (int)ctrl->ctrl_key->len, ctrl->ctrl_key->key);
-#endif
-out_free_hash:
-	if (ret) {
-		if (ctrl->host_key) {
-			nvme_auth_free_key(ctrl->host_key);
-			ctrl->host_key = NULL;
-		}
-		ctrl->shash_id = 0;
-	}
 out_unlock:
 	up_read(&nvmet_config_sem);
 
-- 
2.56.0

[PATCH 65/98] nvmet-auth: take only AUTH_Negotiate for a negotiation

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

nvmet_execute_auth_send() hands a common-type message to
nvmet_auth_negotiate() if its ID is AUTH_Negotiate or matches the step
the queue is in. The second case is wrong: the only other common-type
message a host sends is AUTH_Failure2, and no queue waits in that step.
A host can name the step on purpose though, and its message is then
treated as a new AUTH_Negotiate without the work that goes with a new
transaction, like reloading the keys on the admin queue.

Fail every other common-type message, as AUTH_Failure2 already is.

Fixes: db1312dd9548 ("nvmet: implement basic In-Band Authentication")
Cc: Hannes Reinecke <hare@suse.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/fabrics-cmd-auth.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/target/fabrics-cmd-auth.c b/drivers/nvme/target/fabrics-cmd-auth.c
index a559fec3aa65..5ab943256f38 100644
--- a/drivers/nvme/target/fabrics-cmd-auth.c
+++ b/drivers/nvme/target/fabrics-cmd-auth.c
@@ -350,8 +350,9 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
 			}
 			req->sq->dhchap_step =
 				NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
-		} else if (data->auth_id != req->sq->dhchap_step)
+		} else {
 			goto done_failure1;
+		}
 		/* Validate negotiation parameters */
 		dhchap_status = nvmet_auth_negotiate(req, d, tl);
 		if (dhchap_status == 0)
-- 
2.56.0

[PATCH 66/98] nvmet-auth: free the old data on a new AUTH_Negotiate

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

A host may restart the negotiation at any step. The challenge and the
session key of the abandoned transaction stay in the queue, and
nvmet_auth_challenge() and nvmet_auth_ctrl_sesskey() overwrite the
pointers with new allocations, leaking one or two hash-sized buffers per
round for anyone who can reach the port. The controller challenge stays
too, and if the new transaction brings none but asks for the
controller's response, the old one is hashed with the new hash length:

  BUG: KASAN: slab-out-of-bounds in nvmet_auth_ctrl_hash+0x2d5/0x4b0

Free the old transaction's data when a new one begins.

Fixes: db1312dd9548 ("nvmet: implement basic In-Band Authentication")
Cc: Hannes Reinecke <hare@suse.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/fabrics-cmd-auth.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/nvme/target/fabrics-cmd-auth.c b/drivers/nvme/target/fabrics-cmd-auth.c
index 5ab943256f38..665c56d29d07 100644
--- a/drivers/nvme/target/fabrics-cmd-auth.c
+++ b/drivers/nvme/target/fabrics-cmd-auth.c
@@ -336,6 +336,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
 			/* Restart negotiation */
 			pr_debug("%s: ctrl %d qid %d reset negotiation\n",
 				 __func__, ctrl->cntlid, req->sq->qid);
+			nvmet_auth_sq_free(req->sq);
 			if (!req->sq->qid) {
 				dhchap_status = nvmet_setup_auth(ctrl, req->sq,
 								 true);
-- 
2.56.0

[PATCH 67/98] nvmet-auth: fix the check of the offered DH groups

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

nvmet_auth_negotiate() matches the first DH group in the host's list
that is NOT the configured one, and looks for a fallback among those
that are. So a host offering only groups the target isn't configured
for passes and gets a challenge for a group it never offered, while a
host offering only the NULL group is refused by a target configured for
NULL, since there's no KPP algorithm to fall back to. A fallback can't
work anyway, the DH context and public key belong to the configured
group.

Accept if the configured group is in the list, fail with "DH group
unusable" otherwise.

Fixes: 7a277c37d352 ("nvmet-auth: Diffie-Hellman key exchange support")
Cc: Hannes Reinecke <hare@suse.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/fabrics-cmd-auth.c | 27 +++++---------------------
 1 file changed, 5 insertions(+), 22 deletions(-)

diff --git a/drivers/nvme/target/fabrics-cmd-auth.c b/drivers/nvme/target/fabrics-cmd-auth.c
index 665c56d29d07..8c48140eb542 100644
--- a/drivers/nvme/target/fabrics-cmd-auth.c
+++ b/drivers/nvme/target/fabrics-cmd-auth.c
@@ -47,7 +47,7 @@ static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d, u32 tl)
 {
 	struct nvmet_ctrl *ctrl = req->sq->ctrl;
 	struct nvmf_auth_dhchap_negotiate_data *data = d;
-	int i, hash_id = 0, fallback_hash_id = 0, dhgid, fallback_dhgid;
+	int i, hash_id = 0, fallback_hash_id = 0;
 
 	if (tl < sizeof(*data) +
 			sizeof(struct nvmf_auth_dhchap_protocol_descriptor))
@@ -115,32 +115,15 @@ static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d, u32 tl)
 	}
 	req->sq->shash_id = hash_id;
 
-	dhgid = -1;
-	fallback_dhgid = -1;
 	for (i = 0; i < data->auth_protocol[0].dhchap.dhlen; i++) {
-		int tmp_dhgid = data->auth_protocol[0].dhchap.idlist[i + 30];
-
-		if (tmp_dhgid != ctrl->dh_gid) {
-			dhgid = tmp_dhgid;
+		if (data->auth_protocol[0].dhchap.idlist[i + 30] == ctrl->dh_gid)
 			break;
-		}
-		if (fallback_dhgid < 0) {
-			const char *kpp = nvme_auth_dhgroup_kpp(tmp_dhgid);
-
-			if (crypto_has_kpp(kpp, 0, 0))
-				fallback_dhgid = tmp_dhgid;
-		}
 	}
-	if (dhgid < 0) {
-		if (fallback_dhgid < 0) {
-			pr_debug("%s: ctrl %d qid %d: no usable DH group found\n",
-				 __func__, ctrl->cntlid, req->sq->qid);
-			return NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
-		}
+	if (i == data->auth_protocol[0].dhchap.dhlen) {
 		pr_debug("%s: ctrl %d qid %d: configured DH group %s not found\n",
 			 __func__, ctrl->cntlid, req->sq->qid,
-			 nvme_auth_dhgroup_name(fallback_dhgid));
-		ctrl->dh_gid = fallback_dhgid;
+			 nvme_auth_dhgroup_name(ctrl->dh_gid));
+		return NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
 	}
 	if (ctrl->dh_gid == NVME_AUTH_DHGROUP_NULL && ctrl->concat) {
 		pr_debug("%s: ctrl %d qid %d: NULL DH group invalid "
-- 
2.56.0

[PATCH 68/98] nvmet: accept I/O queues on the controller's port only

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

nvmet_ctrl_find_get() looks the subsystem up through the port of the
request, but then takes the controller from the subsystem list whatever
port it was created on. A controller belongs to one port and one
transport: ctrl->ops is that of the admin queue, and both
->install_queue() and ->delete_ctrl() assume the queues are their own.
So an I/O queue arriving on another port of the same subsystem, possibly
on another transport, is installed into a controller it has no part in,
nvmet_tcp_install_queue() writes to it as a TCP queue, and an error on
it deletes that controller. The Connect comes before any authentication.

Skip controllers of other ports, and compare the transport too since a
port's transport can be changed while it's disabled.

Fixes: a07b4970f464 ("nvmet: add a generic NVMe target")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/core.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 333739f14c2b..d01f8abed66c 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -1507,6 +1507,11 @@ struct nvmet_ctrl *nvmet_ctrl_find_get(const char *subsysnqn,
 				pr_warn("hostnqn mismatch.\n");
 				continue;
 			}
+			/* All queues of a controller come in by its port. */
+			if (ctrl->port != req->port || ctrl->ops != req->ops) {
+				pr_warn("port mismatch.\n");
+				continue;
+			}
 			if (!kref_get_unless_zero(&ctrl->ref))
 				continue;
 
-- 
2.56.0

[PATCH 69/98] nvmet: confine fatal errors of unauthenticated queues

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

The Connect of an I/O queue names its controller by NQNs and cntlid and
needs no secret, DH-HMAC-CHAP comes after it. But a fatal error on any
queue, a failed authentication or a transport error like an unknown PDU
type, takes the whole controller down. So anyone who can reach the port
and knows the NQNs can attach a queue to an authenticated host's
controller and kick that host out, repeatedly.

Add a ->disconnect_queue() transport op and, for an I/O queue that
hasn't passed the authentication asked of it, disconnect just that
queue. Transports without the op (fc, loop) take the controller down as
before. A queue that had authenticated and fails to do so again is the
host's own, there the controller goes down as before too.

[axboe: resolved against the per-queue auth_lock and expiry work that
 is upstream now]

Fixes: db1312dd9548 ("nvmet: implement basic In-Band Authentication")
Cc: Hannes Reinecke <hare@suse.de>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/auth.c             |  9 ++++++---
 drivers/nvme/target/core.c             | 16 ++++++++++++++++
 drivers/nvme/target/fabrics-cmd-auth.c | 20 ++++++++++++++++----
 drivers/nvme/target/nvmet.h            |  7 +++++++
 drivers/nvme/target/rdma.c             |  9 ++++++++-
 drivers/nvme/target/tcp.c              | 11 ++++++++++-
 6 files changed, 63 insertions(+), 9 deletions(-)

diff --git a/drivers/nvme/target/auth.c b/drivers/nvme/target/auth.c
index 5809ce32a2e4..335cf8f3f79e 100644
--- a/drivers/nvme/target/auth.c
+++ b/drivers/nvme/target/auth.c
@@ -261,10 +261,8 @@ void nvmet_destroy_auth(struct nvmet_ctrl *ctrl)
 #endif
 }
 
-bool nvmet_check_auth_status(struct nvmet_req *req)
+bool nvmet_sq_auth_status(struct nvmet_sq *sq)
 {
-	struct nvmet_sq *sq = req->sq;
-
 	if (!sq->ctrl->host_key || sq->authenticated)
 		return true;
 
@@ -272,6 +270,11 @@ bool nvmet_check_auth_status(struct nvmet_req *req)
 	return sq->qid && nvmet_queue_tls_keyid(sq);
 }
 
+bool nvmet_check_auth_status(struct nvmet_req *req)
+{
+	return nvmet_sq_auth_status(req->sq);
+}
+
 int nvmet_auth_host_hash(struct nvmet_req *req, u8 *response,
 			 unsigned int shash_len)
 {
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index d01f8abed66c..eea67fba6f92 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -1828,6 +1828,22 @@ void nvmet_ctrl_fatal_error(struct nvmet_ctrl *ctrl)
 }
 EXPORT_SYMBOL_GPL(nvmet_ctrl_fatal_error);
 
+/*
+ * A fatal error on one queue.  The Connect command of an I/O queue needs no
+ * secret, so until the queue has authenticated it may be anybody's: do not
+ * let it take down the controller of the host whose NQN it has named.
+ */
+void nvmet_sq_fatal_error(struct nvmet_sq *sq)
+{
+	struct nvmet_ctrl *ctrl = sq->ctrl;
+
+	if (sq->qid && ctrl->ops->disconnect_queue && !nvmet_sq_auth_status(sq))
+		ctrl->ops->disconnect_queue(sq);
+	else
+		nvmet_ctrl_fatal_error(ctrl);
+}
+EXPORT_SYMBOL_GPL(nvmet_sq_fatal_error);
+
 ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl,
 		char *traddr, size_t traddr_len)
 {
diff --git a/drivers/nvme/target/fabrics-cmd-auth.c b/drivers/nvme/target/fabrics-cmd-auth.c
index 8c48140eb542..cb6be285b3c4 100644
--- a/drivers/nvme/target/fabrics-cmd-auth.c
+++ b/drivers/nvme/target/fabrics-cmd-auth.c
@@ -33,6 +33,14 @@ static void nvmet_auth_expired_work(struct work_struct *work)
 	mutex_unlock(&sq->auth_lock);
 }
 
+static void nvmet_auth_fatal_error(struct nvmet_sq *sq, bool passed)
+{
+	if (passed)
+		nvmet_ctrl_fatal_error(sq->ctrl);
+	else
+		nvmet_sq_fatal_error(sq);
+}
+
 void nvmet_auth_sq_init(struct nvmet_sq *sq)
 {
 	/* Initialize in-band authentication */
@@ -253,11 +261,13 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
 	void *d;
 	u32 tl;
 	u16 status = 0;
+	bool passed;
 	u8 dhchap_status;
 
 	mutex_lock(&req->sq->auth_lock);
 	mutex_lock(&ctrl->auth_lock);
-
+	/* A queue that fails to authenticate again is no stranger's. */
+	passed = nvmet_sq_auth_status(req->sq);
 	if (req->cmd->auth_send.secp != NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER) {
 		status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
 		req->error_loc =
@@ -430,7 +440,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
 	/* Final states, clear up variables */
 	nvmet_auth_sq_free(req->sq);
 	if (req->sq->dhchap_step == NVME_AUTH_DHCHAP_MESSAGE_FAILURE2)
-		nvmet_ctrl_fatal_error(ctrl);
+		nvmet_auth_fatal_error(req->sq, passed);
 
 complete:
 	mutex_unlock(&ctrl->auth_lock);
@@ -551,10 +561,12 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)
 	void *d;
 	u32 al;
 	u16 status = 0;
+	bool passed;
 
 	mutex_lock(&req->sq->auth_lock);
 	mutex_lock(&ctrl->auth_lock);
-
+	/* A queue that fails to authenticate again is no stranger's. */
+	passed = nvmet_sq_auth_status(req->sq);
 	if (req->cmd->auth_receive.secp != NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER) {
 		status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
 		req->error_loc =
@@ -641,7 +653,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)
 		nvmet_auth_sq_free(req->sq);
 	else if (req->sq->dhchap_step == NVME_AUTH_DHCHAP_MESSAGE_FAILURE1) {
 		nvmet_auth_sq_free(req->sq);
-		nvmet_ctrl_fatal_error(ctrl);
+		nvmet_auth_fatal_error(req->sq, passed);
 	}
 	mutex_unlock(&ctrl->auth_lock);
 	mutex_unlock(&req->sq->auth_lock);
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index cc7fface77de..e354a71514ae 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -441,6 +441,7 @@ struct nvmet_fabrics_ops {
 	int (*add_port)(struct nvmet_port *port);
 	void (*remove_port)(struct nvmet_port *port);
 	void (*delete_ctrl)(struct nvmet_ctrl *ctrl);
+	void (*disconnect_queue)(struct nvmet_sq *sq);
 	void (*disc_traddr)(struct nvmet_req *req,
 			struct nvmet_port *port, char *traddr);
 	ssize_t (*host_traddr)(struct nvmet_ctrl *ctrl,
@@ -622,6 +623,7 @@ void nvmet_sq_destroy(struct nvmet_sq *sq);
 int nvmet_sq_init(struct nvmet_sq *sq, struct nvmet_cq *cq);
 
 void nvmet_ctrl_fatal_error(struct nvmet_ctrl *ctrl);
+void nvmet_sq_fatal_error(struct nvmet_sq *sq);
 
 void nvmet_update_cc(struct nvmet_ctrl *ctrl, u32 new);
 
@@ -989,6 +991,7 @@ void nvmet_destroy_auth(struct nvmet_ctrl *ctrl);
 void nvmet_auth_sq_free(struct nvmet_sq *sq);
 void nvmet_auth_sq_destroy(struct nvmet_sq *sq);
 int nvmet_setup_dhgroup(struct nvmet_ctrl *ctrl, u8 dhgroup_id);
+bool nvmet_sq_auth_status(struct nvmet_sq *sq);
 bool nvmet_check_auth_status(struct nvmet_req *req);
 int nvmet_auth_host_hash(struct nvmet_req *req, u8 *response,
 			 unsigned int hash_len);
@@ -1015,6 +1018,10 @@ static inline void nvmet_auth_sq_init(struct nvmet_sq *sq)
 static inline void nvmet_destroy_auth(struct nvmet_ctrl *ctrl) {};
 static inline void nvmet_auth_sq_free(struct nvmet_sq *sq) {};
 static inline void nvmet_auth_sq_destroy(struct nvmet_sq *sq) {};
+static inline bool nvmet_sq_auth_status(struct nvmet_sq *sq)
+{
+	return true;
+}
 static inline bool nvmet_check_auth_status(struct nvmet_req *req)
 {
 	return true;
diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index ffa1e172c259..c4b848f84465 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -687,7 +687,7 @@ static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp)
 static void nvmet_rdma_error_comp(struct nvmet_rdma_queue *queue)
 {
 	if (queue->nvme_sq.ctrl) {
-		nvmet_ctrl_fatal_error(queue->nvme_sq.ctrl);
+		nvmet_sq_fatal_error(&queue->nvme_sq);
 	} else {
 		/*
 		 * we didn't setup the controller yet in case
@@ -1848,6 +1848,12 @@ static int nvmet_rdma_cm_handler(struct rdma_cm_id *cm_id,
 	return ret;
 }
 
+static void nvmet_rdma_disconnect_queue(struct nvmet_sq *sq)
+{
+	nvmet_rdma_queue_disconnect(container_of(sq, struct nvmet_rdma_queue,
+						 nvme_sq));
+}
+
 static void nvmet_rdma_delete_ctrl(struct nvmet_ctrl *ctrl)
 {
 	struct nvmet_rdma_queue *queue, *n;
@@ -2078,6 +2084,7 @@ static const struct nvmet_fabrics_ops nvmet_rdma_ops = {
 	.remove_port		= nvmet_rdma_remove_port,
 	.queue_response		= nvmet_rdma_queue_response,
 	.delete_ctrl		= nvmet_rdma_delete_ctrl,
+	.disconnect_queue	= nvmet_rdma_disconnect_queue,
 	.disc_traddr		= nvmet_rdma_disc_port_addr,
 	.host_traddr		= nvmet_rdma_host_port_addr,
 	.get_mdts		= nvmet_rdma_get_mdts,
diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index a2a5f39d2bb7..c8f0c52a1b95 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -412,7 +412,7 @@ static void nvmet_tcp_socket_error(struct nvmet_tcp_queue *queue, int status)
 	if (status == -EPIPE || status == -ECONNRESET || !queue->nvme_sq.ctrl)
 		kernel_sock_shutdown(queue->sock, SHUT_RDWR);
 	else
-		nvmet_ctrl_fatal_error(queue->nvme_sq.ctrl);
+		nvmet_sq_fatal_error(&queue->nvme_sq);
 }
 
 static int nvmet_tcp_map_data(struct nvmet_tcp_cmd *cmd)
@@ -2223,6 +2223,14 @@ static void nvmet_tcp_remove_port(struct nvmet_port *nport)
 	kfree(port);
 }
 
+static void nvmet_tcp_disconnect_queue(struct nvmet_sq *sq)
+{
+	struct nvmet_tcp_queue *queue =
+		container_of(sq, struct nvmet_tcp_queue, nvme_sq);
+
+	kernel_sock_shutdown(queue->sock, SHUT_RDWR);
+}
+
 static void nvmet_tcp_delete_ctrl(struct nvmet_ctrl *ctrl)
 {
 	struct nvmet_tcp_queue *queue;
@@ -2300,6 +2308,7 @@ static const struct nvmet_fabrics_ops nvmet_tcp_ops = {
 	.remove_port		= nvmet_tcp_remove_port,
 	.queue_response		= nvmet_tcp_queue_response,
 	.delete_ctrl		= nvmet_tcp_delete_ctrl,
+	.disconnect_queue	= nvmet_tcp_disconnect_queue,
 	.install_queue		= nvmet_tcp_install_queue,
 	.disc_traddr		= nvmet_tcp_disc_port_addr,
 	.host_traddr		= nvmet_tcp_host_port_addr,
-- 
2.56.0

[PATCH 70/98] nvmet-tcp: don't wait for the host in the accept work

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

On a port that offers TLS but doesn't require it,
nvmet_tcp_alloc_queue() peeks at the first bytes of a new connection to
tell an ICReq from a TLS handshake. That receive blocks until the host
sends something or closes, and it runs in the accept work, of which
there is one per port. A connection that stays silent blocks every
other connect on that port for as long as it's held open, no secret
needed. And nvmet_tcp_remove_port() waits for the accept work with
nvmet_config_sem held for writing, so an admin taking the port down
hangs there and no port of the target answers a Connect anymore.

Install the socket callbacks first and peek without blocking from the
io work, which data_ready() reruns. If it's a handshake, restore the
default callbacks for the upcall as before, and make the upcall from
one more run of the io work so no run is left that could steal data
from the handshake. The state change happens under state_lock and only
if the queue isn't already going away. The io work doesn't hold the
port like the accept work did, so note the keyring serial, which is all
the upcall needs from the port, at allocation.

A partially arrived header used to be taken for "no TLS", now the rest
is waited for. A silent connection still holds its queue, as on a port
without TLS.

Fixes: 70525e5d82f6 ("nvmet-tcp: peek icreq before starting TLS")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/tcp.c | 67 ++++++++++++++++++++++++++++++++++-----
 1 file changed, 59 insertions(+), 8 deletions(-)

diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index c8f0c52a1b95..3d24efe3b7c3 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -140,6 +140,7 @@ struct nvmet_tcp_cmd {
 
 enum nvmet_tcp_queue_state {
 	NVMET_TCP_Q_CONNECTING,
+	NVMET_TCP_Q_TLS_PEEK,
 	NVMET_TCP_Q_TLS_HANDSHAKE,
 	NVMET_TCP_Q_LIVE,
 	NVMET_TCP_Q_DISCONNECTING,
@@ -176,6 +177,7 @@ struct nvmet_tcp_queue {
 
 	/* TLS state */
 	key_serial_t		tls_pskid;
+	key_serial_t		tls_keyring;
 	struct delayed_work	tls_handshake_tmo_work;
 
 	unsigned long           poll_end;
@@ -1496,6 +1498,8 @@ static bool nvmet_tcp_check_queue_deadline(struct nvmet_tcp_queue *queue,
 	return !time_after(jiffies, queue->poll_end);
 }
 
+static bool nvmet_tcp_tls_peek(struct nvmet_tcp_queue *queue);
+
 static void nvmet_tcp_io_work(struct work_struct *w)
 {
 	struct nvmet_tcp_queue *queue =
@@ -1503,6 +1507,11 @@ static void nvmet_tcp_io_work(struct work_struct *w)
 	bool pending;
 	int ret, ops = 0;
 
+	if ((queue->state == NVMET_TCP_Q_TLS_PEEK ||
+	     queue->state == NVMET_TCP_Q_TLS_HANDSHAKE) &&
+	    !nvmet_tcp_tls_peek(queue))
+		return;
+
 	do {
 		pending = false;
 
@@ -1828,12 +1837,9 @@ static int nvmet_tcp_try_peek_pdu(struct nvmet_tcp_queue *queue)
 	struct msghdr msg = {
 		.msg_control = cbuf,
 		.msg_controllen = sizeof(cbuf),
-		.msg_flags = MSG_PEEK,
+		.msg_flags = MSG_PEEK | MSG_DONTWAIT,
 	};
 
-	if (nvmet_port_secure_channel_required(queue->port->nport))
-		return 0;
-
 	len = kernel_recvmsg(queue->sock, &msg, &iov, 1,
 			iov.iov_len, msg.msg_flags);
 	if (unlikely(len < 0)) {
@@ -1955,7 +1961,7 @@ static int nvmet_tcp_tls_handshake(struct nvmet_tcp_queue *queue)
 	args.ta_sock = queue->sock;
 	args.ta_done = nvmet_tcp_tls_handshake_done;
 	args.ta_data = queue;
-	args.ta_keyring = key_serial(queue->port->nport->keyring);
+	args.ta_keyring = queue->tls_keyring;
 	args.ta_timeout_ms = tls_handshake_timeout * 1000;
 
 	ret = tls_server_hello_psk(&args, GFP_KERNEL);
@@ -1968,8 +1974,51 @@ static int nvmet_tcp_tls_handshake(struct nvmet_tcp_queue *queue)
 	}
 	return ret;
 }
+
+/*
+ * TLS is offered, but not required: the first bytes tell what the host wants.
+ * Returns true if the queue goes on without TLS.
+ */
+static bool nvmet_tcp_tls_peek(struct nvmet_tcp_queue *queue)
+{
+	int ret;
+
+	if (queue->state == NVMET_TCP_Q_TLS_HANDSHAKE) {
+		if (nvmet_tcp_tls_handshake(queue))
+			nvmet_tcp_schedule_release_queue(queue);
+		return false;
+	}
+
+	/* ->data_ready() brings us back */
+	ret = nvmet_tcp_try_peek_pdu(queue);
+	if (ret == -EAGAIN)
+		return false;
+
+	/* The upcall wants the default callbacks */
+	if (!ret)
+		nvmet_tcp_restore_socket_callbacks(queue);
+
+	spin_lock_bh(&queue->state_lock);
+	if (queue->state != NVMET_TCP_Q_TLS_PEEK) {
+		spin_unlock_bh(&queue->state_lock);
+		return false;
+	}
+	queue->state = ret ? NVMET_TCP_Q_CONNECTING : NVMET_TCP_Q_TLS_HANDSHAKE;
+	spin_unlock_bh(&queue->state_lock);
+	if (ret)
+		return true;
+
+	/*
+	 * ->data_ready() may have queued the io work meanwhile, nothing does
+	 * from now on. The handshake must not start before that run is over,
+	 * so leave it to that run.
+	 */
+	queue_work_on(queue_cpu(queue), nvmet_tcp_wq, &queue->io_work);
+	return false;
+}
 #else
 static void nvmet_tcp_tls_handshake_timeout(struct work_struct *w) {}
+static bool nvmet_tcp_tls_peek(struct nvmet_tcp_queue *queue) { return true; }
 #endif
 
 static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
@@ -2039,14 +2088,16 @@ static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
 		sk->sk_user_data = NULL;
 		sk->sk_data_ready = port->data_ready;
 		write_unlock_bh(&sk->sk_callback_lock);
-		if (!nvmet_tcp_try_peek_pdu(queue)) {
+		/* The port may be gone when the upcall is made */
+		queue->tls_keyring = key_serial(port->nport->keyring);
+		if (nvmet_port_secure_channel_required(port->nport)) {
 			if (!nvmet_tcp_tls_handshake(queue))
 				return;
 			/* TLS handshake failed, terminate the connection */
 			goto out_destroy_sq;
 		}
-		/* Not a TLS connection, continue with normal processing */
-		queue->state = NVMET_TCP_Q_CONNECTING;
+		/* Don't wait for the host here: others want to connect, too */
+		queue->state = NVMET_TCP_Q_TLS_PEEK;
 	}
 #endif
 
-- 
2.56.0

[PATCH 71/98] nvmet: check the host NQN against the TLS PSK identity

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

The identity of a TLS PSK names the host and subsystem the key was
generated for. The target remembers which key a handshake ended with,
but never compares it with the host NQN in the Connect that follows. So
a host holding any PSK in the port's keyring can connect as another
host, sees what that host may see, and isn't asked for DH-HMAC-CHAP
since TLS queues skip it. It can add an I/O queue to that host's TLS
controller the same way.

Refuse a Connect whose host NQN isn't the one in the PSK identity of the
queue. The subsystem NQN isn't compared, a discovery subsystem goes by
two names.

Fixes: 675b453e0241 ("nvmet-tcp: enable TLS handshake upcall")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/fabrics-cmd.c | 33 +++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/drivers/nvme/target/fabrics-cmd.c b/drivers/nvme/target/fabrics-cmd.c
index ddb732f62981..089e3082c388 100644
--- a/drivers/nvme/target/fabrics-cmd.c
+++ b/drivers/nvme/target/fabrics-cmd.c
@@ -264,6 +264,30 @@ static u32 nvmet_connect_result(struct nvmet_ctrl *ctrl, struct nvmet_sq *sq)
 		(needs_auth ? NVME_CONNECT_AUTHREQ_ATR : 0);
 }
 
+/*
+ * The identity of a TLS PSK names the host that the key was made for:
+ * "NVMe<version><type><hash> <host NQN> <subsystem NQN> ...".
+ * Whoever has proven to hold the key is that host, and no other one.
+ */
+static u16 nvmet_check_tls_host(struct nvmet_sq *sq, const char *hostnqn)
+{
+#ifdef CONFIG_NVME_TARGET_TCP_TLS
+	size_t len = strlen(hostnqn);
+	const char *id;
+
+	if (!sq->tls_key)
+		return 0;
+
+	id = strchr(sq->tls_key->description, ' ');
+	if (!id || strncmp(id + 1, hostnqn, len) || id[len + 1] != ' ') {
+		pr_warn("TLS PSK %08x does not belong to host %s\n",
+			key_serial(sq->tls_key), hostnqn);
+		return NVME_SC_CONNECT_INVALID_HOST | NVME_STATUS_DNR;
+	}
+#endif
+	return 0;
+}
+
 static void nvmet_execute_admin_connect(struct nvmet_req *req)
 {
 	struct nvmf_connect_command *c = &req->cmd->connect;
@@ -309,6 +333,10 @@ static void nvmet_execute_admin_connect(struct nvmet_req *req)
 	d->subsysnqn[NVMF_NQN_FIELD_LEN - 1] = '\0';
 	d->hostnqn[NVMF_NQN_FIELD_LEN - 1] = '\0';
 
+	args.status = nvmet_check_tls_host(req->sq, d->hostnqn);
+	if (args.status)
+		goto out;
+
 	args.subsysnqn = d->subsysnqn;
 	args.hostnqn = d->hostnqn;
 	args.hostid = &d->hostid;
@@ -363,6 +391,11 @@ static void nvmet_execute_io_connect(struct nvmet_req *req)
 
 	d->subsysnqn[NVMF_NQN_FIELD_LEN - 1] = '\0';
 	d->hostnqn[NVMF_NQN_FIELD_LEN - 1] = '\0';
+
+	status = nvmet_check_tls_host(req->sq, d->hostnqn);
+	if (status)
+		goto out;
+
 	ctrl = nvmet_ctrl_find_get(d->subsysnqn, d->hostnqn,
 				   le16_to_cpu(d->cntlid), req);
 	if (!ctrl) {
-- 
2.56.0

[PATCH 72/98] nvmet: fix NULL dereference of PCI endpoint targets

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

nvmet_pci_epf_create_ctrl() calls nvmet_alloc_ctrl() without a queue,
the admin queue comes later from the host. nvmet_alloc_ctrl() passes
args->sq to nvmet_queue_tls_keyid(), which with
CONFIG_NVME_TARGET_TCP_TLS dereferences it, so every PCI endpoint
controller creation crashes on such a kernel. The debugfs tls_key file
passes ctrl->sqs[0] to the same function, which is NULL until the admin
queue exists and again once a fabrics host has closed it while keeping
an I/O queue.

A queue that isn't there has no TLS key.

Fixes: fa2e0f8bbc68 ("nvmet-tcp: support secure channel concatenation")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/nvmet.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index e354a71514ae..5c2021b89ec4 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -964,7 +964,8 @@ static inline void nvmet_req_bio_put(struct nvmet_req *req, struct bio *bio)
 #ifdef CONFIG_NVME_TARGET_TCP_TLS
 static inline key_serial_t nvmet_queue_tls_keyid(struct nvmet_sq *sq)
 {
-	return sq->tls_key ? key_serial(sq->tls_key) : 0;
+	/* A PCI endpoint controller is created without a queue */
+	return sq && sq->tls_key ? key_serial(sq->tls_key) : 0;
 }
 static inline void nvmet_sq_put_tls_key(struct nvmet_sq *sq)
 {
-- 
2.56.0

[PATCH 73/98] nvmet-tcp: stop listening when a port is removed

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

nvmet_tcp_accept_work() accepts until there's nothing left. A
connection that is already dead when accepted is torn down right there,
which takes a few ms, and more can arrive meanwhile.
nvmet_tcp_remove_port() waits for the work with cancel_work_sync(), so
someone opening and closing connections in a loop keeps the removal from
finishing. It holds nvmet_config_sem for writing, so nothing can be
configured and no port answers a Connect until the flood stops. No
secret needed, just an admin removing the port at the wrong time.

Shut the listening socket down first. Pending connections are dropped,
as they were on release, and the next kernel_accept() fails and ends the
work.

Fixes: 872d26a391da ("nvmet-tcp: add NVMe over TCP target driver")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/tcp.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index 3d24efe3b7c3..0b70d3647e28 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -2263,6 +2263,8 @@ static void nvmet_tcp_remove_port(struct nvmet_port *nport)
 	port->sock->sk->sk_data_ready = port->data_ready;
 	port->sock->sk->sk_user_data = NULL;
 	write_unlock_bh(&port->sock->sk->sk_callback_lock);
+	/* The accept work goes on for as long as there is something to accept */
+	kernel_sock_shutdown(port->sock, SHUT_RDWR);
 	cancel_work_sync(&port->accept_work);
 	/*
 	 * Destroy the remaining queues, which are not belong to any
-- 
2.56.0

[PATCH 74/98] nvme-tcp: keep the page frag cache with the controller

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

nvme_tcp_init_request() allocates the PDU from the queue's page frag
cache under pf_cache_lock, and it's called not just for the tag set but
for the flush request of every hctx of a new namespace. The cache and
the lock are torn down and reinitialized with the connection though,
and a namespace scan isn't stopped by error recovery, so it allocates
from a cache that's being drained or set up:

  WARNING: kernel/locking/mutex.c:625 at __mutex_lock+0xa25/0x1a30
   nvme_tcp_init_request+0x8c/0x1a0
   blk_mq_alloc_and_init_hctx+0x28e/0x6c0

Without lock debugging a waiter can spin forever on a mutex that's
being reinitialized, an allocation beside the drain can return a
fragment of a freed page, and what's allocated after the drain leaks if
the controller goes away. The target, or anyone who can break the
connection, picks the moment.

The PDUs outlive the connection anyway. Initialize the lock with the
controller and drain the caches when it's freed.

Fixes: a5053c92b3db ("nvme-tcp: fix memory leak when freeing a queue")
Fixes: 36ac05f7cfd5 ("nvme-tcp: fix usage of page_frag_cache")
Cc: Keith Busch <kbusch@kernel.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/host/tcp.c | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index c9531e653a2c..4aa666dd0266 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -162,6 +162,7 @@ struct nvme_tcp_queue_setup_ctx {
 struct nvme_tcp_ctrl {
 	/* read only in the hot path */
 	struct nvme_tcp_queue	*queues;
+	int			nr_queues;	/* allocated, see queue_count */
 	struct blk_mq_tag_set	tag_set;
 
 	/* other member variables */
@@ -1668,8 +1669,6 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
 	if (!test_and_clear_bit(NVME_TCP_Q_ALLOCATED, &queue->flags))
 		return;
 
-	page_frag_cache_drain(&queue->pf_cache);
-
 	/**
 	 * Prevent memory reclaim from triggering block I/O during socket
 	 * teardown. The socket release path fput -> tcp_close ->
@@ -1692,7 +1691,6 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid)
 
 	kfree(queue->pdu);
 	mutex_destroy(&queue->send_mutex);
-	mutex_destroy(&queue->pf_cache_lock);
 }
 
 static int nvme_tcp_init_connection(struct nvme_tcp_queue *queue)
@@ -2021,7 +2019,6 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
 	INIT_LIST_HEAD(&queue->send_list);
 	mutex_init(&queue->send_mutex);
 	INIT_WORK(&queue->io_work, nvme_tcp_io_work);
-	mutex_init(&queue->pf_cache_lock);
 
 	if (qid > 0)
 		queue->cmnd_capsule_len = nctrl->ioccsz * 16;
@@ -2153,7 +2150,6 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
 	queue->sock = NULL;
 err_destroy_mutex:
 	mutex_destroy(&queue->send_mutex);
-	mutex_destroy(&queue->pf_cache_lock);
 	return ret;
 }
 
@@ -2840,6 +2836,10 @@ static void nvme_tcp_stop_ctrl(struct nvme_ctrl *ctrl)
 static void nvme_tcp_free_ctrl(struct nvme_ctrl *nctrl)
 {
 	struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl);
+	int i;
+
+	for (i = 0; i < ctrl->nr_queues; i++)
+		page_frag_cache_drain(&ctrl->queues[i].pf_cache);
 
 	if (list_empty_careful(&ctrl->list))
 		goto free_ctrl;
@@ -3265,9 +3265,15 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev,
 		ret = -ENOMEM;
 		goto out_free_ctrl;
 	}
-	/* A poller may try the lock of a queue that has been freed */
-	for (i = 0; i < ctrl->ctrl.queue_count; i++)
+	ctrl->nr_queues = ctrl->ctrl.queue_count;
+	/*
+	 * A poller may try the lock of a queue that has been freed, and a
+	 * request of a new namespace may be set up for one.
+	 */
+	for (i = 0; i < ctrl->nr_queues; i++) {
 		mutex_init(&ctrl->queues[i].queue_lock);
+		mutex_init(&ctrl->queues[i].pf_cache_lock);
+	}
 
 	ctrl->net = get_net(current->nsproxy->net_ns);
 
-- 
2.56.0

[PATCH 75/98] nvmet-tcp: hold the queue while the TLS timer is armed

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

nvmet_tcp_tls_handshake() hands the queue to tls_server_hello_psk() and
arms the timeout work afterwards. The reference it takes belongs to the
handshake and is dropped by nvmet_tcp_tls_handshake_done(). If the
handshake already ended by then, the callback found no timer to cancel,
and if it failed because the host closed the connection, the queue may
be gone as well. The timer is then armed in freed memory and fires
there. Anyone who can reach a TLS port can start a handshake and close
the connection, it just takes a quick tlshd.

Hold a second reference until the timer is armed, and arm it under
state_lock only if the handshake is still in progress. The callback
changes the state under that lock before it cancels the timer.

Fixes: 675b453e0241 ("nvmet-tcp: enable TLS handshake upcall")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/tcp.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index 0b70d3647e28..908614fa82b0 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -1955,6 +1955,8 @@ static int nvmet_tcp_tls_handshake(struct nvmet_tcp_queue *queue)
 		return -EINVAL;
 	}
 
+	/* One for the handshake, one to keep the queue until we are through */
+	kref_get(&queue->kref);
 	kref_get(&queue->kref);
 	pr_debug("queue %d: TLS ServerHello\n", queue->idx);
 	memset(&args, 0, sizeof(args));
@@ -1969,9 +1971,15 @@ static int nvmet_tcp_tls_handshake(struct nvmet_tcp_queue *queue)
 		kref_put(&queue->kref, nvmet_tcp_release_queue);
 		pr_err("failed to start TLS, err=%d\n", ret);
 	} else {
-		queue_delayed_work(nvmet_wq, &queue->tls_handshake_tmo_work,
-				   tls_handshake_timeout * HZ);
+		/* The handshake may be over, then nobody cancels the timer */
+		spin_lock_bh(&queue->state_lock);
+		if (queue->state == NVMET_TCP_Q_TLS_HANDSHAKE)
+			queue_delayed_work(nvmet_wq,
+					   &queue->tls_handshake_tmo_work,
+					   tls_handshake_timeout * HZ);
+		spin_unlock_bh(&queue->state_lock);
 	}
+	kref_put(&queue->kref, nvmet_tcp_release_queue);
 	return ret;
 }
 
-- 
2.56.0

[PATCH 76/98] nvmet: fix NULL dereference in debugfs host_traddr

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

The host_traddr callbacks of tcp, rdma and fc take the host address from
the transport queue around ctrl->sqs[0], which nvmet_sq_destroy() clears
when the admin queue goes away while the controller lives on with its
I/O queues. A host that closes the admin connection and keeps an I/O
queue leaves such a controller behind, and reading its host_traddr in
debugfs dereferences a near-NULL pointer:

  BUG: kernel NULL pointer dereference, address: 00000000000002e0
  RIP: 0010:nvmet_tcp_host_port_addr+0x19/0x50 [nvmet_tcp]

Don't ask the transport if there's no admin queue. Nothing holds the
queue for debugfs, so a reader that's past the check when the queue
goes away is still on its own.

Fixes: 7e5c3de3f2f2 ("nvmet: add 'host_traddr' callback for debugfs")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/core.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index eea67fba6f92..142df0db5208 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -1849,6 +1849,9 @@ ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl,
 {
 	if (!ctrl->ops->host_traddr)
 		return -EOPNOTSUPP;
+	/* They all take the address from the admin queue */
+	if (!ctrl->sqs[0])
+		return -ENOTCONN;
 	return ctrl->ops->host_traddr(ctrl, traddr, traddr_len);
 }
 
-- 
2.56.0

Re: [PATCH 76/98] nvmet: fix NULL dereference in debugfs host_traddr

[Bart Van Assche]

On 10/28/26 09:42, Jens Axboe wrote:
> @@ -1849,6 +1849,9 @@ ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl,
>  {
>  	if (!ctrl->ops->host_traddr)
>  		return -EOPNOTSUPP;
> +	/* They all take the address from the admin queue */
> +	if (!ctrl->sqs[0])
> +		return -ENOTCONN;
>  	return ctrl->ops->host_traddr(ctrl, traddr, traddr_len);
>  }

As the commit message notes, nothing synchronizes this check with
nvmet_sq_destroy() clearing ctrl->sqs[0] and the transport freeing the
admin queue: nvmet_tcp_host_port_addr(), nvmet_rdma_host_port_addr(),
and nvmet_fc_host_traddr() all re-read ctrl->sqs[0] unlocked and
dereference the container transport queue, leaving both a TOCTOU NULL
dereference and a use-after-free.

Instead of reaching into ctrl->sqs[0] from debugfs, why not format and
save the host transport address once in a char
host_traddr[NVMF_TRADDR_SIZE] buffer on struct nvmet_ctrl when the admin
queue connects, and have nvmet_ctrl_host_traddr() return that string?
That avoids touching the admin queue after setup altogether.

Re: [PATCH 76/98] nvmet: fix NULL dereference in debugfs host_traddr

[Bart Van Assche]

On 10/28/26 09:42, Jens Axboe wrote:
> @@ -1849,6 +1849,9 @@ ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl,
>  {
>  	if (!ctrl->ops->host_traddr)
>  		return -EOPNOTSUPP;
> +	/* They all take the address from the admin queue */
> +	if (!ctrl->sqs[0])
> +		return -ENOTCONN;
>  	return ctrl->ops->host_traddr(ctrl, traddr, traddr_len);
>  }

As the commit message notes, nothing synchronizes this check with
nvmet_sq_destroy() clearing ctrl->sqs[0] and the transport freeing the
admin queue: nvmet_tcp_host_port_addr(), nvmet_rdma_host_port_addr(),
and nvmet_fc_host_traddr() all re-read ctrl->sqs[0] unlocked and
dereference the container transport queue, leaving a TOCTOU NULL
dereference and use-after-free.

Since args->sq is already passed to nvmet_alloc_ctrl() when the admin
queue creates the controller, we can pass struct nvmet_sq * directly to
ops->host_traddr, format the address once into ctrl->host_traddr before
nvmet_debugfs_ctrl_setup(ctrl), and have nvmet_ctrl_host_traddr() copy
out ctrl->host_traddr. That avoids touching ctrl->sqs[0] or the admin
transport queue after controller allocation altogether, and also keeps
debugfs host_traddr working even if the admin connection has closed.

Here is a replacement patch (checked with checkpatch.pl and
syntax-checked with Clang; not runtime-tested):

---
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index eea67fba6f92..b837c591af36 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -1729,6 +1729,10 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)
 		goto init_pr_fail;
 	list_add_tail(&ctrl->subsys_entry, &subsys->ctrls);
 	nvmet_setup_p2p_ns_map(ctrl, args->p2p_client);
+	if (ctrl->ops->host_traddr && args->sq &&
+	    ctrl->ops->host_traddr(args->sq, ctrl->host_traddr,
+				   sizeof(ctrl->host_traddr)) < 0)
+		ctrl->host_traddr[0] = '\0';
 	nvmet_debugfs_ctrl_setup(ctrl);
 	mutex_unlock(&subsys->lock);
 	up_read(&nvmet_config_sem);
@@ -1849,7 +1853,9 @@ ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl,
 {
 	if (!ctrl->ops->host_traddr)
 		return -EOPNOTSUPP;
-	return ctrl->ops->host_traddr(ctrl, traddr, traddr_len);
+	if (!ctrl->host_traddr[0])
+		return -EINVAL;
+	return strscpy(traddr, ctrl->host_traddr, traddr_len);
 }
 
 static struct nvmet_subsys *nvmet_find_get_subsys(struct nvmet_port *port,
diff --git a/drivers/nvme/target/fc.c b/drivers/nvme/target/fc.c
index eecdffefa24f..1e1583c8b7e7 100644
--- a/drivers/nvme/target/fc.c
+++ b/drivers/nvme/target/fc.c
@@ -2960,10 +2960,9 @@ nvmet_fc_discovery_chg(struct nvmet_port *port)
 }
 
 static ssize_t
-nvmet_fc_host_traddr(struct nvmet_ctrl *ctrl,
+nvmet_fc_host_traddr(struct nvmet_sq *sq,
 		char *traddr, size_t traddr_size)
 {
-	struct nvmet_sq *sq = ctrl->sqs[0];
 	struct nvmet_fc_tgt_queue *queue =
 		container_of(sq, struct nvmet_fc_tgt_queue, nvme_sq);
 	struct nvmet_fc_tgtport *tgtport = queue->assoc ? queue->assoc->tgtport : NULL;
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 5c2021b89ec4..e05cbf463751 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -297,6 +297,7 @@ struct nvmet_ctrl {
 	u32			nr_changed_ns;
 
 	char			hostnqn[NVMF_NQN_FIELD_LEN];
+	char			host_traddr[NVMF_TRADDR_SIZE];
 
 	struct device		*p2p_client;
 	struct radix_tree_root	p2p_ns_map;
@@ -444,7 +445,7 @@ struct nvmet_fabrics_ops {
 	void (*disconnect_queue)(struct nvmet_sq *sq);
 	void (*disc_traddr)(struct nvmet_req *req,
 			struct nvmet_port *port, char *traddr);
-	ssize_t (*host_traddr)(struct nvmet_ctrl *ctrl,
+	ssize_t (*host_traddr)(struct nvmet_sq *sq,
 			char *traddr, size_t traddr_len);
 	u16 (*install_queue)(struct nvmet_sq *nvme_sq);
 	void (*discovery_chg)(struct nvmet_port *port);
diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index c4b848f84465..5e24cdb3a71f 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -2050,10 +2050,9 @@ static void nvmet_rdma_disc_port_addr(struct nvmet_req *req,
 	}
 }
 
-static ssize_t nvmet_rdma_host_port_addr(struct nvmet_ctrl *ctrl,
+static ssize_t nvmet_rdma_host_port_addr(struct nvmet_sq *nvme_sq,
 		char *traddr, size_t traddr_len)
 {
-	struct nvmet_sq *nvme_sq = ctrl->sqs[0];
 	struct nvmet_rdma_queue *queue =
 		container_of(nvme_sq, struct nvmet_rdma_queue, nvme_sq);
 
diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index 908614fa82b0..57b00f3ae988 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -2348,10 +2348,9 @@ static void nvmet_tcp_disc_port_addr(struct nvmet_req *req,
 	}
 }
 
-static ssize_t nvmet_tcp_host_port_addr(struct nvmet_ctrl *ctrl,
+static ssize_t nvmet_tcp_host_port_addr(struct nvmet_sq *sq,
 			char *traddr, size_t traddr_len)
 {
-	struct nvmet_sq *sq = ctrl->sqs[0];
 	struct nvmet_tcp_queue *queue =
 		container_of(sq, struct nvmet_tcp_queue, nvme_sq);
 

[PATCH 77/98] nvmet-tcp: let the accept work come to an end

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

nvmet_tcp_accept_work() loops until there's nothing left to accept, and
anyone opening and closing connections fast enough keeps it running
indefinitely, since tearing down a dead connection takes long enough for
more to arrive. Whoever flushes nvmet_wq waits for it: removing any
other port of the target does in nvmet_port_release(), and so does
unloading the rdma and fc modules.

Accept a bounded number of connections per run and requeue the work. A
flush doesn't wait for work queued behind it.

Fixes: 872d26a391da ("nvmet-tcp: add NVMe over TCP target driver")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/tcp.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index 908614fa82b0..a0adcc7f0740 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -85,6 +85,7 @@ MODULE_PARM_DESC(tls_handshake_timeout,
 #define NVMET_TCP_RECV_BUDGET		8
 #define NVMET_TCP_SEND_BUDGET		8
 #define NVMET_TCP_IO_WORK_BUDGET	64
+#define NVMET_TCP_ACCEPT_BUDGET		16
 
 enum nvmet_tcp_send_state {
 	NVMET_TCP_SEND_DATA_PDU,
@@ -2145,9 +2146,9 @@ static void nvmet_tcp_accept_work(struct work_struct *w)
 	struct nvmet_tcp_port *port =
 		container_of(w, struct nvmet_tcp_port, accept_work);
 	struct socket *newsock;
-	int ret;
+	int ret, i;
 
-	while (true) {
+	for (i = 0; i < NVMET_TCP_ACCEPT_BUDGET; i++) {
 		ret = kernel_accept(port->sock, &newsock, O_NONBLOCK);
 		if (ret < 0) {
 			if (ret != -EAGAIN)
@@ -2156,6 +2157,9 @@ static void nvmet_tcp_accept_work(struct work_struct *w)
 		}
 		nvmet_tcp_alloc_queue(port, newsock);
 	}
+
+	/* There may be more. Come to an end for those who flush nvmet_wq. */
+	queue_work(nvmet_wq, &port->accept_work);
 }
 
 static void nvmet_tcp_listen_data_ready(struct sock *sk)
-- 
2.56.0

[PATCH 78/98] nvmet-tcp: don't read a removed port at Connect time

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

nvmet_tcp_alloc_cmd() takes the nvmet port of a command from
queue->port, and the commands of a queue are allocated from the io work
when the Connect arrives. nvmet_tcp_remove_port() shuts down the queues
that have no controller yet and frees the port without waiting for
them, they're released later from a work. An io work handling a Connect
at that moment reads the freed port for every command of the queue and
hands the result to the core.

Note the nvmet port in the queue at allocation. It outlives the
transport port.

Fixes: 872d26a391da ("nvmet-tcp: add NVMe over TCP target driver")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/tcp.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index a0adcc7f0740..4cc65decb02e 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -151,6 +151,7 @@ enum nvmet_tcp_queue_state {
 struct nvmet_tcp_queue {
 	struct socket		*sock;
 	struct nvmet_tcp_port	*port;
+	struct nvmet_port	*nport;
 	struct work_struct	io_work;
 	struct nvmet_cq		nvme_cq;
 	struct nvmet_sq		nvme_sq;
@@ -1544,7 +1545,7 @@ static int nvmet_tcp_alloc_cmd(struct nvmet_tcp_queue *queue,
 	u8 hdgst = nvmet_tcp_hdgst_len(queue);
 
 	c->queue = queue;
-	c->req.port = queue->port->nport;
+	c->req.port = queue->nport;
 
 	c->cmd_pdu = page_frag_alloc(&queue->pf_cache,
 			sizeof(*c->cmd_pdu) + hdgst, GFP_KERNEL | __GFP_ZERO);
@@ -2048,6 +2049,8 @@ static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
 	kref_init(&queue->kref);
 	queue->sock = newsock;
 	queue->port = port;
+	/* The port may be gone when the commands are allocated */
+	queue->nport = port->nport;
 	queue->nr_cmds = 0;
 	spin_lock_init(&queue->state_lock);
 	if (queue->port->nport->disc_addr.tsas.tcp.sectype ==
-- 
2.56.0

[PATCH 79/98] nvme-tcp: start error recovery when a send fails

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

When the target closes a connection, nvme_tcp_state_change() calls
nvme_tcp_error_recovery(), which does nothing unless the controller is
live. While connecting that's fine for the queue being set up, its
Connect fails and the attempt with it. But if an already connected queue
is closed at the end of the attempt, nothing fails, the notification is
lost and the controller goes live on a dead socket. Every request on
that queue then fails with "failed to send request -32" and is retried,
forever with native multipathing, and if it's the admin queue the first
keep-alive fails too. Nothing but a manual reset recovers it. The Linux
target closes its connections at exactly that moment after a fatal
error.

Start error recovery when a request can't be sent, as nvme-rdma does.

Fixes: 3f2304f8c6d6 ("nvme-tcp: add NVMe over TCP host driver")
Cc: Keith Busch <kbusch@kernel.org>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/host/tcp.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 4aa666dd0266..52c7f6dd34d0 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -1461,6 +1461,8 @@ static int nvme_tcp_try_send(struct nvme_tcp_queue *queue)
 			"failed to send request %d\n", ret);
 		nvme_tcp_fail_request(queue->request);
 		nvme_tcp_done_send_req(queue);
+		/* The close may have come when nobody could act on it */
+		nvme_tcp_error_recovery(&queue->ctrl->ctrl);
 	}
 out:
 	memalloc_noreclaim_restore(noreclaim_flag);
-- 
2.56.0

[PATCH 80/98] nvme-loop: don't BUG() if a scan runs beside a reset

[Jens Axboe]

From: Josef Bacik <josef@toxicpanda.com>

A reset sets queue_count to 1 while it destroys the I/O queues and
counts it back up when the new ones are made, and it doesn't stop the
scan work. A new namespace found in that window trips the check in
nvme_loop_init_hctx() for every hctx but the first:

  kernel BUG at drivers/nvme/target/loop.c:224!
   nvme_loop_init_hctx+0x7d/0x80
   __blk_mq_realloc_hw_ctxs+0x3af/0xa80
   nvme_alloc_ns+0x1a2/0x22f0

The check protects nothing. ctrl->queues has nr_io_queues + 1 entries
for the life of the controller, the tag set never has more hardware
queues than that, and nvme_loop_queue_rq() rejects requests until the
controller is live again.

Remove it.

Fixes: 3a85a5de29ea ("nvme-loop: add a NVMe loopback host driver")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/loop.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c
index 93ba6357a850..80b67608eb15 100644
--- a/drivers/nvme/target/loop.c
+++ b/drivers/nvme/target/loop.c
@@ -221,8 +221,6 @@ static int nvme_loop_init_hctx(struct blk_mq_hw_ctx *hctx, void *data,
 	struct nvme_loop_ctrl *ctrl = to_loop_ctrl(data);
 	struct nvme_loop_queue *queue = &ctrl->queues[hctx_idx + 1];
 
-	BUG_ON(hctx_idx >= ctrl->ctrl.queue_count);
-
 	/*
 	 * flush_end_io() can be called recursively for us, so use our own
 	 * lock class key for avoiding lockdep possible recursive locking,
-- 
2.56.0

[PATCH 81/98] nvmet: fix pc_ref use-after-free in preempt and abort

[Jens Axboe]

From: Chris Mason <mason@kernel.org>

With reservations enabled on a namespace, the target keeps a pc_ref per
controller, a percpu_ref counting that controller's I/O on the
namespace. "Preempt and abort" kills the pc_refs of the preempted
controllers and waits for the kill to be confirmed in
nvmet_pr_set_ctrl_to_abort(), then waits for the count to drop to zero
and resurrects them in nvmet_pr_do_abort(). Neither holds a reference
to the pc_ref, only pr_sem. nvmet_ctrl_destroy_pr() frees the pc_refs
under subsys->lock when the controller goes away, so a host that
registers a key, disconnects, and sends preempt and abort with that key
over a second connection frees a pc_ref that the abort is sleeping on:

    preempt and abort on C1            last put of C2
    nvmet_pr_set_ctrl_to_abort()
      percpu_ref_kill_and_confirm()
      wait_for_completion()
                                       nvmet_ctrl_destroy_pr()
                                         percpu_ref_exit()
                                         kfree(pc_ref)
    percpu_ref_switch_to_atomic_rcu()  /* use after free */

Put the pc_refs that an abort has killed on a per-namespace list.
nvmet_ctrl_destroy_pr() only flags a pc_ref on that list, and
nvmet_pr_do_abort() frees flagged ones once it's done with them. The
list and the flag are written under the xarray lock.

Fixes: 5a47c2080a73 ("nvmet: support reservation feature")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/nvmet.h |  5 ++
 drivers/nvme/target/pr.c    | 92 ++++++++++++++++++++++++++-----------
 2 files changed, 69 insertions(+), 28 deletions(-)

diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 5c2021b89ec4..563b966b8f75 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -88,6 +88,8 @@ struct nvmet_pr {
 	 */
 	struct semaphore	pr_sem;
 	struct list_head	registrant_list;
+	/* The pc_refs that a running "preempt and abort" has killed. */
+	struct list_head	abort_list;
 };
 
 struct nvmet_pr_per_ctrl_ref {
@@ -95,6 +97,9 @@ struct nvmet_pr_per_ctrl_ref {
 	struct completion	free_done;
 	struct completion	confirm_done;
 	uuid_t			hostid;
+	/* Both are written under the lock of ns->pr_per_ctrl_refs. */
+	struct list_head	abort_entry;
+	bool			ctrl_gone;
 };
 
 struct nvmet_ns {
diff --git a/drivers/nvme/target/pr.c b/drivers/nvme/target/pr.c
index 14eaa99f98fa..1c0d6cac1a6b 100644
--- a/drivers/nvme/target/pr.c
+++ b/drivers/nvme/target/pr.c
@@ -468,18 +468,54 @@ static void nvmet_pr_confirm_ns_pc_ref(struct percpu_ref *ref)
 	complete(&pc_ref->confirm_done);
 }
 
+static void nvmet_pr_free_pc_ref(struct nvmet_pr_per_ctrl_ref *pc_ref)
+{
+	percpu_ref_exit(&pc_ref->ref);
+	kfree(pc_ref);
+}
+
+/*
+ * The controller goes away. "Preempt and abort" sleeps on the pc_refs on its
+ * list, so leave the free of such a pc_ref to nvmet_pr_do_abort().
+ */
+static void nvmet_pr_erase_pc_ref(struct nvmet_ns *ns, unsigned long cntlid)
+{
+	struct nvmet_pr_per_ctrl_ref *pc_ref;
+
+	xa_lock(&ns->pr_per_ctrl_refs);
+	pc_ref = __xa_erase(&ns->pr_per_ctrl_refs, cntlid);
+	if (pc_ref && !list_empty(&pc_ref->abort_entry)) {
+		pc_ref->ctrl_gone = true;
+		pc_ref = NULL;
+	}
+	xa_unlock(&ns->pr_per_ctrl_refs);
+
+	if (pc_ref)
+		nvmet_pr_free_pc_ref(pc_ref);
+}
+
 static void nvmet_pr_set_ctrl_to_abort(struct nvmet_req *req, uuid_t *hostid)
 {
 	struct nvmet_pr_per_ctrl_ref *pc_ref;
 	struct nvmet_ns *ns = req->ns;
 	unsigned long idx;
 
+	xa_lock(&ns->pr_per_ctrl_refs);
 	xa_for_each(&ns->pr_per_ctrl_refs, idx, pc_ref) {
-		if (uuid_equal(&pc_ref->hostid, hostid)) {
-			percpu_ref_kill_and_confirm(&pc_ref->ref,
-						nvmet_pr_confirm_ns_pc_ref);
-			wait_for_completion(&pc_ref->confirm_done);
-		}
+		if (uuid_equal(&pc_ref->hostid, hostid) &&
+		    list_empty(&pc_ref->abort_entry))
+			list_add_tail(&pc_ref->abort_entry,
+				      &ns->pr.abort_list);
+	}
+	xa_unlock(&ns->pr_per_ctrl_refs);
+
+	list_for_each_entry(pc_ref, &ns->pr.abort_list, abort_entry) {
+		/* Killed for another host that this command preempts. */
+		if (percpu_ref_is_dying(&pc_ref->ref))
+			continue;
+		percpu_ref_kill_and_confirm(&pc_ref->ref,
+					    nvmet_pr_confirm_ns_pc_ref);
+		wait_for_completion(&pc_ref->confirm_done);
 	}
 }
 
@@ -618,20 +654,27 @@ static u16 nvmet_pr_preempt(struct nvmet_req *req,
 static void nvmet_pr_do_abort(struct work_struct *w)
 {
 	struct nvmet_req *req = container_of(w, struct nvmet_req, r.abort_work);
-	struct nvmet_pr_per_ctrl_ref *pc_ref;
+	struct nvmet_pr_per_ctrl_ref *pc_ref, *tmp;
 	struct nvmet_ns *ns = req->ns;
-	unsigned long idx;
+	bool ctrl_gone;
 
 	/*
 	 * The target does not support abort, just wait per-controller ref to 0.
 	 */
-	xa_for_each(&ns->pr_per_ctrl_refs, idx, pc_ref) {
-		if (percpu_ref_is_dying(&pc_ref->ref)) {
-			wait_for_completion(&pc_ref->free_done);
-			reinit_completion(&pc_ref->confirm_done);
-			reinit_completion(&pc_ref->free_done);
-			percpu_ref_resurrect(&pc_ref->ref);
-		}
+	list_for_each_entry_safe(pc_ref, tmp, &ns->pr.abort_list, abort_entry) {
+		wait_for_completion(&pc_ref->free_done);
+		reinit_completion(&pc_ref->confirm_done);
+		reinit_completion(&pc_ref->free_done);
+		percpu_ref_resurrect(&pc_ref->ref);
+
+		/* Off the list, the pc_ref belongs to its controller again. */
+		xa_lock(&ns->pr_per_ctrl_refs);
+		list_del_init(&pc_ref->abort_entry);
+		ctrl_gone = pc_ref->ctrl_gone;
+		xa_unlock(&ns->pr_per_ctrl_refs);
+
+		if (ctrl_gone)
+			nvmet_pr_free_pc_ref(pc_ref);
 	}
 
 	up(&ns->pr.pr_sem);
@@ -1052,6 +1095,8 @@ static int nvmet_pr_alloc_and_insert_pc_ref(struct nvmet_ns *ns,
 	init_completion(&pc_ref->free_done);
 	init_completion(&pc_ref->confirm_done);
 	uuid_copy(&pc_ref->hostid, hostid);
+	INIT_LIST_HEAD(&pc_ref->abort_entry);
+	pc_ref->ctrl_gone = false;
 
 	ret = xa_insert(&ns->pr_per_ctrl_refs, idx, pc_ref, GFP_KERNEL);
 	if (ret)
@@ -1067,7 +1112,6 @@ static int nvmet_pr_alloc_and_insert_pc_ref(struct nvmet_ns *ns,
 int nvmet_ctrl_init_pr(struct nvmet_ctrl *ctrl)
 {
 	struct nvmet_subsys *subsys = ctrl->subsys;
-	struct nvmet_pr_per_ctrl_ref *pc_ref;
 	struct nvmet_ns *ns = NULL;
 	unsigned long idx;
 	int ret;
@@ -1095,19 +1139,14 @@ int nvmet_ctrl_init_pr(struct nvmet_ctrl *ctrl)
 
 free_per_ctrl_refs:
 	nvmet_for_each_enabled_ns(&subsys->namespaces, idx, ns) {
-		if (ns->pr.enable) {
-			pc_ref = xa_erase(&ns->pr_per_ctrl_refs, ctrl->cntlid);
-			if (pc_ref)
-				percpu_ref_exit(&pc_ref->ref);
-			kfree(pc_ref);
-		}
+		if (ns->pr.enable)
+			nvmet_pr_erase_pc_ref(ns, ctrl->cntlid);
 	}
 	return ret;
 }
 
 void nvmet_ctrl_destroy_pr(struct nvmet_ctrl *ctrl)
 {
-	struct nvmet_pr_per_ctrl_ref *pc_ref;
 	struct nvmet_ns *ns;
 	unsigned long idx;
 
@@ -1115,12 +1154,8 @@ void nvmet_ctrl_destroy_pr(struct nvmet_ctrl *ctrl)
 	mutex_destroy(&ctrl->pr_log_mgr.lock);
 
 	nvmet_for_each_enabled_ns(&ctrl->subsys->namespaces, idx, ns) {
-		if (ns->pr.enable) {
-			pc_ref = xa_erase(&ns->pr_per_ctrl_refs, ctrl->cntlid);
-			if (pc_ref)
-				percpu_ref_exit(&pc_ref->ref);
-			kfree(pc_ref);
-		}
+		if (ns->pr.enable)
+			nvmet_pr_erase_pc_ref(ns, ctrl->cntlid);
 	}
 }
 
@@ -1136,6 +1171,7 @@ int nvmet_pr_init_ns(struct nvmet_ns *ns)
 	atomic_set(&ns->pr.generation, 0);
 	sema_init(&ns->pr.pr_sem, 1);
 	INIT_LIST_HEAD(&ns->pr.registrant_list);
+	INIT_LIST_HEAD(&ns->pr.abort_list);
 	ns->pr.notify_mask = 0;
 
 	xa_init(&ns->pr_per_ctrl_refs);
-- 
2.56.0

[PATCH 82/98] nvmet: move preempt and abort work off nvmet-wq

[Jens Axboe]

From: Chris Mason <mason@kernel.org>

nvmet_pr_do_abort() runs on nvmet-wq, holds pr_sem while it waits for
the preempted controllers' I/O, and releases it at the end. nvmet-wq
also runs nvmet_tcp_release_queue_work(), which cancel_work_sync()s the
io_work of a closed connection, and that io_work may be sleeping on
pr_sem in a reservation command. So a host that keeps a write in flight
on a preempted controller, sends a second preempt and abort, and then
closes enough connections with a reservation command in flight fills
nvmet-wq with release works that wait for pr_sem. When the first abort
finishes, the second one is queued on a full workqueue behind them and
never starts, pr_sem is never released, and the accept work doesn't run
either, so nobody can connect.

Queue the abort work on its own workqueue so it starts regardless. The
release works can still fill nvmet-wq while pr_sem is held, and a
file-backed namespace's flush, DSM and write zeroes need nvmet-wq
themselves, so this is a mitigation.

Fixes: 5a47c2080a73 ("nvmet: support reservation feature")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/core.c  | 11 ++++++++++-
 drivers/nvme/target/nvmet.h |  1 +
 drivers/nvme/target/pr.c    |  6 +++++-
 3 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 142df0db5208..f96b21257169 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -29,6 +29,7 @@ struct workqueue_struct *nvmet_wq;
 EXPORT_SYMBOL_GPL(nvmet_wq);
 struct workqueue_struct *nvmet_aen_wq;
 EXPORT_SYMBOL_GPL(nvmet_aen_wq);
+struct workqueue_struct *nvmet_pr_wq;
 
 /*
  * This read/write semaphore is used to synchronize access to configuration
@@ -2034,9 +2035,14 @@ static int __init nvmet_init(void)
 	if (!nvmet_aen_wq)
 		goto out_free_nvmet_work_queue;
 
+	nvmet_pr_wq = alloc_workqueue("nvmet-pr-wq",
+				      WQ_MEM_RECLAIM | WQ_UNBOUND, 0);
+	if (!nvmet_pr_wq)
+		goto out_free_nvmet_aen_work_queue;
+
 	error = nvmet_init_debugfs();
 	if (error)
-		goto out_free_nvmet_aen_work_queue;
+		goto out_free_nvmet_pr_work_queue;
 
 	error = nvmet_init_discovery();
 	if (error)
@@ -2052,6 +2058,8 @@ static int __init nvmet_init(void)
 	nvmet_exit_discovery();
 out_exit_debugfs:
 	nvmet_exit_debugfs();
+out_free_nvmet_pr_work_queue:
+	destroy_workqueue(nvmet_pr_wq);
 out_free_nvmet_aen_work_queue:
 	destroy_workqueue(nvmet_aen_wq);
 out_free_nvmet_work_queue:
@@ -2071,6 +2079,7 @@ static void __exit nvmet_exit(void)
 	nvmet_exit_discovery();
 	nvmet_exit_debugfs();
 	ida_destroy(&cntlid_ida);
+	destroy_workqueue(nvmet_pr_wq);
 	destroy_workqueue(nvmet_aen_wq);
 	destroy_workqueue(nvmet_wq);
 	destroy_workqueue(buffered_io_wq);
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 563b966b8f75..9e8fbb2d1fc1 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -532,6 +532,7 @@ extern struct workqueue_struct *buffered_io_wq;
 extern struct workqueue_struct *zbd_wq;
 extern struct workqueue_struct *nvmet_wq;
 extern struct workqueue_struct *nvmet_aen_wq;
+extern struct workqueue_struct *nvmet_pr_wq;
 
 static inline void nvmet_set_result(struct nvmet_req *req, u32 result)
 {
diff --git a/drivers/nvme/target/pr.c b/drivers/nvme/target/pr.c
index 1c0d6cac1a6b..1a03f74c2e14 100644
--- a/drivers/nvme/target/pr.c
+++ b/drivers/nvme/target/pr.c
@@ -756,7 +756,11 @@ static void nvmet_execute_pr_acquire(struct nvmet_req *req)
 	if (!status && acquire_act == NVME_PR_ACQUIRE_ACT_PREEMPT_AND_ABORT) {
 		kfree(d);
 		INIT_WORK(&req->r.abort_work, nvmet_pr_do_abort);
-		queue_work(nvmet_wq, &req->r.abort_work);
+		/*
+		 * Not nvmet_wq: works on it can wait for pr_sem, which this
+		 * work releases.
+		 */
+		queue_work(nvmet_pr_wq, &req->r.abort_work);
 		return;
 	}
 
-- 
2.56.0

[PATCH 83/98] nvmet-rdma: treat a failed connect as a disconnect

[Jens Axboe]

From: Chris Mason <mason@kernel.org>

Once a connect request has been accepted, a queue is only freed by its
release_work, so that must be queued once. The disconnect paths take
the queue off nvmet_rdma_queue_list first and queue the work only if it
was still on it. nvmet_rdma_queue_connect_fail(), for REJECTED,
UNREACHABLE and CONNECT_ERROR, queues it unconditionally and only warns
if the queue isn't connecting. But the IB CM delivers a REJ from the
peer while the connection is established and even after we've sent a
DREQ, so a peer that causes a disconnect with a short capsule and then
sends a REJ while the release work is running queues it a second time
and the work runs on a freed queue. No credentials needed, just RoCE or
IB reachability, reproduced over rxe with a UDP socket.

Route connect failures through nvmet_rdma_queue_disconnect(), so only
the path that takes the queue off the list queues the work. A REJ on an
established connection still frees the queue, the RDMA CM reports no
disconnect after one.

Fixes: 8f000cac6e7a ("nvmet-rdma: add a NVMe over Fabrics RDMA target driver")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/rdma.c | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index c4b848f84465..13243e3edca5 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -1738,15 +1738,14 @@ static void nvmet_rdma_queue_disconnect(struct nvmet_rdma_queue *queue)
 static void nvmet_rdma_queue_connect_fail(struct rdma_cm_id *cm_id,
 		struct nvmet_rdma_queue *queue)
 {
-	WARN_ON_ONCE(queue->state != NVMET_RDMA_Q_CONNECTING);
-
-	mutex_lock(&nvmet_rdma_queue_mutex);
-	if (!list_empty(&queue->queue_list))
-		list_del_init(&queue->queue_list);
-	mutex_unlock(&nvmet_rdma_queue_mutex);
-
 	pr_err("failed to connect queue %d\n", queue->idx);
-	queue_work(nvmet_wq, &queue->release_work);
+
+	/*
+	 * The peer decides when these events arrive: a REJ is also delivered
+	 * for a connection that is established, or that we are disconnecting.
+	 * Whoever takes the queue off the list queues release_work, once.
+	 */
+	nvmet_rdma_queue_disconnect(queue);
 }
 
 /**
-- 
2.56.0

[PATCH 84/98] nvmet-rdma: disconnect only established queues

[Jens Axboe]

From: Chris Mason <mason@kernel.org>

Over iWARP, rdma_disconnect() sleeps in iw_cm_disconnect() until the
ESTABLISHED event of the connection has been handled, and iw_cm handles
all events of all connections one at a time on iw_cm_wq.
nvmet_rdma_destroy_port_queues(), nvmet_rdma_remove_one() and
nvmet_rdma_delete_ctrl() walk the queue list with
nvmet_rdma_queue_mutex held and call rdma_disconnect() for each, while
nvmet_rdma_cm_handler() takes the same mutex. If the walk sleeps on a
queue whose ESTABLISHED is queued behind another queue's event, both
are stuck and iWARP connection handling is dead until reboot. A peer
sets that up by connecting in a loop, or over siw by asking for MPA
peer-to-peer mode and then staying silent, which delays ESTABLISHED
indefinitely.

Only call rdma_disconnect() for a queue in NVMET_RDMA_Q_LIVE, which the
handler sets when it gets ESTABLISHED. A queue that never got there is
left to the release work: ib_drain_qp() moves the QP to the error
state, which ends the connection, and rdma_destroy_id() tells the peer.
Over IB and RoCE the skipped call did nothing more than that.

Fixes: f1d4ef7d8883 ("nvmet-rdma: register ib_client to not deadlock in device removal")
Fixes: fcf73a804c7d ("nvmet-rdma: fix use-after-free when a port is removed")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/rdma.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index 13243e3edca5..c5970bd2dc29 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -1687,7 +1687,7 @@ static void nvmet_rdma_queue_established(struct nvmet_rdma_queue *queue)
 
 static void __nvmet_rdma_queue_disconnect(struct nvmet_rdma_queue *queue)
 {
-	bool disconnect = false;
+	bool disconnect = false, live = false;
 	unsigned long flags;
 
 	pr_debug("cm_id= %p queue->state= %d\n", queue->cm_id, queue->state);
@@ -1706,6 +1706,7 @@ static void __nvmet_rdma_queue_disconnect(struct nvmet_rdma_queue *queue)
 		}
 		fallthrough;
 	case NVMET_RDMA_Q_LIVE:
+		live = queue->state == NVMET_RDMA_Q_LIVE;
 		queue->state = NVMET_RDMA_Q_DISCONNECTING;
 		disconnect = true;
 		break;
@@ -1715,7 +1716,17 @@ static void __nvmet_rdma_queue_disconnect(struct nvmet_rdma_queue *queue)
 	spin_unlock_irqrestore(&queue->state_lock, flags);
 
 	if (disconnect) {
-		rdma_disconnect(queue->cm_id);
+		/*
+		 * Over iWARP rdma_disconnect() sleeps until the CM has handled
+		 * the ESTABLISHED event of this connection. The CM handles the
+		 * events of all connections one at a time, and our handler
+		 * takes nvmet_rdma_queue_mutex, which callers of this function
+		 * hold. So a queue that has not seen the event is left to the
+		 * release work: ib_drain_qp() moves the queue pair to the error
+		 * state, which ends the connection.
+		 */
+		if (live)
+			rdma_disconnect(queue->cm_id);
 		queue_work(nvmet_wq, &queue->release_work);
 	}
 }
-- 
2.56.0

[PATCH 85/98] nvmet-rdma: take a device off the list last

[Jens Axboe]

From: Chris Mason <mason@kernel.org>

nvmet_rdma_remove_one() disconnects the queues of a device and flushes
nvmet-wq, but since commit a3dd7d0022c3 it returns right away if the
device isn't on device_list. nvmet_rdma_free_dev(), run by the release
work of the last queue, takes the device off that list first and frees
the SRQs and the PD afterwards. A device removal in between finds
nothing on the list, returns, and the RDMA core frees the device under
ib_dealloc_pd().

Free the SRQs and the PD first and unlist the device last, so the
remove callback finds it and its flush waits for the release work.

Fixes: a3dd7d0022c3 ("nvmet-rdma: Don't flush system_wq by default during remove_one")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/rdma.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index c5970bd2dc29..a85e6fa71e0b 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -1190,13 +1190,17 @@ static void nvmet_rdma_free_dev(struct kref *ref)
 	struct nvmet_rdma_device *ndev =
 		container_of(ref, struct nvmet_rdma_device, ref);
 
+	nvmet_rdma_destroy_srqs(ndev);
+	ib_dealloc_pd(ndev->pd);
+
+	/*
+	 * Not earlier: nvmet_rdma_remove_one() waits for the release work that
+	 * got here only if it finds the device on the list.
+	 */
 	mutex_lock(&device_list_mutex);
 	list_del(&ndev->entry);
 	mutex_unlock(&device_list_mutex);
 
-	nvmet_rdma_destroy_srqs(ndev);
-	ib_dealloc_pd(ndev->pd);
-
 	kfree(ndev);
 }
 
-- 
2.56.0

[PATCH 86/98] nvmet-rdma: queue the release work under the mutex

[Jens Axboe]

From: Chris Mason <mason@kernel.org>

nvmet_rdma_remove_one() takes the queue mutex, unlists each queue of
the device and queues its release work, then flushes nvmet-wq. So a
queue that still holds objects on the device must either be on the
list or have its release work queued. nvmet_rdma_queue_disconnect()
breaks that: it unlists the queue under the mutex, drops it, and only
then calls __nvmet_rdma_queue_disconnect() to queue the work. A device
removal in that window finds nothing to wait for, and the release work
then destroys the QP and CQ share on a freed device.

Queue the work before dropping the mutex, as nvmet_rdma_remove_one()
itself does. That's safe now that rdma_disconnect() no longer sleeps on
an iWARP event that needs the same mutex.

Fixes: f1d4ef7d8883 ("nvmet-rdma: register ib_client to not deadlock in device removal")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/rdma.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index a85e6fa71e0b..1f71ef2961ba 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -1737,17 +1737,17 @@ static void __nvmet_rdma_queue_disconnect(struct nvmet_rdma_queue *queue)
 
 static void nvmet_rdma_queue_disconnect(struct nvmet_rdma_queue *queue)
 {
-	bool disconnect = false;
-
+	/*
+	 * The release work is queued with the mutex held, so that
+	 * nvmet_rdma_remove_one() finds the queue on the list or its work on
+	 * the workqueue.
+	 */
 	mutex_lock(&nvmet_rdma_queue_mutex);
 	if (!list_empty(&queue->queue_list)) {
 		list_del_init(&queue->queue_list);
-		disconnect = true;
+		__nvmet_rdma_queue_disconnect(queue);
 	}
 	mutex_unlock(&nvmet_rdma_queue_mutex);
-
-	if (disconnect)
-		__nvmet_rdma_queue_disconnect(queue);
 }
 
 static void nvmet_rdma_queue_connect_fail(struct rdma_cm_id *cm_id,
-- 
2.56.0

[PATCH 87/98] nvmet-rdma: disconnect a queue on device removal

[Jens Axboe]

From: Chris Mason <mason@kernel.org>

cma_remove_one() sends DEVICE_REMOVAL to the owner of every cm_id on
the device and then waits, without timeout, until they're all
destroyed. nvmet_rdma_device_removal() does nothing for a queue cm_id
and leaves the queues to nvmet_rdma_remove_one(), which only works if
that runs first. As a module it does, nvmet-rdma registers after
rdma_cm. Built in, both register at the same initcall level and
drivers/Makefile lists nvme/ ahead of infiniband/, so cma_remove_one()
runs first and waits for cm_ids that only the later callback would
destroy. Since the RDMA CM drops all later events for a cm_id once it
has sent DEVICE_REMOVAL, a peer closing its connection doesn't help
either. One connection from anyone who can reach the port hangs "rdma
link delete" until something else ends the queue.

Call nvmet_rdma_queue_disconnect() for the queue. It unlists the queue
and queues the release work, which destroys the cm_id. Don't wait for
it, rdma_destroy_id() waits for the handler to return, which was the
deadlock before commit f1d4ef7d8883.

Fixes: f1d4ef7d8883 ("nvmet-rdma: register ib_client to not deadlock in device removal")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/rdma.c | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index 1f71ef2961ba..baaec9a36d8d 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -1773,10 +1773,13 @@ static void nvmet_rdma_queue_connect_fail(struct rdma_cm_id *cm_id,
  * queue cm_id and/or a device bound listener cm_id (where in this
  * case queue will be null).
  *
- * We registered an ib_client to handle device removal for queues,
- * so we only need to handle the listening port cm_ids. In this case
- * we nullify the priv to prevent double cm_id destruction and destroying
- * the cm_id implicitly by returning a non-zero rc to the callout.
+ * The RDMA CM waits until each cm_id of the device is destroyed, so a queue
+ * has to go now. Our ib_client also removes the queues, but it cannot be
+ * relied on here: built in, it is called after the RDMA CM's client.
+ *
+ * For a listening port cm_id we nullify the priv to prevent double cm_id
+ * destruction and destroying the cm_id implicitly by returning a non-zero
+ * rc to the callout.
  */
 static int nvmet_rdma_device_removal(struct rdma_cm_id *cm_id,
 		struct nvmet_rdma_queue *queue)
@@ -1784,11 +1787,7 @@ static int nvmet_rdma_device_removal(struct rdma_cm_id *cm_id,
 	struct nvmet_rdma_port *port;
 
 	if (queue) {
-		/*
-		 * This is a queue cm_id. we have registered
-		 * an ib_client to handle queues removal
-		 * so don't interfere and just return.
-		 */
+		nvmet_rdma_queue_disconnect(queue);
 		return 0;
 	}
 
-- 
2.56.0

[PATCH 88/98] nvmet-rdma: flush nvmet-wq on module exit

[Jens Axboe]

From: Chris Mason <mason@kernel.org>

nvmet_rdma_exit() relies on ib_unregister_client() to flush nvmet-wq
through nvmet_rdma_remove_one(), but since commit a3dd7d0022c3 that
returns at once for a device no longer on device_list. The release work
of the last queue takes the device off the list in nvmet_rdma_free_dev()
and still has to drop the mutex, free the struct and return through
nvmet_rdma_release_queue_work(), so an rmmod in that window frees the
module code it's running.

Flush nvmet-wq after ib_unregister_client(). No release work can be
queued after that point.

Fixes: a3dd7d0022c3 ("nvmet-rdma: Don't flush system_wq by default during remove_one")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/rdma.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index baaec9a36d8d..b961d1c4aaa4 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -2170,6 +2170,12 @@ static void __exit nvmet_rdma_exit(void)
 {
 	nvmet_unregister_transport(&nvmet_rdma_ops);
 	ib_unregister_client(&nvmet_rdma_ib_client);
+	/*
+	 * nvmet_rdma_remove_one() does not flush for a device that has left
+	 * device_list, and the release work that took it off still has to
+	 * return from nvmet_rdma_free_dev().
+	 */
+	flush_workqueue(nvmet_wq);
 	WARN_ON_ONCE(!list_empty(&nvmet_rdma_queue_list));
 	ida_destroy(&nvmet_rdma_queue_ida);
 }
-- 
2.56.0

[PATCH 89/98] nvmet-rdma: reject queues larger than max_cqe

[Jens Axboe]

From: Chris Mason <mason@kernel.org>

The host picks hrqsize and hsqsize in its RDMA CM connect request, and
nvmet_rdma_create_queue_ib() asks ib_cq_pool_get() for
recv + 2 * send + 1 CQ entries, up to 196607 for an I/O queue. The
pool limits each CQ it allocates to the device's max_cqe, so if the
request exceeds that no CQ ever fits and ib_alloc_cqs() keeps adding
CQs until an allocation fails. Those CQs stay until the device is
removed. rxe has a max_cqe of 32767, and one connect request from an
unprivileged process OOMs a 1 GiB target in under a second, with
nothing to kill that frees the memory.

Refuse a queue that needs more than max_cqe entries before touching the
pool. The connect is rejected with NVME_RDMA_CM_NO_RSC like any other
queue creation failure.

Fixes: ca0f1a8055be ("nvmet-rdma: use new shared CQ mechanism")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/rdma.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index b961d1c4aaa4..26ea4e1d62b3 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -1286,8 +1286,17 @@ static int nvmet_rdma_create_queue_ib(struct nvmet_rdma_queue *queue)
 	 */
 	nr_cqe = queue->recv_queue_size + 2 * queue->send_queue_size;
 
-	queue->cq = ib_cq_pool_get(ndev->device, nr_cqe + 1,
-				   queue->comp_vector, IB_POLL_WORKQUEUE);
+	/*
+	 * The host picks both sizes. Never ask ib_cq_pool_get() for more
+	 * entries than one CQ of the device can have: no CQ would ever fit,
+	 * and it would allocate new CQs until an allocation fails.
+	 */
+	if (nr_cqe + 1 > ndev->device->attrs.max_cqe)
+		queue->cq = ERR_PTR(-EINVAL);
+	else
+		queue->cq = ib_cq_pool_get(ndev->device, nr_cqe + 1,
+					   queue->comp_vector,
+					   IB_POLL_WORKQUEUE);
 	if (IS_ERR(queue->cq)) {
 		ret = PTR_ERR(queue->cq);
 		pr_err("failed to create CQ cqe= %d ret= %d\n",
-- 
2.56.0

[PATCH 90/98] nvmet-rdma: disable interrupts under rsp_wr_wait_lock

[Jens Axboe]

From: Chris Mason <mason@kernel.org>

rsp_wr_wait_lock protects the list of commands waiting for room in the
send queue, and it's taken with plain spin_lock(). But
nvmet_rdma_queue_response() calls nvmet_rdma_release_rsp() directly if
ib_post_send() fails, and that processes the wait list. The call chain
starts in nvmet_req_complete(), which for a block device namespace runs
from the bio end_io callback, which can be in hard interrupt context.
An interrupt that completes a command on a CPU holding the lock for the
same queue spins forever. Lockdep over siw with a null_blk namespace:

  inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage

Use spin_lock_irqsave() for both takers. The processing loop still
drops the lock, and restores the caller's interrupt state, around
nvmet_rdma_execute_command().

Fixes: 8f000cac6e7a ("nvmet-rdma: add a NVMe over Fabrics RDMA target driver")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/rdma.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index 26ea4e1d62b3..7c97a47e305a 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -513,7 +513,9 @@ static int nvmet_rdma_post_recv(struct nvmet_rdma_device *ndev,
 
 static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue)
 {
-	spin_lock(&queue->rsp_wr_wait_lock);
+	unsigned long flags;
+
+	spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
 	while (!list_empty(&queue->rsp_wr_wait_list)) {
 		struct nvmet_rdma_rsp *rsp;
 		bool ret;
@@ -522,16 +524,16 @@ static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue)
 				struct nvmet_rdma_rsp, wait_list);
 		list_del(&rsp->wait_list);
 
-		spin_unlock(&queue->rsp_wr_wait_lock);
+		spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
 		ret = nvmet_rdma_execute_command(rsp);
-		spin_lock(&queue->rsp_wr_wait_lock);
+		spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
 
 		if (!ret) {
 			list_add(&rsp->wait_list, &queue->rsp_wr_wait_list);
 			break;
 		}
 	}
-	spin_unlock(&queue->rsp_wr_wait_lock);
+	spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
 }
 
 static u16 nvmet_rdma_check_pi_status(struct ib_mr *sig_mr)
@@ -976,6 +978,7 @@ static bool nvmet_rdma_execute_command(struct nvmet_rdma_rsp *rsp)
 static void nvmet_rdma_handle_command(struct nvmet_rdma_queue *queue,
 		struct nvmet_rdma_rsp *cmd)
 {
+	unsigned long flags;
 	u16 status;
 
 	ib_dma_sync_single_for_cpu(queue->dev->device,
@@ -993,9 +996,9 @@ static void nvmet_rdma_handle_command(struct nvmet_rdma_queue *queue,
 		goto out_err;
 
 	if (unlikely(!nvmet_rdma_execute_command(cmd))) {
-		spin_lock(&queue->rsp_wr_wait_lock);
+		spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
 		list_add_tail(&cmd->wait_list, &queue->rsp_wr_wait_list);
-		spin_unlock(&queue->rsp_wr_wait_lock);
+		spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
 	}
 
 	return;
-- 
2.56.0

Re: [PATCH 90/98] nvmet-rdma: disable interrupts under rsp_wr_wait_lock

[Bart Van Assche]

On 10/28/26 09:42, Jens Axboe wrote:
> -		spin_unlock(&queue->rsp_wr_wait_lock);
> +		spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
>  		ret = nvmet_rdma_execute_command(rsp);
> -		spin_lock(&queue->rsp_wr_wait_lock);
> +		spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);

Using spin_lock_irqsave() fixes the rsp_wr_wait_lock deadlock itself,
but when nvmet_rdma_process_wr_wait_list() is reached from a bio end_io
callback in hardirq or softirq context (via nvmet_req_complete() ->
nvmet_rdma_queue_response() -> failed ib_post_send() ->
nvmet_rdma_release_rsp()), spin_unlock_irqrestore() restores that
hardirq/softirq context and calls nvmet_rdma_execute_command(rsp) ->
rsp->req.execute(&rsp->req).

The nvmet ->execute() handlers assume process context:
- nvmet_bdev_execute_rw() allocates bios with GFP_KERNEL and calls
  submit_bio() -> submit_bio_noacct(), which hits might_sleep() and
  manipulates current->bio_list without interrupt synchronization.
- Admin, fabrics, auth, and PR handlers perform GFP_KERNEL allocations
  and take mutexes / rwsems.

Shouldn't rsp_wr_wait_list processing be deferred to a work_struct when
!in_task() (or whenever nvmet_rdma_release_rsp() is called from atomic
context)?

Re: [PATCH 90/98] nvmet-rdma: disable interrupts under rsp_wr_wait_lock

[Bart Van Assche]

On 10/28/26 09:42, Jens Axboe wrote:
> -		spin_unlock(&queue->rsp_wr_wait_lock);
> +		spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
>  		ret = nvmet_rdma_execute_command(rsp);
> -		spin_lock(&queue->rsp_wr_wait_lock);
> +		spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);

Using spin_lock_irqsave() fixes the rsp_wr_wait_lock deadlock itself,
but when nvmet_rdma_process_wr_wait_list() is reached from a bio end_io
callback in hardirq or softirq context (via nvmet_req_complete() ->
nvmet_rdma_queue_response() -> failed ib_post_send() ->
nvmet_rdma_release_rsp()), spin_unlock_irqrestore() restores that
hardirq/softirq context and calls nvmet_rdma_execute_command(rsp) ->
rsp->req.execute(&rsp->req).

The nvmet ->execute() handlers require process context:
- nvmet_bdev_execute_rw() allocates bios with GFP_KERNEL and calls
  submit_bio() -> submit_bio_noacct(), which hits might_sleep() and
  manipulates current->bio_list without interrupt synchronization.
- Admin, fabrics, auth, and PR handlers perform GFP_KERNEL allocations
  and take mutexes / rwsems.

If nvmet_rdma_process_wr_wait_list() checks !in_task() under
rsp_wr_wait_lock and schedules a per-queue wr_wait_work on system_dfl_wq
when rsp_wr_wait_list is non-empty (cancelled after ib_drain_qp() in
nvmet_rdma_destroy_queue_ib()), normal task/CQ-workqueue callers pay no
extra cost while hardirq/softirq completions safely defer command
execution to process context.

Note: patches 91/98 and 92/98 compose cleanly on top by placing the
if (done && done->wait_released) check right above if (!in_task()) and
passing NULL from nvmet_rdma_wr_wait_work().

Here is a replacement patch (checked with checkpatch.pl and
syntax-checked with Clang; not runtime-tested):

---
diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index 26ea4e1d62b3..60ea488db8f4 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -104,6 +104,7 @@ struct nvmet_rdma_queue {
 	struct nvmet_rdma_cmd	*cmds;
 
 	struct work_struct	release_work;
+	struct work_struct	wr_wait_work;
 	struct list_head	rsp_wait_list;
 	struct list_head	rsp_wr_wait_list;
 	spinlock_t		rsp_wr_wait_lock;
@@ -513,7 +514,15 @@ static int nvmet_rdma_post_recv(struct nvmet_rdma_device *ndev,
 
 static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue)
 {
-	spin_lock(&queue->rsp_wr_wait_lock);
+	unsigned long flags;
+
+	spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
+	if (!in_task()) {
+		if (!list_empty(&queue->rsp_wr_wait_list))
+			schedule_work(&queue->wr_wait_work);
+		spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
+		return;
+	}
 	while (!list_empty(&queue->rsp_wr_wait_list)) {
 		struct nvmet_rdma_rsp *rsp;
 		bool ret;
@@ -522,16 +531,24 @@ static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue)
 				struct nvmet_rdma_rsp, wait_list);
 		list_del(&rsp->wait_list);
 
-		spin_unlock(&queue->rsp_wr_wait_lock);
+		spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
 		ret = nvmet_rdma_execute_command(rsp);
-		spin_lock(&queue->rsp_wr_wait_lock);
+		spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
 
 		if (!ret) {
 			list_add(&rsp->wait_list, &queue->rsp_wr_wait_list);
 			break;
 		}
 	}
-	spin_unlock(&queue->rsp_wr_wait_lock);
+	spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
+}
+
+static void nvmet_rdma_wr_wait_work(struct work_struct *w)
+{
+	struct nvmet_rdma_queue *queue =
+		container_of(w, struct nvmet_rdma_queue, wr_wait_work);
+
+	nvmet_rdma_process_wr_wait_list(queue);
 }
 
 static u16 nvmet_rdma_check_pi_status(struct ib_mr *sig_mr)
@@ -976,6 +993,7 @@ static bool nvmet_rdma_execute_command(struct nvmet_rdma_rsp *rsp)
 static void nvmet_rdma_handle_command(struct nvmet_rdma_queue *queue,
 		struct nvmet_rdma_rsp *cmd)
 {
+	unsigned long flags;
 	u16 status;
 
 	ib_dma_sync_single_for_cpu(queue->dev->device,
@@ -993,9 +1011,9 @@ static void nvmet_rdma_handle_command(struct nvmet_rdma_queue *queue,
 		goto out_err;
 
 	if (unlikely(!nvmet_rdma_execute_command(cmd))) {
-		spin_lock(&queue->rsp_wr_wait_lock);
+		spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
 		list_add_tail(&cmd->wait_list, &queue->rsp_wr_wait_list);
-		spin_unlock(&queue->rsp_wr_wait_lock);
+		spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
 	}
 
 	return;
@@ -1374,6 +1392,7 @@ static bool nvmet_rdma_reclaim_rsp(struct sbitmap *sb, unsigned int bitnr,
 static void nvmet_rdma_destroy_queue_ib(struct nvmet_rdma_queue *queue)
 {
 	ib_drain_qp(queue->qp);
+	cancel_work_sync(&queue->wr_wait_work);
 
 	/*
 	 * Reclaim resources of a response that is still in-flight when the
@@ -1494,6 +1513,7 @@ nvmet_rdma_alloc_queue(struct nvmet_rdma_device *ndev,
 	 * inside a CM callback would trigger a deadlock. (great API design..)
 	 */
 	INIT_WORK(&queue->release_work, nvmet_rdma_release_queue_work);
+	INIT_WORK(&queue->wr_wait_work, nvmet_rdma_wr_wait_work);
 	queue->dev = ndev;
 	queue->cm_id = cm_id;
 	queue->port = port->nport;

[PATCH 91/98] nvmet-rdma: don't run the wait list recursively

[Jens Axboe]

From: Chris Mason <mason@kernel.org>

nvmet_rdma_release_rsp() returns the send queue slots of a response and
then runs the wait list, passing each waiting command to
nvmet_rdma_execute_command(). Many commands complete inside their
execute callback, and if the response can't be posted,
nvmet_rdma_queue_response() calls nvmet_rdma_release_rsp() right there,
which starts another run of the wait list inside the first:

    nvmet_rdma_process_wr_wait_list()
      nvmet_rdma_execute_command()
        nvmet_execute_admin_connect()
          nvmet_req_complete()
            nvmet_rdma_queue_response()
              ib_post_send() fails
              nvmet_rdma_release_rsp()
                nvmet_rdma_process_wr_wait_list()

Each waiting command adds a level while posts keep failing, and the host
decides how many commands wait: it sizes the receive queue, can fill
the send queue with commands that nvmet_req_init() rejects without
taking a slot, and over siw can make posts fail by holding back an
RDMA READ. 400 waiting Connects overflow the stack of the CQ worker.

Before the loop executes a command, point the response's new
wait_released at a flag on the loop's stack. A release that finds it
set means the loop hasn't returned from that command yet: it returns
the slots and releases the response, but sets the flag and leaves the
list to the loop instead of running it. The flag tells the loop not to
touch the response again.

Fixes: 8f000cac6e7a ("nvmet-rdma: add a NVMe over Fabrics RDMA target driver")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/rdma.c | 28 +++++++++++++++++++++++++---
 1 file changed, 25 insertions(+), 3 deletions(-)

diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index 7c97a47e305a..b8098ec53d7b 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -77,6 +77,8 @@ struct nvmet_rdma_rsp {
 	u32			invalidate_rkey;
 
 	struct list_head	wait_list;
+	/* set under rsp_wr_wait_lock while the wait list's loop executes us */
+	bool			*wait_released;
 	int			tag;
 };
 
@@ -511,23 +513,42 @@ static int nvmet_rdma_post_recv(struct nvmet_rdma_device *ndev,
 	return ret;
 }
 
-static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue)
+static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue,
+					    struct nvmet_rdma_rsp *done)
 {
 	unsigned long flags;
 
 	spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
+	if (done->wait_released) {
+		/*
+		 * The loop below has not come back from this command yet. It is
+		 * further up this stack if the command completed in place and
+		 * its response could not be posted. Otherwise it runs in
+		 * another context. It goes on with the list, so leave the list
+		 * to it.
+		 */
+		*done->wait_released = true;
+		WRITE_ONCE(done->wait_released, NULL);
+		spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
+		return;
+	}
 	while (!list_empty(&queue->rsp_wr_wait_list)) {
 		struct nvmet_rdma_rsp *rsp;
+		bool released = false;
 		bool ret;
 
 		rsp = list_entry(queue->rsp_wr_wait_list.next,
 				struct nvmet_rdma_rsp, wait_list);
 		list_del(&rsp->wait_list);
+		rsp->wait_released = &released;
 
 		spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
 		ret = nvmet_rdma_execute_command(rsp);
 		spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
 
+		/* A released response can be freed, or in use again. */
+		if (!released)
+			WRITE_ONCE(rsp->wait_released, NULL);
 		if (!ret) {
 			list_add(&rsp->wait_list, &queue->rsp_wr_wait_list);
 			break;
@@ -680,8 +701,9 @@ static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp)
 
 	nvmet_rdma_free_rsp_resources(rsp);
 
-	if (unlikely(!list_empty_careful(&queue->rsp_wr_wait_list)))
-		nvmet_rdma_process_wr_wait_list(queue);
+	if (unlikely(READ_ONCE(rsp->wait_released) ||
+		     !list_empty_careful(&queue->rsp_wr_wait_list)))
+		nvmet_rdma_process_wr_wait_list(queue, rsp);
 
 	nvmet_rdma_put_rsp(rsp);
 }
-- 
2.56.0

[PATCH 92/98] nvmet-rdma: fix a lost run of the wait list

[Jens Axboe]

From: Chris Mason <mason@kernel.org>

A command is checked for free send queue slots before it's on the wait
list, in nvmet_rdma_handle_command() and when the processing loop takes
it off to retry, and nvmet_rdma_release_rsp() looks at the list without
a lock after giving slots back. The CQ worker and the CM thread (via
nvmet_rdma_queue_established()) can handle the commands of one queue
concurrently, so a release can see an empty list while the other side
is between the failed slot check and the list add:

    handle a command                  release a response
    too few free slots
                                      give the slots back
                                      wait list empty: return
    add the command to the list       /* nobody runs the list */

If that was the last response in flight the command stays on the list
forever. It holds sq->ref, so nvmet_sq_destroy() hangs at disconnect
and the queue's memory is never freed. A host hits it by sending more
Connects at once than the send queue has slots.

Look at the slots again after a command is on the list: the loop retries
after a put-back with a full barrier in between, nvmet_rdma_release_rsp()
has a matching barrier between returning the slots and looking at the
list, and nvmet_rdma_handle_command() runs the list once after adding
its command.

Fixes: 8f000cac6e7a ("nvmet-rdma: add a NVMe over Fabrics RDMA target driver")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/rdma.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index b8098ec53d7b..7cbf1caf3208 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -519,7 +519,7 @@ static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue,
 	unsigned long flags;
 
 	spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
-	if (done->wait_released) {
+	if (done && done->wait_released) {
 		/*
 		 * The loop below has not come back from this command yet. It is
 		 * further up this stack if the command completed in place and
@@ -551,6 +551,15 @@ static void nvmet_rdma_process_wr_wait_list(struct nvmet_rdma_queue *queue,
 			WRITE_ONCE(rsp->wait_released, NULL);
 		if (!ret) {
 			list_add(&rsp->wait_list, &queue->rsp_wr_wait_list);
+			/*
+			 * A release that came while the command was off the
+			 * list can have seen an empty list, so look at the
+			 * room again.
+			 * Pairs with the barrier in nvmet_rdma_release_rsp().
+			 */
+			smp_mb();
+			if (atomic_read(&queue->sq_wr_avail) > rsp->n_rdma)
+				continue;
 			break;
 		}
 	}
@@ -698,6 +707,11 @@ static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp)
 	struct nvmet_rdma_queue *queue = rsp->queue;
 
 	atomic_add(1 + rsp->n_rdma, &queue->sq_wr_avail);
+	/*
+	 * Give the room back ahead of the look at the wait list below.
+	 * Pairs with the smp_mb() in nvmet_rdma_process_wr_wait_list().
+	 */
+	smp_mb__after_atomic();
 
 	nvmet_rdma_free_rsp_resources(rsp);
 
@@ -1021,6 +1035,8 @@ static void nvmet_rdma_handle_command(struct nvmet_rdma_queue *queue,
 		spin_lock_irqsave(&queue->rsp_wr_wait_lock, flags);
 		list_add_tail(&cmd->wait_list, &queue->rsp_wr_wait_list);
 		spin_unlock_irqrestore(&queue->rsp_wr_wait_lock, flags);
+		/* The last response may have been released in between. */
+		nvmet_rdma_process_wr_wait_list(queue, NULL);
 	}
 
 	return;
-- 
2.56.0

[PATCH 93/98] nvmet-fc: only send read data for successful commands

[Jens Axboe]

From: Andrea Parri <parri.andrea@gmail.com>

__nvmet_fc_fcp_nvme_cmd_done() pushes the whole data buffer of a read
to the host even when the command failed. The buffer comes from
sgl_alloc() without __GFP_ZERO, and a command rejected by
nvmet_check_transfer_len() never touches it, so the host receives stale
target memory, as much as it asked for.

The tcp and rdma targets only send data when cqe->status is zero. Do
the same for fc. The ERSP then reports fod->offset, which is 0, as the
transferred length, matching what the host checks.

Fixes: c53432030d86 ("nvme-fabrics: Add target support for FC transport")
Cc: stable@vger.kernel.org
Cc: Justin Tee <justin.tee@broadcom.com>
Cc: Naresh Gottumukkala <nareshgottumukkala83@gmail.com>
Cc: Paul Ely <paul.ely@broadcom.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Andrea Parri <parri.andrea@gmail.com>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/fc.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/nvme/target/fc.c b/drivers/nvme/target/fc.c
index eecdffefa24f..b95893adae36 100644
--- a/drivers/nvme/target/fc.c
+++ b/drivers/nvme/target/fc.c
@@ -2487,11 +2487,11 @@ __nvmet_fc_fcp_nvme_cmd_done(struct nvmet_fc_tgtport *tgtport,
 	} else {
 
 		/*
-		 * try to push the data even if the SQE status is non-zero.
-		 * There may be a status where data still was intended to
-		 * be moved
+		 * A failed command may not have filled its read buffer, so
+		 * only push data for success, as the tcp and rdma targets do.
 		 */
-		if ((fod->io_dir == NVMET_FCP_READ) && (fod->data_sg_cnt)) {
+		if (fod->io_dir == NVMET_FCP_READ && fod->data_sg_cnt &&
+		    !cqe->status) {
 			/* push the data over before sending rsp */
 			nvmet_fc_transfer_fcp_data(tgtport, fod,
 						NVMET_FCOP_READDATA);
-- 
2.56.0

[PATCH 94/98] nvmet: wait only for executing I/O in preempt and abort

[Jens Axboe]

From: Chris Mason <mason@kernel.org>

nvmet_parse_io_cmd() takes the pc_ref for an I/O command as soon as it's
parsed, and "preempt and abort" waits for all pc_refs of the preempted
controllers before it releases pr_sem. But a write that still needs its
data can't complete without the host sending it and without the
transport context receiving it, and that same context is where Register,
Acquire and Release sleep on pr_sem. So a host that sends a write and
holds the data back, then preempt and abort with its own key, then
Register on the same queue, deadlocks io_work, the abort work and the
write with each other. Two well-behaved hosts get there by accident
when one fences the other while it re-registers under I/O. The
connection can't be torn down, the namespace can't be disabled, and
with nvmet_config_sem held across that, nobody can connect.

Take the pc_ref when the transport calls ->execute(), once all the data
is there: nvmet_parse_io_cmd() stashes the real execute method and
installs nvmet_pr_execute_io(), which gets the reference, rechecks the
reservation access since the host may have lost it since parse time,
and calls through. Preempt and abort then only waits for commands that
have started executing, which need neither the host nor the transport
to finish. A write whose data arrives after the abort now fails with
Reservation Conflict where it was previously executed, that's the
input the hang needs.

Fixes: 5a47c2080a73 ("nvmet: support reservation feature")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/core.c  |  2 +-
 drivers/nvme/target/nvmet.h |  4 +++-
 drivers/nvme/target/pr.c    | 42 ++++++++++++++++++++++++++++++++++++-
 3 files changed, 45 insertions(+), 3 deletions(-)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index f96b21257169..b78a4d4eb838 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -1171,7 +1171,7 @@ static u16 nvmet_parse_io_cmd(struct nvmet_req *req)
 		if (ret)
 			return ret;
 
-		ret = nvmet_pr_get_ns_pc_ref(req);
+		nvmet_pr_set_execute(req);
 	}
 	return ret;
 }
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 9e8fbb2d1fc1..29105e2a7022 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -517,6 +517,8 @@ struct nvmet_req {
 	struct nvmet_port	*port;
 
 	void (*execute)(struct nvmet_req *req);
+	/* what nvmet_pr_execute_io() calls once it has the pc_ref */
+	void (*pr_execute)(struct nvmet_req *req);
 	const struct nvmet_fabrics_ops *ops;
 
 	struct pci_dev		*p2p_dev;
@@ -1051,7 +1053,7 @@ void nvmet_pr_exit_ns(struct nvmet_ns *ns);
 void nvmet_execute_get_log_page_resv(struct nvmet_req *req);
 u16 nvmet_set_feat_resv_notif_mask(struct nvmet_req *req, u32 mask);
 u16 nvmet_get_feat_resv_notif_mask(struct nvmet_req *req);
-u16 nvmet_pr_get_ns_pc_ref(struct nvmet_req *req);
+void nvmet_pr_set_execute(struct nvmet_req *req);
 static inline void nvmet_pr_put_ns_pc_ref(struct nvmet_pr_per_ctrl_ref *pc_ref)
 {
 	percpu_ref_put(&pc_ref->ref);
diff --git a/drivers/nvme/target/pr.c b/drivers/nvme/target/pr.c
index 1a03f74c2e14..798f5ce9011f 100644
--- a/drivers/nvme/target/pr.c
+++ b/drivers/nvme/target/pr.c
@@ -1060,7 +1060,7 @@ u16 nvmet_pr_check_cmd_access(struct nvmet_req *req)
 	return status;
 }
 
-u16 nvmet_pr_get_ns_pc_ref(struct nvmet_req *req)
+static u16 nvmet_pr_get_ns_pc_ref(struct nvmet_req *req)
 {
 	struct nvmet_pr_per_ctrl_ref *pc_ref;
 
@@ -1072,6 +1072,46 @@ u16 nvmet_pr_get_ns_pc_ref(struct nvmet_req *req)
 	return NVME_SC_SUCCESS;
 }
 
+static void nvmet_pr_execute_io(struct nvmet_req *req)
+{
+	u16 status;
+
+	status = nvmet_pr_get_ns_pc_ref(req);
+	if (!status) {
+		/*
+		 * The host can have lost its registration since the command was
+		 * parsed. A get that succeeds after "preempt and abort" has
+		 * resurrected the pc_ref has to see that. In percpu mode the
+		 * get orders no later load, so this pairs with the
+		 * smp_store_release() in __percpu_ref_switch_to_percpu(). While
+		 * the resurrect is under way the get is a fully ordered
+		 * atomic_long_inc_not_zero(). queue_work() in
+		 * nvmet_execute_pr_acquire() orders the unregistration before
+		 * the resurrect.
+		 */
+		smp_rmb();
+		status = nvmet_pr_check_cmd_access(req);
+	}
+	if (status) {
+		nvmet_req_complete(req, status);
+		return;
+	}
+
+	req->pr_execute(req);
+}
+
+/*
+ * "Preempt and abort" waits for the commands that hold a pc_ref. Until the
+ * transport calls ->execute(), the end of a command can depend on the host,
+ * which may still have to send the data, and on the context and the resources
+ * of the transport. So a command gets its pc_ref only then.
+ */
+void nvmet_pr_set_execute(struct nvmet_req *req)
+{
+	req->pr_execute = req->execute;
+	req->execute = nvmet_pr_execute_io;
+}
+
 static void nvmet_pr_ctrl_ns_all_cmds_done(struct percpu_ref *ref)
 {
 	struct nvmet_pr_per_ctrl_ref *pc_ref =
-- 
2.56.0

[PATCH 95/98] nvmet-rdma: fix double puts after a failed RDMA WRITE

[Jens Axboe]

From: Chris Mason <mason@kernel.org>

For a read with T10-PI the RDMA WRITE is posted alone and
nvmet_rdma_write_data_done() posts the SEND once the metadata check
result is known. By then the request has been completed and its
references put: __nvmet_req_complete() put ns->ref and the pc_ref, and
nvmet_req_complete() sq->ref, right after queuing the response. The
error branch of nvmet_rdma_write_data_done() copies the one from
nvmet_rdma_read_data_done() and calls nvmet_req_uninit(), which puts
all three again. A WRITE fails whenever the connection ends with reads
in flight, which is up to the host. Depending on timing the sq->ref
count goes to zero early and the queue is freed under the remaining
completions, or never reaches zero and nvmet_sq_destroy() sleeps
forever, or percpu_ref_put() dereferences NULL after percpu_ref_exit().
ns->ref stays too low for good, so disabling the namespace hangs too.
Reproduced with KASAN over rxe with a test patch forcing the PI path.

Drop the nvmet_req_uninit(). The branch still releases the RDMA context
and the response, which is all the request owns at that point.

Fixes: b09160c3996c ("nvmet-rdma: add metadata/T10-PI support")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/rdma.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index 7cbf1caf3208..994a668f45d6 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -839,7 +839,10 @@ static void nvmet_rdma_write_data_done(struct ib_cq *cq, struct ib_wc *wc)
 
 	if (unlikely(wc->status != IB_WC_SUCCESS)) {
 		nvmet_rdma_rw_ctx_destroy(rsp);
-		nvmet_req_uninit(&rsp->req);
+		/*
+		 * Unlike in nvmet_rdma_read_data_done(), the request has been
+		 * completed and its references are put: no nvmet_req_uninit().
+		 */
 		nvmet_rdma_release_rsp(rsp);
 		if (wc->status != IB_WC_WR_FLUSH_ERR) {
 			pr_info("RDMA WRITE for CQE failed with status %s (%d).\n",
-- 
2.56.0

[PATCH 96/98] nvmet: take sq->ref in nvmet_sq_fatal_error()

[Jens Axboe]

From: Chris Mason <mason@kernel.org>

nvmet_sq_destroy() kills sq->ref, waits for the commands to drop it,
then puts the controller and clears sq->ctrl. So sq->ctrl may only be
used while holding sq->ref. nvmet_sq_fatal_error() reads it without,
and its transport callers hold nothing either: in nvmet-rdma the
completion handlers run until the QP is drained, which
nvmet_rdma_release_queue_work() does only after nvmet_sq_destroy(), and
in nvmet-tcp io_work can run between the two cancels of the release
work. A short capsule or a failed completion that races the disconnect
then dereferences a NULL sq->ctrl, or locks ctrl->lock and queues
fatal_err_work in a freed controller. The host needs nothing but a
Connect, reproduced over rxe with a sleep added to widen the window.

Take sq->ref with percpu_ref_tryget_live() around the body. If the ref
is already dead the queue is going away and the error is dropped, which
is the mitigation: nothing can tell a call that would have finished in
time from one that wouldn't. tryget_live after percpu_ref_exit() is
fine, it sees __PERCPU_REF_DEAD and touches nothing freed, and
nvmet_req_init() already relies on that. nvmet_auth_fatal_error() runs
in a command that holds the ref, so it calls the body directly.

Fixes: 14c862dbb0a0 ("nvmet-rdma: drain the queue-pair just before freeing it")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Cc: Hannes Reinecke <hare@suse.de>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/core.c             | 21 ++++++++++++++++++++-
 drivers/nvme/target/fabrics-cmd-auth.c |  3 ++-
 drivers/nvme/target/nvmet.h            |  1 +
 3 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index b78a4d4eb838..19f96f1b7e74 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -1833,8 +1833,11 @@ EXPORT_SYMBOL_GPL(nvmet_ctrl_fatal_error);
  * A fatal error on one queue.  The Connect command of an I/O queue needs no
  * secret, so until the queue has authenticated it may be anybody's: do not
  * let it take down the controller of the host whose NQN it has named.
+ *
+ * The caller holds sq->ref, as a command that is being executed does: until it
+ * is put, nvmet_sq_destroy() waits, and does not clear sq->ctrl.
  */
-void nvmet_sq_fatal_error(struct nvmet_sq *sq)
+void __nvmet_sq_fatal_error(struct nvmet_sq *sq)
 {
 	struct nvmet_ctrl *ctrl = sq->ctrl;
 
@@ -1843,6 +1846,22 @@ void nvmet_sq_fatal_error(struct nvmet_sq *sq)
 	else
 		nvmet_ctrl_fatal_error(ctrl);
 }
+
+/*
+ * A transport calls this with no reference of its own, concurrently with
+ * nvmet_sq_destroy(), which puts the controller and clears sq->ctrl.  As in
+ * nvmet_req_init(), the call can come after percpu_ref_exit():
+ * percpu_ref_tryget_live() then sees that the ref is dead and touches nothing
+ * that is freed.
+ */
+void nvmet_sq_fatal_error(struct nvmet_sq *sq)
+{
+	if (!percpu_ref_tryget_live(&sq->ref))
+		return;
+
+	__nvmet_sq_fatal_error(sq);
+	percpu_ref_put(&sq->ref);
+}
 EXPORT_SYMBOL_GPL(nvmet_sq_fatal_error);
 
 ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl,
diff --git a/drivers/nvme/target/fabrics-cmd-auth.c b/drivers/nvme/target/fabrics-cmd-auth.c
index cb6be285b3c4..020a989fe263 100644
--- a/drivers/nvme/target/fabrics-cmd-auth.c
+++ b/drivers/nvme/target/fabrics-cmd-auth.c
@@ -33,12 +33,13 @@ static void nvmet_auth_expired_work(struct work_struct *work)
 	mutex_unlock(&sq->auth_lock);
 }
 
+/* Called from a command that is being executed, which holds sq->ref. */
 static void nvmet_auth_fatal_error(struct nvmet_sq *sq, bool passed)
 {
 	if (passed)
 		nvmet_ctrl_fatal_error(sq->ctrl);
 	else
-		nvmet_sq_fatal_error(sq);
+		__nvmet_sq_fatal_error(sq);
 }
 
 void nvmet_auth_sq_init(struct nvmet_sq *sq)
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 29105e2a7022..3519537705f8 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -631,6 +631,7 @@ void nvmet_sq_destroy(struct nvmet_sq *sq);
 int nvmet_sq_init(struct nvmet_sq *sq, struct nvmet_cq *cq);
 
 void nvmet_ctrl_fatal_error(struct nvmet_ctrl *ctrl);
+void __nvmet_sq_fatal_error(struct nvmet_sq *sq);
 void nvmet_sq_fatal_error(struct nvmet_sq *sq);
 
 void nvmet_update_cc(struct nvmet_ctrl *ctrl, u32 new);
-- 
2.56.0

[PATCH 97/98] nvmet-rdma: hold sq->ref to post a T10-PI Read's SEND

[Jens Axboe]

From: Chris Mason <mason@kernel.org>

For a read with T10-PI, nvmet_rdma_write_data_done() posts the SEND
after the WRITE completes, and by then the request has put sq->ref. So
nvmet_sq_destroy() doesn't wait for it, and the release work can reach
ib_drain_qp() before the SEND is posted. The SEND then goes in behind
the drain's own work request, nothing waits for it, and
nvmet_rdma_send_done() runs against a destroyed QP and freed responses.
Seen once as a use-after-free in sgl_free() with an SRQ over rxe.

Take sq->ref with percpu_ref_tryget_live() while posting the SEND, so
nvmet_sq_destroy() waits for it like for any other response. If the ref
is dead, release the response and post nothing, nobody is left to read
it.

Fixes: b09160c3996c ("nvmet-rdma: add metadata/T10-PI support")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/rdma.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index 994a668f45d6..fe426a0f7e57 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -862,10 +862,23 @@ static void nvmet_rdma_write_data_done(struct ib_cq *cq, struct ib_wc *wc)
 		rsp->req.cqe->status = cpu_to_le16(status << 1);
 	nvmet_rdma_rw_ctx_destroy(rsp);
 
+	/*
+	 * The request has been completed and its reference is put, so nothing
+	 * keeps the release work from draining the queue pair. While we hold
+	 * sq->ref it waits in nvmet_sq_destroy(), and the SEND is posted ahead
+	 * of the drain. If the ref is dead, nobody is left to read a response.
+	 */
+	if (unlikely(!percpu_ref_tryget_live(&queue->nvme_sq.ref))) {
+		nvmet_rdma_release_rsp(rsp);
+		return;
+	}
+
 	if (unlikely(ib_post_send(cm_id->qp, &rsp->send_wr, NULL))) {
 		pr_err("sending cmd response failed\n");
 		nvmet_rdma_release_rsp(rsp);
 	}
+
+	percpu_ref_put(&queue->nvme_sq.ref);
 }
 
 static void nvmet_rdma_use_inline_sg(struct nvmet_rdma_rsp *rsp, u32 len,
-- 
2.56.0

Re: [PATCH 97/98] nvmet-rdma: hold sq->ref to post a T10-PI Read's SEND

[Bart Van Assche]

On 10/28/26 09:42, Jens Axboe wrote:
> +	if (unlikely(!percpu_ref_tryget_live(&queue->nvme_sq.ref))) {
> +		nvmet_rdma_release_rsp(rsp);
> +		return;
> +	}
> +
>  	if (unlikely(ib_post_send(cm_id->qp, &rsp->send_wr, NULL))) {
>  		pr_err("sending cmd response failed\n");
>  		nvmet_rdma_release_rsp(rsp);
>  	}
> +
> +	percpu_ref_put(&queue->nvme_sq.ref);

In nvmet_rdma_free_queue(), nvmet_sq_destroy(&queue->nvme_sq) calls
percpu_ref_exit(&sq->ref) before nvmet_rdma_destroy_queue_ib() calls
ib_drain_qp(queue->qp). Because nvmet_req_complete() still drops sq->ref
immediately after posting &rsp->write_cqe, nvmet_sq_destroy() can reach
percpu_ref_exit(&queue->nvme_sq.ref) while the T10-PI RDMA WRITE is in
flight, and nvmet_rdma_write_data_done() will then call
percpu_ref_tryget_live() concurrently with or after percpu_ref_exit().

percpu_ref_tryget_live() is documented as safe only between
percpu_ref_init() and percpu_ref_exit(), and __percpu_ref_exit() /
percpu_ref_exit() perform unmarked writes to ref->percpu_count_ptr that
race under KCSAN with percpu_ref_tryget_live_rcu()'s unmarked read of
ref->percpu_count_ptr. (Note: patch 96/98 has the same KCSAN race and
should also be squashed into patch 69/98 so commits 69..95 do not leave
nvmet_sq_fatal_error() accessing sq->ctrl without sq->ref.)

Here in nvmet_rdma_write_data_done(), we don't need tryget_live() at
all: nvmet_rdma_queue_response() runs while the request's own sq->ref is
still held, so it can call percpu_ref_get(&rsp->queue->nvme_sq.ref)
before posting &rsp->write_cqe (putting it on synchronous ib_post_send()
failure and at the end of nvmet_rdma_write_data_done()). That keeps
sq->ref live across the entire T10-PI RDMA WRITE -> SEND sequence so
nvmet_sq_destroy() waits cleanly before percpu_ref_exit().

Re: [PATCH 97/98] nvmet-rdma: hold sq->ref to post a T10-PI Read's SEND

[Bart Van Assche]

On 10/28/26 09:42, Jens Axboe wrote:
> +	if (unlikely(!percpu_ref_tryget_live(&queue->nvme_sq.ref))) {
> +		nvmet_rdma_release_rsp(rsp);
> +		return;
> +	}
> +
>  	if (unlikely(ib_post_send(cm_id->qp, &rsp->send_wr, NULL))) {
>  		pr_err("sending cmd response failed\n");
>  		nvmet_rdma_release_rsp(rsp);
>  	}
> +
> +	percpu_ref_put(&queue->nvme_sq.ref);

In nvmet_rdma_free_queue(), nvmet_sq_destroy(&queue->nvme_sq) calls
percpu_ref_exit(&sq->ref) before nvmet_rdma_destroy_queue_ib() calls
ib_drain_qp(queue->qp). Because nvmet_req_complete() still drops sq->ref
immediately after posting &rsp->write_cqe, nvmet_sq_destroy() can reach
percpu_ref_exit(&queue->nvme_sq.ref) while the T10-PI RDMA WRITE is in
flight, and nvmet_rdma_write_data_done() will then call
percpu_ref_tryget_live() concurrently with or after percpu_ref_exit().

percpu_ref_tryget_live() is only valid between percpu_ref_init() and
percpu_ref_exit(): __percpu_ref_exit() and percpu_ref_exit() perform
unmarked writes to ref->percpu_count_ptr that race under KCSAN with
percpu_ref_tryget_live_rcu()'s unmarked read. (As a side note, patch
96/98 has the same tryget_live-vs-exit issue in nvmet_rdma_error_comp()
and should also be folded into patch 69/98.)

Here we do not need percpu_ref_tryget_live() at all:
nvmet_rdma_queue_response() runs while the request's own sq->ref is
still held, so it can take an extra percpu_ref_get() on
queue->nvme_sq.ref when preparing a T10-PI RDMA WRITE and drop it either
on synchronous ib_post_send() failure or at the end of
nvmet_rdma_write_data_done(). Because __nvmet_rdma_queue_disconnect()
calls rdma_disconnect() (transitioning the QP to IB_QPS_ERR and flushing
outstanding WRs) before scheduling release_work -> nvmet_sq_destroy(),
holding sq->ref across the RDMA WRITE cannot block disconnect and keeps
sq->ref live until the SEND has been posted.

Here is a replacement patch (checked with checkpatch.pl and
syntax-checked with Clang; not runtime-tested):

---
diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index 994a668f45d6..ec54e3706e6c 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -758,6 +758,7 @@ static void nvmet_rdma_queue_response(struct nvmet_req *req)
 		container_of(req, struct nvmet_rdma_rsp, req);
 	struct rdma_cm_id *cm_id = rsp->queue->cm_id;
 	struct ib_send_wr *first_wr;
+	bool hold_sq = false;
 
 	if (rsp->invalidate_rkey) {
 		rsp->send_wr.opcode = IB_WR_SEND_WITH_INV;
@@ -767,12 +768,20 @@ static void nvmet_rdma_queue_response(struct nvmet_req *req)
 	}
 
 	if (nvmet_rdma_need_data_out(rsp)) {
-		if (rsp->req.metadata_len)
+		if (IS_ENABLED(CONFIG_BLK_DEV_INTEGRITY) &&
+		    rsp->req.metadata_len) {
+			/*
+			 * Hold sq->ref until nvmet_rdma_write_data_done()
+			 * posts the SEND ahead of ib_drain_qp().
+			 */
+			percpu_ref_get(&rsp->queue->nvme_sq.ref);
+			hold_sq = true;
 			first_wr = rdma_rw_ctx_wrs(&rsp->rw, cm_id->qp,
 					cm_id->port_num, &rsp->write_cqe, NULL);
-		else
+		} else {
 			first_wr = rdma_rw_ctx_wrs(&rsp->rw, cm_id->qp,
 					cm_id->port_num, NULL, &rsp->send_wr);
+		}
 	} else {
 		first_wr = &rsp->send_wr;
 	}
@@ -786,6 +795,8 @@ static void nvmet_rdma_queue_response(struct nvmet_req *req)
 	if (unlikely(ib_post_send(cm_id->qp, first_wr, NULL))) {
 		pr_err("sending cmd response failed\n");
 		nvmet_rdma_release_rsp(rsp);
+		if (hold_sq)
+			percpu_ref_put(&rsp->queue->nvme_sq.ref);
 	}
 }
 
@@ -849,6 +860,7 @@ static void nvmet_rdma_write_data_done(struct ib_cq *cq, struct ib_wc *wc)
 				ib_wc_status_msg(wc->status), wc->status);
 			nvmet_rdma_error_comp(queue);
 		}
+		percpu_ref_put(&queue->nvme_sq.ref);
 		return;
 	}
 
@@ -866,6 +878,7 @@ static void nvmet_rdma_write_data_done(struct ib_cq *cq, struct ib_wc *wc)
 		pr_err("sending cmd response failed\n");
 		nvmet_rdma_release_rsp(rsp);
 	}
+	percpu_ref_put(&queue->nvme_sq.ref);
 }
 
 static void nvmet_rdma_use_inline_sg(struct nvmet_rdma_rsp *rsp, u32 len,

[PATCH 98/98] nvmet: don't clear sq->ctrl for a Connect with sqsize 0

[Jens Axboe]

From: Chris Mason <mason@kernel.org>

nvmet_install_queue() checks sqsize first and jumps to err on zero,
where sq->ctrl is set to NULL. Since commit b71df12605ca the cmpxchg()
that sets sq->ctrl comes after the checks, so err now undoes a bind
that this call never made. Over RDMA a host can send two Connects
together and the second is parsed before the first executes, so a
second Connect with sqsize 0 clears the sq->ctrl the first one set.
The reference it held is never dropped and the controller leaks, with
its cntlid, until reboot. The discovery subsystem lets any host do
that, 20 connections leak around 10 controllers. A pending AER on such
a queue hangs nvmet_sq_destroy() forever, and a queue can keep its
"authenticated" mark across a rebind.

Return the status directly, as the two checks after it do, so err is
only reached after this call's cmpxchg() succeeded.

Fixes: b71df12605ca ("nvmet: avoid duplicate qid in connect cmd")
Cc: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Chaitanya Kulkarni <kch@nvidia.com>
Signed-off-by: Chris Mason <mason@kernel.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
---
 drivers/nvme/target/fabrics-cmd.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/drivers/nvme/target/fabrics-cmd.c b/drivers/nvme/target/fabrics-cmd.c
index 089e3082c388..5614849edece 100644
--- a/drivers/nvme/target/fabrics-cmd.c
+++ b/drivers/nvme/target/fabrics-cmd.c
@@ -186,8 +186,7 @@ static u16 nvmet_install_queue(struct nvmet_ctrl *ctrl, struct nvmet_req *req)
 		pr_warn("queue size zero!\n");
 		req->error_loc = offsetof(struct nvmf_connect_command, sqsize);
 		req->cqe->result.u32 = IPO_IATTR_CONNECT_SQE(sqsize);
-		ret = NVME_SC_CONNECT_INVALID_PARAM | NVME_STATUS_DNR;
-		goto err;
+		return NVME_SC_CONNECT_INVALID_PARAM | NVME_STATUS_DNR;
 	}
 
 	if (ctrl->sqs[qid] != NULL) {
-- 
2.56.0

Re: [PATCHSET 0/98] Various block related fixes

[Keith Busch]

On Wed, Oct 28, 2026 at 10:41:03AM -0600, Jens Axboe wrote:
> Got a giant pile of stuff dumped on me, and went through and did some
> vetting and, particularly, commit message rewriting to get it to a
> somewhat comprehensible shape. If you're CC'ed on a patch, please
> review it. Feel free to queue it up, or reply back why the particular
> patch is garbage, if it is garbage.
> 
> Thanks!

Neat! This looks like it is tackling my top complaint about sashiko: it
flags pre-existing issues only when a patch is submitted touching code
in the general area instead of proactively identifying and proposing
fixes to those pre-existing bugs.

Patches 66 and 93 have identical commits from other contributors that I
have already staged in the nvme tree, and patch 54 is fixed differently
than proposed here. Pull request coming this afternoon.

For the rest, lacking any block layer dependencies, I'd prefer splitting
the nvme subsystem specific patches into their own series and they go
through our nvme tree.

Re: [PATCHSET 0/98] Various block related fixes

[Jens Axboe]

On 10/28/26 11:11 AM, Keith Busch wrote:
> On Wed, Oct 28, 2026 at 10:41:03AM -0600, Jens Axboe wrote:
>> Got a giant pile of stuff dumped on me, and went through and did some
>> vetting and, particularly, commit message rewriting to get it to a
>> somewhat comprehensible shape. If you're CC'ed on a patch, please
>> review it. Feel free to queue it up, or reply back why the particular
>> patch is garbage, if it is garbage.
>>
>> Thanks!
> 
> Neat! This looks like it is tackling my top complaint about sashiko: it
> flags pre-existing issues only when a patch is submitted touching code
> in the general area instead of proactively identifying and proposing
> fixes to those pre-existing bugs.
> 
> Patches 66 and 93 have identical commits from other contributors that I
> have already staged in the nvme tree, and patch 54 is fixed differently
> than proposed here. Pull request coming this afternoon.

Ah excellent - I have no issues dropping whatever patches, just wanted
to flush out this queue.

> For the rest, lacking any block layer dependencies, I'd prefer splitting
> the nvme subsystem specific patches into their own series and they go
> through our nvme tree.

No dependencies between eg nvme patches and block patches. For nvme,
ideally you just grab all of them, and I can drop them locally. They
only exist locally because they had to go somewhere first.

-- 
Jens Axboe

Re: [PATCHSET 0/98] Various block related fixes

[Keith Busch]

On Wed, Oct 28, 2026 at 10:41:03AM -0600, Jens Axboe wrote:
> Got a giant pile of stuff dumped on me, and went through and did some
> vetting and, particularly, commit message rewriting to get it to a
> somewhat comprehensible shape. If you're CC'ed on a patch, please
> review it. Feel free to queue it up, or reply back why the particular
> patch is garbage, if it is garbage.

cc'ing nvme; original thread:

  https://lore.kernel.org/linux-block/20261028164415.1262013-1-axboe@kernel.dk/

I've picked a few of the obvious ones for nvme from Josef and Chris: 45,
65, 72, 73, 76, 78, 80, 88, 90, 95, and 98. The rest want a more careful
look.

Re: [PATCHSET 0/98] Various block related fixes

[Bart Van Assche]

[-- Attachment #1: Type: text/plain, Size: 574 bytes --]

On 10/28/26 9:41 AM, Jens Axboe wrote:
> Got a giant pile of stuff dumped on me, and went through and did some
> vetting and, particularly, commit message rewriting to get it to a
> somewhat comprehensible shape. If you're CC'ed on a patch, please
> review it. Feel free to queue it up, or reply back why the particular
> patch is garbage, if it is garbage.

Hi Jens,

Reviewing 98 patches at once is too much for a human. So I asked an LLM
to take a look. That LLM identified 9 must fix issues and 15 minor
issues. Please take a look at the attached report.

Thanks,

Bart.

[-- Attachment #2: block_fixes_98_review.md --]
[-- Type: text/markdown, Size: 62962 bytes --]

# Review of `[PATCHSET 0/98] Various block related fixes`

- **Series**: https://lore.kernel.org/linux-block/20261028164415.1262013-1-axboe@kernel.dk/
- **Poster**: Jens Axboe `<axboe@kernel.dk>` (77 patches from Josef Bacik, 20 from Chris Mason, 1 from Andrea Parri)
- **Baseline Verified Against**: `origin/master` (`d135ab453a74`) — all 98/98 patches apply cleanly in sequence across 45 files; all `Fixes:` commit SHAs and titles resolve accurately in git history (with 1 minor root-cause SHA note on Patch 06).

> [!IMPORTANT]
> **High-Level Assessment ("Valid Fixes vs. Garbage")**: **0 of the 98 patches target imaginary or unreachable bugs.** Every patch in this batch diagnoses a real upstream defect — including numerous remotely triggerable use-after-frees, out-of-bounds memory accesses, authentication/TLS bypasses, and lockups across `blk-mq`, `blk-cgroup`, `blk-zoned`, `rnbd-srv`, `nvme-tcp`, `nvmet-auth`, `nvmet-tcp`, `nvmet-rdma`, and `nvmet` persistent reservations.
>
> However, **9 patches have substantive bugs, regressions, or incomplete fixes** in their proposed implementation (most critically **Patches 28, 41, 48, 90, and 97**), and **15 patches have minor edge cases or cleanups** worth raising on `linux-block`.

## Verdict Breakdown

| Verdict | Count | Patches |
| :--- | ---: | :--- |
| **Needs Changes / Incomplete (`NEEDS WORK`)** | **9** | `18, 20, 22, 28, 41, 48, 76, 90, 97` |
| **Valid Fix w/ Minor Note (`MINOR`)** | **15** | `06, 10, 11, 17, 21, 23, 29, 39, 49, 53, 55, 61, 71, 78, 84` |
| **Clean & Ready (`LGTM`)** | **74** | All remaining 74 patches (see Section 5) |

---

## 1. Patches Requiring Changes (`NEEDS WORK`)

### 1.1 Patch 28/98 — `blk-iocost: check again that the disk is alive` (Buggy Unwind -> `request_queue` UAF)
- **Patch**: [28/98](https://lore.kernel.org/linux-block/20261028164415.1262013-29-axboe@kernel.dk/) (Josef Bacik) — [`ioc_qos_write()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/block/blk-iocost.c#L3293-L3448) in [blk-iocost.c](file:///usr/local/google/home/bvanassche/software/linux-kernel/block/blk-iocost.c#L3317-L3442)
- **Root-Cause Diagnosis**: Valid. `ioc_qos_write()` drops `rq_qos_mutex` before `blk_mq_freeze_queue()`, allowing `del_gendisk()` + `blk_mq_free_tag_set()` to run in between so `disk->queue->tag_set` is freed before `blk_mq_quiesce_queue()`.
- **Bug in Patch**: When `!disk_live(disk)` is true (meaning `del_gendisk()` has run and the driver may already have called `put_disk()`), jumping to `close_bdev:` executes:
  ```c
  close_bdev:
  	q = ctx.bdev->bd_queue;
  	blkg_conf_close_bdev(&ctx);
  	blk_mq_unfreeze_queue(q, memflags);
  ```
  [`blkg_conf_close_bdev()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/block/blk-cgroup.c#L956-L963) unlocks `rq_qos_mutex` and calls `blkdev_put_no_open(ctx.bdev)`. When `ctx.bdev` holds the final reference to `bdev`/`disk`, `blkdev_put_no_open()` synchronously triggers `disk_release()` -> `blk_put_queue(q)`, **freeing `q` while it is still frozen**, and the very next line `blk_mq_unfreeze_queue(q, memflags)` **dereferences the freed `struct request_queue` (`q->mq_freeze_lock`, `q->q_usage_counter`, `q->io_lockdep_map`)**.
- **Suggested Fix**: Unlock `q->rq_qos_mutex` and call `blk_mq_unfreeze_queue(q, memflags)` *before* `blkdev_put_no_open(ctx.bdev)` so `ctx.bdev` pins `disk` and `q` across the unfreeze:
  ```c
  close_bdev:
  	q = ctx.bdev->bd_queue;
  	mutex_unlock(&q->rq_qos_mutex);
  	blk_mq_unfreeze_queue(q, memflags);
  	blkdev_put_no_open(ctx.bdev);
  	return ret ?: nbytes;
  ```

### 1.2 Patch 41/98 — `zram: check a bio's range against the device size` (NULL Deref on Empty Bio + 4 KiB Page Overrun / Infoleak)
- **Patch**: [41/98](https://lore.kernel.org/linux-block/20261028164415.1262013-42-axboe@kernel.dk/) (Chris Mason) — [`zram_submit_bio()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/block/zram/zram_drv.c#L2800-L2819) in [zram_drv.c](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/block/zram/zram_drv.c#L212-L223)
- **Root-Cause Diagnosis**: Valid. Commit `9fe95babc742 ("zram: remove valid_io_request")` relied on `bio_check_eod()`, which truncates `bi_size` to whole 512-byte sectors (`bio_sectors(bio)`), letting sub-sector bios or sub-sector tails step past `zram->table` (or dereference `zram->table == NULL` on an uninitialized zram disk).
- **Two Bugs in `zram_bio_in_range()`**:
  1. **Empty `REQ_OP_READ` / `REQ_OP_WRITE` bios crash immediately**: The new comment states *"An empty bio has to start inside the device as well: the loops for reads and writes run once before their test"*, and `zram_bio_in_range()` returns `true` when `bi_size == 0` and `start < nr_sectors` (`len == 0 <= nr_sectors - start`). Entering [`zram_bio_read()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/block/zram/zram_drv.c#L2736-L2765) or [`zram_bio_write()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/block/zram/zram_drv.c#L2767-L2795) with `bi_size == 0` (`bi_io_vec == NULL`) immediately executes `bio_iter_iovec(bio, iter)`, **dereferencing `NULL` `bio->bi_io_vec`**.
  2. **Sub-4K / unaligned bios corrupt or leak 4 KiB pages on `PAGE_SIZE == 4096` kernels**: On `PAGE_SIZE == 4096` (x86\_64, arm64-4K), `ZRAM_LOGICAL_BLOCK_SIZE` is `4096` and [`is_partial_io()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/block/zram/zram_drv.c#L212-L223) is hardcoded to `return false;`. If an in-kernel bio builder (e.g., `rnbd-srv`, the exact threat model in the commit message) submits a 512-byte or sub-sector bio inside the device range, `zram_bio_in_range()` allows it through:
     - [`zram_bvec_read()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/block/zram/zram_drv.c#L2229-L2235) calls `zram_read_page(zram, bvec->bv_page, index, bio)`, ignoring `bv_offset` and `bv_len` and overwriting all 4096 bytes of `bv_page` from offset 0.
     - `zram_bvec_write()` calls `zram_write_page(zram, bvec->bv_page, index)`, compressing all 4096 bytes of `bv_page` into the zram slot — corrupting the rest of the 4 KiB block and **leaking up to 4095 bytes of adjacent kernel page memory** on a subsequent read.
- **Suggested Fix**: Reinstate the `ZRAM_LOGICAL_BLOCK_SIZE` (`4096`) alignment check on both `bi_sector` and `bi_size` (just as `valid_io_request()` originally did) and reject `bi_size == 0` before entering `zram_bio_read()` / `zram_bio_write()`.

### 1.3 Patch 48/98 — `nvmet-tcp: rewind the iterator after non-data records` (`TLS_RECORD_TYPE_ALERT` Corrupts / Misreads `ITER_BVEC` & Short Alerts)
- **Patch**: [48/98](https://lore.kernel.org/linux-block/20261028164415.1262013-49-axboe@kernel.dk/) (Josef Bacik) — [`nvmet_tcp_tls_record_ok()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/tcp.c#L1176-L1208) in [tcp.c](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/tcp.c#L1176-L1305)
- **Root-Cause Diagnosis**: Valid. Rewinding `&msg->msg_iter` by `len` prevents non-data TLS records from advancing the destination iterator and causing KASAN OOB / command data corruption.
- **Two Remaining Bugs on `TLS_RECORD_TYPE_ALERT`**:
  1. **Type confusion (`ITER_BVEC` cast to `struct kvec *`) in `nvmet_tcp_try_recv_data()`**: [`nvmet_tcp_try_recv_data()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/tcp.c#L1281-L1310) initializes `cmd->recv_msg.msg_iter` via `iov_iter_bvec(&cmd->recv_msg.msg_iter, ITER_DEST, cmd->req.sg, ...)` (`ITER_BVEC`). When a TLS Alert arrives during `NVMET_TCP_RECV_DATA`, `nvmet_tcp_tls_record_ok()` calls [`tls_alert_recv()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/net/handshake/alert.c#L97-L110), which blindly reads union member `msg->msg_iter.kvec->iov_base`! For an `ITER_BVEC`, `iov->iov_base` aliases `bvec->bv_page` (`struct page *` in `vmemmap`), so `tls_alert_recv()` reads `((u8 *)bv_page)[0..1]` (`page->flags`!) instead of the received alert payload — silently ignoring fatal TLS alerts (`level != 2`) and returning `-EAGAIN`! (Plus the alert payload overwrote up to 2 bytes of the command's SG page before `iov_iter_revert()`, which matters if a warning alert is ignored and not followed by full command data.)
  2. **1-byte OOB / stale byte read on `len < 2`**: If `queue->left == 1` when a 2-byte TLS Alert arrives, `sock_recvmsg()` receives `len == 1` byte (with `MSG_EOR` cleared), and `tls_alert_recv()` unconditionally reads `data[0]` and `data[1]` without checking `len >= 2` — reading 1 byte past a 1-byte buffer or reading stale stack memory.
- **Suggested Fix**: Fix `tls_alert_recv()` (or avoid calling it on `ITER_BVEC` / when `len < 2`; note that under TLS 1.3, RFC 8446 §6 treats all alerts other than `close_notify` and `user_canceled` as fatal anyway, so any alert during NVMe/TCP PDU/data transfer can tear down the connection without inspecting `ITER_BVEC` memory).

### 1.4 Patch 90/98 — `nvmet-rdma: disable interrupts under rsp_wr_wait_lock` (Executes `req.execute()` & `mutex_lock()` in HardIRQ/SoftIRQ Context)
- **Patch**: [90/98](https://lore.kernel.org/linux-block/20261028164415.1262013-91-axboe@kernel.dk/) (Chris Mason) — [`nvmet_rdma_process_wr_wait_list()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/rdma.c#L514-L537) in [rdma.c](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/rdma.c#L514-L537)
- **Root-Cause Diagnosis**: Valid. `nvmet_bio_done()` -> `nvmet_req_complete()` -> [`nvmet_rdma_queue_response()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/rdma.c#L717-L752) can run in hardirq or softirq context; when `ib_post_send()` fails, `nvmet_rdma_release_rsp()` calls `nvmet_rdma_process_wr_wait_list()`, which previously took `rsp_wr_wait_lock` with plain `spin_lock()`.
- **Why the Fix Is Incomplete**: As the commit message itself notes, `nvmet_rdma_process_wr_wait_list()` drops `rsp_wr_wait_lock` (restoring the caller's hardirq/softirq state) and calls `nvmet_rdma_execute_command(rsp)` -> `rsp->req.execute(&rsp->req)` **directly in hardirq/softirq context**! Almost every `nvmet` `req.execute` handler assumes process context:
  - `nvmet_bdev_execute_rw()` calls `bio_alloc(..., GFP_KERNEL)` and `submit_bio()` -> `submit_bio_noacct()`, which hits `might_sleep()` and manipulates `current->bio_list` without interrupt protection (corrupting any interrupted task's `current->bio_list`!).
  - Admin/fabrics/auth/PR handlers (`nvmet_execute_identify_*`, `nvmet_execute_admin_connect`, `nvmet_execute_auth_*`, `nvmet_execute_pr_*`) call `kzalloc(..., GFP_KERNEL)`, `mutex_lock()`, and `down_read()`.
  - Furthermore, if **Patch 69** is applied and `ib_post_send()` fails on a dying queue in hardirq/softirq context (`nvmet_rdma_queue_response()` doesn't currently call `nvmet_rdma_error_comp()`, though `nvmet_rdma_Disconnect` paths take `mutex_lock(&nvmet_rdma_queue_mutex)`), any fatal error path assuming process context will sleep in atomic.
- **Suggested Fix**: Offload `nvmet_rdma_process_wr_wait_list()` to a `work_struct` on `nvmet_wq` (or queue the work whenever `!in_task()`) instead of calling `nvmet_rdma_execute_command()` in interrupt context.

### 1.5 Patch 97/98 (and Patch 96/98 note) — `nvmet-rdma: hold sq->ref to post a T10-PI Read's SEND` (`percpu_ref_tryget_live()` Concurrent With / After `percpu_ref_exit()`)
- **Patches**: [97/98](https://lore.kernel.org/linux-block/20261028164415.1262013-98-axboe@kernel.dk/) and [96/98](https://lore.kernel.org/linux-block/20261028164415.1262013-97-axboe@kernel.dk/) (Chris Mason) — [`nvmet_rdma_write_data_done()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/rdma.c#L787-L828) in [rdma.c](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/rdma.c#L1361-L1395) and [`nvmet_sq_destroy()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/core.c#L985-L1023) in [core.c](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/core.c#L985-L1023)
- **Root-Cause Diagnosis**: Valid. For a T10-PI Read, `nvmet_req_complete()` drops `sq->ref` right after posting the `RDMA WRITE`, so `nvmet_sq_destroy()` doesn't wait for `nvmet_rdma_write_data_done()` to post `&rsp->send_wr`, allowing `&rsp->send_wr` to be posted *behind* `ib_drain_qp()`'s `sdrain` WR.
- **API Contract Violation & KCSAN Race**:
  - In [`nvmet_rdma_free_queue()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/rdma.c#L1379-L1395), `nvmet_sq_destroy(&queue->nvme_sq)` (which calls [`percpu_ref_exit(&sq->ref)`](file:///usr/local/google/home/bvanassche/software/linux-kernel/lib/percpu-refcount.c#L130-L148)) runs **before** `nvmet_rdma_destroy_queue_ib(queue)` -> `ib_drain_qp(queue->qp)`.
  - While Patch 96's commit message notes that `percpu_ref_tryget_live()` happens not to dereference `ref->data` (`NULL`) after `percpu_ref_exit()` because `__PERCPU_REF_ATOMIC_DEAD` remains set in `ref->percpu_count_ptr`, [`percpu_ref_tryget_live()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/include/linux/percpu-refcount.h#L276-L315) explicitly documents *"This function is safe to call as long as @ref is between init and exit"*, and [`__percpu_ref_exit()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/lib/percpu-refcount.c#L108-L147) / `percpu_ref_exit()` perform unmarked plain writes to `ref->percpu_count_ptr` while `percpu_ref_tryget_live_rcu()` performs an unmarked read (`!(ref->percpu_count_ptr & __PERCPU_REF_DEAD)`), which is a **KCSAN data race**.
  - Specifically in **Patch 97**, calling `percpu_ref_tryget_live()` after `sq->ref` was dropped is completely avoidable: [`nvmet_rdma_queue_response()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/rdma.c#L717-L752) runs while the request's own `sq->ref` is still held! Simply calling `percpu_ref_get(&rsp->queue->nvme_sq.ref)` before posting `&rsp->write_cqe` (putting it on synchronous `ib_post_send` failure and at the end of `nvmet_rdma_write_data_done()`) keeps `sq->ref` genuinely live across the entire T10-PI `RDMA WRITE` -> `SEND` handoff so `nvmet_sq_destroy()` waits cleanly before `percpu_ref_exit()`.

### 1.6 Patches 20/98 & 22/98 — `mtip32xx` Surprise-Removal Ordering Deadlock (`blk_mark_disk_dead` / `kthread_stop` Before Aborting In-Flight Commands)
- **Patches**: [20/98](https://lore.kernel.org/linux-block/20261028164415.1262013-21-axboe@kernel.dk/) and [22/98](https://lore.kernel.org/linux-block/20261028164415.1262013-23-axboe@kernel.dk/) (Josef Bacik) — [`mtip_service_thread()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/block/mtip32xx/mtip32xx.c#L2465-L2535) and [`mtip_pci_remove()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/block/mtip32xx/mtip32xx.c#L3874-L3955) in [mtip32xx.c](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/block/mtip32xx/mtip32xx.c#L133-L150)
- **Root-Cause Diagnosis**: Valid. Calling `blk_mark_disk_dead()` from `mtip_handle_irq()` sleeps in hardirq (`mutex_lock(&disk->part0->bd_holder_lock)`), and `del_gendisk()` hangs on surprise removal if in-flight NCQ commands are never aborted.
- **Remaining Deadlock & Race**:
  1. When a filesystem is mounted on `/dev/rssd*`, `blk_mark_disk_dead()` invokes `fs_bdev_mark_dead()` -> `bdev_super_lock(bdev, ...)` (`down_write(&sb->s_umount)`) and `sync_filesystem()`. In **Patch 20**, `mtip_service_thread()` calls `blk_mark_disk_dead(dd->disk)` *before* aborting in-flight NCQ commands; if another task holds `s_umount` waiting on an uncompleted NCQ command on the dead PCIe device, `mtip_service_thread` blocks on `s_umount` and **Patch 22**'s `mtip_stop_service_thread()` (`kthread_stop()`) blocks waiting for `mtip_service_thread` *before* reaching `blk_mq_tagset_busy_iter(&dd->tags, mtip_abort_cmd, dd)`!
  2. Conversely, if PCIe link loss (`dd->sr = true`) occurs *after* `mtip_pci_remove()` has already passed `if (dd->sr)` and entered `del_gendisk(dd->disk)`, `MTIP_DDF_REMOVE_PENDING_BIT` causes `mtip_service_thread()` to exit immediately without aborting in-flight commands, hanging `del_gendisk()` anyway.
- **Suggested Fix**: In `mtip_service_thread()`, quiesce `dd->queue` and abort in-flight commands (`blk_mq_tagset_busy_iter(&dd->tags, mtip_abort_cmd, dd)`) *before* calling `blk_mark_disk_dead(dd->disk)`, and do not let `MTIP_DDF_REMOVE_PENDING_BIT` skip surprise-removal command abortion while `del_gendisk()` is still draining I/O.

### 1.7 Patch 18/98 — `aoe: don't free a device with its work queued` (`DEVFL_TKILL` Set Too Late After `aoedev_downdev()`)
- **Patch**: [18/98](https://lore.kernel.org/linux-block/20261028164415.1262013-19-axboe@kernel.dk/) (Josef Bacik) — [`ataid_complete()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/block/aoe/aoecmd.c#L930-L950) in [aoecmd.c](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/block/aoe/aoecmd.c#L898-L950) and [`flush()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/block/aoe/aoedev.c#L315-L370) in [aoedev.c](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/block/aoe/aoedev.c#L272-L370)
- **Root-Cause Diagnosis**: Valid. Commit `430380b4637a` dropped `d->lock` across `aoedev_downdev(d)` without preventing `ataid_complete()` from scheduling `d->work`, leading to UAF on `d->work` when `freedev(d)` freed `d`.
- **Remaining Race**: Patch 18 sets `d->flags |= DEVFL_TKILL` *after* `aoedev_downdev(d)` returns rather than *before* dropping `d->lock` at line 356. During the unlocked `aoedev_downdev(d)` window, an incoming ATA IDENTIFY reply still sees `!(d->flags & DEVFL_TKILL)` in `ataid_complete()`, schedules [`aoecmd_sleepwork()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/block/aoe/aoecmd.c#L898-L928) (which runs `aoedisk_add_disk(d)` and sets `d->flags |= DEVFL_UP` without checking `DEVFL_TKILL`!), re-bringing the device UP *after* `aoedev_downdev(d)` finished so `freedev()` can receive new bios without anyone failing them during `del_gendisk()`. (Or if `aoecmd_sleepwork` hasn't run yet when `flush()` restarts, `flush()` hits `WARN_ON(d->flags & DEVFL_TKILL)` when skipping `DEVFL_GDALLOC | DEVFL_NEWSIZE`.)
- **Suggested Fix**: Set `d->flags |= DEVFL_TKILL` *before* dropping `d->lock` prior to `aoedev_downdev(d)` in `flush()` (and remove the then-unreachable `WARN_ON(d->flags & DEVFL_TKILL)` on the `restart1` loop).

### 1.8 Patch 76/98 — `nvmet: fix NULL dereference in debugfs host_traddr` (Unlocked TOCTOU NULL-Deref & UAF Remains)
- **Patch**: [76/98](https://lore.kernel.org/linux-block/20261028164415.1262013-77-axboe@kernel.dk/) (Josef Bacik) — [`nvmet_ctrl_host_traddr()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/core.c#L1822-L1830) in [core.c](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/core.c#L1822-L1830)
- **Root-Cause Diagnosis**: Valid. When the admin queue (`ctrl->sqs[0]`) disconnects while I/O queues remain open, `nvmet_sq_destroy()` sets `ctrl->sqs[0] = NULL`, so reading `host_traddr` in debugfs crashes with a near-NULL dereference (`container_of(ctrl->sqs[0], ...)`).
- **Incomplete Synchronization**: As the commit message candidly admits (*"Nothing holds the queue for debugfs, so a reader that's past the check when the queue goes away is still on its own"*), `if (!ctrl->sqs[0])` is a plain unlocked load, and `nvmet_tcp_host_port_addr()` / `nvmet_rdma_host_port_addr()` / `nvmet_fc_host_traddr()` immediately perform a second unlocked load of `ctrl->sqs[0]` and dereference the transport queue (`queue->sockaddr_peer`, `nvme_rdma_cm_id`, `assoc->hostport`) concurrently with `nvmet_sq_destroy()` freeing the queue (`kfree(queue)` -> **TOCTOU NULL deref and KASAN UAF**).
- **Suggested Fix**: Snapshot the formatted `host_traddr` string into a small `char host_traddr[NVMF_TRADDR_SIZE]` buffer in `struct nvmet_ctrl` during admin queue `Connect` (`nvmet_alloc_ctrl()` / `install_queue`), and have `nvmet_ctrl_host_traddr()` copy that immutable buffer. That eliminates all transport queue lifetime races with ~4 lines of code.

---

## 2. Valid Fixes With Minor Issues or Follow-up Notes (`MINOR`)

1. **Patch 06/98** ([06/98](https://lore.kernel.org/linux-block/20261028164415.1262013-7-axboe@kernel.dk/) — `blk-mq: don't batch allocate tags from an inactive hctx`):
   - Code is spot-on. `Fixes:` points to `47c122e35d7e` (when `blk_mq_is_sbitmap_shared()` renamed the check), whereas the actual bug originated in `349302da8352 ("blk-mq: support batching dispatch in case of io scheduler")` which omitted `blk_mq_hctx_stopped(hctx)` from the batch path.
2. **Patch 10/98** ([10/98](https://lore.kernel.org/linux-block/20261028164415.1262013-11-axboe@kernel.dk/) — `block: don't unplug a zone under a failed write`):
   - Valid fix. Note that moving `zwplug->flags &= ~BLK_ZONE_WPLUG_PLUGGED` under `if (blk_queue_zoned_qd1_writes(disk->queue))` inside `disk_zone_wplug_abort()` itself would achieve the same invariant with a smaller diff while also protecting future callers of `disk_zone_wplug_abort()`. **Dependency**: relies on Patches 07, 08, and 09.
3. **Patch 11/98** ([11/98](https://lore.kernel.org/linux-block/20261028164415.1262013-12-axboe@kernel.dk/) — `block: check dma alignment of bvec iters in async dio`):
   - Valid fix (`iov_iter_alignment()` checked length alignment on `ITER_BVEC` but missed `bv_offset` address alignment). Note a pre-existing quirk in `bio_iov_bvec_aligned()`: `bvec_iter_is_aligned()` short-circuits to `true` when `!IS_ENABLED(CONFIG_DEBUG_KERNEL)` (`/* assume all kernel users know what they are doing */`), even though `IORING_OP_READV_FIXED` / `WRITEV_FIXED` creates `ITER_BVEC` slices from arbitrary user-supplied offsets inside registered buffers.
4. **Patches 17/98 & 39/98** ([17/98](https://lore.kernel.org/linux-block/20261028164415.1262013-18-axboe@kernel.dk/) & [39/98](https://lore.kernel.org/linux-block/20261028164415.1262013-40-axboe@kernel.dk/) — `rnbd-srv` `sess_dev` kobject teardown):
   - Valid fixes for `xa_lock` under `srv_sess->lock` sleep-in-atomic/UAF and double-`CLOSE` UAF. Worth noting that if `CONFIG_DEBUG_KOBJECT_RELEASE=y` is enabled, `kobject_put(&sess_dev->kobj)` delays `.release` (`xa_erase(&srv_sess->index_idr, ...)`) by up to 1–4 seconds after `srv_sess->lock` is dropped; moving `xa_erase()` out of `rnbd_srv_dev_release()` into explicit teardown (once final ref drops or at `CLOSE`) would decouple XArray visibility from `CONFIG_DEBUG_KOBJECT_RELEASE`.
5. **Patch 21/98** ([21/98](https://lore.kernel.org/linux-block/20261028164415.1262013-22-axboe@kernel.dk/) — `mtip32xx: abort timed out commands of a removed device`):
   - Valid fix; note `mtip_timeout_function()` returns `BLK_EH_RESET_TIMER` after calling `mtip_abort_cmd()` -> `blk_mq_complete_request()`, which works safely because `blk_mq_complete_request_remote()` clears `REQ_ATOM_STARTED`, though `BLK_EH_DONE` when all commands are aborted is cleaner.
6. **Patch 23/98** ([23/98](https://lore.kernel.org/linux-block/20261028164415.1262013-24-axboe@kernel.dk/) — `blk-iolatency: flush enable_work after going offline`):
   - Valid fix for `enable_work` UAF. Using `disable_work_sync(&blkiolat->enable_work)` *before* `blkcg_deactivate_policy()` (instead of `cancel_work_sync()` after it) would also prevent re-arming while avoiding a theoretical lockdep cycle (`disable_work_sync` -> `freeze_lock` vs `freeze_lock` -> `blkcg_mutex`) on `!GD_OWNS_QUEUE` drivers.
7. **Patch 29/98** ([29/98](https://lore.kernel.org/linux-block/20261028164415.1262013-30-axboe@kernel.dk/) — `blk-mq: keep sharing tags fairly after a queue update`):
   - Valid fix for adding queues when `BLK_MQ_F_TAG_QUEUE_SHARED` is set. Asymmetric edge case: if a tagset transitions `2 -> 1 -> 2` queues while shrinking/growing `nr_hw_queues`, an `hctx` recycled from `q->unused_hctx_list` retains `BLK_MQ_F_TAG_QUEUE_SHARED` during the `1`-queue phase because [`blk_mq_update_tag_set_shared()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/block/blk-mq.c#L4312-L4330) doesn't touch `unused_hctx_list`. Calling `queue_set_hctx_shared(q, set->flags & BLK_MQ_F_TAG_QUEUE_SHARED)` unconditionally (instead of under `if (set->flags & BLK_MQ_F_TAG_QUEUE_SHARED)`) handles both directions cleanly in 1 line.
8. **Patch 49/98** ([49/98](https://lore.kernel.org/linux-block/20261028164415.1262013-50-axboe@kernel.dk/) — `nvmet-tcp: ask for the TLS record type on every receive`):
   - Valid fix (`put_cmsg()` overwrites `msg_controllen = 17` on the first receive, causing subsequent `recvmsg()` calls on the persistent `cmd->recv_msg` to hit `MSG_CTRUNC` with stale `cbuf`). Leaves a now-redundant `if (queue->state == NVMET_TCP_Q_TLS_HANDSHAKE)` `msg_control` setup block in [`nvmet_tcp_alloc_cmd()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/tcp.c#L1492-L1525) (`tcp.c:1508-1511`) that can be deleted.
9. **Patch 53/98** ([53/98](https://lore.kernel.org/linux-block/20261028164415.1262013-54-axboe@kernel.dk/) — `nvme-tcp: don't get stuck on a TLS control record`):
   - Valid fix for kTLS `-EIO` soft-lockup on post-handshake `NewSessionTicket` / `Alert`. Minor cleanups in [`nvme_tcp_recv_pdu()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/host/tcp.c#L886-L915): `char cbuf[CMSG_SPACE(sizeof(char))]` lacks `__aligned(__alignof__(struct cmsghdr))` before casting to `(struct cmsghdr *)cbuf`, and `tls_alert_recv()` is called before checking `ret >= 2` (same 1-byte alert caveat as Patch 48).
10. **Patch 55/98** ([55/98](https://lore.kernel.org/linux-block/20261028164415.1262013-56-axboe@kernel.dk/) — `nvme-tcp: don't act on PDUs for a command being sent`):
    - Great catch for malicious target early C2HData/R2T/CapsuleResp race while the host is still sending the command! One KCSAN buglet: [`nvme_tcp_try_send()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/host/tcp.c#L1371-L1405) writes `queue->request = nvme_tcp_fetch_request(queue);`, performing a redundant **plain unmarked store** to `queue->request` right *after* `nvme_tcp_fetch_request()` already published it via `smp_store_release(&queue->request, req)`! Change `nvme_tcp_fetch_request()` to `void` (or assign only on `NULL`) so `smp_load_acquire(&queue->request)` on the RX CPU only ever pairs with `smp_store_release()`.
11. **Patch 61/98** ([61/98](https://lore.kernel.org/linux-block/20261028164415.1262013-62-axboe@kernel.dk/) — `nvmet-auth: authenticate the I/O queues again`):
    - Code is completely right (critical auth-bypass fix after commit `be86843fd640`). Commit message nit: the sentence *"On the admin queue, Property Set runs before the check and lets the host enable or shut down the controller"* describes an older bug already fixed upstream by commit `44512543ff04 ("nvmet: move fabrics command authentication check before command execution")`.
12. **Patch 71/98** ([71/98](https://lore.kernel.org/linux-block/20261028164415.1262013-72-axboe@kernel.dk/) — `nvmet: check the host NQN against the TLS PSK identity`):
    - Valid fix preventing PSK host-NQN spoofing across hosts. Note that deliberately skipping `<subsysnqn>` comparison in `NVMe0R01 <hostnqn> <subsysnqn>` allows a host with a valid PSK for subsystem `A` on a multi-subsystem TLS port to connect to subsystem `B` if `B` also permits `<hostnqn>` (acceptable for Discovery, worth documenting or restricting to `nvmet_is_disc_subsys()`).
13. **Patch 78/98** ([78/98](https://lore.kernel.org/linux-block/20261028164415.1262013-79-axboe@kernel.dk/) — `nvmet-tcp: don't read a removed port at Connect time`):
    - Valid UAF fix (`nvmet_tcp_install_queue` reading `queue->port->priv` after `nvmet_tcp_remove_port` freed `port`). Note a sibling UAF exists in [`nvmet_tcp_disc_port_addr()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/tcp.c#L2227-L2240) (`struct nvmet_tcp_port *port = nport->priv;` on Discovery Log Page reads racing with `nvmet_tcp_remove_port()`), and `queue->port` in `struct nvmet_tcp_queue` now has zero remaining readers and can be removed.
14. **Patch 84/98** ([84/98](https://lore.kernel.org/linux-block/20261028164415.1262013-85-axboe@kernel.dk/) — `nvmet-rdma: disconnect only established queues`):
    - Valid `nvmet-rdma` fix (and prerequisite for Patch 86). Commit message nit: when a silent MPAv2 peer-to-peer initiator triggers the path over `siw`, `siw` itself still hits a separate driver `BUG()` in `cm_close_handler()` (`drivers/infiniband/sw/siw/iwcm.c:966`) when `ib_drain_qp()` transitions the QP out of `SIW_EPSTATE_RDMA_MODE` before `IW_CM_EVENT_ESTABLISHED`.

---

## 3. Intra-Series Dependencies (Important for Partial Cherry-Picking / Backports)

If subsets of this 98-patch series are applied selectively or backported to stable trees, keep these dependencies together:

- **`blk-zoned` / `blk-mq` (`07 -> 08 -> 09 -> 10`)**: Patch 10 relies on Patches 07, 08, and 09 so failed/keyslot-aborted zoned writes clean up `zwplug` without leaking references or unplugging active zones.
- **`blk-mq` `tag_list_lock` (`36 -> 37 -> 38`)**: Patches 36, 37, and 38 stack directly on `blk_mq_add_queue_tag_set()` / `blk_mq_del_queue_tag_set()` / `blk_mq_init_allocated_queue()`.
- **`rnbd-srv` (`16 -> 17 -> 39 -> 40`)**: Sequential fixes to `sess_dev` lifecycle and `srv_sess->index_idr` publication/removal.
- **`mtip32xx` (`20 -> 21 -> 22`)**: Stack on `dd->sr` and `mtip_service_thread()`.
- **`nvme-tcp` (`52 -> 74`, `53 -> 54`, `55 -> 56`)**: Patch 74 modifies `nvme_tcp_free_queue()` touched by Patch 52; Patch 54 extends the TLS recv switch added in Patch 53; Patch 56 uses `nvme_tcp_Req_cmd_sent` concepts adjacent to Patch 55.
- **`nvmet` fatal error (`69 + 96`)**: **Patch 96 fixes a UAF/NULL-deref introduced by Patch 69** (`nvmet_sq_fatal_error()` reading `sq->ctrl` without `sq->ref`). **Squash Patch 96 into Patch 69** before merging so commits `69..95` are bisection-safe!
- **`nvmet` PR (`81 -> 82 -> 94`)**: Stack on `nvmet_pr_preempt_and_abort()`.
- **`nvmet-rdma` (`83 -> 84 -> 85 -> 86 -> 87 -> 88 -> 89 -> 90 -> 91 -> 92 -> 95 -> 97`)**:
  - Patch 86 **semantically requires** Patch 84 (calling `rdma_disconnect()` under `nvmet_rdma_queue_mutex` on an unestablished `siw` `cm_id` would synchronously re-enter `nvmet_rdma_cm_handler()` on the same thread and self-deadlock on `nvmet_rdma_queue_mutex` without Patch 84!).
  - Patch 92 builds directly on Patch 91's `nvmet_rdma_process_wr_wait_list(queue, rsp)` signature.

---

## 4. `scripts/checkpatch.pl` Audit

Running `scripts/checkpatch.pl` across all 98 patches produced **zero C coding-style errors** and only commit-citation format complaints (`GIT_COMMIT_ID`: 12-char inline commit SHA in commit message body without `("commit title")`) on **15 patches**:

- Patches `01, 02, 07, 10, 25, 34, 36, 41, 43, 61, 62, 85, 87, 88, 98`.

All 98 `Fixes:` tags have valid 12-char SHAs and exact matching commit subjects in `origin/master`.

---

## 5. Complete 98-Patch Subsystem Review Table

### 5.1 Block Cgroup, QoS (`iocost`, `iolatency`, `blk-throttle`)
| # | Subject | Author | Verdict | Technical Summary |
| :-: | :--- | :--- | :-: | :--- |
| [01](https://lore.kernel.org/linux-block/20261028164415.1262013-2-axboe@kernel.dk/) | `blk-cgroup: keep the old blkg while reassociating a bio` | Josef Bacik | **LGTM** | Fixes `blkg` UAF in `blk_throtl_assoc_bio()` when `bio_associate_blkg_from_css()` drops the old `bi_blkg` ref before `.bio_associate` runs. |
| [02](https://lore.kernel.org/linux-block/20261028164415.1262013-3-axboe@kernel.dk/) | `blk-iocost: free struct ioc after a grace period` | Josef Bacik | **LGTM** | Fixes `struct ioc` UAF by RCU-freeing `ioc` (`kfree_rcu`) for RCU readers in `ioc_Comm`/`blkg` paths. |
| [03](https://lore.kernel.org/linux-block/20261028164415.1262013-4-axboe@kernel.dk/) | `blk-throttle: don't truncate the cgroup v1 iops limits` | Josef Bacik | **LGTM** | Fixes `unsigned int` truncation of `u64` cgroup v1 IOPS limits (`U64_MAX` truncated to `UINT_MAX`). |
| [13](https://lore.kernel.org/linux-block/20261028164415.1262013-14-axboe@kernel.dk/) | `blk-cgroup: fix root io.stat byte counts on 32-bit` | Josef Bacik | **LGTM** | Casts `sector_t` / `unsigned long` sector count to `u64` before `<< SECTOR_SHIFT` in `blkcg_print_one_stat()`. |
| [19](https://lore.kernel.org/linux-block/20261028164415.1262013-20-axboe@kernel.dk/) | `blk-cgroup: don't free a blkg with its punt work queued` | Josef Bacik | **LGTM** | Flushes `async_bio_work` before freeing `blkg` in `blkg_free_workfn()` instead of freeing with queued work. |
| [23](https://lore.kernel.org/linux-block/20261028164415.1262013-24-axboe@kernel.dk/) | `blk-iolatency: flush enable_work after going offline` | Josef Bacik | **MINOR** | Fixes `enable_work` UAF on teardown; `disable_work_sync()` before `blkcg_deactivate_policy()` is slightly cleaner (see §2.6). |
| [25](https://lore.kernel.org/linux-block/20261028164415.1262013-26-axboe@kernel.dk/) | `blk-throttle: pass the I/O counts of a dead group on` | Josef Bacik | **LGTM** | Propagates local `stat_bytes`/`stat_ios` to parent `tg` on `throtl_pd_offline()` before freeing child `tg`. |
| [26](https://lore.kernel.org/linux-block/20261028164415.1262013-27-axboe@kernel.dk/) | `blk-throttle: make blkio.reset_stats reset I/O counts` | Josef Bacik | **LGTM** | Resets `tg`'s per-CPU `stat_bytes`/`stat_ios` counters alongside `blkg` stats on `blkio.reset_stats`. |
| [28](https://lore.kernel.org/linux-block/20261028164415.1262013-29-axboe@kernel.dk/) | `blk-iocost: check again that the disk is alive` | Josef Bacik | **NEEDS WORK** | Valid `del_gendisk()` race, **but `goto close_bdev` puts `ctx.bdev` (freeing `q`) before `blk_mq_unfreeze_queue(q)`** (see §1.1). |
| [34](https://lore.kernel.org/linux-block/20261028164415.1262013-35-axboe@kernel.dk/) | `block: don't enter reclaim under rq_qos_mutex` | Josef Bacik | **LGTM** | Wraps `rq_qos_mutex` scopes in `memalloc_noio_save()` where allocations run under `rq_qos_mutex`, fixing `rq_qos_mutex -> fs_reclaim` lockdep deadlock. |

### 5.2 Block Core, Zoned, Integrity, Partitions, Blktrace & `blk-mq`
| # | Subject | Author | Verdict | Technical Summary |
| :-: | :--- | :--- | :-: | :--- |
| [04](https://lore.kernel.org/linux-block/20261028164415.1262013-5-axboe@kernel.dk/) | `block: fix use-after-free of the disk in sync_bdevs()` | Josef Bacik | **LGTM** | Holds an extra `bdev`/`inode` reference across dropping `blockdev_superblock->s_inode_list_lock` so `bdev->bd_disk` isn't freed before `put_disk()`. |
| [05](https://lore.kernel.org/linux-block/20261028164415.1262013-6-axboe@kernel.dk/) | `block: don't lose the status of a failed flush request` | Josef Bacik | **LGTM** | Preserves non-zero `error` in `blk_flush_complete_seq()` instead of overwriting with `BLK_STS_OK` on `REQ_FSEQ_DATA` completion. |
| [06](https://lore.kernel.org/linux-block/20261028164415.1262013-7-axboe@kernel.dk/) | `blk-mq: don't batch allocate tags from an inactive hctx` | Josef Bacik | **MINOR** | Checks `blk_mq_hctx_stopped(hctx)` in `blk_mq_get_driver_tag()` batch path; `Fixes:` SHA could be `349302da8352` (see §2.1). |
| [07](https://lore.kernel.org/linux-block/20261028164415.1262013-8-axboe@kernel.dk/) | `block: keep zone write plugs with plugged BIOs alive` | Josef Bacik | **LGTM** | Keeps `zwplug` reference elevated while `bio_list` is non-empty so `disk_put_zone_wplug()` can't free an active plug. |
| [08](https://lore.kernel.org/linux-block/20261028164415.1262013-9-axboe@kernel.dk/) | `block: keep the zone write plug of a failed write` | Josef Bacik | **LGTM** | Prevents double-put / premature free of `zwplug` when a zoned write fails before reaching normal completion. |
| [09](https://lore.kernel.org/linux-block/20261028164415.1262013-10-axboe@kernel.dk/) | `blk-mq: end the request if getting a keyslot fails` | Josef Bacik | **LGTM** | Calls `blk_mq_end_request()` instead of `bio_endio()` when `blk_crypto_rq_get_keyslot()` fails so `rq` & zoned state aren't leaked. |
| [10](https://lore.kernel.org/linux-block/20261028164415.1262013-11-axboe@kernel.dk/) | `block: don't unplug a zone under a failed write` | Josef Bacik | **MINOR** | Avoids clearing `BLK_ZONE_WPLUG_PLUGGED` while another write is still in flight on `qd > 1` zoned queues (see §2.2). |
| [11](https://lore.kernel.org/linux-block/20261028164415.1262013-12-axboe@kernel.dk/) | `block: check dma alignment of bvec iters in async dio` | Josef Bacik | **MINOR** | Adds `bio_iov_bvec_aligned()` check in `blkdev_bio_iter_aligned()` for async DIO `ITER_BVEC` (see §2.3). |
| [27](https://lore.kernel.org/linux-block/20261028164415.1262013-28-axboe@kernel.dk/) | `blk-mq: don't trust the cookie in blk_mq_poll()` | Josef Bacik | **LGTM** | Bounds-checks `qc` queue index against `q->nr_hw_queues` and NULL/poll state in `blk_mq_poll()`, fixing OOB crash after `nr_hw_queues` shrink. |
| [29](https://lore.kernel.org/linux-block/20261028164415.1262013-30-axboe@kernel.dk/) | `blk-mq: keep sharing tags fairly after a queue update` | Josef Bacik | **MINOR** | Restores `BLK_MQ_F_TAG_QUEUE_SHARED` on reallocated `hctx`s; unconditional `queue_set_hctx_shared()` also covers `unused_hctx_list` reuse (see §2.7). |
| [30](https://lore.kernel.org/linux-block/20261028164415.1262013-31-axboe@kernel.dk/) | `blk-mq: don't bring back flags of a completed request` | Josef Bacik | **LGTM** | Fixes non-atomic RMW on `rq->rq_flags` racing with remote request completion (`RQF_MQ_INFLIGHT` clobber / accounting leak). |
| [31](https://lore.kernel.org/linux-block/20261028164415.1262013-32-axboe@kernel.dk/) | `blk-mq: don't forget a timer the driver has restarted` | Josef Bacik | **LGTM** | Re-arms `q->timeout` if a timeout handler returns `BLK_EH_RESET_TIMER` (or `blk_add_timer` runs concurrently) instead of stranding requests forever. |
| [32](https://lore.kernel.org/linux-block/20261028164415.1262013-33-axboe@kernel.dk/) | `blk-mq-dma: initialize the dma state for P2P, too` | Josef Bacik | **LGTM** | Zero-initializes `iter->iod` fields on the PCI P2PDMA mapping path in `blk_rq_dma_map_iova()`. |
| [33](https://lore.kernel.org/linux-block/20261028164415.1262013-34-axboe@kernel.dk/) | `blk-mq-dma: map the metadata a passthrough request has` | Josef Bacik | **LGTM** | Checks `blk_rq_integrity_map_user` / `rq->nr_integrity_segments` rather than `BLK_INTEGRITY_DEVICE_CAPABLE` so passthrough metadata is mapped. |
| [35](https://lore.kernel.org/linux-block/20261028164415.1262013-36-axboe@kernel.dk/) | `blktrace: remove the debugfs files before the channel` | Josef Bacik | **LGTM** | Calls `debugfs_remove()` on `msg_file`/`dropped_file` before `relay_close(bt->rchan)` to prevent debugfs UAF on `bt->rchan`. |
| [36](https://lore.kernel.org/linux-block/20261028164415.1262013-37-axboe@kernel.dk/) | `blk-mq: quiesce a queue that joins a quiesced tag set` | Josef Bacik | **LGTM** | Tracks `set->quiesce_depth` under `set->quiesce_lock` so newly added queues inherit tagset quiesce state (e.g., `nvme_scan_work` vs `nvme_reset_work`). |
| [37](https://lore.kernel.org/linux-block/20261028164415.1262013-38-axboe@kernel.dk/) | `blk-mq: exit the hctxs of a queue under tag_list_lock` | Josef Bacik | **LGTM** | Serializes `blk_mq_exit_hw_queues()` + `blk_mq_del_queue_tag_set()` under `set->tag_list_lock` against `blk_mq_update_nr_hw_queues()` (`hctx->fq` NULL deref). |
| [38](https://lore.kernel.org/linux-block/20261028164415.1262013-39-axboe@kernel.dk/) | `blk-mq: set up a queue under tag_list_lock` | Josef Bacik | **LGTM** | Holds `set->tag_list_lock` (with `memalloc_noio_save()`) across `blk_mq_init_allocated_queue()`, preventing `set->tags` UAF during `nr_hw_queues` updates. |
| [42](https://lore.kernel.org/linux-block/20261028164415.1262013-43-axboe@kernel.dk/) | `block: partitions/mac: compute the sectors in 64 bits` | Josef Bacik | **LGTM** | Widens `secsize / 512` multiplication to `u64`/`sector_t` in Mac partition parser to prevent 32-bit overflow on >2 TiB disks. |
| [43](https://lore.kernel.org/linux-block/20261028164415.1262013-44-axboe@kernel.dk/) | `blk-mq: free the bitmaps of tags in the SRCU callback` | Josef Bacik | **LGTM** | Moves `sbitmap_queue_free()` into `blk_mq_free_tags_callback()`, fixing `sb->map` NULL/UAF in concurrent `blk_mq_tagset_busy_iter()` SRCU readers. |
| [44](https://lore.kernel.org/linux-block/20261028164415.1262013-45-axboe@kernel.dk/) | `block: apply the segment size to the integrity buffer` | Josef Bacik | **LGTM** | Enforces `queue_max_segment_size(q)` when coalescing contiguous integrity `bio_vec`s in `blk_rq_map_integrity_sg()` / DMA iterators. |

### 5.3 Block Drivers (`drbd`, `zloop`, `rnbd-srv`, `aoe`, `mtip32xx`, `zram`)
| # | Subject | Author | Verdict | Technical Summary |
| :-: | :--- | :--- | :-: | :--- |
| [12](https://lore.kernel.org/linux-block/20261028164415.1262013-13-axboe@kernel.dk/) | `drbd: stop the start_resync timer of a dying device` | Josef Bacik | **LGTM** | Calls `timer_delete_sync(&device->start_resync_timer)` in `drbd_destroy_device()` before freeing `device`. |
| [14](https://lore.kernel.org/linux-block/20261028164415.1262013-15-axboe@kernel.dk/) | `drbd: take req_lock to skip the initial sync as peer` | Josef Bacik | **LGTM** | Acquires `resource->req_lock` (`spin_lock_irq`) around `__change_cstate()` in `receive_uuids()`. |
| [15](https://lore.kernel.org/linux-block/20261028164415.1262013-16-axboe@kernel.dk/) | `zloop: refuse zone files without read/write_iter` | Josef Bacik | **LGTM** | Rejects backing zone files missing `f_op->read_iter` / `write_iter` on open instead of NULL-dereferencing in I/O worker. |
| [16](https://lore.kernel.org/linux-block/20261028164415.1262013-17-axboe@kernel.dk/) | `block/rnbd-srv: don't free the device on a sysfs error` | Josef Bacik | **LGTM** | Fixes double-free of `sess_dev` when `rnbd_srv_create_dev_session_sysfs()` fails after `kobject_init_and_add()`. |
| [17](https://lore.kernel.org/linux-block/20261028164415.1262013-18-axboe@kernel.dk/) | `block/rnbd-srv: drop the sess_dev kobject unlocked` | Josef Bacik | **MINOR** | Drops `sess_dev->kobj` outside `xa_lock` / `srv_sess->lock` to avoid sleeping in atomic (see `CONFIG_DEBUG_KOBJECT_RELEASE` note in §2.4). |
| [18](https://lore.kernel.org/linux-block/20261028164415.1262013-19-axboe@kernel.dk/) | `aoe: don't free a device with its work queued` | Josef Bacik | **NEEDS WORK** | Cancels `d->work`, **but sets `DEVFL_TKILL` *after* `aoedev_downdev(d)` instead of before dropping `d->lock`** (see §1.7). |
| [20](https://lore.kernel.org/linux-block/20261028164415.1262013-21-axboe@kernel.dk/) | `mtip32xx: don't mark the disk dead in the irq handler` | Josef Bacik | **NEEDS WORK** | Defers `blk_mark_disk_dead()` out of hardirq to `mtip_service_thread()`, **but calling it before aborting NCQ commands can deadlock on `s_umount`** (see §1.6). |
| [21](https://lore.kernel.org/linux-block/20261028164415.1262013-22-axboe@kernel.dk/) | `mtip32xx: abort timed out commands of a removed device` | Josef Bacik | **MINOR** | Aborts timed-out commands directly when `mtip_device_removed(dd)` is true (see §2.5). |
| [22](https://lore.kernel.org/linux-block/20261028164415.1262013-23-axboe@kernel.dk/) | `mtip32xx: abort commands in flight on surprise removal` | Josef Bacik | **NEEDS WORK** | Aborts commands before `del_gendisk()`, **but `mtip_stop_service_thread()` before `mtip_abort_cmd` can deadlock with Patch 20, and misses `sr` during `del_gendisk()`** (see §1.6). |
| [24](https://lore.kernel.org/linux-block/20261028164415.1262013-25-axboe@kernel.dk/) | `zram: accept lz4 dictionaries of more than 64 KiB` | Josef Bacik | **LGTM** | Clamps LZ4 dictionary view to the last 64 KiB (matching LZ4 spec) instead of failing stream init on `dict_size > 64 KiB`. |
| [39](https://lore.kernel.org/linux-block/20261028164415.1262013-40-axboe@kernel.dk/) | `rnbd-srv: fix use-after-free on two CLOSE messages` | Chris Mason | **MINOR** | Serializes `CLOSE` with `open_write_lock` / `sess_dev->closed` flag to prevent double `rnbd_srv_destroy_dev_session_sysfs()` (see §2.4). |
| [40](https://lore.kernel.org/linux-block/20261028164415.1262013-41-axboe@kernel.dk/) | `rnbd-srv: publish the sess_dev at the end of OPEN` | Chris Mason | **LGTM** | Inserts `sess_dev` into `srv_sess->index_idr` only after full initialization so early I/O/CLOSE messages cannot see half-built `sess_dev`. |
| [41](https://lore.kernel.org/linux-block/20261028164415.1262013-42-axboe@kernel.dk/) | `zram: check a bio's range against the device size` | Chris Mason | **NEEDS WORK** | **Allows empty bios (`bi_io_vec` NULL deref) and sub-4K bios on 4K-page kernels (4 KiB page buffer overrun / kernel memory infoleak)** (see §1.2). |

### 5.4 NVMe Host (`core`, `fabrics`, `loop`, `tcp`)
| # | Subject | Author | Verdict | Technical Summary |
| :-: | :--- | :--- | :-: | :--- |
| [46](https://lore.kernel.org/linux-block/20261028164415.1262013-47-axboe@kernel.dk/) | `nvme-loop: delete the controller if a reset fails` | Josef Bacik | **LGTM** | Calls `nvme_delete_ctrl(&ctrl->ctrl)` on `nvme_loop_reset_ctrl_work()` failure (matching rdma/tcp/fc) instead of stranding a dead controller. |
| [47](https://lore.kernel.org/linux-block/20261028164415.1262013-48-axboe@kernel.dk/) | `nvme: refuse metadata sizes above 255 bytes` | Josef Bacik | **LGTM** | Rejects namespaces whose `ms > U8_MAX` before assigning to `u8 tuple_size` (`blk_integrity`), preventing integer wraparound / OOB DMA. |
| [50](https://lore.kernel.org/linux-block/20261028164415.1262013-51-axboe@kernel.dk/) | `nvme-tcp: wait for an in-flight TLS handshake callback` | Josef Bacik | **LGTM** | Uses `tls_handshake_cancel()` + `wait_for_completion()` on timeout so `nvme_tcp_tls_done()` cannot UAF the stack `Handshake` context or `queue`. |
| [52](https://lore.kernel.org/linux-block/20261028164415.1262013-53-axboe@kernel.dk/) | `nvme-tcp: keep the socket while the queue is polled` | Josef Bacik | **LGTM** | Defers `sock_release(queue->sock)` until `nvme_tcp_free_queue()` (after tagset/queue teardown) so `nvme_tcp_poll()` cannot UAF `queue->sock`. |
| [53](https://lore.kernel.org/linux-block/20261028164415.1262013-54-axboe@kernel.dk/) | `nvme-tcp: don't get stuck on a TLS control record` | Josef Bacik | **MINOR** | Consumes non-data TLS records via `sock_recvmsg()` with cmsg when `tcp_read_sock()` returns `-EIO`; minor `cbuf` alignment & `ret < 2` alert notes (see §2.9). |
| [54](https://lore.kernel.org/linux-block/20261028164415.1262013-55-axboe@kernel.dk/) | `nvme-tcp: start error recovery after a TLS KeyUpdate` | Josef Bacik | **LGTM** | Triggers `nvme_tcp_error_recovery()` on `TLS_RECORD_TYPE_HANDSHAKE` (`KeyUpdate` unsupported in kTLS without userspace rekeying). |
| [55](https://lore.kernel.org/linux-block/20261028164415.1262013-56-axboe@kernel.dk/) | `nvme-tcp: don't act on PDUs for a command being sent` | Josef Bacik | **MINOR** | Rejects unsolicited C2H/R2T/CapsuleResp PDUs while `req` is still on the TX path (`UAF`/double-complete fix); minor KCSAN plain store on `queue->request` (see §2.10). |
| [56](https://lore.kernel.org/linux-block/20261028164415.1262013-57-axboe@kernel.dk/) | `nvme-tcp: don't queue the AER while it is being sent` | Josef Bacik | **LGTM** | Tracks AER TX completion before re-queuing `async_event_work` so an immediate CQE cannot re-submit `&ctrl->async_req` while TX still reads it. |
| [57](https://lore.kernel.org/linux-block/20261028164415.1262013-58-axboe@kernel.dk/) | `nvme-tcp: don't send SIGPIPE to whoever submits I/O` | Josef Bacik | **LGTM** | Passes `MSG_NOSIGNAL` on synchronous `nvme_tcp_try_send_*()` `sock_sendmsg()` paths so broken sockets don't deliver `SIGPIPE` to user tasks. |
| [58](https://lore.kernel.org/linux-block/20261028164415.1262013-59-axboe@kernel.dk/) | `nvme-fabrics: reconnect if the association is gone` | Josef Bacik | **LGTM** | Triggers controller reset/reconnect on `NVME_SC_FABRICS_ASSOC_LOST` / invalid association errors instead of failing forever. |
| [74](https://lore.kernel.org/linux-block/20261028164415.1262013-75-axboe@kernel.dk/) | `nvme-tcp: keep the page frag cache with the controller` | Josef Bacik | **LGTM** | Drains `page_frag_cache` in `nvme_tcp_free_queue()` instead of `nvme_tcp_stop_queue()` so TX pages still referenced by skbs aren't freed early. |
| [79](https://lore.kernel.org/linux-block/20261028164415.1262013-80-axboe@kernel.dk/) | `nvme-tcp: start error recovery when a send fails` | Josef Bacik | **LGTM** | Starts `nvme_tcp_error_recovery()` when `nvme_tcp_try_send()` fails with a fatal socket error instead of waiting up to 30s for I/O timeout. |
| [80](https://lore.kernel.org/linux-block/20261028164415.1262013-81-axboe@kernel.dk/) | `nvme-loop: don't BUG() if a scan runs beside a reset` | Josef Bacik | **LGTM** | Returns `BLK_STS_IOERR` in `nvme_loop_queue_rq()` when `queue` isn't `LIVE` instead of `BUG_ON(!test_bit(NVME_LOOP_Q_LIVE, ...))`. |

### 5.5 NVMe Target — Auth, Core, PR & FC (`nvmet`, `nvmet-auth`, `nvmet-fc`)
| # | Subject | Author | Verdict | Technical Summary |
| :-: | :--- | :--- | :-: | :--- |
| [45](https://lore.kernel.org/linux-block/20261028164415.1262013-46-axboe@kernel.dk/) | `nvmet: don't truncate the length of big discards` | Josef Bacik | **LGTM** | Accumulates discard range sector lengths in `u64` / `sector_t` instead of `u32`, fixing 2 TiB truncation on multi-range Dataset Management discards. |
| [59](https://lore.kernel.org/linux-block/20261028164415.1262013-60-axboe@kernel.dk/) | `nvmet-auth: serialize the commands of a controller` | Josef Bacik | **LGTM** | Adds `ctrl->auth_lock` inside `sq->auth_lock` to serialize concurrent `AUTH_Send`/`AUTH_Receive` mutations of shared `ctrl` DH-HMAC-CHAP fields (`UAF` fix). |
| [60](https://lore.kernel.org/linux-block/20261028164415.1262013-61-axboe@kernel.dk/) | `nvmet-auth: keep the negotiated hash per queue` | Josef Bacik | **LGTM** | Moves `shash_id` from `nvmet_ctrl` to `nvmet_sq` so an unauthenticated I/O queue cannot clobber another queue's negotiated HMAC hash. |
| [61](https://lore.kernel.org/linux-block/20261028164415.1262013-62-axboe@kernel.dk/) | `nvmet-auth: authenticate the I/O queues again` | Josef Bacik | **MINOR** | Fixes complete I/O queue DH-HMAC-CHAP authentication bypass introduced in `be86843fd640`; minor commit-message nit on `Property Set` (see §2.11). |
| [62](https://lore.kernel.org/linux-block/20261028164415.1262013-63-axboe@kernel.dk/) | `nvmet: don't mix queues with and without TLS` | Josef Bacik | **LGTM** | Enforces uniform TLS usage across admin and I/O queues of a controller (`!!ctrl->tls_key == !!nvmet_queue_tls_keyid(sq)`), preventing plaintext I/O queue injection. |
| [63](https://lore.kernel.org/linux-block/20261028164415.1262013-64-axboe@kernel.dk/) | `nvmet-auth: refuse a reply without a DH value` | Josef Bacik | **LGTM** | Rejects `DH-HMAC-CHAP_Reply` with `hl == 0` when a DH group was negotiated, preventing downgrade to static-PSK-only authentication. |
| [64](https://lore.kernel.org/linux-block/20261028164415.1262013-65-axboe@kernel.dk/) | `nvmet-auth: keep the old keys if the new ones fail` | Josef Bacik | **LGTM** | Stages updated configfs host/ctrl DHCHAP keys before replacing live `ctrl->host_key`/`ctrl_key` so invalid key strings don't wipe active auth (`!ctrl->host_key` bypass!). |
| [65](https://lore.kernel.org/linux-block/20261028164415.1262013-66-axboe@kernel.dk/) | `nvmet-auth: take only AUTH_Negotiate for a negotiation` | Josef Bacik | **LGTM** | Requires `auth_id == NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE` when `dhchap_step == NEGOTIATE`, closing state-machine skip. |
| [66](https://lore.kernel.org/linux-block/20261028164415.1262013-67-axboe@kernel.dk/) | `nvmet-auth: free the old data on a new AUTH_Negotiate` | Josef Bacik | **LGTM** | Frees prior `sq->dhchap_s1/s2/c1/c2` buffers on repeated `AUTH_Negotiate` commands, fixing remote kmemleak. |
| [67](https://lore.kernel.org/linux-block/20261028164415.1262013-68-axboe@kernel.dk/) | `nvmet-auth: fix the check of the offered DH groups` | Josef Bacik | **LGTM** | Validates bounds and exact `dhl` length of `auth_protocol[0].dhchap` DH group list in `AUTH_Negotiate` (OOB read fix). |
| [68](https://lore.kernel.org/linux-block/20261028164415.1262013-69-axboe@kernel.dk/) | `nvmet: accept I/O queues on the controller's port only` | Josef Bacik | **LGTM** | Verifies `req->port == ctrl->port` on I/O queue `Connect` so an I/O queue on an unencrypted/unrestricted port cannot attach to a restricted port's `ctrl`. |
| [69](https://lore.kernel.org/linux-block/20261028164415.1262013-70-axboe@kernel.dk/) | `nvmet: confine fatal errors of unauthenticated queues` | Josef Bacik | **LGTM** | Disconnects only the unauthenticated I/O queue instead of killing the authenticated host's `ctrl` (**must be squashed/paired with Patch 96**, see §3). |
| [71](https://lore.kernel.org/linux-block/20261028164415.1262013-72-axboe@kernel.dk/) | `nvmet: check the host NQN against the TLS PSK identity` | Josef Bacik | **MINOR** | Verifies `Connect` `hostnqn` matches the negotiated TLS PSK identity (`NVMe0R01 <hostnqn> ...`); see cross-subsys note in §2.12. |
| [72](https://lore.kernel.org/linux-block/20261028164415.1262013-73-axboe@kernel.dk/) | `nvmet: fix NULL dereference of PCI endpoint targets` | Josef Bacik | **LGTM** | Guards `req->port` dereferences for `nvmet-pci-epf` controllers where `req->port` / `ctrl->port` is legitimately `NULL` or lacks fabric addr ops. |
| [76](https://lore.kernel.org/linux-block/20261028164415.1262013-77-axboe@kernel.dk/) | `nvmet: fix NULL dereference in debugfs host_traddr` | Josef Bacik | **NEEDS WORK** | Fixes steady-state `ctrl->sqs[0] == NULL`, **but leaves unlocked TOCTOU NULL deref & queue UAF against `nvmet_sq_destroy()`** (see §1.8). |
| [81](https://lore.kernel.org/linux-block/20261028164415.1262013-82-axboe@kernel.dk/) | `nvmet: fix pc_ref use-after-free in preempt and abort` | Chris Mason | **LGTM** | Completes `pc_ref` barrier and calls `percpu_ref_exit()` safely before freeing preempted `nvmet_pr_per_ctrl_ref` entries. |
| [82](https://lore.kernel.org/linux-block/20261028164415.1262013-83-axboe@kernel.dk/) | `nvmet: move preempt and abort work off nvmet-wq` | Chris Mason | **LGTM** | Moves `preempt_and_abort` work off `nvmet_wq` so waiting for I/O / controller teardown on `nvmet_wq` cannot deadlock workqueue workers. |
| [93](https://lore.kernel.org/linux-block/20261028164415.1262013-94-axboe@kernel.dk/) | `nvmet-fc: only send read data for successful commands` | Andrea Parri | **LGTM** | Skips `FCP_IUI_DATA_IN` DMA transfer when a read command failed (`req->cqe->status != 0`), preventing uninitialized kernel memory leak to host. |
| [94](https://lore.kernel.org/linux-block/20261028164415.1262013-95-axboe@kernel.dk/) | `nvmet: wait only for executing I/O in preempt and abort` | Chris Mason | **LGTM** | Drops `pc_ref` before queuing response (or scopes `pc_ref` to command execution) so stalled TX responses don't hang `Preempt and Abort`. |
| [96](https://lore.kernel.org/linux-block/20261028164415.1262013-97-axboe@kernel.dk/) | `nvmet: take sq->ref in nvmet_sq_fatal_error()` | Chris Mason | **LGTM** | Fixes `sq->ctrl` NULL/UAF in Patch 69's `nvmet_sq_fatal_error()` (**squash into Patch 69**; see also `percpu_ref_exit` note in §1.5). |
| [98](https://lore.kernel.org/linux-block/20261028164415.1262013-99-axboe@kernel.dk/) | `nvmet: don't clear sq->ctrl for a Connect with sqsize 0` | Chris Mason | **LGTM** | Checks `sqsize == 0` before checking/mutating `sq->ctrl` or fails cleanly without orphaning an already-installed `sq->ctrl` reference. |

### 5.6 NVMe Target — TCP (`nvmet-tcp`)
| # | Subject | Author | Verdict | Technical Summary |
| :-: | :--- | :--- | :-: | :--- |
| [48](https://lore.kernel.org/linux-block/20261028164415.1262013-49-axboe@kernel.dk/) | `nvmet-tcp: rewind the iterator after non-data records` | Josef Bacik | **NEEDS WORK** | Rewinds `msg_iter`, **but `tls_alert_recv()` casts `ITER_BVEC` `bv_page` to `struct kvec *` in `try_recv_data()` and reads 2 bytes when `len == 1`** (see §1.3). |
| [49](https://lore.kernel.org/linux-block/20261028164415.1262013-50-axboe@kernel.dk/) | `nvmet-tcp: ask for the TLS record type on every receive` | Josef Bacik | **MINOR** | Resets `msg_control` / `msg_controllen` before every `sock_recvmsg()` in `nvmet_tcp_try_recv_data()`; leaves dead init in `nvmet_tcp_alloc_cmd()` (see §2.8). |
| [51](https://lore.kernel.org/linux-block/20261028164415.1262013-52-axboe@kernel.dk/) | `nvmet-tcp: look again after a short read with TLS` | Josef Bacik | **LGTM** | Loops on short kTLS record reads instead of breaking out of `io_work` when decrypted bytes from the next record are already queued (`sk_data_ready` already fired). |
| [70](https://lore.kernel.org/linux-block/20261028164415.1262013-71-axboe@kernel.dk/) | `nvmet-tcp: don't wait for the host in the accept work` | Josef Bacik | **LGTM** | Moves TLS `tls_server_hello_psk()` initiation / handshake kickoff out of synchronous `accept_work` so one stalled client cannot DoS the listener port. |
| [73](https://lore.kernel.org/linux-block/20261028164415.1262013-74-axboe@kernel.dk/) | `nvmet-tcp: stop listening when a port is removed` | Josef Bacik | **LGTM** | Shuts down the listening socket before draining queues in `nvmet_tcp_remove_port()`, preventing new connections from spawning queues after teardown. |
| [75](https://lore.kernel.org/linux-block/20261028164415.1262013-76-axboe@kernel.dk/) | `nvmet-tcp: hold the queue while the TLS timer is armed` | Josef Bacik | **LGTM** | Cancels `tls_handshake_tmo_work` synchronously in `nvmet_tcp_release_queue_work()` before `kfree(queue)` (`UAF` fix). |
| [77](https://lore.kernel.org/linux-block/20261028164415.1262013-78-axboe@kernel.dk/) | `nvmet-tcp: let the accept work come to an end` | Josef Bacik | **LGTM** | Caps per-invocation `kernel_accept()` iterations and reschedules `accept_work` so connection floods cannot starve `cancel_work_sync(&port->accept_work)`. |
| [78](https://lore.kernel.org/linux-block/20261028164415.1262013-79-axboe@kernel.dk/) | `nvmet-tcp: don't read a removed port at Connect time` | Josef Bacik | **MINOR** | Snapshots `pi_enable` onto `nvmet_tcp_queue` at accept time instead of reading freed `queue->port->priv` at Connect; sibling `disc_port_addr` UAF remains (see §2.13). |

### 5.7 NVMe Target — RDMA (`nvmet-rdma`)
| # | Subject | Author | Verdict | Technical Summary |
| :-: | :--- | :--- | :-: | :--- |
| [83](https://lore.kernel.org/linux-block/20261028164415.1262013-84-axboe@kernel.dk/) | `nvmet-rdma: treat a failed connect as a disconnect` | Chris Mason | **LGTM** | Handles `RDMA_CM_EVENT_CONNECT_ERROR` / `UNREACHABLE` / `REJECTED` after `RDMA_CM_EVENT_CONNECT_REQUEST` by disconnecting/releasing the allocated `queue` (leak/hang fix). |
| [84](https://lore.kernel.org/linux-block/20261028164415.1262013-85-axboe@kernel.dk/) | `nvmet-rdma: disconnect only established queues` | Chris Mason | **MINOR** | Calls `rdma_disconnect()` only when the CM connection was accepted/established (`NVMET_RDMA_Q_LIVE` / connecting state); see `siw` note in §2.14. |
| [85](https://lore.kernel.org/linux-block/20261028164415.1262013-86-axboe@kernel.dk/) | `nvmet-rdma: take a device off the list last` | Chris Mason | **LGTM** | Removes `ndev` from `device_list` under `device_list_mutex` properly relative to `kref` teardown, preventing duplicate `ndev` / SRQ collisions during device churn. |
| [86](https://lore.kernel.org/linux-block/20261028164415.1262013-87-axboe@kernel.dk/) | `nvmet-rdma: queue the release work under the mutex` | Chris Mason | **LGTM** | Serializes `list_del_init(&queue->queue_list)` + `queue_work(nvmet_wq, &queue->release_work)` under `nvmet_rdma_queue_mutex` (requires Patch 84). |
| [87](https://lore.kernel.org/linux-block/20261028164415.1262013-88-axboe@kernel.dk/) | `nvmet-rdma: disconnect a queue on device removal` | Chris Mason | **LGTM** | Properly disconnects and flushes queues using `ndev` in `nvmet_rdma_remove_one()`, fixing UAF on `ib_device` / PD / CQ removal. |
| [88](https://lore.kernel.org/linux-block/20261028164415.1262013-89-axboe@kernel.dk/) | `nvmet-rdma: flush nvmet-wq on module exit` | Chris Mason | **LGTM** | Calls `flush_workqueue(nvmet_wq)` in `nvmet_rdma_exit()` so queued `release_work` callbacks finish before module text unmaps. |
| [89](https://lore.kernel.org/linux-block/20261028164415.1262013-90-axboe@kernel.dk/) | `nvmet-rdma: reject queues larger than max_cqe` | Chris Mason | **LGTM** | Validates `recv_queue_size + 2 * send_queue_size + 1 <= max_cqe` (with overflow-safe arithmetic) before `ib_cq_pool_get()`. |
| [90](https://lore.kernel.org/linux-block/20261028164415.1262013-91-axboe@kernel.dk/) | `nvmet-rdma: disable interrupts under rsp_wr_wait_lock` | Chris Mason | **NEEDS WORK** | Fixes `rsp_wr_wait_lock` IRQ deadlock, **but still invokes `nvmet_rdma_execute_command()` -> `req.execute()` (`GFP_KERNEL`, `mutex_lock`, `submit_bio`) in hardirq/softirq** (see §1.4). |
| [91](https://lore.kernel.org/linux-block/20261028164415.1262013-92-axboe@kernel.dk/) | `nvmet-rdma: don't run the wait list recursively` | Chris Mason | **LGTM** | Prevents unbounded kernel stack recursion when synchronous completions inside `nvmet_rdma_execute_command()` re-enter `nvmet_rdma_process_wr_wait_list()`. |
| [92](https://lore.kernel.org/linux-block/20261028164415.1262013-93-axboe@kernel.dk/) | `nvmet-rdma: fix a lost run of the wait list` | Chris Mason | **LGTM** | Adds paired `smp_mb()` / `smp_mb__after_atomic()` + recheck between `sq_wr_avail` updates and `rsp_wr_wait_list` insertion so commands cannot stall forever. |
| [95](https://lore.kernel.org/linux-block/20261028164415.1262013-96-axboe@kernel.dk/) | `nvmet-rdma: fix double puts after a failed RDMA WRITE` | Chris Mason | **LGTM** | Removes erroneous `nvmet_req_uninit(&rsp->req)` in `nvmet_rdma_write_data_done()` error path (`nvmet_req_complete()` already dropped `ns`, `pc_ref`, and `sq->ref`). |
| [97](https://lore.kernel.org/linux-block/20261028164415.1262013-98-axboe@kernel.dk/) | `nvmet-rdma: hold sq->ref to post a T10-PI Read's SEND` | Chris Mason | **NEEDS WORK** | Valid `ib_drain_qp()` race, **but calls `percpu_ref_tryget_live(&sq->ref)` concurrently with / after `percpu_ref_exit()` instead of holding `sq->ref` across `write_cqe`** (see §1.5). |

Re: [PATCHSET 0/98] Various block related fixes

[Jens Axboe]

On 10/28/26 1:22 PM, Bart Van Assche wrote:
> On 10/28/26 9:41 AM, Jens Axboe wrote:
>> Got a giant pile of stuff dumped on me, and went through and did some
>> vetting and, particularly, commit message rewriting to get it to a
>> somewhat comprehensible shape. If you're CC'ed on a patch, please
>> review it. Feel free to queue it up, or reply back why the particular
>> patch is garbage, if it is garbage.
> 
> Hi Jens,
> 
> Reviewing 98 patches at once is too much for a human.

Yeah, no kidding...

> So I asked an LLM to take a look. That LLM identified 9 must fix
> issues and 15 minor issues. Please take a look at the attached report.

And likewise, reading through 60kb of text is also too much. The plan
here is that people that are resonsible for their area/driver/whatever
will grab the patches. Hence I think any kind of review should be
per-patch, eg reply to the patch with the review. I'm not going to even
open a giant text file... I might, however, read a targeted review of
any given patch, but it has to show up as a reply to that patch.

-- 
Jens Axboe

Re: [PATCHSET 0/98] Various block related fixes

[Bart Van Assche]

On 10/28/26 12:29 PM, Jens Axboe wrote:
> And likewise, reading through 60kb of text is also too much.

You don't have to read all the text. Please take at least a look at the
must fix findings. One of the findings is that a patch in this series
includes a new use-after-free bug.

> The plan
> here is that people that are resonsible for their area/driver/whatever
> will grab the patches. Hence I think any kind of review should be
> per-patch, eg reply to the patch with the review. I'm not going to even
> open a giant text file... I might, however, read a targeted review of
> any given patch, but it has to show up as a reply to that patch.

I will convert the must-fix issues into per-patch replies.

Thanks,

Bart.

Re: [PATCHSET 0/98] Various block related fixes

[Jens Axboe]

On 10/28/26 1:47 PM, Bart Van Assche wrote:
> On 10/28/26 12:29 PM, Jens Axboe wrote:
>> And likewise, reading through 60kb of text is also too much.
> 
> You don't have to read all the text. Please take at least a look at the
> must fix findings. One of the findings is that a patch in this series
> includes a new use-after-free bug.

When you get a wall of text, eyes gloss over. There's zero reason for me
to do that work, when tooling can just split it and reply appropriately!
I can see you're starting to send them in reply to the patches, that is
much better.

And as a reminder, these are not my patches to review, it's on all of us
to get these sorted and merged. Hence for the replies, also have your
LLM actually suggest a fixed patch instead. I'm not going to personally
handle all of these, I would however be quite happy to swap a broken
patch for a fixed one. If you do so, feel free to grab attribution for
that patch and just put the original author in a Reported-by tag.

>> The plan
>> here is that people that are resonsible for their area/driver/whatever
>> will grab the patches. Hence I think any kind of review should be
>> per-patch, eg reply to the patch with the review. I'm not going to even
>> open a giant text file... I might, however, read a targeted review of
>> any given patch, but it has to show up as a reply to that patch.
> 
> I will convert the must-fix issues into per-patch replies.

Sounds good, also see above for actual patches.

-- 
Jens Axboe

Re: [PATCHSET 0/98] Various block related fixes

[Bart Van Assche]

On 10/28/26 2:06 PM, Jens Axboe wrote:
> Hence for the replies, also have your LLM actually suggest a fixed patch instead.
Done.

Bart.

Re: [PATCHSET 0/98] Various block related fixes

[Keith Busch]

On Wed, Oct 28, 2026 at 12:22:53PM -0700, Bart Van Assche wrote:
> On 10/28/26 9:41 AM, Jens Axboe wrote:
> > Got a giant pile of stuff dumped on me, and went through and did some
> > vetting and, particularly, commit message rewriting to get it to a
> > somewhat comprehensible shape. If you're CC'ed on a patch, please
> > review it. Feel free to queue it up, or reply back why the particular
> > patch is garbage, if it is garbage.
> 
> Hi Jens,
> 
> Reviewing 98 patches at once is too much for a human. So I asked an LLM
> to take a look. That LLM identified 9 must fix issues and 15 minor
> issues. Please take a look at the attached report.

Thanks, Bart.

Your review flagged a couple patches I had already queued up, so let me
explain why I think it's safe to proceed with them.

> ### 1.4 Patch 90/98 - `nvmet-rdma: disable interrupts under rsp_wr_wait_lock` (Executes `req.execute()` & `mutex_lock()` in HardIRQ/SoftIRQ Context)
> - **Patch**: [90/98](https://lore.kernel.org/linux-block/20261028164415.1262013-91-axboe@kernel.dk/) (Chris Mason) - [`nvmet_rdma_process_wr_wait_list()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/rdma.c#L514-L537) in [rdma.c](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/rdma.c#L514-L537)
> - **Root-Cause Diagnosis**: Valid. `nvmet_bio_done()` -> `nvmet_req_complete()` -> [`nvmet_rdma_queue_response()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/rdma.c#L717-L752) can run in hardirq or softirq context; when `ib_post_send()` fails, `nvmet_rdma_release_rsp()` calls `nvmet_rdma_process_wr_wait_list()`, which previously took `rsp_wr_wait_lock` with plain `spin_lock()`.
> - **Why the Fix Is Incomplete**: As the commit message itself notes, `nvmet_rdma_process_wr_wait_list()` drops `rsp_wr_wait_lock` (restoring the caller's hardirq/softirq state) and calls `nvmet_rdma_execute_command(rsp)` -> `rsp->req.execute(&rsp->req)` **directly in hardirq/softirq context**! Almost every `nvmet` `req.execute` handler assumes process context:
>   - `nvmet_bdev_execute_rw()` calls `bio_alloc(..., GFP_KERNEL)` and `submit_bio()` -> `submit_bio_noacct()`, which hits `might_sleep()` and manipulates `current->bio_list` without interrupt protection (corrupting any interrupted task's `current->bio_list`!).
>   - Admin/fabrics/auth/PR handlers (`nvmet_execute_identify_*`, `nvmet_execute_admin_connect`, `nvmet_execute_auth_*`, `nvmet_execute_pr_*`) call `kzalloc(..., GFP_KERNEL)`, `mutex_lock()`, and `down_read()`.
>   - Furthermore, if **Patch 69** is applied and `ib_post_send()` fails on a dying queue in hardirq/softirq context (`nvmet_rdma_queue_response()` doesn't currently call `nvmet_rdma_error_comp()`, though `nvmet_rdma_Disconnect` paths take `mutex_lock(&nvmet_rdma_queue_mutex)`), any fatal error path assuming process context will sleep in atomic.
> - **Suggested Fix**: Offload `nvmet_rdma_process_wr_wait_list()` to a `work_struct` on `nvmet_wq` (or queue the work whenever `!in_task()`) instead of calling `nvmet_rdma_execute_command()` in interrupt context.

I agree the wait list shouldn't run from the completion context, but
that issue predates this patch which only fixes the lock inversion that
lockdep found. Patch seems a safe improvement to proceed with.

> ### 1.8 Patch 76/98 - `nvmet: fix NULL dereference in debugfs host_traddr` (Unlocked TOCTOU NULL-Deref & UAF Remains)
> - **Patch**: [76/98](https://lore.kernel.org/linux-block/20261028164415.1262013-77-axboe@kernel.dk/) (Josef Bacik) - [`nvmet_ctrl_host_traddr()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/core.c#L1822-L1830) in [core.c](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/core.c#L1822-L1830)
> - **Root-Cause Diagnosis**: Valid. When the admin queue (`ctrl->sqs[0]`) disconnects while I/O queues remain open, `nvmet_sq_destroy()` sets `ctrl->sqs[0] = NULL`, so reading `host_traddr` in debugfs crashes with a near-NULL dereference (`container_of(ctrl->sqs[0], ...)`).
> - **Incomplete Synchronization**: As the commit message candidly admits (*"Nothing holds the queue for debugfs, so a reader that's past the check when the queue goes away is still on its own"*), `if (!ctrl->sqs[0])` is a plain unlocked load, and `nvmet_tcp_host_port_addr()` / `nvmet_rdma_host_port_addr()` / `nvmet_fc_host_traddr()` immediately perform a second unlocked load of `ctrl->sqs[0]` and dereference the transport queue (`queue->sockaddr_peer`, `nvme_rdma_cm_id`, `assoc->hostport`) concurrently with `nvmet_sq_destroy()` freeing the queue (`kfree(queue)` -> **TOCTOU NULL deref and KASAN UAF**).
> - **Suggested Fix**: Snapshot the formatted `host_traddr` string into a small `char host_traddr[NVMF_TRADDR_SIZE]` buffer in `struct nvmet_ctrl` during admin queue `Connect` (`nvmet_alloc_ctrl()` / `install_queue`), and have `nvmet_ctrl_host_traddr()` copy that immutable buffer. That eliminates all transport queue lifetime races with ~4 lines of code.

Yeah, this fixes the deterministic crash, but a reader can still get
past the check and race the teardown into the transport callback. I
don't think a snapshot is needed though: the controller is being torn
down, so showing a stale address isn't useful, and returning an error
is fine.

Clearing ctrl->sqs[qid] under ctrl->lock in nvmet_sq_destroy() and
holding it across the check and the ->host_traddr() call should close
the race. Guixin is working on the same race for the tls_key attribute,
so it'd make sense to fix both the same way.

Re: (subset) [PATCHSET 0/98] Various block related fixes

[Jens Axboe]

On Wed, 28 Oct 2026 10:41:03 -0600, Jens Axboe wrote:
> Got a giant pile of stuff dumped on me, and went through and did some
> vetting and, particularly, commit message rewriting to get it to a
> somewhat comprehensible shape. If you're CC'ed on a patch, please
> review it. Feel free to queue it up, or reply back why the particular
> patch is garbage, if it is garbage.
> 
> Thanks!
> 
> [...]

Applied, thanks!

[01/98] blk-cgroup: keep the old blkg while reassociating a bio
        commit: 0e1daaf9b20df2125452846f768a68d8ea280adf
[02/98] blk-iocost: free struct ioc after a grace period
        commit: 05ce1df3756773862c056ca8698f8ec4cdd7fe98
[03/98] blk-throttle: don't truncate the cgroup v1 iops limits
        commit: 7d96874ba80ba7a2ab578ea9139c2af79c353b88
[13/98] blk-cgroup: fix root io.stat byte counts on 32-bit
        commit: e335b853d919f2c6246ca0993115285742a2acc1
[19/98] blk-cgroup: don't free a blkg with its punt work queued
        commit: 4def096314cf29fb6940b74c087a0b923185580e
[23/98] blk-iolatency: flush enable_work after going offline
        commit: 020f61242d6ebeba5f8f2ef6e429aeffdd63f1c1
[25/98] blk-throttle: pass the I/O counts of a dead group on
        commit: 587edb931fdda032bcd63fdfb0db4ded75159ee9
[26/98] blk-throttle: make blkio.reset_stats reset I/O counts
        commit: ac81a07a0bd5e75e8edc0eb5be563be1b2d56554
[34/98] block: don't enter reclaim under rq_qos_mutex
        commit: 045cbfc369f5929ed473f7c200dc2e8146569ce5

Best regards,
-- 
Jens Axboe