From: Keith Busch <kbusch@kernel.org>
To: Bart Van Assche <bvanassche@acm.org>
Cc: Jens Axboe <axboe@kernel.dk>, linux-block@vger.kernel.org
Subject: Re: [PATCHSET 0/98] Various block related fixes
Date: Wed, 28 Oct 2026 13:52:00 -0600 [thread overview]
Message-ID: <auJSYF4iz0EXquwp@kbusch-mbp> (raw)
In-Reply-To: <c16356fc-662e-4939-ad76-355f3a62ed4c@acm.org>
On Wed, Oct 28, 2026 at 12:22:53PM -0700, Bart Van Assche wrote:
> On 10/28/26 9:41 AM, Jens Axboe wrote:
> > Got a giant pile of stuff dumped on me, and went through and did some
> > vetting and, particularly, commit message rewriting to get it to a
> > somewhat comprehensible shape. If you're CC'ed on a patch, please
> > review it. Feel free to queue it up, or reply back why the particular
> > patch is garbage, if it is garbage.
>
> Hi Jens,
>
> Reviewing 98 patches at once is too much for a human. So I asked an LLM
> to take a look. That LLM identified 9 must fix issues and 15 minor
> issues. Please take a look at the attached report.
Thanks, Bart.
Your review flagged a couple patches I had already queued up, so let me
explain why I think it's safe to proceed with them.
> ### 1.4 Patch 90/98 - `nvmet-rdma: disable interrupts under rsp_wr_wait_lock` (Executes `req.execute()` & `mutex_lock()` in HardIRQ/SoftIRQ Context)
> - **Patch**: [90/98](https://lore.kernel.org/linux-block/20261028164415.1262013-91-axboe@kernel.dk/) (Chris Mason) - [`nvmet_rdma_process_wr_wait_list()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/rdma.c#L514-L537) in [rdma.c](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/rdma.c#L514-L537)
> - **Root-Cause Diagnosis**: Valid. `nvmet_bio_done()` -> `nvmet_req_complete()` -> [`nvmet_rdma_queue_response()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/rdma.c#L717-L752) can run in hardirq or softirq context; when `ib_post_send()` fails, `nvmet_rdma_release_rsp()` calls `nvmet_rdma_process_wr_wait_list()`, which previously took `rsp_wr_wait_lock` with plain `spin_lock()`.
> - **Why the Fix Is Incomplete**: As the commit message itself notes, `nvmet_rdma_process_wr_wait_list()` drops `rsp_wr_wait_lock` (restoring the caller's hardirq/softirq state) and calls `nvmet_rdma_execute_command(rsp)` -> `rsp->req.execute(&rsp->req)` **directly in hardirq/softirq context**! Almost every `nvmet` `req.execute` handler assumes process context:
> - `nvmet_bdev_execute_rw()` calls `bio_alloc(..., GFP_KERNEL)` and `submit_bio()` -> `submit_bio_noacct()`, which hits `might_sleep()` and manipulates `current->bio_list` without interrupt protection (corrupting any interrupted task's `current->bio_list`!).
> - Admin/fabrics/auth/PR handlers (`nvmet_execute_identify_*`, `nvmet_execute_admin_connect`, `nvmet_execute_auth_*`, `nvmet_execute_pr_*`) call `kzalloc(..., GFP_KERNEL)`, `mutex_lock()`, and `down_read()`.
> - Furthermore, if **Patch 69** is applied and `ib_post_send()` fails on a dying queue in hardirq/softirq context (`nvmet_rdma_queue_response()` doesn't currently call `nvmet_rdma_error_comp()`, though `nvmet_rdma_Disconnect` paths take `mutex_lock(&nvmet_rdma_queue_mutex)`), any fatal error path assuming process context will sleep in atomic.
> - **Suggested Fix**: Offload `nvmet_rdma_process_wr_wait_list()` to a `work_struct` on `nvmet_wq` (or queue the work whenever `!in_task()`) instead of calling `nvmet_rdma_execute_command()` in interrupt context.
I agree the wait list shouldn't run from the completion context, but
that issue predates this patch which only fixes the lock inversion that
lockdep found. Patch seems a safe improvement to proceed with.
> ### 1.8 Patch 76/98 - `nvmet: fix NULL dereference in debugfs host_traddr` (Unlocked TOCTOU NULL-Deref & UAF Remains)
> - **Patch**: [76/98](https://lore.kernel.org/linux-block/20261028164415.1262013-77-axboe@kernel.dk/) (Josef Bacik) - [`nvmet_ctrl_host_traddr()`](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/core.c#L1822-L1830) in [core.c](file:///usr/local/google/home/bvanassche/software/linux-kernel/drivers/nvme/target/core.c#L1822-L1830)
> - **Root-Cause Diagnosis**: Valid. When the admin queue (`ctrl->sqs[0]`) disconnects while I/O queues remain open, `nvmet_sq_destroy()` sets `ctrl->sqs[0] = NULL`, so reading `host_traddr` in debugfs crashes with a near-NULL dereference (`container_of(ctrl->sqs[0], ...)`).
> - **Incomplete Synchronization**: As the commit message candidly admits (*"Nothing holds the queue for debugfs, so a reader that's past the check when the queue goes away is still on its own"*), `if (!ctrl->sqs[0])` is a plain unlocked load, and `nvmet_tcp_host_port_addr()` / `nvmet_rdma_host_port_addr()` / `nvmet_fc_host_traddr()` immediately perform a second unlocked load of `ctrl->sqs[0]` and dereference the transport queue (`queue->sockaddr_peer`, `nvme_rdma_cm_id`, `assoc->hostport`) concurrently with `nvmet_sq_destroy()` freeing the queue (`kfree(queue)` -> **TOCTOU NULL deref and KASAN UAF**).
> - **Suggested Fix**: Snapshot the formatted `host_traddr` string into a small `char host_traddr[NVMF_TRADDR_SIZE]` buffer in `struct nvmet_ctrl` during admin queue `Connect` (`nvmet_alloc_ctrl()` / `install_queue`), and have `nvmet_ctrl_host_traddr()` copy that immutable buffer. That eliminates all transport queue lifetime races with ~4 lines of code.
Yeah, this fixes the deterministic crash, but a reader can still get
past the check and race the teardown into the transport callback. I
don't think a snapshot is needed though: the controller is being torn
down, so showing a stale address isn't useful, and returning an error
is fine.
Clearing ctrl->sqs[qid] under ctrl->lock in nvmet_sq_destroy() and
holding it across the check and the ->host_traddr() call should close
the race. Guixin is working on the same race for the tls_key attribute,
so it'd make sense to fix both the same way.
next prev parent reply other threads:[~2026-10-28 19:52 UTC|newest]
Thread overview: 166+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-28 16:41 [PATCHSET 0/98] Various block related fixes Jens Axboe
2026-10-28 16:41 ` [PATCH 01/98] blk-cgroup: keep the old blkg while reassociating a bio Jens Axboe
2026-10-28 19:52 ` Tejun Heo
2026-10-28 16:41 ` [PATCH 02/98] blk-iocost: free struct ioc after a grace period Jens Axboe
2026-10-28 20:00 ` Tejun Heo
2026-10-28 16:41 ` [PATCH 03/98] blk-throttle: don't truncate the cgroup v1 iops limits Jens Axboe
2026-10-28 20:00 ` Tejun Heo
2026-10-28 16:41 ` [PATCH 04/98] block: fix use-after-free of the disk in sync_bdevs() Jens Axboe
2026-10-28 16:41 ` [PATCH 05/98] block: don't lose the status of a failed flush request Jens Axboe
2026-10-28 16:41 ` [PATCH 06/98] blk-mq: don't batch allocate tags from an inactive hctx Jens Axboe
2026-10-28 16:41 ` [PATCH 07/98] block: keep zone write plugs with plugged BIOs alive Jens Axboe
2026-10-28 16:41 ` [PATCH 08/98] block: keep the zone write plug of a failed write Jens Axboe
2026-10-28 16:41 ` [PATCH 09/98] blk-mq: end the request if getting a keyslot fails Jens Axboe
2026-10-28 16:41 ` [PATCH 10/98] block: don't unplug a zone under a failed write Jens Axboe
2026-10-28 16:41 ` [PATCH 11/98] block: check dma alignment of bvec iters in async dio Jens Axboe
2026-10-28 16:41 ` [PATCH 12/98] drbd: stop the start_resync timer of a dying device Jens Axboe
2026-10-28 16:41 ` [PATCH 13/98] blk-cgroup: fix root io.stat byte counts on 32-bit Jens Axboe
2026-10-28 19:52 ` Tejun Heo
2026-10-28 16:41 ` [PATCH 14/98] drbd: take req_lock to skip the initial sync as peer Jens Axboe
2026-10-28 16:41 ` [PATCH 15/98] zloop: refuse zone files without read/write_iter Jens Axboe
2026-10-29 7:09 ` Johannes Thumshirn
2026-10-28 16:41 ` [PATCH 16/98] block/rnbd-srv: don't free the device on a sysfs error Jens Axboe
2026-10-28 16:41 ` [PATCH 17/98] block/rnbd-srv: drop the sess_dev kobject unlocked Jens Axboe
2026-10-28 16:41 ` [PATCH 18/98] aoe: don't free a device with its work queued Jens Axboe
2026-10-28 19:57 ` Bart Van Assche
2026-10-28 23:49 ` Bart Van Assche
2026-10-28 16:41 ` [PATCH 19/98] blk-cgroup: don't free a blkg with its punt " Jens Axboe
2026-10-28 19:52 ` Tejun Heo
2026-10-28 16:41 ` [PATCH 20/98] mtip32xx: don't mark the disk dead in the irq handler Jens Axboe
2026-10-28 19:57 ` Bart Van Assche
2026-10-28 23:49 ` Bart Van Assche
2026-10-28 16:41 ` [PATCH 21/98] mtip32xx: abort timed out commands of a removed device Jens Axboe
2026-10-28 16:41 ` [PATCH 22/98] mtip32xx: abort commands in flight on surprise removal Jens Axboe
2026-10-28 19:57 ` Bart Van Assche
2026-10-28 23:49 ` Bart Van Assche
2026-10-28 16:41 ` [PATCH 23/98] blk-iolatency: flush enable_work after going offline Jens Axboe
2026-10-28 19:52 ` Tejun Heo
2026-10-28 16:41 ` [PATCH 24/98] zram: accept lz4 dictionaries of more than 64 KiB Jens Axboe
2026-10-29 1:20 ` Sergey Senozhatsky
2026-10-29 1:54 ` Jens Axboe
2026-10-29 1:55 ` Sergey Senozhatsky
2026-10-29 2:03 ` Jens Axboe
2026-10-28 16:41 ` [PATCH 25/98] blk-throttle: pass the I/O counts of a dead group on Jens Axboe
2026-10-28 20:00 ` Tejun Heo
2026-10-28 16:41 ` [PATCH 26/98] blk-throttle: make blkio.reset_stats reset I/O counts Jens Axboe
2026-10-28 19:52 ` Tejun Heo
2026-10-28 16:41 ` [PATCH 27/98] blk-mq: don't trust the cookie in blk_mq_poll() Jens Axboe
2026-10-28 16:41 ` [PATCH 28/98] blk-iocost: check again that the disk is alive Jens Axboe
2026-10-28 19:57 ` Bart Van Assche
2026-10-28 23:49 ` Bart Van Assche
2026-10-28 16:41 ` [PATCH 29/98] blk-mq: keep sharing tags fairly after a queue update Jens Axboe
2026-10-28 16:41 ` [PATCH 30/98] blk-mq: don't bring back flags of a completed request Jens Axboe
2026-10-28 16:41 ` [PATCH 31/98] blk-mq: don't forget a timer the driver has restarted Jens Axboe
2026-10-28 16:41 ` [PATCH 32/98] blk-mq-dma: initialize the dma state for P2P, too Jens Axboe
2026-10-28 16:41 ` [PATCH 33/98] blk-mq-dma: map the metadata a passthrough request has Jens Axboe
2026-10-28 16:41 ` [PATCH 34/98] block: don't enter reclaim under rq_qos_mutex Jens Axboe
2026-10-28 20:00 ` Tejun Heo
2026-10-28 16:41 ` [PATCH 35/98] blktrace: remove the debugfs files before the channel Jens Axboe
2026-10-28 16:41 ` [PATCH 36/98] blk-mq: quiesce a queue that joins a quiesced tag set Jens Axboe
2026-10-28 16:41 ` [PATCH 37/98] blk-mq: exit the hctxs of a queue under tag_list_lock Jens Axboe
2026-10-28 16:41 ` [PATCH 38/98] blk-mq: set up " Jens Axboe
2026-10-28 16:41 ` [PATCH 39/98] rnbd-srv: fix use-after-free on two CLOSE messages Jens Axboe
2026-10-28 16:41 ` [PATCH 40/98] rnbd-srv: publish the sess_dev at the end of OPEN Jens Axboe
2026-10-28 16:41 ` [PATCH 41/98] zram: check a bio's range against the device size Jens Axboe
2026-10-28 19:57 ` Bart Van Assche
2026-10-28 23:49 ` Bart Van Assche
2026-10-28 16:41 ` [PATCH 42/98] block: partitions/mac: compute the sectors in 64 bits Jens Axboe
2026-10-28 16:41 ` [PATCH 43/98] blk-mq: free the bitmaps of tags in the SRCU callback Jens Axboe
2026-10-29 7:24 ` Hannes Reinecke
2026-10-28 16:41 ` [PATCH 44/98] block: apply the segment size to the integrity buffer Jens Axboe
2026-10-28 16:41 ` [PATCH 45/98] nvmet: don't truncate the length of big discards Jens Axboe
2026-10-29 7:25 ` Hannes Reinecke
2026-10-28 16:41 ` [PATCH 46/98] nvme-loop: delete the controller if a reset fails Jens Axboe
2026-10-28 16:41 ` [PATCH 47/98] nvme: refuse metadata sizes above 255 bytes Jens Axboe
2026-10-28 17:40 ` Keith Busch
2026-10-28 16:41 ` [PATCH 48/98] nvmet-tcp: rewind the iterator after non-data records Jens Axboe
2026-10-28 19:57 ` Bart Van Assche
2026-10-29 8:58 ` Hannes Reinecke
2026-10-28 23:49 ` Bart Van Assche
2026-10-28 16:41 ` [PATCH 49/98] nvmet-tcp: ask for the TLS record type on every receive Jens Axboe
2026-10-29 7:38 ` Hannes Reinecke
2026-10-28 16:41 ` [PATCH 50/98] nvme-tcp: wait for an in-flight TLS handshake callback Jens Axboe
2026-10-29 7:40 ` Hannes Reinecke
2026-10-28 16:41 ` [PATCH 51/98] nvmet-tcp: look again after a short read with TLS Jens Axboe
2026-10-28 16:41 ` [PATCH 52/98] nvme-tcp: keep the socket while the queue is polled Jens Axboe
2026-10-28 16:41 ` [PATCH 53/98] nvme-tcp: don't get stuck on a TLS control record Jens Axboe
2026-10-29 7:44 ` Hannes Reinecke
2026-10-28 16:41 ` [PATCH 54/98] nvme-tcp: start error recovery after a TLS KeyUpdate Jens Axboe
2026-10-29 7:49 ` Hannes Reinecke
2026-10-28 16:41 ` [PATCH 55/98] nvme-tcp: don't act on PDUs for a command being sent Jens Axboe
2026-10-28 16:41 ` [PATCH 56/98] nvme-tcp: don't queue the AER while it is " Jens Axboe
2026-10-28 16:42 ` [PATCH 57/98] nvme-tcp: don't send SIGPIPE to whoever submits I/O Jens Axboe
2026-10-28 16:42 ` [PATCH 58/98] nvme-fabrics: reconnect if the association is gone Jens Axboe
2026-10-28 16:42 ` [PATCH 59/98] nvmet-auth: serialize the commands of a controller Jens Axboe
2026-10-29 9:33 ` Hannes Reinecke
2026-10-28 16:42 ` [PATCH 60/98] nvmet-auth: keep the negotiated hash per queue Jens Axboe
2026-10-29 9:43 ` Hannes Reinecke
2026-10-28 16:42 ` [PATCH 61/98] nvmet-auth: authenticate the I/O queues again Jens Axboe
2026-10-28 16:42 ` [PATCH 62/98] nvmet: don't mix queues with and without TLS Jens Axboe
2026-10-29 9:52 ` Hannes Reinecke
2026-10-28 16:42 ` [PATCH 63/98] nvmet-auth: refuse a reply without a DH value Jens Axboe
2026-10-29 9:45 ` Hannes Reinecke
2026-10-28 16:42 ` [PATCH 64/98] nvmet-auth: keep the old keys if the new ones fail Jens Axboe
2026-10-29 9:53 ` Hannes Reinecke
2026-10-28 16:42 ` [PATCH 65/98] nvmet-auth: take only AUTH_Negotiate for a negotiation Jens Axboe
2026-10-29 9:55 ` Hannes Reinecke
2026-10-28 16:42 ` [PATCH 66/98] nvmet-auth: free the old data on a new AUTH_Negotiate Jens Axboe
2026-10-29 9:56 ` Hannes Reinecke
2026-10-28 16:42 ` [PATCH 67/98] nvmet-auth: fix the check of the offered DH groups Jens Axboe
2026-10-29 9:57 ` Hannes Reinecke
2026-10-28 16:42 ` [PATCH 68/98] nvmet: accept I/O queues on the controller's port only Jens Axboe
2026-10-29 9:58 ` Hannes Reinecke
2026-10-28 16:42 ` [PATCH 69/98] nvmet: confine fatal errors of unauthenticated queues Jens Axboe
2026-10-29 10:23 ` Hannes Reinecke
2026-10-28 16:42 ` [PATCH 70/98] nvmet-tcp: don't wait for the host in the accept work Jens Axboe
2026-10-28 16:42 ` [PATCH 71/98] nvmet: check the host NQN against the TLS PSK identity Jens Axboe
2026-10-28 16:42 ` [PATCH 72/98] nvmet: fix NULL dereference of PCI endpoint targets Jens Axboe
2026-10-28 16:42 ` [PATCH 73/98] nvmet-tcp: stop listening when a port is removed Jens Axboe
2026-10-29 10:02 ` Hannes Reinecke
2026-10-28 16:42 ` [PATCH 74/98] nvme-tcp: keep the page frag cache with the controller Jens Axboe
2026-10-28 16:42 ` [PATCH 75/98] nvmet-tcp: hold the queue while the TLS timer is armed Jens Axboe
2026-10-28 16:42 ` [PATCH 76/98] nvmet: fix NULL dereference in debugfs host_traddr Jens Axboe
2026-10-28 19:57 ` Bart Van Assche
2026-10-28 23:49 ` Bart Van Assche
2026-10-28 16:42 ` [PATCH 77/98] nvmet-tcp: let the accept work come to an end Jens Axboe
2026-10-29 10:06 ` Hannes Reinecke
2026-10-28 16:42 ` [PATCH 78/98] nvmet-tcp: don't read a removed port at Connect time Jens Axboe
2026-10-28 16:42 ` [PATCH 79/98] nvme-tcp: start error recovery when a send fails Jens Axboe
2026-10-29 10:08 ` Hannes Reinecke
2026-10-28 16:42 ` [PATCH 80/98] nvme-loop: don't BUG() if a scan runs beside a reset Jens Axboe
2026-10-29 10:08 ` Hannes Reinecke
2026-10-28 16:42 ` [PATCH 81/98] nvmet: fix pc_ref use-after-free in preempt and abort Jens Axboe
2026-10-28 16:42 ` [PATCH 82/98] nvmet: move preempt and abort work off nvmet-wq Jens Axboe
2026-10-28 16:42 ` [PATCH 83/98] nvmet-rdma: treat a failed connect as a disconnect Jens Axboe
2026-10-28 16:42 ` [PATCH 84/98] nvmet-rdma: disconnect only established queues Jens Axboe
2026-10-28 16:42 ` [PATCH 85/98] nvmet-rdma: take a device off the list last Jens Axboe
2026-10-28 16:42 ` [PATCH 86/98] nvmet-rdma: queue the release work under the mutex Jens Axboe
2026-10-28 16:42 ` [PATCH 87/98] nvmet-rdma: disconnect a queue on device removal Jens Axboe
2026-10-28 16:42 ` [PATCH 88/98] nvmet-rdma: flush nvmet-wq on module exit Jens Axboe
2026-10-28 16:42 ` [PATCH 89/98] nvmet-rdma: reject queues larger than max_cqe Jens Axboe
2026-10-28 16:42 ` [PATCH 90/98] nvmet-rdma: disable interrupts under rsp_wr_wait_lock Jens Axboe
2026-10-28 19:57 ` Bart Van Assche
2026-10-28 23:49 ` Bart Van Assche
2026-10-28 16:42 ` [PATCH 91/98] nvmet-rdma: don't run the wait list recursively Jens Axboe
2026-10-28 16:42 ` [PATCH 92/98] nvmet-rdma: fix a lost run of the wait list Jens Axboe
2026-10-28 16:42 ` [PATCH 93/98] nvmet-fc: only send read data for successful commands Jens Axboe
2026-10-29 10:11 ` Hannes Reinecke
2026-10-28 16:42 ` [PATCH 94/98] nvmet: wait only for executing I/O in preempt and abort Jens Axboe
2026-10-28 16:42 ` [PATCH 95/98] nvmet-rdma: fix double puts after a failed RDMA WRITE Jens Axboe
2026-10-28 16:42 ` [PATCH 96/98] nvmet: take sq->ref in nvmet_sq_fatal_error() Jens Axboe
2026-10-29 10:11 ` Hannes Reinecke
2026-10-28 16:42 ` [PATCH 97/98] nvmet-rdma: hold sq->ref to post a T10-PI Read's SEND Jens Axboe
2026-10-28 19:57 ` Bart Van Assche
2026-10-28 23:49 ` Bart Van Assche
2026-10-28 16:42 ` [PATCH 98/98] nvmet: don't clear sq->ctrl for a Connect with sqsize 0 Jens Axboe
2026-10-29 10:13 ` Hannes Reinecke
2026-10-28 17:11 ` [PATCHSET 0/98] Various block related fixes Keith Busch
2026-10-28 17:13 ` Jens Axboe
2026-10-28 19:12 ` Keith Busch
2026-10-28 19:22 ` Bart Van Assche
2026-10-28 19:29 ` Jens Axboe
2026-10-28 19:47 ` Bart Van Assche
2026-10-28 21:06 ` Jens Axboe
2026-10-28 23:51 ` Bart Van Assche
2026-10-28 19:52 ` Keith Busch [this message]
2026-10-28 21:41 ` (subset) " Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=auJSYF4iz0EXquwp@kbusch-mbp \
--to=kbusch@kernel.org \
--cc=axboe@kernel.dk \
--cc=bvanassche@acm.org \
--cc=linux-block@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox