[PATCH V2 0/2] ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel

public inbox for linux-block@vger.kernel.org
 help / color / mirror / Atom feed

* [PATCH V2 0/2] ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd
@ 2025-04-25  1:37 Ming Lei
  2025-04-25  1:37 ` [PATCH V2 1/2] ublk: call ublk_dispatch_req() for handling UBLK_U_IO_NEED_GET_DATA Ming Lei
                   ` (2 more replies)
  0 siblings, 3 replies; 9+ messages in thread
From: Ming Lei @ 2025-04-25  1:37 UTC (permalink / raw)
  To: Jens Axboe, linux-block
  Cc: Uday Shankar, Caleb Sander Mateos, Guy Eisenberg, Jared Holzman,
	Yoav Cohen, Omri Levi, Ofer Oshri, Ming Lei

Hello Jens,

The 2 patches try to fix race between between io_uring_cmd_complete_in_task
and ublk_cancel_cmd.

Thanks,
Ming

V2:
	- improve comment and commit log
	- remove useless memory barrier(Caleb Sander Mateos)
	- add tested-by fixes tag

Ming Lei (2):
  ublk: call ublk_dispatch_req() for handling UBLK_U_IO_NEED_GET_DATA
  ublk: fix race between io_uring_cmd_complete_in_task and
    ublk_cancel_cmd

 drivers/block/ublk_drv.c | 41 +++++++++++++++++++++++-----------------
 1 file changed, 24 insertions(+), 17 deletions(-)

-- 
2.47.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH V2 1/2] ublk: call ublk_dispatch_req() for handling UBLK_U_IO_NEED_GET_DATA
  2025-04-25  1:37 [PATCH V2 0/2] ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd Ming Lei
@ 2025-04-25  1:37 ` Ming Lei
  2025-04-25  1:37 ` [PATCH V2 2/2] ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd Ming Lei
  2025-04-25  1:53 ` [PATCH V2 0/2] " Jens Axboe
  2 siblings, 0 replies; 9+ messages in thread
From: Ming Lei @ 2025-04-25  1:37 UTC (permalink / raw)
  To: Jens Axboe, linux-block
  Cc: Uday Shankar, Caleb Sander Mateos, Guy Eisenberg, Jared Holzman,
	Yoav Cohen, Omri Levi, Ofer Oshri, Ming Lei

We call io_uring_cmd_complete_in_task() to schedule task_work for handling
UBLK_U_IO_NEED_GET_DATA.

This way is really not necessary because the current context is exactly
the ublk queue context, so call ublk_dispatch_req() directly for handling
UBLK_U_IO_NEED_GET_DATA.

Fixes: 216c8f5ef0f2 ("ublk: replace monitor with cancelable uring_cmd")
Tested-by: Jared Holzman <jholzman@nvidia.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
 drivers/block/ublk_drv.c | 14 +++-----------
 1 file changed, 3 insertions(+), 11 deletions(-)

diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c
index 2de7b2bd409d..c4d4be4f6fbd 100644
--- a/drivers/block/ublk_drv.c
+++ b/drivers/block/ublk_drv.c
@@ -1886,15 +1886,6 @@ static void ublk_mark_io_ready(struct ublk_device *ub, struct ublk_queue *ubq)
 	}
 }
 
-static void ublk_handle_need_get_data(struct ublk_device *ub, int q_id,
-		int tag)
-{
-	struct ublk_queue *ubq = ublk_get_queue(ub, q_id);
-	struct request *req = blk_mq_tag_to_rq(ub->tag_set.tags[q_id], tag);
-
-	ublk_queue_cmd(ubq, req);
-}
-
 static inline int ublk_check_cmd_op(u32 cmd_op)
 {
 	u32 ioc_type = _IOC_TYPE(cmd_op);
@@ -2103,8 +2094,9 @@ static int __ublk_ch_uring_cmd(struct io_uring_cmd *cmd,
 		if (!(io->flags & UBLK_IO_FLAG_OWNED_BY_SRV))
 			goto out;
 		ublk_fill_io_cmd(io, cmd, ub_cmd->addr);
-		ublk_handle_need_get_data(ub, ub_cmd->q_id, ub_cmd->tag);
-		break;
+		req = blk_mq_tag_to_rq(ub->tag_set.tags[ub_cmd->q_id], tag);
+		ublk_dispatch_req(ubq, req, issue_flags);
+		return -EIOCBQUEUED;
 	default:
 		goto out;
 	}
-- 
2.47.0


^ permalink raw reply related	[flat|nested] 9+ messages in thread

* [PATCH V2 2/2] ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd
  2025-04-25  1:37 [PATCH V2 0/2] ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd Ming Lei
  2025-04-25  1:37 ` [PATCH V2 1/2] ublk: call ublk_dispatch_req() for handling UBLK_U_IO_NEED_GET_DATA Ming Lei
@ 2025-04-25  1:37 ` Ming Lei
  2025-05-14  1:50   ` Shinichiro Kawasaki
  2025-04-25  1:53 ` [PATCH V2 0/2] " Jens Axboe
  2 siblings, 1 reply; 9+ messages in thread
From: Ming Lei @ 2025-04-25  1:37 UTC (permalink / raw)
  To: Jens Axboe, linux-block
  Cc: Uday Shankar, Caleb Sander Mateos, Guy Eisenberg, Jared Holzman,
	Yoav Cohen, Omri Levi, Ofer Oshri, Ming Lei

ublk_cancel_cmd() calls io_uring_cmd_done() to complete uring_cmd, but
we may have scheduled task work via io_uring_cmd_complete_in_task() for
dispatching request, then kernel crash can be triggered.

Fix it by not trying to canceling the command if ublk block request is
started.

Fixes: 216c8f5ef0f2 ("ublk: replace monitor with cancelable uring_cmd")
Reported-by: Jared Holzman <jholzman@nvidia.com>
Tested-by: Jared Holzman <jholzman@nvidia.com>
Closes: https://lore.kernel.org/linux-block/d2179120-171b-47ba-b664-23242981ef19@nvidia.com/
Signed-off-by: Ming Lei <ming.lei@redhat.com>
---
 drivers/block/ublk_drv.c | 27 +++++++++++++++++++++------
 1 file changed, 21 insertions(+), 6 deletions(-)

diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c
index c4d4be4f6fbd..40f971a66d3e 100644
--- a/drivers/block/ublk_drv.c
+++ b/drivers/block/ublk_drv.c
@@ -1683,14 +1683,31 @@ static void ublk_start_cancel(struct ublk_queue *ubq)
 	ublk_put_disk(disk);
 }
 
-static void ublk_cancel_cmd(struct ublk_queue *ubq, struct ublk_io *io,
+static void ublk_cancel_cmd(struct ublk_queue *ubq, unsigned tag,
 		unsigned int issue_flags)
 {
+	struct ublk_io *io = &ubq->ios[tag];
+	struct ublk_device *ub = ubq->dev;
+	struct request *req;
 	bool done;
 
 	if (!(io->flags & UBLK_IO_FLAG_ACTIVE))
 		return;
 
+	/*
+	 * Don't try to cancel this command if the request is started for
+	 * avoiding race between io_uring_cmd_done() and
+	 * io_uring_cmd_complete_in_task().
+	 *
+	 * Either the started request will be aborted via __ublk_abort_rq(),
+	 * then this uring_cmd is canceled next time, or it will be done in
+	 * task work function ublk_dispatch_req() because io_uring guarantees
+	 * that ublk_dispatch_req() is always called
+	 */
+	req = blk_mq_tag_to_rq(ub->tag_set.tags[ubq->q_id], tag);
+	if (req && blk_mq_request_started(req))
+		return;
+
 	spin_lock(&ubq->cancel_lock);
 	done = !!(io->flags & UBLK_IO_FLAG_CANCELED);
 	if (!done)
@@ -1722,7 +1739,6 @@ static void ublk_uring_cmd_cancel_fn(struct io_uring_cmd *cmd,
 	struct ublk_uring_cmd_pdu *pdu = ublk_get_uring_cmd_pdu(cmd);
 	struct ublk_queue *ubq = pdu->ubq;
 	struct task_struct *task;
-	struct ublk_io *io;
 
 	if (WARN_ON_ONCE(!ubq))
 		return;
@@ -1737,9 +1753,8 @@ static void ublk_uring_cmd_cancel_fn(struct io_uring_cmd *cmd,
 	if (!ubq->canceling)
 		ublk_start_cancel(ubq);
 
-	io = &ubq->ios[pdu->tag];
-	WARN_ON_ONCE(io->cmd != cmd);
-	ublk_cancel_cmd(ubq, io, issue_flags);
+	WARN_ON_ONCE(ubq->ios[pdu->tag].cmd != cmd);
+	ublk_cancel_cmd(ubq, pdu->tag, issue_flags);
 }
 
 static inline bool ublk_queue_ready(struct ublk_queue *ubq)
@@ -1752,7 +1767,7 @@ static void ublk_cancel_queue(struct ublk_queue *ubq)
 	int i;
 
 	for (i = 0; i < ubq->q_depth; i++)
-		ublk_cancel_cmd(ubq, &ubq->ios[i], IO_URING_F_UNLOCKED);
+		ublk_cancel_cmd(ubq, i, IO_URING_F_UNLOCKED);
 }
 
 /* Cancel all pending commands, must be called after del_gendisk() returns */
-- 
2.47.0


^ permalink raw reply related	[flat|nested] 9+ messages in thread

* Re: [PATCH V2 2/2] ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd
  2025-04-25  1:37 ` [PATCH V2 2/2] ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd Ming Lei
@ 2025-05-14  1:50   ` Shinichiro Kawasaki
  2025-05-18 19:12     ` Jared Holzman
  0 siblings, 1 reply; 9+ messages in thread
From: Shinichiro Kawasaki @ 2025-05-14  1:50 UTC (permalink / raw)
  To: Ming Lei
  Cc: Jens Axboe, linux-block@vger.kernel.org, Uday Shankar,
	Caleb Sander Mateos, Guy Eisenberg, Jared Holzman, Yoav Cohen,
	Omri Levi, Ofer Oshri

On Apr 25, 2025 / 09:37, Ming Lei wrote:
> ublk_cancel_cmd() calls io_uring_cmd_done() to complete uring_cmd, but
> we may have scheduled task work via io_uring_cmd_complete_in_task() for
> dispatching request, then kernel crash can be triggered.
> 
> Fix it by not trying to canceling the command if ublk block request is
> started.

I found that the blktests test case ublk/002 often hangs with the recent
v6.15-rcX kernel tags with the INFO at iou-wrk-X [1]. The hang is recreated
in stable manner when I repeat the test case a few times.

I bisected and this patch as the commit f40139fde527 triggers the hang.
When I reverted the commit from the kernel v6.15-rc6, the hang disappeared.
(I repeated ublk/002 20 times, and observed no hang.)

The hang was observed with test systems with Fedora 42. I do not observe
the hang with Fedora 41, but not sure why. liburing version difference
could be the reason (v2.6 for Fedora 41, v2.9 for Fedora 42).

Actions for fix will be appreciated.

[1]

[ 4497.777695] [ T130863] run blktests ublk/002 at 2025-05-07 14:48:32
[ 4499.983130] [  T67084] blk_print_req_error: 58 callbacks suppressed
[ 4499.983136] [  T67084] I/O error, dev ublkb0, sector 106830432 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 4499.985230] [  T67084] I/O error, dev ublkb0, sector 165364248 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 4499.998194] [  T57173] I/O error, dev ublkb0, sector 251279584 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
[ 4671.208372] [     T46] INFO: task iou-wrk-131061:131062 blocked for more than 122 seconds.
[ 4671.209417] [     T46]       Not tainted 6.15.0-rc5 #317
[ 4671.210182] [     T46] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4671.211194] [     T46] task:iou-wrk-131061  state:D stack:0     pid:131062 tgid:131061 ppid:131060 task_flags:0x404150 flags:0x00004000
[ 4671.212573] [     T46] Call Trace:
[ 4671.213364] [     T46]  <TASK>
[ 4671.214047] [     T46]  __schedule+0xf8c/0x5c80
[ 4671.214920] [     T46]  ? rcu_is_watching+0x11/0xb0
[ 4671.215831] [     T46]  ? __mutex_lock+0x2d9/0x1b50
[ 4671.216760] [     T46]  ? rcu_is_watching+0x11/0xb0
[ 4671.217718] [     T46]  ? lock_release+0x244/0x300
[ 4671.218621] [     T46]  ? lock_release+0x244/0x300
[ 4671.219521] [     T46]  ? __pfx___schedule+0x10/0x10
[ 4671.220433] [     T46]  ? do_raw_spin_lock+0x124/0x260
[ 4671.221425] [     T46]  ? rcu_is_watching+0x11/0xb0
[ 4671.222292] [     T46]  ? rcu_is_watching+0x11/0xb0
[ 4671.223121] [     T46]  ? lock_release+0x244/0x300
[ 4671.223984] [     T46]  ? lock_release+0x244/0x300
[ 4671.224844] [     T46]  schedule+0xda/0x390
[ 4671.225667] [     T46]  blk_mq_freeze_queue_wait+0x121/0x170
[ 4671.226653] [     T46]  ? __pfx_blk_mq_freeze_queue_wait+0x10/0x10
[ 4671.227656] [     T46]  ? __pfx_autoremove_wake_function+0x10/0x10
[ 4671.228656] [     T46]  ? trace_hardirqs_on+0x14/0x150
[ 4671.229564] [     T46]  ? _raw_spin_unlock_irq+0x34/0x50
[ 4671.230484] [     T46]  del_gendisk+0x4f9/0xa20
[ 4671.231320] [     T46]  ? ublk_stop_dev+0x24/0x200 [ublk_drv]
[ 4671.232379] [     T46]  ? __pfx_del_gendisk+0x10/0x10
[ 4671.233197] [     T46]  ? __pfx___mutex_lock+0x10/0x10
[ 4671.234094] [     T46]  ublk_stop_dev_unlocked.part.0+0xb6/0x580 [ublk_drv]
[ 4671.235153] [     T46]  ? __pfx_ublk_stop_dev_unlocked.part.0+0x10/0x10 [ublk_drv]
[ 4671.236303] [     T46]  ? kvm_sched_clock_read+0xd/0x20
[ 4671.237156] [     T46]  ublk_stop_dev+0x62/0x200 [ublk_drv]
[ 4671.238081] [     T46]  ? security_capable+0x79/0x110
[ 4671.238964] [     T46]  ublk_ctrl_uring_cmd+0xe87/0x2a70 [ublk_drv]
[ 4671.239967] [     T46]  ? __pfx_avc_has_perm+0x10/0x10
[ 4671.240858] [     T46]  ? lock_release+0x244/0x300
[ 4671.241759] [     T46]  ? rcu_is_watching+0x11/0xb0
[ 4671.242677] [     T46]  ? __pfx_ublk_ctrl_uring_cmd+0x10/0x10 [ublk_drv]
[ 4671.243751] [     T46]  ? finish_task_switch.isra.0+0x202/0x850
[ 4671.244728] [     T46]  ? selinux_uring_cmd+0x1c3/0x280
[ 4671.245623] [     T46]  ? __schedule+0x2f21/0x5c80
[ 4671.246504] [     T46]  ? __pfx_selinux_uring_cmd+0x10/0x10
[ 4671.247438] [     T46]  ? rcu_is_watching+0x11/0xb0
[ 4671.248228] [     T46]  ? __pfx_do_raw_spin_lock+0x10/0x10
[ 4671.249109] [     T46]  io_uring_cmd+0x221/0x3c0
[ 4671.249952] [     T46]  io_issue_sqe+0x6ab/0x1800
[ 4671.250767] [     T46]  ? trace_hardirqs_on+0x14/0x150
[ 4671.251634] [     T46]  ? __pfx_io_issue_sqe+0x10/0x10
[ 4671.252502] [     T46]  ? __timer_delete_sync+0x16c/0x270
[ 4671.253392] [     T46]  ? __pfx___timer_delete_sync+0x10/0x10
[ 4671.254402] [     T46]  ? rcu_is_watching+0x11/0xb0
[ 4671.255162] [     T46]  io_wq_submit_work+0x3ea/0xf70
[ 4671.255990] [     T46]  io_worker_handle_work+0x615/0x1280
[ 4671.256854] [     T46]  io_wq_worker+0x2ee/0xbd0
[ 4671.257670] [     T46]  ? __pfx_io_wq_worker+0x10/0x10
[ 4671.258491] [     T46]  ? rcu_is_watching+0x11/0xb0
[ 4671.259285] [     T46]  ? lock_acquire+0x2af/0x310
[ 4671.260035] [     T46]  ? rcu_is_watching+0x11/0xb0
[ 4671.260800] [     T46]  ? do_raw_spin_lock+0x124/0x260
[ 4671.261604] [     T46]  ? lock_acquire+0x2af/0x310
[ 4671.262376] [     T46]  ? __pfx_do_raw_spin_lock+0x10/0x10
[ 4671.263186] [     T46]  ? lock_release+0x244/0x300
[ 4671.263947] [     T46]  ? rcu_is_watching+0x11/0xb0
[ 4671.264718] [     T46]  ? __pfx_io_wq_worker+0x10/0x10
[ 4671.265529] [     T46]  ret_from_fork+0x30/0x70
[ 4671.266277] [     T46]  ? __pfx_io_wq_worker+0x10/0x10
[ 4671.267070] [     T46]  ret_from_fork_asm+0x1a/0x30
[ 4671.267855] [     T46]  </TASK>
[ 4671.268495] [     T46] INFO: lockdep is turned off.
[ 4794.088765] [     T46] INFO: task iou-wrk-131061:131062 blocked for more than 245 seconds.
[ 4794.090310] [     T46]       Not tainted 6.15.0-rc5 #317
[ 4794.091453] [     T46] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 4794.093379] [     T46] task:iou-wrk-131061  state:D stack:0     pid:131062 tgid:131061 ppid:131060 task_flags:0x404150 flags:0x00004000
[ 4794.095353] [     T46] Call Trace:
[ 4794.096142] [     T46]  <TASK>
[ 4794.096929] [     T46]  __schedule+0xf8c/0x5c80
[ 4794.097878] [     T46]  ? rcu_is_watching+0x11/0xb0
[ 4794.098869] [     T46]  ? __mutex_lock+0x2d9/0x1b50
[ 4794.099866] [     T46]  ? rcu_is_watching+0x11/0xb0
[ 4794.100870] [     T46]  ? lock_release+0x244/0x300
[ 4794.101857] [     T46]  ? lock_release+0x244/0x300
[ 4794.102797] [     T46]  ? __pfx___schedule+0x10/0x10
[ 4794.103726] [     T46]  ? do_raw_spin_lock+0x124/0x260
[ 4794.104752] [     T46]  ? rcu_is_watching+0x11/0xb0
[ 4794.105744] [     T46]  ? rcu_is_watching+0x11/0xb0
[ 4794.106678] [     T46]  ? lock_release+0x244/0x300
[ 4794.107575] [     T46]  ? lock_release+0x244/0x300
[ 4794.108528] [     T46]  schedule+0xda/0x390
[ 4794.109391] [     T46]  blk_mq_freeze_queue_wait+0x121/0x170
[ 4794.110437] [     T46]  ? __pfx_blk_mq_freeze_queue_wait+0x10/0x10
[ 4794.111527] [     T46]  ? __pfx_autoremove_wake_function+0x10/0x10
[ 4794.112662] [     T46]  ? trace_hardirqs_on+0x14/0x150
[ 4794.113626] [     T46]  ? _raw_spin_unlock_irq+0x34/0x50
[ 4794.114644] [     T46]  del_gendisk+0x4f9/0xa20
[ 4794.115518] [     T46]  ? ublk_stop_dev+0x24/0x200 [ublk_drv]
[ 4794.116548] [     T46]  ? __pfx_del_gendisk+0x10/0x10
[ 4794.117536] [     T46]  ? __pfx___mutex_lock+0x10/0x10
[ 4794.118521] [     T46]  ublk_stop_dev_unlocked.part.0+0xb6/0x580 [ublk_drv]
[ 4794.119749] [     T46]  ? __pfx_ublk_stop_dev_unlocked.part.0+0x10/0x10 [ublk_drv]
[ 4794.121060] [     T46]  ? kvm_sched_clock_read+0xd/0x20
[ 4794.122081] [     T46]  ublk_stop_dev+0x62/0x200 [ublk_drv]
[ 4794.123102] [     T46]  ? security_capable+0x79/0x110
[ 4794.124139] [     T46]  ublk_ctrl_uring_cmd+0xe87/0x2a70 [ublk_drv]
[ 4794.125225] [     T46]  ? __pfx_avc_has_perm+0x10/0x10
[ 4794.126054] [     T46]  ? lock_release+0x244/0x300
[ 4794.126850] [     T46]  ? rcu_is_watching+0x11/0xb0
[ 4794.127579] [     T46]  ? __pfx_ublk_ctrl_uring_cmd+0x10/0x10 [ublk_drv]
[ 4794.128512] [     T46]  ? finish_task_switch.isra.0+0x202/0x850
[ 4794.129379] [     T46]  ? selinux_uring_cmd+0x1c3/0x280
[ 4794.130178] [     T46]  ? __schedule+0x2f21/0x5c80
[ 4794.130926] [     T46]  ? __pfx_selinux_uring_cmd+0x10/0x10
[ 4794.131701] [     T46]  ? rcu_is_watching+0x11/0xb0
[ 4794.132357] [     T46]  ? __pfx_do_raw_spin_lock+0x10/0x10
[ 4794.133159] [     T46]  io_uring_cmd+0x221/0x3c0
[ 4794.133931] [     T46]  io_issue_sqe+0x6ab/0x1800
[ 4794.134653] [     T46]  ? trace_hardirqs_on+0x14/0x150
[ 4794.135417] [     T46]  ? __pfx_io_issue_sqe+0x10/0x10
[ 4794.136191] [     T46]  ? __timer_delete_sync+0x16c/0x270
[ 4794.136984] [     T46]  ? __pfx___timer_delete_sync+0x10/0x10
[ 4794.137833] [     T46]  ? rcu_is_watching+0x11/0xb0
[ 4794.138527] [     T46]  io_wq_submit_work+0x3ea/0xf70
[ 4794.139268] [     T46]  io_worker_handle_work+0x615/0x1280
[ 4794.140062] [     T46]  io_wq_worker+0x2ee/0xbd0
[ 4794.140806] [     T46]  ? __pfx_io_wq_worker+0x10/0x10
[ 4794.141478] [     T46]  ? rcu_is_watching+0x11/0xb0
[ 4794.142189] [     T46]  ? lock_acquire+0x2af/0x310
[ 4794.142930] [     T46]  ? rcu_is_watching+0x11/0xb0
[ 4794.143663] [     T46]  ? do_raw_spin_lock+0x124/0x260
[ 4794.144363] [     T46]  ? lock_acquire+0x2af/0x310
[ 4794.145087] [     T46]  ? __pfx_do_raw_spin_lock+0x10/0x10
[ 4794.145897] [     T46]  ? lock_release+0x244/0x300
[ 4794.146617] [     T46]  ? rcu_is_watching+0x11/0xb0
[ 4794.147340] [     T46]  ? __pfx_io_wq_worker+0x10/0x10
[ 4794.148070] [     T46]  ret_from_fork+0x30/0x70
[ 4794.148817] [     T46]  ? __pfx_io_wq_worker+0x10/0x10
[ 4794.149576] [     T46]  ret_from_fork_asm+0x1a/0x30
[ 4794.150307] [     T46]  </TASK>
[ 4794.150959] [     T46] INFO: lockdep is turned off.
...

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH V2 2/2] ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd
  2025-05-14  1:50   ` Shinichiro Kawasaki
@ 2025-05-18 19:12     ` Jared Holzman
  2025-05-18 23:54       ` Shinichiro Kawasaki
  2025-05-19  2:45       ` Ming Lei
  0 siblings, 2 replies; 9+ messages in thread
From: Jared Holzman @ 2025-05-18 19:12 UTC (permalink / raw)
  To: Shinichiro Kawasaki, Ming Lei
  Cc: Jens Axboe, linux-block@vger.kernel.org, Uday Shankar,
	Caleb Sander Mateos, Guy Eisenberg, Yoav Cohen, Omri Levi,
	Ofer Oshri

On 14/05/2025 4:50, Shinichiro Kawasaki wrote:
> On Apr 25, 2025 / 09:37, Ming Lei wrote:
>> ublk_cancel_cmd() calls io_uring_cmd_done() to complete uring_cmd, but
>> we may have scheduled task work via io_uring_cmd_complete_in_task() for
>> dispatching request, then kernel crash can be triggered.
>>
>> Fix it by not trying to canceling the command if ublk block request is
>> started.
> 
> I found that the blktests test case ublk/002 often hangs with the recent
> v6.15-rcX kernel tags with the INFO at iou-wrk-X [1]. The hang is recreated
> in stable manner when I repeat the test case a few times.
> 
> I bisected and this patch as the commit f40139fde527 triggers the hang.
> When I reverted the commit from the kernel v6.15-rc6, the hang disappeared.
> (I repeated ublk/002 20 times, and observed no hang.)
> 
> The hang was observed with test systems with Fedora 42. I do not observe
> the hang with Fedora 41, but not sure why. liburing version difference
> could be the reason (v2.6 for Fedora 41, v2.9 for Fedora 42).
> 
> Actions for fix will be appreciated.
> 
> [1]
> 
> [ 4497.777695] [ T130863] run blktests ublk/002 at 2025-05-07 14:48:32
> [ 4499.983130] [  T67084] blk_print_req_error: 58 callbacks suppressed
> [ 4499.983136] [  T67084] I/O error, dev ublkb0, sector 106830432 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
> [ 4499.985230] [  T67084] I/O error, dev ublkb0, sector 165364248 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
> [ 4499.998194] [  T57173] I/O error, dev ublkb0, sector 251279584 op 0x0:(READ) flags 0x800 phys_seg 1 prio class 0
> [ 4671.208372] [     T46] INFO: task iou-wrk-131061:131062 blocked for more than 122 seconds.
> [ 4671.209417] [     T46]       Not tainted 6.15.0-rc5 #317
> [ 4671.210182] [     T46] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 4671.211194] [     T46] task:iou-wrk-131061  state:D stack:0     pid:131062 tgid:131061 ppid:131060 task_flags:0x404150 flags:0x00004000
> [ 4671.212573] [     T46] Call Trace:
> [ 4671.213364] [     T46]  <TASK>
> [ 4671.214047] [     T46]  __schedule+0xf8c/0x5c80
> [ 4671.214920] [     T46]  ? rcu_is_watching+0x11/0xb0
> [ 4671.215831] [     T46]  ? __mutex_lock+0x2d9/0x1b50
> [ 4671.216760] [     T46]  ? rcu_is_watching+0x11/0xb0
> [ 4671.217718] [     T46]  ? lock_release+0x244/0x300
> [ 4671.218621] [     T46]  ? lock_release+0x244/0x300
> [ 4671.219521] [     T46]  ? __pfx___schedule+0x10/0x10
> [ 4671.220433] [     T46]  ? do_raw_spin_lock+0x124/0x260
> [ 4671.221425] [     T46]  ? rcu_is_watching+0x11/0xb0
> [ 4671.222292] [     T46]  ? rcu_is_watching+0x11/0xb0
> [ 4671.223121] [     T46]  ? lock_release+0x244/0x300
> [ 4671.223984] [     T46]  ? lock_release+0x244/0x300
> [ 4671.224844] [     T46]  schedule+0xda/0x390
> [ 4671.225667] [     T46]  blk_mq_freeze_queue_wait+0x121/0x170
> [ 4671.226653] [     T46]  ? __pfx_blk_mq_freeze_queue_wait+0x10/0x10
> [ 4671.227656] [     T46]  ? __pfx_autoremove_wake_function+0x10/0x10
> [ 4671.228656] [     T46]  ? trace_hardirqs_on+0x14/0x150
> [ 4671.229564] [     T46]  ? _raw_spin_unlock_irq+0x34/0x50
> [ 4671.230484] [     T46]  del_gendisk+0x4f9/0xa20
> [ 4671.231320] [     T46]  ? ublk_stop_dev+0x24/0x200 [ublk_drv]
> [ 4671.232379] [     T46]  ? __pfx_del_gendisk+0x10/0x10
> [ 4671.233197] [     T46]  ? __pfx___mutex_lock+0x10/0x10
> [ 4671.234094] [     T46]  ublk_stop_dev_unlocked.part.0+0xb6/0x580 [ublk_drv]
> [ 4671.235153] [     T46]  ? __pfx_ublk_stop_dev_unlocked.part.0+0x10/0x10 [ublk_drv]
> [ 4671.236303] [     T46]  ? kvm_sched_clock_read+0xd/0x20
> [ 4671.237156] [     T46]  ublk_stop_dev+0x62/0x200 [ublk_drv]
> [ 4671.238081] [     T46]  ? security_capable+0x79/0x110
> [ 4671.238964] [     T46]  ublk_ctrl_uring_cmd+0xe87/0x2a70 [ublk_drv]
> [ 4671.239967] [     T46]  ? __pfx_avc_has_perm+0x10/0x10
> [ 4671.240858] [     T46]  ? lock_release+0x244/0x300
> [ 4671.241759] [     T46]  ? rcu_is_watching+0x11/0xb0
> [ 4671.242677] [     T46]  ? __pfx_ublk_ctrl_uring_cmd+0x10/0x10 [ublk_drv]
> [ 4671.243751] [     T46]  ? finish_task_switch.isra.0+0x202/0x850
> [ 4671.244728] [     T46]  ? selinux_uring_cmd+0x1c3/0x280
> [ 4671.245623] [     T46]  ? __schedule+0x2f21/0x5c80
> [ 4671.246504] [     T46]  ? __pfx_selinux_uring_cmd+0x10/0x10
> [ 4671.247438] [     T46]  ? rcu_is_watching+0x11/0xb0
> [ 4671.248228] [     T46]  ? __pfx_do_raw_spin_lock+0x10/0x10
> [ 4671.249109] [     T46]  io_uring_cmd+0x221/0x3c0
> [ 4671.249952] [     T46]  io_issue_sqe+0x6ab/0x1800
> [ 4671.250767] [     T46]  ? trace_hardirqs_on+0x14/0x150
> [ 4671.251634] [     T46]  ? __pfx_io_issue_sqe+0x10/0x10
> [ 4671.252502] [     T46]  ? __timer_delete_sync+0x16c/0x270
> [ 4671.253392] [     T46]  ? __pfx___timer_delete_sync+0x10/0x10
> [ 4671.254402] [     T46]  ? rcu_is_watching+0x11/0xb0
> [ 4671.255162] [     T46]  io_wq_submit_work+0x3ea/0xf70
> [ 4671.255990] [     T46]  io_worker_handle_work+0x615/0x1280
> [ 4671.256854] [     T46]  io_wq_worker+0x2ee/0xbd0
> [ 4671.257670] [     T46]  ? __pfx_io_wq_worker+0x10/0x10
> [ 4671.258491] [     T46]  ? rcu_is_watching+0x11/0xb0
> [ 4671.259285] [     T46]  ? lock_acquire+0x2af/0x310
> [ 4671.260035] [     T46]  ? rcu_is_watching+0x11/0xb0
> [ 4671.260800] [     T46]  ? do_raw_spin_lock+0x124/0x260
> [ 4671.261604] [     T46]  ? lock_acquire+0x2af/0x310
> [ 4671.262376] [     T46]  ? __pfx_do_raw_spin_lock+0x10/0x10
> [ 4671.263186] [     T46]  ? lock_release+0x244/0x300
> [ 4671.263947] [     T46]  ? rcu_is_watching+0x11/0xb0
> [ 4671.264718] [     T46]  ? __pfx_io_wq_worker+0x10/0x10
> [ 4671.265529] [     T46]  ret_from_fork+0x30/0x70
> [ 4671.266277] [     T46]  ? __pfx_io_wq_worker+0x10/0x10
> [ 4671.267070] [     T46]  ret_from_fork_asm+0x1a/0x30
> [ 4671.267855] [     T46]  </TASK>
> [ 4671.268495] [     T46] INFO: lockdep is turned off.
> [ 4794.088765] [     T46] INFO: task iou-wrk-131061:131062 blocked for more than 245 seconds.
> [ 4794.090310] [     T46]       Not tainted 6.15.0-rc5 #317
> [ 4794.091453] [     T46] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 4794.093379] [     T46] task:iou-wrk-131061  state:D stack:0     pid:131062 tgid:131061 ppid:131060 task_flags:0x404150 flags:0x00004000
> [ 4794.095353] [     T46] Call Trace:
> [ 4794.096142] [     T46]  <TASK>
> [ 4794.096929] [     T46]  __schedule+0xf8c/0x5c80
> [ 4794.097878] [     T46]  ? rcu_is_watching+0x11/0xb0
> [ 4794.098869] [     T46]  ? __mutex_lock+0x2d9/0x1b50
> [ 4794.099866] [     T46]  ? rcu_is_watching+0x11/0xb0
> [ 4794.100870] [     T46]  ? lock_release+0x244/0x300
> [ 4794.101857] [     T46]  ? lock_release+0x244/0x300
> [ 4794.102797] [     T46]  ? __pfx___schedule+0x10/0x10
> [ 4794.103726] [     T46]  ? do_raw_spin_lock+0x124/0x260
> [ 4794.104752] [     T46]  ? rcu_is_watching+0x11/0xb0
> [ 4794.105744] [     T46]  ? rcu_is_watching+0x11/0xb0
> [ 4794.106678] [     T46]  ? lock_release+0x244/0x300
> [ 4794.107575] [     T46]  ? lock_release+0x244/0x300
> [ 4794.108528] [     T46]  schedule+0xda/0x390
> [ 4794.109391] [     T46]  blk_mq_freeze_queue_wait+0x121/0x170
> [ 4794.110437] [     T46]  ? __pfx_blk_mq_freeze_queue_wait+0x10/0x10
> [ 4794.111527] [     T46]  ? __pfx_autoremove_wake_function+0x10/0x10
> [ 4794.112662] [     T46]  ? trace_hardirqs_on+0x14/0x150
> [ 4794.113626] [     T46]  ? _raw_spin_unlock_irq+0x34/0x50
> [ 4794.114644] [     T46]  del_gendisk+0x4f9/0xa20
> [ 4794.115518] [     T46]  ? ublk_stop_dev+0x24/0x200 [ublk_drv]
> [ 4794.116548] [     T46]  ? __pfx_del_gendisk+0x10/0x10
> [ 4794.117536] [     T46]  ? __pfx___mutex_lock+0x10/0x10
> [ 4794.118521] [     T46]  ublk_stop_dev_unlocked.part.0+0xb6/0x580 [ublk_drv]
> [ 4794.119749] [     T46]  ? __pfx_ublk_stop_dev_unlocked.part.0+0x10/0x10 [ublk_drv]
> [ 4794.121060] [     T46]  ? kvm_sched_clock_read+0xd/0x20
> [ 4794.122081] [     T46]  ublk_stop_dev+0x62/0x200 [ublk_drv]
> [ 4794.123102] [     T46]  ? security_capable+0x79/0x110
> [ 4794.124139] [     T46]  ublk_ctrl_uring_cmd+0xe87/0x2a70 [ublk_drv]
> [ 4794.125225] [     T46]  ? __pfx_avc_has_perm+0x10/0x10
> [ 4794.126054] [     T46]  ? lock_release+0x244/0x300
> [ 4794.126850] [     T46]  ? rcu_is_watching+0x11/0xb0
> [ 4794.127579] [     T46]  ? __pfx_ublk_ctrl_uring_cmd+0x10/0x10 [ublk_drv]
> [ 4794.128512] [     T46]  ? finish_task_switch.isra.0+0x202/0x850
> [ 4794.129379] [     T46]  ? selinux_uring_cmd+0x1c3/0x280
> [ 4794.130178] [     T46]  ? __schedule+0x2f21/0x5c80
> [ 4794.130926] [     T46]  ? __pfx_selinux_uring_cmd+0x10/0x10
> [ 4794.131701] [     T46]  ? rcu_is_watching+0x11/0xb0
> [ 4794.132357] [     T46]  ? __pfx_do_raw_spin_lock+0x10/0x10
> [ 4794.133159] [     T46]  io_uring_cmd+0x221/0x3c0
> [ 4794.133931] [     T46]  io_issue_sqe+0x6ab/0x1800
> [ 4794.134653] [     T46]  ? trace_hardirqs_on+0x14/0x150
> [ 4794.135417] [     T46]  ? __pfx_io_issue_sqe+0x10/0x10
> [ 4794.136191] [     T46]  ? __timer_delete_sync+0x16c/0x270
> [ 4794.136984] [     T46]  ? __pfx___timer_delete_sync+0x10/0x10
> [ 4794.137833] [     T46]  ? rcu_is_watching+0x11/0xb0
> [ 4794.138527] [     T46]  io_wq_submit_work+0x3ea/0xf70
> [ 4794.139268] [     T46]  io_worker_handle_work+0x615/0x1280
> [ 4794.140062] [     T46]  io_wq_worker+0x2ee/0xbd0
> [ 4794.140806] [     T46]  ? __pfx_io_wq_worker+0x10/0x10
> [ 4794.141478] [     T46]  ? rcu_is_watching+0x11/0xb0
> [ 4794.142189] [     T46]  ? lock_acquire+0x2af/0x310
> [ 4794.142930] [     T46]  ? rcu_is_watching+0x11/0xb0
> [ 4794.143663] [     T46]  ? do_raw_spin_lock+0x124/0x260
> [ 4794.144363] [     T46]  ? lock_acquire+0x2af/0x310
> [ 4794.145087] [     T46]  ? __pfx_do_raw_spin_lock+0x10/0x10
> [ 4794.145897] [     T46]  ? lock_release+0x244/0x300
> [ 4794.146617] [     T46]  ? rcu_is_watching+0x11/0xb0
> [ 4794.147340] [     T46]  ? __pfx_io_wq_worker+0x10/0x10
> [ 4794.148070] [     T46]  ret_from_fork+0x30/0x70
> [ 4794.148817] [     T46]  ? __pfx_io_wq_worker+0x10/0x10
> [ 4794.149576] [     T46]  ret_from_fork_asm+0x1a/0x30
> [ 4794.150307] [     T46]  </TASK>
> [ 4794.150959] [     T46] INFO: lockdep is turned off.
> ...

Hi Shinichiro,

Are you referring to test_generic_02.sh in tools/testing/selftests/ublk?

I tried running it 20 times and it didn't get stuck

Kernel: 6.15.0-rc6
liburing: v2.9

Jared

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH V2 2/2] ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd
  2025-05-18 19:12     ` Jared Holzman
@ 2025-05-18 23:54       ` Shinichiro Kawasaki
  2025-05-19  2:45       ` Ming Lei
  1 sibling, 0 replies; 9+ messages in thread
From: Shinichiro Kawasaki @ 2025-05-18 23:54 UTC (permalink / raw)
  To: Jared Holzman
  Cc: Ming Lei, Jens Axboe, linux-block@vger.kernel.org, Uday Shankar,
	Caleb Sander Mateos, Guy Eisenberg, Yoav Cohen, Omri Levi,
	Ofer Oshri

On May 18, 2025 / 22:12, Jared Holzman wrote:
> On 14/05/2025 4:50, Shinichiro Kawasaki wrote:
> > On Apr 25, 2025 / 09:37, Ming Lei wrote:
> >> ublk_cancel_cmd() calls io_uring_cmd_done() to complete uring_cmd, but
> >> we may have scheduled task work via io_uring_cmd_complete_in_task() for
> >> dispatching request, then kernel crash can be triggered.
> >>
> >> Fix it by not trying to canceling the command if ublk block request is
> >> started.
> > 
> > I found that the blktests test case ublk/002 often hangs with the recent
> > v6.15-rcX kernel tags with the INFO at iou-wrk-X [1]. The hang is recreated
> > in stable manner when I repeat the test case a few times.
> > 
> > I bisected and this patch as the commit f40139fde527 triggers the hang.
> > When I reverted the commit from the kernel v6.15-rc6, the hang disappeared.
> > (I repeated ublk/002 20 times, and observed no hang.)
> > 
> > The hang was observed with test systems with Fedora 42. I do not observe
> > the hang with Fedora 41, but not sure why. liburing version difference
> > could be the reason (v2.6 for Fedora 41, v2.9 for Fedora 42).
> > 
> > Actions for fix will be appreciated.
> > 
> > [1]
> > 
> > [ 4497.777695] [ T130863] run blktests ublk/002 at 2025-05-07 14:48:32
> > [ 4499.983130] [  T67084] blk_print_req_error: 58 callbacks suppressed
> > [ 4499.983136] [  T67084] I/O error, dev ublkb0, sector 106830432 op 0x0:(READ) flags 0x800 phys_

[...]

> > ...
> 
> Hi Shinichiro,
> 
> Are you referring to test_generic_02.sh in tools/testing/selftests/ublk?

No. As I noted above, I refer to blktests [2] and its test case ublk/002 [3].

[2] https://github.com/osandov/blktests
[3] https://github.com/osandov/blktests/blob/master/tests/ublk/002

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH V2 2/2] ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd
  2025-05-18 19:12     ` Jared Holzman
  2025-05-18 23:54       ` Shinichiro Kawasaki
@ 2025-05-19  2:45       ` Ming Lei
  1 sibling, 0 replies; 9+ messages in thread
From: Ming Lei @ 2025-05-19  2:45 UTC (permalink / raw)
  To: Jared Holzman
  Cc: Shinichiro Kawasaki, Ming Lei, Jens Axboe,
	linux-block@vger.kernel.org, Uday Shankar, Caleb Sander Mateos,
	Guy Eisenberg, Yoav Cohen, Omri Levi, Ofer Oshri

On Sun, May 18, 2025 at 10:12:29PM +0300, Jared Holzman wrote:
> On 14/05/2025 4:50, Shinichiro Kawasaki wrote:

...

> 
> Hi Shinichiro,
> 
> Are you referring to test_generic_02.sh in tools/testing/selftests/ublk?
> 
> I tried running it 20 times and it didn't get stuck

It is supposed to be triggered in stress_02.sh, but turns out it isn't
stressful enough, and the hang can be duplicated with the following change:

- single queue

- random IO

- --numjobs="$(nproc)"

diff --git a/tools/testing/selftests/ublk/test_common.sh b/tools/testing/selftests/ublk/test_common.sh
index a81210ca3e99..c17fd66b73ac 100755
--- a/tools/testing/selftests/ublk/test_common.sh
+++ b/tools/testing/selftests/ublk/test_common.sh
@@ -251,7 +251,7 @@ __run_io_and_remove()
        local kill_server=$3

        fio --name=job1 --filename=/dev/ublkb"${dev_id}" --ioengine=libaio \
-               --rw=readwrite --iodepth=256 --size="${size}" --numjobs=4 \
+               --rw=randrw --norandommap --iodepth=256 --size="${size}" --numjobs="$(nproc)" \
                --runtime=20 --time_based > /dev/null 2>&1 &
        sleep 2
        if [ "${kill_server}" = "yes" ]; then
diff --git a/tools/testing/selftests/ublk/test_stress_02.sh b/tools/testing/selftests/ublk/test_stress_02.sh
index 1a9065125ae1..e532ca6bd569 100755
--- a/tools/testing/selftests/ublk/test_stress_02.sh
+++ b/tools/testing/selftests/ublk/test_stress_02.sh
@@ -30,5 +30,7 @@ ublk_io_and_kill_daemon 256M -t loop -q 4 "${UBLK_BACKFILES[0]}" &
 ublk_io_and_kill_daemon 256M -t stripe -q 4 "${UBLK_BACKFILES[1]}" "${UBLK_BACKFILES[2]}" &
 wait

+ublk_io_and_kill_daemon 8G -t null -q 1
+
 _cleanup_test "stress"
 _show_result $TID $ERR_CODE



Thanks,
Ming

^ permalink raw reply related	[flat|nested] 9+ messages in thread

* Re: [PATCH V2 0/2] ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd
  2025-04-25  1:37 [PATCH V2 0/2] ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd Ming Lei
  2025-04-25  1:37 ` [PATCH V2 1/2] ublk: call ublk_dispatch_req() for handling UBLK_U_IO_NEED_GET_DATA Ming Lei
  2025-04-25  1:37 ` [PATCH V2 2/2] ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd Ming Lei
@ 2025-04-25  1:53 ` Jens Axboe
  2025-04-28 13:31   ` Jared Holzman
  2 siblings, 1 reply; 9+ messages in thread
From: Jens Axboe @ 2025-04-25  1:53 UTC (permalink / raw)
  To: linux-block, Ming Lei
  Cc: Uday Shankar, Caleb Sander Mateos, Guy Eisenberg, Jared Holzman,
	Yoav Cohen, Omri Levi, Ofer Oshri


On Fri, 25 Apr 2025 09:37:38 +0800, Ming Lei wrote:
> The 2 patches try to fix race between between io_uring_cmd_complete_in_task
> and ublk_cancel_cmd.
> 
> Thanks,
> Ming
> 
> V2:
> 	- improve comment and commit log
> 	- remove useless memory barrier(Caleb Sander Mateos)
> 	- add tested-by fixes tag
> 
> [...]

Applied, thanks!

[1/2] ublk: call ublk_dispatch_req() for handling UBLK_U_IO_NEED_GET_DATA
      commit: d6aa0c178bf81f30ae4a780b2bca653daa2eb633
[2/2] ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd
      commit: f40139fde5278d81af3227444fd6e76a76b9506d

Best regards,
-- 
Jens Axboe




^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH V2 0/2] ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd
  2025-04-25  1:53 ` [PATCH V2 0/2] " Jens Axboe
@ 2025-04-28 13:31   ` Jared Holzman
  0 siblings, 0 replies; 9+ messages in thread
From: Jared Holzman @ 2025-04-28 13:31 UTC (permalink / raw)
  To: Jens Axboe, linux-block, Ming Lei
  Cc: Uday Shankar, Caleb Sander Mateos, Guy Eisenberg, Yoav Cohen,
	Omri Levi, Ofer Oshri

On 25/04/2025 4:53, Jens Axboe wrote:
> 
> On Fri, 25 Apr 2025 09:37:38 +0800, Ming Lei wrote:
>> The 2 patches try to fix race between between io_uring_cmd_complete_in_task
>> and ublk_cancel_cmd.
>>
>> Thanks,
>> Ming
>>
>> V2:
>> 	- improve comment and commit log
>> 	- remove useless memory barrier(Caleb Sander Mateos)
>> 	- add tested-by fixes tag
>>
>> [...]
> 
> Applied, thanks!
> 
> [1/2] ublk: call ublk_dispatch_req() for handling UBLK_U_IO_NEED_GET_DATA
>       commit: d6aa0c178bf81f30ae4a780b2bca653daa2eb633
> [2/2] ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd
>       commit: f40139fde5278d81af3227444fd6e76a76b9506d
> 
> Best regards,

Confirmed that this version also fixes the issue for us.

Thanks,
Jared

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2025-05-19  2:45 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-04-25  1:37 [PATCH V2 0/2] ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd Ming Lei
2025-04-25  1:37 ` [PATCH V2 1/2] ublk: call ublk_dispatch_req() for handling UBLK_U_IO_NEED_GET_DATA Ming Lei
2025-04-25  1:37 ` [PATCH V2 2/2] ublk: fix race between io_uring_cmd_complete_in_task and ublk_cancel_cmd Ming Lei
2025-05-14  1:50   ` Shinichiro Kawasaki
2025-05-18 19:12     ` Jared Holzman
2025-05-18 23:54       ` Shinichiro Kawasaki
2025-05-19  2:45       ` Ming Lei
2025-04-25  1:53 ` [PATCH V2 0/2] " Jens Axboe
2025-04-28 13:31   ` Jared Holzman

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox