dm-devel.redhat.com archive mirror
 help / color / mirror / Atom feed
* [PATCH-next 0/2] dm: fix quiesce asymmetry and null pointer in suspend/load race
@ 2025-08-26  7:42 Zheng Qixing
  2025-08-26  7:42 ` [PATCH-next 1/2] dm: fix queue start/stop imbalance under suspend/load/resume races Zheng Qixing
                   ` (2 more replies)
  0 siblings, 3 replies; 4+ messages in thread
From: Zheng Qixing @ 2025-08-26  7:42 UTC (permalink / raw)
  To: agk, snitzer, mpatocka, axboe, ming.lei
  Cc: dm-devel, linux-kernel, yukuai3, yi.zhang, yangerkun, houtao1,
	zhengqixing

From: Zheng Qixing <zhengqixing@huawei.com>

Hello,

During reliability testing of the dm module, we discovered two bugs when
performing create/remove and suspend/resume operations at high frequency.

Analysis revealed that race conditions between suspend and table load
operations cause asymmetric handling of q->quiesce_depth and null pointer
dereference issues with q->tag_set.

To address the q->tag_set null pointer issue, patch 2 determines whether
to perform certain suspend actions for request-based devices based on the
existence of the table (map).

For detailed information, please refer to the patch commits.

Additionally, since modern Linux kernels have indeed transitioned to
multi-queue as the default (and single-queue is largely deprecated),
is using queue_is_mq() to determine dm_request_based() obsolete?

Zheng Qixing (2):
  dm: fix queue start/stop imbalance under suspend/load/resume races
  dm: fix NULL pointer dereference in __dm_suspend()

 drivers/md/dm-core.h |  1 +
 drivers/md/dm.c      | 13 ++++++++-----
 2 files changed, 9 insertions(+), 5 deletions(-)

-- 
2.39.2


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH-next 1/2] dm: fix queue start/stop imbalance under suspend/load/resume races
  2025-08-26  7:42 [PATCH-next 0/2] dm: fix quiesce asymmetry and null pointer in suspend/load race Zheng Qixing
@ 2025-08-26  7:42 ` Zheng Qixing
  2025-08-26  7:42 ` [PATCH-next 2/2] dm: fix NULL pointer dereference in __dm_suspend() Zheng Qixing
  2025-08-26 14:50 ` [PATCH-next 0/2] dm: fix quiesce asymmetry and null pointer in suspend/load race Mikulas Patocka
  2 siblings, 0 replies; 4+ messages in thread
From: Zheng Qixing @ 2025-08-26  7:42 UTC (permalink / raw)
  To: agk, snitzer, mpatocka, axboe, ming.lei
  Cc: dm-devel, linux-kernel, yukuai3, yi.zhang, yangerkun, houtao1,
	zhengqixing

From: Zheng Qixing <zhengqixing@huawei.com>

When suspend and load run concurrently, before q->mq_ops is set in
blk_mq_init_allocated_queue(), __dm_suspend() skip dm_stop_queue(). As a
result, the queue's quiesce depth is not incremented.

Later, once table load has finished and __dm_resume() runs, which triggers
q->quiesce_depth ==0 warning in blk_mq_unquiesce_queue():
Call Trace:
 <TASK>
 dm_start_queue+0x16/0x20 [dm_mod]
 __dm_resume+0xac/0xb0 [dm_mod]
 dm_resume+0x12d/0x150 [dm_mod]
 do_resume+0x2c2/0x420 [dm_mod]
 dev_suspend+0x30/0x130 [dm_mod]
 ctl_ioctl+0x402/0x570 [dm_mod]
 dm_ctl_ioctl+0x23/0x30 [dm_mod]

Fix this by explicitly tracking whether the request queue was
stopped in __dm_suspend() via a new DMF_QUEUE_STOPPED flag.
Only call dm_start_queue() in __dm_resume() if the queue was
actually stopped.

Fixes: e70feb8b3e68 ("blk-mq: support concurrent queue quiesce/unquiesce")
Signed-off-by: Zheng Qixing <zhengqixing@huawei.com>
---
 drivers/md/dm-core.h | 1 +
 drivers/md/dm.c      | 8 +++++---
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/drivers/md/dm-core.h b/drivers/md/dm-core.h
index c889332e533b..0070e4462ee2 100644
--- a/drivers/md/dm-core.h
+++ b/drivers/md/dm-core.h
@@ -162,6 +162,7 @@ struct mapped_device {
 #define DMF_SUSPENDED_INTERNALLY 7
 #define DMF_POST_SUSPENDING 8
 #define DMF_EMULATE_ZONE_APPEND 9
+#define DMF_QUEUE_STOPPED 10
 
 static inline sector_t dm_get_size(struct mapped_device *md)
 {
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index a44e8c2dccee..7222f20c1a83 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -2960,8 +2960,10 @@ static int __dm_suspend(struct mapped_device *md, struct dm_table *map,
 	 * Stop md->queue before flushing md->wq in case request-based
 	 * dm defers requests to md->wq from md->queue.
 	 */
-	if (dm_request_based(md))
+	if (dm_request_based(md)) {
 		dm_stop_queue(md->queue);
+		set_bit(DMF_QUEUE_STOPPED, &md->flags);
+	}
 
 	flush_workqueue(md->wq);
 
@@ -2983,7 +2985,7 @@ static int __dm_suspend(struct mapped_device *md, struct dm_table *map,
 	if (r < 0) {
 		dm_queue_flush(md);
 
-		if (dm_request_based(md))
+		if (test_and_clear_bit(DMF_QUEUE_STOPPED, &md->flags))
 			dm_start_queue(md->queue);
 
 		unlock_fs(md);
@@ -3067,7 +3069,7 @@ static int __dm_resume(struct mapped_device *md, struct dm_table *map)
 	 * so that mapping of targets can work correctly.
 	 * Request-based dm is queueing the deferred I/Os in its request_queue.
 	 */
-	if (dm_request_based(md))
+	if (test_and_clear_bit(DMF_QUEUE_STOPPED, &md->flags))
 		dm_start_queue(md->queue);
 
 	unlock_fs(md);
-- 
2.39.2


^ permalink raw reply related	[flat|nested] 4+ messages in thread

* [PATCH-next 2/2] dm: fix NULL pointer dereference in __dm_suspend()
  2025-08-26  7:42 [PATCH-next 0/2] dm: fix quiesce asymmetry and null pointer in suspend/load race Zheng Qixing
  2025-08-26  7:42 ` [PATCH-next 1/2] dm: fix queue start/stop imbalance under suspend/load/resume races Zheng Qixing
@ 2025-08-26  7:42 ` Zheng Qixing
  2025-08-26 14:50 ` [PATCH-next 0/2] dm: fix quiesce asymmetry and null pointer in suspend/load race Mikulas Patocka
  2 siblings, 0 replies; 4+ messages in thread
From: Zheng Qixing @ 2025-08-26  7:42 UTC (permalink / raw)
  To: agk, snitzer, mpatocka, axboe, ming.lei
  Cc: dm-devel, linux-kernel, yukuai3, yi.zhang, yangerkun, houtao1,
	zhengqixing

From: Zheng Qixing <zhengqixing@huawei.com>

There is a race condition between dm device suspend and table load that
can lead to null pointer dereference. The issue occurs when suspend is
invoked before table load completes:

BUG: kernel NULL pointer dereference, address: 0000000000000054
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 6 PID: 6798 Comm: dmsetup Not tainted 6.6.0-g7e52f5f0ca9b #62
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.1-2.fc37 04/01/2014
RIP: 0010:blk_mq_wait_quiesce_done+0x0/0x50
Call Trace:
  <TASK>
  blk_mq_quiesce_queue+0x2c/0x50
  dm_stop_queue+0xd/0x20
  __dm_suspend+0x130/0x330
  dm_suspend+0x11a/0x180
  dev_suspend+0x27e/0x560
  ctl_ioctl+0x4cf/0x850
  dm_ctl_ioctl+0xd/0x20
  vfs_ioctl+0x1d/0x50
  __se_sys_ioctl+0x9b/0xc0
  __x64_sys_ioctl+0x19/0x30
  x64_sys_call+0x2c4a/0x4620
  do_syscall_64+0x9e/0x1b0

The issue can be triggered as below:

T1 						T2
dm_suspend					table_load
__dm_suspend					dm_setup_md_queue
						dm_mq_init_request_queue
						blk_mq_init_allocated_queue
						=> q->mq_ops = set->ops; (1)
dm_stop_queue / dm_wait_for_completion
=> q->tag_set NULL pointer!	(2)
						=> q->tag_set = set; (3)

Fix this by checking if a valid table (map) exists before performing
request-based suspend and waiting for target I/O. When map is NULL,
skip these table-dependent suspend steps.

Even when map is NULL, no I/O can reach any target because there is
no table loaded; I/O submitted in this state will fail early in the
DM layer. Skipping the table-dependent suspend logic in this case
is safe and avoids NULL pointer dereferences.

Fixes: c4576aed8d85 ("dm: fix request-based dm's use of dm_wait_for_completion")
Signed-off-by: Zheng Qixing <zhengqixing@huawei.com>
---
 drivers/md/dm.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 7222f20c1a83..66dd5f6ce778 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -2908,7 +2908,7 @@ static int __dm_suspend(struct mapped_device *md, struct dm_table *map,
 {
 	bool do_lockfs = suspend_flags & DM_SUSPEND_LOCKFS_FLAG;
 	bool noflush = suspend_flags & DM_SUSPEND_NOFLUSH_FLAG;
-	int r;
+	int r = 0;
 
 	lockdep_assert_held(&md->suspend_lock);
 
@@ -2960,7 +2960,7 @@ static int __dm_suspend(struct mapped_device *md, struct dm_table *map,
 	 * Stop md->queue before flushing md->wq in case request-based
 	 * dm defers requests to md->wq from md->queue.
 	 */
-	if (dm_request_based(md)) {
+	if (map && dm_request_based(md)) {
 		dm_stop_queue(md->queue);
 		set_bit(DMF_QUEUE_STOPPED, &md->flags);
 	}
@@ -2972,7 +2972,8 @@ static int __dm_suspend(struct mapped_device *md, struct dm_table *map,
 	 * We call dm_wait_for_completion to wait for all existing requests
 	 * to finish.
 	 */
-	r = dm_wait_for_completion(md, task_state);
+	if (map)
+		r = dm_wait_for_completion(md, task_state);
 	if (!r)
 		set_bit(dmf_suspended_flag, &md->flags);
 
-- 
2.39.2


^ permalink raw reply related	[flat|nested] 4+ messages in thread

* Re: [PATCH-next 0/2] dm: fix quiesce asymmetry and null pointer in suspend/load race
  2025-08-26  7:42 [PATCH-next 0/2] dm: fix quiesce asymmetry and null pointer in suspend/load race Zheng Qixing
  2025-08-26  7:42 ` [PATCH-next 1/2] dm: fix queue start/stop imbalance under suspend/load/resume races Zheng Qixing
  2025-08-26  7:42 ` [PATCH-next 2/2] dm: fix NULL pointer dereference in __dm_suspend() Zheng Qixing
@ 2025-08-26 14:50 ` Mikulas Patocka
  2 siblings, 0 replies; 4+ messages in thread
From: Mikulas Patocka @ 2025-08-26 14:50 UTC (permalink / raw)
  To: Zheng Qixing
  Cc: agk, snitzer, axboe, ming.lei, dm-devel, linux-kernel, yukuai3,
	yi.zhang, yangerkun, houtao1, zhengqixing



On Tue, 26 Aug 2025, Zheng Qixing wrote:

> From: Zheng Qixing <zhengqixing@huawei.com>
> 
> Hello,
> 
> During reliability testing of the dm module, we discovered two bugs when
> performing create/remove and suspend/resume operations at high frequency.
> 
> Analysis revealed that race conditions between suspend and table load
> operations cause asymmetric handling of q->quiesce_depth and null pointer
> dereference issues with q->tag_set.
> 
> To address the q->tag_set null pointer issue, patch 2 determines whether
> to perform certain suspend actions for request-based devices based on the
> existence of the table (map).
> 
> For detailed information, please refer to the patch commits.
> 
> Additionally, since modern Linux kernels have indeed transitioned to
> multi-queue as the default (and single-queue is largely deprecated),
> is using queue_is_mq() to determine dm_request_based() obsolete?
> 
> Zheng Qixing (2):
>   dm: fix queue start/stop imbalance under suspend/load/resume races
>   dm: fix NULL pointer dereference in __dm_suspend()
> 
>  drivers/md/dm-core.h |  1 +
>  drivers/md/dm.c      | 13 ++++++++-----
>  2 files changed, 9 insertions(+), 5 deletions(-)
> 
> -- 
> 2.39.2

Both patches applied, thanks.

Mikulas


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2025-08-26 14:50 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-08-26  7:42 [PATCH-next 0/2] dm: fix quiesce asymmetry and null pointer in suspend/load race Zheng Qixing
2025-08-26  7:42 ` [PATCH-next 1/2] dm: fix queue start/stop imbalance under suspend/load/resume races Zheng Qixing
2025-08-26  7:42 ` [PATCH-next 2/2] dm: fix NULL pointer dereference in __dm_suspend() Zheng Qixing
2025-08-26 14:50 ` [PATCH-next 0/2] dm: fix quiesce asymmetry and null pointer in suspend/load race Mikulas Patocka

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).