* [PATCH-next 0/2] dm: fix quiesce asymmetry and null pointer in suspend/load race
@ 2025-08-26 7:42 Zheng Qixing
2025-08-26 7:42 ` [PATCH-next 1/2] dm: fix queue start/stop imbalance under suspend/load/resume races Zheng Qixing
` (2 more replies)
0 siblings, 3 replies; 4+ messages in thread
From: Zheng Qixing @ 2025-08-26 7:42 UTC (permalink / raw)
To: agk, snitzer, mpatocka, axboe, ming.lei
Cc: dm-devel, linux-kernel, yukuai3, yi.zhang, yangerkun, houtao1,
zhengqixing
From: Zheng Qixing <zhengqixing@huawei.com>
Hello,
During reliability testing of the dm module, we discovered two bugs when
performing create/remove and suspend/resume operations at high frequency.
Analysis revealed that race conditions between suspend and table load
operations cause asymmetric handling of q->quiesce_depth and null pointer
dereference issues with q->tag_set.
To address the q->tag_set null pointer issue, patch 2 determines whether
to perform certain suspend actions for request-based devices based on the
existence of the table (map).
For detailed information, please refer to the patch commits.
Additionally, since modern Linux kernels have indeed transitioned to
multi-queue as the default (and single-queue is largely deprecated),
is using queue_is_mq() to determine dm_request_based() obsolete?
Zheng Qixing (2):
dm: fix queue start/stop imbalance under suspend/load/resume races
dm: fix NULL pointer dereference in __dm_suspend()
drivers/md/dm-core.h | 1 +
drivers/md/dm.c | 13 ++++++++-----
2 files changed, 9 insertions(+), 5 deletions(-)
--
2.39.2
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH-next 1/2] dm: fix queue start/stop imbalance under suspend/load/resume races
2025-08-26 7:42 [PATCH-next 0/2] dm: fix quiesce asymmetry and null pointer in suspend/load race Zheng Qixing
@ 2025-08-26 7:42 ` Zheng Qixing
2025-08-26 7:42 ` [PATCH-next 2/2] dm: fix NULL pointer dereference in __dm_suspend() Zheng Qixing
2025-08-26 14:50 ` [PATCH-next 0/2] dm: fix quiesce asymmetry and null pointer in suspend/load race Mikulas Patocka
2 siblings, 0 replies; 4+ messages in thread
From: Zheng Qixing @ 2025-08-26 7:42 UTC (permalink / raw)
To: agk, snitzer, mpatocka, axboe, ming.lei
Cc: dm-devel, linux-kernel, yukuai3, yi.zhang, yangerkun, houtao1,
zhengqixing
From: Zheng Qixing <zhengqixing@huawei.com>
When suspend and load run concurrently, before q->mq_ops is set in
blk_mq_init_allocated_queue(), __dm_suspend() skip dm_stop_queue(). As a
result, the queue's quiesce depth is not incremented.
Later, once table load has finished and __dm_resume() runs, which triggers
q->quiesce_depth ==0 warning in blk_mq_unquiesce_queue():
Call Trace:
<TASK>
dm_start_queue+0x16/0x20 [dm_mod]
__dm_resume+0xac/0xb0 [dm_mod]
dm_resume+0x12d/0x150 [dm_mod]
do_resume+0x2c2/0x420 [dm_mod]
dev_suspend+0x30/0x130 [dm_mod]
ctl_ioctl+0x402/0x570 [dm_mod]
dm_ctl_ioctl+0x23/0x30 [dm_mod]
Fix this by explicitly tracking whether the request queue was
stopped in __dm_suspend() via a new DMF_QUEUE_STOPPED flag.
Only call dm_start_queue() in __dm_resume() if the queue was
actually stopped.
Fixes: e70feb8b3e68 ("blk-mq: support concurrent queue quiesce/unquiesce")
Signed-off-by: Zheng Qixing <zhengqixing@huawei.com>
---
drivers/md/dm-core.h | 1 +
drivers/md/dm.c | 8 +++++---
2 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/drivers/md/dm-core.h b/drivers/md/dm-core.h
index c889332e533b..0070e4462ee2 100644
--- a/drivers/md/dm-core.h
+++ b/drivers/md/dm-core.h
@@ -162,6 +162,7 @@ struct mapped_device {
#define DMF_SUSPENDED_INTERNALLY 7
#define DMF_POST_SUSPENDING 8
#define DMF_EMULATE_ZONE_APPEND 9
+#define DMF_QUEUE_STOPPED 10
static inline sector_t dm_get_size(struct mapped_device *md)
{
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index a44e8c2dccee..7222f20c1a83 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -2960,8 +2960,10 @@ static int __dm_suspend(struct mapped_device *md, struct dm_table *map,
* Stop md->queue before flushing md->wq in case request-based
* dm defers requests to md->wq from md->queue.
*/
- if (dm_request_based(md))
+ if (dm_request_based(md)) {
dm_stop_queue(md->queue);
+ set_bit(DMF_QUEUE_STOPPED, &md->flags);
+ }
flush_workqueue(md->wq);
@@ -2983,7 +2985,7 @@ static int __dm_suspend(struct mapped_device *md, struct dm_table *map,
if (r < 0) {
dm_queue_flush(md);
- if (dm_request_based(md))
+ if (test_and_clear_bit(DMF_QUEUE_STOPPED, &md->flags))
dm_start_queue(md->queue);
unlock_fs(md);
@@ -3067,7 +3069,7 @@ static int __dm_resume(struct mapped_device *md, struct dm_table *map)
* so that mapping of targets can work correctly.
* Request-based dm is queueing the deferred I/Os in its request_queue.
*/
- if (dm_request_based(md))
+ if (test_and_clear_bit(DMF_QUEUE_STOPPED, &md->flags))
dm_start_queue(md->queue);
unlock_fs(md);
--
2.39.2
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [PATCH-next 2/2] dm: fix NULL pointer dereference in __dm_suspend()
2025-08-26 7:42 [PATCH-next 0/2] dm: fix quiesce asymmetry and null pointer in suspend/load race Zheng Qixing
2025-08-26 7:42 ` [PATCH-next 1/2] dm: fix queue start/stop imbalance under suspend/load/resume races Zheng Qixing
@ 2025-08-26 7:42 ` Zheng Qixing
2025-08-26 14:50 ` [PATCH-next 0/2] dm: fix quiesce asymmetry and null pointer in suspend/load race Mikulas Patocka
2 siblings, 0 replies; 4+ messages in thread
From: Zheng Qixing @ 2025-08-26 7:42 UTC (permalink / raw)
To: agk, snitzer, mpatocka, axboe, ming.lei
Cc: dm-devel, linux-kernel, yukuai3, yi.zhang, yangerkun, houtao1,
zhengqixing
From: Zheng Qixing <zhengqixing@huawei.com>
There is a race condition between dm device suspend and table load that
can lead to null pointer dereference. The issue occurs when suspend is
invoked before table load completes:
BUG: kernel NULL pointer dereference, address: 0000000000000054
Oops: 0000 [#1] PREEMPT SMP PTI
CPU: 6 PID: 6798 Comm: dmsetup Not tainted 6.6.0-g7e52f5f0ca9b #62
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.1-2.fc37 04/01/2014
RIP: 0010:blk_mq_wait_quiesce_done+0x0/0x50
Call Trace:
<TASK>
blk_mq_quiesce_queue+0x2c/0x50
dm_stop_queue+0xd/0x20
__dm_suspend+0x130/0x330
dm_suspend+0x11a/0x180
dev_suspend+0x27e/0x560
ctl_ioctl+0x4cf/0x850
dm_ctl_ioctl+0xd/0x20
vfs_ioctl+0x1d/0x50
__se_sys_ioctl+0x9b/0xc0
__x64_sys_ioctl+0x19/0x30
x64_sys_call+0x2c4a/0x4620
do_syscall_64+0x9e/0x1b0
The issue can be triggered as below:
T1 T2
dm_suspend table_load
__dm_suspend dm_setup_md_queue
dm_mq_init_request_queue
blk_mq_init_allocated_queue
=> q->mq_ops = set->ops; (1)
dm_stop_queue / dm_wait_for_completion
=> q->tag_set NULL pointer! (2)
=> q->tag_set = set; (3)
Fix this by checking if a valid table (map) exists before performing
request-based suspend and waiting for target I/O. When map is NULL,
skip these table-dependent suspend steps.
Even when map is NULL, no I/O can reach any target because there is
no table loaded; I/O submitted in this state will fail early in the
DM layer. Skipping the table-dependent suspend logic in this case
is safe and avoids NULL pointer dereferences.
Fixes: c4576aed8d85 ("dm: fix request-based dm's use of dm_wait_for_completion")
Signed-off-by: Zheng Qixing <zhengqixing@huawei.com>
---
drivers/md/dm.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 7222f20c1a83..66dd5f6ce778 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -2908,7 +2908,7 @@ static int __dm_suspend(struct mapped_device *md, struct dm_table *map,
{
bool do_lockfs = suspend_flags & DM_SUSPEND_LOCKFS_FLAG;
bool noflush = suspend_flags & DM_SUSPEND_NOFLUSH_FLAG;
- int r;
+ int r = 0;
lockdep_assert_held(&md->suspend_lock);
@@ -2960,7 +2960,7 @@ static int __dm_suspend(struct mapped_device *md, struct dm_table *map,
* Stop md->queue before flushing md->wq in case request-based
* dm defers requests to md->wq from md->queue.
*/
- if (dm_request_based(md)) {
+ if (map && dm_request_based(md)) {
dm_stop_queue(md->queue);
set_bit(DMF_QUEUE_STOPPED, &md->flags);
}
@@ -2972,7 +2972,8 @@ static int __dm_suspend(struct mapped_device *md, struct dm_table *map,
* We call dm_wait_for_completion to wait for all existing requests
* to finish.
*/
- r = dm_wait_for_completion(md, task_state);
+ if (map)
+ r = dm_wait_for_completion(md, task_state);
if (!r)
set_bit(dmf_suspended_flag, &md->flags);
--
2.39.2
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH-next 0/2] dm: fix quiesce asymmetry and null pointer in suspend/load race
2025-08-26 7:42 [PATCH-next 0/2] dm: fix quiesce asymmetry and null pointer in suspend/load race Zheng Qixing
2025-08-26 7:42 ` [PATCH-next 1/2] dm: fix queue start/stop imbalance under suspend/load/resume races Zheng Qixing
2025-08-26 7:42 ` [PATCH-next 2/2] dm: fix NULL pointer dereference in __dm_suspend() Zheng Qixing
@ 2025-08-26 14:50 ` Mikulas Patocka
2 siblings, 0 replies; 4+ messages in thread
From: Mikulas Patocka @ 2025-08-26 14:50 UTC (permalink / raw)
To: Zheng Qixing
Cc: agk, snitzer, axboe, ming.lei, dm-devel, linux-kernel, yukuai3,
yi.zhang, yangerkun, houtao1, zhengqixing
On Tue, 26 Aug 2025, Zheng Qixing wrote:
> From: Zheng Qixing <zhengqixing@huawei.com>
>
> Hello,
>
> During reliability testing of the dm module, we discovered two bugs when
> performing create/remove and suspend/resume operations at high frequency.
>
> Analysis revealed that race conditions between suspend and table load
> operations cause asymmetric handling of q->quiesce_depth and null pointer
> dereference issues with q->tag_set.
>
> To address the q->tag_set null pointer issue, patch 2 determines whether
> to perform certain suspend actions for request-based devices based on the
> existence of the table (map).
>
> For detailed information, please refer to the patch commits.
>
> Additionally, since modern Linux kernels have indeed transitioned to
> multi-queue as the default (and single-queue is largely deprecated),
> is using queue_is_mq() to determine dm_request_based() obsolete?
>
> Zheng Qixing (2):
> dm: fix queue start/stop imbalance under suspend/load/resume races
> dm: fix NULL pointer dereference in __dm_suspend()
>
> drivers/md/dm-core.h | 1 +
> drivers/md/dm.c | 13 ++++++++-----
> 2 files changed, 9 insertions(+), 5 deletions(-)
>
> --
> 2.39.2
Both patches applied, thanks.
Mikulas
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2025-08-26 14:50 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-08-26 7:42 [PATCH-next 0/2] dm: fix quiesce asymmetry and null pointer in suspend/load race Zheng Qixing
2025-08-26 7:42 ` [PATCH-next 1/2] dm: fix queue start/stop imbalance under suspend/load/resume races Zheng Qixing
2025-08-26 7:42 ` [PATCH-next 2/2] dm: fix NULL pointer dereference in __dm_suspend() Zheng Qixing
2025-08-26 14:50 ` [PATCH-next 0/2] dm: fix quiesce asymmetry and null pointer in suspend/load race Mikulas Patocka
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).