[PATCH] md: wake raid456 reshape waiters before suspend

[PATCH] md: wake raid456 reshape waiters before suspend

[Yu Kuai]

During raid456 reshape, direct IO across the reshape position can sleep
in raid5_make_request() waiting for reshape progress while still
holding an active_io reference. If userspace then freezes reshape and
writes md/suspend_lo or md/suspend_hi, mddev_suspend() kills active_io
and waits for all in-flight IO to drain.

This can deadlock: the IO needs reshape progress to continue, but the
reshape thread is already frozen, so the active_io reference is never
dropped and suspend never completes.

raid5_prepare_suspend() already wakes wait_for_reshape for dm-raid. Do
the same for normal md suspend when reshape is already interrupted, so
waiting raid456 IO can abort, drop its reference, and let suspend
finish.

The mdadm test tests/25raid456-reshape-deadlock reproduces the hang.

Fixes: 714d20150ed8 ("md: add new helpers to suspend/resume array")
Signed-off-by: Yu Kuai <yukuai@fnnas.com>
---
 drivers/md/md.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/md/md.c b/drivers/md/md.c
index 65c85973092a..3ae5f36b5baf 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -489,6 +489,17 @@ int mddev_suspend(struct mddev *mddev, bool interruptible)
 	}
 
 	percpu_ref_kill(&mddev->active_io);
+
+	/*
+	 * RAID456 IO can sleep in wait_for_reshape while still holding an
+	 * active_io reference. If reshape is already interrupted or frozen,
+	 * wake those waiters so they can abort and drop the reference instead
+	 * of deadlocking suspend.
+	 */
+	if (mddev->pers && mddev->pers->prepare_suspend &&
+	    reshape_interrupted(mddev))
+		mddev->pers->prepare_suspend(mddev);
+
 	if (interruptible)
 		err = wait_event_interruptible(mddev->sb_wait,
 				percpu_ref_is_zero(&mddev->active_io));
-- 
2.51.0

Re: [PATCH] md: wake raid456 reshape waiters before suspend

[Li Nan]

在 2026/3/27 22:07, Yu Kuai 写道:
> During raid456 reshape, direct IO across the reshape position can sleep
> in raid5_make_request() waiting for reshape progress while still
> holding an active_io reference. If userspace then freezes reshape and
> writes md/suspend_lo or md/suspend_hi, mddev_suspend() kills active_io
> and waits for all in-flight IO to drain.
> 
> This can deadlock: the IO needs reshape progress to continue, but the
> reshape thread is already frozen, so the active_io reference is never
> dropped and suspend never completes.
> 
> raid5_prepare_suspend() already wakes wait_for_reshape for dm-raid. Do
> the same for normal md suspend when reshape is already interrupted, so
> waiting raid456 IO can abort, drop its reference, and let suspend
> finish.
> 
> The mdadm test tests/25raid456-reshape-deadlock reproduces the hang.
> 
> Fixes: 714d20150ed8 ("md: add new helpers to suspend/resume array")
> Signed-off-by: Yu Kuai <yukuai@fnnas.com>
> ---
>   drivers/md/md.c | 11 +++++++++++
>   1 file changed, 11 insertions(+)
> 
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 65c85973092a..3ae5f36b5baf 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -489,6 +489,17 @@ int mddev_suspend(struct mddev *mddev, bool interruptible)
>   	}
>   
>   	percpu_ref_kill(&mddev->active_io);
> +
> +	/*
> +	 * RAID456 IO can sleep in wait_for_reshape while still holding an
> +	 * active_io reference. If reshape is already interrupted or frozen,
> +	 * wake those waiters so they can abort and drop the reference instead
> +	 * of deadlocking suspend.
> +	 */
> +	if (mddev->pers && mddev->pers->prepare_suspend &&
> +	    reshape_interrupted(mddev))
> +		mddev->pers->prepare_suspend(mddev);
> +
>   	if (interruptible)
>   		err = wait_event_interruptible(mddev->sb_wait,
>   				percpu_ref_is_zero(&mddev->active_io));

If I only freeze raid5 without suspend, the IO still hangs and cannot
complete. Should we fix this scenario?

-- 
Thanks,
Nan

Re: [PATCH] md: wake raid456 reshape waiters before suspend

[Yu Kuai]

Hi,

在 2026/4/2 10:53, Li Nan 写道:
>
>
> 在 2026/3/27 22:07, Yu Kuai 写道:
>> During raid456 reshape, direct IO across the reshape position can sleep
>> in raid5_make_request() waiting for reshape progress while still
>> holding an active_io reference. If userspace then freezes reshape and
>> writes md/suspend_lo or md/suspend_hi, mddev_suspend() kills active_io
>> and waits for all in-flight IO to drain.
>>
>> This can deadlock: the IO needs reshape progress to continue, but the
>> reshape thread is already frozen, so the active_io reference is never
>> dropped and suspend never completes.
>>
>> raid5_prepare_suspend() already wakes wait_for_reshape for dm-raid. Do
>> the same for normal md suspend when reshape is already interrupted, so
>> waiting raid456 IO can abort, drop its reference, and let suspend
>> finish.
>>
>> The mdadm test tests/25raid456-reshape-deadlock reproduces the hang.
>>
>> Fixes: 714d20150ed8 ("md: add new helpers to suspend/resume array")
>> Signed-off-by: Yu Kuai <yukuai@fnnas.com>
>> ---
>>   drivers/md/md.c | 11 +++++++++++
>>   1 file changed, 11 insertions(+)
>>
>> diff --git a/drivers/md/md.c b/drivers/md/md.c
>> index 65c85973092a..3ae5f36b5baf 100644
>> --- a/drivers/md/md.c
>> +++ b/drivers/md/md.c
>> @@ -489,6 +489,17 @@ int mddev_suspend(struct mddev *mddev, bool 
>> interruptible)
>>       }
>>         percpu_ref_kill(&mddev->active_io);
>> +
>> +    /*
>> +     * RAID456 IO can sleep in wait_for_reshape while still holding an
>> +     * active_io reference. If reshape is already interrupted or 
>> frozen,
>> +     * wake those waiters so they can abort and drop the reference 
>> instead
>> +     * of deadlocking suspend.
>> +     */
>> +    if (mddev->pers && mddev->pers->prepare_suspend &&
>> +        reshape_interrupted(mddev))
>> +        mddev->pers->prepare_suspend(mddev);
>> +
>>       if (interruptible)
>>           err = wait_event_interruptible(mddev->sb_wait,
>> percpu_ref_is_zero(&mddev->active_io));
>
> If I only freeze raid5 without suspend, the IO still hangs and cannot
> complete. Should we fix this scenario?
Yes, for other path can freeze raid5 directly, we should fix it as well.
>
> -- 
> Thanks,
> Nan
>
-- 
Thansk,
Kuai

Re: [PATCH] md: wake raid456 reshape waiters before suspend

[Yu Kuai]

在 2026/3/27 22:07, Yu Kuai 写道:

> During raid456 reshape, direct IO across the reshape position can sleep
> in raid5_make_request() waiting for reshape progress while still
> holding an active_io reference. If userspace then freezes reshape and
> writes md/suspend_lo or md/suspend_hi, mddev_suspend() kills active_io
> and waits for all in-flight IO to drain.
>
> This can deadlock: the IO needs reshape progress to continue, but the
> reshape thread is already frozen, so the active_io reference is never
> dropped and suspend never completes.
>
> raid5_prepare_suspend() already wakes wait_for_reshape for dm-raid. Do
> the same for normal md suspend when reshape is already interrupted, so
> waiting raid456 IO can abort, drop its reference, and let suspend
> finish.
>
> The mdadm test tests/25raid456-reshape-deadlock reproduces the hang.
>
> Fixes: 714d20150ed8 ("md: add new helpers to suspend/resume array")
> Signed-off-by: Yu Kuai<yukuai@fnnas.com>
> ---
>   drivers/md/md.c | 11 +++++++++++
>   1 file changed, 11 insertions(+)
Applied to md-7.1

-- 
Thansk,
Kuai