From: Xiao Ni <xni@redhat.com>
To: Yu Kuai <yukuai1@huaweicloud.com>
Cc: agk@redhat.com, snitzer@kernel.org, dm-devel@redhat.com,
song@kernel.org, linux-kernel@vger.kernel.org,
linux-raid@vger.kernel.org, yukuai3@huawei.com,
yi.zhang@huawei.com, yangerkun@huawei.com
Subject: Re: [PATCH -next v2 03/28] md: add new helpers to suspend/resume array
Date: Mon, 25 Sep 2023 15:21:45 +0800 [thread overview]
Message-ID: <CALTww2-5F=C5N6YZ-3weD9xSWhpT6Mx8NkaevfXZWqR6=Bwc4A@mail.gmail.com> (raw)
In-Reply-To: <20230828020021.2489641-4-yukuai1@huaweicloud.com>
On Mon, Aug 28, 2023 at 10:04 AM Yu Kuai <yukuai1@huaweicloud.com> wrote:
>
> From: Yu Kuai <yukuai3@huawei.com>
>
> Advantages for new apis:
> - reconfig_mutex is not required;
> - the weird logical that suspend array hold 'reconfig_mutex' for
> mddev_check_recovery() to update superblock is not needed;
> - the specail handling, 'pers->prepare_suspend', for raid456 is not
> needed;
> - It's safe to be called at any time once mddev is allocated, and it's
> designed to be used from slow path where array configuration is changed;
>
> Signed-off-by: Yu Kuai <yukuai3@huawei.com>
> ---
> drivers/md/md.c | 85 +++++++++++++++++++++++++++++++++++++++++++++++--
> drivers/md/md.h | 3 ++
> 2 files changed, 86 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 7fa311a14317..6236e2e395c1 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -443,12 +443,22 @@ void mddev_suspend(struct mddev *mddev)
> lockdep_is_held(&mddev->reconfig_mutex));
>
> WARN_ON_ONCE(thread && current == thread->tsk);
> - if (mddev->suspended++)
> +
> + /* can't concurrent with __mddev_suspend() and __mddev_resume() */
> + mutex_lock(&mddev->suspend_mutex);
> + if (mddev->suspended++) {
> + mutex_unlock(&mddev->suspend_mutex);
> return;
> + }
> +
> wake_up(&mddev->sb_wait);
> set_bit(MD_ALLOW_SB_UPDATE, &mddev->flags);
> percpu_ref_kill(&mddev->active_io);
>
> + /*
> + * TODO: cleanup 'pers->prepare_suspend after all callers are replaced
> + * by __mddev_suspend().
> + */
> if (mddev->pers && mddev->pers->prepare_suspend)
> mddev->pers->prepare_suspend(mddev);
>
> @@ -459,14 +469,21 @@ void mddev_suspend(struct mddev *mddev)
> del_timer_sync(&mddev->safemode_timer);
> /* restrict memory reclaim I/O during raid array is suspend */
> mddev->noio_flag = memalloc_noio_save();
> +
> + mutex_unlock(&mddev->suspend_mutex);
> }
> EXPORT_SYMBOL_GPL(mddev_suspend);
>
> void mddev_resume(struct mddev *mddev)
> {
> lockdep_assert_held(&mddev->reconfig_mutex);
> - if (--mddev->suspended)
> +
> + /* can't concurrent with __mddev_suspend() and __mddev_resume() */
> + mutex_lock(&mddev->suspend_mutex);
> + if (--mddev->suspended) {
> + mutex_unlock(&mddev->suspend_mutex);
> return;
> + }
>
> /* entred the memalloc scope from mddev_suspend() */
> memalloc_noio_restore(mddev->noio_flag);
> @@ -477,9 +494,72 @@ void mddev_resume(struct mddev *mddev)
> set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
> md_wakeup_thread(mddev->thread);
> md_wakeup_thread(mddev->sync_thread); /* possibly kick off a reshape */
> +
> + mutex_unlock(&mddev->suspend_mutex);
> }
> EXPORT_SYMBOL_GPL(mddev_resume);
>
> +void __mddev_suspend(struct mddev *mddev)
> +{
> +
> + /*
> + * hold reconfig_mutex to wait for normal io will deadlock, because
> + * other context can't update super_block, and normal io can rely on
> + * updating super_block.
> + */
> + lockdep_assert_not_held(&mddev->reconfig_mutex);
> +
> + mutex_lock(&mddev->suspend_mutex);
> +
> + if (mddev->suspended) {
> + WRITE_ONCE(mddev->suspended, mddev->suspended + 1);
> + mutex_unlock(&mddev->suspend_mutex);
> + return;
> + }
> +
> + percpu_ref_kill(&mddev->active_io);
> + wait_event(mddev->sb_wait, percpu_ref_is_zero(&mddev->active_io));
> +
> + /*
> + * For raid456, io might be waiting for reshape to make progress,
> + * allow new reshape to start while waiting for io to be done to
> + * prevent deadlock.
> + */
> + WRITE_ONCE(mddev->suspended, mddev->suspended + 1);
It changes the order of setting suspended and checking active_io.
suspended is used to stop I/O. Now it checks active_io first and then
adds suspended, if the i/o doesn't stop, it looks like active_io can't
be 0. So it will stuck at waiting active_io to be 0?
Best Regards
Xiao
> +
> + del_timer_sync(&mddev->safemode_timer);
> + /* restrict memory reclaim I/O during raid array is suspend */
> + mddev->noio_flag = memalloc_noio_save();
> +
> + mutex_unlock(&mddev->suspend_mutex);
> +}
> +EXPORT_SYMBOL_GPL(__mddev_suspend);
> +
> +void __mddev_resume(struct mddev *mddev)
> +{
> + lockdep_assert_not_held(&mddev->reconfig_mutex);
> +
> + mutex_lock(&mddev->suspend_mutex);
> + WRITE_ONCE(mddev->suspended, mddev->suspended - 1);
> + if (mddev->suspended) {
> + mutex_unlock(&mddev->suspend_mutex);
> + return;
> + }
> +
> + /* entred the memalloc scope from __mddev_suspend() */
> + memalloc_noio_restore(mddev->noio_flag);
> +
> + percpu_ref_resurrect(&mddev->active_io);
> + wake_up(&mddev->sb_wait);
> +
> + set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
> + md_wakeup_thread(mddev->thread);
> + md_wakeup_thread(mddev->sync_thread); /* possibly kick off a reshape */
> +
> + mutex_unlock(&mddev->suspend_mutex);
> +}
> +EXPORT_SYMBOL_GPL(__mddev_resume);
> +
> /*
> * Generic flush handling for md
> */
> @@ -667,6 +747,7 @@ int mddev_init(struct mddev *mddev)
> mutex_init(&mddev->open_mutex);
> mutex_init(&mddev->reconfig_mutex);
> mutex_init(&mddev->sync_mutex);
> + mutex_init(&mddev->suspend_mutex);
> mutex_init(&mddev->bitmap_info.mutex);
> INIT_LIST_HEAD(&mddev->disks);
> INIT_LIST_HEAD(&mddev->all_mddevs);
> diff --git a/drivers/md/md.h b/drivers/md/md.h
> index fb3b123f16dd..1103e6b08ad9 100644
> --- a/drivers/md/md.h
> +++ b/drivers/md/md.h
> @@ -316,6 +316,7 @@ struct mddev {
> unsigned long sb_flags;
>
> int suspended;
> + struct mutex suspend_mutex;
> struct percpu_ref active_io;
> int ro;
> int sysfs_active; /* set when sysfs deletes
> @@ -811,6 +812,8 @@ extern void md_rdev_clear(struct md_rdev *rdev);
> extern void md_handle_request(struct mddev *mddev, struct bio *bio);
> extern void mddev_suspend(struct mddev *mddev);
> extern void mddev_resume(struct mddev *mddev);
> +extern void __mddev_suspend(struct mddev *mddev);
> +extern void __mddev_resume(struct mddev *mddev);
>
> extern void md_reload_sb(struct mddev *mddev, int raid_disk);
> extern void md_update_sb(struct mddev *mddev, int force);
> --
> 2.39.2
>
next prev parent reply other threads:[~2023-09-25 7:22 UTC|newest]
Thread overview: 37+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-08-28 1:59 [PATCH -next v2 00/28] md: synchronize io with array reconfiguration Yu Kuai
2023-08-28 1:59 ` [PATCH -next v2 01/28] md: use READ_ONCE/WRITE_ONCE for 'suspend_lo' and 'suspend_hi' Yu Kuai
2023-09-14 2:53 ` Xiao Ni
2023-09-25 1:18 ` Yu Kuai
2023-08-28 1:59 ` [PATCH -next v2 02/28] md: use 'mddev->suspended' for is_md_suspended() Yu Kuai
2023-09-20 8:46 ` Xiao Ni
2023-09-25 1:34 ` Yu Kuai
2023-08-28 1:59 ` [PATCH -next v2 03/28] md: add new helpers to suspend/resume array Yu Kuai
2023-09-25 7:21 ` Xiao Ni [this message]
2023-09-25 7:23 ` Xiao Ni
2023-08-28 1:59 ` [PATCH -next v2 04/28] md: add new helpers to suspend/resume and lock/unlock array Yu Kuai
2023-08-28 1:59 ` [PATCH -next v2 05/28] md: use new apis to suspend array for suspend_lo/hi_store() Yu Kuai
2023-08-28 1:59 ` [PATCH -next v2 06/28] md: use new apis to suspend array for level_store() Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 07/28] md: use new apis to suspend array for serialize_policy_store() Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 08/28] md/dm-raid: use new apis to suspend array Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 09/28] md/md-bitmap: use new apis to suspend array for location_store() Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 10/28] md/raid5-cache: use READ_ONCE/WRITE_ONCE for 'conf->log' Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 11/28] md/raid5-cache: use new apis to suspend array for r5c_disable_writeback_async() Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 12/28] md/raid5-cache: use new apis to suspend array for r5c_journal_mode_store() Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 13/28] md/raid5: use new apis to suspend array for raid5_store_stripe_size() Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 14/28] md/raid5: use new apis to suspend array for raid5_store_skip_copy() Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 15/28] md/raid5: use new apis to suspend array for raid5_store_group_thread_cnt() Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 16/28] md/raid5: use new apis to suspend array for raid5_change_consistency_policy() Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 17/28] md/raid5: replace suspend with quiesce() callback Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 18/28] md: quiesce before md_kick_rdev_from_array() for md-cluster Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 19/28] md: use new apis to suspend array for ioctls involed array reconfiguration Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 20/28] md: use new apis to suspend array for adding/removing rdev from state_store() Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 21/28] md: use new apis to suspend array for bind_rdev_to_array() Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 22/28] md: use new apis to suspend array related to serial pool in state_store() Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 23/28] md: use new apis to suspend array in backlog_store() Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 24/28] md: suspend array in md_start_sync() if array need reconfiguration Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 25/28] md: cleanup mddev_create/destroy_serial_pool() Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 26/28] md/md-linear: cleanup linear_add() Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 27/28] md: remove old apis to suspend the array Yu Kuai
2023-08-28 2:00 ` [PATCH -next v2 28/28] md: rename __mddev_suspend/resume() back to mddev_suspend/resume() Yu Kuai
2023-09-25 15:45 ` [PATCH -next v2 00/28] md: synchronize io with array reconfiguration Song Liu
2023-09-26 0:55 ` Yu Kuai
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CALTww2-5F=C5N6YZ-3weD9xSWhpT6Mx8NkaevfXZWqR6=Bwc4A@mail.gmail.com' \
--to=xni@redhat.com \
--cc=agk@redhat.com \
--cc=dm-devel@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-raid@vger.kernel.org \
--cc=snitzer@kernel.org \
--cc=song@kernel.org \
--cc=yangerkun@huawei.com \
--cc=yi.zhang@huawei.com \
--cc=yukuai1@huaweicloud.com \
--cc=yukuai3@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).