Re: [md PATCH 1/5 v2] md: always hold reconfig_mutex when calling mddev_suspend()

[Shaohua Li <shli@kernel.org>]

On Thu, Oct 19, 2017 at 05:29:22PM +1100, Neil Brown wrote:
> On Wed, Oct 18 2017, Shaohua Li wrote:
> 
> > On Thu, Oct 19, 2017 at 12:17:16PM +1100, Neil Brown wrote:
> >> 
> >> Most often mddev_suspend() is called with
> >> reconfig_mutex held.  Make this a requirement in
> >> preparation a subsequent patch.  Also require
> >> reconfig_mutex to be held for mddev_resume(),
> >> partly for symmetry and partly to guarantee
> >> no races with incr/decr of mddev->suspend.
> >> 
> >> Taking the mutex in r5c_disable_writeback_async() is
> >> a little tricky as this is called from a work queue
> >> via log->disable_writeback_work, and flush_work()
> >> is called on that while holding ->reconfig_mutex.
> >> If the work item hasn't run before flush_work()
> >> is called, the work function will not be able to
> >> get the mutex.
> >> 
> >> So we use mddev_trylock() inside the wait_event() call, and have that
> >> abort when conf->log is set to NULL, which happens before
> >> flush_work() is called.
> >> We wait in mddev->sb_wait and ensure this is woken
> >> when any of the conditions change.  This requires
> >> waking mddev->sb_wait in mddev_unlock().  This is only
> >> like to trigger extra wake_ups of threads that needn't
> >> be woken when metadata is being written, and that
> >> doesn't happen often enough that the cost would be
> >> noticeable.
> >> 
> >> Signed-off-by: NeilBrown <neilb@suse.com>
> >
> > --snip--
> >
> > Applied the other patches in the series including the 'remove quiesce(2)' one.
> 
> Thanks.
> 
> >
> >> index 0b7406ac8ce1..6a631dd21f0b 100644
> >> --- a/drivers/md/raid5-cache.c
> >> +++ b/drivers/md/raid5-cache.c
> >> @@ -693,6 +693,8 @@ static void r5c_disable_writeback_async(struct work_struct *work)
> >>  	struct r5l_log *log = container_of(work, struct r5l_log,
> >>  					   disable_writeback_work);
> >>  	struct mddev *mddev = log->rdev->mddev;
> >> +	struct r5conf *conf = mddev->private;
> >> +	int locked = 0;
> >>  
> >>  	if (log->r5c_journal_mode == R5C_JOURNAL_MODE_WRITE_THROUGH)
> >>  		return;
> >> @@ -701,11 +703,15 @@ static void r5c_disable_writeback_async(struct work_struct *work)
> >>  
> >>  	/* wait superblock change before suspend */
> >>  	wait_event(mddev->sb_wait,
> >> -		   !test_bit(MD_SB_CHANGE_PENDING, &mddev->sb_flags));
> >> -
> >> -	mddev_suspend(mddev);
> >> -	log->r5c_journal_mode = R5C_JOURNAL_MODE_WRITE_THROUGH;
> >> -	mddev_resume(mddev);
> >> +		   conf->log == NULL ||
> >> +		   (!test_bit(MD_SB_CHANGE_PENDING, &mddev->sb_flags) &&
> >> +		    (locked = mddev_trylock(mddev))));
> >> +	if (locked) {
> >> +		mddev_suspend(mddev);
> >> +		log->r5c_journal_mode = R5C_JOURNAL_MODE_WRITE_THROUGH;
> >> +		mddev_resume(mddev);
> >> +		mddev_unlock(mddev);
> >> +	}
> >
> > For this one, my point is:
> >
> > 	wait_event(mddev->sb_wait, conf->log == NULL ||
> > 		   !test_bit(MD_SB_CHANGE_PENDING, &mddev->sb_flags));
> > 	if (conf->log == NULL)
> > 		return;
> >
> > 	mddev_suspend(mddev);
> > 	log->r5c_journal_mode = R5C_JOURNAL_MODE_WRITE_THROUGH;
> > 	mddev_resume(mddev);
> >
> > does it work?
> 
> The
> 	lockdep_assert_held(&mddev->reconfig_mutex);
> in mddev_suspend() will complain.
> 
> If you put an mddev_lock() call in there to stop the complaint, and if
> the work item doesn't start before the reconfig_mutex is taken prior to
> stopping the array, then r5l_exit_log() can deadlock at
> 	flush_work(&log->disable_writeback_work);

Ok, got it now. But really don't like this patch. The mddev_unlock is strange,
r5c_disable_writeback_async could skip disabling writeback too. Could we add a
new callback like .prepare_free, and flush workqueue there. After we drop the
reconfig_mutex in do_md_stop, we call the prepare_free. We can probably set a
flag, so later r5c_disable_writeback_async will bail out doing nothing. I think
this should work, right?

Thanks,
Shaohua