* [PATCH v2] xfs: check kthread_should_stop() after the setting of task state
@ 2017-09-12 8:45 Hou Tao
2017-09-18 18:52 ` Darrick J. Wong
0 siblings, 1 reply; 4+ messages in thread
From: Hou Tao @ 2017-09-12 8:45 UTC (permalink / raw)
To: linux-xfs; +Cc: darrick.wong
A umount hang is possible when a race occurs between the umount
process and the xfsaild kthread. The following sequences outline
the race:
xfsaild: kthread_should_stop()
=> return false, so xfsaild continue
umount: set_bit(KTHREAD_SHOULD_STOP, &kthread->flags)
=> by kthread_stop()
umount: wake_up_process()
=> because xfsaild is still running, so 0 is returned
xfsaild: __set_current_state(TASK_INTERRUPTIBLE)
xfsaild: schedule()
=> now, xfsaild will wait indefinitely
umount: wait_for_completion()
=> and umount will hang
To fix that, we need to check kthread_should_stop() after we set
the task state, so the xfsaild will either see the stop bit and
exit or the task state is reset to runnable by wake_up_process()
such that it isn't scheduled out indefinitely and detects the stop
bit at the next iteration.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Brian Foster <bfoster@redhat.com>
Signed-off-by: Hou Tao <houtao1@huawei.com>
---
v2:
* comment updates suggested by Brain Foster
v1:
* http://www.spinics.net/lists/linux-xfs/msg10285.html
---
fs/xfs/xfs_trans_ail.c | 21 ++++++++++++++++++---
1 file changed, 18 insertions(+), 3 deletions(-)
diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c
index 9056c0f..2d77d9c 100644
--- a/fs/xfs/xfs_trans_ail.c
+++ b/fs/xfs/xfs_trans_ail.c
@@ -499,11 +499,26 @@ xfsaild(
current->flags |= PF_MEMALLOC;
set_freezable();
- while (!kthread_should_stop()) {
+ while (1) {
if (tout && tout <= 20)
- __set_current_state(TASK_KILLABLE);
+ set_current_state(TASK_KILLABLE);
else
- __set_current_state(TASK_INTERRUPTIBLE);
+ set_current_state(TASK_INTERRUPTIBLE);
+
+ /*
+ * Check kthread_should_stop() after we set the task state
+ * to guarantee that we either see the stop bit and exit or
+ * the task state is reset to runnable such that it's not
+ * scheduled out indefinitely and detects the stop bit at
+ * next iteration.
+ *
+ * A memory barrier is included in above task state set to
+ * serialize again kthread_stop().
+ */
+ if (kthread_should_stop()) {
+ __set_current_state(TASK_RUNNING);
+ break;
+ }
spin_lock(&ailp->xa_lock);
--
2.5.0
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH v2] xfs: check kthread_should_stop() after the setting of task state
2017-09-12 8:45 [PATCH v2] xfs: check kthread_should_stop() after the setting of task state Hou Tao
@ 2017-09-18 18:52 ` Darrick J. Wong
2017-09-22 10:57 ` Hou Tao
0 siblings, 1 reply; 4+ messages in thread
From: Darrick J. Wong @ 2017-09-18 18:52 UTC (permalink / raw)
To: Hou Tao; +Cc: linux-xfs
On Tue, Sep 12, 2017 at 04:45:30PM +0800, Hou Tao wrote:
> A umount hang is possible when a race occurs between the umount
> process and the xfsaild kthread. The following sequences outline
> the race:
>
> xfsaild: kthread_should_stop()
> => return false, so xfsaild continue
>
> umount: set_bit(KTHREAD_SHOULD_STOP, &kthread->flags)
> => by kthread_stop()
> umount: wake_up_process()
> => because xfsaild is still running, so 0 is returned
>
> xfsaild: __set_current_state(TASK_INTERRUPTIBLE)
> xfsaild: schedule()
> => now, xfsaild will wait indefinitely
>
> umount: wait_for_completion()
> => and umount will hang
>
> To fix that, we need to check kthread_should_stop() after we set
> the task state, so the xfsaild will either see the stop bit and
> exit or the task state is reset to runnable by wake_up_process()
> such that it isn't scheduled out indefinitely and detects the stop
> bit at the next iteration.
>
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> Reviewed-by: Brian Foster <bfoster@redhat.com>
> Signed-off-by: Hou Tao <houtao1@huawei.com>
Looks ok, I guess, but just to reiterate what I said in another email
15 minutes ago, can this be turned into a regression test?
--D
> ---
> v2:
> * comment updates suggested by Brain Foster
> v1:
> * http://www.spinics.net/lists/linux-xfs/msg10285.html
> ---
> fs/xfs/xfs_trans_ail.c | 21 ++++++++++++++++++---
> 1 file changed, 18 insertions(+), 3 deletions(-)
>
> diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c
> index 9056c0f..2d77d9c 100644
> --- a/fs/xfs/xfs_trans_ail.c
> +++ b/fs/xfs/xfs_trans_ail.c
> @@ -499,11 +499,26 @@ xfsaild(
> current->flags |= PF_MEMALLOC;
> set_freezable();
>
> - while (!kthread_should_stop()) {
> + while (1) {
> if (tout && tout <= 20)
> - __set_current_state(TASK_KILLABLE);
> + set_current_state(TASK_KILLABLE);
> else
> - __set_current_state(TASK_INTERRUPTIBLE);
> + set_current_state(TASK_INTERRUPTIBLE);
> +
> + /*
> + * Check kthread_should_stop() after we set the task state
> + * to guarantee that we either see the stop bit and exit or
> + * the task state is reset to runnable such that it's not
> + * scheduled out indefinitely and detects the stop bit at
> + * next iteration.
> + *
> + * A memory barrier is included in above task state set to
> + * serialize again kthread_stop().
> + */
> + if (kthread_should_stop()) {
> + __set_current_state(TASK_RUNNING);
> + break;
> + }
>
> spin_lock(&ailp->xa_lock);
>
> --
> 2.5.0
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH v2] xfs: check kthread_should_stop() after the setting of task state
2017-09-18 18:52 ` Darrick J. Wong
@ 2017-09-22 10:57 ` Hou Tao
2017-09-22 15:14 ` Darrick J. Wong
0 siblings, 1 reply; 4+ messages in thread
From: Hou Tao @ 2017-09-22 10:57 UTC (permalink / raw)
To: Darrick J. Wong; +Cc: linux-xfs
Hi Darrick,
On 2017/9/19 2:52, Darrick J. Wong wrote:
> On Tue, Sep 12, 2017 at 04:45:30PM +0800, Hou Tao wrote:
>> A umount hang is possible when a race occurs between the umount
>> process and the xfsaild kthread. The following sequences outline
>> the race:
>>
>> xfsaild: kthread_should_stop()
>> => return false, so xfsaild continue
>>
>> umount: set_bit(KTHREAD_SHOULD_STOP, &kthread->flags)
>> => by kthread_stop()
>> umount: wake_up_process()
>> => because xfsaild is still running, so 0 is returned
>>
>> xfsaild: __set_current_state(TASK_INTERRUPTIBLE)
>> xfsaild: schedule()
>> => now, xfsaild will wait indefinitely
>>
>> umount: wait_for_completion()
>> => and umount will hang
>>
>> To fix that, we need to check kthread_should_stop() after we set
>> the task state, so the xfsaild will either see the stop bit and
>> exit or the task state is reset to runnable by wake_up_process()
>> such that it isn't scheduled out indefinitely and detects the stop
>> bit at the next iteration.
>>
>> Reviewed-by: Christoph Hellwig <hch@lst.de>
>> Reviewed-by: Brian Foster <bfoster@redhat.com>
>> Signed-off-by: Hou Tao <houtao1@huawei.com>
>
> Looks ok, I guess, but just to reiterate what I said in another email
> 15 minutes ago, can this be turned into a regression test?
It's difficult to create an always-happened test for the race, and
I had test the patch by adding artificial delays (as suggested by Brian Foster) in
kernel source code. I also have tried to reproduce the problem by lifting up the
schedule class/priority of the umount process and lifting down the schedule
class/priorityof the xfsaild kthread, but still can not reproduce the problem,
so any ideas or suggestions ?
Regard
> --D
>
>> ---
>> v2:
>> * comment updates suggested by Brain Foster
>> v1:
>> * http://www.spinics.net/lists/linux-xfs/msg10285.html
>> ---
>> fs/xfs/xfs_trans_ail.c | 21 ++++++++++++++++++---
>> 1 file changed, 18 insertions(+), 3 deletions(-)
>>
>> diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c
>> index 9056c0f..2d77d9c 100644
>> --- a/fs/xfs/xfs_trans_ail.c
>> +++ b/fs/xfs/xfs_trans_ail.c
>> @@ -499,11 +499,26 @@ xfsaild(
>> current->flags |= PF_MEMALLOC;
>> set_freezable();
>>
>> - while (!kthread_should_stop()) {
>> + while (1) {
>> if (tout && tout <= 20)
>> - __set_current_state(TASK_KILLABLE);
>> + set_current_state(TASK_KILLABLE);
>> else
>> - __set_current_state(TASK_INTERRUPTIBLE);
>> + set_current_state(TASK_INTERRUPTIBLE);
>> +
>> + /*
>> + * Check kthread_should_stop() after we set the task state
>> + * to guarantee that we either see the stop bit and exit or
>> + * the task state is reset to runnable such that it's not
>> + * scheduled out indefinitely and detects the stop bit at
>> + * next iteration.
>> + *
>> + * A memory barrier is included in above task state set to
>> + * serialize again kthread_stop().
>> + */
>> + if (kthread_should_stop()) {
>> + __set_current_state(TASK_RUNNING);
>> + break;
>> + }
>>
>> spin_lock(&ailp->xa_lock);
>>
>> --
>> 2.5.0
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
> .
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH v2] xfs: check kthread_should_stop() after the setting of task state
2017-09-22 10:57 ` Hou Tao
@ 2017-09-22 15:14 ` Darrick J. Wong
0 siblings, 0 replies; 4+ messages in thread
From: Darrick J. Wong @ 2017-09-22 15:14 UTC (permalink / raw)
To: Hou Tao; +Cc: linux-xfs
On Fri, Sep 22, 2017 at 06:57:00PM +0800, Hou Tao wrote:
> Hi Darrick,
>
> On 2017/9/19 2:52, Darrick J. Wong wrote:
> > On Tue, Sep 12, 2017 at 04:45:30PM +0800, Hou Tao wrote:
> >> A umount hang is possible when a race occurs between the umount
> >> process and the xfsaild kthread. The following sequences outline
> >> the race:
> >>
> >> xfsaild: kthread_should_stop()
> >> => return false, so xfsaild continue
> >>
> >> umount: set_bit(KTHREAD_SHOULD_STOP, &kthread->flags)
> >> => by kthread_stop()
> >> umount: wake_up_process()
> >> => because xfsaild is still running, so 0 is returned
> >>
> >> xfsaild: __set_current_state(TASK_INTERRUPTIBLE)
> >> xfsaild: schedule()
> >> => now, xfsaild will wait indefinitely
> >>
> >> umount: wait_for_completion()
> >> => and umount will hang
> >>
> >> To fix that, we need to check kthread_should_stop() after we set
> >> the task state, so the xfsaild will either see the stop bit and
> >> exit or the task state is reset to runnable by wake_up_process()
> >> such that it isn't scheduled out indefinitely and detects the stop
> >> bit at the next iteration.
> >>
> >> Reviewed-by: Christoph Hellwig <hch@lst.de>
> >> Reviewed-by: Brian Foster <bfoster@redhat.com>
> >> Signed-off-by: Hou Tao <houtao1@huawei.com>
> >
> > Looks ok, I guess, but just to reiterate what I said in another email
> > 15 minutes ago, can this be turned into a regression test?
> It's difficult to create an always-happened test for the race, and
> I had test the patch by adding artificial delays (as suggested by Brian Foster) in
> kernel source code. I also have tried to reproduce the problem by lifting up the
> schedule class/priority of the umount process and lifting down the schedule
> class/priorityof the xfsaild kthread, but still can not reproduce the problem,
> so any ideas or suggestions ?
Do the artificial delays make it so that the race happens reliably?
If so, we could just add a debug knob to inject delays, which you could
then call from a xfstest script.
--D
>
> Regard
>
>
> > --D
> >
> >> ---
> >> v2:
> >> * comment updates suggested by Brain Foster
> >> v1:
> >> * http://www.spinics.net/lists/linux-xfs/msg10285.html
> >> ---
> >> fs/xfs/xfs_trans_ail.c | 21 ++++++++++++++++++---
> >> 1 file changed, 18 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c
> >> index 9056c0f..2d77d9c 100644
> >> --- a/fs/xfs/xfs_trans_ail.c
> >> +++ b/fs/xfs/xfs_trans_ail.c
> >> @@ -499,11 +499,26 @@ xfsaild(
> >> current->flags |= PF_MEMALLOC;
> >> set_freezable();
> >>
> >> - while (!kthread_should_stop()) {
> >> + while (1) {
> >> if (tout && tout <= 20)
> >> - __set_current_state(TASK_KILLABLE);
> >> + set_current_state(TASK_KILLABLE);
> >> else
> >> - __set_current_state(TASK_INTERRUPTIBLE);
> >> + set_current_state(TASK_INTERRUPTIBLE);
> >> +
> >> + /*
> >> + * Check kthread_should_stop() after we set the task state
> >> + * to guarantee that we either see the stop bit and exit or
> >> + * the task state is reset to runnable such that it's not
> >> + * scheduled out indefinitely and detects the stop bit at
> >> + * next iteration.
> >> + *
> >> + * A memory barrier is included in above task state set to
> >> + * serialize again kthread_stop().
> >> + */
> >> + if (kthread_should_stop()) {
> >> + __set_current_state(TASK_RUNNING);
> >> + break;
> >> + }
> >>
> >> spin_lock(&ailp->xa_lock);
> >>
> >> --
> >> 2.5.0
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> > .
> >
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2017-09-22 15:14 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-09-12 8:45 [PATCH v2] xfs: check kthread_should_stop() after the setting of task state Hou Tao
2017-09-18 18:52 ` Darrick J. Wong
2017-09-22 10:57 ` Hou Tao
2017-09-22 15:14 ` Darrick J. Wong
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).