[PATCH v2] xfs: check kthread_should_stop() after the setting of task state

[PATCH v2] xfs: check kthread_should_stop() after the setting of task state

[Hou Tao]

A umount hang is possible when a race occurs between the umount
process and the xfsaild kthread. The following sequences outline
the race:

    xfsaild: kthread_should_stop()
	     => return false, so xfsaild continue

    umount: set_bit(KTHREAD_SHOULD_STOP, &kthread->flags)
	    => by kthread_stop()
    umount: wake_up_process()
	    => because xfsaild is still running, so 0 is returned

    xfsaild: __set_current_state(TASK_INTERRUPTIBLE)
    xfsaild: schedule()
	    => now, xfsaild will wait indefinitely

    umount: wait_for_completion()
	    => and umount will hang

To fix that, we need to check kthread_should_stop() after we set
the task state, so the xfsaild will either see the stop bit and
exit or the task state is reset to runnable by wake_up_process()
such that it isn't scheduled out indefinitely and detects the stop
bit at the next iteration.

Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Brian Foster <bfoster@redhat.com>
Signed-off-by: Hou Tao <houtao1@huawei.com>
---
v2:
	* comment updates suggested by Brain Foster
v1:
	* http://www.spinics.net/lists/linux-xfs/msg10285.html
---
 fs/xfs/xfs_trans_ail.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c
index 9056c0f..2d77d9c 100644
--- a/fs/xfs/xfs_trans_ail.c
+++ b/fs/xfs/xfs_trans_ail.c
@@ -499,11 +499,26 @@ xfsaild(
 	current->flags |= PF_MEMALLOC;
 	set_freezable();
 
-	while (!kthread_should_stop()) {
+	while (1) {
 		if (tout && tout <= 20)
-			__set_current_state(TASK_KILLABLE);
+			set_current_state(TASK_KILLABLE);
 		else
-			__set_current_state(TASK_INTERRUPTIBLE);
+			set_current_state(TASK_INTERRUPTIBLE);
+
+		/*
+		 * Check kthread_should_stop() after we set the task state
+		 * to guarantee that we either see the stop bit and exit or
+		 * the task state is reset to runnable such that it's not
+		 * scheduled out indefinitely and detects the stop bit at
+		 * next iteration.
+		 *
+		 * A memory barrier is included in above task state set to
+		 * serialize again kthread_stop().
+		 */
+		if (kthread_should_stop()) {
+			__set_current_state(TASK_RUNNING);
+			break;
+		}
 
 		spin_lock(&ailp->xa_lock);
 
-- 
2.5.0

Re: [PATCH v2] xfs: check kthread_should_stop() after the setting of task state

[Darrick J. Wong]

On Tue, Sep 12, 2017 at 04:45:30PM +0800, Hou Tao wrote:
> A umount hang is possible when a race occurs between the umount
> process and the xfsaild kthread. The following sequences outline
> the race:
> 
>     xfsaild: kthread_should_stop()
> 	     => return false, so xfsaild continue
> 
>     umount: set_bit(KTHREAD_SHOULD_STOP, &kthread->flags)
> 	    => by kthread_stop()
>     umount: wake_up_process()
> 	    => because xfsaild is still running, so 0 is returned
> 
>     xfsaild: __set_current_state(TASK_INTERRUPTIBLE)
>     xfsaild: schedule()
> 	    => now, xfsaild will wait indefinitely
> 
>     umount: wait_for_completion()
> 	    => and umount will hang
> 
> To fix that, we need to check kthread_should_stop() after we set
> the task state, so the xfsaild will either see the stop bit and
> exit or the task state is reset to runnable by wake_up_process()
> such that it isn't scheduled out indefinitely and detects the stop
> bit at the next iteration.
> 
> Reviewed-by: Christoph Hellwig <hch@lst.de>
> Reviewed-by: Brian Foster <bfoster@redhat.com>
> Signed-off-by: Hou Tao <houtao1@huawei.com>

Looks ok, I guess, but just to reiterate what I said in another email
15 minutes ago, can this be turned into a regression test?

--D

> ---
> v2:
> 	* comment updates suggested by Brain Foster
> v1:
> 	* http://www.spinics.net/lists/linux-xfs/msg10285.html
> ---
>  fs/xfs/xfs_trans_ail.c | 21 ++++++++++++++++++---
>  1 file changed, 18 insertions(+), 3 deletions(-)
> 
> diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c
> index 9056c0f..2d77d9c 100644
> --- a/fs/xfs/xfs_trans_ail.c
> +++ b/fs/xfs/xfs_trans_ail.c
> @@ -499,11 +499,26 @@ xfsaild(
>  	current->flags |= PF_MEMALLOC;
>  	set_freezable();
>  
> -	while (!kthread_should_stop()) {
> +	while (1) {
>  		if (tout && tout <= 20)
> -			__set_current_state(TASK_KILLABLE);
> +			set_current_state(TASK_KILLABLE);
>  		else
> -			__set_current_state(TASK_INTERRUPTIBLE);
> +			set_current_state(TASK_INTERRUPTIBLE);
> +
> +		/*
> +		 * Check kthread_should_stop() after we set the task state
> +		 * to guarantee that we either see the stop bit and exit or
> +		 * the task state is reset to runnable such that it's not
> +		 * scheduled out indefinitely and detects the stop bit at
> +		 * next iteration.
> +		 *
> +		 * A memory barrier is included in above task state set to
> +		 * serialize again kthread_stop().
> +		 */
> +		if (kthread_should_stop()) {
> +			__set_current_state(TASK_RUNNING);
> +			break;
> +		}
>  
>  		spin_lock(&ailp->xa_lock);
>  
> -- 
> 2.5.0
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2] xfs: check kthread_should_stop() after the setting of task state

[Hou Tao]

Hi Darrick,

On 2017/9/19 2:52, Darrick J. Wong wrote:
> On Tue, Sep 12, 2017 at 04:45:30PM +0800, Hou Tao wrote:
>> A umount hang is possible when a race occurs between the umount
>> process and the xfsaild kthread. The following sequences outline
>> the race:
>>
>>     xfsaild: kthread_should_stop()
>> 	     => return false, so xfsaild continue
>>
>>     umount: set_bit(KTHREAD_SHOULD_STOP, &kthread->flags)
>> 	    => by kthread_stop()
>>     umount: wake_up_process()
>> 	    => because xfsaild is still running, so 0 is returned
>>
>>     xfsaild: __set_current_state(TASK_INTERRUPTIBLE)
>>     xfsaild: schedule()
>> 	    => now, xfsaild will wait indefinitely
>>
>>     umount: wait_for_completion()
>> 	    => and umount will hang
>>
>> To fix that, we need to check kthread_should_stop() after we set
>> the task state, so the xfsaild will either see the stop bit and
>> exit or the task state is reset to runnable by wake_up_process()
>> such that it isn't scheduled out indefinitely and detects the stop
>> bit at the next iteration.
>>
>> Reviewed-by: Christoph Hellwig <hch@lst.de>
>> Reviewed-by: Brian Foster <bfoster@redhat.com>
>> Signed-off-by: Hou Tao <houtao1@huawei.com>
> 
> Looks ok, I guess, but just to reiterate what I said in another email
> 15 minutes ago, can this be turned into a regression test?
It's difficult to create an always-happened test for the race, and
I had test the patch by adding artificial delays (as suggested by Brian Foster) in
kernel source code. I also have tried to reproduce the problem by lifting up the
schedule class/priority of the umount process and lifting down the schedule
class/priorityof the xfsaild kthread, but still can not reproduce the problem,
so any ideas or suggestions ?

Regard


> --D
> 
>> ---
>> v2:
>> 	* comment updates suggested by Brain Foster
>> v1:
>> 	* http://www.spinics.net/lists/linux-xfs/msg10285.html
>> ---
>>  fs/xfs/xfs_trans_ail.c | 21 ++++++++++++++++++---
>>  1 file changed, 18 insertions(+), 3 deletions(-)
>>
>> diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c
>> index 9056c0f..2d77d9c 100644
>> --- a/fs/xfs/xfs_trans_ail.c
>> +++ b/fs/xfs/xfs_trans_ail.c
>> @@ -499,11 +499,26 @@ xfsaild(
>>  	current->flags |= PF_MEMALLOC;
>>  	set_freezable();
>>  
>> -	while (!kthread_should_stop()) {
>> +	while (1) {
>>  		if (tout && tout <= 20)
>> -			__set_current_state(TASK_KILLABLE);
>> +			set_current_state(TASK_KILLABLE);
>>  		else
>> -			__set_current_state(TASK_INTERRUPTIBLE);
>> +			set_current_state(TASK_INTERRUPTIBLE);
>> +
>> +		/*
>> +		 * Check kthread_should_stop() after we set the task state
>> +		 * to guarantee that we either see the stop bit and exit or
>> +		 * the task state is reset to runnable such that it's not
>> +		 * scheduled out indefinitely and detects the stop bit at
>> +		 * next iteration.
>> +		 *
>> +		 * A memory barrier is included in above task state set to
>> +		 * serialize again kthread_stop().
>> +		 */
>> +		if (kthread_should_stop()) {
>> +			__set_current_state(TASK_RUNNING);
>> +			break;
>> +		}
>>  
>>  		spin_lock(&ailp->xa_lock);
>>  
>> -- 
>> 2.5.0
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> .
> 

Re: [PATCH v2] xfs: check kthread_should_stop() after the setting of task state

[Darrick J. Wong]

On Fri, Sep 22, 2017 at 06:57:00PM +0800, Hou Tao wrote:
> Hi Darrick,
> 
> On 2017/9/19 2:52, Darrick J. Wong wrote:
> > On Tue, Sep 12, 2017 at 04:45:30PM +0800, Hou Tao wrote:
> >> A umount hang is possible when a race occurs between the umount
> >> process and the xfsaild kthread. The following sequences outline
> >> the race:
> >>
> >>     xfsaild: kthread_should_stop()
> >> 	     => return false, so xfsaild continue
> >>
> >>     umount: set_bit(KTHREAD_SHOULD_STOP, &kthread->flags)
> >> 	    => by kthread_stop()
> >>     umount: wake_up_process()
> >> 	    => because xfsaild is still running, so 0 is returned
> >>
> >>     xfsaild: __set_current_state(TASK_INTERRUPTIBLE)
> >>     xfsaild: schedule()
> >> 	    => now, xfsaild will wait indefinitely
> >>
> >>     umount: wait_for_completion()
> >> 	    => and umount will hang
> >>
> >> To fix that, we need to check kthread_should_stop() after we set
> >> the task state, so the xfsaild will either see the stop bit and
> >> exit or the task state is reset to runnable by wake_up_process()
> >> such that it isn't scheduled out indefinitely and detects the stop
> >> bit at the next iteration.
> >>
> >> Reviewed-by: Christoph Hellwig <hch@lst.de>
> >> Reviewed-by: Brian Foster <bfoster@redhat.com>
> >> Signed-off-by: Hou Tao <houtao1@huawei.com>
> > 
> > Looks ok, I guess, but just to reiterate what I said in another email
> > 15 minutes ago, can this be turned into a regression test?
> It's difficult to create an always-happened test for the race, and
> I had test the patch by adding artificial delays (as suggested by Brian Foster) in
> kernel source code. I also have tried to reproduce the problem by lifting up the
> schedule class/priority of the umount process and lifting down the schedule
> class/priorityof the xfsaild kthread, but still can not reproduce the problem,
> so any ideas or suggestions ?

Do the artificial delays make it so that the race happens reliably?
If so, we could just add a debug knob to inject delays, which you could
then call from a xfstest script.

--D

> 
> Regard
> 
> 
> > --D
> > 
> >> ---
> >> v2:
> >> 	* comment updates suggested by Brain Foster
> >> v1:
> >> 	* http://www.spinics.net/lists/linux-xfs/msg10285.html
> >> ---
> >>  fs/xfs/xfs_trans_ail.c | 21 ++++++++++++++++++---
> >>  1 file changed, 18 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/fs/xfs/xfs_trans_ail.c b/fs/xfs/xfs_trans_ail.c
> >> index 9056c0f..2d77d9c 100644
> >> --- a/fs/xfs/xfs_trans_ail.c
> >> +++ b/fs/xfs/xfs_trans_ail.c
> >> @@ -499,11 +499,26 @@ xfsaild(
> >>  	current->flags |= PF_MEMALLOC;
> >>  	set_freezable();
> >>  
> >> -	while (!kthread_should_stop()) {
> >> +	while (1) {
> >>  		if (tout && tout <= 20)
> >> -			__set_current_state(TASK_KILLABLE);
> >> +			set_current_state(TASK_KILLABLE);
> >>  		else
> >> -			__set_current_state(TASK_INTERRUPTIBLE);
> >> +			set_current_state(TASK_INTERRUPTIBLE);
> >> +
> >> +		/*
> >> +		 * Check kthread_should_stop() after we set the task state
> >> +		 * to guarantee that we either see the stop bit and exit or
> >> +		 * the task state is reset to runnable such that it's not
> >> +		 * scheduled out indefinitely and detects the stop bit at
> >> +		 * next iteration.
> >> +		 *
> >> +		 * A memory barrier is included in above task state set to
> >> +		 * serialize again kthread_stop().
> >> +		 */
> >> +		if (kthread_should_stop()) {
> >> +			__set_current_state(TASK_RUNNING);
> >> +			break;
> >> +		}
> >>  
> >>  		spin_lock(&ailp->xa_lock);
> >>  
> >> -- 
> >> 2.5.0
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> > .
> > 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html