Re: [PATCH] f2fs: avoid hungtask problem caused by losing wake_up

[He YunLei <heyunlei@huawei.com>]

On 2016/2/23 17:15, Chao Yu wrote:
Hi Chao,

> Hi Yunlei,
>
>> -----Original Message-----
>> From: He YunLei [mailto:heyunlei@huawei.com]
>> Sent: Tuesday, February 23, 2016 3:03 PM
>> To: Chao Yu; jaegeuk@kernel.org; linux-f2fs-devel@lists.sourceforge.net
>> Cc: bintian.wang@huawei.com; 'Biao He'
>> Subject: Re: [f2fs-dev] [PATCH] f2fs: avoid hungtask problem caused by losing wake_up
>>
>> On 2016/2/23 13:44, Chao Yu wrote:
>>> Hi Yunlei,
>> Hi Chao,
>>>
>>>> -----Original Message-----
>>>> From: Yunlei He [mailto:heyunlei@huawei.com]
>>>> Sent: Tuesday, February 23, 2016 12:08 PM
>>>> To: chao2.yu@samsung.com; jaegeuk@kernel.org; linux-f2fs-devel@lists.sourceforge.net
>>>> Cc: bintian.wang@huawei.com; Yunlei He; Biao He
>>>> Subject: [f2fs-dev] [PATCH] f2fs: avoid hungtask problem caused by losing wake_up
>>>>
>>>> The D state of wait_on_all_pages_writeback should be waken by
>>>> function f2fs_write_end_io when all writeback pages have been
>>>> succesfully written to device. It's possible that wake_up comes
>>>> between get_pages and io_schedule. Maybe in this case it will
>>>> lost wake_up and still in D state even if all pages have been
>>>> write back to device, and finally, the whole system will be into
>>>> the hungtask state.
>>>
>>> I haven't encountered such issue so far, do you suffer this in real
>>> world?
>>>
>> yes, I have encounter it, the whole file system is blocked at function
>> wait_on_all_pages_writeback beyond 120s when write cp, and no error reported
>> by storage device driver.
>
> Could this reproducible? If it could, could you please share the details.
> And did this occur in a huge size f2fs image?
>
>>>>
>>>>                   if (!get_pages(sbi, F2FS_WRITEBACK))
>>>>                            break;
>>>> 					<---------  wake_up
>>>
>>> wake_up will put all tasks linked in sbi->cp_wait on run-queue, so
>>> here it should be save to call io_schedule, after being rescheduled,
>>> it will get the chance to check above condition to break out.
>>>
>>> Thanks,
>>
>> Here, we just doubt something weird may cause wait_on_all_pages_writeback
>> could not be waken. Wake_up trigger only one time by last bio's end_io
>> function, if the thread happen to miss it, the thread will be in D state
>> forever. So we change the code to make wait_on_all_pages_writeback awaken
>> periodically, then check the condition.
>
> Got it.
>
> The patch can fix issue that checkpointer will wait forever in case of
> write_end_io was failed to call wake_up for some reason.
>
> But I doubt more that the reason we are stuck is there are remained pages
> cached in bio buffer without being submitted. To make sure, maybe in
> wait_on_all_pages_writeback we could add print info to see whether
> sbi->write_io[].bio is valid or not.
>
We use tool dump f2fs_sb_info information and find that:

	write_io[DATA].bio = 0;
	write_io[NODE].bio = 0;
	write_io[META].bio = 0;

	nr_pages[F2FS_WRITEBACK] = 0;
	nr_pages[F2FS_DIRTY_DENTS] = 0;
	nr_pages[F2FS_DIRTY_NODES] = 13;
	nr_pages[F2FS_DIRTY_META] = 0;
	nr_pages[F2FS_INMEM_PAGES] = 0;

So we believe that the block device is ok!

Thanks,

> Thanks,
>
>>
>>>
>>>>                   io_schedule();
>>>>
>>>> Signed-off-by: Yunlei He <heyunlei@huawei.com>
>>>> Signed-off-by: Biao He <hebiao6@huawei.com>
>>>> ---
>>>>    fs/f2fs/checkpoint.c | 2 +-
>>>>    1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c
>>>> index 2bac8a1..f55355d 100644
>>>> --- a/fs/f2fs/checkpoint.c
>>>> +++ b/fs/f2fs/checkpoint.c
>>>> @@ -920,7 +920,7 @@ static void wait_on_all_pages_writeback(struct f2fs_sb_info *sbi)
>>>>    		if (!get_pages(sbi, F2FS_WRITEBACK))
>>>>    			break;
>>>>
>>>> -		io_schedule();
>>>> +		io_schedule_timeout(5*HZ);
>>>>    	}
>>>>    	finish_wait(&sbi->cp_wait, &wait);
>>>>    }
>>>> --
>>>> 1.9.1
>>>
>>>
>>>
>>> .
>>>
>
>
>
> .
>


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140