Re: [RFC] Using squashfs, kernel will hung task with no free memory?

["long.wanglong" <long.wanglong@huawei.com>]

On 2015/1/23 2:19, Phillip Lougher wrote:
> On 22/01/15 02:28, long.wanglong wrote:
>> hi，
>>
>> I have encountered kernel hung task when running stability and stress test.
>>
>> test scenarios:
>>     1)the kernel hungtask settings are following:
>>         hung_task_panic = 1
>>         hung_task_timeout_secs = 120
>>     2)the rootfs type is squashfs(read-only)
>>     
>> what the test does is to fork many child process and each process will alloc memory.
>> when there is no free memory in the system, OOM killer is triggerred. and then the kernel triggers hung task(after about five  minutes) .
>> the reason for hung task is that some process keep D states for 120 seconds.
>>
>> if there is no free memory in the system, many process state is D, they enter into D state by kernel path  `squashfs_cache_get()--->wait_event()`.
>> the backtrace is:
>>
>> [  313.950118] [<c02d2014>] (__schedule+0x448/0x5cc) from [<c014e510>] (squashfs_cache_get+0x120/0x3ec)
>> [  314.059660] [<c014e510>] (squashfs_cache_get+0x120/0x3ec) from [<c014fd1c>] (squashfs_readpage+0x748/0xa2c)
>> [  314.176497] [<c014fd1c>] (squashfs_readpage+0x748/0xa2c) from [<c00b7be0>] (__do_page_cache_readahead+0x1ac/0x200)
>> [  314.300621] [<c00b7be0>] (__do_page_cache_readahead+0x1ac/0x200) from [<c00b7e98>] (ra_submit+0x24/0x28)
>> [  314.414325] [<c00b7e98>] (ra_submit+0x24/0x28) from [<c00b043c>] (filemap_fault+0x16c/0x3f0)
>> [  314.515521] [<c00b043c>] (filemap_fault+0x16c/0x3f0) from [<c00c94e0>] (__do_fault+0xc0/0x570)
>> [  314.618802] [<c00c94e0>] (__do_fault+0xc0/0x570) from [<c00cbdc4>] (handle_pte_fault+0x47c/0x1048)
>> [  314.726250] [<c00cbdc4>] (handle_pte_fault+0x47c/0x1048) from [<c00cd928>] (handle_mm_fault+0x164/0x218)
>> [  314.839959] [<c00cd928>] (handle_mm_fault+0x164/0x218) from [<c02d4878>] (do_page_fault.part.7+0x108/0x360)
>> [  314.956788] [<c02d4878>] (do_page_fault.part.7+0x108/0x360) from [<c02d4afc>] (do_page_fault+0x2c/0x70)
>> [  315.069442] [<c02d4afc>] (do_page_fault+0x2c/0x70) from [<c00084cc>] (do_PrefetchAbort+0x2c/0x90)
>> [  315.175850] [<c00084cc>] (do_PrefetchAbort+0x2c/0x90) from [<c02d3674>] (ret_from_exception+0x0/0x10)
>>
>> when a task is already exiting because of OOM killer,the next time OOM killer will kill the same task.
>> so, if the first time of OOM killer select a task(A) that in D state (the task ingore exit signal beacuse of D state).
>> then the next time of OOM killer will also kill task A. In this scenario, oom killer will not free memory.
>>
>> with no free memory, many process sleep in function squashfs_cache_get. about 2 minutes, the system hung task and panic.
>> because of OOM feature and squashfs, on heavy system, This problem is easily reproduce.
>>
>> Is this a problem about squashfs or about the OOM killer. Can anyone give me some good ideas about this?
> 
> This is not a Squashfs issue, it is a well known problem with
> the OOM killer trying to kill tasks which are slow to exit (being
> in D state).  Just google "OOM hung task" to see how long this
> issue has been around.
> 
> The OOM killer is worse than useless in embedded systems because
> its behaviour is unpredictable and can leave a system in a
> zombified or half zombified state. Due to this reason many
> embedded systems disable the OOM killer entirely, and ensure
> there is adequate memory backed up by a watchdog which reboots
> a hung system.
> 
> Phillip
> 

Thanks

>>
>> Best Regards
>> Wang Long
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> .
>>
> 
> 
> .
>