Re: kswapd hogging in lowmem_shrink

Re: kswapd hogging in lowmem_shrink

[Rik van Riel]

On 03/13/2015 10:25 AM, Vaibhav Shinde wrote:
> 
> On low memory situation, I see various shrinkers being invoked, but in
> lowmem_shrink() case, kswapd is found to be hogging for around 150msecs.
> 
> Due to this my application suffer latency issue, as the cpu was not
> released by kswapd0.
> 
> I took below traces with vmscan events, that show lowmem_shrink taking
> such long time for execution.

This is the Android low memory killer, which kills the
task with the lowest priority in the system.

The low memory killer will iterate over all the tasks
in the system to identify the task to kill.

This is not a problem in Android systems, and other
small systems where this piece of code is used.

What kind of system are you trying to use the low
memory killer on?

How many tasks are you running?

Re: kswapd hogging in lowmem_shrink

[Rik van Riel]

On 03/16/2015 03:45 AM, Vaibhav Shinde wrote:
> 
> 
> On Fri, Mar 13, 2015 at 7:52 AM, Rik van Riel <riel@redhat.com
> <mailto:riel@redhat.com>> wrote:
>>
>> On 03/13/2015 10:25 AM, Vaibhav Shinde wrote:
>> >
>> > On low memory situation, I see various shrinkers being invoked, but in
>> > lowmem_shrink() case, kswapd is found to be hogging for around 150msecs.
>> >
>> > Due to this my application suffer latency issue, as the cpu was not
>> > released by kswapd0.
>> >
>> > I took below traces with vmscan events, that show lowmem_shrink taking
>> > such long time for execution.
>>
>> This is the Android low memory killer, which kills the
>> task with the lowest priority in the system.
>>
>> The low memory killer will iterate over all the tasks
>> in the system to identify the task to kill.
>>
>> This is not a problem in Android systems, and other
>> small systems where this piece of code is used.
>>
>> What kind of system are you trying to use the low
>> memory killer on?
>>
>> How many tasks are you running?
>>
> yes, lowmemorykiller kills the task depending on its oom_score, I am
> using a embedded device with 2GB memory, there are task running that
> cause lowmemory situation - no issue about it.
> 
> But my concern is kswapd takes too long to iterate through all the
> processes(lowmem_shrink() => for_each_process()), the time taken is
> around 150msec, due to which my high priority application suffer system
> latency that cause malfunctioning.

If it is an issue for you, you will have to fix that.

-- 
All rights reversed

Re: kswapd hogging in lowmem_shrink

[Michal Hocko]

On Fri 13-03-15 19:55:27, Vaibhav Shinde wrote:
> On low memory situation, I see various shrinkers being invoked, but in
> lowmem_shrink() case, kswapd is found to be hogging for around 150msecs.
> 
> Due to this my application suffer latency issue, as the cpu was not
> released by kswapd0.
> 
> I took below traces with vmscan events, that show lowmem_shrink taking such
> long time for execution.
> 
> kswapd0-67 [003] ...1  1501.987110: mm_shrink_slab_start:
> lowmem_shrink+0x0/0x580 c0ee8e34: objects to shrink 122 gfp_flags
> GFP_KERNEL pgs_scanned 83 lru_pgs 241753 cache items 241754 delta 10
> total_scan 132
> kswapd0-67 [003] ...1  1502.020827: mm_shrink_slab_end:
> lowmem_shrink+0x0/0x580 c0ee8e34: unused scan count 122 new scan count 4
> total_scan -118 last shrinker return val 237339
> 
> Please provide inputs on the same.

I would strongly discourage from using lowmemory killer. It is broken by
design IMHO. It can spend a lot of time looping on a large machine. Why
do you use it in the first place?

-- 
Michal Hocko
SUSE Labs