kswapd hogging in lowmem_shrink

kswapd hogging in lowmem_shrink

[Vaibhav Shinde]

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]

On low memory situation, I see various shrinkers being invoked, but in
lowmem_shrink() case, kswapd is found to be hogging for around 150msecs.

Due to this my application suffer latency issue, as the cpu was not
released by kswapd0.

I took below traces with vmscan events, that show lowmem_shrink taking such
long time for execution.

kswapd0-67 [003] ...1  1501.987110: mm_shrink_slab_start:
lowmem_shrink+0x0/0x580 c0ee8e34: objects to shrink 122 gfp_flags
GFP_KERNEL pgs_scanned 83 lru_pgs 241753 cache items 241754 delta 10
total_scan 132
kswapd0-67 [003] ...1  1502.020827: mm_shrink_slab_end:
lowmem_shrink+0x0/0x580 c0ee8e34: unused scan count 122 new scan count 4
total_scan -118 last shrinker return val 237339

Please provide inputs on the same.

Thanks and Regards,
Vaibhav

[-- Attachment #2: Type: text/html, Size: 855 bytes --]

Re: kswapd hogging in lowmem_shrink

[Rik van Riel]

On 03/13/2015 10:25 AM, Vaibhav Shinde wrote:
> 
> On low memory situation, I see various shrinkers being invoked, but in
> lowmem_shrink() case, kswapd is found to be hogging for around 150msecs.
> 
> Due to this my application suffer latency issue, as the cpu was not
> released by kswapd0.
> 
> I took below traces with vmscan events, that show lowmem_shrink taking
> such long time for execution.

This is the Android low memory killer, which kills the
task with the lowest priority in the system.

The low memory killer will iterate over all the tasks
in the system to identify the task to kill.

This is not a problem in Android systems, and other
small systems where this piece of code is used.

What kind of system are you trying to use the low
memory killer on?

How many tasks are you running?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: kswapd hogging in lowmem_shrink

[Vaibhav Shinde]

[-- Attachment #1: Type: text/plain, Size: 1348 bytes --]

On Fri, Mar 13, 2015 at 8:22 PM, Rik van Riel <riel@redhat.com> wrote:

> On 03/13/2015 10:25 AM, Vaibhav Shinde wrote:
> >
> > On low memory situation, I see various shrinkers being invoked, but in
> > lowmem_shrink() case, kswapd is found to be hogging for around 150msecs.
> >
> > Due to this my application suffer latency issue, as the cpu was not
> > released by kswapd0.
> >
> > I took below traces with vmscan events, that show lowmem_shrink taking
> > such long time for execution.
>
> This is the Android low memory killer, which kills the
> task with the lowest priority in the system.
>
> The low memory killer will iterate over all the tasks
> in the system to identify the task to kill.
>
> This is not a problem in Android systems, and other
> small systems where this piece of code is used.
>
> What kind of system are you trying to use the low
> memory killer on?
>
> How many tasks are you running?
>
> yes, lowmemorykiller kills the task depending on its oom_score, I am using
a embedded device with 2GB memory, there are task running that cause
lowmemory situation - no issue about it.

But my concern is kswapd takes too long to iterate through all the
processes(lowmem_shrink() => for_each_process()), the time taken is around
150msec, due to which my high priority application suffer system latency
that cause malfunctioning.

[-- Attachment #2: Type: text/html, Size: 1858 bytes --]

Re: kswapd hogging in lowmem_shrink

[Vaibhav Shinde]

[-- Attachment #1: Type: text/plain, Size: 1364 bytes --]

On Fri, Mar 13, 2015 at 7:52 AM, Rik van Riel <riel@redhat.com> wrote:
>
> On 03/13/2015 10:25 AM, Vaibhav Shinde wrote:
> >
> > On low memory situation, I see various shrinkers being invoked, but in
> > lowmem_shrink() case, kswapd is found to be hogging for around 150msecs.
> >
> > Due to this my application suffer latency issue, as the cpu was not
> > released by kswapd0.
> >
> > I took below traces with vmscan events, that show lowmem_shrink taking
> > such long time for execution.
>
> This is the Android low memory killer, which kills the
> task with the lowest priority in the system.
>
> The low memory killer will iterate over all the tasks
> in the system to identify the task to kill.
>
> This is not a problem in Android systems, and other
> small systems where this piece of code is used.
>
> What kind of system are you trying to use the low
> memory killer on?
>
> How many tasks are you running?
>
yes, lowmemorykiller kills the task depending on its oom_score, I am using
a embedded device with 2GB memory, there are task running that cause
lowmemory situation - no issue about it.

But my concern is kswapd takes too long to iterate through all the
processes(lowmem_shrink() => for_each_process()), the time taken is around
150msec, due to which my high priority application suffer system latency
that cause malfunctioning.

Thanks,
Vaibhav

[-- Attachment #2: Type: text/html, Size: 1645 bytes --]

Re: kswapd hogging in lowmem_shrink

[Rik van Riel]

On 03/16/2015 03:45 AM, Vaibhav Shinde wrote:
> 
> 
> On Fri, Mar 13, 2015 at 7:52 AM, Rik van Riel <riel@redhat.com
> <mailto:riel@redhat.com>> wrote:
>>
>> On 03/13/2015 10:25 AM, Vaibhav Shinde wrote:
>> >
>> > On low memory situation, I see various shrinkers being invoked, but in
>> > lowmem_shrink() case, kswapd is found to be hogging for around 150msecs.
>> >
>> > Due to this my application suffer latency issue, as the cpu was not
>> > released by kswapd0.
>> >
>> > I took below traces with vmscan events, that show lowmem_shrink taking
>> > such long time for execution.
>>
>> This is the Android low memory killer, which kills the
>> task with the lowest priority in the system.
>>
>> The low memory killer will iterate over all the tasks
>> in the system to identify the task to kill.
>>
>> This is not a problem in Android systems, and other
>> small systems where this piece of code is used.
>>
>> What kind of system are you trying to use the low
>> memory killer on?
>>
>> How many tasks are you running?
>>
> yes, lowmemorykiller kills the task depending on its oom_score, I am
> using a embedded device with 2GB memory, there are task running that
> cause lowmemory situation - no issue about it.
> 
> But my concern is kswapd takes too long to iterate through all the
> processes(lowmem_shrink() => for_each_process()), the time taken is
> around 150msec, due to which my high priority application suffer system
> latency that cause malfunctioning.

If it is an issue for you, you will have to fix that.

-- 
All rights reversed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: kswapd hogging in lowmem_shrink

[Michal Hocko]

On Fri 13-03-15 19:55:27, Vaibhav Shinde wrote:
> On low memory situation, I see various shrinkers being invoked, but in
> lowmem_shrink() case, kswapd is found to be hogging for around 150msecs.
> 
> Due to this my application suffer latency issue, as the cpu was not
> released by kswapd0.
> 
> I took below traces with vmscan events, that show lowmem_shrink taking such
> long time for execution.
> 
> kswapd0-67 [003] ...1  1501.987110: mm_shrink_slab_start:
> lowmem_shrink+0x0/0x580 c0ee8e34: objects to shrink 122 gfp_flags
> GFP_KERNEL pgs_scanned 83 lru_pgs 241753 cache items 241754 delta 10
> total_scan 132
> kswapd0-67 [003] ...1  1502.020827: mm_shrink_slab_end:
> lowmem_shrink+0x0/0x580 c0ee8e34: unused scan count 122 new scan count 4
> total_scan -118 last shrinker return val 237339
> 
> Please provide inputs on the same.

I would strongly discourage from using lowmemory killer. It is broken by
design IMHO. It can spend a lot of time looping on a large machine. Why
do you use it in the first place?

-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>