isolate_freepages_block(): very high intermittent overhead

isolate_freepages_block(): very high intermittent overhead

[Markus Trippelsdorf]

On my v3.18-rc2 kernel isolate_freepages_block() sometimes shows up very
high (>20%) in perf top during the configuration phase of software
builds. It increases build time considerably.

Unfortunately the issue is not 100% reproducible, because it appears
only intermittently. And the symptoms vanish after a few minutes.

I think the "mm, compaction" series from Vlastimil is to blame, but it's
hard to be sure when bisection doesn't work.

-- 
Markus

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: isolate_freepages_block(): very high intermittent overhead

[Markus Trippelsdorf]

On 2014.10.27 at 21:40 +0100, Markus Trippelsdorf wrote:
> On my v3.18-rc2 kernel isolate_freepages_block() sometimes shows up very
> high (>20%) in perf top during the configuration phase of software
> builds. It increases build time considerably.
> 
> Unfortunately the issue is not 100% reproducible, because it appears
> only intermittently. And the symptoms vanish after a few minutes.
> 
> I think the "mm, compaction" series from Vlastimil is to blame, but it's
> hard to be sure when bisection doesn't work.

Here is an example:

Overhead  Shared Object      Symbol
59.12%    [kernel]           [k] isolate_freepages_block
4.75%     [kernel]           [k] amd_e400_idle
1.10%     [kernel]           [k] clear_page_c
0.89%     [kernel]           [k] get_pfnblock_flags_mask
0.64%     ld-2.19.90.so      [.] do_lookup_x
0.64%     [kernel]           [k] unmap_single_vma
0.58%     [kernel]           [k] copy_page_rep
0.54%     libc-2.19.90.so    [.] memcpy@@GLIBC_2.14
0.52%     [kernel]           [k] filemap_map_pages
0.47%     [kernel]           [k] _cond_resched

-- 
Markus

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: isolate_freepages_block(): very high intermittent overhead

[Vlastimil Babka]

On 10/27/2014 09:40 PM, Markus Trippelsdorf wrote:
> On my v3.18-rc2 kernel isolate_freepages_block() sometimes shows up very
> high (>20%) in perf top during the configuration phase of software
> builds. It increases build time considerably.
> 
> Unfortunately the issue is not 100% reproducible, because it appears
> only intermittently. And the symptoms vanish after a few minutes.

Does it happen for long enough so you can capture it by perf record -g ?

Vlastimil

> I think the "mm, compaction" series from Vlastimil is to blame, but it's
> hard to be sure when bisection doesn't work.
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: isolate_freepages_block(): very high intermittent overhead

[Markus Trippelsdorf]

On 2014.10.27 at 23:01 +0100, Vlastimil Babka wrote:
> On 10/27/2014 09:40 PM, Markus Trippelsdorf wrote:
> > On my v3.18-rc2 kernel isolate_freepages_block() sometimes shows up very
> > high (>20%) in perf top during the configuration phase of software
> > builds. It increases build time considerably.
> > 
> > Unfortunately the issue is not 100% reproducible, because it appears
> > only intermittently. And the symptoms vanish after a few minutes.
> 
> Does it happen for long enough so you can capture it by perf record -g ?

It only happens when I use the "Lockless Allocator":
http://locklessinc.com/downloads/lockless_allocator_src.tgz

I use: LD_PRELOAD=/usr/lib/libllalloc.so.1.3 when building software,
because it gives me a ~8% speed boost over glibc's malloc.

Unfortunately, I don't have time to debug this further and have disabled 
"Transparent Hugepage Support" for now.

-- 
Markus

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: isolate_freepages_block(): very high intermittent overhead

[Vlastimil Babka]

On 10/28/2014 09:59 AM, Markus Trippelsdorf wrote:
> On 2014.10.27 at 23:01 +0100, Vlastimil Babka wrote:
>> On 10/27/2014 09:40 PM, Markus Trippelsdorf wrote:
>>> On my v3.18-rc2 kernel isolate_freepages_block() sometimes shows up very
>>> high (>20%) in perf top during the configuration phase of software
>>> builds. It increases build time considerably.
>>>
>>> Unfortunately the issue is not 100% reproducible, because it appears
>>> only intermittently. And the symptoms vanish after a few minutes.
>>
>> Does it happen for long enough so you can capture it by perf record -g ?
>
> It only happens when I use the "Lockless Allocator":
> http://locklessinc.com/downloads/lockless_allocator_src.tgz
>
> I use: LD_PRELOAD=/usr/lib/libllalloc.so.1.3 when building software,
> because it gives me a ~8% speed boost over glibc's malloc.

Hm I see. I'll try to test that.

> Unfortunately, I don't have time to debug this further and have disabled
> "Transparent Hugepage Support" for now.

That's unfortunate indeed. Commit 
e14c720efdd73c6d69cd8d07fa894bcd11fe1973 "mm, compaction: remember 
position within pageblock in free pages scanner" would be the most 
suspicious one here I guess, so testing at least a kernel with this 
patch reverted would be very useful. Simple git revert seems to apply 
cleanly here.

Thanks,
Vlastimil

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: isolate_freepages_block(): very high intermittent overhead

[Vlastimil Babka]

On 10/28/2014 09:59 AM, Markus Trippelsdorf wrote:
> On 2014.10.27 at 23:01 +0100, Vlastimil Babka wrote:
>> On 10/27/2014 09:40 PM, Markus Trippelsdorf wrote:
>> > On my v3.18-rc2 kernel isolate_freepages_block() sometimes shows up very
>> > high (>20%) in perf top during the configuration phase of software
>> > builds. It increases build time considerably.
>> > 
>> > Unfortunately the issue is not 100% reproducible, because it appears
>> > only intermittently. And the symptoms vanish after a few minutes.
>> 
>> Does it happen for long enough so you can capture it by perf record -g ?
> 
> It only happens when I use the "Lockless Allocator":
> http://locklessinc.com/downloads/lockless_allocator_src.tgz
> 
> I use: LD_PRELOAD=/usr/lib/libllalloc.so.1.3 when building software,
> because it gives me a ~8% speed boost over glibc's malloc.

I tried the allocator while updating my gentoo desktop with 3.18-rc3 and adding
some extra memory pressure, but didn't observe anything like this. It could be
system specific. If you do't have the time to debug, can you at least send me
output of "cat /proc/zoneinfo"?

Thanks,
Vlastimil

> Unfortunately, I don't have time to debug this further and have disabled 
> "Transparent Hugepage Support" for now.
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: isolate_freepages_block(): very high intermittent overhead

[Markus Trippelsdorf]

On 2014.11.06 at 12:42 +0100, Vlastimil Babka wrote:
> On 10/28/2014 09:59 AM, Markus Trippelsdorf wrote:
> > On 2014.10.27 at 23:01 +0100, Vlastimil Babka wrote:
> >> On 10/27/2014 09:40 PM, Markus Trippelsdorf wrote:
> >> > On my v3.18-rc2 kernel isolate_freepages_block() sometimes shows up very
> >> > high (>20%) in perf top during the configuration phase of software
> >> > builds. It increases build time considerably.
> >> > 
> >> > Unfortunately the issue is not 100% reproducible, because it appears
> >> > only intermittently. And the symptoms vanish after a few minutes.
> >> 
> >> Does it happen for long enough so you can capture it by perf record -g ?
> > 
> > It only happens when I use the "Lockless Allocator":
> > http://locklessinc.com/downloads/lockless_allocator_src.tgz
> > 
> > I use: LD_PRELOAD=/usr/lib/libllalloc.so.1.3 when building software,
> > because it gives me a ~8% speed boost over glibc's malloc.
> 
> I tried the allocator while updating my gentoo desktop with 3.18-rc3 and adding
> some extra memory pressure, but didn't observe anything like this. It could be
> system specific. If you do't have the time to debug, can you at least send me
> output of "cat /proc/zoneinfo"?

I will try to debug this further this weekend.
BTW there is an interesting thread on LKML that might be related to this
issue: https://lkml.org/lkml/2014/11/4/904

-- 
Markus

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>