From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lb0-f174.google.com (mail-lb0-f174.google.com [209.85.217.174]) by kanga.kvack.org (Postfix) with ESMTP id 22F31900021 for ; Mon, 27 Oct 2014 16:40:07 -0400 (EDT) Received: by mail-lb0-f174.google.com with SMTP id p9so6422833lbv.19 for ; Mon, 27 Oct 2014 13:40:06 -0700 (PDT) Received: from mail.ud10.udmedia.de (ud10.udmedia.de. [194.117.254.50]) by mx.google.com with ESMTPS id j9si21754399lab.13.2014.10.27.13.40.04 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 27 Oct 2014 13:40:05 -0700 (PDT) Date: Mon, 27 Oct 2014 21:40:03 +0100 From: Markus Trippelsdorf Subject: isolate_freepages_block(): very high intermittent overhead Message-ID: <20141027204003.GB348@x4> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Sender: owner-linux-mm@kvack.org List-ID: To: linux-mm@kvack.org Cc: Vlastimil Babka On my v3.18-rc2 kernel isolate_freepages_block() sometimes shows up very high (>20%) in perf top during the configuration phase of software builds. It increases build time considerably. Unfortunately the issue is not 100% reproducible, because it appears only intermittently. And the symptoms vanish after a few minutes. I think the "mm, compaction" series from Vlastimil is to blame, but it's hard to be sure when bisection doesn't work. -- Markus -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lb0-f179.google.com (mail-lb0-f179.google.com [209.85.217.179]) by kanga.kvack.org (Postfix) with ESMTP id 0B140900021 for ; Mon, 27 Oct 2014 17:06:59 -0400 (EDT) Received: by mail-lb0-f179.google.com with SMTP id w7so1873031lbi.38 for ; Mon, 27 Oct 2014 14:06:59 -0700 (PDT) Received: from mail.ud10.udmedia.de (ud10.udmedia.de. [194.117.254.50]) by mx.google.com with ESMTPS id t13si20631657lal.121.2014.10.27.14.06.57 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 27 Oct 2014 14:06:58 -0700 (PDT) Date: Mon, 27 Oct 2014 22:06:56 +0100 From: Markus Trippelsdorf Subject: Re: isolate_freepages_block(): very high intermittent overhead Message-ID: <20141027210656.GC348@x4> References: <20141027204003.GB348@x4> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20141027204003.GB348@x4> Sender: owner-linux-mm@kvack.org List-ID: To: linux-mm@kvack.org Cc: Vlastimil Babka On 2014.10.27 at 21:40 +0100, Markus Trippelsdorf wrote: > On my v3.18-rc2 kernel isolate_freepages_block() sometimes shows up very > high (>20%) in perf top during the configuration phase of software > builds. It increases build time considerably. > > Unfortunately the issue is not 100% reproducible, because it appears > only intermittently. And the symptoms vanish after a few minutes. > > I think the "mm, compaction" series from Vlastimil is to blame, but it's > hard to be sure when bisection doesn't work. Here is an example: Overhead Shared Object Symbol 59.12% [kernel] [k] isolate_freepages_block 4.75% [kernel] [k] amd_e400_idle 1.10% [kernel] [k] clear_page_c 0.89% [kernel] [k] get_pfnblock_flags_mask 0.64% ld-2.19.90.so [.] do_lookup_x 0.64% [kernel] [k] unmap_single_vma 0.58% [kernel] [k] copy_page_rep 0.54% libc-2.19.90.so [.] memcpy@@GLIBC_2.14 0.52% [kernel] [k] filemap_map_pages 0.47% [kernel] [k] _cond_resched -- Markus -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-la0-f44.google.com (mail-la0-f44.google.com [209.85.215.44]) by kanga.kvack.org (Postfix) with ESMTP id F35E4900021 for ; Mon, 27 Oct 2014 18:01:44 -0400 (EDT) Received: by mail-la0-f44.google.com with SMTP id gf13so380239lab.31 for ; Mon, 27 Oct 2014 15:01:44 -0700 (PDT) Received: from mx2.suse.de (cantor2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id g7si22011422lbs.0.2014.10.27.15.01.42 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 27 Oct 2014 15:01:43 -0700 (PDT) Message-ID: <544EC0C5.7050808@suse.cz> Date: Mon, 27 Oct 2014 23:01:41 +0100 From: Vlastimil Babka MIME-Version: 1.0 Subject: Re: isolate_freepages_block(): very high intermittent overhead References: <20141027204003.GB348@x4> In-Reply-To: <20141027204003.GB348@x4> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Markus Trippelsdorf , linux-mm@kvack.org On 10/27/2014 09:40 PM, Markus Trippelsdorf wrote: > On my v3.18-rc2 kernel isolate_freepages_block() sometimes shows up very > high (>20%) in perf top during the configuration phase of software > builds. It increases build time considerably. > > Unfortunately the issue is not 100% reproducible, because it appears > only intermittently. And the symptoms vanish after a few minutes. Does it happen for long enough so you can capture it by perf record -g ? Vlastimil > I think the "mm, compaction" series from Vlastimil is to blame, but it's > hard to be sure when bisection doesn't work. > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lb0-f179.google.com (mail-lb0-f179.google.com [209.85.217.179]) by kanga.kvack.org (Postfix) with ESMTP id F2BC6900021 for ; Tue, 28 Oct 2014 04:59:19 -0400 (EDT) Received: by mail-lb0-f179.google.com with SMTP id w7so177587lbi.24 for ; Tue, 28 Oct 2014 01:59:19 -0700 (PDT) Received: from mail.ud10.udmedia.de (ud10.udmedia.de. [194.117.254.50]) by mx.google.com with ESMTPS id ra5si1295010lbb.137.2014.10.28.01.59.17 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Oct 2014 01:59:18 -0700 (PDT) Date: Tue, 28 Oct 2014 09:59:16 +0100 From: Markus Trippelsdorf Subject: Re: isolate_freepages_block(): very high intermittent overhead Message-ID: <20141028085916.GA337@x4> References: <20141027204003.GB348@x4> <544EC0C5.7050808@suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <544EC0C5.7050808@suse.cz> Sender: owner-linux-mm@kvack.org List-ID: To: Vlastimil Babka Cc: linux-mm@kvack.org On 2014.10.27 at 23:01 +0100, Vlastimil Babka wrote: > On 10/27/2014 09:40 PM, Markus Trippelsdorf wrote: > > On my v3.18-rc2 kernel isolate_freepages_block() sometimes shows up very > > high (>20%) in perf top during the configuration phase of software > > builds. It increases build time considerably. > > > > Unfortunately the issue is not 100% reproducible, because it appears > > only intermittently. And the symptoms vanish after a few minutes. > > Does it happen for long enough so you can capture it by perf record -g ? It only happens when I use the "Lockless Allocator": http://locklessinc.com/downloads/lockless_allocator_src.tgz I use: LD_PRELOAD=/usr/lib/libllalloc.so.1.3 when building software, because it gives me a ~8% speed boost over glibc's malloc. Unfortunately, I don't have time to debug this further and have disabled "Transparent Hugepage Support" for now. -- Markus -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lb0-f174.google.com (mail-lb0-f174.google.com [209.85.217.174]) by kanga.kvack.org (Postfix) with ESMTP id 7B616900021 for ; Wed, 29 Oct 2014 11:08:12 -0400 (EDT) Received: by mail-lb0-f174.google.com with SMTP id z11so665803lbi.19 for ; Wed, 29 Oct 2014 08:08:11 -0700 (PDT) Received: from mx2.suse.de (cantor2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id m7si7574180lah.97.2014.10.29.08.08.09 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 29 Oct 2014 08:08:09 -0700 (PDT) Message-ID: <545102D1.6080908@suse.cz> Date: Wed, 29 Oct 2014 16:08:01 +0100 From: Vlastimil Babka MIME-Version: 1.0 Subject: Re: isolate_freepages_block(): very high intermittent overhead References: <20141027204003.GB348@x4> <544EC0C5.7050808@suse.cz> <20141028085916.GA337@x4> In-Reply-To: <20141028085916.GA337@x4> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Markus Trippelsdorf Cc: linux-mm@kvack.org On 10/28/2014 09:59 AM, Markus Trippelsdorf wrote: > On 2014.10.27 at 23:01 +0100, Vlastimil Babka wrote: >> On 10/27/2014 09:40 PM, Markus Trippelsdorf wrote: >>> On my v3.18-rc2 kernel isolate_freepages_block() sometimes shows up very >>> high (>20%) in perf top during the configuration phase of software >>> builds. It increases build time considerably. >>> >>> Unfortunately the issue is not 100% reproducible, because it appears >>> only intermittently. And the symptoms vanish after a few minutes. >> >> Does it happen for long enough so you can capture it by perf record -g ? > > It only happens when I use the "Lockless Allocator": > http://locklessinc.com/downloads/lockless_allocator_src.tgz > > I use: LD_PRELOAD=/usr/lib/libllalloc.so.1.3 when building software, > because it gives me a ~8% speed boost over glibc's malloc. Hm I see. I'll try to test that. > Unfortunately, I don't have time to debug this further and have disabled > "Transparent Hugepage Support" for now. That's unfortunate indeed. Commit e14c720efdd73c6d69cd8d07fa894bcd11fe1973 "mm, compaction: remember position within pageblock in free pages scanner" would be the most suspicious one here I guess, so testing at least a kernel with this patch reverted would be very useful. Simple git revert seems to apply cleanly here. Thanks, Vlastimil -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f172.google.com (mail-wi0-f172.google.com [209.85.212.172]) by kanga.kvack.org (Postfix) with ESMTP id 348F66B0096 for ; Thu, 6 Nov 2014 06:42:11 -0500 (EST) Received: by mail-wi0-f172.google.com with SMTP id bs8so1173633wib.17 for ; Thu, 06 Nov 2014 03:42:10 -0800 (PST) Received: from mx2.suse.de (cantor2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id p5si9468972wij.23.2014.11.06.03.42.09 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 06 Nov 2014 03:42:09 -0800 (PST) Message-ID: <545B5E90.6070902@suse.cz> Date: Thu, 06 Nov 2014 12:42:08 +0100 From: Vlastimil Babka MIME-Version: 1.0 Subject: Re: isolate_freepages_block(): very high intermittent overhead References: <20141027204003.GB348@x4> <544EC0C5.7050808@suse.cz> <20141028085916.GA337@x4> In-Reply-To: <20141028085916.GA337@x4> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Markus Trippelsdorf Cc: linux-mm@kvack.org On 10/28/2014 09:59 AM, Markus Trippelsdorf wrote: > On 2014.10.27 at 23:01 +0100, Vlastimil Babka wrote: >> On 10/27/2014 09:40 PM, Markus Trippelsdorf wrote: >> > On my v3.18-rc2 kernel isolate_freepages_block() sometimes shows up very >> > high (>20%) in perf top during the configuration phase of software >> > builds. It increases build time considerably. >> > >> > Unfortunately the issue is not 100% reproducible, because it appears >> > only intermittently. And the symptoms vanish after a few minutes. >> >> Does it happen for long enough so you can capture it by perf record -g ? > > It only happens when I use the "Lockless Allocator": > http://locklessinc.com/downloads/lockless_allocator_src.tgz > > I use: LD_PRELOAD=/usr/lib/libllalloc.so.1.3 when building software, > because it gives me a ~8% speed boost over glibc's malloc. I tried the allocator while updating my gentoo desktop with 3.18-rc3 and adding some extra memory pressure, but didn't observe anything like this. It could be system specific. If you do't have the time to debug, can you at least send me output of "cat /proc/zoneinfo"? Thanks, Vlastimil > Unfortunately, I don't have time to debug this further and have disabled > "Transparent Hugepage Support" for now. > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f169.google.com (mail-wi0-f169.google.com [209.85.212.169]) by kanga.kvack.org (Postfix) with ESMTP id 743536B0099 for ; Thu, 6 Nov 2014 06:53:25 -0500 (EST) Received: by mail-wi0-f169.google.com with SMTP id n3so1180623wiv.2 for ; Thu, 06 Nov 2014 03:53:24 -0800 (PST) Received: from mail.ud10.udmedia.de (ud10.udmedia.de. [194.117.254.50]) by mx.google.com with ESMTPS id j16si9496797wic.43.2014.11.06.03.53.24 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Nov 2014 03:53:24 -0800 (PST) Date: Thu, 6 Nov 2014 12:53:22 +0100 From: Markus Trippelsdorf Subject: Re: isolate_freepages_block(): very high intermittent overhead Message-ID: <20141106115322.GA17467@x4> References: <20141027204003.GB348@x4> <544EC0C5.7050808@suse.cz> <20141028085916.GA337@x4> <545B5E90.6070902@suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <545B5E90.6070902@suse.cz> Sender: owner-linux-mm@kvack.org List-ID: To: Vlastimil Babka Cc: linux-mm@kvack.org On 2014.11.06 at 12:42 +0100, Vlastimil Babka wrote: > On 10/28/2014 09:59 AM, Markus Trippelsdorf wrote: > > On 2014.10.27 at 23:01 +0100, Vlastimil Babka wrote: > >> On 10/27/2014 09:40 PM, Markus Trippelsdorf wrote: > >> > On my v3.18-rc2 kernel isolate_freepages_block() sometimes shows up very > >> > high (>20%) in perf top during the configuration phase of software > >> > builds. It increases build time considerably. > >> > > >> > Unfortunately the issue is not 100% reproducible, because it appears > >> > only intermittently. And the symptoms vanish after a few minutes. > >> > >> Does it happen for long enough so you can capture it by perf record -g ? > > > > It only happens when I use the "Lockless Allocator": > > http://locklessinc.com/downloads/lockless_allocator_src.tgz > > > > I use: LD_PRELOAD=/usr/lib/libllalloc.so.1.3 when building software, > > because it gives me a ~8% speed boost over glibc's malloc. > > I tried the allocator while updating my gentoo desktop with 3.18-rc3 and adding > some extra memory pressure, but didn't observe anything like this. It could be > system specific. If you do't have the time to debug, can you at least send me > output of "cat /proc/zoneinfo"? I will try to debug this further this weekend. BTW there is an interesting thread on LKML that might be related to this issue: https://lkml.org/lkml/2014/11/4/904 -- Markus -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org