From: Ingo Molnar <mingo@kernel.org>
To: Mel Gorman <mgorman@suse.de>
Cc: Rik van Riel <riel@redhat.com>,
Thomas Gleixner <tglx@linutronix.de>,
linux-kernel@vger.kernel.org, linux-mm@kvack.org,
Peter Zijlstra <a.p.zijlstra@chello.nl>,
Paul Turner <pjt@google.com>,
Lee Schermerhorn <Lee.Schermerhorn@hp.com>,
Christoph Lameter <cl@linux.com>,
Andrew Morton <akpm@linux-foundation.org>,
Andrea Arcangeli <aarcange@redhat.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
Johannes Weiner <hannes@cmpxchg.org>,
Hugh Dickins <hughd@google.com>
Subject: Re: [GIT TREE] Unified NUMA balancing tree, v3
Date: Mon, 10 Dec 2012 21:07:55 +0100 [thread overview]
Message-ID: <20121210200755.GA15097@gmail.com> (raw)
In-Reply-To: <20121210192828.GL1009@suse.de>
* Mel Gorman <mgorman@suse.de> wrote:
> On Mon, Dec 10, 2012 at 08:15:45PM +0100, Ingo Molnar wrote:
> >
> > * Rik van Riel <riel@redhat.com> wrote:
> >
> > > On 12/10/2012 01:22 PM, Thomas Gleixner wrote:
> > >
> > > > So autonuma and numacore are basically on the same page,
> > > > with a slight advantage for numacore in the THP enabled
> > > > case. balancenuma is closer to mainline than to
> > > > autonuma/numacore.
> > >
> > > Indeed, when the system is fully loaded, numacore does very
> > > well.
> >
> > Note that the latest (-v3) code also does well in under-loaded
> > situations:
> >
> > http://lkml.org/lkml/2012/12/7/331
> >
> > Here's the 'perf bench numa' comparison to 'balancenuma':
> >
> > balancenuma | NUMA-tip
> > [test unit] : -v10 | -v3
> > ------------------------------------------------------------
> > 2x1-bw-process : 6.136 | 9.647: 57.2%
> > 3x1-bw-process : 7.250 | 14.528: 100.4%
> > 4x1-bw-process : 6.867 | 18.903: 175.3%
> > 8x1-bw-process : 7.974 | 26.829: 236.5%
> > 8x1-bw-process-NOTHP : 5.937 | 22.237: 274.5%
> > 16x1-bw-process : 5.592 | 29.294: 423.9%
> > 4x1-bw-thread : 13.598 | 19.290: 41.9%
> > 8x1-bw-thread : 16.356 | 26.391: 61.4%
> > 16x1-bw-thread : 24.608 | 29.557: 20.1%
> > 32x1-bw-thread : 25.477 | 30.232: 18.7%
> > 2x3-bw-thread : 8.785 | 15.327: 74.5%
> > 4x4-bw-thread : 6.366 | 27.957: 339.2%
> > 4x6-bw-thread : 6.287 | 27.877: 343.4%
> > 4x8-bw-thread : 5.860 | 28.439: 385.3%
> > 4x8-bw-thread-NOTHP : 6.167 | 25.067: 306.5%
> > 3x3-bw-thread : 8.235 | 21.560: 161.8%
> > 5x5-bw-thread : 5.762 | 26.081: 352.6%
> > 2x16-bw-thread : 5.920 | 23.269: 293.1%
> > 1x32-bw-thread : 5.828 | 18.985: 225.8%
> > numa02-bw : 29.054 | 31.431: 8.2%
> > numa02-bw-NOTHP : 27.064 | 29.104: 7.5%
> > numa01-bw-thread : 20.338 | 28.607: 40.7%
> > numa01-bw-thread-NOTHP : 18.528 | 21.119: 14.0%
> > ------------------------------------------------------------
> >
> > More than half of these testcases are under-loaded situations.
> >
> > > The main issues that have been observed with numacore are when
> > > the system is only partially loaded. Something strange seems
> > > to be going on that causes performance regressions in that
> > > situation.
> >
> > I haven't seen such reports with -v3 yet, which is what Thomas
> > tested. Mel has not tested -v3 yet AFAICS.
> >
>
> Yes, I have. The drop I took and the results I posted to you
> were based on a tip/master pull from December 9th. v3 was
> released on December 7th and your release said to test based
> on tip/master. The results are here
> https://lkml.org/lkml/2012/12/9/108 . Look at the columns
> marked numafix-20121209 which is tip/master with a bodge on
> top to remove the "if (p->nr_cpus_allowed !=
> num_online_cpus())" check.
Ah, indeed - I saw those results but the 'numafix' tag threw me
off.
Looks like at least in terms of AutoNUMA-benchmark numbers you
measured the best-ever results with the -v3 tree? That aspect is
obviously good news.
This part isn't:
> > If there are any such instances left then I'll investigate,
> > but right now it's looking pretty good.
>
> If you had read that report, you would know that I didn't have
> results for specjbb with THP enabled due to the JVM crashing
> with null pointer exceptions.
Hm, it's the unified tree where most of the mm/ bits are the
AutoNUMA bits from your tree. (It does not match 100%, because
your tree has an ancient version of key memory usage statistics
that the scheduler needs for its convergence model. I'll take a
look at the differences.)
Given how well the unified kernel performs, and given that the
segfaults occur on your box, would you be willing to debug this
a bit and help me out fixing the bug? Thanks!
Ingo
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
WARNING: multiple messages have this Message-ID (diff)
From: Ingo Molnar <mingo@kernel.org>
To: Mel Gorman <mgorman@suse.de>
Cc: Rik van Riel <riel@redhat.com>,
Thomas Gleixner <tglx@linutronix.de>,
linux-kernel@vger.kernel.org, linux-mm@kvack.org,
Peter Zijlstra <a.p.zijlstra@chello.nl>,
Paul Turner <pjt@google.com>,
Lee Schermerhorn <Lee.Schermerhorn@hp.com>,
Christoph Lameter <cl@linux.com>,
Andrew Morton <akpm@linux-foundation.org>,
Andrea Arcangeli <aarcange@redhat.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
Johannes Weiner <hannes@cmpxchg.org>,
Hugh Dickins <hughd@google.com>
Subject: Re: [GIT TREE] Unified NUMA balancing tree, v3
Date: Mon, 10 Dec 2012 21:07:55 +0100 [thread overview]
Message-ID: <20121210200755.GA15097@gmail.com> (raw)
In-Reply-To: <20121210192828.GL1009@suse.de>
* Mel Gorman <mgorman@suse.de> wrote:
> On Mon, Dec 10, 2012 at 08:15:45PM +0100, Ingo Molnar wrote:
> >
> > * Rik van Riel <riel@redhat.com> wrote:
> >
> > > On 12/10/2012 01:22 PM, Thomas Gleixner wrote:
> > >
> > > > So autonuma and numacore are basically on the same page,
> > > > with a slight advantage for numacore in the THP enabled
> > > > case. balancenuma is closer to mainline than to
> > > > autonuma/numacore.
> > >
> > > Indeed, when the system is fully loaded, numacore does very
> > > well.
> >
> > Note that the latest (-v3) code also does well in under-loaded
> > situations:
> >
> > http://lkml.org/lkml/2012/12/7/331
> >
> > Here's the 'perf bench numa' comparison to 'balancenuma':
> >
> > balancenuma | NUMA-tip
> > [test unit] : -v10 | -v3
> > ------------------------------------------------------------
> > 2x1-bw-process : 6.136 | 9.647: 57.2%
> > 3x1-bw-process : 7.250 | 14.528: 100.4%
> > 4x1-bw-process : 6.867 | 18.903: 175.3%
> > 8x1-bw-process : 7.974 | 26.829: 236.5%
> > 8x1-bw-process-NOTHP : 5.937 | 22.237: 274.5%
> > 16x1-bw-process : 5.592 | 29.294: 423.9%
> > 4x1-bw-thread : 13.598 | 19.290: 41.9%
> > 8x1-bw-thread : 16.356 | 26.391: 61.4%
> > 16x1-bw-thread : 24.608 | 29.557: 20.1%
> > 32x1-bw-thread : 25.477 | 30.232: 18.7%
> > 2x3-bw-thread : 8.785 | 15.327: 74.5%
> > 4x4-bw-thread : 6.366 | 27.957: 339.2%
> > 4x6-bw-thread : 6.287 | 27.877: 343.4%
> > 4x8-bw-thread : 5.860 | 28.439: 385.3%
> > 4x8-bw-thread-NOTHP : 6.167 | 25.067: 306.5%
> > 3x3-bw-thread : 8.235 | 21.560: 161.8%
> > 5x5-bw-thread : 5.762 | 26.081: 352.6%
> > 2x16-bw-thread : 5.920 | 23.269: 293.1%
> > 1x32-bw-thread : 5.828 | 18.985: 225.8%
> > numa02-bw : 29.054 | 31.431: 8.2%
> > numa02-bw-NOTHP : 27.064 | 29.104: 7.5%
> > numa01-bw-thread : 20.338 | 28.607: 40.7%
> > numa01-bw-thread-NOTHP : 18.528 | 21.119: 14.0%
> > ------------------------------------------------------------
> >
> > More than half of these testcases are under-loaded situations.
> >
> > > The main issues that have been observed with numacore are when
> > > the system is only partially loaded. Something strange seems
> > > to be going on that causes performance regressions in that
> > > situation.
> >
> > I haven't seen such reports with -v3 yet, which is what Thomas
> > tested. Mel has not tested -v3 yet AFAICS.
> >
>
> Yes, I have. The drop I took and the results I posted to you
> were based on a tip/master pull from December 9th. v3 was
> released on December 7th and your release said to test based
> on tip/master. The results are here
> https://lkml.org/lkml/2012/12/9/108 . Look at the columns
> marked numafix-20121209 which is tip/master with a bodge on
> top to remove the "if (p->nr_cpus_allowed !=
> num_online_cpus())" check.
Ah, indeed - I saw those results but the 'numafix' tag threw me
off.
Looks like at least in terms of AutoNUMA-benchmark numbers you
measured the best-ever results with the -v3 tree? That aspect is
obviously good news.
This part isn't:
> > If there are any such instances left then I'll investigate,
> > but right now it's looking pretty good.
>
> If you had read that report, you would know that I didn't have
> results for specjbb with THP enabled due to the JVM crashing
> with null pointer exceptions.
Hm, it's the unified tree where most of the mm/ bits are the
AutoNUMA bits from your tree. (It does not match 100%, because
your tree has an ancient version of key memory usage statistics
that the scheduler needs for its convergence model. I'll take a
look at the differences.)
Given how well the unified kernel performs, and given that the
segfaults occur on your box, would you be willing to debug this
a bit and help me out fixing the bug? Thanks!
Ingo
next prev parent reply other threads:[~2012-12-10 20:08 UTC|newest]
Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-12-07 0:19 [GIT TREE] Unified NUMA balancing tree, v3 Ingo Molnar
2012-12-07 0:19 ` Ingo Molnar
2012-12-07 0:19 ` [PATCH 1/9] numa, sched: Fix NUMA tick ->numa_shared setting Ingo Molnar
2012-12-07 0:19 ` Ingo Molnar
2012-12-07 0:19 ` [PATCH 2/9] numa, sched: Add tracking of runnable NUMA tasks Ingo Molnar
2012-12-07 0:19 ` Ingo Molnar
2012-12-07 0:19 ` [PATCH 3/9] numa, sched: Implement wake-cpu migration support Ingo Molnar
2012-12-07 0:19 ` Ingo Molnar
2012-12-07 0:19 ` [PATCH 4/9] numa, mm, sched: Implement last-CPU+PID hash tracking Ingo Molnar
2012-12-07 0:19 ` Ingo Molnar
2012-12-07 0:19 ` [PATCH 5/9] numa, mm, sched: Fix NUMA affinity tracking logic Ingo Molnar
2012-12-07 0:19 ` Ingo Molnar
2012-12-07 0:19 ` [PATCH 6/9] numa, mm: Fix !THP, 4K-pte "2M-emu" NUMA fault handling Ingo Molnar
2012-12-07 0:19 ` Ingo Molnar
2012-12-07 0:19 ` [PATCH 7/9] numa, sched: Improve staggered convergence Ingo Molnar
2012-12-07 0:19 ` Ingo Molnar
2012-12-07 0:19 ` [PATCH 8/9] numa, sched: Improve directed convergence Ingo Molnar
2012-12-07 0:19 ` Ingo Molnar
2012-12-07 0:19 ` [PATCH 9/9] numa, sched: Streamline and fix numa_allow_migration() use Ingo Molnar
2012-12-07 0:19 ` Ingo Molnar
2012-12-10 18:22 ` [GIT TREE] Unified NUMA balancing tree, v3 Thomas Gleixner
2012-12-10 18:22 ` Thomas Gleixner
2012-12-10 18:41 ` Rik van Riel
2012-12-10 18:41 ` Rik van Riel
2012-12-10 19:15 ` Ingo Molnar
2012-12-10 19:15 ` Ingo Molnar
2012-12-10 19:28 ` Mel Gorman
2012-12-10 19:28 ` Mel Gorman
2012-12-10 20:07 ` Ingo Molnar [this message]
2012-12-10 20:07 ` Ingo Molnar
2012-12-10 20:10 ` Ingo Molnar
2012-12-10 20:10 ` Ingo Molnar
2012-12-10 21:03 ` Ingo Molnar
2012-12-10 21:03 ` Ingo Molnar
2012-12-10 22:19 ` Mel Gorman
2012-12-10 22:19 ` Mel Gorman
2012-12-10 19:32 ` Mel Gorman
2012-12-10 19:32 ` Mel Gorman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20121210200755.GA15097@gmail.com \
--to=mingo@kernel.org \
--cc=Lee.Schermerhorn@hp.com \
--cc=a.p.zijlstra@chello.nl \
--cc=aarcange@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=cl@linux.com \
--cc=hannes@cmpxchg.org \
--cc=hughd@google.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@suse.de \
--cc=pjt@google.com \
--cc=riel@redhat.com \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.