From: Mel Gorman <mgorman@suse.de>
To: Peter Zijlstra <peterz@infradead.org>
Cc: Nicholas Piggin <npiggin@gmail.com>,
linux-kernel@vger.kernel.org, Ingo Molnar <mingo@redhat.com>,
Juri Lelli <juri.lelli@redhat.com>,
Vincent Guittot <vincent.guittot@linaro.org>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ben Segall <bsegall@google.com>,
Daniel Bristot de Oliveira <bristot@redhat.com>,
Vaidyanathan Srinivasan <svaidy@linux.ibm.com>,
Srikar Dronamraju <srikar@linux.vnet.ibm.com>,
Frederic Weisbecker <fweisbec@gmail.com>
Subject: Re: [PATCH] nohz: nohz idle balancing per node
Date: Thu, 1 Jul 2021 14:11:39 +0100 [thread overview]
Message-ID: <20210701131139.GD3772@suse.de> (raw)
In-Reply-To: <YN2Wav1CSVq+6cS+@hirez.programming.kicks-ass.net>
On Thu, Jul 01, 2021 at 12:18:18PM +0200, Peter Zijlstra wrote:
> On Thu, Jul 01, 2021 at 03:53:23PM +1000, Nicholas Piggin wrote:
> > Currently a single nohz idle CPU is designated to perform balancing on
> > behalf of all other nohz idle CPUs in the system. Implement a per node
> > nohz balancer to minimize cross-node memory accesses and runqueue lock
> > acquisitions.
> >
> > On a 4 node system, this improves performance by 9.3% on a 'pgbench -N'
> > with 32 clients/jobs (which is about where throughput maxes out due to
> > IO and contention in postgres).
>
> Hmm, Suresh tried something like this around 2010 and then we ran into
> trouble that when once node went completely idle and another node was
> fully busy, the completely idle node would not run ILB and the node
> would forever stay idle.
>
An effect like that *might* be visible at
https://beta.suse.com/private/mgorman/melt/v5.13/3-perf-test/sched/sched-nohznuma-v1r1/html/network-tbench/hardy2/
at the CPU usage heatmaps ordered by topology at the very bottom of
the page.
The heatmap covers all client counts so there are "blocks" of activity for
each client count tested. The third block is for 8 thread counts so a node
is not fully busy yet. However, with the vanilla kernel, there is some
load on each node but with the patch all the load is on one node. This
did not happen on the two other test machines so the observation is not
reliable and could be a total coincidence.
That said, there were some gains but large losses depending on the client
count across the 3 machines for tbench which is a concern. Other results,
like pgbench mentioned in the changelog, will not complete until tomorrow
to see if it is a general pattern or tbench-specific.
https://beta.suse.com/private/mgorman/melt/v5.13/3-perf-test/sched/sched-nohznuma-v1r1/html/network-tbench/bing2/
https://beta.suse.com/private/mgorman/melt/v5.13/3-perf-test/sched/sched-nohznuma-v1r1/html/network-tbench/hardy2/
https://beta.suse.com/private/mgorman/melt/v5.13/3-perf-test/sched/sched-nohznuma-v1r1/html/network-tbench/marvin2/
--
Mel Gorman
SUSE Labs
next prev parent reply other threads:[~2021-07-01 13:11 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-07-01 5:53 [PATCH] nohz: nohz idle balancing per node Nicholas Piggin
2021-07-01 10:18 ` Peter Zijlstra
2021-07-01 13:11 ` Mel Gorman [this message]
2021-07-01 23:33 ` Nicholas Piggin
2021-07-02 8:49 ` Mel Gorman
2021-07-01 23:20 ` Nicholas Piggin
2021-07-01 16:41 ` kernel test robot
2021-07-01 16:48 ` kernel test robot
2021-07-07 7:50 ` Vincent Guittot
2021-07-12 13:57 ` Vincent Guittot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210701131139.GD3772@suse.de \
--to=mgorman@suse.de \
--cc=bristot@redhat.com \
--cc=bsegall@google.com \
--cc=dietmar.eggemann@arm.com \
--cc=fweisbec@gmail.com \
--cc=juri.lelli@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=npiggin@gmail.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=srikar@linux.vnet.ibm.com \
--cc=svaidy@linux.ibm.com \
--cc=vincent.guittot@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox