linux-arm-kernel.lists.infradead.org archive mirror
 help / color / mirror / Atom feed
From: Mel Gorman <mgorman@techsingularity.net>
To: Vincent Guittot <vincent.guittot@linaro.org>
Cc: Juri Lelli <juri.lelli@redhat.com>,
	Barry Song <song.bao.hua@hisilicon.com>,
	Peter Ziljstra <peterz@infradead.org>,
	Aubrey Li <aubrey.li@linux.intel.com>,
	LKML <linux-kernel@vger.kernel.org>,
	Ingo Molnar <mingo@redhat.com>,
	Valentin Schneider <valentin.schneider@arm.com>,
	Linux-ARM <linux-arm-kernel@lists.infradead.org>
Subject: Re: [PATCH 0/4] Reduce scanning of runqueues in select_idle_sibling
Date: Thu, 10 Dec 2020 11:04:24 +0000	[thread overview]
Message-ID: <20201210110424.GR3371@techsingularity.net> (raw)
In-Reply-To: <CAKfTPtBigdckTCpYyVD9R0U9Mm+ENX7BJ3gqrTkPW3m64iPCVA@mail.gmail.com>

On Thu, Dec 10, 2020 at 10:38:37AM +0100, Vincent Guittot wrote:
> > while testing your patchset and Aubrey one on top of tip, I'm facing
> > some perf regression on my arm64 numa system on hackbench and reaim.
> > The regression seems to comes from your patchset but i don't know
> > which patch in particular yet
> >
> > hackbench -l 256000 -g 1
> >
> > v5.10-rc7 + tip/sched/core 13,255(+/- 3.22%)
> > with your patchset         15.368(+/- 2.74)  -15.9%
> >
> > I'm also seeing perf regression on reaim but this one needs more
> > investigation before confirming
> >
> > TBH, I was not expecting regressions. I'm running more test to find
> > which patch is the culprit
> 
> The regression comes from patch 3: sched/fair: Do not replace
> recent_used_cpu with the new target
> 

That's not entirely surprising. The intent of the patch is to increase the
hit rate of p->recent_used_cpu but it's not a guaranteed win due to two
corner cases. If multiple tasks have the same p->recent_used_cpu, they can
race to use that CPU and stack as a result instead of searching the domain.
If SMT is enabled then p->recent_used_cpu can point to an idle CPU that has
a busy sibling which the search would have avoided in select_idle_core().

I think you are using processes and sockets for hackbench but as you'll
see later, hackbench can be used both to show losses and gains.

I originally tested with 6 machines covering Broadwell (2 socket), Haswell
(2 socket), Skylake (1 socket), Cascadelake (2 socket), EPYC (2 socket)
and EPYC 2 (2 socket) with a range of workloads including hackbench. Of
those, just one reported a major problem with 1 group -- the EPYC 1 machine

EPYC hackbench process-sockets
                          5.10.0-rc6             5.10.0-rc6
                       baseline-v2r2         altrecent-v2r5
Amean     1        1.0607 (   0.00%)      1.1480 (  -8.23%)
Amean     4        1.3277 (   0.00%)      1.3117 (   1.21%)
Amean     7        1.6940 (   0.00%)      1.6950 (  -0.06%)
Amean     12       2.1600 (   0.00%)      2.1367 (   1.08%)
Amean     21       3.2450 (   0.00%)      3.5883 ( -10.58%)
Amean     30       4.1673 (   0.00%)      3.9653 (   4.85%)
Amean     48       4.9257 (   0.00%)      5.0000 (  -1.51%)
Amean     79       7.4950 (   0.00%)      7.4563 (   0.52%)
Amean     110     10.4233 (   0.00%)     10.4727 (  -0.47%)
Amean     141     13.4690 (   0.00%)     13.4563 (   0.09%)
Amean     172     16.6450 (   0.00%)     16.6033 (   0.25%)
Amean     203     19.4873 (   0.00%)     19.7893 *  -1.55%*
Amean     234     22.5507 (   0.00%)     22.8033 (  -1.12%)
Amean     265     25.3380 (   0.00%)     25.6490 (  -1.23%)
Amean     296     28.0070 (   0.00%)     28.1270 (  -0.43%)

That's showing an 8% loss for 1 group and also a problem with 21 groups.
Otherwise, it was more or less flat. EPYC 2 also showed a 2% loss for 1
group and 9% loss for 21 groups (probably related to the size of the LLC
domain as there are many LLCs per socket on EPYC*).

For the *same* machine running hackbench using pipes instead of sockets
we get

EPYC hackbench process-pipes
Amean     1        0.9497 (   0.00%)      0.9517 (  -0.21%)
Amean     4        1.2253 (   0.00%)      1.1387 (   7.07%)
Amean     7        2.0677 (   0.00%)      1.7460 *  15.56%*
Amean     12       2.8717 (   0.00%)      2.4797 *  13.65%*
Amean     21       4.4053 (   0.00%)      3.7463 *  14.96%*
Amean     30       5.3983 (   0.00%)      4.1097 *  23.87%*
Amean     48       6.1050 (   0.00%)      4.6873 *  23.22%*
Amean     79       7.5640 (   0.00%)      6.8493 (   9.45%)
Amean     110     12.2627 (   0.00%)      9.4613 *  22.84%*
Amean     141     16.9980 (   0.00%)     13.8327 *  18.62%*
Amean     172     21.5280 (   0.00%)     17.3693 *  19.32%*
Amean     203     25.4480 (   0.00%)     20.9947 *  17.50%*
Amean     234     29.6570 (   0.00%)     24.9613 *  15.83%*
Amean     265     33.0713 (   0.00%)     28.1103 *  15.00%*
Amean     296     37.4443 (   0.00%)     31.8757 *  14.87%*

So even on the *same hardware*, hackbench can show very different results
depending on how it is run.

The rest of the machines were more or less neutral for this patch. Once
hackbench saturates the machine, the hit rate on recent_used_cpu is going
to be low

1-socket skylake
Amean     1       1.3183 (   0.00%)      1.2827 *   2.71%*
Amean     3       3.6750 (   0.00%)      3.6610 (   0.38%)
Amean     5       6.1003 (   0.00%)      6.0190 *   1.33%*
Amean     7       8.6063 (   0.00%)      8.6047 (   0.02%)
Amean     12     14.9480 (   0.00%)     15.0327 (  -0.57%)
Amean     18     22.3430 (   0.00%)     22.6680 (  -1.45%)
Amean     24     29.4970 (   0.00%)     29.6677 (  -0.58%)
Amean     30     36.7373 (   0.00%)     36.3687 (   1.00%)
Amean     32     39.0973 (   0.00%)     39.4417 (  -0.88%)

Shows a 2.71% gain for one group, otherwise more or less neutral

2-socket CascadeLake

Amean     1        0.3663 (   0.00%)      0.3657 (   0.18%)
Amean     4        0.7510 (   0.00%)      0.7793 (  -3.77%)
Amean     7        1.2650 (   0.00%)      1.2583 (   0.53%)
Amean     12       1.9510 (   0.00%)      1.9450 (   0.31%)
Amean     21       2.9677 (   0.00%)      3.0277 (  -2.02%)
Amean     30       4.2993 (   0.00%)      4.0237 *   6.41%*
Amean     48       6.5373 (   0.00%)      6.2987 *   3.65%*
Amean     79      10.5513 (   0.00%)     10.3280 (   2.12%)
Amean     110     15.8567 (   0.00%)     13.9817 (  11.82%)
Amean     141     17.4243 (   0.00%)     17.3177 (   0.61%)
Amean     172     21.0473 (   0.00%)     20.9760 (   0.34%)
Amean     203     25.1070 (   0.00%)     25.1150 (  -0.03%)
Amean     234     28.6753 (   0.00%)     28.9383 (  -0.92%)
Amean     265     32.7970 (   0.00%)     32.9663 (  -0.52%)
Amean     296     36.6510 (   0.00%)     36.6753 (  -0.07%)

Neutral for 1 group, small regression for 4 groups, few gains around the
middle, neutral when over-saturated.

select_idle_sibling is a curse because it's very rare that a change to
it is a universal win. On balance, I think it's better to avoid searching
the domain at all where possible even if there are cases where searching
can have a benefit such as finding an idle core instead of picking an
idle CPU with a busy sibling via p->recent_used_cpu.

-- 
Mel Gorman
SUSE Labs

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

  reply	other threads:[~2020-12-10 11:05 UTC|newest]

Thread overview: 28+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-12-08 15:34 [PATCH 0/4] Reduce scanning of runqueues in select_idle_sibling Mel Gorman
2020-12-08 15:34 ` [PATCH 1/4] sched/fair: Remove SIS_AVG_CPU Mel Gorman
2020-12-08 16:13   ` Vincent Guittot
2020-12-08 15:34 ` [PATCH 2/4] sched/fair: Move avg_scan_cost calculations under SIS_PROP Mel Gorman
2020-12-08 16:03   ` Vincent Guittot
2020-12-08 16:30     ` Mel Gorman
2020-12-09  5:28     ` Li, Aubrey
2020-12-09  9:05       ` Mel Gorman
2020-12-09 11:07         ` Li, Aubrey
2020-12-09 11:33           ` Mel Gorman
2020-12-10  5:18   ` Li, Aubrey
2020-12-10  9:32     ` Mel Gorman
2020-12-08 15:35 ` [PATCH 3/4] sched/fair: Do not replace recent_used_cpu with the new target Mel Gorman
2020-12-08 16:14   ` Vincent Guittot
2020-12-10  9:40     ` Vincent Guittot
2020-12-11  6:25   ` Hillf Danton
2020-12-11  9:02     ` Mel Gorman
2020-12-11  9:34       ` Hillf Danton
2020-12-11  9:45         ` Mel Gorman
2020-12-08 15:35 ` [PATCH 4/4] sched/fair: Return an idle cpu if one is found after a failed search for an idle core Mel Gorman
2020-12-08 16:15   ` Vincent Guittot
2020-12-09 14:37 ` [PATCH 0/4] Reduce scanning of runqueues in select_idle_sibling Mel Gorman
2020-12-10  8:00   ` Vincent Guittot
2020-12-10  9:38     ` Vincent Guittot
2020-12-10 11:04       ` Mel Gorman [this message]
2020-12-11  9:51         ` Vincent Guittot
2020-12-11 10:23           ` Mel Gorman
2020-12-12 10:02             ` Vincent Guittot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20201210110424.GR3371@techsingularity.net \
    --to=mgorman@techsingularity.net \
    --cc=aubrey.li@linux.intel.com \
    --cc=juri.lelli@redhat.com \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@redhat.com \
    --cc=peterz@infradead.org \
    --cc=song.bao.hua@hisilicon.com \
    --cc=valentin.schneider@arm.com \
    --cc=vincent.guittot@linaro.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).