From: Chen Yu <yu.c.chen@intel.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: <oe-lkp@lists.linux.dev>, <lkp@intel.com>,
Oliver Sang <oliver.sang@intel.com>,
Chen Yu <yu.chen.surf@gmail.com>
Subject: Re: [peterz-queue:sched/eevdf] [sched/fair] 23669fce72: aim7.jobs-per-min -18.6% regression
Date: Thu, 23 Mar 2023 20:23:21 +0800 [thread overview]
Message-ID: <ZBxEuboumEifedjG@chenyu5-mobl1> (raw)
In-Reply-To: <20230321090318.GB2234901@hirez.programming.kicks-ass.net>
On 2023-03-21 at 10:03:18 +0100, Peter Zijlstra wrote:
> On Tue, Mar 21, 2023 at 04:04:19PM +0800, Chen Yu wrote:
> > On 2023-03-21 at 15:46:28 +0800, Oliver Sang wrote:
> > > Hi Peter Zijlstra,
> > >
> > > On Mon, Mar 20, 2023 at 08:58:50AM +0100, Peter Zijlstra wrote:
> > > > On Mon, Mar 20, 2023 at 03:46:57PM +0800, kernel test robot wrote:
> > > > >
> > > > > Greeting,
> > > > >
> > > > > FYI, we noticed a -18.6% regression of aim7.jobs-per-min due to commit:
> > > > >
> > > > >
> > > >
> > > > Hi Oliver,
> > > >
> > > > Could you do a full performance run on the current sched/eevdf
> > > > 04c54aa28eed ("sched/eevdf: Tuning...") branch?
> > >
> > > got it.
> > >
> > > we will test netperf/stress-ng/schbench/hackbench/tbench
> > >
> > Every test we will launch
> > 25%, 50%, 75%, 100%, 125%, 150%, 175%, 200% of instances per the number
> > of CPUs.
> > Both the latency and througput will be monitored.
>
> Excellent!
>
> > However currently we do not enable the latency nice for these tests. Later
> > we can enabled this once above tests finish.
>
> Yeah, just the normal numbers, thanks! No point looking at shiny new
> features when the base is all messed up ;-)
>
We have finished the first cycle of tests on a Ice Lake Server. It is a
32C * 2 Sockets system, and 128 CPUs in total.
The baseline commit is v6.3-rc2, the test commit is on top of sched/eevdf
commit 04c54aa28eed ("sched/eevdf: Tuning...")
The raw benchmark test results with perf profiling comparison are uploaded here:
schbench:
https://raw.githubusercontent.com/yu-chen-surf/schedtests/master/results/eevdf/benchmarks/schbench-lkp-icl-2sp6.log
hackbench:
https://raw.githubusercontent.com/yu-chen-surf/schedtests/master/results/eevdf/benchmarks/hackbench-lkp-icl-2sp5.log
stress-ng:
https://raw.githubusercontent.com/yu-chen-surf/schedtests/master/results/eevdf/benchmarks/stress-ng-lkp-icl-2sp6.log
unixbench:
https://raw.githubusercontent.com/yu-chen-surf/schedtests/master/results/eevdf/benchmarks/unixbench-lkp-icl-2sp2.log
netperf:
https://raw.githubusercontent.com/yu-chen-surf/schedtests/master/results/eevdf/benchmarks/netperf-lkp-icl-2sp5.log
The followings are the summary per my understanding:
schbench (95% tail latency, lower is better)
=================================================================================
case nr_instance baseline(std%) compare%( std%)
normal 25% 1.00 (2.49%) -81.2% (4.27%)
normal 50% 1.00 (2.47%) -84.5% (0.47%)
normal 75% 1.00 (2.5%) -81.3% (1.27%)
normal 100% 1.00 (3.14%) -79.2% (0.72%)
normal 125% 1.00 (3.07%) -77.5% (0.85%)
normal 150% 1.00 (3.35%) -76.4% (0.10%)
normal 175% 1.00 (3.06%) -76.2% (0.56%)
normal 200% 1.00 (3.11%) -76.3% (0.39%)
There is a universal huge win in schbench in every load range. I think this is because
the wakee is more likely to preempt current task after we reduce the vruntime by
lag during task wake up.
==================================================================================
hackbench (throughput, higher is better)
==============================================================================
case nr_instance baseline(std%) compare%( std%)
threads-pipe 25% 1.00 (<2%) -17.5 (<2%)
threads-socket 25% 1.00 (<2%) -1.9 (<2%)
threads-pipe 50% 1.00 (<2%) +6.7 (<2%)
threads-socket 50% 1.00 (<2%) -6.3 (<2%)
threads-pipe 100% 1.00 (3%) +110.1 (3%)
threads-socket 100% 1.00 (<2%) -40.2 (<2%)
threads-pipe 150% 1.00 (<2%) +125.4 (<2%)
threads-socket 150% 1.00 (<2%) -24.7 (<2%)
threads-pipe 200% 1.00 (<2%) -89.5 (<2%)
threads-socket 200% 1.00 (<2%) -27.4 (<2%)
process-pipe 25% 1.00 (<2%) -15.0 (<2%)
process-socket 25% 1.00 (<2%) -3.9 (<2%)
process-pipe 50% 1.00 (<2%) -0.4 (<2%)
process-socket 50% 1.00 (<2%) -5.3 (<2%)
process-pipe 100% 1.00 (<2%) +62.0 (<2%)
process-socket 100% 1.00 (<2%) -39.5 (<2%)
process-pipe 150% 1.00 (<2%) +70.0 (<2%)
process-socket 150% 1.00 (<2%) -20.3 (<2%)
process-pipe 200% 1.00 (<2%) +79.2 (<2%)
process-socket 200% 1.00 (<2%) -22.4 (<2%)
There are pros and cons on hackbench. Most benefits come from pipe mode. While
in socket mode, most cases have regression. I think this is related to the behavior
of hackbench: hackbench does not want to be preempted, and this
patch set somehow make wakee easier to preempt the current. For example, in 150% nr_instance
case, we saw 21.4% increasement of preemption:
75747402 ± 2% +21.4% 91946710 ± 5% hackbench.time.involuntary_context_switches
which is close to the throughput decrease ratio:
511478 -20.3% 407468 hackbench.throughput
The behavior of hackbench is that, each instance open 20 pipes, and wakes up 20 workers
via the 20 pipes. I have a stupid thought, if the wakee->wakee_flips is too high, should
we still give lag bonus to this wakee? If we give lag bonus to it, it might introduce
contention because it will bring another 20 tasks into the room.
=======================================================================================
stress-ng (throughput, higher is better)
==============================================================================
case nr_instance baseline(std%) compare%( std%)
switch 25% 1.00 (<2%) -6.5 (<2%)
switch 50% 1.00 (<2%) -9.2 (<2%)
switch 75% 1.00 (<2%) -1.2 (<2%)
switch 100% 1.00 (<2%) +11.1 (<2%)
switch 125% 1.00 (<2%) -16.7% (9%)
switch 150% 1.00 (<2%) -13.6 (<2%)
switch 175% 1.00 (<2%) -16.2 (<2%)
switch 200% 1.00 (<2%) -19.4% (<2%)
fork 50% 1.00 (<2%) -0.1 (<2%)
fork 75% 1.00 (<2%) -0.3 (<2%)
fork 100% 1.00 (<2%) -0.1 (<2%)
fork 125% 1.00 (<2%) -6.9 (<2%)
fork 150% 1.00 (<2%) -8.8 (<2%)
fork 200% 1.00 (<2%) -3.3 (<2%)
futex 25% 1.00 (<2%) -3.2 (<2%)
futex 50% 1.00 (3%) -19.9 (5%)
futex 75% 1.00 (6%) -19.1 (2%)
futex 100% 1.00 (16%) -30.5 (10%)
futex 125% 1.00 (25%) -39.3 (11%)
futex 150% 1.00 (20%) -27.2% (17%)
futex 175% 1.00 (<2%) -18.6 (<2%)
futex 200% 1.00 (<2%) -47.5 (<2%)
nanosleep 25% 1.00 (<2%) -0.1 (<2%)
nanosleep 50% 1.00 (<2%) -0.0% (<2%)
nanosleep 75% 1.00 (<2%) +15.2% (<2%)
nanosleep 100% 1.00 (<2%) -26.4 (<2%)
nanosleep 125% 1.00 (<2%) -1.3 (<2%)
nanosleep 150% 1.00 (<2%) +2.1 (<2%)
nanosleep 175% 1.00 (<2%) +8.3 (<2%)
nanosleep 200% 1.00 (<2%) +2.0% (<2%)
It seems that when the load increases, there would be regression in "switch" and
"futex" case. In the futex case, the regression seems to be caused by fewer context
switch. The stress-ng futex would create a lot of 1:1 futex_wait/futex_wake pairs.
And it seems that with the patch applied, there are more wakeup, but less successful
wakeup. It is possible that the wakers are stacked on 1 CPU which delay the
wakeup.
For example, more wakeup attempts:
49.27 ± 4% +13.4 62.63 perf-profile.calltrace.cycles-pp.futex_wake.do_futex
However less successful wakeups(context switch):
852533 ± 18% -35.0% 553996 ± 9% sched_debug.cpu.nr_switches.avg
1.01e+08 ± 24% -36.2% 64471512 ± 9% stress-ng.time.involuntary_context_switches
1.271e+08 ± 15% -34.0% 83868905 ± 8% stress-ng.time.voluntary_context_switches
BTW, I thought this is a use case for short task wakeup placement. Waking
up the short task on current CPU when the system is overloaded might mitigate
this issue.
===============================================================================
unixbench (throughput, higher is better)
==============================================================================
case nr_instance baseline(std%) compare%( std%)
spawn 125% 1.00 (<2%) +8.1 (<2%)
context1 100% 1.00 (6%) +17.4 (6%)
context1 75% 1.00 (13%) +18.8 (8%)
We tested spawn, pipe, context1 and execl cases. Most cases did not show <= 3%
difference in terms of throughput. And context1 case show significant improvement
when the system is near overloaded.
=================================================================================
netperf (throughput, higher is better)
===========================================================================
case nr_instance baseline(std%) compare%( std%)
UDP_RR 25% 1.00 (<2%) -1.5% (<2%)
UDP_RR 50% 1.00 (<2%) -0.3% (<2%)
UDP_RR 75% 1.00 (<2%) +12.5% (<2%)
UDP_RR 100% 1.00 (<2%) -4.3% (<2%)
UDP_RR 125% 1.00 (<2%) -4.9% (<2%)
UDP_RR 150% 1.00 (<2%) -4.7% (<2%)
UDP_RR 175% 1.00 (<2%) -6.1% (<2%)
UDP_RR 200% 1.00 (<2%) -6.6% (<2%)
TCP_RR 25% 1.00 (<2%) -1.4% (<2%)
TCP_RR 50% 1.00 (<2%) -0.2% (<2%)
TCP_RR 75% 1.00 (<2%) -3.9% (<2%)
TCP_RR 100% 1.00 (2%) +3.6% (5%)
TCP_RR 125% 1.00 (<2%) -4.2% (<2%)
TCP_RR 150% 1.00 (<2%) -6.0% (<2%)
TCP_RR 175% 1.00 (<2%) -7.4% (<2%)
TCP_RR 200% 1.00 (<2%) -8.4% (<2%)
It seems that there is no much impact on netperf, except for 75% case.
==========================================================================
thanks,
Chenyu
next prev parent reply other threads:[~2023-03-23 12:23 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-03-20 7:46 [peterz-queue:sched/eevdf] [sched/fair] 23669fce72: aim7.jobs-per-min -18.6% regression kernel test robot
2023-03-20 7:58 ` Peter Zijlstra
2023-03-21 7:46 ` Oliver Sang
2023-03-21 8:04 ` Chen Yu
2023-03-21 9:03 ` Peter Zijlstra
2023-03-23 12:23 ` Chen Yu [this message]
2023-03-23 15:30 ` Peter Zijlstra
2023-03-26 11:00 ` Peter Zijlstra
2023-03-26 13:38 ` Peter Zijlstra
2023-03-27 13:39 ` Chen Yu
2023-03-27 15:18 ` Peter Zijlstra
2023-03-27 13:51 ` Chen Yu
2023-03-27 15:30 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=ZBxEuboumEifedjG@chenyu5-mobl1 \
--to=yu.c.chen@intel.com \
--cc=lkp@intel.com \
--cc=oe-lkp@lists.linux.dev \
--cc=oliver.sang@intel.com \
--cc=peterz@infradead.org \
--cc=yu.chen.surf@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.