From: Lin Ming <ming.m.lin@intel.com>
To: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: linux-kernel <linux-kernel@vger.kernel.org>,
"Zhang, Yanmin" <yanmin_zhang@linux.intel.com>,
mingo <mingo@elte.hu>, Gregory Haskins <ghaskins@novell.com>
Subject: Re: oltp ~10% regression with 2.6.27-rc5 on stoakley machine
Date: Thu, 04 Sep 2008 20:42:03 +0800 [thread overview]
Message-ID: <1220532123.12955.3.camel@minggr> (raw)
In-Reply-To: <1220531187.8609.217.camel@twins>
On Thu, 2008-09-04 at 14:26 +0200, Peter Zijlstra wrote:
> On Thu, 2008-09-04 at 20:12 +0800, Lin Ming wrote:
> > > > > Thats bizarre... that just indicates the better clock, which should give
> > > > > better (read fairer) scheduling hurts your workload.
> > > > >
> > > > > Is there anything I can run to see if we can fix the scheduler perhaps?
> > > >
> > > > I observed schedstats of sysbench, there's more
> > > > "nr_failed_migrations_hot"
> > > >
> > > > 2.6.27-rc4: se.nr_failed_migrations_hot 11
> > > > 2.6.27-rc5: se.nr_failed_migrations_hot 95
> > > >
> > > > task migration failed because of task_hot, the system is un-balanced?
> > >
> > > Ah, that makes sense, a more accurate clock could indeed make more tasks
> > > hot.
> > >
> > > Can you try fiddling with: /proc/sys/kernel/sched_migration_cost ?
> >
> > sched_migration_cost regression
> > ---------------------- -------------
> > 50000 ~6%
> > 0 ~8%
> > 500000 (default) ~10%
> > 5000000 ~14%
> > -1 ~19%
>
> at 50000 (~6%), is the predominant difference in schedstats still the
> nr_failed_migrations_hot?
Yes, it's strange that nr_failed_migrations_hot (cost=50000) is larger
than (cost=500000)
sched_migration_cost = 50000
----------------------------------------------------------
se.exec_start : 3475158.689868
se.vruntime : 935215.526989
se.sum_exec_runtime : 14358.431942
se.avg_overlap : 0.000000
se.wait_start : 0.000000
se.sleep_start : 3475158.689868
se.block_start : 0.000000
se.sleep_max : 37.910779
se.block_max : 0.033796
se.exec_max : 0.121028
se.slice_max : 0.000000
se.wait_max : 6.007209
se.wait_sum : 25029.649588
se.wait_count : 1954948
sched_info.bkl_count : 0
se.nr_migrations : 2521
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 147
se.nr_failed_migrations_hot : 123
se.nr_forced_migrations : 0
se.nr_forced2_migrations : 1236
se.nr_wakeups : 1952827
se.nr_wakeups_sync : 1944785
se.nr_wakeups_migrate : 2479
se.nr_wakeups_local : 1930122
se.nr_wakeups_remote : 22705
se.nr_wakeups_affine : 13252
se.nr_wakeups_affine_attempts : 825950
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.007355
avg_per_cpu : 5.695530
nr_switches : 1951994
nr_voluntary_switches : 1949963
nr_involuntary_switches : 2031
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 81
sched_migration_cost = 500000
----------------------------------------------------------
se.exec_start : 3719706.806593
se.vruntime : 1049301.798712
se.sum_exec_runtime : 14636.764384
se.avg_overlap : 0.000121
se.wait_start : 0.000000
se.sleep_start : 3719706.806593
se.block_start : 0.000000
se.sleep_max : 214.989106
se.block_max : 0.036240
se.exec_max : 0.147829
se.slice_max : 0.000000
se.wait_max : 6.475864
se.wait_sum : 24932.530233
se.wait_count : 1922627
sched_info.bkl_count : 0
se.nr_migrations : 2314
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 71
se.nr_failed_migrations_hot : 76
se.nr_forced_migrations : 0
se.nr_forced2_migrations : 1735
se.nr_wakeups : 1920337
se.nr_wakeups_sync : 1908767
se.nr_wakeups_migrate : 2272
se.nr_wakeups_local : 1898194
se.nr_wakeups_remote : 22143
se.nr_wakeups_affine : 1218
se.nr_wakeups_affine_attempts : 820118
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.007624
avg_per_cpu : 6.325308
nr_switches : 1919596
nr_voluntary_switches : 1917396
nr_involuntary_switches : 2200
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 69
next prev parent reply other threads:[~2008-09-04 12:42 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-09-04 8:51 oltp ~10% regression with 2.6.27-rc5 on stoakley machine Lin Ming
2008-09-04 9:03 ` Peter Zijlstra
2008-09-04 10:52 ` Lin Ming
2008-09-04 11:06 ` Peter Zijlstra
2008-09-04 12:12 ` Lin Ming
2008-09-04 12:26 ` Peter Zijlstra
2008-09-04 12:42 ` Lin Ming [this message]
2008-09-04 13:50 ` Gregory Haskins
2008-09-04 13:50 ` [PATCH 1/4] revert "sched: sched_cacheflush is now unused" Gregory Haskins
2008-09-04 13:50 ` [PATCH 2/4] Revert "[PATCH] sched: remove cache_hot_time" Gregory Haskins
2008-09-04 13:50 ` [PATCH 3/4] Revert "sched: zap the migration init / cache-hot balancing code" Gregory Haskins
2008-09-04 13:50 ` [PATCH 4/4] sched: make task_hot() once again use sd->cache_hot_time Gregory Haskins
2008-09-04 11:09 ` oltp ~10% regression with 2.6.27-rc5 on stoakley machine Ingo Molnar
2008-09-04 11:30 ` Lin Ming
2008-09-04 11:35 ` Ingo Molnar
2008-09-04 12:19 ` Lin Ming
2008-09-05 1:26 ` Lin Ming
2008-09-20 21:38 ` Peter Zijlstra
2008-09-26 2:00 ` Lin Ming
-- strict thread matches above, loose matches on Subject: below --
2008-09-04 7:06 Lin Ming
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1220532123.12955.3.camel@minggr \
--to=ming.m.lin@intel.com \
--cc=a.p.zijlstra@chello.nl \
--cc=ghaskins@novell.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@elte.hu \
--cc=yanmin_zhang@linux.intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox