From: Lin Ming <ming.m.lin@intel.com>
To: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: linux-kernel <linux-kernel@vger.kernel.org>,
"Zhang, Yanmin" <yanmin_zhang@linux.intel.com>,
mingo <mingo@elte.hu>, Gregory Haskins <ghaskins@novell.com>
Subject: Re: oltp ~10% regression with 2.6.27-rc5 on stoakley machine
Date: Thu, 04 Sep 2008 20:42:03 +0800 [thread overview]
Message-ID: <1220532123.12955.3.camel@minggr> (raw)
In-Reply-To: <1220531187.8609.217.camel@twins>
On Thu, 2008-09-04 at 14:26 +0200, Peter Zijlstra wrote:
> On Thu, 2008-09-04 at 20:12 +0800, Lin Ming wrote:
> > > > > Thats bizarre... that just indicates the better clock, which should give
> > > > > better (read fairer) scheduling hurts your workload.
> > > > >
> > > > > Is there anything I can run to see if we can fix the scheduler perhaps?
> > > >
> > > > I observed schedstats of sysbench, there's more
> > > > "nr_failed_migrations_hot"
> > > >
> > > > 2.6.27-rc4: se.nr_failed_migrations_hot 11
> > > > 2.6.27-rc5: se.nr_failed_migrations_hot 95
> > > >
> > > > task migration failed because of task_hot, the system is un-balanced?
> > >
> > > Ah, that makes sense, a more accurate clock could indeed make more tasks
> > > hot.
> > >
> > > Can you try fiddling with: /proc/sys/kernel/sched_migration_cost ?
> >
> > sched_migration_cost regression
> > ---------------------- -------------
> > 50000 ~6%
> > 0 ~8%
> > 500000 (default) ~10%
> > 5000000 ~14%
> > -1 ~19%
>
> at 50000 (~6%), is the predominant difference in schedstats still the
> nr_failed_migrations_hot?
Yes, it's strange that nr_failed_migrations_hot (cost=50000) is larger
than (cost=500000)
sched_migration_cost = 50000
----------------------------------------------------------
se.exec_start : 3475158.689868
se.vruntime : 935215.526989
se.sum_exec_runtime : 14358.431942
se.avg_overlap : 0.000000
se.wait_start : 0.000000
se.sleep_start : 3475158.689868
se.block_start : 0.000000
se.sleep_max : 37.910779
se.block_max : 0.033796
se.exec_max : 0.121028
se.slice_max : 0.000000
se.wait_max : 6.007209
se.wait_sum : 25029.649588
se.wait_count : 1954948
sched_info.bkl_count : 0
se.nr_migrations : 2521
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 147
se.nr_failed_migrations_hot : 123
se.nr_forced_migrations : 0
se.nr_forced2_migrations : 1236
se.nr_wakeups : 1952827
se.nr_wakeups_sync : 1944785
se.nr_wakeups_migrate : 2479
se.nr_wakeups_local : 1930122
se.nr_wakeups_remote : 22705
se.nr_wakeups_affine : 13252
se.nr_wakeups_affine_attempts : 825950
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.007355
avg_per_cpu : 5.695530
nr_switches : 1951994
nr_voluntary_switches : 1949963
nr_involuntary_switches : 2031
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 81
sched_migration_cost = 500000
----------------------------------------------------------
se.exec_start : 3719706.806593
se.vruntime : 1049301.798712
se.sum_exec_runtime : 14636.764384
se.avg_overlap : 0.000121
se.wait_start : 0.000000
se.sleep_start : 3719706.806593
se.block_start : 0.000000
se.sleep_max : 214.989106
se.block_max : 0.036240
se.exec_max : 0.147829
se.slice_max : 0.000000
se.wait_max : 6.475864
se.wait_sum : 24932.530233
se.wait_count : 1922627
sched_info.bkl_count : 0
se.nr_migrations : 2314
se.nr_migrations_cold : 0
se.nr_failed_migrations_affine : 0
se.nr_failed_migrations_running : 71
se.nr_failed_migrations_hot : 76
se.nr_forced_migrations : 0
se.nr_forced2_migrations : 1735
se.nr_wakeups : 1920337
se.nr_wakeups_sync : 1908767
se.nr_wakeups_migrate : 2272
se.nr_wakeups_local : 1898194
se.nr_wakeups_remote : 22143
se.nr_wakeups_affine : 1218
se.nr_wakeups_affine_attempts : 820118
se.nr_wakeups_passive : 0
se.nr_wakeups_idle : 0
avg_atom : 0.007624
avg_per_cpu : 6.325308
nr_switches : 1919596
nr_voluntary_switches : 1917396
nr_involuntary_switches : 2200
se.load.weight : 1024
policy : 0
prio : 120
clock-delta : 69
next prev parent reply other threads:[~2008-09-04 12:42 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-09-04 8:51 oltp ~10% regression with 2.6.27-rc5 on stoakley machine Lin Ming
2008-09-04 9:03 ` Peter Zijlstra
2008-09-04 10:52 ` Lin Ming
2008-09-04 11:06 ` Peter Zijlstra
2008-09-04 12:12 ` Lin Ming
2008-09-04 12:26 ` Peter Zijlstra
2008-09-04 12:42 ` Lin Ming [this message]
2008-09-04 13:50 ` Gregory Haskins
2008-09-04 13:50 ` [PATCH 1/4] revert "sched: sched_cacheflush is now unused" Gregory Haskins
2008-09-04 13:50 ` [PATCH 2/4] Revert "[PATCH] sched: remove cache_hot_time" Gregory Haskins
2008-09-04 13:50 ` [PATCH 3/4] Revert "sched: zap the migration init / cache-hot balancing code" Gregory Haskins
2008-09-04 13:50 ` [PATCH 4/4] sched: make task_hot() once again use sd->cache_hot_time Gregory Haskins
2008-09-04 11:09 ` oltp ~10% regression with 2.6.27-rc5 on stoakley machine Ingo Molnar
2008-09-04 11:30 ` Lin Ming
2008-09-04 11:35 ` Ingo Molnar
2008-09-04 12:19 ` Lin Ming
2008-09-05 1:26 ` Lin Ming
2008-09-20 21:38 ` Peter Zijlstra
2008-09-26 2:00 ` Lin Ming
-- strict thread matches above, loose matches on Subject: below --
2008-09-04 7:06 Lin Ming
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1220532123.12955.3.camel@minggr \
--to=ming.m.lin@intel.com \
--cc=a.p.zijlstra@chello.nl \
--cc=ghaskins@novell.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@elte.hu \
--cc=yanmin_zhang@linux.intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.