Re: [RFC PATCH 00/19] Cleanup and optimise the page allocator V2

Ingo Molnar
2009-03-06T09:39:56Z

* Lin Ming <ming.m.lin@intel.com> wrote:

> Thanks, I have used "perfstat -s" to collect cache misses
> data.
>
> 2.6.29-rc7-tip: tip/perfcounters/core (b5e8acf)
> 2.6.29-rc7-tip-mg2: v2 patches applied to tip/perfcounters/core
>
> I collected 5 times netperf UDP-U-4k data with and without
> mg-v2 patches applied to tip/perfcounters/core on a 4p
> quad-core tigerton machine, as below "value" means UDP-U-4k
> test result.
>
> 2.6.29-rc7-tip
> ---------------
> value cache misses CPU migrations cachemisses/migrations
> 5329.71 391094656 1710 228710
> 5641.59 239552767 2138 112045
> 5580.87 132474745 2172 60992
> 5547.19 86911457 2099 41406
> 5626.38 196751217 2050 95976
>
> 2.6.29-rc7-tip-mg2
> -------------------
> value cache misses CPU migrations cachemisses/migrations
> 4749.80 649929463 1132 574142
> 4327.06 484100170 1252 386661
> 4649.51 374201508 1489 251310
> 5655.82 405511551 1848 219432
> 5571.58 90222256 2159 41788
>
> Lin Ming

Hm, these numbers look really interesting and give us insight
into this workload. The workload is fluctuating but by measuring
3 metrics at once instead of just one we see the following
patterns:

- Less CPU migrations means more cache misses and less
performance.

The lowest-score runs had the lowest CPU migrations count,
coupled with a high amount of cachemisses.

This _probably_ means that in this workload migrations are
desired: the sooner two related tasks migrate to the same CPU
the better. If they stay separate (migration count is low) then
they interact with each other from different CPUs, creating a
lot of cachemisses and reducing performance.

You can reduce the migration barrier of the system by enabling
CONFIG_SCHED_DEBUG=y and setting sched_migration_cost to zero:

echo 0 > /proc/sys/kernel/sched_migration_cost

This will hurt other workloads - but if this improves the
numbers then it proves that what this particular workload wants
is easy migrations.

Now the question is, why does the mg2 patchset reduce the number
of migrations? It might not be an inherent property of the mg2
patches: maybe just unlucky timings push the workload across
sched_migration_cost.

Setting sched_migration_cost to either zero or to a very high
value and repeating the test will eliminate this source of noise
and will tell us about other properties of the mg2 patchset.

There might be other effects i'm missing. For example what kind
of UDP transport is used - localhost networking? That means that
sender and receiver really wants to be coupled strongly and what
controls this workload is whether such a 'pair' of tasks can
properly migrate to the same CPU.

Ingo