Large increase in context switch rate

[Jeremy Fitzhardinge <jeremy@goop.org>]

Hi Ingo,

We have Alex Nixon doing some profiling of Xen kernels, comparing 
current pvops Xen and native with the last "official" Xen kernel 
2.6.18.8-xen.

One obvious difference is that the kernbench context switch rate is way 
up, from about 30k to 110k.  Also, the user time went up from about 375s 
to 390s - and that's comparing pvops native to 2.6.18.8-xen (pvops Xen 
was more or less identical).

I wonder if the user time increase is related to the context switch 
rate, because the actual context switch time itself is accounted to the 
process, or because of secondary things like cache and tlb misses.  Or 
perhaps the new scheduler accounts for things differently?

Anyway, I'm wondering:

    * is the increased context switch rate expected?
    * what tunables are there so we can try and make them have
      comparable context switch rates?

This is an issue because the Xen/pvops kernel is showing a fairly large 
overall performance regression, and the context switches a specifically 
slow compared to the old Xen kernel, and the high switch rate is 
presumably compounding the problem.  It would be nice to have some knobs 
to turn to see what the underlying performance characteristics are.

Thanks,
    J