Re: schedulers and topology exposing questions

[George Dunlap <george.dunlap@citrix.com>]

On 27/01/16 14:33, Konrad Rzeszutek Wilk wrote:
> On Tue, Jan 26, 2016 at 11:21:36AM +0000, George Dunlap wrote:
>> On 22/01/16 16:54, Elena Ufimtseva wrote:
>>> Hello all!
>>>
>>> Dario, Gerorge or anyone else,  your help will be appreciated.
>>>
>>> Let me put some intro to our findings. I may forget something or put something
>>> not too explicit, please ask me.
>>>
>>> Customer filled a bug where some of the applications were running slow in their HVM DomU setups.
>>> These running times were compared against baremetal running same kernel version as HVM DomU.
>>>
>>> After some investigation by different parties, the test case scenario was found
>>> where the problem was easily seen. The test app is a udp server/client pair where
>>> client passes some message n number of times.
>>> The test case was executed on baremetal and Xen DomU with kernel version 2.6.39.
>>> Bare metal showed 2x times better result that DomU.
>>>
>>> Konrad came up with a workaround that was setting the flag for domain scheduler in linux
>>> As the guest is not aware of SMT-related topology, it has a flat topology initialized.
>>> Kernel has domain scheduler flags for scheduling domain CPU set to 4143 for 2.6.39.
>>> Konrad discovered that changing the flag for CPU sched domain to 4655
>>> works as a workaround and makes Linux think that the topology has SMT threads.
>>> This workaround makes the test to complete almost in same time as on baremetal (or insignificantly worse).
>>>
>>> This workaround is not suitable for kernels of higher versions as we discovered.
>>>
>>> The hackish way of making domU linux think that it has SMT threads (along with matching cpuid)
>>> made us thinks that the problem comes from the fact that cpu topology is not exposed to
>>> guest and Linux scheduler cannot make intelligent decision on scheduling.
>>>
>>> Joao Martins from Oracle developed set of patches that fixed the smt/core/cashe
>>> topology numbering and provided matching pinning of vcpus and enabling options,
>>> allows to expose to guest correct topology.
>>> I guess Joao will be posting it at some point.
>>>
>>> With this patches we decided to test the performance impact on different kernel versionand Xen versions.
>>>
>>> The test described above was labeled as IO-bound test.
>>
>> So just to clarify: The client sends a request (presumably not much more
>> than a ping) to the server, and waits for the server to respond before
>> sending another one; and the server does the reverse -- receives a
>> request, responds, and then waits for the next request.  Is that right?
> 
> Yes.
>>
>> How much data is transferred?
> 
> 1 packet, UDP
>>
>> If the amount of data transferred is tiny, then the bottleneck for the
>> test is probably the IPI time, and I'd call this a "ping-pong"
>> benchmark[1].  I would only call this "io-bound" if you're actually
>> copying large amounts of data.
> 
> What we found is that on baremetal the scheduler would put both apps
> on the same CPU and schedule them right after each other. This would
> have a high IPI as the scheduler would poke itself.
> On Xen it would put the two applications on seperate CPUs - and there
> would be hardly any IPI.

Sorry -- why would the scheduler send itself an IPI if it's on the same
logical cpu (which seems pretty pointless), but *not* send an IPI to the
*other* processor when it was actually waking up another task?

Or do you mean high context switch rate?

> Digging deeper in the code I found out that if you do an UDP sendmsg
> without any timeouts - it would put it in a queue and just call schedule.

You mean, it would mark the other process as runnable somehow, but not
actually send an IPI to wake it up?  Is that a new "feature" designed
for large systems, to reduce the IPI traffic or something?

> On baremetal the schedule would result in scheduler picking up the other
> task, and starting it - which would dequeue immediately.
> 
> On Xen - the schedule() would go HLT.. and then later be woken up by the
> VIRQ_TIMER. And since the two applications were on seperate CPUs - the
> single packet would just stick in the queue until the VIRQ_TIMER arrived.

I'm not sure I understand the situation right, but it sounds a bit like
what you're seeing is just a quirk of the fact that Linux doesn't always
send IPIs to wake other processes up (either by design or by accident),
but relies on scheduling timers to check for work to do.  Presumably
they knew that low performance on ping-pong workloads might be a
possibility when they wrote the code that way; I don't see a reason why
we should try to work around that in Xen.

 -George