Re: PV-vNUMA issue: topology is misinterpreted by the guest

[George Dunlap <george.dunlap@citrix.com>]

On 07/27/2015 11:54 AM, Juergen Gross wrote:
> On 07/27/2015 12:43 PM, George Dunlap wrote:
>> On Mon, Jul 27, 2015 at 5:35 AM, Juergen Gross <jgross@suse.com> wrote:
>>> On 07/24/2015 06:44 PM, Boris Ostrovsky wrote:
>>>>
>>>> On 07/24/2015 12:39 PM, Juergen Gross wrote:
>>>>>
>>>>>
>>>>>
>>>>> I don't say mangling cpuids can't solve the scheduling problem. It
>>>>> surely can. But it can't solve the scheduling problem without hiding
>>>>> information like number of sockets or cores which might be required
>>>>> for license purposes. If we don't care, fine.
>>>>>
>>>>
>>>> (this is somewhat repeating the email I just sent)
>>>>
>>>> Why can's we construct socket/core info with CPUID (and *possibly* ACPI
>>>> changes) that we present a reasonable (licensing-wise) picture?
>>>>
>>>> Can you suggest an example where it will not work and then maybe we can
>>>> figure something out?
>>>
>>>
>>> Let's assume a software with license based on core count. You have a
>>> system with a 2 8 core processors and hyperthreads enabled, summing up
>>> to 32 logical processors. Your license is valid for up to 16 cores, so
>>> running the software on bare metal on your system is fine.
>>>
>>> Now you are running the software inside a virtual machine with 24 vcpus
>>> in a cpupool with 24 logical cpus limited to 12 cores (6 cores of each
>>> processor). As we have to hide hyperthreading in order to not to have
>>> to pin each vcpu to just a single logical processor, the topology
>>> resulting from this picture will have to present 24 cores. The license
>>> will not cover this hardware.
>>
>> But how does doing a PV topology help this situation?  Because we're
>> telling one thing to the OS (via our PV interface) and another thing
>> to applications (via direct CPUID access)?
> 
> Exactly.
> 
> In my example it would even work to not modify the cpuid information at
> all. The kernel wouldn't try to be extremely clever regarding scheduling
> and the user land would see the cpuid information from the real hardware
> (only the 12 cores it is running on, of course).

Right; so it seems

1. Userspace applications are in the habit of reading CPUID to determine
the topology of the system they're running on

2. Many use the topology information to help themselves make better
scheduling decisions.  Because a vcpu is not typically pinned to a
specific pcpu, we may need to lie here slightly (e.g., not mention
threads) to get the optimal behavior overall.

3. Others use the topology information to implement licensing
restrictions.  Because threads are treated differently to cores, we want
to tell the truth here (i.e., make sure we mention that some of these
are threads) to get the optimal behavior overall.

Numbers #2 and #3 lead to contradictory courses of action; we cannot
optimize for both at the same time.

I think at some level we need to just try to accommodate both -- if the
user doesn't have licensing issues, or prefers performance over
licensing, then present a unified topology in PVH / HVM using CPUID,
ACPI, &c.  I think this should be the default.

If the user has licensing issues, and doesn't mind having wonky or
unreliable topology to its guests, then let the raw CPUID through.  But
it would, in this case, be good to try to give the guest OS scheduler a
hint that it shouldn't really bother trying to read the topology or do
placement as a result, as any decisions will be unreliable.

Or alternately, if the user wants to give up on the "consolidation"
aspect of virtualization, they can pin vcpus to pcpus and then pass in
the actual host topology (hyperthreads and all).

 -George