Re: PV-vNUMA issue: topology is misinterpreted by the guest

[Juergen Gross <jgross@suse.com>]

On 07/24/2015 05:14 PM, Juergen Gross wrote:
> On 07/24/2015 04:44 PM, Dario Faggioli wrote:
>> On Fri, 2015-07-24 at 12:28 +0200, Juergen Gross wrote:
>>> On 07/23/2015 04:07 PM, Dario Faggioli wrote:
>>
>>>> FWIW, I was thinking that the kernel were a better place, as Juergen is
>>>> saying, while now I'm more convinced that tools would be more
>>>> appropriate, as Boris is saying.
>>>
>>> I've collected some information from the linux kernel sources as a base
>>> for the discussion:
>>>
>> That's great, thanks for this!
>>
>>> The complete numa information (cpu->node and memory->node relations) is
>>> taken from the acpi tables (srat, slit for "distances").
>>>
>> Ok. And I already have a question (as I lost track of things a bit).
>> What you just said about ACPI tables is certainly true for baremetal and
>> HVM guests, but for PV? At the time I was looking into it, together with
>> Elena, there were Linux patches being produced for the PV case, which
>> makes sense.
>> However, ISTR that both Wei and Elena mentioned recently that those
>> patches have not been upstreamed in Linux yet... Is that the case? Maybe
>> not all, but at least some of them are there? Because if not, I'm not
>> sure I see how a PV guest would even see a vNUMA topology (which it
>> does).
>>
>> Of course, I can go and check, but since you just looked, you may have
>> it fresh and clear already. :-)
>
> I checked "bottom up", so when I found the acpi scan stuff I stopped
> searching how the kernel obtains numa info. During my search I found no
> clue of an pv-numa stuff in the kernel. And a quick "grep -i numa" in
> arch/x86/xen and drivers/xen didn't reveal anything. Same for a complete
> kernel source search for "vnuma".
>
>>
>>> The topology information is obtained via:
>>> - intel:
>>>     + cpuid leaf b with subleafs, leaf 4
>>>     + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available
>>> - amd:
>>>     + cpuid leaf 8000001e, leaf 8000001d, leaf 4
>>>     + msr c001100c
>>>     + cpuid leaf 2 and/or leaf 1 if leaf b and/or 4 isn't available
>>>
>>> The scheduler is aware of:
>>> - smt siblings (from topology)
>>> - last-level-cache siblings (from topology)
>>> - node siblings (from numa information)
>>>
>> Right. So, this confirms what we were guessing: we need to "reconcile"
>> these two sources of information (from the guest point of view).
>>
>> Both the 'in kernel' and 'in toolstack' approach should have all the
>> necessary information to make things match, I think. In fact, in
>> toolstack, we know what the vNUMA topology is (we're parsing and
>> actually putting it in place!). In kernel, we know it as we read it from
>> tables or hypercalls (isn't that so, for PV guest?).
>>
>> In fact, I think that it is the topology, i.e., what comes from MSRs,
>> that needs to adapt, and follow vNUMA, as much as possible. Do we agree
>> on this?
>
> I think we have to be very careful here. I see two possible scenarios:
>
> 1) The vcpus are not pinned 1:1 on physical cpus. The hypervisor will
>     try to schedule the vcpus according to their numa affinity. So they
>     can change pcpus at any time in case of very busy guests. I don't
>     think the linux kernel should treat the cpus differently in this
>     case as it will be in vane regarding the Xen scheduler's activity.
>     So we should use the "null" topology in this case.

Sorry, the topology should reflect the vcpu<->numa-node relations, of
course, but nothing else (so flat topolgy in each numa node).

>
> 2) The vcpus of the guest are all pinned 1:1 to physical cpus. The Xen
>     scheduler can't move vcpus between pcpus, so the linux kernel should
>     see the real topology of the used pcpus in order to optimize for this
>     picture.
>
> This only covers the scheduling aspect, of course.
>
>>
>> IMO, the thing boils down to these:
>>
>>   1) from where (kernel vs. toolstack) is it the most easy and effective
>>      to enact the CPUID fiddling? As in, can we do that in toolstack?
>>      (Andrew was not so sure, and Boris found issues, although Jan seems
>>      to think they're no show stopper.)
>>      I'm quite certain that we can do that from inside the kernel,
>>      although, how early would we need to be doing it? Do we have the
>>      vNUMA info already?
>>
>>   2) when tweaking the values of CPUID and other MSRs, are there other
>>      vNUMA (and topology in general) constraints and requirements we
>>      should take into account? For instance, do we want, for licensing
>>      reasons, all (or most) of the vcpus to be siblings, rather than full
>>      sockets? Etc.
>>       2a) if yes, how and where are these constraints specified?
>>
>> If looking at 1) only, it still looks to me that doing things within the
>> kernel would be the way to go.
>>
>> When looking at 2), OTOH, toolstacks variants start to be more
>> appealing. Especially depending on our answer to 2a). In fact,
>> in case we want to give the user a way to specify this
>> siblings-vs-cores-vs-sockets information, it IMO would be good to deal
>> with that in tools, rather than having to involve Xen or Linux!
>>
>>> It will especially move tasks from one cpu to another first between smt
>>> siblings, second between llc siblings, third between node siblings and
>>> last all cpus.
>>>
>> Yep, this part, I knew.
>>
>> Maybe, there is room for "fixing" this at this level, hooking up inside
>> the scheduler code... but I'm shooting in the dark, without having check
>> whether and how this could be really feasible, should I?
>
> Uuh, I don't think a change of the scheduler on behalf of Xen is really
> appreciated. :-)
>
> I'd rather fiddle with the cpu masks on the different levels to let the
> scheduler do the right thing.
>
>> One thing I don't like about this approach is that it would potentially
>> solve vNUMA and other scheduling anomalies, but...
>>
>>> cpuid instruction is available for user mode as well.
>>>
>> ...it would not do any good for other subsystems, and user level code
>> and apps.
>
> Indeed. I think the optimal solution would be two-fold: give the
> scheduler the information it is needing to react correctly via a
> kernel patch not relying on cpuid values and fiddle with the cpuid
> values from xen tools according to any needs of other subsystems and/or
> user code (e.g. licensing).


Juergen