Re: arm64: Approach for DT based NUMA and issues

[Julien Grall <julien.grall@arm.com>]

Hi Dario,

On 27/11/2016 01:01, Dario Faggioli wrote:
> On Sat, 2016-11-26 at 12:29 +0530, Vijay Kilari wrote:
>> 4) Generating memory nodes for DOM0
>> ---------------------------------------------------------
>> Linux kernel device drivers that uses devm_zalloc(), tries to
>> allocate memory
>> from local memory node. So Dom0 needs to have memory allocated on all
>> the
>> available nodes of the system.
>>
> So, first of all, I know next to nothing about ARM, NUMA or non-NUMA.
>
> That being said, providing a guest with a NUMA layout is what we call
> "virtual NUMA" (vNUMA). It is implemented (although it still has some
> issues) for x86 already (so you can have a look), but only for DomU.
>
> I agree that we need to support vNUMA for Dom0 sooner rather than
> later, and I agree that Dom0 is a bit special, so some tricks may be
> necessary. But until we don't implement vNUMA for Dom0, Dom0 is just a
> non-NUMA virtual machine, and the kernel running inside that should
> just behave like it behaves on a non-NUMA box.
>
> Again, I don't know much about ARM, but I think that, until we don't
> have vNUMA for Dom0, that devm_zalloc() thing will just see 1 and only
> 1 NUMA node from which to allocate memory.

I would rather divide the NUMA work for ARM in 2 distinct tasks:
	- Make Xen NUMA-aware
	- Make DOM0 NUMA-aware

Vijay, if I understood correctly what Dario said, on x86 DOM0 is not yet 
NUMA-aware. Aside performance improvement, will there be any technical 
problem (such as PCI devices will not work) if we don't expose NUMA to 
DOM0 from the beginning?

>
> That being said, and for what it's worth...
>
>> Ex: SMMU driver of device on node 1 tries to allocate memory
>> on node 1.
>>
>> ISSUE:
>>  - Dom0's memory should be split across all the available memory
>> nodes
>>    of the system and memory nodes should be generated accordingly.
>>
> ...This is the default behavior, at least on x86.

Are you speaking about the command line parameter dom0_nodes?

>
>>  - Memory DT node generated by Xen for Dom0 should populate numa-
>> node-id
>>    information.
>>
> Generating DT nodes for Dom0 is exactly what I mean when I say
> "implementing / enabling vNUMA for Dom0" (in this case on ARM).
>
> So, yes, let's do it, but let's discuss how to do it properly (e.g., if
> there's anything that can be common between archs, such as some bits of
> the interface).

I would expect vNUMA for Dom0 to be common between x86 and ARM.

Regards,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel