Re: [PATCH 02/10] numa: introduce MachineClass::forbid_asymmetrical_numa

[Daniel Henrique Barboza <danielhb413@gmail.com>]

On 8/24/20 8:49 PM, David Gibson wrote:
> On Mon, Aug 24, 2020 at 08:45:12AM -0300, Daniel Henrique Barboza wrote:
>>
>>

[...]

>>>> LOPAPR support a somewhat asymmetrical NUMA setup in its current
>>>> form,
>>>
>>> Huh, I didn't even realize that.  What's the mechanism?
>>
>> LOPAPR mentions that a single resource/node can have multiple associativity
>> arrays. The idea is to contemplate the situations where the node has
>> more than one connection with the board.
>>
>> I say "somewhat" because, right after mentioning that, the spec also says that
>> the OS should consider that the distance between two nodes must always be
>> the shortest one of all available arrays. I'll copy/paste the except here
>> (end of section 15.2, "Numa Resource Associativity":
> 
> Ah.  I didn't think that's what "asymmetric NUMA" meant... but come to
> think of it, I'm not very sure about that.


This was a poor attempt of my part to cut PAPR some slack.

TBH, even if current PAPR allows for some form of NUMA asymmetry, I don't think
it's worth implementing at all. It'll be more complexity on top of what I
already added here, and the best case scenario will be the kernel ignoring it
(worst case - kernel blowing it up because we're adding more associativity
arrays in each CPU and so on).



Thanks,


DHB

> 
>> -----
>>
>> The reason that the “ibm,associativity” property may contain multiple associativity
>> lists is that a resource may be multiply connected into the platform. This resource
>> then has a different associativity characteristics relative to its multiple connections.
>> To determine the associativity between any two resources, the OS scans down the two
>> resources associativity lists in all pair wise combinations counting how many domains
>> are the same until the first domain where the two list do not agree. The highest such
>> count is the associativity between the two resources.
>>
>> ----
>>
>>
>> DHB
>>
>>
>>>
>>>> but
>>>> the Linux kernel doesn't support it. The effort to implement it in the current
>>>> spapr machine code, given that Linux wouldn't mind it, is not worth it. This
>>>> is why I chose to invalidate it for pseries.
>>>
>>> Igor,
>>>
>>> It's kind of difficult to answer that question - PAPR doesn't
>>> specifically describe limitations, it's just that the representation
>>> it uses is inherently limited.  Instead of the obvious, simple and
>>> pretty much universal method (used in the generic kernel and qemu) of
>>> having a matrix of distance between all the nodes, it instead
>>> describes the hierarchy of components that give rise to the different
>>> distances.
>>>
>>> So, for each NUMA relevant object (cpu, memory block, host bridge,
>>> etc.) there is a vector of IDs.  Each number in the vector gives one
>>> level of the objects location in the heirarchy.
>>>
>>> So, for example the first number might be the physical chip/socket.
>>> the second one which group of cores & memory interfaces sharing an Ln
>>> cache, the third one the specific core number.  So to work out how far
>>> objects are from each other you essentially look at how long a prefix
>>> of their vector they share, which tells you how far above in the
>>> hierarchy you have to go to reach it.
>>>
>>> There's a bunch of complicating details, but that's the gist of it.
>>>
>>>>>> Perhaps a warning would be better in this case?
>>>>>>
>>>>>> In either case, it sounds like this won't be a common constraint
>>>>>> and I now agree with your original suggestion of doing this in
>>>>>> machine initialization code.
>>>>> Agreed, if it goes to spapr specific machine code I will not object much.
>>>>> (it will burden just spapr maintainers, so it's about convincing
>>>>> David in the end)
>>>>
>>>> I believe he's ok with it given that he suggested it in his first reply.
>>>>
>>>> I'll move this verification to spapr machine_init in the next version.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> DHB
>>>>
>>>>
>>>>>
>>>>
>>>
>>
>