Re: [RFC PATCH 7/8] powerpc/pseries: Add support for FORM2 associativity

["Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>]

On 6/17/21 1:20 PM, David Gibson wrote:
> On Tue, Jun 15, 2021 at 01:10:27PM +0530, Aneesh Kumar K.V wrote:
>> David Gibson <david@gibson.dropbear.id.au> writes:


....

>>>> PAPR defines "most significant" as below
>>>>
>>>> When the “ibm,architecture-vec-5” property byte 5 bit 0 has the value of one, the “ibm,associativ-
>>>> ity-reference-points” property indicates boundaries between associativity domains presented by the
>>>> “ibm,associativity” property containing “near” and “far” resources. The
>>>> first such boundary in the list represents the 1 based ordinal in the
>>>> associativity lists of the most significant boundary, with subsequent
>>>> entries indicating progressively less significant boundaries
>>>
>>> No... that's not a definition.  Like your draft PAPR uses the term
>>> while entirely failing to define it.  From what I can tell about how
>>> it is used the "most significant" boundary corresponds to what Linux
>>> simply thinks of as the node id.  But intuitively, I'd think of that
>>> as the "least significant" boundary, since that's basically the
>>> smallest granularity at which we care about NUMA distances.
>>>
>>>
>>>> I would interpret it as the boundary where we start defining NUMA
>>>> nodes.
>>>
>>> That isn't any clearer to me.
>>
>> How about calling it least significant boundary then?
> 
> Heck, no.  My whole point here is that the meaning is unclear: my
> first guess at the meaning is different from whoever wrote that text.
> We need to come up with a way of describing it that's clearer.
> 
>> The “ibm,associativity-reference-points” property contains one or more list of numbers
>> (domainID index) that represents the 1 based ordinal in the associativity lists of the
>> least significant boundary, with subsequent entries indicating progressively higher
>> significant boundaries.
>>
>> ex:
>> { primary domainID index, secondary domainID index, tertiary domainID index.. }
>>
>> Linux kernel uses the domainID of the least significant boundary (aka primary domain)
>> as the NUMA node id. Linux kernel computes NUMA distance between two domains by
>> recursively comparing if they belong to the same higher-level domains. For mismatch
>> at every higher level of the resource group, the kernel doubles the NUMA distance between
>> the comparing domains.
>>
>

Any suggestion on how to reword the above section then? We could say
associativity-reference-points is list of domainID index representing 
increasing hierarchy of resource group. I am not sure that explains it 
any better?

....
>>>> For ex: With domainID 0, 4, 5 we could do a 5x5 matrix to represent the
>>>> numa distance. Instead ibm,numa-lookup-index-table allows us to present
>>>> the same in a 3x3 matrix  distance[index0][index1] is the  distance
>>>> between NUMA node 0 and 4 and distance[index0][index2] is the distance
>>>> between NUMA node 0 and 5
>>>
>>> Right, I get the purpose of it, and I realized I misphrashed my
>>> question.  My point is that in a Form2 world, the *only* thing the
>>> associativity array is used for is to deduce its position in
>>> lookup-index-table.  Once you have have that for each resource, you
>>> have everything you need, yes?
>>
>>
>> ibm,associativity is used find the domainID/NUMA node id of the
>> resource.
>>
>> ibm,lookup-index-table is used compute the distance information between
>> NUMA nodes using ibm,numa-distance-table.
> 
> I get that you need to use lookup-index-table to work out how to
> interpret numa-distance-table.  My point is that IIUC once you've done
> the lookup in lookup-index-table once for each associativity array
> value, the number you get out (which just a compacted version of the
> node id) should be all you need ever again.
> 

That is correct. We will continue to use the index to nodeid map during 
DLPAR, if such an operation adds a new numa node. update_numa_distance() 
shows the detail.

-aneesh