Re: [PATCH v2 1/8] powerpc/xive: Use cpu_to_node() instead of ibm,chip-id property

[Daniel Henrique Barboza <danielhb@linux.ibm.com>]

On 3/9/21 12:33 PM, Cédric Le Goater wrote:
> On 3/8/21 6:13 PM, Greg Kurz wrote:
>> On Wed, 3 Mar 2021 18:48:50 +0100
>> Cédric Le Goater <clg@kaod.org> wrote:
>>
>>> The 'chip_id' field of the XIVE CPU structure is used to choose a
>>> target for a source located on the same chip when possible. This field
>>> is assigned on the PowerNV platform using the "ibm,chip-id" property
>>> on pSeries under KVM when NUMA nodes are defined but it is undefined
>>
>> This sentence seems to have a syntax problem... like it is missing an
>> 'and' before 'on pSeries'.
> 
> ah yes, or simply a comma.
> 
>>> under PowerVM. The XIVE source structure has a similar field
>>> 'src_chip' which is only assigned on the PowerNV platform.
>>>
>>> cpu_to_node() returns a compatible value on all platforms, 0 being the
>>> default node. It will also give us the opportunity to set the affinity
>>> of a source on pSeries when we can localize them.
>>>
>>
>> IIUC this relies on the fact that the NUMA node id is == to chip id
>> on PowerNV, i.e. xc->chip_id which is passed to OPAL remain stable
>> with this change.
> 
> Linux sets the NUMA node in numa_setup_cpu(). On pseries, the hcall
> H_HOME_NODE_ASSOCIATIVITY returns the node id if I am correct (Daniel
> in Cc:)

That's correct. H_HOME_NODE_ASSOCIATIVITY returns not only the node_id, but
a list with the ibm,associativity domains of the CPU that "proc-no" (processor
identifier) is mapped to inside QEMU.

node_id in this case, considering that we're working with a reference-points
of size 4, is the 4th element of the returned list. The last element is
"procno" itself.


> 
> On PowerNV, Linux uses "ibm,associativity" property of the CPU to find
> the node id. This value is built from the chip id in OPAL, so the
> value returned by cpu_to_node(cpu) and the value of the "ibm,chip-id"
> property are unlikely to be different.
> 
> cpu_to_node(cpu) is used in many places to allocate the structures
> locally to the owning node. XIVE is not an exception (see below in the
> same patch), it is better to be consistent and get the same information
> (node id) using the same routine.
> 
> 
> In Linux, "ibm,chip-id" is only used in low level PowerNV drivers :
> LPC, XSCOM, RNG, VAS, NX. XIVE should be in that list also but skiboot
> unifies the controllers of the system to only expose one the OS. This
> is problematic and should be changed but it's another topic.
> 
> 
>> On the other hand, you have the pSeries case under PowerVM that
>> doesn't xc->chip_id, which isn't passed to any hcall AFAICT.
> 
> yes "ibm,chip-id" is an OPAL concept unfortunately and it has no meaning
> under PAPR. xc->chip_id on pseries (PowerVM) will contains an invalid
> chip id.
> 
> QEMU/KVM exposes "ibm,chip-id" but it's not used. (its value is not
> always correct btw)


If you have a way to reliably reproduce this, let me know and I'll fix it
up in QEMU.



Thanks,


DHB


> 
>> It looks like the chip id is only used for localization purpose in
>> this case, right ?
> 
> Yes and PAPR sources are not localized. So it's not used. MSI sources
> could be if we rewrote the MSI driver.
> 
>> In this case, what about doing this change for pSeries only,
>> somewhere in spapr.c ?
> 
> The IPI code is common to all platforms and all have the same issue.
> I rather not.
> 
> Thanks,
> 
> C.
>   
>>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>>> ---
>>>   arch/powerpc/sysdev/xive/common.c | 7 +------
>>>   1 file changed, 1 insertion(+), 6 deletions(-)
>>>
>>> diff --git a/arch/powerpc/sysdev/xive/common.c b/arch/powerpc/sysdev/xive/common.c
>>> index 595310e056f4..b8e456da28aa 100644
>>> --- a/arch/powerpc/sysdev/xive/common.c
>>> +++ b/arch/powerpc/sysdev/xive/common.c
>>> @@ -1335,16 +1335,11 @@ static int xive_prepare_cpu(unsigned int cpu)
>>>   
>>>   	xc = per_cpu(xive_cpu, cpu);
>>>   	if (!xc) {
>>> -		struct device_node *np;
>>> -
>>>   		xc = kzalloc_node(sizeof(struct xive_cpu),
>>>   				  GFP_KERNEL, cpu_to_node(cpu));
>>>   		if (!xc)
>>>   			return -ENOMEM;
>>> -		np = of_get_cpu_node(cpu, NULL);
>>> -		if (np)
>>> -			xc->chip_id = of_get_ibm_chip_id(np);
>>> -		of_node_put(np);
>>> +		xc->chip_id = cpu_to_node(cpu);
>>>   		xc->hw_ipi = XIVE_BAD_IRQ;
>>>   
>>>   		per_cpu(xive_cpu, cpu) = xc;
>>
>