Re: [PATCH v3] PCI: pci_call_probe: call local_pci_probe() when selected cpu is offline

[Ethan Zhao <haifeng.zhao@linux.intel.com>]

On 7/24/2024 11:09 AM, Hongchen Zhang wrote:
> Hi Ethan,
>
> On 2024/7/24 上午10:47, Ethan Zhao wrote:
>> On 7/24/2024 9:58 AM, Hongchen Zhang wrote:
>>> Hi Ethan,
>>> On 2024/7/22 PM 3:39, Ethan Zhao wrote:
>>>>
>>>> On 6/13/2024 3:42 PM, Hongchen Zhang wrote:
>>>>> Call work_on_cpu(cpu, fn, arg) in pci_call_probe() while the argument
>>>>> @cpu is a offline cpu would cause system stuck forever.
>>>>>
>>>>> This can be happen if a node is online while all its CPUs are
>>>>> offline (We can use "maxcpus=1" without "nr_cpus=1" to reproduce it).
>>>>>
>>>>> So, in the above case, let pci_call_probe() call local_pci_probe()
>>>>> instead of work_on_cpu() when the best selected cpu is offline.
>>>>>
>>>>> Fixes: 69a18b18699b ("PCI: Restrict probe functions to 
>>>>> housekeeping CPUs")
>>>>> Cc: <stable@vger.kernel.org>
>>>>> Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
>>>>> Signed-off-by: Hongchen Zhang <zhanghongchen@loongson.cn>
>>>>> ---
>>>>> v2 -> v3: Modify commit message according to Markus's suggestion
>>>>> v1 -> v2: Add a method to reproduce the problem
>>>>> ---
>>>>>   drivers/pci/pci-driver.c | 2 +-
>>>>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>
>>>>> diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c
>>>>> index af2996d0d17f..32a99828e6a3 100644
>>>>> --- a/drivers/pci/pci-driver.c
>>>>> +++ b/drivers/pci/pci-driver.c
>>>>> @@ -386,7 +386,7 @@ static int pci_call_probe(struct pci_driver 
>>>>> *drv, struct pci_dev *dev,
>>>>>           free_cpumask_var(wq_domain_mask);
>>>>>       }
>>>>> -    if (cpu < nr_cpu_ids)
>>>>
>>>> Why not choose the right cpu to callwork_on_cpu() ? the one that is 
>>>> online. Thanks, Ethan
>>> Yes, let housekeeping_cpumask() return online cpu is a good idea, 
>>> but it may be changed by command line. so the simplest way is to 
>>> call local_pci_probe when the best selected cpu is offline.
>>
>> Hmm..... housekeeping_cpumask() should never return offline CPU, so
>> I guess you didn't hit issue with the CPU isolation, but the following
>> code seems not good.
> The issue is the dev node is online but the best selected cpu is 
> offline, so it seems that there is no better way to directly set the 
> cpu to nr_cpu_ids.

I mean where the bug is ? you should debug more about that.
just add one cpu_online(cpu) check there might suggest there
is bug in the cpu selection stage.


if (node < 0 || node >= MAX_NUMNODES || !node_online(node) ||
	    pci_physfn_is_probed(dev)) {
		cpu = nr_cpu_ids; // <----- if you hit here, then local_pci_probe() should be called.
             
	} else {
		cpumask_var_t wq_domain_mask;

		if (!zalloc_cpumask_var(&wq_domain_mask, GFP_KERNEL)) {
			error = -ENOMEM;
			goto out;
		}
		cpumask_and(wq_domain_mask,
			    housekeeping_cpumask(HK_TYPE_WQ),
			    housekeeping_cpumask(HK_TYPE_DOMAIN));

		cpu = cpumask_any_and(cpumask_of_node(node),
				      wq_domain_mask);
		free_cpumask_var(wq_domain_mask);
                 // <----- if you hit here, then work_on_cpu(cpu, local_pci_probe, &ddi) should be called.
                 // do you mean there one offline cpu is selected ?

	}

	if (cpu < nr_cpu_ids)
		error = work_on_cpu(cpu, local_pci_probe, &ddi);
	else
		error = local_pci_probe(&ddi);


Thanks,
Ethan

>>
>> ...
>>
>> if (node < 0 || node >= MAX_NUMNODES || !node_online(node) ||
>>          pci_physfn_is_probed(dev)) {
>>          cpu = nr_cpu_ids;
>>      } else {
>>
>> ....
>>
>> perhaps you could change the logic there and fix it  ?
>>
>> Thanks
>> Ethan
>>
>>
>>
>>>>
>>>>> +    if ((cpu < nr_cpu_ids) && cpu_online(cpu))
>>>>>           error = work_on_cpu(cpu, local_pci_probe, &ddi);
>>>>>       else
>>>>>           error = local_pci_probe(&ddi);
>>>
>>>
>
>