Re: [PATCH V8 2/3] PCI: Create device tree node for selected devices

[Lizhi Hou <lizhi.hou@amd.com>]

On 4/25/23 08:02, Rob Herring wrote:
> On Thu, Apr 20, 2023 at 11:05 AM Lizhi Hou <lizhi.hou@amd.com> wrote:
>> On 4/19/23 16:11, Rob Herring wrote:
>>> On Tue, Apr 18, 2023 at 09:19:53PM -0700, Lizhi Hou wrote:
>>>> The PCI endpoint device such as Xilinx Alveo PCI card maps the register
>>>> spaces from multiple hardware peripherals to its PCI BAR. Normally,
>>>> the PCI core discovers devices and BARs using the PCI enumeration process.
>>>> There is no infrastructure to discover the hardware peripherals that are
>>>> present in a PCI device, and which can be accessed through the PCI BARs.
> [...]
>
>>>> diff --git a/drivers/pci/of.c b/drivers/pci/of.c
>>>> index 196834ed44fe..42a5cfac2d34 100644
>>>> --- a/drivers/pci/of.c
>>>> +++ b/drivers/pci/of.c
>>>> @@ -469,6 +469,8 @@ static int of_irq_parse_pci(const struct pci_dev *pdev, struct of_phandle_args *
>>>>               } else {
>>>>                       /* We found a P2P bridge, check if it has a node */
>>>>                       ppnode = pci_device_to_OF_node(ppdev);
>>>> +                    if (ppnode && of_node_check_flag(ppnode, OF_DYNAMIC))
>>>> +                            ppnode = NULL;
>>> Again, different behavior if dynamic. I'm not seeing why you need this
>>> change.
>> This change is required. For dynamic generated node, we do not generate
>> interrupt routing related properties. Thus we need to fallback to use
>> pci_swizzle_interrupt_pin(). Generating interrupt routing related
>> properties might be difficult. I think we can differ it to the future
>> patches. Or just use pci_swizzle_interrupt_pin() which is much simpler.
> I don't think we need to generate anything else in the DT. I think we
> need to break from the loop if (ppnode && of_property_present(ppnode,
> "interrupt-map")) instead.
Sure. I will use 'interrupt-map' instead.
>
>
>>>> +static int of_pci_prop_reg(struct pci_dev *pdev, struct of_changeset *ocs,
>>>> +                       struct device_node *np)
>>>> +{
>>>> +    struct of_pci_addr_pair *reg;
>>>> +    int i = 1, resno, ret = 0;
>>>> +    u32 flags, base_addr;
>>>> +    resource_size_t sz;
>>>> +
>>>> +    reg = kcalloc(PCI_STD_NUM_BARS + 1, sizeof(*reg), GFP_KERNEL);
>>>> +    if (!reg)
>>>> +            return -ENOMEM;
>>>> +
>>>> +    /* configuration space */
>>>> +    of_pci_set_address(pdev, reg[0].phys_addr, 0, 0, 0, true);
>>>> +
>>>> +    base_addr = PCI_BASE_ADDRESS_0;
>>>> +    for (resno = PCI_STD_RESOURCES; resno <= PCI_STD_RESOURCE_END;
>>>> +         resno++, base_addr += 4) {
>>>> +            sz = pci_resource_len(pdev, resno);
>>>> +            if (!sz)
>>>> +                    continue;
>>>> +
>>>> +            ret = of_pci_get_addr_flags(&pdev->resource[resno], &flags);
>>>> +            if (ret)
>>>> +                    continue;
>>>> +
>>>> +            of_pci_set_address(pdev, reg[i].phys_addr, 0, base_addr, flags,
>>>> +                               true);
>>>> +            reg[i].size[0] = FIELD_GET(OF_PCI_SIZE_HI, (u64)sz);
>>>> +            reg[i].size[1] = FIELD_GET(OF_PCI_SIZE_LO, (u64)sz);
>>>> +            i++;
>>>> +    }
>>>> +
>>>> +    ret = of_changeset_add_prop_u32_array(ocs, np, "reg", (u32 *)reg,
>>> I believe this should be 'assigned-addresses' rather than 'reg'. But the
>>> config space entry above does go in 'reg'.
>> Do you mean I need to add 'assigned-addresses' in this patch?
> Yes, but on further thought, I think they can just be omitted. They
> are only needed
> if we need of_pci_address_to_resource() to work.
Got it.
>
>> For 'reg', it needs to have pairs for memory space or I/O space. Here is
>> what I saw in IEEE1275:
>>
>> "In the first such pair, the phys-addr component shall be the
>> Configuration Space address of the
>> beginning of the function's set of configuration registers (i.e. the
>> rrrrrrrr field is zero) and the size component shall
>> be zero. Each additional (phys-addr, size) pair shall specify the
>> address of an addressable region of Memory Space or I/
>> O Space associated with the function. In these pairs, if the "n" bit of
>> phys.hi is 0, reflecting a relocatable address, then
>> phys.mid and phys.lo specify an address relative to the value of the
>> associated base register. In general this value will be
>> zero, specifying an address range corresponding directly to the
>> hardware's. If the "n" bit of phys.hi is 1, reflecting a nonrelocatable
>> address, then phys.mid and phys.hi specify an absolute PCI address."
> I think this is a case where true OpenFirmware and FDT differ
> slightly. In OF, the DT reflects everything the firmware discovered
> and configured. FDT is more just what's static and not discoverable.
> (Though generating nodes here is more OF like.) For example, we don't
> put the bus numbers in the DT as those are dynamic and assigned by the
> OS. The purpose of the BAR registers in reg is to define the BAR size
> (and address only if fixed). We don't need that unless what's
> discoverable is wrong and we want to override it.
Thanks for the comments. I will remove the memory and I/O pairs.
>
>
>>>> diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c
>>>> index 57ddcc59af30..9120ca63a82a 100644
>>>> --- a/drivers/pci/pci-driver.c
>>>> +++ b/drivers/pci/pci-driver.c
>>>> @@ -1634,7 +1634,8 @@ static int pci_dma_configure(struct device *dev)
>>>>       bridge = pci_get_host_bridge_device(to_pci_dev(dev));
>>>>
>>>>       if (IS_ENABLED(CONFIG_OF) && bridge->parent &&
>>>> -        bridge->parent->of_node) {
>>>> +        bridge->parent->of_node &&
>>>> +        !of_node_check_flag(bridge->parent->of_node, OF_DYNAMIC)) {
>>> Again, I don't think changing behavior for dynamic case is right. I
>>> haven't dug into what an ACPI+DT case would look like here. (Hint:
>>> someone that wants this merged can dig into that)
>> I think this is required. Without dynamic node, on pure DT system,
>> has_acpi_companion() will return false. Then "ret" is 0 and the
>> following iommu_device_use_default_domain() might be called.
>>
>> With dynamic node, of_dma_configure() might return error because dma
>> related properties are not generated. Thus, "ret" is none zero and the
>> following iommu_device_use_default_domain() will be skipped.
> Again, dynamic is the wrong thing to key off of. If we need
> properties, then they should be added. However, I think the host
> bridge should have what's needed. If the code needs to handle this
> case, then we need to figure out the right thing to do.

I see. I will remove this change. It is not needed for pure DT case.


Thanks,

Lizhi

>
> Rob