Re: [PATCH v3 00/10] ARM: tegra: Add PCIe device tree support

[Stephen Warren <swarren@wwwdotorg.org>]

On 08/14/2012 05:51 PM, Stephen Warren wrote:
> On 08/14/2012 04:58 PM, Stephen Warren wrote:
>> On 08/14/2012 03:55 PM, Bjorn Helgaas wrote:
>>> On Tue, Aug 14, 2012 at 12:58 PM, Thierry Reding
>>> <thierry.reding@avionic-design.de> wrote:
>>>> On Tue, Aug 14, 2012 at 01:39:23PM -0600, Stephen Warren wrote:
>>>>> On 08/13/2012 05:18 PM, Bjorn Helgaas wrote:
>>>>>> On Mon, Aug 13, 2012 at 11:47 AM, Stephen Warren <swarren@wwwdotorg.org> wrote:
>>>>> ...
>>>>>>> whereas for a device tree boot:
>>>>>>>
>>>>>>> (same):
>>>>>>>> [    2.112217] pci 0000:01:00.0: reg 10: [io  0x0000-0x00ff]
>>>>>>>> [    2.117635] pci 0000:01:00.0: reg 18: [mem 0x00000000-0x00000fff 64bit pref]
>>>>>>>> [    2.124690] pci 0000:01:00.0: reg 20: [mem 0x00000000-0x00003fff 64bit pref]
>>>>>>>> [    2.131731] pci 0000:01:00.0: reg 30: [mem 0x00000000-0x0001ffff pref]
>>>>>>> ... (request region happens early)
>>>>>>>> [    2.179838] r8169 0000:01:00.0: BAR 0: requesting [io  0x0000-0x00ff]
>>>>>>>> [    2.193312] r8169 0000:01:00.0: BAR 2: requesting [mem 0x00000000-0x00000fff 64bit pref]
>>>>>>>> [    2.201397] r8169 0000:01:00.0: BAR 2: can't reserve [mem 0x00000000-0x00000fff 64bit pref]
>>>>>>>> [    2.209742] r8169 0000:01:00.0: (unregistered net_device): could not request regions
>>>>>>> ... (same, just happens too late)
>>>>>>>> [    2.236818] pci 0000:01:00.0: BAR 6: assigned [mem 0xa0000000-0xa001ffff pref]
>>>>>>>> [    2.244027] pci 0000:01:00.0: BAR 4: assigned [mem 0xa0020000-0xa0023fff 64bit pref]
>>>>>>>> [    2.251794] pci 0000:01:00.0: BAR 2: assigned [mem 0xa0024000-0xa0024fff 64bit pref]
>>>>>>>> [    2.259542] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
>>>>>>>
>>>>>>> I suspect this is all still related to the PCI devices themselves being
>>>>>>> probed much earlier in the overall PCI initialization sequence when the
>>>>>>> PCI controller is probed later in the boot sequence, whereas PCI device
>>>>>>> probe is deferred until the overall PCI initialization sequence is
>>>>>>> complete if the PCI controller is probed very early in the boot sequence.
>>>>>>
>>>>>> I don't know what to apply your patches to (they don't apply cleanly
>>>>>> to v3.6-rc2), so I can't see exactly what you're doing.  But it looks
>>>>>> like you might be calling pci_bus_add_devices() before
>>>>>> pci_bus_assign_resource(), which isn't going to work.
>>>>>
>>>>> Yes, that's exactly what is happening.
>>>>>
>>>>> PCIe initialization starts in arch/arm/mach-tegra/pci.e
>>>>> tegra_pcie_init() which calls arch/arm/kernel/bios32.c
>>>>> pci_common_init(). That function first calls pcibios_init_hw() (in the
>>>>> same file, more about this later) and then loops over PCI buses, calling
>>>>> amongst other things pci_bus_assign_resources() then pci_bus_add_devices().
>>>>>
>>>>> The problem is that ARM's pcibios_init_hw() calls pci_scan_root_bus()
>>>>> (or a host-driver-specific function which that also calls
>>>>> pci_scan_root_bus() in Tegra's case) which in turn calls
>>>>> pci_bus_add_devices() right at the end, before control has returned to
>>>>> pci_common_init() and hence before pci_bus_assign_resources() has been
>>>>> called.
>>>>>
>>>>> If I modify pci_scan_root_bus() and remove the call to
>>>>> pci_bus_add_devices(), everything works as expected.
>>>>>
>>>>> So, I guess the question is: Should ARM's pcibios_init_hw() not be
>>>>> calling pci_scan_root_bus(), or at least presumably the ARM PCI code
>>>>> needs to do things in a slightly different order?
>>>
>>> I think you need to do something like this instead of using pci_scan_root_bus():
>>>
>>>     pci_create_root_bus()
>>>     pci_scan_child_bus()
>>>     pci_bus_assign_resources()
>>>     pci_bus_add_devices()
>>>
>>> This is the effective order used by most of the pci_create_root_bus() callers.
>>
>> That would pretty much duplicate everything in pci_scan_root_bus(). That
>> might cause divergence down the road.
>>
>> Can't we make the call to pci_bus_add_devices() optional in
>> pci_scan_root_bus() somehow; one of:
> 
> Sigh, that turns out not to work correctly; it solves at least this part
> of the problem when booting using device tree, but when booting using a
> board file, it causes the IRQ number passed to the PCIe device to be
> bogus:-(

The reason this fails is because the following happens after
pci_scan_bus_root():

	pci_fixup_irqs(pcibios_swizzle, pcibios_map_irq);

(in arch/arm/kernel/bios32.c:pci_common_init())

So, if I replace the call to pci_scan_root_bus() with the code you wrote
above, then the IRQ numbers aren't assigned into the pci_dev structures
until after they're device_add()ed, and hence probed().

Ideally, I could just add a call to pci_fixup_irqs() right before the
call to pci_bus_add_devices() in the code you wrote above. However, that
won't work, because pci_fixup_irqs() finds all the PCI devices to fix up
by searching the list of devices that pci_bus_add_devices() adds to:-(

I guess it's a pretty basic premise of the current PCI code that all the
PCI scanning happens well before any device drivers are registered,
which in turn means that device_add() doesn't trigger the device's
probe() until much later, after all the fixups and resource assignments
are done?