pcie-iproc: broken 2nd (& 3rd?) controller support by c3245a566400 ("PCI: iproc: Request host bridge window resources")

pcie-iproc: broken 2nd (& 3rd?) controller support by c3245a566400 ("PCI: iproc: Request host bridge window resources")

[Rafał Miłecki]

Hi,

I just tried upgrading BCM5301X from 4.4 to 4.9 and noticed I don't see card
connected to the 2nd controller.

[    2.593534] pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
[    2.599786] pci_bus 0000:00: root bus resource [mem 0x08000000-0x0fffffff]
[    2.606663] pcie_iproc_bcma bcma0:7: link: UP
[    2.611316] PCI: bus0: Fast back to back transfers disabled
[    2.616899] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.625395] PCI: bus1: Fast back to back transfers disabled
[    2.631011] pci 0000:00:00.0: BAR 8: assigned [mem 0x08000000-0x080fffff]
[    2.637795] pci 0000:01:00.0: BAR 0: assigned [mem 0x08000000-0x08007fff 64bit]
[    2.645091] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.650042] pci 0000:00:00.0:   bridge window [mem 0x08000000-0x080fffff]

[    2.657199] pcie_iproc_bcma bcma0:8: resource collision: [mem 0x40000000-0x47ffffff] conflicts with PCIe MEM space [mem 0x40000000-0x47ffffff]
[    2.669946] pcie_iproc_bcma bcma0:8: PCIe controller setup failed
[    2.676032] pcie_iproc_bcma: probe of bcma0:8 failed with error -16


This used to work with older kernels because there wasn't any collision check:

[    2.587117] pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
[    2.593378] pci_bus 0000:00: root bus resource [mem 0x08000000-0x0fffffff]
[    2.600256] pcie_iproc_bcma bcma0:7: link: UP
[    2.604888] PCI: bus0: Fast back to back transfers disabled
[    2.610474] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.618973] PCI: bus1: Fast back to back transfers disabled
[    2.624591] pci 0000:00:00.0: BAR 8: assigned [mem 0x08000000-0x080fffff]
[    2.631382] pci 0000:01:00.0: BAR 0: assigned [mem 0x08000000-0x08007fff 64bit]
[    2.638686] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.643633] pci 0000:00:00.0:   bridge window [mem 0x08000000-0x080fffff]

[    2.777118] pcie_iproc_bcma bcma0:8: PCI host bridge to bus 0001:00
[    2.783367] pci_bus 0001:00: root bus resource [mem 0x40000000-0x47ffffff]
[    2.790245] pcie_iproc_bcma bcma0:8: link: UP
[    2.794862] PCI: bus0: Fast back to back transfers disabled
[    2.800452] pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.808946] PCI: bus1: Fast back to back transfers disabled
[    2.814558] pci 0001:00:00.0: BAR 8: assigned [mem 0x40000000-0x400fffff]
[    2.821352] pci 0001:01:00.0: BAR 0: assigned [mem 0x40000000-0x40007fff 64bit]
[    2.828650] pci 0001:00:00.0: PCI bridge to [bus 01]
[    2.833600] pci 0001:00:00.0:   bridge window [mem 0x40000000-0x400fffff]


I guess the check is OK after all and the real problem is iproc driver assigning
the same resource.

Broadcom team: could you take a look at this, please?

-- 
Rafa?

pcie-iproc: broken 2nd (& 3rd?) controller support by c3245a566400 ("PCI: iproc: Request host bridge window resources")

[Ray Jui]

Hi Rafal,

On 3/8/2017 4:56 AM, Rafa? Mi?ecki wrote:
> Hi,
> 
> I just tried upgrading BCM5301X from 4.4 to 4.9 and noticed I don't see
> card
> connected to the 2nd controller.
> 
> [    2.593534] pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
> [    2.599786] pci_bus 0000:00: root bus resource [mem
> 0x08000000-0x0fffffff]
> [    2.606663] pcie_iproc_bcma bcma0:7: link: UP
> [    2.611316] PCI: bus0: Fast back to back transfers disabled
> [    2.616899] pci 0000:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
> [    2.625395] PCI: bus1: Fast back to back transfers disabled
> [    2.631011] pci 0000:00:00.0: BAR 8: assigned [mem
> 0x08000000-0x080fffff]
> [    2.637795] pci 0000:01:00.0: BAR 0: assigned [mem
> 0x08000000-0x08007fff 64bit]
> [    2.645091] pci 0000:00:00.0: PCI bridge to [bus 01]
> [    2.650042] pci 0000:00:00.0:   bridge window [mem
> 0x08000000-0x080fffff]
> 
> [    2.657199] pcie_iproc_bcma bcma0:8: resource collision: [mem
> 0x40000000-0x47ffffff] conflicts with PCIe MEM space [mem
> 0x40000000-0x47ffffff]
> [    2.669946] pcie_iproc_bcma bcma0:8: PCIe controller setup failed
> [    2.676032] pcie_iproc_bcma: probe of bcma0:8 failed with error -16
> 

Would you be able to add some debugging trace to see when the resource
was allocated the first time, and when the resource was allocated the
second time and therefore results in the conflict?

In addition, I'm puzzled why this resource conflict issue for BCMA is
not seen with the first controller with outbound memory in the range of
0x08000000 - 0x0fffffff? Maybe that provides a valuable data point here.

Thanks,

Ray

> 
> This used to work with older kernels because there wasn't any collision
> check:
> 
> [    2.587117] pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
> [    2.593378] pci_bus 0000:00: root bus resource [mem
> 0x08000000-0x0fffffff]
> [    2.600256] pcie_iproc_bcma bcma0:7: link: UP
> [    2.604888] PCI: bus0: Fast back to back transfers disabled
> [    2.610474] pci 0000:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
> [    2.618973] PCI: bus1: Fast back to back transfers disabled
> [    2.624591] pci 0000:00:00.0: BAR 8: assigned [mem
> 0x08000000-0x080fffff]
> [    2.631382] pci 0000:01:00.0: BAR 0: assigned [mem
> 0x08000000-0x08007fff 64bit]
> [    2.638686] pci 0000:00:00.0: PCI bridge to [bus 01]
> [    2.643633] pci 0000:00:00.0:   bridge window [mem
> 0x08000000-0x080fffff]
> 
> [    2.777118] pcie_iproc_bcma bcma0:8: PCI host bridge to bus 0001:00
> [    2.783367] pci_bus 0001:00: root bus resource [mem
> 0x40000000-0x47ffffff]
> [    2.790245] pcie_iproc_bcma bcma0:8: link: UP
> [    2.794862] PCI: bus0: Fast back to back transfers disabled
> [    2.800452] pci 0001:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
> [    2.808946] PCI: bus1: Fast back to back transfers disabled
> [    2.814558] pci 0001:00:00.0: BAR 8: assigned [mem
> 0x40000000-0x400fffff]
> [    2.821352] pci 0001:01:00.0: BAR 0: assigned [mem
> 0x40000000-0x40007fff 64bit]
> [    2.828650] pci 0001:00:00.0: PCI bridge to [bus 01]
> [    2.833600] pci 0001:00:00.0:   bridge window [mem
> 0x40000000-0x400fffff]
> 
> 
> I guess the check is OK after all and the real problem is iproc driver
> assigning
> the same resource.
> 
> Broadcom team: could you take a look at this, please?
> 

pcie-iproc: broken 2nd (& 3rd?) controller support by c3245a566400 ("PCI: iproc: Request host bridge window resources")

[Rafał Miłecki]

On 03/08/2017 06:22 PM, Ray Jui wrote:
> On 3/8/2017 4:56 AM, Rafa? Mi?ecki wrote:
>> I just tried upgrading BCM5301X from 4.4 to 4.9 and noticed I don't see
>> card
>> connected to the 2nd controller.
>>
>> [    2.593534] pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
>> [    2.599786] pci_bus 0000:00: root bus resource [mem
>> 0x08000000-0x0fffffff]
>> [    2.606663] pcie_iproc_bcma bcma0:7: link: UP
>> [    2.611316] PCI: bus0: Fast back to back transfers disabled
>> [    2.616899] pci 0000:00:00.0: bridge configuration invalid ([bus
>> 00-00]), reconfiguring
>> [    2.625395] PCI: bus1: Fast back to back transfers disabled
>> [    2.631011] pci 0000:00:00.0: BAR 8: assigned [mem
>> 0x08000000-0x080fffff]
>> [    2.637795] pci 0000:01:00.0: BAR 0: assigned [mem
>> 0x08000000-0x08007fff 64bit]
>> [    2.645091] pci 0000:00:00.0: PCI bridge to [bus 01]
>> [    2.650042] pci 0000:00:00.0:   bridge window [mem
>> 0x08000000-0x080fffff]
>>
>> [    2.657199] pcie_iproc_bcma bcma0:8: resource collision: [mem
>> 0x40000000-0x47ffffff] conflicts with PCIe MEM space [mem
>> 0x40000000-0x47ffffff]
>> [    2.669946] pcie_iproc_bcma bcma0:8: PCIe controller setup failed
>> [    2.676032] pcie_iproc_bcma: probe of bcma0:8 failed with error -16
>>
>
> Would you be able to add some debugging trace to see when the resource
> was allocated the first time, and when the resource was allocated the
> second time and therefore results in the conflict?
>
> In addition, I'm puzzled why this resource conflict issue for BCMA is
> not seen with the first controller with outbound memory in the range of
> 0x08000000 - 0x0fffffff? Maybe that provides a valuable data point here.

This is clearly a conflict between iproc on 1st controller and iproc on 2nd controller.

I added following code to the pcie-iproc-bcma.c:
if (bdev->core_unit == 0) {
	dev_info(dev, "[%s] aborting\n", __func__);
	return -ENOTSUPP;
}

And it resulted in 2nd controller (bcma0:8) working again:
[    2.472562] pcie_iproc_bcma bcma0:7: [iproc_pcie_bcma_probe] aborting
[    2.479027] pcie_iproc_bcma: probe of bcma0:7 failed with error -524
[    2.607425] pcie_iproc_bcma bcma0:8: PCI host bridge to bus 0000:00
[    2.613678] pci_bus 0000:00: root bus resource [mem 0x40000000-0x47ffffff]
[    2.620557] pcie_iproc_bcma bcma0:8: link: UP
[    2.625206] PCI: bus0: Fast back to back transfers disabled
[    2.630789] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.639260] PCI: bus1: Fast back to back transfers disabled
[    2.644871] pci 0000:00:00.0: BAR 8: assigned [mem 0x40000000-0x400fffff]
[    2.651665] pci 0000:01:00.0: BAR 0: assigned [mem 0x40000000-0x40007fff 64bit]
[    2.658962] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.663911] pci 0000:00:00.0:   bridge window [mem 0x40000000-0x400fffff]

I'm not exactly sure where this conflict is coming from. For debugging purposes I added:
dev_info(dev, "[%s] res_mem:%pR\n", __func__, &res_mem);

Which resulted in:
[    2.474558] pcie_iproc_bcma bcma0:7: [iproc_pcie_bcma_probe] res_mem:[mem 0x08000000-0x0fffffff]
[    2.597246] pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
[    2.603500] pci_bus 0000:00: root bus resource [mem 0x08000000-0x0fffffff]
[    2.610382] pcie_iproc_bcma bcma0:7: link: UP
[    2.615032] PCI: bus0: Fast back to back transfers disabled
[    2.620619] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.629131] PCI: bus1: Fast back to back transfers disabled
[    2.634745] pci 0000:00:00.0: BAR 8: assigned [mem 0x08000000-0x080fffff]
[    2.641537] pci 0000:01:00.0: BAR 0: assigned [mem 0x08000000-0x08007fff 64bit]
[    2.648835] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.653783] pci 0000:00:00.0:   bridge window [mem 0x08000000-0x080fffff]
[    2.660947] pcie_iproc_bcma bcma0:8: [iproc_pcie_bcma_probe] res_mem:[mem 0x40000000-0x47ffffff]
[    2.669732] pcie_iproc_bcma bcma0:8: resource collision: [mem 0x40000000-0x47ffffff] conflicts with PCIe MEM space [mem 0x40000000-0x47ffffff]
[    2.682470] pcie_iproc_bcma bcma0:8: PCIe controller setup failed
[    2.688559] pcie_iproc_bcma: probe of bcma0:8 failed with error -16

There shouldn't be any conflict between 0x08000000-0x0fffffff and 0x40000000-0x47ffffff, right?

For bcma we do this:
res_mem.start = bdev->addr_s[0];
res_mem.end = bdev->addr_s[0] + SZ_128M - 1;

In case of BCM4708 there are following addresses:
[    2.285430] bcma: bus0: Core 7 found: PCIe Gen 2 (manuf 0x4BF, id 0x501, rev 0x01, class 0x0)	addr:0x18012000 wrap:0x18101000 addr_s[0]:0x08000000 addr_s[1]:0x00000000
[    2.300384] bcma: bus0: Core 8 found: PCIe Gen 2 (manuf 0x4BF, id 0x501, rev 0x01, class 0x0)	addr:0x18013000 wrap:0x18102000 addr_s[0]:0x40000000 addr_s[1]:0x00000000
which matches what we see in the boot log.

pcie-iproc: broken 2nd (& 3rd?) controller support by c3245a566400 ("PCI: iproc: Request host bridge window resources")

[Ray Jui]

On 3/8/2017 3:28 PM, Rafa? Mi?ecki wrote:
> On 03/08/2017 06:22 PM, Ray Jui wrote:
>> On 3/8/2017 4:56 AM, Rafa? Mi?ecki wrote:
>>> I just tried upgrading BCM5301X from 4.4 to 4.9 and noticed I don't see
>>> card
>>> connected to the 2nd controller.
>>>
>>> [    2.593534] pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
>>> [    2.599786] pci_bus 0000:00: root bus resource [mem
>>> 0x08000000-0x0fffffff]
>>> [    2.606663] pcie_iproc_bcma bcma0:7: link: UP
>>> [    2.611316] PCI: bus0: Fast back to back transfers disabled
>>> [    2.616899] pci 0000:00:00.0: bridge configuration invalid ([bus
>>> 00-00]), reconfiguring
>>> [    2.625395] PCI: bus1: Fast back to back transfers disabled
>>> [    2.631011] pci 0000:00:00.0: BAR 8: assigned [mem
>>> 0x08000000-0x080fffff]
>>> [    2.637795] pci 0000:01:00.0: BAR 0: assigned [mem
>>> 0x08000000-0x08007fff 64bit]
>>> [    2.645091] pci 0000:00:00.0: PCI bridge to [bus 01]
>>> [    2.650042] pci 0000:00:00.0:   bridge window [mem
>>> 0x08000000-0x080fffff]
>>>
>>> [    2.657199] pcie_iproc_bcma bcma0:8: resource collision: [mem
>>> 0x40000000-0x47ffffff] conflicts with PCIe MEM space [mem
>>> 0x40000000-0x47ffffff]
>>> [    2.669946] pcie_iproc_bcma bcma0:8: PCIe controller setup failed
>>> [    2.676032] pcie_iproc_bcma: probe of bcma0:8 failed with error -16
>>>
>>
>> Would you be able to add some debugging trace to see when the resource
>> was allocated the first time, and when the resource was allocated the
>> second time and therefore results in the conflict?
>>
>> In addition, I'm puzzled why this resource conflict issue for BCMA is
>> not seen with the first controller with outbound memory in the range of
>> 0x08000000 - 0x0fffffff? Maybe that provides a valuable data point here.
> 
> This is clearly a conflict between iproc on 1st controller and iproc on
> 2nd controller.
> 
> I added following code to the pcie-iproc-bcma.c:
> if (bdev->core_unit == 0) {
>     dev_info(dev, "[%s] aborting\n", __func__);
>     return -ENOTSUPP;
> }
> 
> And it resulted in 2nd controller (bcma0:8) working again:

This is what puzzled me. The first controller is using the following
outbound window: 0x8000000 - 0x0fffffff

The 2nd controller is using:
0x40000000 - 0x400fffff

They should not conflict with each other.

> [    2.472562] pcie_iproc_bcma bcma0:7: [iproc_pcie_bcma_probe] aborting
> [    2.479027] pcie_iproc_bcma: probe of bcma0:7 failed with error -524
> [    2.607425] pcie_iproc_bcma bcma0:8: PCI host bridge to bus 0000:00
> [    2.613678] pci_bus 0000:00: root bus resource [mem
> 0x40000000-0x47ffffff]
> [    2.620557] pcie_iproc_bcma bcma0:8: link: UP
> [    2.625206] PCI: bus0: Fast back to back transfers disabled
> [    2.630789] pci 0000:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
> [    2.639260] PCI: bus1: Fast back to back transfers disabled
> [    2.644871] pci 0000:00:00.0: BAR 8: assigned [mem
> 0x40000000-0x400fffff]
> [    2.651665] pci 0000:01:00.0: BAR 0: assigned [mem
> 0x40000000-0x40007fff 64bit]
> [    2.658962] pci 0000:00:00.0: PCI bridge to [bus 01]
> [    2.663911] pci 0000:00:00.0:   bridge window [mem
> 0x40000000-0x400fffff]
> 
> I'm not exactly sure where this conflict is coming from. For debugging
> purposes I added:
> dev_info(dev, "[%s] res_mem:%pR\n", __func__, &res_mem);
> 
> Which resulted in:
> [    2.474558] pcie_iproc_bcma bcma0:7: [iproc_pcie_bcma_probe]
> res_mem:[mem 0x08000000-0x0fffffff]
> [    2.597246] pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
> [    2.603500] pci_bus 0000:00: root bus resource [mem
> 0x08000000-0x0fffffff]
> [    2.610382] pcie_iproc_bcma bcma0:7: link: UP
> [    2.615032] PCI: bus0: Fast back to back transfers disabled
> [    2.620619] pci 0000:00:00.0: bridge configuration invalid ([bus
> 00-00]), reconfiguring
> [    2.629131] PCI: bus1: Fast back to back transfers disabled
> [    2.634745] pci 0000:00:00.0: BAR 8: assigned [mem
> 0x08000000-0x080fffff]
> [    2.641537] pci 0000:01:00.0: BAR 0: assigned [mem
> 0x08000000-0x08007fff 64bit]
> [    2.648835] pci 0000:00:00.0: PCI bridge to [bus 01]
> [    2.653783] pci 0000:00:00.0:   bridge window [mem
> 0x08000000-0x080fffff]
> [    2.660947] pcie_iproc_bcma bcma0:8: [iproc_pcie_bcma_probe]
> res_mem:[mem 0x40000000-0x47ffffff]
> [    2.669732] pcie_iproc_bcma bcma0:8: resource collision: [mem
> 0x40000000-0x47ffffff] conflicts with PCIe MEM space [mem
> 0x40000000-0x47ffffff]

Someone must have reserved 0x40000000-0x47ffffff silently somewhere, as
this is what the kernel complains about.

Btw, this code is from devm_request_resource in kernel/resource.c

> [    2.682470] pcie_iproc_bcma bcma0:8: PCIe controller setup failed
> [    2.688559] pcie_iproc_bcma: probe of bcma0:8 failed with error -16
> 
> There shouldn't be any conflict between 0x08000000-0x0fffffff and
> 0x40000000-0x47ffffff, right?

Correct. You need to figure out when 0x40000000-0x47ffffff was reserved
previously.

> 
> For bcma we do this:
> res_mem.start = bdev->addr_s[0];
> res_mem.end = bdev->addr_s[0] + SZ_128M - 1;
> 
> In case of BCM4708 there are following addresses:
> [    2.285430] bcma: bus0: Core 7 found: PCIe Gen 2 (manuf 0x4BF, id
> 0x501, rev 0x01, class 0x0)    addr:0x18012000 wrap:0x18101000
> addr_s[0]:0x08000000 addr_s[1]:0x00000000
> [    2.300384] bcma: bus0: Core 8 found: PCIe Gen 2 (manuf 0x4BF, id
> 0x501, rev 0x01, class 0x0)    addr:0x18013000 wrap:0x18102000
> addr_s[0]:0x40000000 addr_s[1]:0x00000000
> which matches what we see in the boot log.

pcie-iproc: broken 2nd (& 3rd?) controller support by c3245a566400 ("PCI: iproc: Request host bridge window resources")

[Rafał Miłecki]

On 03/08/2017 01:56 PM, Rafa? Mi?ecki wrote:
> I just tried upgrading BCM5301X from 4.4 to 4.9 and noticed I don't see card
> connected to the 2nd controller.
>
> [    2.593534] pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
> [    2.599786] pci_bus 0000:00: root bus resource [mem 0x08000000-0x0fffffff]
> [    2.606663] pcie_iproc_bcma bcma0:7: link: UP
> [    2.611316] PCI: bus0: Fast back to back transfers disabled
> [    2.616899] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> [    2.625395] PCI: bus1: Fast back to back transfers disabled
> [    2.631011] pci 0000:00:00.0: BAR 8: assigned [mem 0x08000000-0x080fffff]
> [    2.637795] pci 0000:01:00.0: BAR 0: assigned [mem 0x08000000-0x08007fff 64bit]
> [    2.645091] pci 0000:00:00.0: PCI bridge to [bus 01]
> [    2.650042] pci 0000:00:00.0:   bridge window [mem 0x08000000-0x080fffff]
>
> [    2.657199] pcie_iproc_bcma bcma0:8: resource collision: [mem 0x40000000-0x47ffffff] conflicts with PCIe MEM space [mem 0x40000000-0x47ffffff]
> [    2.669946] pcie_iproc_bcma bcma0:8: PCIe controller setup failed
> [    2.676032] pcie_iproc_bcma: probe of bcma0:8 failed with error -16
>
>
> This used to work with older kernels because there wasn't any collision check:
>
> [    2.587117] pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
> [    2.593378] pci_bus 0000:00: root bus resource [mem 0x08000000-0x0fffffff]
> [    2.600256] pcie_iproc_bcma bcma0:7: link: UP
> [    2.604888] PCI: bus0: Fast back to back transfers disabled
> [    2.610474] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> [    2.618973] PCI: bus1: Fast back to back transfers disabled
> [    2.624591] pci 0000:00:00.0: BAR 8: assigned [mem 0x08000000-0x080fffff]
> [    2.631382] pci 0000:01:00.0: BAR 0: assigned [mem 0x08000000-0x08007fff 64bit]
> [    2.638686] pci 0000:00:00.0: PCI bridge to [bus 01]
> [    2.643633] pci 0000:00:00.0:   bridge window [mem 0x08000000-0x080fffff]
>
> [    2.777118] pcie_iproc_bcma bcma0:8: PCI host bridge to bus 0001:00
> [    2.783367] pci_bus 0001:00: root bus resource [mem 0x40000000-0x47ffffff]
> [    2.790245] pcie_iproc_bcma bcma0:8: link: UP
> [    2.794862] PCI: bus0: Fast back to back transfers disabled
> [    2.800452] pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> [    2.808946] PCI: bus1: Fast back to back transfers disabled
> [    2.814558] pci 0001:00:00.0: BAR 8: assigned [mem 0x40000000-0x400fffff]
> [    2.821352] pci 0001:01:00.0: BAR 0: assigned [mem 0x40000000-0x40007fff 64bit]
> [    2.828650] pci 0001:00:00.0: PCI bridge to [bus 01]
> [    2.833600] pci 0001:00:00.0:   bridge window [mem 0x40000000-0x400fffff]
>
>
> I guess the check is OK after all and the real problem is iproc driver assigning
> the same resource.
>
> Broadcom team: could you take a look at this, please?

I found a reason of this conflict (and probably random crashes I started
seeing with 4.9). I believe we have a memory corruption.

So that commit c3245a566400 ("PCI: iproc: Request host bridge window
resources") adds call to the devm_request_pci_bus_resources passing "res"
pointer. The problem is "res" points to the *local* variable of
iproc_pcie_bcma_probe function.
As soon as the iproc_pcie_bcma_probe exits that resource variable is not
accessible anymore, yet it's used as a child for &iomem_resource. Things go
even worse on another iproc_pcie_bcma_probe call. Its local variable res_mem
gets the same memory address as the one already used for &iomem_resource. We
modify local variable modifying already-added resource at the same time!

So this is where this whole conflicts comes from. What is stored as resource
for [mem 0x08000000-0x0fffffff] range gets modified as "local" variable to the
[mem 0x40000000-0x47ffffff] and then we try to re-request the same resource.

I'm pasting log & patch that allowed me to debug & notice this problem so you
can confirm my observations.

[    2.615055] resource: [__request_resource] root:[mem 0x00000000-0xffffffff]
[    2.621995] resource: [__request_resource]  new:c7839be0 new:[mem 0x08000000-0x0fffffff]
[    2.630060] resource: [__request_resource]  tmp:c7ffee00 tmp:[mem 0x00000000-0x07ffffff]
[    2.638122] resource: [__request_resource]  tmp:c7adf180 tmp:[mem 0x18000300-0x180003ff]
[    2.764941] pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
[    2.771194] pci_bus 0000:00: root bus resource [mem 0x08000000-0x0fffffff]
[    2.778066] pcie_iproc_bcma bcma0:7: link: UP
[    2.782693] PCI: bus0: Fast back to back transfers disabled
[    2.788267] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    2.796767] PCI: bus1: Fast back to back transfers disabled
[    2.802375] resource: [__request_resource] root:[mem 0x08000000-0x0fffffff]
[    2.809324] resource: [__request_resource]  new:c7b08e64 new:[mem size 0x00100000]
[    2.816865] resource: [__request_resource]  tmp:  (null) tmp:  (null)
[    2.823282] pci 0000:00:00.0: BAR 8: assigned [mem 0x08000000-0x080fffff]
[    2.830049] resource: [__request_resource] root:[mem 0x08000000-0x080fffff]
[    2.836986] resource: [__request_resource]  new:c7b09164 new:[mem size 0x00008000 64bit]
[    2.845047] resource: [__request_resource]  tmp:  (null) tmp:  (null)
[    2.851464] pci 0000:01:00.0: BAR 0: assigned [mem 0x08000000-0x08007fff 64bit]
[    2.858758] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.863700] pci 0000:00:00.0:   bridge window [mem 0x08000000-0x080fffff]

[    2.870834] resource: [__request_resource] root:[mem 0x00000000-0xffffffff]
[    2.877791] resource: [__request_resource]  new:c7839be0 new:[mem 0x40000000-0x47ffffff]
[    2.885855] resource: [__request_resource]  tmp:c7ffee00 tmp:[mem 0x00000000-0x07ffffff]
[    2.893911] resource: [__request_resource]  tmp:c7839be0 tmp:[mem 0x40000000-0x47ffffff]
[    2.901971] resource: [__request_resource] Found collision with tmp
[    2.908225] pcie_iproc_bcma bcma0:8: resource collision: [mem 0x40000000-0x47ffffff] conflicts with PCIe MEM space [mem 0x40000000-0x47ffffff]
[    2.920952] pcie_iproc_bcma bcma0:8: PCIe controller setup failed
[    2.927037] pcie_iproc_bcma: probe of bcma0:8 failed with error -16

As you can see, for the first controller following resource has been requested:
c7839be0 [mem 0x08000000-0x0fffffff]

Then when we try to probe second controller there are following resources
already in use:
c7ffee00 [mem 0x00000000-0x07ffffff]
c7839be0 [mem 0x40000000-0x47ffffff]
and we try to request:
c7839be0 [mem 0x40000000-0x47ffffff]


diff --git a/kernel/resource.c b/kernel/resource.c
index 9b5f044..fab9405 100644
--- a/kernel/resource.c
+++ b/kernel/resource.c
@@ -220,6 +220,8 @@ static struct resource * __request_resource(struct resource *root, struct resour
  	resource_size_t end = new->end;
  	struct resource *tmp, **p;

+	if (new->flags & IORESOURCE_MEM) pr_info("[%s] root:%pR\n", __func__, root);
+	if (new->flags & IORESOURCE_MEM) pr_info("[%s]  new:%p new:%pR\n", __func__, new, new);
  	if (end < start)
  		return root;
  	if (start < root->start)
@@ -229,6 +231,7 @@ static struct resource * __request_resource(struct resource *root, struct resour
  	p = &root->child;
  	for (;;) {
  		tmp = *p;
+		if (new->flags & IORESOURCE_MEM) pr_info("[%s]  tmp:%p tmp:%pR\n", __func__, tmp, tmp);
  		if (!tmp || tmp->start > end) {
  			new->sibling = tmp;
  			*p = new;
@@ -238,6 +241,7 @@ static struct resource * __request_resource(struct resource *root, struct resour
  		p = &tmp->sibling;
  		if (tmp->end < start)
  			continue;
+		if (new->flags & IORESOURCE_MEM) pr_info("[%s] Found collision with tmp\n", __func__);
  		return tmp;
  	}
  }

pcie-iproc: broken 2nd (& 3rd?) controller support by c3245a566400 ("PCI: iproc: Request host bridge window resources")

[Bjorn Helgaas]

On Thu, Mar 09, 2017 at 08:39:07AM +0100, Rafa? Mi?ecki wrote:
> On 03/08/2017 01:56 PM, Rafa? Mi?ecki wrote:
> >I just tried upgrading BCM5301X from 4.4 to 4.9 and noticed I don't see card
> >connected to the 2nd controller.
> >
> > pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
> > pci_bus 0000:00: root bus resource [mem 0x08000000-0x0fffffff]
> > pcie_iproc_bcma bcma0:7: link: UP
> > PCI: bus0: Fast back to back transfers disabled
> > pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> > PCI: bus1: Fast back to back transfers disabled
> > pci 0000:00:00.0: BAR 8: assigned [mem 0x08000000-0x080fffff]
> > pci 0000:01:00.0: BAR 0: assigned [mem 0x08000000-0x08007fff 64bit]
> > pci 0000:00:00.0: PCI bridge to [bus 01]
> > pci 0000:00:00.0:   bridge window [mem 0x08000000-0x080fffff]
> >
> > pcie_iproc_bcma bcma0:8: resource collision: [mem 0x40000000-0x47ffffff] conflicts with PCIe MEM space [mem 0x40000000-0x47ffffff]
> > pcie_iproc_bcma bcma0:8: PCIe controller setup failed
> > pcie_iproc_bcma: probe of bcma0:8 failed with error -16
> >
> >
> >This used to work with older kernels because there wasn't any collision check:
> >
> > pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
> > pci_bus 0000:00: root bus resource [mem 0x08000000-0x0fffffff]
> > pcie_iproc_bcma bcma0:7: link: UP
> > PCI: bus0: Fast back to back transfers disabled
> > pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> > PCI: bus1: Fast back to back transfers disabled
> > pci 0000:00:00.0: BAR 8: assigned [mem 0x08000000-0x080fffff]
> > pci 0000:01:00.0: BAR 0: assigned [mem 0x08000000-0x08007fff 64bit]
> > pci 0000:00:00.0: PCI bridge to [bus 01]
> > pci 0000:00:00.0:   bridge window [mem 0x08000000-0x080fffff]
> >
> > pcie_iproc_bcma bcma0:8: PCI host bridge to bus 0001:00
> > pci_bus 0001:00: root bus resource [mem 0x40000000-0x47ffffff]
> > pcie_iproc_bcma bcma0:8: link: UP
> > PCI: bus0: Fast back to back transfers disabled
> > pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> > PCI: bus1: Fast back to back transfers disabled
> > pci 0001:00:00.0: BAR 8: assigned [mem 0x40000000-0x400fffff]
> > pci 0001:01:00.0: BAR 0: assigned [mem 0x40000000-0x40007fff 64bit]
> > pci 0001:00:00.0: PCI bridge to [bus 01]
> > pci 0001:00:00.0:   bridge window [mem 0x40000000-0x400fffff]
> >
> >
> >I guess the check is OK after all and the real problem is iproc driver assigning
> >the same resource.
> >
> >Broadcom team: could you take a look at this, please?
> 
> I found a reason of this conflict (and probably random crashes I started
> seeing with 4.9). I believe we have a memory corruption.

Yep, we're using a resource structure on the stack when we shouldn't.  Can
you try the patch below?

diff --git a/drivers/pci/host/pcie-iproc-bcma.c b/drivers/pci/host/pcie-iproc-bcma.c
index bd4c9ec25edc..8cebbbff1e72 100644
--- a/drivers/pci/host/pcie-iproc-bcma.c
+++ b/drivers/pci/host/pcie-iproc-bcma.c
@@ -44,8 +44,7 @@ static int iproc_pcie_bcma_probe(struct bcma_device *bdev)
 {
 	struct device *dev = &bdev->dev;
 	struct iproc_pcie *pcie;
-	LIST_HEAD(res);
-	struct resource res_mem;
+	LIST_HEAD(resources);
 	int ret;
 
 	pcie = devm_kzalloc(dev, sizeof(*pcie), GFP_KERNEL);
@@ -63,22 +62,23 @@ static int iproc_pcie_bcma_probe(struct bcma_device *bdev)
 
 	pcie->base_addr = bdev->addr;
 
-	res_mem.start = bdev->addr_s[0];
-	res_mem.end = bdev->addr_s[0] + SZ_128M - 1;
-	res_mem.name = "PCIe MEM space";
-	res_mem.flags = IORESOURCE_MEM;
-	pci_add_resource(&res, &res_mem);
+	pcie->mem.start = bdev->addr_s[0];
+	pcie->mem.end = bdev->addr_s[0] + SZ_128M - 1;
+	pcie->mem.name = "PCIe MEM space";
+	pcie->mem.flags = IORESOURCE_MEM;
+	pci_add_resource(&resources, &pcie->mem);
 
 	pcie->map_irq = iproc_pcie_bcma_map_irq;
 
-	ret = iproc_pcie_setup(pcie, &res);
+	ret = iproc_pcie_setup(pcie, &resources);
 	if (ret)
 		dev_err(dev, "PCIe controller setup failed\n");
-
-	pci_free_resource_list(&res);
+		pci_free_resource_list(&resources);
+		return ret;
+	}
 
 	bcma_set_drvdata(bdev, pcie);
-	return ret;
+	return 0;
 }
 
 static void iproc_pcie_bcma_remove(struct bcma_device *bdev)
diff --git a/drivers/pci/host/pcie-iproc-platform.c b/drivers/pci/host/pcie-iproc-platform.c
index f4909bb0b2ad..5f6361f27c69 100644
--- a/drivers/pci/host/pcie-iproc-platform.c
+++ b/drivers/pci/host/pcie-iproc-platform.c
@@ -51,7 +51,7 @@ static int iproc_pcie_pltfm_probe(struct platform_device *pdev)
 	struct device_node *np = dev->of_node;
 	struct resource reg;
 	resource_size_t iobase = 0;
-	LIST_HEAD(res);
+	LIST_HEAD(resources);
 	int ret;
 
 	pcie = devm_kzalloc(dev, sizeof(*pcie), GFP_KERNEL);
@@ -96,10 +96,10 @@ static int iproc_pcie_pltfm_probe(struct platform_device *pdev)
 		pcie->phy = NULL;
 	}
 
-	ret = of_pci_get_host_bridge_resources(np, 0, 0xff, &res, &iobase);
+	ret = of_pci_get_host_bridge_resources(np, 0, 0xff, &resources,
+					       &iobase);
 	if (ret) {
-		dev_err(dev,
-			"unable to get PCI host bridge resources\n");
+		dev_err(dev, "unable to get PCI host bridge resources\n");
 		return ret;
 	}
 
@@ -112,14 +112,15 @@ static int iproc_pcie_pltfm_probe(struct platform_device *pdev)
 		pcie->map_irq = of_irq_parse_and_map_pci;
 	}
 
-	ret = iproc_pcie_setup(pcie, &res);
+	ret = iproc_pcie_setup(pcie, &resources);
 	if (ret)
 		dev_err(dev, "PCIe controller setup failed\n");
-
-	pci_free_resource_list(&res);
+		pci_free_resource_list(&resources);
+		return ret;
+	}
 
 	platform_set_drvdata(pdev, pcie);
-	return ret;
+	return 0;
 }
 
 static int iproc_pcie_pltfm_remove(struct platform_device *pdev)
diff --git a/drivers/pci/host/pcie-iproc.h b/drivers/pci/host/pcie-iproc.h
index 04fed8e907f1..0bbe2ea44f3e 100644
--- a/drivers/pci/host/pcie-iproc.h
+++ b/drivers/pci/host/pcie-iproc.h
@@ -90,6 +90,7 @@ struct iproc_pcie {
 #ifdef CONFIG_ARM
 	struct pci_sys_data sysdata;
 #endif
+	struct resource mem;
 	struct pci_bus *root_bus;
 	struct phy *phy;
 	int (*map_irq)(const struct pci_dev *, u8, u8);

pcie-iproc: broken 2nd (& 3rd?) controller support by c3245a566400 ("PCI: iproc: Request host bridge window resources")

[Rafał Miłecki]

On 03/09/2017 07:22 PM, Bjorn Helgaas wrote:
> On Thu, Mar 09, 2017 at 08:39:07AM +0100, Rafa? Mi?ecki wrote:
>> On 03/08/2017 01:56 PM, Rafa? Mi?ecki wrote:
>>> I just tried upgrading BCM5301X from 4.4 to 4.9 and noticed I don't see card
>>> connected to the 2nd controller.
>>>
>>> pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
>>> pci_bus 0000:00: root bus resource [mem 0x08000000-0x0fffffff]
>>> pcie_iproc_bcma bcma0:7: link: UP
>>> PCI: bus0: Fast back to back transfers disabled
>>> pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
>>> PCI: bus1: Fast back to back transfers disabled
>>> pci 0000:00:00.0: BAR 8: assigned [mem 0x08000000-0x080fffff]
>>> pci 0000:01:00.0: BAR 0: assigned [mem 0x08000000-0x08007fff 64bit]
>>> pci 0000:00:00.0: PCI bridge to [bus 01]
>>> pci 0000:00:00.0:   bridge window [mem 0x08000000-0x080fffff]
>>>
>>> pcie_iproc_bcma bcma0:8: resource collision: [mem 0x40000000-0x47ffffff] conflicts with PCIe MEM space [mem 0x40000000-0x47ffffff]
>>> pcie_iproc_bcma bcma0:8: PCIe controller setup failed
>>> pcie_iproc_bcma: probe of bcma0:8 failed with error -16
>>>
>>>
>>> This used to work with older kernels because there wasn't any collision check:
>>>
>>> pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
>>> pci_bus 0000:00: root bus resource [mem 0x08000000-0x0fffffff]
>>> pcie_iproc_bcma bcma0:7: link: UP
>>> PCI: bus0: Fast back to back transfers disabled
>>> pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
>>> PCI: bus1: Fast back to back transfers disabled
>>> pci 0000:00:00.0: BAR 8: assigned [mem 0x08000000-0x080fffff]
>>> pci 0000:01:00.0: BAR 0: assigned [mem 0x08000000-0x08007fff 64bit]
>>> pci 0000:00:00.0: PCI bridge to [bus 01]
>>> pci 0000:00:00.0:   bridge window [mem 0x08000000-0x080fffff]
>>>
>>> pcie_iproc_bcma bcma0:8: PCI host bridge to bus 0001:00
>>> pci_bus 0001:00: root bus resource [mem 0x40000000-0x47ffffff]
>>> pcie_iproc_bcma bcma0:8: link: UP
>>> PCI: bus0: Fast back to back transfers disabled
>>> pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
>>> PCI: bus1: Fast back to back transfers disabled
>>> pci 0001:00:00.0: BAR 8: assigned [mem 0x40000000-0x400fffff]
>>> pci 0001:01:00.0: BAR 0: assigned [mem 0x40000000-0x40007fff 64bit]
>>> pci 0001:00:00.0: PCI bridge to [bus 01]
>>> pci 0001:00:00.0:   bridge window [mem 0x40000000-0x400fffff]
>>>
>>>
>>> I guess the check is OK after all and the real problem is iproc driver assigning
>>> the same resource.
>>>
>>> Broadcom team: could you take a look at this, please?
>>
>> I found a reason of this conflict (and probably random crashes I started
>> seeing with 4.9). I believe we have a memory corruption.
>
> Yep, we're using a resource structure on the stack when we shouldn't.  Can
> you try the patch below?

Works great! Will you push it with
Cc: stable at vger.kernel.org # v4.8+
please?

Thanks for looking at this!

Also see below...


> (...)
>  	if (ret)
>  		dev_err(dev, "PCIe controller setup failed\n");
> -
> -	pci_free_resource_list(&res);
> +		pci_free_resource_list(&resources);
> +		return ret;
> +	}

Works great, except it doesn't compile (braces) ;)

pcie-iproc: broken 2nd (& 3rd?) controller support by c3245a566400 ("PCI: iproc: Request host bridge window resources")

[Ray Jui]

Hi Bjorn,

Thanks for helping take care of this.

Ray

On 3/10/2017 8:02 AM, Rafa? Mi?ecki wrote:
> On 03/09/2017 07:22 PM, Bjorn Helgaas wrote:
>> On Thu, Mar 09, 2017 at 08:39:07AM +0100, Rafa? Mi?ecki wrote:
>>> On 03/08/2017 01:56 PM, Rafa? Mi?ecki wrote:
>>>> I just tried upgrading BCM5301X from 4.4 to 4.9 and noticed I don't
>>>> see card
>>>> connected to the 2nd controller.
>>>>
>>>> pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
>>>> pci_bus 0000:00: root bus resource [mem 0x08000000-0x0fffffff]
>>>> pcie_iproc_bcma bcma0:7: link: UP
>>>> PCI: bus0: Fast back to back transfers disabled
>>>> pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]),
>>>> reconfiguring
>>>> PCI: bus1: Fast back to back transfers disabled
>>>> pci 0000:00:00.0: BAR 8: assigned [mem 0x08000000-0x080fffff]
>>>> pci 0000:01:00.0: BAR 0: assigned [mem 0x08000000-0x08007fff 64bit]
>>>> pci 0000:00:00.0: PCI bridge to [bus 01]
>>>> pci 0000:00:00.0:   bridge window [mem 0x08000000-0x080fffff]
>>>>
>>>> pcie_iproc_bcma bcma0:8: resource collision: [mem
>>>> 0x40000000-0x47ffffff] conflicts with PCIe MEM space [mem
>>>> 0x40000000-0x47ffffff]
>>>> pcie_iproc_bcma bcma0:8: PCIe controller setup failed
>>>> pcie_iproc_bcma: probe of bcma0:8 failed with error -16
>>>>
>>>>
>>>> This used to work with older kernels because there wasn't any
>>>> collision check:
>>>>
>>>> pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
>>>> pci_bus 0000:00: root bus resource [mem 0x08000000-0x0fffffff]
>>>> pcie_iproc_bcma bcma0:7: link: UP
>>>> PCI: bus0: Fast back to back transfers disabled
>>>> pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]),
>>>> reconfiguring
>>>> PCI: bus1: Fast back to back transfers disabled
>>>> pci 0000:00:00.0: BAR 8: assigned [mem 0x08000000-0x080fffff]
>>>> pci 0000:01:00.0: BAR 0: assigned [mem 0x08000000-0x08007fff 64bit]
>>>> pci 0000:00:00.0: PCI bridge to [bus 01]
>>>> pci 0000:00:00.0:   bridge window [mem 0x08000000-0x080fffff]
>>>>
>>>> pcie_iproc_bcma bcma0:8: PCI host bridge to bus 0001:00
>>>> pci_bus 0001:00: root bus resource [mem 0x40000000-0x47ffffff]
>>>> pcie_iproc_bcma bcma0:8: link: UP
>>>> PCI: bus0: Fast back to back transfers disabled
>>>> pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]),
>>>> reconfiguring
>>>> PCI: bus1: Fast back to back transfers disabled
>>>> pci 0001:00:00.0: BAR 8: assigned [mem 0x40000000-0x400fffff]
>>>> pci 0001:01:00.0: BAR 0: assigned [mem 0x40000000-0x40007fff 64bit]
>>>> pci 0001:00:00.0: PCI bridge to [bus 01]
>>>> pci 0001:00:00.0:   bridge window [mem 0x40000000-0x400fffff]
>>>>
>>>>
>>>> I guess the check is OK after all and the real problem is iproc
>>>> driver assigning
>>>> the same resource.
>>>>
>>>> Broadcom team: could you take a look at this, please?
>>>
>>> I found a reason of this conflict (and probably random crashes I started
>>> seeing with 4.9). I believe we have a memory corruption.
>>
>> Yep, we're using a resource structure on the stack when we shouldn't. 
>> Can
>> you try the patch below?
> 
> Works great! Will you push it with
> Cc: stable at vger.kernel.org # v4.8+
> please?
> 
> Thanks for looking at this!
> 
> Also see below...
> 
> 
>> (...)
>>      if (ret)
>>          dev_err(dev, "PCIe controller setup failed\n");
>> -
>> -    pci_free_resource_list(&res);
>> +        pci_free_resource_list(&resources);
>> +        return ret;
>> +    }
> 
> Works great, except it doesn't compile (braces) ;)

pcie-iproc: broken 2nd (& 3rd?) controller support by c3245a566400 ("PCI: iproc: Request host bridge window resources")

[Bjorn Helgaas]

On Fri, Mar 10, 2017 at 05:02:48PM +0100, Rafa? Mi?ecki wrote:
> On 03/09/2017 07:22 PM, Bjorn Helgaas wrote:
> >On Thu, Mar 09, 2017 at 08:39:07AM +0100, Rafa? Mi?ecki wrote:
> >>On 03/08/2017 01:56 PM, Rafa? Mi?ecki wrote:
> >>>I just tried upgrading BCM5301X from 4.4 to 4.9 and noticed I don't see card
> >>>connected to the 2nd controller.
> >>>
> >>>pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
> >>>pci_bus 0000:00: root bus resource [mem 0x08000000-0x0fffffff]
> >>>pcie_iproc_bcma bcma0:7: link: UP
> >>>PCI: bus0: Fast back to back transfers disabled
> >>>pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> >>>PCI: bus1: Fast back to back transfers disabled
> >>>pci 0000:00:00.0: BAR 8: assigned [mem 0x08000000-0x080fffff]
> >>>pci 0000:01:00.0: BAR 0: assigned [mem 0x08000000-0x08007fff 64bit]
> >>>pci 0000:00:00.0: PCI bridge to [bus 01]
> >>>pci 0000:00:00.0:   bridge window [mem 0x08000000-0x080fffff]
> >>>
> >>>pcie_iproc_bcma bcma0:8: resource collision: [mem 0x40000000-0x47ffffff] conflicts with PCIe MEM space [mem 0x40000000-0x47ffffff]
> >>>pcie_iproc_bcma bcma0:8: PCIe controller setup failed
> >>>pcie_iproc_bcma: probe of bcma0:8 failed with error -16
> >>>
> >>>
> >>>This used to work with older kernels because there wasn't any collision check:
> >>>
> >>>pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
> >>>pci_bus 0000:00: root bus resource [mem 0x08000000-0x0fffffff]
> >>>pcie_iproc_bcma bcma0:7: link: UP
> >>>PCI: bus0: Fast back to back transfers disabled
> >>>pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> >>>PCI: bus1: Fast back to back transfers disabled
> >>>pci 0000:00:00.0: BAR 8: assigned [mem 0x08000000-0x080fffff]
> >>>pci 0000:01:00.0: BAR 0: assigned [mem 0x08000000-0x08007fff 64bit]
> >>>pci 0000:00:00.0: PCI bridge to [bus 01]
> >>>pci 0000:00:00.0:   bridge window [mem 0x08000000-0x080fffff]
> >>>
> >>>pcie_iproc_bcma bcma0:8: PCI host bridge to bus 0001:00
> >>>pci_bus 0001:00: root bus resource [mem 0x40000000-0x47ffffff]
> >>>pcie_iproc_bcma bcma0:8: link: UP
> >>>PCI: bus0: Fast back to back transfers disabled
> >>>pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
> >>>PCI: bus1: Fast back to back transfers disabled
> >>>pci 0001:00:00.0: BAR 8: assigned [mem 0x40000000-0x400fffff]
> >>>pci 0001:01:00.0: BAR 0: assigned [mem 0x40000000-0x40007fff 64bit]
> >>>pci 0001:00:00.0: PCI bridge to [bus 01]
> >>>pci 0001:00:00.0:   bridge window [mem 0x40000000-0x400fffff]
> >>>
> >>>
> >>>I guess the check is OK after all and the real problem is iproc driver assigning
> >>>the same resource.
> >>>
> >>>Broadcom team: could you take a look at this, please?
> >>
> >>I found a reason of this conflict (and probably random crashes I started
> >>seeing with 4.9). I believe we have a memory corruption.
> >
> >Yep, we're using a resource structure on the stack when we shouldn't.  Can
> >you try the patch below?
> 
> Works great! Will you push it with
> Cc: stable at vger.kernel.org # v4.8+
> please?

Thanks for testing this.  I applied the patch (with the braces fix) to
for-linus for v4.11.

> >(...)
> > 	if (ret)
> > 		dev_err(dev, "PCIe controller setup failed\n");
> >-
> >-	pci_free_resource_list(&res);
> >+		pci_free_resource_list(&resources);
> >+		return ret;
> >+	}
> 
> Works great, except it doesn't compile (braces) ;)

pcie-iproc: broken 2nd (& 3rd?) controller support by c3245a566400 ("PCI: iproc: Request host bridge window resources")

[Rafał Miłecki]

On 03/10/2017 06:47 PM, Bjorn Helgaas wrote:
> On Fri, Mar 10, 2017 at 05:02:48PM +0100, Rafa? Mi?ecki wrote:
>> On 03/09/2017 07:22 PM, Bjorn Helgaas wrote:
>>> On Thu, Mar 09, 2017 at 08:39:07AM +0100, Rafa? Mi?ecki wrote:
>>>> On 03/08/2017 01:56 PM, Rafa? Mi?ecki wrote:
>>>>> I just tried upgrading BCM5301X from 4.4 to 4.9 and noticed I don't see card
>>>>> connected to the 2nd controller.
>>>>>
>>>>> pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
>>>>> pci_bus 0000:00: root bus resource [mem 0x08000000-0x0fffffff]
>>>>> pcie_iproc_bcma bcma0:7: link: UP
>>>>> PCI: bus0: Fast back to back transfers disabled
>>>>> pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
>>>>> PCI: bus1: Fast back to back transfers disabled
>>>>> pci 0000:00:00.0: BAR 8: assigned [mem 0x08000000-0x080fffff]
>>>>> pci 0000:01:00.0: BAR 0: assigned [mem 0x08000000-0x08007fff 64bit]
>>>>> pci 0000:00:00.0: PCI bridge to [bus 01]
>>>>> pci 0000:00:00.0:   bridge window [mem 0x08000000-0x080fffff]
>>>>>
>>>>> pcie_iproc_bcma bcma0:8: resource collision: [mem 0x40000000-0x47ffffff] conflicts with PCIe MEM space [mem 0x40000000-0x47ffffff]
>>>>> pcie_iproc_bcma bcma0:8: PCIe controller setup failed
>>>>> pcie_iproc_bcma: probe of bcma0:8 failed with error -16
>>>>>
>>>>>
>>>>> This used to work with older kernels because there wasn't any collision check:
>>>>>
>>>>> pcie_iproc_bcma bcma0:7: PCI host bridge to bus 0000:00
>>>>> pci_bus 0000:00: root bus resource [mem 0x08000000-0x0fffffff]
>>>>> pcie_iproc_bcma bcma0:7: link: UP
>>>>> PCI: bus0: Fast back to back transfers disabled
>>>>> pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
>>>>> PCI: bus1: Fast back to back transfers disabled
>>>>> pci 0000:00:00.0: BAR 8: assigned [mem 0x08000000-0x080fffff]
>>>>> pci 0000:01:00.0: BAR 0: assigned [mem 0x08000000-0x08007fff 64bit]
>>>>> pci 0000:00:00.0: PCI bridge to [bus 01]
>>>>> pci 0000:00:00.0:   bridge window [mem 0x08000000-0x080fffff]
>>>>>
>>>>> pcie_iproc_bcma bcma0:8: PCI host bridge to bus 0001:00
>>>>> pci_bus 0001:00: root bus resource [mem 0x40000000-0x47ffffff]
>>>>> pcie_iproc_bcma bcma0:8: link: UP
>>>>> PCI: bus0: Fast back to back transfers disabled
>>>>> pci 0001:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
>>>>> PCI: bus1: Fast back to back transfers disabled
>>>>> pci 0001:00:00.0: BAR 8: assigned [mem 0x40000000-0x400fffff]
>>>>> pci 0001:01:00.0: BAR 0: assigned [mem 0x40000000-0x40007fff 64bit]
>>>>> pci 0001:00:00.0: PCI bridge to [bus 01]
>>>>> pci 0001:00:00.0:   bridge window [mem 0x40000000-0x400fffff]
>>>>>
>>>>>
>>>>> I guess the check is OK after all and the real problem is iproc driver assigning
>>>>> the same resource.
>>>>>
>>>>> Broadcom team: could you take a look at this, please?
>>>>
>>>> I found a reason of this conflict (and probably random crashes I started
>>>> seeing with 4.9). I believe we have a memory corruption.
>>>
>>> Yep, we're using a resource structure on the stack when we shouldn't.  Can
>>> you try the patch below?
>>
>> Works great! Will you push it with
>> Cc: stable at vger.kernel.org # v4.8+
>> please?
>
> Thanks for testing this.  I applied the patch (with the braces fix) to
> for-linus for v4.11.

Thanks!

Btw. this is how it looks now (all good it seems):

root at LEDE:/# cat /proc/iomem
00000000-07ffffff : System RAM
   00008000-0043cfff : Kernel code
   00476000-004d95df : Kernel data
08000000-0fffffff : PCIe MEM space
   08000000-085fffff : PCI Bus 0000:01
     08000000-083fffff : 0000:01:00.0
     08400000-08407fff : 0000:01:00.0
18000300-180003ff : serial
18003000-18003fff : ccb-mii
1800c000-1800cfff : dmu
18021000-18021fff : /axi at 18000000/usb2 at 21000/ehci at 21000
18022000-18022fff : /axi at 18000000/usb2 at 21000/ohci at 22000
18023000-18023fff : /axi at 18000000/usb3 at 23000/xhci at 23000
18028000-180285ff : nand
18028f00-18028f1f : iproc-ext
18105000-18105fff : dmp
1811a408-1811aa07 : iproc-idm
40000000-47ffffff : PCIe MEM space
   40000000-405fffff : PCI Bus 0001:01
     40000000-403fffff : 0001:01:00.0
     40400000-40407fff : 0001:01:00.0
88000000-8fffffff : System RAM