From: Khalid Aziz <khalid.aziz@oracle.com>
To: Yinghai Lu <yinghai@kernel.org>
Cc: Bjorn Helgaas <bhelgaas@google.com>,
David Miller <davem@davemloft.net>,
Benjamin Herrenschmidt <benh@kernel.crashing.org>,
Wei Yang <weiyang@linux.vnet.ibm.com>, TJ <linux@iam.tj>,
Yijing Wang <wangyijing@huawei.com>,
linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v6 01/53] sparc/PCI: Add mem64 resource parsing for root bus
Date: Fri, 02 Oct 2015 14:00:07 -0600 [thread overview]
Message-ID: <1443816007.7936.24.camel@oracle.com> (raw)
In-Reply-To: <1443678807-786-2-git-send-email-yinghai@kernel.org>
On Wed, 2015-09-30 at 22:52 -0700, Yinghai Lu wrote:
> Found "no compatible bridge window" warning in boot log from T5-8.
>
> pci 0000:00:01.0: can't claim BAR 15 [mem 0x100000000-0x4afffffff pref]: no compatible bridge window
>
> That resource is above 4G, but does not get offset correctly as
> root bus only report io and mem32.
>
> pci_sun4v f02dbcfc: PCI host bridge to bus 0000:00
> pci_bus 0000:00: root bus resource [io 0x804000000000-0x80400fffffff] (bus address [0x0000-0xfffffff])
> pci_bus 0000:00: root bus resource [mem 0x800000000000-0x80007effffff] (bus address [0x00000000-0x7effffff])
> pci_bus 0000:00: root bus resource [bus 00-77]
>
> Add mem64 handling in pci_common for sparc, so we can have 64bit resource
> registered for root bus at first.
>
> After patch, will have:
> pci_sun4v f02dbcfc: PCI host bridge to bus 0000:00
> pci_bus 0000:00: root bus resource [io 0x804000000000-0x80400fffffff] (bus address [0x0000-0xfffffff])
> pci_bus 0000:00: root bus resource [mem 0x800000000000-0x80007effffff] (bus address [0x00000000-0x7effffff])
> pci_bus 0000:00: root bus resource [mem 0x800100000000-0x8007ffffffff] (bus address [0x100000000-0x7ffffffff])
> pci_bus 0000:00: root bus resource [bus 00-77]
>
> -v2: mem64_space should use mem_space.start as offset.
> -v3: add IORESOURCE_MEM_64 flag
>
> Fixes: commit d63e2e1f3df9 ("sparc/PCI: Clip bridge windows to fit in upstream windows")
> Link: http://lkml.kernel.org/r/CAE9FiQU1gJY1LYrxs+ma5LCTEEe4xmtjRG0aXJ9K_Tsu+m9Wuw@mail.gmail.com
> Reported-by: David Ahern <david.ahern@oracle.com>
> Tested-by: David Ahern <david.ahern@oracle.com>
> Signed-off-by: Yinghai Lu <yinghai@kernel.org>
I am still seeing a large number of "can't claim BAR" and "no compatible
bridge window" messages on sparc after applying the entire patch series
to 4.3-rc3:
# dmesg | grep "n't claim"
pci 0000:71:00.0: can't claim BAR 0 [mem 0x2000480000000-0x20004800fffff
64bit]: no compatible bridge window
pci 0004:01:00.0: can't claim BAR 1 [mem
0x2004000100000-0x2004000103fff]: no compatible bridge window
pci 0004:01:00.0: can't claim BAR 6 [mem
0x2004000110000-0x200400011ffff]: no compatible bridge window
pci 0006:01:00.0: can't claim BAR 1 [mem 0x2021100000000-0x202110000ffff
64bit]: no compatible bridge window
pci 0006:01:00.0: can't claim BAR 3 [mem 0x2021100040000-0x202110007ffff
64bit]: no compatible bridge window
pci 0006:01:00.0: can't claim BAR 6 [mem
0x2021000100000-0x20210001fffff]: no compatible bridge window
pci 0009:01:00.0: can't claim BAR 0 [mem 0x2024100000000-0x20241001fffff
64bit pref]: no compatible bridge window
pci 0009:01:00.0: can't claim BAR 7 [mem 0x2024100304000-0x2024100403fff
64bit]: no compatible bridge window
pci 0009:01:00.0: can't claim BAR 10 [mem
0x2024100404000-0x2024100503fff 64bit]: no compatible bridge window
pci 0009:01:00.1: can't claim BAR 7 [mem 0x2024100508000-0x2024100607fff
64bit]: no compatible bridge window
pci 0009:01:00.1: can't claim BAR 10 [mem
0x2024100900000-0x20241009fffff 64bit]: no compatible bridge window
pci 000b:01:00.0: can't claim BAR 0 [mem 0x2041100000000-0x20411001fffff
64bit pref]: no compatible bridge window
pci 000b:01:00.0: can't claim BAR 7 [mem 0x2041100304000-0x2041100403fff
64bit]: no compatible bridge window
pci 000b:01:00.0: can't claim BAR 10 [mem
0x2041100404000-0x2041100503fff 64bit]: no compatible bridge window
pci 000b:01:00.1: can't claim BAR 7 [mem 0x2041100508000-0x2041100607fff
64bit]: no compatible bridge window
pci 000b:01:00.1: can't claim BAR 10 [mem
0x2041100900000-0x20411009fffff 64bit]: no compatible bridge window
pci 000b:02:00.0: can't claim BAR 1 [mem 0x2041110000000-0x204111000ffff
64bit]: no compatible bridge window
pci 000b:02:00.0: can't claim BAR 3 [mem 0x2041110040000-0x204111007ffff
64bit]: no compatible bridge window
pci 000b:02:00.0: can't claim BAR 6 [mem
0x2041000100000-0x20410001fffff]: no compatible bridge window
pci 000e:01:00.0: can't claim BAR 0 [mem 0x2044100000000-0x204410000ffff
64bit]: no compatible bridge window
pci 000e:01:00.0: can't claim BAR 2 [mem 0x2044100010000-0x2044100011fff
64bit]: no compatible bridge window
pci 0013:01:00.0: can't claim BAR 0 [mem 0x2064100000000-0x206410000ffff
64bit]: no compatible bridge window
pci 0013:01:00.0: can't claim BAR 2 [mem 0x2064100010000-0x2064100011fff
64bit]: no compatible bridge window
pci 0006:01:00.0: can't claim BAR 1 [mem size 0x00010000 64bit]: no
address assigned
pci 0006:01:00.0: can't claim BAR 3 [mem size 0x00040000 64bit]: no
address assigned
pci 000b:02:00.0: can't claim BAR 1 [mem size 0x00010000 64bit]: no
address assigned
pci 000b:02:00.0: can't claim BAR 3 [mem size 0x00040000 64bit]: no
address assigned
After applying this patch series, ixgbe driver failed to attach to two
of the NICs (PCI devices 0009:01:00.0 and 000b:01:00.0) on my sparc
machine. I tracked down the failure for 0009:01:00.0 and it looks like
BAR 0 could not be mapped by the driver. BAR 0 for this device wants to
be mapped at 0x2024100000000-0x20241001fffff:
PCI: Claiming 0009:01:00.0: Resource 0: 0002024100000000..00020241001fffff [10220c]
but 0002024100100000..00020241100fffff was allocated to the bridge above
this device which overlaps this address range:
PCI: Claiming 0009:00:01.0: Resource 14: 0002024100100000..00020241100fffff [10220c]
It seems when the bridge 0009:00:01.0 is scanned by
of_scan_pci_bridge(), it asks for range
0000000100000000-000000010fffffff:
RAW Range[c3000000:00000001:00000000:c3000000:00000001:00000000:00000000:100
00000]
Using flags[0010220c] start[0000000100000000] size[0000000010000000]
which pcibios_bus_to_resource() maps to
0002024100100000..00020241100fffff without considering this range might
overlap address range required by a device underneath the bridge later
when devices are scanned. This causes BAR0 mapping for the device under
the bridge to fail and driver fails to attach to the device.
Bus 9 on this machine looks like:
+-[0009:00]---01.0-[01]--+-00.0 Intel Corporation Ethernet Controller 10-Gigabit X540-AT2
| \-00.1 Intel Corporation Ethernet Controller 10-Gigabit X540-AT2
dmesg output with "debug ignore_loglevel ofpci_debug=1" for scan of bus
9 is below:
/pci@301: SUN4V PCI Bus Module
/pci@301: On NUMA node -1
/pci@301: PCI IO[2027e40000000] MEM[2024000100000] MEM64[2024100000000]
/pci@301: Unable to request IOMMU resource.
/pci@301: MSI Queue first[0] num[56] count[128] devino[0x6]
/pci@301: MSI first[0] num[1024] mask[0x3ff] width[32]
/pci@301: MSI addr32[0x7f000000:0x800000] addr64[0xe00000003f000000:0x800000]
/pci@301: MSI queues at RA [0000401f85a80000]
PCI: Scanning PBM /pci@301
pci_sun4v f0339c2c: PCI host bridge to bus 0009:00
pci_bus 0009:00: root bus resource [io 0x2027e40000000-0x2027e4fffffff] (bus address [0x0000-0xfffffff])
pci_bus 0009:00: root bus resource [mem 0x2024000100000-0x202407effffff] (bus address [0x00000000-0x7eefffff])
pci_bus 0009:00: root bus resource [mem 0x2024100000000-0x2024dffffffff] (bus address [0xfff00000-0xdffefffff])
pci_bus 0009:00: root bus resource [bus 00-01]
PCI: scan_bus[/pci@301] bus no 0
* /pci@301/pci@1
create device, devfn: 8, type: pciex
class: 0x60400 device name: 0009:00:01.0
adding to system ...
pci 0009:00:01.0: PME# supported from D0 D3hot D3cold
PCI: dev header type: 1
of_scan_pci_bridge(/pci@301/pci@1)
Bridge bus range [1 --> 1]
Bridge ranges[fff8401fdfca5180] simba[0]
RAW Range[81000000:00000000:00000000:81000000:00000000:00000000:00000000:00001000]
Using flags[00000101] start[0000000000000000] size[0000000000001000]
RAW Range[c3000000:00000001:00000000:c3000000:00000001:00000000:00000000:10000000]
Using flags[0010220c] start[0000000100000000] size[0000000010000000]
bus name: PCI Bus 0009:01
PCI: scan_bus[/pci@301/pci@1] bus no 1
* /pci@301/pci@1/network@0
create device, devfn: 0, type: network
class: 0x20000 device name: 0009:01:00.0
parse addresses (60 bytes) @ fff8401fdfca3cc0
start: 2024100000000, end: 20241001fffff, i: 10
start: 2027e40000000, end: 2027e4000001f, i: 18
start: 2024100200000, end: 2024100203fff, i: 20
adding to system ...
pci 0009:01:00.0: PME# supported from D0 D3hot D3cold
pci 0009:01:00.0: reg 0x184: [mem 0x2024100304000-0x2024100307fff 64bit]
pci 0009:01:00.0: VF(n) BAR0 space: [mem 0x2024100304000-0x2024100403fff 64bit] (contains BAR0 for 64 VFs)
pci 0009:01:00.0: reg 0x190: [mem 0x2024100404000-0x2024100407fff 64bit]
pci 0009:01:00.0: VF(n) BAR3 space: [mem 0x2024100404000-0x2024100503fff 64bit] (contains BAR3 for 64 VFs)
PCI: dev header type: 0
* /pci@301/pci@1/network@0,1
create device, devfn: 1, type: network
class: 0x20000 device name: 0009:01:00.1
parse addresses (60 bytes) @ fff8401fdfca2080
start: 2024100600000, end: 20241007fffff, i: 10
start: 2027e40000020, end: 2027e4000003f, i: 18
start: 2024100404000, end: 2024100407fff, i: 20
adding to system ...
pci 0009:01:00.1: PME# supported from D0 D3hot D3cold
pci 0009:01:00.1: reg 0x184: [mem 0x2024100508000-0x202410050bfff 64bit]
pci 0009:01:00.1: VF(n) BAR0 space: [mem 0x2024100508000-0x2024100607fff 64bit] (contains BAR0 for 64 VFs)
pci 0009:01:00.1: reg 0x190: [mem 0x2024100900000-0x2024100903fff 64bit]
pci 0009:01:00.1: VF(n) BAR3 space: [mem 0x2024100900000-0x20241009fffff 64bit] (contains BAR3 for 64 VFs)
PCI: dev header type: 0
PCI: Claiming 0009:00:01.0: Resource 13: 0002027e40000000..0002027e40000fff [101]
PCI: Claiming 0009:00:01.0: Resource 14: 0002024100100000..00020241100fffff [10220c]
PCI: Claiming 0009:01:00.0: Resource 0: 0002024100000000..00020241001fffff [10220c]
pci 0009:01:00.0: can't claim BAR 0 [mem 0x2024100000000-0x20241001fffff 64bit pref]: no compatible bridge window
PCI: Claiming 0009:01:00.0: Resource 2: 0002027e40000000..0002027e4000001f [101]
PCI: Claiming 0009:01:00.0: Resource 4: 0002024100200000..0002024100203fff [10220c]
PCI: Claiming 0009:01:00.0: Resource 7: 0002024100304000..0002024100403fff [140204]
pci 0009:01:00.0: can't claim BAR 7 [mem 0x2024100304000-0x2024100403fff 64bit]: no compatible bridge window
PCI: Claiming 0009:01:00.0: Resource 10: 0002024100404000..0002024100503fff [140204]
pci 0009:01:00.0: can't claim BAR 10 [mem 0x2024100404000-0x2024100503fff 64bit]: no compatible bridge window
PCI: Claiming 0009:01:00.1: Resource 0: 0002024100600000..00020241007fffff [10220c]
PCI: Claiming 0009:01:00.1: Resource 2: 0002027e40000020..0002027e4000003f [101]
PCI: Claiming 0009:01:00.1: Resource 4: 0002024100404000..0002024100407fff [10220c]
PCI: Claiming 0009:01:00.1: Resource 7: 0002024100508000..0002024100607fff [140204]
pci 0009:01:00.1: can't claim BAR 7 [mem 0x2024100508000-0x2024100607fff 64bit]: no compatible bridge window
PCI: Claiming 0009:01:00.1: Resource 10: 0002024100900000..00020241009fffff [140204]
pci 0009:01:00.1: can't claim BAR 10 [mem 0x2024100900000-0x20241009fffff 64bit]: no compatible bridge window
--
Khalid
next prev parent reply other threads:[~2015-10-02 20:00 UTC|newest]
Thread overview: 59+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-10-01 5:52 [PATCH v6 00/53] PCI: Resource allocation cleanup for v4.4 Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 01/53] sparc/PCI: Add mem64 resource parsing for root bus Yinghai Lu
2015-10-02 20:00 ` Khalid Aziz [this message]
2015-10-02 22:05 ` Yinghai Lu
2015-10-02 23:05 ` Khalid Aziz
2015-10-02 23:16 ` Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 02/53] sparc/PCI: Add IORESOURCE_MEM_64 for 64-bit resource in OF parsing Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 03/53] powerpc/PCI: " Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 04/53] OF/PCI: Add IORESOURCE_MEM_64 for 64-bit resource Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 05/53] PCI: Don't release fixed resource for realloc Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 06/53] PCI: Claim fixed resource during remove/rescan path Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 07/53] PCI: Set resource to FIXED for LSI devices Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 08/53] PCI: Separate realloc list checking after allocation Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 09/53] PCI: Treat optional as required in first try for bridge rescan Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 10/53] PCI: Get new realloc size for bridge for last try Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 11/53] PCI: Don't release sibling bridge resources during hotplug Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 12/53] PCI: Cleanup res_to_dev_res() printout Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 13/53] PCI: Reuse res_to_dev_res() in reassign_resources_sorted() Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 14/53] PCI: Use correct align for optional only resources during sorting Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 15/53] PCI: Optimize bus min_align/size calculation during sizing Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 16/53] PCI: Optimize bus align/size calculation for optional " Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 17/53] PCI: Don't add too much optional size for hotplug bridge MMIO Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 18/53] PCI: Reorder resources list for required/optional resources Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 19/53] PCI: Remove duplicated code for resource sorting Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 20/53] PCI: Rename pdev_sort_resources() to pdev_assign_resources_prepare() Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 21/53] PCI: Treat ROM resource as optional during realloc Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 22/53] PCI: Add debug printout during releasing partial assigned resources Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 23/53] PCI: Simplify res reference using in __assign_resources_sorted() Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 24/53] PCI: Add __add_to_list() Yinghai Lu
2015-10-01 5:52 ` [PATCH v6 25/53] PCI: Cache window alignment value during bus sizing Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 26/53] PCI: Check if resource is allocated before trying to assign one Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 27/53] PCI: Separate out save_resources()/restore_resources() Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 28/53] PCI: Move comment to pci_need_to_release() Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 29/53] PCI: Separate required+optional assigning to another function Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 30/53] PCI: Skip required+optional if there is no optional Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 31/53] PCI: Move saved required resource list out of required+optional assigning Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 32/53] PCI: Add alt_size ressource allocation support Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 33/53] PCI: Add support for more than two alt_size entries under same bridge Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 34/53] PCI: Fix size calculation with old_size on rescan path Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 35/53] PCI: Don't add too much optional size for hotplug bridge io Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 36/53] PCI: Move ISA io port align out of calculate_iosize() Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 37/53] PCI: Don't add too much io port for hotplug bridge with old size Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 38/53] PCI: Unify calculate_size() for io port and MMIO Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 39/53] PCI: Allow bridge optional only io port resource required size to be 0 Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 40/53] PCI: Unify skip_ioresource_align() Yinghai Lu
2015-10-01 8:17 ` Thomas Gleixner
2015-10-01 5:53 ` [PATCH v6 41/53] PCI: Kill macro checking for bus io port sizing Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 42/53] resources: Split out __allocate_resource() Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 43/53] resources: Make allocate_resource() return best fit resource Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 44/53] PCI, x86: Allocate from high in available window for MMIO Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 45/53] PCI: Add debug print out for min_align and alt_size Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 46/53] PCI: Check pref compatible bit for mem64 resource of PCIe device Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 47/53] PCI: Only treat non-pref mmio64 as pref if all bridges have MEM_64 Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 48/53] PCI: Add has_mem64 for struct host_bridge Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 49/53] PCI: Only treat non-pref mmio64 as pref if host bridge has mmio64 Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 50/53] PCI: Restore pref MMIO allocation logic for host bridge without mmio64 Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 51/53] PCI, x86: Add pci=assign_pref_bars to reallocate pref BARs Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 52/53] PCI: Introduce resource_disabled() Yinghai Lu
2015-10-01 5:53 ` [PATCH v6 53/53] PCI: Don't set flags to 0 when assign resource fail Yinghai Lu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1443816007.7936.24.camel@oracle.com \
--to=khalid.aziz@oracle.com \
--cc=benh@kernel.crashing.org \
--cc=bhelgaas@google.com \
--cc=davem@davemloft.net \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux@iam.tj \
--cc=wangyijing@huawei.com \
--cc=weiyang@linux.vnet.ibm.com \
--cc=yinghai@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).