Re: SPARC64: getting "no compatible bridge window" errors :/

[Bjorn Helgaas <helgaas@kernel.org>]

On Fri, Oct 21, 2022 at 05:47:59AM +0200, Richard Rogalski wrote:
> Hello, very very sorry about the late reply. Life has been hectic. Also, not sure if this is how I reply to one of these, sorry if I screwed it up :)
> 
> > This is great, thanks a lot for your report!  Is this a regression?
> 
> Believe it or not, I am a brand new SPARC user :). So I can't say
> right now. Should I try a few old kernel releases to check?

I wouldn't bother trying older kernels.  In fact, I just noticed that
you're running a 5.15 kernel, which is about a year old.  It would be
much more interesting to try to reproduce the problem on a current
kernel, e.g., v6.0.

At https://packages.gentoo.org/packages/sys-kernel/gentoo-kernel, it
doesn't look like sparc gets much attention ;)

> > Any chance you could collect a dmesg log with "ofpci_debug=1"?
> 
> https://gitlab.freedesktop.org/drm/amd/uploads/0ed3c92921d7f88b06654b5f46e9756d/dmesg
> 
> > Do the devices we complain about (NICs and storage HBAs 09:00.0, 
> > 09:00.1, 0d:00.0, 0d:00.1, 0e:00.0, 0f:00.0, 0001:03:00.0, 
> > 0001:03:00.1, 0001:0:00.0, 0001:0a:00.1) work?
> 
> Well, I don't have any fiber optic equipment: these just came with
> the server. Also it has wayy too many NICs. I can't quite say.
> However... for the HBAs, that's where my root is :O. This is mildly
> concerning :D.

I spent way too long looking at these PCI resource weirdnesses.
Bottom line: ignore them.

From your ofpci_debug dmesg log (annotated with logging the PCI core
would do if it were doing this instead of the sparc OF code):

  pci@400: PCI MEM   [mem 0x84000100000-0x8407f7fffff] offset 84000000000
  pci@400: PCI MEM64 [mem 0x84100000000-0x84dffffffff] offset 80000000000
  pci_bus 0000:00: root bus resource [mem 0x84000100000-0x8407f7fffff] (bus address [0x00100000-0x7f7fffff])
  pci_bus 0000:00: root bus resource [mem 0x84100000000-0x84dffffffff] (bus address [0x4100000000-0x4dffffffff])

  pci 0000:04:00.0: can't claim VGA legacy [mem 0x000a0000-0x000bffff]: no compatible bridge window

    This one happens because according to OF, there is no bridge
    aperture to the PCI bus 0xa0000-0xbffff region.  The only accessible
    PCI bus regions are [0x00100000-0x7f7fffff] and
    [0x4100000000-0x4dffffffff].  Probably an OF defect.

  pci 0000:02:0c.0: PCI bridge to [bus 09]
  pci 0000:02:0c.0:       Using flags[0010220c] start[0000004120000000] size[0000000010000000]
  pci 0000:02:0c.0:   bridge window [mem 0x84120000000-0x8412fffffff 64bit pref]
  pci 0000:09:00.0: can't claim BAR 0 [mem 0x84120000000-0x8412007ffff 64bit]: no compatible bridge window

    These and similar warnings happen because OF says the upstream
    bridge window is prefetchable, but this is a non-prefetchable BAR.
    These likely work fine because in most cases prefetching will not
    occur on PCIe, even though the bridge window allows it.

So the warnings above are mostly harmless.  If you were to hot-add
something, there could be issues because we aren't keeping track of
the space these devices use.

lspci on sparc is unusual: it shows PCI bus addresses, not CPU
physical addresses like other arches [1], which means we see things
like this in dmesg, which shows the CPU physical address:

  pci_bus 0000:00: root bus resource [mem 0x84000100000-0x8407f7fffff] (bus address [0x00100000-0x7f7fffff])
  pci 0000:04:00.0: reg 0x10: [mem 0x84000800000-0x84000ffffff]

and this in lspci, which is the PCI bus address:

  0000:04:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 10) (prog-if 00 [VGA controller])
      Region 0: Memory at 00800000 (32-bit, non-prefetchable) [size=8M]

Annoying but harmless.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/?id=v5.18#n1