Re: [RFC v2 PATCH 1/1] PCI: override BIOS/firmware resource allocation

[Ram Pai <linuxram@us.ibm.com>]

On Thu, Oct 07, 2010 at 03:41:04PM -0600, Bjorn Helgaas wrote:
> On Thursday, October 07, 2010 02:42:13 pm Ram Pai wrote:
> > On Wed, Oct 06, 2010 at 10:13:02PM -0600, Bjorn Helgaas wrote:
> > > On Wed, Oct 06, 2010 at 05:30:41PM -0700, Ram Pai wrote:
> > > > On Wed, Oct 06, 2010 at 05:39:53PM -0600, Bjorn Helgaas wrote:
> > > > > On Wed, Oct 06, 2010 at 03:58:34PM -0700, Ram Pai wrote:
> > > > > >         PCI: override BIOS/firmware memory resource allocation
> > > > > > 		through command line parameters
> > > > > > 
> > > > > > 	Platforms that are unaware of  SRIOV  BARs  fail to allocate MMIO
> > > > > > 	resources  to SRIOV PCIe  devices. Hence  on  such  platforms the
> > > > > > 	OS fails to  enable  SRIOV.
> > > > > > 	Some  platforms  where  BIOS/uEFI resource   allocations  conflict
> > > > > > 	the conflicting devices are disabled.
> > > > > > 
> > > > > > 	Ideally we  would  want  the  OS  to detect and fix automatically
> > > > > > 	such problems and conflicts.  However previous  attempts to do so
> > > > > > 	have led to regression on legacy platforms.
> > > > > 
> > > > > I'm sorry to be a nay-sayer, but I think we just haven't tried hard
> > > > > enough.  Our ACPI/PCI/e820 resource management is not well integrated,
> > > > > and I suspect if we straightened that out, we could avoid some of the
> > > > > regressions we saw with previous attempts.
> > > > 
> > > > Can you be more specific as to what can be done to fix it automatically?
> > > > 
> > > > Neither accepting this approach nor telling what needs to be straightened out
> > > > to automatically fix all the systems out there, is just a deadend.
> > > 
> > > Yeah, I guess that wasn't really fair, sorry.  And keep in mind that I'm
> > > not the PCI maintainer, so these are just my opinions, nothing like an
> > > official "nack."
> > > 
> > > I did look at this dmesg log from the thread you referenced:
> > >     http://marc.info/?l=linux-kernel&m=127178918128740&w=2
> > > but it looks to me like we just completely botched it.  I don't see an
> > > SRIOV device or anything else that didn't have resources, so as far as I
> > > can tell, we started with working resource assignments from the BIOS,
> > > threw them away, and started over from scratch.  We failed because we
> > > tried to assign I/O port space to bridges with nothing behind them, and
> > > there was nothing left by the time we got to the 0000:09:04.0 device
> > > that actually *did* need the space.
> > 
> > hmm.. is that possible? Yinghai's patch sized the resource requirement of each
> > of the bridges, before actually allocating them. Which means a bridge with
> > no device behind it would not get any i/o space.
> 
> Here's what I see in the dmesg log referenced above:
> 
>     ACPI: PCI Root Bridge [PCI0] (0000:00)
>     pci_root PNP0A08:00: host bridge window [io 0x0000-0x0cf7]
>     pci_root PNP0A08:00: host bridge window [io 0x0d00-0xffff]
>     pci 0000:00:1c.0: PCI bridge to [bus 04-09]
>     pci 0000:00:1c.0:   bridge window [io 0xd000-0xdfff]
>     pci 0000:04:00.0: PCI bridge to [bus 05-09]
>     pci 0000:04:00.0:   bridge window [io 0xd000-0xdfff]
>     pci 0000:05:01.0: PCI bridge to [bus 08-09]
>     pci 0000:05:01.0:   bridge window [io 0xd000-0xdfff]
>     pci 0000:08:00.0: PCI bridge to [bus 09-09]
>     pci 0000:08:00.0:   bridge window [io 0xd000-0xdfff]
>     pci 0000:09:04.0: found [13f6:8788] class 000401 header type 00
>     pci 0000:09:04.0: reg 10: [io  0xd800-0xd8ff]
>     pci 0000:05:02.0: PCI bridge to [bus 07-07]
>     pci 0000:05:02.0:   bridge window [io 0xf000-0x0000] (disabled)
>     pci 0000:05:03.0: PCI bridge to [bus 06-06]
>     pci 0000:05:03.0:   bridge window [io 0xf000-0x0000] (disabled)
>     
> The above is the state as we got it from BIOS.  Despite all the bridges,
> 09:04.0 is the only device below the 00:1c.0 bridge, and it requires only
> 0x100 I/O ports.
> 
> There are no devices on buses 06 (below 05:03.0) or 07 (below 05:02.0).
> 
> I didn't look at Yinghai's patch to figure out *why*, but it sure looks like
> we released the 09:04.0 space, then tried to assign 0x2000 ports to 05:01.0 
> (which needs 0x100 and had 0x1000 originally), 0x1000 to 05:02.0 (which needs
> none), and 0x1000 to 05:03.0 (which also needs none):
> 
>     PCI: No. 3 try to assign unassigned res
>     release child resource [io  0xd800-0xd8ff]
>     pci 0000:08:00.0: resource 7 [io 0xd000-0xdfff] released
>     pci 0000:04:00.0: BAR 7: can't assign io (size 0x4000)
>     pci 0000:05:01.0: BAR 7: can't assign io (size 0x2000)
>     pci 0000:05:02.0: BAR 7: can't assign io (size 0x1000)
>     pci 0000:05:03.0: BAR 7: can't assign io (size 0x1000)
>     pci 0000:08:00.0: BAR 7: can't assign io (size 0x1000)

Actually the message preceeding to them  are even more surprising:

Apr 20 20:31:42 [kernel] pci 0000:04:00.0: BAR 8: can't assign mem (size
0xc00000)
Apr 20 20:31:42 [kernel] pci 0000:05:01.0: BAR 8: can't assign mem (size
0x200000)
Apr 20 20:31:42 [kernel] pci 0000:05:01.0: BAR 9: can't assign mem pref
(size 0x200000)
Apr 20 20:31:42 [kernel] pci 0000:05:02.0: BAR 8: can't assign mem (size
0x400000)
Apr 20 20:31:42 [kernel] pci 0000:05:03.0: BAR 7: can't assign io (size
0x1000)

Do these bridges have IOV BARs and those BARs are demanding i/o resources?
Something is really funny with this machine. Or I am reading this wrong?

RP