[Discussion] PCI Resource Resizing & Assignment

[Gavin Shan <gwshan@linux.vnet.ibm.com>]

Hi Yinghai,

The purpose of this is going to have some discussion on current PCI resource
resizing and assignment implementation, mainly related to __pci_bus_size_bridges()
and pbus_size_mem().

In current implementation, all 64-bits and prefetchable BARs are going to be covered
by 64-bits and prefetchable bridge windows, and all other types of BARs are covered
by 32-bits bridge windows. It means 64-bits and non-prefetchable BARs are assigned
from 32-bits bridge windows which is usually limited and 2GB on our PowerNV platform.
It seems the attribute ("prefetchable") plays important role in the design so that
the final resource layout is compatible with: non-prefetchable BARs cannot be covered
by prefetchable bridge windows. It seems that's not true any more in PCI express domain,
meaning non-prefetchable BARs can live under prefetchable bridge windows. If so, I
think the "prefetchable" attribute isn't important and shouldn't affect the implementation
so much as we had. PCI/PCIx domain still have the problem: non-prefetchable BARs cannot
live behind a prefetchable bridge window. Apart from PCI/PCIx, we probably need enhance
current implementation for PCIe domain: 64-bits BARs are covered by 64-bits bridge windows
and 32-bits BARs are covered by 32-bits bridge windows, meaning the attribute "prefetchable"
won't have effect during resizing/assignment in PCIe domain. We will benefit from the
enhancement: (A) 64-bits non-prefetchable BARs will be covered by 64-bits prefetchable
bridge windows instead of 32-bits bridge windows, avoid 32-bits resource to be exhausted
quickly. (B) VFs whose BARs isn't 64-bits and prefetchable cannot be enabled on PowerNV
platform. It's not a issue with this enhancement.

The situation becomes complicated when we have combination of PCIe and PCI/PCIx domains.
We still need follow the rule in this scenario: non-prefetchable BARs cannot be covered
by prefetchable bridge windows. It might be more complicated with PCI hotplug is under
consideration. However, the platform (e.g. PowerNV) could have clear idea if PCI/PCIx
is going to be supported on particular PHB. I think the above enhancement can be applied
if PCI/PCIx won't be supported on particular PHB.

Thanks,
Gavin