Re: Re: [PATCH] PCI: Enable io space 1k granularity for intel cpu root port

[Bjorn Helgaas <helgaas@kernel.org>]

On Sun, Jun 30, 2024 at 02:52:25AM +0000, Zhou Shengqing wrote:
> > On Thu, Jun 27, 2024 at 12:58:56AM +0000, Zhou Shengqing wrote:
> > > This patch add 1k granularity for intel root port bridge.Intel latest
> > > server CPU support 1K granularity,And there is an BIOS setup item named
> > > "EN1K",but linux doesn't support it. if an IIO has 5 IOU (SPR has 5 IOUs)
> > > all are bifurcated 2x8.In a 2P server system,There are 20 P2P bridges
> > > present.if keep 4K granularity allocation,it need 20*4=80k io space,
> > > exceeding 64k.I test it in a 16*nvidia 4090s system under intel eaglestrem
> > > platform.There are six 4090s that cannot be allocated I/O resources.
> > > So I applied this patch.And I found a similar implementation in quirks.c,
> > > but it only targets the Intel P64H2 platform.
> > > 
> > > Signed-off-by: Zhou Shengqing <zhoushengqing@ttyinfo.com>
> > > ---
> > >  drivers/pci/probe.c | 23 +++++++++++++++++++++++
> > >  1 file changed, 23 insertions(+)
> > > 
> > > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> > > index 5fbabb4e3425..909962795311 100644
> > > --- a/drivers/pci/probe.c
> > > +++ b/drivers/pci/probe.c
> > > @@ -461,6 +461,9 @@ static void pci_read_bridge_windows(struct pci_dev *bridge)
> > >  	u32 buses;
> > >  	u16 io;
> > >  	u32 pmem, tmp;
> > > +	u16 ven_id, dev_id;
> > > +	u16 en1k = 0;
> > > +	struct pci_dev *dev = NULL;
> > >  	struct resource res;
> > >  
> > >  	pci_read_config_dword(bridge, PCI_PRIMARY_BUS, &buses);
> > > @@ -478,6 +481,26 @@ static void pci_read_bridge_windows(struct pci_dev *bridge)
> > >  	}
> > >  	if (io) {
> > >  		bridge->io_window = 1;
> > > +		if (pci_is_root_bus(bridge->bus)) {
> > > +			list_for_each_entry(dev, &bridge->bus->devices, bus_list) {
> > > +				pci_read_config_word(dev, PCI_VENDOR_ID, &ven_id);
> > > +				pci_read_config_word(dev, PCI_DEVICE_ID, &dev_id);
> > > +				if (ven_id == PCI_VENDOR_ID_INTEL && dev_id == 0x09a2) {
> > > +					/*IIO MISC Control offset 0x1c0*/
> > > +					pci_read_config_word(dev, 0x1c0, &en1k);
> > > +				}
> > > +			}
> > > +		/*
> > > +		 *Intel ICX SPR EMR GNR
> > > +		 *IIO MISC Control (IIOMISCCTRL_1_5_0_CFG) — Offset 1C0h
> > > +		 *bit 2:Enable 1K (EN1K)
> > > +		 *This bit when set, enables 1K granularity for I/O space decode
> > > +		 *in each of the virtual P2P bridges
> > > +		 *corresponding to root ports, and DMI ports.
> > > +		 */
> > > +		if (en1k & 0x4)
> > > +			bridge->io_window_1k = 1;
> > > +		}
> > 
> > I still think this is not going to work because I don't want this kind
> > of device-specific clutter in this generic path. The pcibios_*
> > interfaces are history that we'd like to get rid of also, but it would
> > be better than putting it here.
> 
> Do you think it should be putted to the pci_bios* interface?
> 
> And if there is no suitable place to apply this patch, 
> then let's just ignore this issue.

What are the device/function addresses of the IIO and the Root Ports?

They must all be on the root bus, and if the IIO is enumerated first
(i.e., if it has a smaller device/function number), you may be able to 
make a quirk that applies to the Root Port vendor/device ID but reads
the config bit from the IIO.

quirk_passive_release(), quirk_vialatency(), quirk_amd_780_apc_msi(),
etc. do similar things.

Bjorn