Re: [PATCH] PCI: vmd: Enable Hotplug based on BIOS setting on VMD rootports

[Nirmal Patel <nirmal.patel@linux.intel.com>]

On Tue, 2023-10-31 at 10:31 -0500, Bjorn Helgaas wrote:
> On Mon, Oct 30, 2023 at 04:16:54PM -0400, Nirmal Patel wrote:
> > VMD Hotplug should be enabled or disabled based on VMD rootports'
> > Hotplug configuration in BIOS. is_hotplug_bridge is set on each
> > VMD rootport based on Hotplug capable bit in SltCap in probe.c.
> > Check is_hotplug_bridge and enable or disable native_pcie_hotplug
> > based on that value.
> > 
> > Currently VMD driver copies ACPI settings or platform
> > configurations
> > for Hotplug, AER, DPC, PM, etc and enables or disables these
> > features
> > on VMD bridge which is not correct in case of Hotplug.
> 
> This needs some background about why it's correct to copy the ACPI
> settings in the case of AER, DPC, PM, etc, but incorrect for hotplug.
> 
> > Also during the Guest boot up, ACPI settings along with VMD UEFI
> > driver are not present in Guest BIOS which results in assigning
> > default values to Hotplug, AER, DPC, etc. As a result Hotplug is
> > disabled on VMD in the Guest OS.
> > 
> > This patch will make sure that Hotplug is enabled properly in Host
> > as well as in VM.
> 
> Did we come to some consensus about how or whether _OSC for the host
> bridge above the VMD device should apply to devices in the separate
> domain below the VMD?
We are not able to come to any consensus. Someone suggested to copy
either all _OSC flags or none. But logic behind that assumption is
that the VMD is a bridge device which is not completely true. VMD is an
endpoint device and it owns its domain.

Also please keep this in your consideration, since Guest BIOS
doesn't have _OSC implementation, all of the flags Hotplug, AER, DPC
are set to power state default value and VMD's very important hotplug
functionality is broken.

The patch 04b12ef163d1 assumes VMD is a bridge device and borrows/
*imposes system settings* for AER, DPC, Hotplug, PM, etc on VMD.
VMD is*type 0 PCI endpoint* device and all the PCI devices under VMD
are *privately *owned by VMD not by the OS.

Also VMD has its own Hotplug setting for its rootports in BIOS under
VMD settings that are different from global BIOS system settings. It is
these settings that give VMD its own unique functionality.

That is why I suggested three solutions but never got any confirmation.

#1: Revert the patch 04b12ef163d1 which was added under wrong
assumption. This patch didn't need to be added to VMD code if AER
was disabled from BIOS platform settings.

#2: VMD driver disables AER by copying AER BIOS system settings
which the patch 04b12ef163d1 does but do not change Hotplug.
I proposed this patch and didn't get approval.

#3: If hotplug is enabled on VMD root ports, make sure hotplug is
enabled on the bridge rootports are connected to. The proposed patch
does that.

Can we please come to some decision? VM Hotplug is an important part of
VMD and customers are reporting VM Hotplug issues. I would like to get
some progress on this one.

Thanks
nirmal
> I think this warrants some clarification and possibly discussion in
> the PCI firmware SIG.
> 
> At the very least, the commit log should mention _OSC and say
> something about the fact that this is assuming PCIe hotplug ownership
> for devices below VMD, regardless of what the upstream _OSC said.
> 
> > Signed-off-by: Nirmal Patel <nirmal.patel@linux.intel.com>
> > ---
> > ---
> >  drivers/pci/controller/vmd.c | 11 ++++++++++-
> >  1 file changed, 10 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/pci/controller/vmd.c
> > b/drivers/pci/controller/vmd.c
> > index 769eedeb8802..e39eaef5549a 100644
> > --- a/drivers/pci/controller/vmd.c
> > +++ b/drivers/pci/controller/vmd.c
> > @@ -720,6 +720,7 @@ static int vmd_enable_domain(struct vmd_dev
> > *vmd, unsigned long features)
> >  	resource_size_t membar2_offset = 0x2000;
> >  	struct pci_bus *child;
> >  	struct pci_dev *dev;
> > +	struct pci_host_bridge *vmd_bridge;
> >  	int ret;
> >  
> >  	/*
> > @@ -886,8 +887,16 @@ static int vmd_enable_domain(struct vmd_dev
> > *vmd, unsigned long features)
> >  	 * and will fail pcie_bus_configure_settings() early. It can
> > instead be
> >  	 * run on each of the real root ports.
> >  	 */
> > -	list_for_each_entry(child, &vmd->bus->children, node)
> > +	vmd_bridge = to_pci_host_bridge(vmd->bus->bridge);
> > +	list_for_each_entry(child, &vmd->bus->children, node) {
> >  		pcie_bus_configure_settings(child);
> > +		/*
> > +		 * When Hotplug is enabled on vmd root-port, enable it
> > on vmd
> > +		 * bridge.
> > +		 */
> > +		if (child->self->is_hotplug_bridge)
> > +			vmd_bridge->native_pcie_hotplug = 1;
> > +	}
> >  
> >  	pci_bus_add_devices(vmd->bus);
> >  
> > -- 
> > 2.31.1
> >