Re: [PATCH v2] PCI: vmd: Enable Hotplug based on BIOS setting on VMD rootports

[Bjorn Helgaas <helgaas@kernel.org>]

On Tue, Jan 16, 2024 at 01:37:32PM -0700, Nirmal Patel wrote:
> On Fri, 2024-01-12 at 16:55 -0600, Bjorn Helgaas wrote:
> ...

> > I think we're converging on the idea that since VMD is effectively
> > *not* an ACPI host bridge and doesn't have its own _OSC, the _OSC
> > that applies to the VMD endpoint does *not* apply to the hierarchy
> > below the VMD.  In that case, the default is that the OS owns all
> > the features (hotplug, AER, etc) below the VMD.
>
> Well there will be few problems if VMD owns/assigns all the flags by
> itself. We discussed all of the potential problems but due to the
> holidays I think I should summarize them again.
>
> #1 : Currently there is no way to pass the information about AER,
> DPC, etc to VMD driver from BIOS or from boot parameter. For
> example, if VMD blindly enables AER and platform has AER disabled,
> then we will see AERs from devices under VMD but user have no way to
> toggle it since we rejected idea of adding boot parameter for AER,
> DPC under VMD. I believe you also didn't like the idea of sysfs knob
> suggested by Kai-Heng.
> 
> #2 : Even if we use VMD hardware register to store AER, DPC and make
> UEFI VMD driver to write information about Hotplug, DPC, AER, we
> still dont have a way to change the setting if user wants to alter
> them. Also the issue will still persist in client platforms since we
> don't own their UEFI VMD driver. It will be a huge effort.
> 
> #3 : At this moment, user can enable/disable only Hotplug in VMD
> BIOS settings (meaning no AER, DPC, LTR, etc)and VMD driver can read
> it from SltCap register. This means BIOS needs to add other settings
> and VMD needs to figure out to read them which at this moment VMD
> can't do it.
> 
> #4 : consistency with Host OS and Guest OS.
> 
> I believe the current proposed patch is the best option which
> requires minimal changes without breaking other platform features
> and unblock our customers. This issues has been a blocker for us.
> 
> For your concerns regarding how VMD can own all the _OSC features, i
> am open to other ideas and will discuss with the architect if they
> have any suggestions.

As I understand it, the basic model of PCIe features is:

  - If platform doesn't have a way to negotiate ownership of PCIe
    features, the OS owns them by default, e.g., on non-ACPI systems.

  - If platform does have a way to negotiate, e.g., ACPI _OSC, the
    platform owns the features until it grants ownership to the OS.

  - If the OS has ownership (either by default or granted by
    platform), it can use the feature if the hardware device
    advertises support.

I think this model applies to all PCIe features, including hotplug,
AER, DPC, etc., and the OS uses _OSC and the Capabilities in device
config space to determine whether to use the features.

_OSC is the obvious way for a platform to use BIOS settings to
influence what the OS does.  I think the problem with VMD is that you
have a guest OS running on a platform (the guest VM) where you want a
host BIOS setting to control things in that guest platform, but
there's currently no way to influence the _OSC seen by the guest.

I think we need to:

  - Clarify whether _OSC only applies in the domain of the host bridge
    where it appears, i.e., an _OSC in a host bridge to domain 0000
    obviously applies to a VMD Endpoint in domain 0000; does it also
    apply to devices in the domain 10000 *below* the VMD Endpoint?

  - Describe what the VMD bridge does with events below it.  E.g.,
    normally a Root Port that receives an error Message generates an
    interrupt depending on its Interrupt Disable and Error Reporting
    Enable bits.  What happens when it's a VMD Root Port?  Does it
    forward an error Message up via the VMD Endpoint?  Generate an
    interrupt via the VMD Endpoint?  If so, which interrupt?

The question of where _OSC applies feels like an architectural thing.

The question of how AER, DPC, hotplug, etc. events are forwarded
across the VMD Root Port/Endpoint might be, too, or maybe that's all
driver-specific, I dunno.  Either way, I think it's worth documenting
somehow.

Bjorn