Re: [PATCHv3 2/2] x86/vmd: Add PCI domain specific LED option

[Bjorn Helgaas <helgaas@kernel.org>]

[+cc Nirmal, Jim, Paul, Blazej]

On Tue, Sep 13, 2016 at 09:05:40AM -0600, Keith Busch wrote:
> This patch adds a new function to set PCI domain specific options as
> devices are added. The usage included in this patch is for LED indicator
> control in VMD domains, but may be extended in the future as new domain
> specific options are required.
> 
> PCIe LED Slot Control in a VMD domain is repurposed to a non-standard
> implementation. As such, all devices in a VMD domain will be flagged so
> pciehp does not attempt to use LED indicators. This user_led flag
> has pciehp provide a different sysfs entry for user exclusive control
> over the domain's slot indicators.
> 
> In order to determine if a bus is within a PCI domain, the patch appends
> a bool to the pci_sysdata structure that the VMD driver sets during
> initialization.

This eventually turned into https://git.kernel.org/linus/3161832d58c7
("x86/PCI: VMD: Request userspace control of PCIe hotplug indicators")

More questions about this, prompted by Blazej's recent regression
report:
https://lore.kernel.org/r/20240722141440.7210-1-blazej.kucman@intel.com

I assume this patch was prompted by NVMe devices behind a VMD?  And
the non-standard slot indicator usage is specifically related to VMD
Root Ports?  Isn't it possible to add non-NVMe devices behind VMD,
e.g., a switch in an external enclosure where pciehp manages a switch
Downstream Port with standard slot indicators?

I'm wondering if pdev->hotplug_user_indicators should be more narrowly
targeted to just VMD Root Ports.

If there's any possibility of a Downstream Port behind VMD with
standard indicators, users are going to be very confused when the
sysfs "attention" file is basically backwards from normal.  IIUC
writing 0 to "attention" when hotplug_user_indicators is set writes 0
("reserved") to AIC, when it would otherwise write 11b ("off").

I'm also wondering whether there's a way to do this in the vmd driver
instead of in arch/x86/pci/common.c, but that's a secondary question.

>  arch/x86/include/asm/pci.h | 14 ++++++++++++++
>  arch/x86/pci/common.c      |  7 +++++++
>  arch/x86/pci/vmd.c         |  1 +
>  3 files changed, 22 insertions(+)
> 
> diff --git a/arch/x86/include/asm/pci.h b/arch/x86/include/asm/pci.h
> index 9ab7507..1411dbe 100644
> --- a/arch/x86/include/asm/pci.h
> +++ b/arch/x86/include/asm/pci.h
> @@ -23,6 +23,9 @@ struct pci_sysdata {
>  #ifdef CONFIG_PCI_MSI_IRQ_DOMAIN
>  	void		*fwnode;	/* IRQ domain for MSI assignment */
>  #endif
> +#if IS_ENABLED(CONFIG_VMD)
> +	bool vmd_domain;		/* True if in Intel VMD domain */
> +#endif
>  };
>  
>  extern int pci_routeirq;
> @@ -56,6 +59,17 @@ static inline void *_pci_root_bus_fwnode(struct pci_bus *bus)
>  #define pci_root_bus_fwnode	_pci_root_bus_fwnode
>  #endif
>  
> +static inline bool is_vmd(struct pci_bus *bus)
> +{
> +#if IS_ENABLED(CONFIG_VMD)
> +	struct pci_sysdata *sd = bus->sysdata;
> +
> +	return sd->vmd_domain;
> +#else
> +	return false;
> +#endif
> +}
> +
>  /* Can be used to override the logic in pci_scan_bus for skipping
>     already-configured bus numbers - to be used for buggy BIOSes
>     or architectures with incomplete PCI setup by the loader */
> diff --git a/arch/x86/pci/common.c b/arch/x86/pci/common.c
> index 7b6a9d1..ccf696c 100644
> --- a/arch/x86/pci/common.c
> +++ b/arch/x86/pci/common.c
> @@ -677,6 +677,12 @@ static void set_dma_domain_ops(struct pci_dev *pdev)
>  static void set_dma_domain_ops(struct pci_dev *pdev) {}
>  #endif
>  
> +static void set_dev_domain_options(struct pci_dev *pdev)
> +{
> +	if (is_vmd(pdev->bus))
> +		pdev->user_leds = 1;
> +}
> +
>  int pcibios_add_device(struct pci_dev *dev)
>  {
>  	struct setup_data *data;
> @@ -707,6 +713,7 @@ int pcibios_add_device(struct pci_dev *dev)
>  		iounmap(data);
>  	}
>  	set_dma_domain_ops(dev);
> +	set_dev_domain_options(dev);
>  	return 0;
>  }
>  
> diff --git a/arch/x86/pci/vmd.c b/arch/x86/pci/vmd.c
> index b814ca6..a021b7b 100644
> --- a/arch/x86/pci/vmd.c
> +++ b/arch/x86/pci/vmd.c
> @@ -596,6 +596,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd)
>  		.parent = res,
>  	};
>  
> +	sd->vmd_domain = true;
>  	sd->domain = vmd_find_free_domain();
>  	if (sd->domain < 0)
>  		return sd->domain;
> -- 
> 2.7.2