[PATCH v2 0/2] PCI: device link quirk for NVIDIA GPU

[PATCH v2 0/2] PCI: device link quirk for NVIDIA GPU

[Abhishek Sahu]

* v2:

1. Make the pci device link helper function generic which can be
   used for other multi-function PCI devices also.
2. Minor changes in comments and commit logs.

* v1:

NVIDIA Turing GPU [1] has hardware support for USB Type-C and
VirtualLink [2]. The Turing GPU is a multi-function PCI device
which has the following four functions:

	- VGA display controller (Function 0)
	- Audio controller (Function 1)
	- USB xHCI Host controller (Function 2)
	- USB Type-C USCI controller (Function 3)

Currently NVIDIA and Nouveau GPU drivers only manage function 0.
Rest of the functions are managed by other drivers. These functions
internally in the hardware are tightly coupled. When function 0 goes
in runtime suspended state, then it will do power gating for most of
the hardware blocks. Some of these hardware blocks are used by
the other PCI functions, which leads to functional failure. In the
mainline kernel, the device link is present between
function 0 and function 1.  This patch series deals with creating
a similar kind of device link between function 0 and
functions 2 and 3.

[1] https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/technologies/turing-architecture/NVIDIA-Turing-Architecture-Whitepaper.pdf
[2] https://en.wikipedia.org/wiki/VirtualLink

Abhishek Sahu (2):
  PCI: Code reorganization for creating device link
  PCI: Create device link for NVIDIA GPU

 drivers/pci/quirks.c | 79 +++++++++++++++++++++++++++++++++-----------
 1 file changed, 59 insertions(+), 20 deletions(-)

-- 
2.17.1

[PATCH v2 1/2] PCI: Code reorganization for creating device link

[Abhishek Sahu]

In Multi-function PCI device, one function (consumer) can have
hardware functional dependencies on another function (supplier).
Whenever the consumer is active and in D0 state, the supplier should
also be in D0 state. Currently, the device link is being created from
HDA function to VGA function for GPU's. This patch does minor code
reorganization. It introduces a helper function which creates device
link from consumer pci device to supplier pci device and uses this
helper function for creating device link from HDA to VGA. This helper
function can be used in future for creating device link from one
function to another function.

Signed-off-by: Abhishek Sahu <abhsahu@nvidia.com>
---
* Changes from v1:

  1. Make the helper function generic which takes supplier class,
     class shift and function number also.
  2. Minor changes in commit log

 drivers/pci/quirks.c | 53 +++++++++++++++++++++++++++-----------------
 1 file changed, 33 insertions(+), 20 deletions(-)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index a077f67fe1da..379cd7fbcb12 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -4916,35 +4916,48 @@ static void quirk_fsl_no_msi(struct pci_dev *pdev)
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_FREESCALE, PCI_ANY_ID, quirk_fsl_no_msi);
 
 /*
- * GPUs with integrated HDA controller for streaming audio to attached displays
- * need a device link from the HDA controller (consumer) to the GPU (supplier)
- * so that the GPU is powered up whenever the HDA controller is accessed.
- * The GPU and HDA controller are functions 0 and 1 of the same PCI device.
- * The device link stays in place until shutdown (or removal of the PCI device
- * if it's hotplugged).  Runtime PM is allowed by default on the HDA controller
- * to prevent it from permanently keeping the GPU awake.
+ * Multi-function PCI devices can have hardware functional dependencies from
+ * one function (consumer) to another function (supplier). Whenever the
+ * consumer is in D0 state, the supplier should also be in D0 state. This is
+ * a helper function which creates device link from the consumer to the
+ * supplier.  The device link stays in place until shutdown (or removal of
+ * the PCI device if it's hotplugged). Runtime PM is allowed by default on
+ * consumers to prevent it from permanently keeping the supplier awake.
  */
-static void quirk_gpu_hda(struct pci_dev *hda)
+static void pci_create_device_link(struct pci_dev *pdev, unsigned int consumer,
+				   unsigned int supplier, unsigned int class,
+				   unsigned int class_shift)
 {
-	struct pci_dev *gpu;
+	struct pci_dev *supplier_pdev;
 
-	if (PCI_FUNC(hda->devfn) != 1)
+	if (PCI_FUNC(pdev->devfn) != consumer)
 		return;
 
-	gpu = pci_get_domain_bus_and_slot(pci_domain_nr(hda->bus),
-					  hda->bus->number,
-					  PCI_DEVFN(PCI_SLOT(hda->devfn), 0));
-	if (!gpu || (gpu->class >> 16) != PCI_BASE_CLASS_DISPLAY) {
-		pci_dev_put(gpu);
+	supplier_pdev = pci_get_domain_bus_and_slot(pci_domain_nr(pdev->bus),
+				pdev->bus->number,
+				PCI_DEVFN(PCI_SLOT(pdev->devfn), supplier));
+	if (!supplier_pdev || (supplier_pdev->class >> class_shift) != class) {
+		pci_dev_put(supplier_pdev);
 		return;
 	}
 
-	if (!device_link_add(&hda->dev, &gpu->dev,
-			     DL_FLAG_STATELESS | DL_FLAG_PM_RUNTIME))
-		pci_err(hda, "cannot link HDA to GPU %s\n", pci_name(gpu));
+	if (device_link_add(&pdev->dev, &supplier_pdev->dev,
+			    DL_FLAG_STATELESS | DL_FLAG_PM_RUNTIME))
+		pci_info(pdev, "Linked with %s\n", pci_name(supplier_pdev));
+	else
+		pci_err(pdev, "Cannot link with %s\n", pci_name(supplier_pdev));
+
+	pm_runtime_allow(&pdev->dev);
+	pci_dev_put(supplier_pdev);
+}
 
-	pm_runtime_allow(&hda->dev);
-	pci_dev_put(gpu);
+/*
+ * Create device link for GPUs with integrated HDA controller for streaming
+ * audio to attached displays.
+ */
+static void quirk_gpu_hda(struct pci_dev *hda)
+{
+	pci_create_device_link(hda, 1, 0, PCI_BASE_CLASS_DISPLAY, 16);
 }
 DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_ATI, PCI_ANY_ID,
 			      PCI_CLASS_MULTIMEDIA_HD_AUDIO, 8, quirk_gpu_hda);
-- 
2.17.1

[PATCH v2 2/2] PCI: Create device link for NVIDIA GPU

[Abhishek Sahu]

NVIDIA Turing GPUs include hardware support for USB Type-C and
VirtualLink. It helps in delivering the power, display, and data
required to power VR headsets through a single USB Type-C connector.
The Turing GPU is a multi-function PCI device. It has the following
four functions:

	- VGA display controller (Function 0)
	- Audio controller (Function 1)
	- USB xHCI Host controller (Function 2)
	- USB Type-C USCI controller (Function 3)

The function 0 is tightly coupled with other functions in the
hardware. When function 0 goes in D3 state, then it will do
power gating for most of the hardware blocks. Some of these
hardware blocks are being used by other functions which
leads to functional failure. So if any of these functions (1/2/3)
are in D0 state, then function 0 should also be in D0 state.

'commit 07f4f97d7b4b ("vga_switcheroo: Use device link for
HDA controller")' creates the device link from function 1 to
function 0. A similar kind of device link needs to be created
between function 0 and functions 2 and 3 for NVIDIA Turing GPU.

This patch does the same and creates the required device links. It
will make function 0 to be D0 state if any other function is in D0
state.

Signed-off-by: Abhishek Sahu <abhsahu@nvidia.com>
---
* Changes from v1:

  1. Minor changes in commit log
  2. used pci_create_device_link() helper function

 drivers/pci/quirks.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 379cd7fbcb12..b9182c4e5e42 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -4966,6 +4966,32 @@ DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_AMD, PCI_ANY_ID,
 DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,
 			      PCI_CLASS_MULTIMEDIA_HD_AUDIO, 8, quirk_gpu_hda);
 
+/*
+ * Create device link for NVIDIA GPU with integrated USB xHCI Host
+ * controller to VGA.
+ */
+static void quirk_gpu_usb(struct pci_dev *usb)
+{
+	pci_create_device_link(usb, 2, 0, PCI_BASE_CLASS_DISPLAY, 16);
+}
+DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,
+			      PCI_CLASS_SERIAL_USB, 8, quirk_gpu_usb);
+
+/*
+ * Create device link for NVIDIA GPU with integrated Type-C UCSI controller
+ * to VGA. Currently there is no class code defined for UCSI device over PCI
+ * so using UNKNOWN class for now and it will be updated when UCSI
+ * over PCI gets a class code.
+ */
+#define PCI_CLASS_SERIAL_UNKNOWN	0x0c80
+static void quirk_gpu_usb_typec_ucsi(struct pci_dev *ucsi)
+{
+	pci_create_device_link(ucsi, 3, 0, PCI_BASE_CLASS_DISPLAY, 16);
+}
+DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,
+			      PCI_CLASS_SERIAL_UNKNOWN, 8,
+			      quirk_gpu_usb_typec_ucsi);
+
 /*
  * Some IDT switches incorrectly flag an ACS Source Validation error on
  * completions for config read requests even though PCIe r4.0, sec
-- 
2.17.1

Re: [PATCH v2 0/2] PCI: device link quirk for NVIDIA GPU

[Bjorn Helgaas]

On Thu, Jun 06, 2019 at 02:52:23PM +0530, Abhishek Sahu wrote:
> * v2:
> 
> 1. Make the pci device link helper function generic which can be
>    used for other multi-function PCI devices also.
> 2. Minor changes in comments and commit logs.
> 
> * v1:
> 
> NVIDIA Turing GPU [1] has hardware support for USB Type-C and
> VirtualLink [2]. The Turing GPU is a multi-function PCI device
> which has the following four functions:
> 
> 	- VGA display controller (Function 0)
> 	- Audio controller (Function 1)
> 	- USB xHCI Host controller (Function 2)
> 	- USB Type-C USCI controller (Function 3)
> 
> Currently NVIDIA and Nouveau GPU drivers only manage function 0.
> Rest of the functions are managed by other drivers. These functions
> internally in the hardware are tightly coupled. When function 0 goes
> in runtime suspended state, then it will do power gating for most of
> the hardware blocks. Some of these hardware blocks are used by
> the other PCI functions, which leads to functional failure. In the
> mainline kernel, the device link is present between
> function 0 and function 1.  This patch series deals with creating
> a similar kind of device link between function 0 and
> functions 2 and 3.
> 
> [1] https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/technologies/turing-architecture/NVIDIA-Turing-Architecture-Whitepaper.pdf
> [2] https://en.wikipedia.org/wiki/VirtualLink
> 
> Abhishek Sahu (2):
>   PCI: Code reorganization for creating device link
>   PCI: Create device link for NVIDIA GPU

Applied to pci/misc for v5.3, thanks!

Re: [PATCH v2 0/2] PCI: device link quirk for NVIDIA GPU

[Abhishek Sahu]

On 6/14/2019 2:27 AM, Bjorn Helgaas wrote:
> On Thu, Jun 06, 2019 at 02:52:23PM +0530, Abhishek Sahu wrote:
>> * v2:
>>
>> 1. Make the pci device link helper function generic which can be
>>    used for other multi-function PCI devices also.
>> 2. Minor changes in comments and commit logs.
>>
>> * v1:
>>
>> NVIDIA Turing GPU [1] has hardware support for USB Type-C and
>> VirtualLink [2]. The Turing GPU is a multi-function PCI device
>> which has the following four functions:
>>
>> 	- VGA display controller (Function 0)
>> 	- Audio controller (Function 1)
>> 	- USB xHCI Host controller (Function 2)
>> 	- USB Type-C USCI controller (Function 3)
>>
>> Currently NVIDIA and Nouveau GPU drivers only manage function 0.
>> Rest of the functions are managed by other drivers. These functions
>> internally in the hardware are tightly coupled. When function 0 goes
>> in runtime suspended state, then it will do power gating for most of
>> the hardware blocks. Some of these hardware blocks are used by
>> the other PCI functions, which leads to functional failure. In the
>> mainline kernel, the device link is present between
>> function 0 and function 1.  This patch series deals with creating
>> a similar kind of device link between function 0 and
>> functions 2 and 3.
>>
>> [1] https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/technologies/turing-architecture/NVIDIA-Turing-Architecture-Whitepaper.pdf
>> [2] https://en.wikipedia.org/wiki/VirtualLink
>>
>> Abhishek Sahu (2):
>>   PCI: Code reorganization for creating device link
>>   PCI: Create device link for NVIDIA GPU
> 
> Applied to pci/misc for v5.3, thanks!

 Thanks Bjorn for your review and support!

 The runtime PM changes in USB Type-C USCI driver is also
 applied for v5.3

 https://marc.info/?l=linux-usb&m=155994544705901&w=2

 It will help in achieving run-time PM for Turing GPUs
 in v5.3.

 Regards,
 Abhishek