[PATCH] PCI: Disable PTM on Upstream Ports during suspend

[Bjorn Helgaas <helgaas@kernel.org>]

From: Bjorn Helgaas <bhelgaas@google.com>

During suspend, we want to disable PTM on Root Ports because that allows
some chips, e.g., Intel mobile chips since Coffee Lake, to enter a
lower-power PM state.

Previously we only disabled PTM for Root Ports, but PCIe r6.0, sec 2.2.8,
strongly recommends that functions support generation of Messages in non-D0
states, so we must assume that Switch Upstream Ports or Endpoints may send
PTM Request Messages while in D1, D2, and D3hot.

A PTM Message received by a Downstream Port, e.g., a Root Port, with PTM
disabled must be treated as an Unsupported Request (sec 6.21.3).

With this topology:

  0000:00:1d.0 Root Port              to [bus 08-71]
  0000:08:00.0 Switch Upstream Port   to [bus 09-71]

Kai-Heng reported errors like this:

  pcieport 0000:00:1d.0: AER: Uncorrected (Non-Fatal) error received: 0000:00:1d.0
  pcieport 0000:00:1d.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID)
  pcieport 0000:00:1d.0:   device [8086:7ab0] error status/mask=00100000/00004000
  pcieport 0000:00:1d.0:    [20] UnsupReq               (First)
  pcieport 0000:00:1d.0: AER:   TLP Header: 34000000 08000052 00000000 00000000

Decoding this (from PCIe r6.0, sec 2.2.1.1, 2.2.8.10) shows that 00:1d.0
logged an Unsupported Request error when it received a PTM Request with
Requester ID 08:00.0.

To prevent this error, disable PTM when suspending Upstream Ports
(including those on Endpoints), not just Root Ports.

Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=215453
Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=216210
Based-on: https://lore.kernel.org/r/20220706123244.18056-1-kai.heng.feng@canonical.com
Based-on-patch-by: Kai-Heng Feng <kai.heng.feng@canonical.com>
Reported-by: Kai-Heng Feng <kai.heng.feng@canonical.com>
Tested-by: Kai-Heng Feng <kai.heng.feng@canonical.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Cc: David E. Box <david.e.box@linux.intel.com>
Cc: Sathyanarayanan Kuppuswamy <sathyanarayanan.kuppuswamy@linux.intel.com>
---
 drivers/pci/pci.c      | 30 ++++++++++++++----------------
 drivers/pci/pcie/ptm.c |  8 +++++++-
 2 files changed, 21 insertions(+), 17 deletions(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 95bc329e74c0..96487a9ce5bf 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -2707,14 +2707,19 @@ int pci_prepare_to_sleep(struct pci_dev *dev)
 		return -EIO;
 
 	/*
-	 * There are systems (for example, Intel mobile chips since Coffee
-	 * Lake) where the power drawn while suspended can be significantly
-	 * reduced by disabling PTM on PCIe root ports as this allows the
-	 * port to enter a lower-power PM state and the SoC to reach a
-	 * lower-power idle state as a whole.
+	 * We want to disable PTM on Root Ports because that allows some
+	 * chips, e.g., Intel mobile chips since Coffee Lake, to enter a
+	 * lower-power PM state.
+	 *
+	 * PCIe r6.0, sec 2.2.8, strongly recommends that functions support
+	 * generation of messages in non-D0 states, so we assume Switch
+	 * Upstream Ports or Endpoints may send PTM Requests while in D1,
+	 * D2, and D3hot.  A PTM message received by a Downstream Port
+	 * (including a Root Port) with PTM disabled must be treated as an
+	 * Unsupported Request (sec 6.21.3).  To prevent this error,
+	 * disable PTM in *all* devices, not just Root Ports.
 	 */
-	if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT)
-		pci_disable_ptm(dev);
+	pci_disable_ptm(dev);
 
 	pci_enable_wake(dev, target_state, wakeup);
 
@@ -2764,15 +2769,8 @@ int pci_finish_runtime_suspend(struct pci_dev *dev)
 	if (target_state == PCI_POWER_ERROR)
 		return -EIO;
 
-	/*
-	 * There are systems (for example, Intel mobile chips since Coffee
-	 * Lake) where the power drawn while suspended can be significantly
-	 * reduced by disabling PTM on PCIe root ports as this allows the
-	 * port to enter a lower-power PM state and the SoC to reach a
-	 * lower-power idle state as a whole.
-	 */
-	if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT)
-		pci_disable_ptm(dev);
+	/* See rationale above for disabling PTM */
+	pci_disable_ptm(dev);
 
 	__pci_enable_wake(dev, target_state, pci_dev_run_wake(dev));
 
diff --git a/drivers/pci/pcie/ptm.c b/drivers/pci/pcie/ptm.c
index 368a254e3124..ec338470d13f 100644
--- a/drivers/pci/pcie/ptm.c
+++ b/drivers/pci/pcie/ptm.c
@@ -31,12 +31,18 @@ static void pci_ptm_info(struct pci_dev *dev)
 
 void pci_disable_ptm(struct pci_dev *dev)
 {
-	int ptm;
+	int type, ptm;
 	u16 ctrl;
 
 	if (!pci_is_pcie(dev))
 		return;
 
+	type = pci_pcie_type(dev);
+	if (!(type == PCI_EXP_TYPE_ROOT_PORT ||
+	      type == PCI_EXP_TYPE_UPSTREAM ||
+	      type == PCI_EXP_TYPE_ENDPOINT))
+		return;
+
 	ptm = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_PTM);
 	if (!ptm)
 		return;
-- 
2.25.1