[PATCH v7 0/3] PCI: brcmstb: Configure appropriate HW CLKREQ# mode

[PATCH v7 0/3] PCI: brcmstb: Configure appropriate HW CLKREQ# mode

[Jim Quinlan]

[-- Attachment #1: Type: text/plain, Size: 3844 bytes --]

v7 -- Manivannan Sadhasivam suggested (a) making the property look like a
      network phy-mode and (b) keeping the code simple (not counting clkreq
      signal appearances, un-advertising capabilites, etc).  This is
      what I have done.  The property is now "brcm,clkreq-mode" and
      the values may be one of "safe", "default", and "no-l1ss".  The
      default setting is to employ the most capable power savings mode.

v6 -- No code has been changed.
   -- Changed commit subject and comment in "#PERST" commit (Bjorn, Cyril)
   -- Changed sign-off and author email address for all commits.
      This was due to a change in Broadcom's upstreaming policy.

v5 -- Remove DT property "brcm,completion-timeout-us" from	 
      "DT bindings" commit.  Although this error may be reported	 
      as a completion timeout, its cause was traced to an	 
      internal bus timeout which may occur even when there is	 
      no PCIe access being processed.  We set a timeout of four	 
      seconds only if we are operating in "L1SS CLKREQ#" mode.
   -- Correct CEM 2.0 reference provided by HW engineer,
      s/3.2.5.2.5/3.2.5.2.2/ (Bjorn)
   -- Add newline to dev_info() string (Stefan)
   -- Change variable rval to unsigned (Stefan)
   -- s/implementaion/implementation/ (Bjorn)
   -- s/superpowersave/powersupersave/ (Bjorn)
   -- Slightly modify message on "PERST#" commit.
   -- Rebase to torvalds master

v4 -- New commit that asserts PERST# for 2711/RPi SOCs at PCIe RC
      driver probe() time.  This is done in Raspian Linux and its
      absence may be the cause of a failing test case.
   -- New commit that removes stale comment.

v3 -- Rewrote commit msgs and comments refering panics if L1SS
      is enabled/disabled; the code snippet that unadvertises L1SS
      eliminates the panic scenario. (Bjorn)
   -- Add reference for "400ns of CLKREQ# assertion" blurb (Bjorn)
   -- Put binding names in DT commit Subject (Bjorn)
   -- Add a verb to a commit's subject line (Bjorn)
   -- s/accomodat(\w+)/accommodat$1/g (Bjorn)
   -- Rewrote commit msgs and comments refering panics if L1SS
      is enabled/disabled; the code snippet that unadvertises L1SS
      eliminates the panic scenario. (Bjorn)

v2 -- Changed binding property 'brcm,completion-timeout-msec' to
      'brcm,completion-timeout-us'.  (StefanW for standard suffix).
   -- Warn when clamping timeout value, and include clamped
      region in message. Also add min and max in YAML. (StefanW)
   -- Qualify description of "brcm,completion-timeout-us" so that
      it refers to PCIe transactions. (StefanW)
   -- Remvove mention of Linux specifics in binding description. (StefanW)
   -- s/clkreq#/CLKREQ#/g (Bjorn)
   -- Refactor completion-timeout-us code to compare max and min to
      value given by the property (as opposed to the computed value).

v1 -- The current driver assumes the downstream devices can
      provide CLKREQ# for ASPM.  These commits accomodate devices
      w/ or w/o clkreq# and also handle L1SS-capable devices.

   -- The Raspian Linux folks have already been using a PCIe RC
      property "brcm,enable-l1ss".  These commits use the same
      property, in a backward-compatible manner, and the implementaion
      adds more detail and also automatically identifies devices w/o
      a clkreq# signal, i.e. most devices plugged into an RPi CM4
      IO board.


Jim Quinlan (3):
  dt-bindings: PCI: brcmstb: Add property "brcm,clkreq-mode"
  PCI: brcmstb: Configure HW CLKREQ# mode appropriate for downstream
    device
  PCI: brcmstb: Set higher value for internal bus timeout

 .../bindings/pci/brcm,stb-pcie.yaml           | 21 +++++
 drivers/pci/controller/pcie-brcmstb.c         | 81 ++++++++++++++++---
 2 files changed, 92 insertions(+), 10 deletions(-)


base-commit: 305230142ae0637213bf6e04f6d9f10bbcb74af8
-- 
2.17.1


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4210 bytes --]

[PATCH v7 1/3] dt-bindings: PCI: brcmstb: Add property "brcm,clkreq-mode"

[Jim Quinlan]

[-- Attachment #1: Type: text/plain, Size: 2061 bytes --]

The Broadcom STB/CM PCIe HW -- a core that is also used by RPi SOCs --
requires the driver to deliberately place the RC HW one of three CLKREQ#
modes.  The "brcm,clkreq-mode" property allows the user to override the
default setting.  If this property is omitted, the default mode shall be
"default".

Signed-off-by: Jim Quinlan <james.quinlan@broadcom.com>
---
 .../bindings/pci/brcm,stb-pcie.yaml           | 21 +++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml b/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
index 7e15aae7d69e..992b35e915a5 100644
--- a/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
+++ b/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
@@ -64,6 +64,27 @@ properties:
 
   aspm-no-l0s: true
 
+  brcm,clkreq-mode:
+    description: A string that determines the operating
+      clkreq mode of the PCIe RC HW WRT controlling the refclk signal.
+      There are three different modes --
+      "safe", which drives the
+      refclk signal unconditionally and will work for all devices but does
+      not provide any power savings;
+      "no-l1ss" -- which provides Clock Power Management, L0s, and
+      L1, but cannot provide L1 substate (L1SS) power
+      savings. If the downstream device connected to the RC is
+      L1SS capable AND the OS enables L1SS, all PCIe traffic
+      may abruptly halt, potentially hanging the system;
+      "default" -- which provides L0s, L1, and L1SS, but not
+      compliant to provide Clock Power Management;
+      specifically, may not be able to meet the Tclron max
+      timing of 400ns as specified in "Dynamic Clock Control",
+      section 3.2.5.2.2 of the PCIe spec.  This situation is
+      atypical and should happen only with older devices.
+    $ref: /schemas/types.yaml#/definitions/string
+    enum: [ safe, no-l1ss, default ]
+
   brcm,scb-sizes:
     description: u64 giving the 64bit PCIe memory
       viewport size of a memory controller.  There may be up to
-- 
2.17.1


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4210 bytes --]

[PATCH v7 2/3] PCI: brcmstb: Configure HW CLKREQ# mode appropriate for downstream device

[Jim Quinlan]

[-- Attachment #1: Type: text/plain, Size: 5375 bytes --]

The Broadcom STB/CM PCIe HW core, which is also used in RPi SOCs, must be
deliberately set by the PCIe RC HW into one of three mutually exclusive
modes:

"safe" -- No CLKREQ# expected or required, refclk is always provided.  This
    mode should work for all devices but is not be capable of any refclk
    power savings.

"no-l1ss" -- CLKREQ# is expected to be driven by the downstream device for
    CPM and ASPM L0s and L1.  Provides Clock Power Management, L0s, and L1,
    but cannot provide L1 substate (L1SS) power savings. If the downstream
    device connected to the RC is L1SS capable AND the OS enables L1SS, all
    PCIe traffic may abruptly halt, potentially hanging the system.

"default" -- Bidirectional CLKREQ# between the RC and downstream device.
    Provides ASPM L0s, L1, and L1SS, but not compliant to provide Clock
    Power Management; specifically, may not be able to meet the Tclron max
    timing of 400ns as specified in "Dynamic Clock Control", section
    3.2.5.2.2 of the PCIe spec.  This situation is atypical and should
    happen only with older devices.

Previously, this driver always set the mode to "no-l1ss", as almost all
STB/CM boards operate in this mode.  But now there is interest in
activating L1SS power savings from STB/CM customers, which requires "aspm"
mode.  In addition, a bug was filed for RPi4 CM platform because most
devices did not work in "no-l1ss" mode.

Note that the mode is specified by the DT property "brcm,clkreq-mode".  If
this property is omitted, then "default" mode is chosen.

Link: https://bugzilla.kernel.org/show_bug.cgi?id=217276

Signed-off-by: Jim Quinlan <james.quinlan@broadcom.com>
---
 drivers/pci/controller/pcie-brcmstb.c | 65 ++++++++++++++++++++++-----
 1 file changed, 55 insertions(+), 10 deletions(-)

diff --git a/drivers/pci/controller/pcie-brcmstb.c b/drivers/pci/controller/pcie-brcmstb.c
index f9dd6622fe10..f45c5d0168d3 100644
--- a/drivers/pci/controller/pcie-brcmstb.c
+++ b/drivers/pci/controller/pcie-brcmstb.c
@@ -121,9 +121,12 @@
 
 #define PCIE_MISC_HARD_PCIE_HARD_DEBUG					0x4204
 #define  PCIE_MISC_HARD_PCIE_HARD_DEBUG_CLKREQ_DEBUG_ENABLE_MASK	0x2
+#define  PCIE_MISC_HARD_PCIE_HARD_DEBUG_L1SS_ENABLE_MASK		0x200000
 #define  PCIE_MISC_HARD_PCIE_HARD_DEBUG_SERDES_IDDQ_MASK		0x08000000
 #define  PCIE_BMIPS_MISC_HARD_PCIE_HARD_DEBUG_SERDES_IDDQ_MASK		0x00800000
-
+#define  PCIE_CLKREQ_MASK \
+	  (PCIE_MISC_HARD_PCIE_HARD_DEBUG_CLKREQ_DEBUG_ENABLE_MASK | \
+	   PCIE_MISC_HARD_PCIE_HARD_DEBUG_L1SS_ENABLE_MASK)
 
 #define PCIE_INTR2_CPU_BASE		0x4300
 #define PCIE_MSI_INTR2_BASE		0x4500
@@ -1028,13 +1031,61 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
 	return 0;
 }
 
+static void brcm_config_clkreq(struct brcm_pcie *pcie)
+{
+	static const char err_msg[] = "invalid 'brcm,clkreq-mode' DT string\n";
+	const char *mode = "default";
+	u32 clkreq_cntl;
+	int ret;
+
+	ret = of_property_read_string(pcie->np, "brcm,clkreq-mode", &mode);
+	if (ret && ret != -EINVAL) {
+		dev_err(pcie->dev, err_msg);
+		mode = "safe";
+	}
+
+	/* Start out assuming safe mode (both mode bits cleared) */
+	clkreq_cntl = readl(pcie->base + PCIE_MISC_HARD_PCIE_HARD_DEBUG);
+	clkreq_cntl &= ~PCIE_CLKREQ_MASK;
+
+	if (strcmp(mode, "no-l1ss") == 0) {
+		/*
+		 * "no-l1ss" -- Provides Clock Power Management, L0s, and
+		 * L1, but cannot provide L1 substate (L1SS) power
+		 * savings. If the downstream device connected to the RC is
+		 * L1SS capable AND the OS enables L1SS, all PCIe traffic
+		 * may abruptly halt, potentially hanging the system.
+		 */
+		clkreq_cntl |= PCIE_MISC_HARD_PCIE_HARD_DEBUG_CLKREQ_DEBUG_ENABLE_MASK;
+	} else if (strcmp(mode, "default") == 0) {
+		/*
+		 * "default" -- Provides L0s, L1, and L1SS, but not
+		 * compliant to provide Clock Power Management;
+		 * specifically, may not be able to meet the Tclron max
+		 * timing of 400ns as specified in "Dynamic Clock Control",
+		 * section 3.2.5.2.2 of the PCIe spec.  This situation is
+		 * atypical and should happen only with older devices.
+		 */
+		clkreq_cntl |= PCIE_MISC_HARD_PCIE_HARD_DEBUG_L1SS_ENABLE_MASK;
+	} else {
+		/*
+		 * "safe" -- No power savings; refclk is driven by RC
+		 * unconditionally.
+		 */
+		if (strcmp(mode, "safe") != 0)
+			dev_err(pcie->dev, err_msg);
+		mode = "safe";
+	}
+	writel(clkreq_cntl, pcie->base + PCIE_MISC_HARD_PCIE_HARD_DEBUG);
+	dev_info(pcie->dev, "clkreq-mode set to %s\n", mode);
+}
+
 static int brcm_pcie_start_link(struct brcm_pcie *pcie)
 {
 	struct device *dev = pcie->dev;
 	void __iomem *base = pcie->base;
 	u16 nlw, cls, lnksta;
 	bool ssc_good = false;
-	u32 tmp;
 	int ret, i;
 
 	/* Unassert the fundamental reset */
@@ -1059,6 +1110,8 @@ static int brcm_pcie_start_link(struct brcm_pcie *pcie)
 		return -ENODEV;
 	}
 
+	brcm_config_clkreq(pcie);
+
 	if (pcie->gen)
 		brcm_pcie_set_gen(pcie, pcie->gen);
 
@@ -1077,14 +1130,6 @@ static int brcm_pcie_start_link(struct brcm_pcie *pcie)
 		 pci_speed_string(pcie_link_speed[cls]), nlw,
 		 ssc_good ? "(SSC)" : "(!SSC)");
 
-	/*
-	 * Refclk from RC should be gated with CLKREQ# input when ASPM L0s,L1
-	 * is enabled => setting the CLKREQ_DEBUG_ENABLE field to 1.
-	 */
-	tmp = readl(base + PCIE_MISC_HARD_PCIE_HARD_DEBUG);
-	tmp |= PCIE_MISC_HARD_PCIE_HARD_DEBUG_CLKREQ_DEBUG_ENABLE_MASK;
-	writel(tmp, base + PCIE_MISC_HARD_PCIE_HARD_DEBUG);
-
 	return 0;
 }
 
-- 
2.17.1


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4210 bytes --]

[PATCH v7 3/3] PCI: brcmstb: Set higher value for internal bus timeout

[Jim Quinlan]

[-- Attachment #1: Type: text/plain, Size: 1726 bytes --]

During long periods of the PCIe RC HW being in an L1SS sleep state, there
may be a timeout on an internal bus access, even though there may not be
any PCIe access involved.  Such a timeout will cause a subsequent CPU
abort.

Signed-off-by: Jim Quinlan <james.quinlan@broadcom.com>
---
 drivers/pci/controller/pcie-brcmstb.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/pci/controller/pcie-brcmstb.c b/drivers/pci/controller/pcie-brcmstb.c
index f45c5d0168d3..f82a3e1a843a 100644
--- a/drivers/pci/controller/pcie-brcmstb.c
+++ b/drivers/pci/controller/pcie-brcmstb.c
@@ -1031,6 +1031,21 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
 	return 0;
 }
 
+/*
+ * This extends the timeout period for an access to an internal bus.  This
+ * access timeout may occur during L1SS sleep periods, even without the
+ * presence of a PCIe access.
+ */
+static void brcm_extend_rbus_timeout(struct brcm_pcie *pcie)
+{
+	/* TIMEOUT register is two registers before RGR1_SW_INIT_1 */
+	const unsigned int REG_OFFSET = PCIE_RGR1_SW_INIT_1(pcie) - 8;
+	u32 timeout_us = 4000000; /* 4 seconds, our setting for L1SS */
+
+	/* Each unit in timeout register is 1/216,000,000 seconds */
+	writel(216 * timeout_us, pcie->base + REG_OFFSET);
+}
+
 static void brcm_config_clkreq(struct brcm_pcie *pcie)
 {
 	static const char err_msg[] = "invalid 'brcm,clkreq-mode' DT string\n";
@@ -1067,6 +1082,7 @@ static void brcm_config_clkreq(struct brcm_pcie *pcie)
 		 * atypical and should happen only with older devices.
 		 */
 		clkreq_cntl |= PCIE_MISC_HARD_PCIE_HARD_DEBUG_L1SS_ENABLE_MASK;
+		brcm_extend_rbus_timeout(pcie);
 	} else {
 		/*
 		 * "safe" -- No power savings; refclk is driven by RC
-- 
2.17.1


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4210 bytes --]

Re: [PATCH v7 2/3] PCI: brcmstb: Configure HW CLKREQ# mode appropriate for downstream device

[Bjorn Helgaas]

On Thu, Nov 09, 2023 at 02:13:53PM -0500, Jim Quinlan wrote:
> The Broadcom STB/CM PCIe HW core, which is also used in RPi SOCs, must be
> deliberately set by the PCIe RC HW into one of three mutually exclusive
> modes:
> 
> "safe" -- No CLKREQ# expected or required, refclk is always provided.  This
>     mode should work for all devices but is not be capable of any refclk
>     power savings.
> 
> "no-l1ss" -- CLKREQ# is expected to be driven by the downstream device for
>     CPM and ASPM L0s and L1.  Provides Clock Power Management, L0s, and L1,
>     but cannot provide L1 substate (L1SS) power savings. If the downstream
>     device connected to the RC is L1SS capable AND the OS enables L1SS, all
>     PCIe traffic may abruptly halt, potentially hanging the system.
> 
> "default" -- Bidirectional CLKREQ# between the RC and downstream device.
>     Provides ASPM L0s, L1, and L1SS, but not compliant to provide Clock
>     Power Management; specifically, may not be able to meet the Tclron max
>     timing of 400ns as specified in "Dynamic Clock Control", section
>     3.2.5.2.2 of the PCIe spec.  This situation is atypical and should
>     happen only with older devices.

The PCIe base spec r6.0 has no section 3.2.5.2.2.  Looks like this
could be:

  PCIe Mini CEM r2.1, sec 3.2.5.2.2 (December, 2016), or
  PCIe CEM r5.1, sec 2.8.2 (August, 2023)

I don't know the relationship between the "Mini CEM" and the "CEM"
specs, but CEM r5.1 seems to have the same text as the Mini CEM r2.1
about Dynamic Clock Control.

We're hampered by the lack of clear subscripts here, but the text in
both capitalizes the "CRL" part, e.g., "T_CLRon".

> Previously, this driver always set the mode to "no-l1ss", as almost all
> STB/CM boards operate in this mode.  But now there is interest in
> activating L1SS power savings from STB/CM customers, which requires "aspm"
> mode.  In addition, a bug was filed for RPi4 CM platform because most
> devices did not work in "no-l1ss" mode.
> 
> Note that the mode is specified by the DT property "brcm,clkreq-mode".  If
> this property is omitted, then "default" mode is chosen.
> 
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=217276
> 
> Signed-off-by: Jim Quinlan <james.quinlan@broadcom.com>
> ---
>  drivers/pci/controller/pcie-brcmstb.c | 65 ++++++++++++++++++++++-----
>  1 file changed, 55 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/pci/controller/pcie-brcmstb.c b/drivers/pci/controller/pcie-brcmstb.c
> index f9dd6622fe10..f45c5d0168d3 100644
> --- a/drivers/pci/controller/pcie-brcmstb.c
> +++ b/drivers/pci/controller/pcie-brcmstb.c
> @@ -121,9 +121,12 @@
>  
>  #define PCIE_MISC_HARD_PCIE_HARD_DEBUG					0x4204
>  #define  PCIE_MISC_HARD_PCIE_HARD_DEBUG_CLKREQ_DEBUG_ENABLE_MASK	0x2
> +#define  PCIE_MISC_HARD_PCIE_HARD_DEBUG_L1SS_ENABLE_MASK		0x200000
>  #define  PCIE_MISC_HARD_PCIE_HARD_DEBUG_SERDES_IDDQ_MASK		0x08000000
>  #define  PCIE_BMIPS_MISC_HARD_PCIE_HARD_DEBUG_SERDES_IDDQ_MASK		0x00800000
> -
> +#define  PCIE_CLKREQ_MASK \
> +	  (PCIE_MISC_HARD_PCIE_HARD_DEBUG_CLKREQ_DEBUG_ENABLE_MASK | \
> +	   PCIE_MISC_HARD_PCIE_HARD_DEBUG_L1SS_ENABLE_MASK)
>  
>  #define PCIE_INTR2_CPU_BASE		0x4300
>  #define PCIE_MSI_INTR2_BASE		0x4500
> @@ -1028,13 +1031,61 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
>  	return 0;
>  }
>  
> +static void brcm_config_clkreq(struct brcm_pcie *pcie)
> +{
> +	static const char err_msg[] = "invalid 'brcm,clkreq-mode' DT string\n";
> +	const char *mode = "default";
> +	u32 clkreq_cntl;
> +	int ret;
> +
> +	ret = of_property_read_string(pcie->np, "brcm,clkreq-mode", &mode);
> +	if (ret && ret != -EINVAL) {
> +		dev_err(pcie->dev, err_msg);
> +		mode = "safe";
> +	}
> +
> +	/* Start out assuming safe mode (both mode bits cleared) */
> +	clkreq_cntl = readl(pcie->base + PCIE_MISC_HARD_PCIE_HARD_DEBUG);
> +	clkreq_cntl &= ~PCIE_CLKREQ_MASK;
> +
> +	if (strcmp(mode, "no-l1ss") == 0) {
> +		/*
> +		 * "no-l1ss" -- Provides Clock Power Management, L0s, and
> +		 * L1, but cannot provide L1 substate (L1SS) power
> +		 * savings. If the downstream device connected to the RC is
> +		 * L1SS capable AND the OS enables L1SS, all PCIe traffic
> +		 * may abruptly halt, potentially hanging the system.
> +		 */
> +		clkreq_cntl |= PCIE_MISC_HARD_PCIE_HARD_DEBUG_CLKREQ_DEBUG_ENABLE_MASK;

Does this somehow change the features advertised by the Root Port,
e.g., does it hide the L1 PM Substates Capability completely, or at
least clear the L1 PM Substates Supported bit?

It it doesn't, the PCI core may enable L1SS and cause this hang.
Every feature advertised in config space is expected to work.

> +	} else if (strcmp(mode, "default") == 0) {
> +		/*
> +		 * "default" -- Provides L0s, L1, and L1SS, but not
> +		 * compliant to provide Clock Power Management;
> +		 * specifically, may not be able to meet the Tclron max
> +		 * timing of 400ns as specified in "Dynamic Clock Control",
> +		 * section 3.2.5.2.2 of the PCIe spec.  This situation is
> +		 * atypical and should happen only with older devices.
> +		 */
> +		clkreq_cntl |= PCIE_MISC_HARD_PCIE_HARD_DEBUG_L1SS_ENABLE_MASK;
> +	} else {
> +		/*
> +		 * "safe" -- No power savings; refclk is driven by RC
> +		 * unconditionally.
> +		 */
> +		if (strcmp(mode, "safe") != 0)
> +			dev_err(pcie->dev, err_msg);
> +		mode = "safe";
> +	}
> +	writel(clkreq_cntl, pcie->base + PCIE_MISC_HARD_PCIE_HARD_DEBUG);
> +	dev_info(pcie->dev, "clkreq-mode set to %s\n", mode);
> +}
> +
>  static int brcm_pcie_start_link(struct brcm_pcie *pcie)
>  {
>  	struct device *dev = pcie->dev;
>  	void __iomem *base = pcie->base;
>  	u16 nlw, cls, lnksta;
>  	bool ssc_good = false;
> -	u32 tmp;
>  	int ret, i;
>  
>  	/* Unassert the fundamental reset */
> @@ -1059,6 +1110,8 @@ static int brcm_pcie_start_link(struct brcm_pcie *pcie)
>  		return -ENODEV;
>  	}
>  
> +	brcm_config_clkreq(pcie);
> +
>  	if (pcie->gen)
>  		brcm_pcie_set_gen(pcie, pcie->gen);
>  
> @@ -1077,14 +1130,6 @@ static int brcm_pcie_start_link(struct brcm_pcie *pcie)
>  		 pci_speed_string(pcie_link_speed[cls]), nlw,
>  		 ssc_good ? "(SSC)" : "(!SSC)");
>  
> -	/*
> -	 * Refclk from RC should be gated with CLKREQ# input when ASPM L0s,L1
> -	 * is enabled => setting the CLKREQ_DEBUG_ENABLE field to 1.
> -	 */
> -	tmp = readl(base + PCIE_MISC_HARD_PCIE_HARD_DEBUG);
> -	tmp |= PCIE_MISC_HARD_PCIE_HARD_DEBUG_CLKREQ_DEBUG_ENABLE_MASK;
> -	writel(tmp, base + PCIE_MISC_HARD_PCIE_HARD_DEBUG);
> -
>  	return 0;
>  }

Re: [PATCH v7 3/3] PCI: brcmstb: Set higher value for internal bus timeout

[Bjorn Helgaas]

On Thu, Nov 09, 2023 at 02:13:54PM -0500, Jim Quinlan wrote:
> During long periods of the PCIe RC HW being in an L1SS sleep state, there
> may be a timeout on an internal bus access, even though there may not be
> any PCIe access involved.  Such a timeout will cause a subsequent CPU
> abort.
> 
> Signed-off-by: Jim Quinlan <james.quinlan@broadcom.com>
> ---
>  drivers/pci/controller/pcie-brcmstb.c | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
> 
> diff --git a/drivers/pci/controller/pcie-brcmstb.c b/drivers/pci/controller/pcie-brcmstb.c
> index f45c5d0168d3..f82a3e1a843a 100644
> --- a/drivers/pci/controller/pcie-brcmstb.c
> +++ b/drivers/pci/controller/pcie-brcmstb.c
> @@ -1031,6 +1031,21 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
>  	return 0;
>  }
>  
> +/*
> + * This extends the timeout period for an access to an internal bus.  This
> + * access timeout may occur during L1SS sleep periods, even without the
> + * presence of a PCIe access.
> + */
> +static void brcm_extend_rbus_timeout(struct brcm_pcie *pcie)
> +{
> +	/* TIMEOUT register is two registers before RGR1_SW_INIT_1 */
> +	const unsigned int REG_OFFSET = PCIE_RGR1_SW_INIT_1(pcie) - 8;
> +	u32 timeout_us = 4000000; /* 4 seconds, our setting for L1SS */
> +
> +	/* Each unit in timeout register is 1/216,000,000 seconds */
> +	writel(216 * timeout_us, pcie->base + REG_OFFSET);
> +}
> +
>  static void brcm_config_clkreq(struct brcm_pcie *pcie)
>  {
>  	static const char err_msg[] = "invalid 'brcm,clkreq-mode' DT string\n";
> @@ -1067,6 +1082,7 @@ static void brcm_config_clkreq(struct brcm_pcie *pcie)
>  		 * atypical and should happen only with older devices.
>  		 */
>  		clkreq_cntl |= PCIE_MISC_HARD_PCIE_HARD_DEBUG_L1SS_ENABLE_MASK;
> +		brcm_extend_rbus_timeout(pcie);

It looks like this should be squashed into the previous patch, which
added brcm_config_clkreq().  Otherwise there's a bisection hole where
somebody testing at the previous patch could see the CPU abort.

>  	} else {
>  		/*
>  		 * "safe" -- No power savings; refclk is driven by RC

Re: [PATCH v7 1/3] dt-bindings: PCI: brcmstb: Add property "brcm,clkreq-mode"

[Bjorn Helgaas]

On Thu, Nov 09, 2023 at 02:13:52PM -0500, Jim Quinlan wrote:
> The Broadcom STB/CM PCIe HW -- a core that is also used by RPi SOCs --
> requires the driver to deliberately place the RC HW one of three CLKREQ#
> modes.  The "brcm,clkreq-mode" property allows the user to override the
> default setting.  If this property is omitted, the default mode shall be
> "default".
> 
> Signed-off-by: Jim Quinlan <james.quinlan@broadcom.com>
> ---
>  .../bindings/pci/brcm,stb-pcie.yaml           | 21 +++++++++++++++++++
>  1 file changed, 21 insertions(+)
> 
> diff --git a/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml b/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
> index 7e15aae7d69e..992b35e915a5 100644
> --- a/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
> +++ b/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
> @@ -64,6 +64,27 @@ properties:
>  
>    aspm-no-l0s: true
>  
> +  brcm,clkreq-mode:
> +    description: A string that determines the operating
> +      clkreq mode of the PCIe RC HW WRT controlling the refclk signal.

I assume "WRT" is shorthand for "with respect to", but it's slightly
confusing following all the other acronyms.

> +      There are three different modes --
> +      "safe", which drives the
> +      refclk signal unconditionally and will work for all devices but does
> +      not provide any power savings;
> +      "no-l1ss" -- which provides Clock Power Management, L0s, and
> +      L1, but cannot provide L1 substate (L1SS) power
> +      savings. If the downstream device connected to the RC is
> +      L1SS capable AND the OS enables L1SS, all PCIe traffic
> +      may abruptly halt, potentially hanging the system;
> +      "default" -- which provides L0s, L1, and L1SS, but not
> +      compliant to provide Clock Power Management;
> +      specifically, may not be able to meet the Tclron max
> +      timing of 400ns as specified in "Dynamic Clock Control",
> +      section 3.2.5.2.2 of the PCIe spec.  This situation is
> +      atypical and should happen only with older devices.

These are all weirdly wrapped.  Really no reason to use lines shorter
than 80.

Same spec citation question as in patch 2/3.

> +    $ref: /schemas/types.yaml#/definitions/string
> +    enum: [ safe, no-l1ss, default ]
> +
>    brcm,scb-sizes:
>      description: u64 giving the 64bit PCIe memory
>        viewport size of a memory controller.  There may be up to

Re: [PATCH v7 2/3] PCI: brcmstb: Configure HW CLKREQ# mode appropriate for downstream device

[Jim Quinlan]

[-- Attachment #1: Type: text/plain, Size: 8048 bytes --]

On Thu, Nov 9, 2023 at 4:27 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Thu, Nov 09, 2023 at 02:13:53PM -0500, Jim Quinlan wrote:
> > The Broadcom STB/CM PCIe HW core, which is also used in RPi SOCs, must be
> > deliberately set by the PCIe RC HW into one of three mutually exclusive
> > modes:
> >
> > "safe" -- No CLKREQ# expected or required, refclk is always provided.  This
> >     mode should work for all devices but is not be capable of any refclk
> >     power savings.
> >
> > "no-l1ss" -- CLKREQ# is expected to be driven by the downstream device for
> >     CPM and ASPM L0s and L1.  Provides Clock Power Management, L0s, and L1,
> >     but cannot provide L1 substate (L1SS) power savings. If the downstream
> >     device connected to the RC is L1SS capable AND the OS enables L1SS, all
> >     PCIe traffic may abruptly halt, potentially hanging the system.
> >
> > "default" -- Bidirectional CLKREQ# between the RC and downstream device.
> >     Provides ASPM L0s, L1, and L1SS, but not compliant to provide Clock
> >     Power Management; specifically, may not be able to meet the Tclron max
> >     timing of 400ns as specified in "Dynamic Clock Control", section
> >     3.2.5.2.2 of the PCIe spec.  This situation is atypical and should
> >     happen only with older devices.
>
> The PCIe base spec r6.0 has no section 3.2.5.2.2.  Looks like this
> could be:
>
>   PCIe Mini CEM r2.1, sec 3.2.5.2.2 (December, 2016), or
>   PCIe CEM r5.1, sec 2.8.2 (August, 2023)
>
> I don't know the relationship between the "Mini CEM" and the "CEM"
> specs, but CEM r5.1 seems to have the same text as the Mini CEM r2.1
> about Dynamic Clock Control.
>
> We're hampered by the lack of clear subscripts here, but the text in
> both capitalizes the "CRL" part, e.g., "T_CLRon".
>
> > Previously, this driver always set the mode to "no-l1ss", as almost all
> > STB/CM boards operate in this mode.  But now there is interest in
> > activating L1SS power savings from STB/CM customers, which requires "aspm"
> > mode.  In addition, a bug was filed for RPi4 CM platform because most
> > devices did not work in "no-l1ss" mode.
> >
> > Note that the mode is specified by the DT property "brcm,clkreq-mode".  If
> > this property is omitted, then "default" mode is chosen.
> >
> > Link: https://bugzilla.kernel.org/show_bug.cgi?id=217276
> >
> > Signed-off-by: Jim Quinlan <james.quinlan@broadcom.com>
> > ---
> >  drivers/pci/controller/pcie-brcmstb.c | 65 ++++++++++++++++++++++-----
> >  1 file changed, 55 insertions(+), 10 deletions(-)
> >
> > diff --git a/drivers/pci/controller/pcie-brcmstb.c b/drivers/pci/controller/pcie-brcmstb.c
> > index f9dd6622fe10..f45c5d0168d3 100644
> > --- a/drivers/pci/controller/pcie-brcmstb.c
> > +++ b/drivers/pci/controller/pcie-brcmstb.c
> > @@ -121,9 +121,12 @@
> >
> >  #define PCIE_MISC_HARD_PCIE_HARD_DEBUG                                       0x4204
> >  #define  PCIE_MISC_HARD_PCIE_HARD_DEBUG_CLKREQ_DEBUG_ENABLE_MASK     0x2
> > +#define  PCIE_MISC_HARD_PCIE_HARD_DEBUG_L1SS_ENABLE_MASK             0x200000
> >  #define  PCIE_MISC_HARD_PCIE_HARD_DEBUG_SERDES_IDDQ_MASK             0x08000000
> >  #define  PCIE_BMIPS_MISC_HARD_PCIE_HARD_DEBUG_SERDES_IDDQ_MASK               0x00800000
> > -
> > +#define  PCIE_CLKREQ_MASK \
> > +       (PCIE_MISC_HARD_PCIE_HARD_DEBUG_CLKREQ_DEBUG_ENABLE_MASK | \
> > +        PCIE_MISC_HARD_PCIE_HARD_DEBUG_L1SS_ENABLE_MASK)
> >
> >  #define PCIE_INTR2_CPU_BASE          0x4300
> >  #define PCIE_MSI_INTR2_BASE          0x4500
> > @@ -1028,13 +1031,61 @@ static int brcm_pcie_setup(struct brcm_pcie *pcie)
> >       return 0;
> >  }
> >
> > +static void brcm_config_clkreq(struct brcm_pcie *pcie)
> > +{
> > +     static const char err_msg[] = "invalid 'brcm,clkreq-mode' DT string\n";
> > +     const char *mode = "default";
> > +     u32 clkreq_cntl;
> > +     int ret;
> > +
> > +     ret = of_property_read_string(pcie->np, "brcm,clkreq-mode", &mode);
> > +     if (ret && ret != -EINVAL) {
> > +             dev_err(pcie->dev, err_msg);
> > +             mode = "safe";
> > +     }
> > +
> > +     /* Start out assuming safe mode (both mode bits cleared) */
> > +     clkreq_cntl = readl(pcie->base + PCIE_MISC_HARD_PCIE_HARD_DEBUG);
> > +     clkreq_cntl &= ~PCIE_CLKREQ_MASK;
> > +
> > +     if (strcmp(mode, "no-l1ss") == 0) {
> > +             /*
> > +              * "no-l1ss" -- Provides Clock Power Management, L0s, and
> > +              * L1, but cannot provide L1 substate (L1SS) power
> > +              * savings. If the downstream device connected to the RC is
> > +              * L1SS capable AND the OS enables L1SS, all PCIe traffic
> > +              * may abruptly halt, potentially hanging the system.
> > +              */
> > +             clkreq_cntl |= PCIE_MISC_HARD_PCIE_HARD_DEBUG_CLKREQ_DEBUG_ENABLE_MASK;
>
> Does this somehow change the features advertised by the Root Port,
> e.g., does it hide the L1 PM Substates Capability completely, or at
> least clear the L1 PM Substates Supported bit?

>
> It it doesn't, the PCI core may enable L1SS and cause this hang.
> Every feature advertised in config space is expected to work.
Agree, I'll put this back in and also reconcile your other comments.

BTW, besides the RPi4, I haven't been able to find a Linux platform
where I can do

        echo $POLICY > /sys/module/pcie_aspm/parameters/policy

It seems that the FW/ACPI typically locks this down.  I did see a
comment somewhere that
said that the reason it was locked down is because too many devices
cannot handle it.
FWIW.

Regards,
Jim Quinlan
Broadcom STB/CM

>
> > +     } else if (strcmp(mode, "default") == 0) {
> > +             /*
> > +              * "default" -- Provides L0s, L1, and L1SS, but not
> > +              * compliant to provide Clock Power Management;
> > +              * specifically, may not be able to meet the Tclron max
> > +              * timing of 400ns as specified in "Dynamic Clock Control",
> > +              * section 3.2.5.2.2 of the PCIe spec.  This situation is
> > +              * atypical and should happen only with older devices.
> > +              */
> > +             clkreq_cntl |= PCIE_MISC_HARD_PCIE_HARD_DEBUG_L1SS_ENABLE_MASK;
> > +     } else {
> > +             /*
> > +              * "safe" -- No power savings; refclk is driven by RC
> > +              * unconditionally.
> > +              */
> > +             if (strcmp(mode, "safe") != 0)
> > +                     dev_err(pcie->dev, err_msg);
> > +             mode = "safe";
> > +     }
> > +     writel(clkreq_cntl, pcie->base + PCIE_MISC_HARD_PCIE_HARD_DEBUG);
> > +     dev_info(pcie->dev, "clkreq-mode set to %s\n", mode);
> > +}
> > +
> >  static int brcm_pcie_start_link(struct brcm_pcie *pcie)
> >  {
> >       struct device *dev = pcie->dev;
> >       void __iomem *base = pcie->base;
> >       u16 nlw, cls, lnksta;
> >       bool ssc_good = false;
> > -     u32 tmp;
> >       int ret, i;
> >
> >       /* Unassert the fundamental reset */
> > @@ -1059,6 +1110,8 @@ static int brcm_pcie_start_link(struct brcm_pcie *pcie)
> >               return -ENODEV;
> >       }
> >
> > +     brcm_config_clkreq(pcie);
> > +
> >       if (pcie->gen)
> >               brcm_pcie_set_gen(pcie, pcie->gen);
> >
> > @@ -1077,14 +1130,6 @@ static int brcm_pcie_start_link(struct brcm_pcie *pcie)
> >                pci_speed_string(pcie_link_speed[cls]), nlw,
> >                ssc_good ? "(SSC)" : "(!SSC)");
> >
> > -     /*
> > -      * Refclk from RC should be gated with CLKREQ# input when ASPM L0s,L1
> > -      * is enabled => setting the CLKREQ_DEBUG_ENABLE field to 1.
> > -      */
> > -     tmp = readl(base + PCIE_MISC_HARD_PCIE_HARD_DEBUG);
> > -     tmp |= PCIE_MISC_HARD_PCIE_HARD_DEBUG_CLKREQ_DEBUG_ENABLE_MASK;
> > -     writel(tmp, base + PCIE_MISC_HARD_PCIE_HARD_DEBUG);
> > -
> >       return 0;
> >  }

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4210 bytes --]

Re: [PATCH v7 2/3] PCI: brcmstb: Configure HW CLKREQ# mode appropriate for downstream device

[Bjorn Helgaas]

On Thu, Nov 09, 2023 at 05:06:15PM -0500, Jim Quinlan wrote:
> ...

> BTW, besides the RPi4, I haven't been able to find a Linux platform
> where I can do
> 
>         echo $POLICY > /sys/module/pcie_aspm/parameters/policy

This sounds like something we should fix.  What exactly happens?  I
think this should be handled at pcie_aspm_set_policy(), so:

  /sys/module/pcie_aspm/parameters/policy doesn't exist (seems
  unlikely)?

  Returns -EPERM (would indicate aspm_disabled)?

  Returns -EINVAL (would indicate $POLICY doesn't match anything in
  policy_str[])?

  Returns 0 with no action (would indicate $POLICY is the same as the
  current aspm_policy)?

> It seems that the FW/ACPI typically locks this down.  I did see a
> comment somewhere that
> said that the reason it was locked down is because too many devices
> cannot handle it.

Do you have any details about FW/ACPI locking this down?
aspm_disabled is set by the kernel "pcie_aspm=off" parameter (I assume
you're not referring to this), if the FADT has ACPI_FADT_NO_ASPM set,
or if a host bridge's _OSC is missing or failed (maybe [1] is the
comment you saw?)

These all *should* be unusual cases, so I'd be surprised if you're
tripping over one of these.  I would NOT be surprised if we had some
issue in pcie_config_aspm_link() or pcie_set_clkpm() that meant the
policy change didn't work as intended, though.

Bjorn

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/acpi/pci_root.c?id=v6.6#n617
  

Re: [PATCH v7 2/3] PCI: brcmstb: Configure HW CLKREQ# mode appropriate for downstream device

[Jim Quinlan]

[-- Attachment #1: Type: text/plain, Size: 2302 bytes --]

On Thu, Nov 9, 2023 at 5:31 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Thu, Nov 09, 2023 at 05:06:15PM -0500, Jim Quinlan wrote:
> > ...
>
> > BTW, besides the RPi4, I haven't been able to find a Linux platform
> > where I can do
> >
> >         echo $POLICY > /sys/module/pcie_aspm/parameters/policy
>
> This sounds like something we should fix.  What exactly happens?  I
> think this should be handled at pcie_aspm_set_policy(), so:

Well, I've tried changing the ASPM policy on my x86 Ubuntu system and
IIRC a Fedora system.
In both cases it says "illegal write operation" but I am root and the
"policy" file does have
rw perms for root, so I have no idea how it comes back with that
error.  Some machines
allow one to change the setting in the BIOS , FWIW.

Right now on my CM4, "echo $POLICY  > policy" actually works.  Perhaps
when I was testing this I did not yet apply my commits, or perhaps it was with
a specific endpoint device.  Regardless, I'll let you know with a
backtrace if I see
this again.

Regards,
Jim


>
>   /sys/module/pcie_aspm/parameters/policy doesn't exist (seems
>   unlikely)?
>
>   Returns -EPERM (would indicate aspm_disabled)?
>
>   Returns -EINVAL (would indicate $POLICY doesn't match anything in
>   policy_str[])?
>
>   Returns 0 with no action (would indicate $POLICY is the same as the
>   current aspm_policy)?
>
> > It seems that the FW/ACPI typically locks this down.  I did see a
> > comment somewhere that
> > said that the reason it was locked down is because too many devices
> > cannot handle it.
>
> Do you have any details about FW/ACPI locking this down?
> aspm_disabled is set by the kernel "pcie_aspm=off" parameter (I assume
> you're not referring to this), if the FADT has ACPI_FADT_NO_ASPM set,
> or if a host bridge's _OSC is missing or failed (maybe [1] is the
> comment you saw?)
>
> These all *should* be unusual cases, so I'd be surprised if you're
> tripping over one of these.  I would NOT be surprised if we had some
> issue in pcie_config_aspm_link() or pcie_set_clkpm() that meant the
> policy change didn't work as intended, though.
>
> Bjorn
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/acpi/pci_root.c?id=v6.6#n617
>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4210 bytes --]

Re: [PATCH v7 2/3] PCI: brcmstb: Configure HW CLKREQ# mode appropriate for downstream device

[Bjorn Helgaas]

On Fri, Nov 10, 2023 at 08:01:23AM -0500, Jim Quinlan wrote:
> On Thu, Nov 9, 2023 at 5:31 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> >
> > On Thu, Nov 09, 2023 at 05:06:15PM -0500, Jim Quinlan wrote:
> > > ...
> >
> > > BTW, besides the RPi4, I haven't been able to find a Linux platform
> > > where I can do
> > >
> > >         echo $POLICY > /sys/module/pcie_aspm/parameters/policy
> >
> > This sounds like something we should fix.  What exactly happens?  I
> > think this should be handled at pcie_aspm_set_policy(), so:
> 
> Well, I've tried changing the ASPM policy on my x86 Ubuntu system
> and IIRC a Fedora system.  In both cases it says "illegal write
> operation" but I am root and the "policy" file does have rw perms
> for root, so I have no idea how it comes back with that error.  Some
> machines allow one to change the setting in the BIOS, FWIW.

BIOS settings like that are potentially misleading unless the BIOS
*also* retains ownership of ASPM or changes the ASPM features
advertised by devices.  If BIOS grants ASPM ownership to the OS, BIOS
should not assume anything about how the OS will configure ASPM.

> Right now on my CM4, "echo $POLICY  > policy" actually works.
> Perhaps when I was testing this I did not yet apply my commits, or
> perhaps it was with a specific endpoint device.  Regardless, I'll
> let you know with a backtrace if I see this again.

Great, thanks!

Bjorn

Re: [PATCH v7 0/3] PCI: brcmstb: Configure appropriate HW CLKREQ# mode

[Florian Fainelli]

[-- Attachment #1: Type: text/plain, Size: 4208 bytes --]

On 11/9/23 11:13, Jim Quinlan wrote:
> v7 -- Manivannan Sadhasivam suggested (a) making the property look like a
>        network phy-mode and (b) keeping the code simple (not counting clkreq
>        signal appearances, un-advertising capabilites, etc).  This is
>        what I have done.  The property is now "brcm,clkreq-mode" and
>        the values may be one of "safe", "default", and "no-l1ss".  The
>        default setting is to employ the most capable power savings mode.
> 
> v6 -- No code has been changed.
>     -- Changed commit subject and comment in "#PERST" commit (Bjorn, Cyril)
>     -- Changed sign-off and author email address for all commits.
>        This was due to a change in Broadcom's upstreaming policy.
> 
> v5 -- Remove DT property "brcm,completion-timeout-us" from	
>        "DT bindings" commit.  Although this error may be reported	
>        as a completion timeout, its cause was traced to an	
>        internal bus timeout which may occur even when there is	
>        no PCIe access being processed.  We set a timeout of four	
>        seconds only if we are operating in "L1SS CLKREQ#" mode.
>     -- Correct CEM 2.0 reference provided by HW engineer,
>        s/3.2.5.2.5/3.2.5.2.2/ (Bjorn)
>     -- Add newline to dev_info() string (Stefan)
>     -- Change variable rval to unsigned (Stefan)
>     -- s/implementaion/implementation/ (Bjorn)
>     -- s/superpowersave/powersupersave/ (Bjorn)
>     -- Slightly modify message on "PERST#" commit.
>     -- Rebase to torvalds master
> 
> v4 -- New commit that asserts PERST# for 2711/RPi SOCs at PCIe RC
>        driver probe() time.  This is done in Raspian Linux and its
>        absence may be the cause of a failing test case.
>     -- New commit that removes stale comment.
> 
> v3 -- Rewrote commit msgs and comments refering panics if L1SS
>        is enabled/disabled; the code snippet that unadvertises L1SS
>        eliminates the panic scenario. (Bjorn)
>     -- Add reference for "400ns of CLKREQ# assertion" blurb (Bjorn)
>     -- Put binding names in DT commit Subject (Bjorn)
>     -- Add a verb to a commit's subject line (Bjorn)
>     -- s/accomodat(\w+)/accommodat$1/g (Bjorn)
>     -- Rewrote commit msgs and comments refering panics if L1SS
>        is enabled/disabled; the code snippet that unadvertises L1SS
>        eliminates the panic scenario. (Bjorn)
> 
> v2 -- Changed binding property 'brcm,completion-timeout-msec' to
>        'brcm,completion-timeout-us'.  (StefanW for standard suffix).
>     -- Warn when clamping timeout value, and include clamped
>        region in message. Also add min and max in YAML. (StefanW)
>     -- Qualify description of "brcm,completion-timeout-us" so that
>        it refers to PCIe transactions. (StefanW)
>     -- Remvove mention of Linux specifics in binding description. (StefanW)
>     -- s/clkreq#/CLKREQ#/g (Bjorn)
>     -- Refactor completion-timeout-us code to compare max and min to
>        value given by the property (as opposed to the computed value).
> 
> v1 -- The current driver assumes the downstream devices can
>        provide CLKREQ# for ASPM.  These commits accomodate devices
>        w/ or w/o clkreq# and also handle L1SS-capable devices.
> 
>     -- The Raspian Linux folks have already been using a PCIe RC
>        property "brcm,enable-l1ss".  These commits use the same
>        property, in a backward-compatible manner, and the implementaion
>        adds more detail and also automatically identifies devices w/o
>        a clkreq# signal, i.e. most devices plugged into an RPi CM4
>        IO board.

For the entire series:

Tested-by: Florian Fainelli <florian.fainelli@broadcom.com>

on all of my available devices:

- SUNIX Co., Ltd. Multiport serial controller
- Broadcom BCM4331 Wi-Fi
- Broadcom BCM43224 Wi-Fi
- Broadcom BCM4322 Wi-Fi
- Qualcomm Atheros AR5008
- Broadcom BCM4366 Wi-Fi
- Marvell Technology Group Ltd. 88SE9125 PCIe SATA 6.0 Gb/s controller
- Intel 7260 Wi-Fi
- Intel Corporation 82574L Gigabit Network Connection
- Broadcom NetXtreme BCM5751 Gigabit Ethernet
- Pepperl+Fuchs RocketPort EXPRESS 8-port w/Octa Cable
- Micron/Crucial Technology P2 NVMe PCIe SSD
- ASM1184e PCIe Switch Port
-- 
Florian


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4221 bytes --]