[PATCH v8 0/2] PCI: brcmstb: Configure appropriate HW CLKREQ# mode

[PATCH v8 0/2] PCI: brcmstb: Configure appropriate HW CLKREQ# mode

[Jim Quinlan]

[-- Attachment #1: Type: text/plain, Size: 4151 bytes --]

V8 -- Un-advertise L1SS capability when in "no-l1ss" mode (Bjorn)
   -- Squashed last two commits of v7 (Bjorn)
   -- Fix DT binding description text wrapping (Bjorn)
   -- Fix incorrect Spec reference (Bjorn)
         s/PCIe Spec/PCIe Express Mini CEM 2.1 specification/
   -- Text substitutions (Bjorn)
         s/WRT/With respect to/ 
         s/Tclron/T_CLRon/

v7 -- Manivannan Sadhasivam suggested (a) making the property look like a
      network phy-mode and (b) keeping the code simple (not counting clkreq
      signal appearances, un-advertising capabilites, etc).  This is
      what I have done.  The property is now "brcm,clkreq-mode" and
      the values may be one of "safe", "default", and "no-l1ss".  The
      default setting is to employ the most capable power savings mode.

v6 -- No code has been changed.
   -- Changed commit subject and comment in "#PERST" commit (Bjorn, Cyril)
   -- Changed sign-off and author email address for all commits.
      This was due to a change in Broadcom's upstreaming policy.

v5 -- Remove DT property "brcm,completion-timeout-us" from	 
      "DT bindings" commit.  Although this error may be reported	 
      as a completion timeout, its cause was traced to an	 
      internal bus timeout which may occur even when there is	 
      no PCIe access being processed.  We set a timeout of four	 
      seconds only if we are operating in "L1SS CLKREQ#" mode.
   -- Correct CEM 2.0 reference provided by HW engineer,
      s/3.2.5.2.5/3.2.5.2.2/ (Bjorn)
   -- Add newline to dev_info() string (Stefan)
   -- Change variable rval to unsigned (Stefan)
   -- s/implementaion/implementation/ (Bjorn)
   -- s/superpowersave/powersupersave/ (Bjorn)
   -- Slightly modify message on "PERST#" commit.
   -- Rebase to torvalds master

v4 -- New commit that asserts PERST# for 2711/RPi SOCs at PCIe RC
      driver probe() time.  This is done in Raspian Linux and its
      absence may be the cause of a failing test case.
   -- New commit that removes stale comment.

v3 -- Rewrote commit msgs and comments refering panics if L1SS
      is enabled/disabled; the code snippet that unadvertises L1SS
      eliminates the panic scenario. (Bjorn)
   -- Add reference for "400ns of CLKREQ# assertion" blurb (Bjorn)
   -- Put binding names in DT commit Subject (Bjorn)
   -- Add a verb to a commit's subject line (Bjorn)
   -- s/accomodat(\w+)/accommodat$1/g (Bjorn)
   -- Rewrote commit msgs and comments refering panics if L1SS
      is enabled/disabled; the code snippet that unadvertises L1SS
      eliminates the panic scenario. (Bjorn)

v2 -- Changed binding property 'brcm,completion-timeout-msec' to
      'brcm,completion-timeout-us'.  (StefanW for standard suffix).
   -- Warn when clamping timeout value, and include clamped
      region in message. Also add min and max in YAML. (StefanW)
   -- Qualify description of "brcm,completion-timeout-us" so that
      it refers to PCIe transactions. (StefanW)
   -- Remvove mention of Linux specifics in binding description. (StefanW)
   -- s/clkreq#/CLKREQ#/g (Bjorn)
   -- Refactor completion-timeout-us code to compare max and min to
      value given by the property (as opposed to the computed value).

v1 -- The current driver assumes the downstream devices can
      provide CLKREQ# for ASPM.  These commits accomodate devices
      w/ or w/o clkreq# and also handle L1SS-capable devices.

   -- The Raspian Linux folks have already been using a PCIe RC
      property "brcm,enable-l1ss".  These commits use the same
      property, in a backward-compatible manner, and the implementaion
      adds more detail and also automatically identifies devices w/o
      a clkreq# signal, i.e. most devices plugged into an RPi CM4
      IO board.

Jim Quinlan (2):
  dt-bindings: PCI: brcmstb: Add property "brcm,clkreq-mode"
  PCI: brcmstb: Configure HW CLKREQ# mode appropriate for downstream
    device

 .../bindings/pci/brcm,stb-pcie.yaml           | 18 ++++
 drivers/pci/controller/pcie-brcmstb.c         | 96 +++++++++++++++++--
 2 files changed, 104 insertions(+), 10 deletions(-)


base-commit: 305230142ae0637213bf6e04f6d9f10bbcb74af8
-- 
2.17.1


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4210 bytes --]

[PATCH v8 1/2] dt-bindings: PCI: brcmstb: Add property "brcm,clkreq-mode"

[Jim Quinlan]

[-- Attachment #1: Type: text/plain, Size: 2076 bytes --]

The Broadcom STB/CM PCIe HW -- a core that is also used by RPi SOCs --
requires the driver to deliberately place the RC HW one of three CLKREQ#
modes.  The "brcm,clkreq-mode" property allows the user to override the
default setting.  If this property is omitted, the default mode shall be
"default".

Signed-off-by: Jim Quinlan <james.quinlan@broadcom.com>
---
 .../devicetree/bindings/pci/brcm,stb-pcie.yaml | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml b/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
index 7e15aae7d69e..22491f7f8852 100644
--- a/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
+++ b/Documentation/devicetree/bindings/pci/brcm,stb-pcie.yaml
@@ -64,6 +64,24 @@ properties:
 
   aspm-no-l0s: true
 
+  brcm,clkreq-mode:
+    description: A string that determines the operating
+      clkreq mode of the PCIe RC HW with respect to controlling the refclk
+      signal.  There are three different modes -- "safe", which drives the
+      refclk signal unconditionally and will work for all devices but does
+      not provide any power savings; "no-l1ss" -- which provides Clock
+      Power Management, L0s, and L1, but cannot provide L1 substate (L1SS)
+      power savings. If the downstream device connected to the RC is L1SS
+      capable AND the OS enables L1SS, all PCIe traffic may abruptly halt,
+      potentially hanging the system; "default" -- which provides L0s, L1,
+      and L1SS, but not compliant to provide Clock Power Management;
+      specifically, may not be able to meet the T_CLRon max timing of 400ns
+      as specified in "Dynamic Clock Control", section 3.2.5.2.2 PCI
+      Express Mini CEM 2.1 specification.  This situation is atypical and
+      should happen only with older devices.
+    $ref: /schemas/types.yaml#/definitions/string
+    enum: [ safe, no-l1ss, default ]
+
   brcm,scb-sizes:
     description: u64 giving the 64bit PCIe memory
       viewport size of a memory controller.  There may be up to
-- 
2.17.1


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4210 bytes --]

Re: [PATCH v8 1/2] dt-bindings: PCI: brcmstb: Add property "brcm,clkreq-mode"

[Conor Dooley]

[-- Attachment #1: Type: text/plain, Size: 513 bytes --]

On Mon, Nov 13, 2023 at 01:56:05PM -0500, Jim Quinlan wrote:
> The Broadcom STB/CM PCIe HW -- a core that is also used by RPi SOCs --
> requires the driver to deliberately place the RC HW one of three CLKREQ#
> modes.  The "brcm,clkreq-mode" property allows the user to override the
> default setting.  If this property is omitted, the default mode shall be
> "default".
> 
> Signed-off-by: Jim Quinlan <james.quinlan@broadcom.com>

Acked-by: Conor Dooley <conor.dooley@microchip.com>

Cheers,
Conor

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v8 1/2] dt-bindings: PCI: brcmstb: Add property "brcm,clkreq-mode"

[Rob Herring]

On Mon, 13 Nov 2023 13:56:05 -0500, Jim Quinlan wrote:
> The Broadcom STB/CM PCIe HW -- a core that is also used by RPi SOCs --
> requires the driver to deliberately place the RC HW one of three CLKREQ#
> modes.  The "brcm,clkreq-mode" property allows the user to override the
> default setting.  If this property is omitted, the default mode shall be
> "default".
> 
> Signed-off-by: Jim Quinlan <james.quinlan@broadcom.com>
> ---
>  .../devicetree/bindings/pci/brcm,stb-pcie.yaml | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)
> 

Reviewed-by: Rob Herring <robh@kernel.org>

Re: [PATCH v8 0/2] PCI: brcmstb: Configure appropriate HW CLKREQ# mode

[Cyril Brulebois]

[-- Attachment #1: Type: text/plain, Size: 2219 bytes --]

Hi Jim,

Jim Quinlan <james.quinlan@broadcom.com> (2023-11-13):
> V8 -- Un-advertise L1SS capability when in "no-l1ss" mode (Bjorn)
>    -- Squashed last two commits of v7 (Bjorn)
>    -- Fix DT binding description text wrapping (Bjorn)
>    -- Fix incorrect Spec reference (Bjorn)
>          s/PCIe Spec/PCIe Express Mini CEM 2.1 specification/
>    -- Text substitutions (Bjorn)
>          s/WRT/With respect to/ 
>          s/Tclron/T_CLRon/
> 
> v7 -- Manivannan Sadhasivam suggested (a) making the property look like a
>       network phy-mode and (b) keeping the code simple (not counting clkreq
>       signal appearances, un-advertising capabilites, etc).  This is
>       what I have done.  The property is now "brcm,clkreq-mode" and
>       the values may be one of "safe", "default", and "no-l1ss".  The
>       default setting is to employ the most capable power savings mode.

Still:

Tested-by: Cyril Brulebois <cyril@debamax.com>


I've just run my big test matrix again, and I can confirm everything is
still looking good with the updated series and the updated base commit.


Test setup:
-----------

 - using a $CM with the 20230111 EEPROM
 - on the same CM4 IO Board
 - with a $PCIE board (PCIe to multiple USB ports)
 - and the same Samsung USB flash drive + Logitech keyboard.

where $CM is one of:

 - CM4 Lite Rev 1.0
 - CM4 8/32 Rev 1.0
 - CM4 4/32 Rev 1.1

and $PCIE is one of:

 - SupaHub PCE6U1C-R02, VER 006
 - SupaHub PCE6U1C-R02, VER 006S
 - Waveshare VIA VL805/806-based


Results:
--------

 1. With an unpatched kernel, I'm getting either a successful boot
    *without* seeing the devices plugged on the PCIe-to-USB board
    or the dreaded SError in most cases, using a locally-built
    v6.4-rc7-194-g8a28a0b6f1a1d kernel.

 2. With a patched kernel (v6.6-15365-g305230142ae0 + this series),
    for all $CM/$PCIE combinations, I'm getting a system that boots,
    sees the flash drive, and gives decent read/write performance on
    it (plus a functional keyboard).


Cheers,
-- 
Cyril Brulebois (kibi@debian.org)            <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v8 0/2] PCI: brcmstb: Configure appropriate HW CLKREQ# mode

[Florian Fainelli]

[-- Attachment #1: Type: text/plain, Size: 1115 bytes --]

On 11/26/23 12:19, Cyril Brulebois wrote:
> Hi Jim,
> 
> Jim Quinlan <james.quinlan@broadcom.com> (2023-11-13):
>> V8 -- Un-advertise L1SS capability when in "no-l1ss" mode (Bjorn)
>>     -- Squashed last two commits of v7 (Bjorn)
>>     -- Fix DT binding description text wrapping (Bjorn)
>>     -- Fix incorrect Spec reference (Bjorn)
>>           s/PCIe Spec/PCIe Express Mini CEM 2.1 specification/
>>     -- Text substitutions (Bjorn)
>>           s/WRT/With respect to/
>>           s/Tclron/T_CLRon/
>>
>> v7 -- Manivannan Sadhasivam suggested (a) making the property look like a
>>        network phy-mode and (b) keeping the code simple (not counting clkreq
>>        signal appearances, un-advertising capabilites, etc).  This is
>>        what I have done.  The property is now "brcm,clkreq-mode" and
>>        the values may be one of "safe", "default", and "no-l1ss".  The
>>        default setting is to employ the most capable power savings mode.
> 
> Still:
> 
> Tested-by: Cyril Brulebois <cyril@debamax.com>

Thanks Cyril! Bjorn, Lorenzo, any chance this can be applied soon? Thanks!
-- 
Florian


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4221 bytes --]

Re: [PATCH v8 0/2] PCI: brcmstb: Configure appropriate HW CLKREQ# mode

[Bjorn Helgaas]

On Tue, Dec 12, 2023 at 03:51:12PM -0800, Florian Fainelli wrote:
> On 11/26/23 12:19, Cyril Brulebois wrote:
> > Hi Jim,
> > 
> > Jim Quinlan <james.quinlan@broadcom.com> (2023-11-13):
> > > V8 -- Un-advertise L1SS capability when in "no-l1ss" mode (Bjorn)
> > >     -- Squashed last two commits of v7 (Bjorn)
> > >     -- Fix DT binding description text wrapping (Bjorn)
> > >     -- Fix incorrect Spec reference (Bjorn)
> > >           s/PCIe Spec/PCIe Express Mini CEM 2.1 specification/
> > >     -- Text substitutions (Bjorn)
> > >           s/WRT/With respect to/
> > >           s/Tclron/T_CLRon/
> > > 
> > > v7 -- Manivannan Sadhasivam suggested (a) making the property look like a
> > >        network phy-mode and (b) keeping the code simple (not counting clkreq
> > >        signal appearances, un-advertising capabilites, etc).  This is
> > >        what I have done.  The property is now "brcm,clkreq-mode" and
> > >        the values may be one of "safe", "default", and "no-l1ss".  The
> > >        default setting is to employ the most capable power savings mode.
> > 
> > Still:
> > 
> > Tested-by: Cyril Brulebois <cyril@debamax.com>
> 
> Thanks Cyril! Bjorn, Lorenzo, any chance this can be applied soon? Thanks!

Seems OK to me if Lorenzo or Krzysztof W. are OK with it.

Re: [PATCH v8 0/2] PCI: brcmstb: Configure appropriate HW CLKREQ# mode

[Jim Quinlan]

On Wed, Dec 13, 2023 at 2:59 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> On Tue, Dec 12, 2023 at 03:51:12PM -0800, Florian Fainelli wrote:
> > On 11/26/23 12:19, Cyril Brulebois wrote:
> > > Hi Jim,
> > >
> > > Jim Quinlan <james.quinlan@broadcom.com> (2023-11-13):
> > > > V8 -- Un-advertise L1SS capability when in "no-l1ss" mode (Bjorn)
> > > >     -- Squashed last two commits of v7 (Bjorn)
> > > >     -- Fix DT binding description text wrapping (Bjorn)
> > > >     -- Fix incorrect Spec reference (Bjorn)
> > > >           s/PCIe Spec/PCIe Express Mini CEM 2.1 specification/
> > > >     -- Text substitutions (Bjorn)
> > > >           s/WRT/With respect to/
> > > >           s/Tclron/T_CLRon/
> > > >
> > > > v7 -- Manivannan Sadhasivam suggested (a) making the property look like a
> > > >        network phy-mode and (b) keeping the code simple (not counting clkreq
> > > >        signal appearances, un-advertising capabilites, etc).  This is
> > > >        what I have done.  The property is now "brcm,clkreq-mode" and
> > > >        the values may be one of "safe", "default", and "no-l1ss".  The
> > > >        default setting is to employ the most capable power savings mode.
> > >
> > > Still:
> > >
> > > Tested-by: Cyril Brulebois <cyril@debamax.com>
> >
> > Thanks Cyril! Bjorn, Lorenzo, any chance this can be applied soon? Thanks!
>
> Seems OK to me if Lorenzo or Krzysztof W. are OK with it.
>
Bjorn,

What is the status of this submission?  Stock Linux on RPi4 CM4
systems with a PCIe device  panic on boot until this commit is
applied.

Regards,
Jim Quinilan
Broadcom STB/CM

Re: [PATCH v8 0/2] PCI: brcmstb: Configure appropriate HW CLKREQ# mode

[Krzysztof Wilczyński]

Hello,

> V8 -- Un-advertise L1SS capability when in "no-l1ss" mode (Bjorn)
>    -- Squashed last two commits of v7 (Bjorn)
>    -- Fix DT binding description text wrapping (Bjorn)
>    -- Fix incorrect Spec reference (Bjorn)
>          s/PCIe Spec/PCIe Express Mini CEM 2.1 specification/
>    -- Text substitutions (Bjorn)
>          s/WRT/With respect to/ 
>          s/Tclron/T_CLRon/
> 
> v7 -- Manivannan Sadhasivam suggested (a) making the property look like a
>       network phy-mode and (b) keeping the code simple (not counting clkreq
>       signal appearances, un-advertising capabilites, etc).  This is
>       what I have done.  The property is now "brcm,clkreq-mode" and
>       the values may be one of "safe", "default", and "no-l1ss".  The
>       default setting is to employ the most capable power savings mode.
> 
> v6 -- No code has been changed.
>    -- Changed commit subject and comment in "#PERST" commit (Bjorn, Cyril)
>    -- Changed sign-off and author email address for all commits.
>       This was due to a change in Broadcom's upstreaming policy.
> 
> v5 -- Remove DT property "brcm,completion-timeout-us" from	 
>       "DT bindings" commit.  Although this error may be reported	 
>       as a completion timeout, its cause was traced to an	 
>       internal bus timeout which may occur even when there is	 
>       no PCIe access being processed.  We set a timeout of four	 
>       seconds only if we are operating in "L1SS CLKREQ#" mode.
>    -- Correct CEM 2.0 reference provided by HW engineer,
>       s/3.2.5.2.5/3.2.5.2.2/ (Bjorn)
>    -- Add newline to dev_info() string (Stefan)
>    -- Change variable rval to unsigned (Stefan)
>    -- s/implementaion/implementation/ (Bjorn)
>    -- s/superpowersave/powersupersave/ (Bjorn)
>    -- Slightly modify message on "PERST#" commit.
>    -- Rebase to torvalds master
> 
> v4 -- New commit that asserts PERST# for 2711/RPi SOCs at PCIe RC
>       driver probe() time.  This is done in Raspian Linux and its
>       absence may be the cause of a failing test case.
>    -- New commit that removes stale comment.
> 
> v3 -- Rewrote commit msgs and comments refering panics if L1SS
>       is enabled/disabled; the code snippet that unadvertises L1SS
>       eliminates the panic scenario. (Bjorn)
>    -- Add reference for "400ns of CLKREQ# assertion" blurb (Bjorn)
>    -- Put binding names in DT commit Subject (Bjorn)
>    -- Add a verb to a commit's subject line (Bjorn)
>    -- s/accomodat(\w+)/accommodat$1/g (Bjorn)
>    -- Rewrote commit msgs and comments refering panics if L1SS
>       is enabled/disabled; the code snippet that unadvertises L1SS
>       eliminates the panic scenario. (Bjorn)
> 
> v2 -- Changed binding property 'brcm,completion-timeout-msec' to
>       'brcm,completion-timeout-us'.  (StefanW for standard suffix).
>    -- Warn when clamping timeout value, and include clamped
>       region in message. Also add min and max in YAML. (StefanW)
>    -- Qualify description of "brcm,completion-timeout-us" so that
>       it refers to PCIe transactions. (StefanW)
>    -- Remvove mention of Linux specifics in binding description. (StefanW)
>    -- s/clkreq#/CLKREQ#/g (Bjorn)
>    -- Refactor completion-timeout-us code to compare max and min to
>       value given by the property (as opposed to the computed value).
> 
> v1 -- The current driver assumes the downstream devices can
>       provide CLKREQ# for ASPM.  These commits accomodate devices
>       w/ or w/o clkreq# and also handle L1SS-capable devices.
> 
>    -- The Raspian Linux folks have already been using a PCIe RC
>       property "brcm,enable-l1ss".  These commits use the same
>       property, in a backward-compatible manner, and the implementaion
>       adds more detail and also automatically identifies devices w/o
>       a clkreq# signal, i.e. most devices plugged into an RPi CM4
>       IO board.

Applied to controller/broadcom, thank you!

[01/02] dt-bindings: PCI: brcmstb: Add property "brcm,clkreq-mode"
        https://git.kernel.org/pci/pci/c/14b15aeb3628
[02/02] PCI: brcmstb: Configure HW CLKREQ# mode appropriate for downstream device
        https://git.kernel.org/pci/pci/c/e2596dcf1e9d

	Krzysztof

Re: [PATCH v8 0/2] PCI: brcmstb: Configure appropriate HW CLKREQ# mode

[Krzysztof Wilczyński]

Hello,

[...]
> What is the status of this submission?

Looks good!  I apologise for the delay.

Bjorn is keen to pick this up for 6.8 with other changes, so it should land
there shortly, given that we have a merge window open now.

	Krzysztof