[PATCH net-next v9 0/5] r8169: Temporarily disable ASPM on NAPI poll

[PATCH net-next v9 0/5] r8169: Temporarily disable ASPM on NAPI poll

[Kai-Heng Feng]

The series is to temporarily disable ASPM on NAPI poll, so the NIC can
"regain" the performace loss when ASPM is enabled. The idea is from
Realtek vendor driver's feature "dynamic ASPM" .

We have "dynamic ASPM" mechanism in Ubuntu 22.04 LTS kernel for quite a
while, and AFAIK it hasn't introduced any regression so far. 

A very similar issue was observed on Realtek wireless NIC, and it was
resolved by disabling ASPM during NAPI poll. So in v8 and v9, we use the
same approach, which is more straightforward, instead of toggling ASPM
based on packet count.

In addition to that, The series also enables ASPM on more systems where
BIOS doesn't grant OS ASPM control.

v8:
https://lore.kernel.org/netdev/20230221023849.1906728-1-kai.heng.feng@canonical.com/

v7:
https://lore.kernel.org/netdev/20211016075442.650311-1-kai.heng.feng@canonical.com/

v6:
https://lore.kernel.org/netdev/20211007161552.272771-1-kai.heng.feng@canonical.com/

v5:
https://lore.kernel.org/netdev/20210916154417.664323-1-kai.heng.feng@canonical.com/

v4:
https://lore.kernel.org/netdev/20210827171452.217123-1-kai.heng.feng@canonical.com/

v3:
https://lore.kernel.org/netdev/20210819054542.608745-1-kai.heng.feng@canonical.com/

v2:
https://lore.kernel.org/netdev/20210812155341.817031-1-kai.heng.feng@canonical.com/

v1:
https://lore.kernel.org/netdev/20210803152823.515849-1-kai.heng.feng@canonical.com/

Kai-Heng Feng (5):
  Revert "PCI/ASPM: Unexport pcie_aspm_support_enabled()"
  PCI/ASPM: Add pcie_aspm_capable() helper
  r8169: Consider chip-specific ASPM can be enabled on more cases
  r8169: Use spinlock to guard config register locking
  r8169: Disable ASPM while doing NAPI poll

 drivers/net/ethernet/realtek/r8169_main.c | 42 +++++++++++++++++++----
 drivers/pci/pcie/aspm.c                   | 12 +++++++
 include/linux/pci.h                       |  2 ++
 3 files changed, 50 insertions(+), 6 deletions(-)

-- 
2.34.1

[PATCH net-next v9 1/5] Revert "PCI/ASPM: Unexport pcie_aspm_support_enabled()"

[Kai-Heng Feng]

This reverts commit ba13d4575da5e656a3cbc18583e0da5c5d865417.

This will be used by module once again.

Signed-off-by: Kai-Heng Feng <kai.heng.feng@canonical.com>
---
v9:
 - No change.

v8:
 - New patch.

 drivers/pci/pcie/aspm.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c
index 4b4184563a927..692d6953f0970 100644
--- a/drivers/pci/pcie/aspm.c
+++ b/drivers/pci/pcie/aspm.c
@@ -1372,3 +1372,4 @@ bool pcie_aspm_support_enabled(void)
 {
 	return aspm_support_enabled;
 }
+EXPORT_SYMBOL(pcie_aspm_support_enabled);
-- 
2.34.1

Re: [PATCH net-next v9 1/5] Revert "PCI/ASPM: Unexport pcie_aspm_support_enabled()"

[Sathyanarayanan Kuppuswamy]

On 2/24/23 7:46 PM, Kai-Heng Feng wrote:
> This reverts commit ba13d4575da5e656a3cbc18583e0da5c5d865417.
> 
> This will be used by module once again.

Since this is a single line change, why revert it? It should be simpler
to export it in the patch that needs it right?

> 
> Signed-off-by: Kai-Heng Feng <kai.heng.feng@canonical.com>
> ---
> v9:
>  - No change.
> 
> v8:
>  - New patch.
> 
>  drivers/pci/pcie/aspm.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c
> index 4b4184563a927..692d6953f0970 100644
> --- a/drivers/pci/pcie/aspm.c
> +++ b/drivers/pci/pcie/aspm.c
> @@ -1372,3 +1372,4 @@ bool pcie_aspm_support_enabled(void)
>  {
>  	return aspm_support_enabled;
>  }
> +EXPORT_SYMBOL(pcie_aspm_support_enabled);

-- 
Sathyanarayanan Kuppuswamy
Linux Kernel Developer

[PATCH net-next v9 2/5] PCI/ASPM: Add pcie_aspm_capable() helper

[Kai-Heng Feng]

Introduce a new helper, pcie_aspm_capable(), to report ASPM capability.

The user will be introduced by next patch.

Acked-by: Bjorn Helgaas <bhelgaas@google.com>
Suggested-by: Bjorn Helgaas <bhelgaas@google.com>
Signed-off-by: Kai-Heng Feng <kai.heng.feng@canonical.com>
---
v9:
v8:
 - No change.

v7:
 - Change subject.

v6:
 - No change.

v5:
 - No change.

v4:
 - Report aspm_capable instead.

v3:
 - This is a new patch

 drivers/pci/pcie/aspm.c | 11 +++++++++++
 include/linux/pci.h     |  2 ++
 2 files changed, 13 insertions(+)

diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c
index 692d6953f0970..d96bf0a362aa2 100644
--- a/drivers/pci/pcie/aspm.c
+++ b/drivers/pci/pcie/aspm.c
@@ -1199,6 +1199,17 @@ bool pcie_aspm_enabled(struct pci_dev *pdev)
 }
 EXPORT_SYMBOL_GPL(pcie_aspm_enabled);
 
+bool pcie_aspm_capable(struct pci_dev *pdev)
+{
+	struct pcie_link_state *link = pcie_aspm_get_link(pdev);
+
+	if (!link)
+		return false;
+
+	return link->aspm_capable;
+}
+EXPORT_SYMBOL_GPL(pcie_aspm_capable);
+
 static ssize_t aspm_attr_show_common(struct device *dev,
 				     struct device_attribute *attr,
 				     char *buf, u8 state)
diff --git a/include/linux/pci.h b/include/linux/pci.h
index adffd65e84b4e..fd56872883e14 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1692,6 +1692,7 @@ int pci_disable_link_state_locked(struct pci_dev *pdev, int state);
 void pcie_no_aspm(void);
 bool pcie_aspm_support_enabled(void);
 bool pcie_aspm_enabled(struct pci_dev *pdev);
+bool pcie_aspm_capable(struct pci_dev *pdev);
 #else
 static inline int pci_disable_link_state(struct pci_dev *pdev, int state)
 { return 0; }
@@ -1700,6 +1701,7 @@ static inline int pci_disable_link_state_locked(struct pci_dev *pdev, int state)
 static inline void pcie_no_aspm(void) { }
 static inline bool pcie_aspm_support_enabled(void) { return false; }
 static inline bool pcie_aspm_enabled(struct pci_dev *pdev) { return false; }
+static inline bool pcie_aspm_capable(struct pci_dev *pdev) { return false; }
 #endif
 
 #ifdef CONFIG_PCIEAER
-- 
2.34.1

Re: [PATCH net-next v9 2/5] PCI/ASPM: Add pcie_aspm_capable() helper

[Sathyanarayanan Kuppuswamy]

On 2/24/23 7:46 PM, Kai-Heng Feng wrote:
> Introduce a new helper, pcie_aspm_capable(), to report ASPM capability.
> 
> The user will be introduced by next patch.

Instead of just saying next patch, just say which driver or use case is
going to use it.

> 
> Acked-by: Bjorn Helgaas <bhelgaas@google.com>
> Suggested-by: Bjorn Helgaas <bhelgaas@google.com>
> Signed-off-by: Kai-Heng Feng <kai.heng.feng@canonical.com>
> ---
> v9:
> v8:
>  - No change.
> 
> v7:
>  - Change subject.
> 
> v6:
>  - No change.
> 
> v5:
>  - No change.
> 
> v4:
>  - Report aspm_capable instead.
> 
> v3:
>  - This is a new patch
> 
>  drivers/pci/pcie/aspm.c | 11 +++++++++++
>  include/linux/pci.h     |  2 ++
>  2 files changed, 13 insertions(+)
> 
> diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c
> index 692d6953f0970..d96bf0a362aa2 100644
> --- a/drivers/pci/pcie/aspm.c
> +++ b/drivers/pci/pcie/aspm.c
> @@ -1199,6 +1199,17 @@ bool pcie_aspm_enabled(struct pci_dev *pdev)
>  }
>  EXPORT_SYMBOL_GPL(pcie_aspm_enabled);
>  
> +bool pcie_aspm_capable(struct pci_dev *pdev)
> +{
> +	struct pcie_link_state *link = pcie_aspm_get_link(pdev);
> +
> +	if (!link)
> +		return false;
> +
> +	return link->aspm_capable;
> +}
> +EXPORT_SYMBOL_GPL(pcie_aspm_capable);
> +
>  static ssize_t aspm_attr_show_common(struct device *dev,
>  				     struct device_attribute *attr,
>  				     char *buf, u8 state)
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index adffd65e84b4e..fd56872883e14 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -1692,6 +1692,7 @@ int pci_disable_link_state_locked(struct pci_dev *pdev, int state);
>  void pcie_no_aspm(void);
>  bool pcie_aspm_support_enabled(void);
>  bool pcie_aspm_enabled(struct pci_dev *pdev);
> +bool pcie_aspm_capable(struct pci_dev *pdev);
>  #else
>  static inline int pci_disable_link_state(struct pci_dev *pdev, int state)
>  { return 0; }
> @@ -1700,6 +1701,7 @@ static inline int pci_disable_link_state_locked(struct pci_dev *pdev, int state)
>  static inline void pcie_no_aspm(void) { }
>  static inline bool pcie_aspm_support_enabled(void) { return false; }
>  static inline bool pcie_aspm_enabled(struct pci_dev *pdev) { return false; }
> +static inline bool pcie_aspm_capable(struct pci_dev *pdev) { return false; }
>  #endif
>  
>  #ifdef CONFIG_PCIEAER

-- 
Sathyanarayanan Kuppuswamy
Linux Kernel Developer

[PATCH net-next v9 3/5] r8169: Consider chip-specific ASPM can be enabled on more cases

[Kai-Heng Feng]

To really enable ASPM on r8169 NICs, both standard PCIe ASPM and
chip-specific ASPM have to be enabled at the same time.

Before enabling ASPM at chip side, make sure the following conditions
are met:
1) Use pcie_aspm_support_enabled() to check if ASPM is disabled by
   kernel parameter.
2) Use pcie_aspm_capable() to see if the device is capable to perform
   PCIe ASPM.
3) Check the return value of pci_disable_link_state(). If it's -EPERM,
   it means BIOS doesn't grant ASPM control to OS, and device should use
   the ASPM setting as is.

Consider ASPM is manageable when those conditions are met.

While at it, disable ASPM at chip-side for TX timeout reset, since
pci_disable_link_state() doesn't have any effect when OS isn't granted
with ASPM control.

Signed-off-by: Kai-Heng Feng <kai.heng.feng@canonical.com>
---
v9:
 - No change.

v8:
 - Enable chip-side ASPM only when PCIe ASPM is already available.
 - Wording.

v7:
 - No change.

v6:
 - Unconditionally enable chip-specific ASPM.

v5:
 - New patch.

 drivers/net/ethernet/realtek/r8169_main.c | 22 ++++++++++++++++++----
 1 file changed, 18 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c
index 45147a1016bec..a857650c2e82b 100644
--- a/drivers/net/ethernet/realtek/r8169_main.c
+++ b/drivers/net/ethernet/realtek/r8169_main.c
@@ -2675,8 +2675,11 @@ static void rtl_disable_exit_l1(struct rtl8169_private *tp)
 
 static void rtl_hw_aspm_clkreq_enable(struct rtl8169_private *tp, bool enable)
 {
-	/* Don't enable ASPM in the chip if OS can't control ASPM */
-	if (enable && tp->aspm_manageable) {
+	/* Skip if PCIe ASPM isn't possible */
+	if (!tp->aspm_manageable)
+		return;
+
+	if (enable) {
 		RTL_W8(tp, Config5, RTL_R8(tp, Config5) | ASPM_en);
 		RTL_W8(tp, Config2, RTL_R8(tp, Config2) | ClkReqEn);
 
@@ -4545,8 +4548,13 @@ static void rtl_task(struct work_struct *work)
 		/* ASPM compatibility issues are a typical reason for tx timeouts */
 		ret = pci_disable_link_state(tp->pci_dev, PCIE_LINK_STATE_L1 |
 							  PCIE_LINK_STATE_L0S);
+
+		/* OS may not be granted to control PCIe ASPM, prevent the driver from using it */
+		tp->aspm_manageable = 0;
+
 		if (!ret)
 			netdev_warn_once(tp->dev, "ASPM disabled on Tx timeout\n");
+
 		goto reset;
 	}
 
@@ -5227,13 +5235,19 @@ static int rtl_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 	 * Chips from RTL8168h partially have issues with L1.2, but seem
 	 * to work fine with L1 and L1.1.
 	 */
-	if (rtl_aspm_is_safe(tp))
+	if (!pcie_aspm_support_enabled() || !pcie_aspm_capable(pdev))
+		rc = -EINVAL;
+	else if (rtl_aspm_is_safe(tp))
 		rc = 0;
 	else if (tp->mac_version >= RTL_GIGA_MAC_VER_46)
 		rc = pci_disable_link_state(pdev, PCIE_LINK_STATE_L1_2);
 	else
 		rc = pci_disable_link_state(pdev, PCIE_LINK_STATE_L1);
-	tp->aspm_manageable = !rc;
+
+	/* -EPERM means BIOS doesn't grant OS ASPM control, ASPM should be use
+	 * as is. Honor it.
+	 */
+	tp->aspm_manageable = (rc == -EPERM) ? 1 : !rc;
 
 	tp->dash_type = rtl_check_dash(tp);
 
-- 
2.34.1

Re: [PATCH net-next v9 3/5] r8169: Consider chip-specific ASPM can be enabled on more cases

[Sathyanarayanan Kuppuswamy]

On 2/24/23 7:46 PM, Kai-Heng Feng wrote:
> To really enable ASPM on r8169 NICs, both standard PCIe ASPM and
> chip-specific ASPM have to be enabled at the same time.
> 
> Before enabling ASPM at chip side, make sure the following conditions
> are met:
> 1) Use pcie_aspm_support_enabled() to check if ASPM is disabled by
>    kernel parameter.
> 2) Use pcie_aspm_capable() to see if the device is capable to perform
>    PCIe ASPM.

Why not club the support check within pcie_aspm_capable()?

> 3) Check the return value of pci_disable_link_state(). If it's -EPERM,
>    it means BIOS doesn't grant ASPM control to OS, and device should use
>    the ASPM setting as is.
> 
> Consider ASPM is manageable when those conditions are met.
> 
> While at it, disable ASPM at chip-side for TX timeout reset, since
> pci_disable_link_state() doesn't have any effect when OS isn't granted
> with ASPM control.
> 
> Signed-off-by: Kai-Heng Feng <kai.heng.feng@canonical.com>
> ---
> v9:
>  - No change.
> 
> v8:
>  - Enable chip-side ASPM only when PCIe ASPM is already available.
>  - Wording.
> 
> v7:
>  - No change.
> 
> v6:
>  - Unconditionally enable chip-specific ASPM.
> 
> v5:
>  - New patch.
> 
>  drivers/net/ethernet/realtek/r8169_main.c | 22 ++++++++++++++++++----
>  1 file changed, 18 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c
> index 45147a1016bec..a857650c2e82b 100644
> --- a/drivers/net/ethernet/realtek/r8169_main.c
> +++ b/drivers/net/ethernet/realtek/r8169_main.c
> @@ -2675,8 +2675,11 @@ static void rtl_disable_exit_l1(struct rtl8169_private *tp)
>  
>  static void rtl_hw_aspm_clkreq_enable(struct rtl8169_private *tp, bool enable)
>  {
> -	/* Don't enable ASPM in the chip if OS can't control ASPM */
> -	if (enable && tp->aspm_manageable) {
> +	/* Skip if PCIe ASPM isn't possible */
> +	if (!tp->aspm_manageable)
> +		return;
> +
> +	if (enable) {
>  		RTL_W8(tp, Config5, RTL_R8(tp, Config5) | ASPM_en);
>  		RTL_W8(tp, Config2, RTL_R8(tp, Config2) | ClkReqEn);
>  
> @@ -4545,8 +4548,13 @@ static void rtl_task(struct work_struct *work)
>  		/* ASPM compatibility issues are a typical reason for tx timeouts */
>  		ret = pci_disable_link_state(tp->pci_dev, PCIE_LINK_STATE_L1 |
>  							  PCIE_LINK_STATE_L0S);
> +
> +		/* OS may not be granted to control PCIe ASPM, prevent the driver from using it */
> +		tp->aspm_manageable = 0;
> +
>  		if (!ret)
>  			netdev_warn_once(tp->dev, "ASPM disabled on Tx timeout\n");
> +
>  		goto reset;
>  	}
>  
> @@ -5227,13 +5235,19 @@ static int rtl_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
>  	 * Chips from RTL8168h partially have issues with L1.2, but seem
>  	 * to work fine with L1 and L1.1.
>  	 */
> -	if (rtl_aspm_is_safe(tp))
> +	if (!pcie_aspm_support_enabled() || !pcie_aspm_capable(pdev))
> +		rc = -EINVAL;
> +	else if (rtl_aspm_is_safe(tp))
>  		rc = 0;
>  	else if (tp->mac_version >= RTL_GIGA_MAC_VER_46)
>  		rc = pci_disable_link_state(pdev, PCIE_LINK_STATE_L1_2);
>  	else
>  		rc = pci_disable_link_state(pdev, PCIE_LINK_STATE_L1);
> -	tp->aspm_manageable = !rc;
> +
> +	/* -EPERM means BIOS doesn't grant OS ASPM control, ASPM should be use
> +	 * as is. Honor it.
> +	 */
> +	tp->aspm_manageable = (rc == -EPERM) ? 1 : !rc;
>  
>  	tp->dash_type = rtl_check_dash(tp);
>  

-- 
Sathyanarayanan Kuppuswamy
Linux Kernel Developer

Re: [PATCH net-next v9 3/5] r8169: Consider chip-specific ASPM can be enabled on more cases

[Bjorn Helgaas]

On Sat, Feb 25, 2023 at 11:46:33AM +0800, Kai-Heng Feng wrote:
> To really enable ASPM on r8169 NICs, both standard PCIe ASPM and
> chip-specific ASPM have to be enabled at the same time.
> 
> Before enabling ASPM at chip side, make sure the following conditions
> are met:
> 1) Use pcie_aspm_support_enabled() to check if ASPM is disabled by
>    kernel parameter.
> 2) Use pcie_aspm_capable() to see if the device is capable to perform
>    PCIe ASPM.
> 3) Check the return value of pci_disable_link_state(). If it's -EPERM,
>    it means BIOS doesn't grant ASPM control to OS, and device should use
>    the ASPM setting as is.
> 
> Consider ASPM is manageable when those conditions are met.
> 
> While at it, disable ASPM at chip-side for TX timeout reset, since
> pci_disable_link_state() doesn't have any effect when OS isn't granted
> with ASPM control.

1) "While at it, ..." is always a hint that maybe this part could be
split to a separate patch.

2) The mix of chip-specific and standard PCIe ASPM configuration is a
mess.  Does it *have* to be intermixed at run-time, or could all the
chip-specific stuff be done once, e.g., maybe chip-specific ASPM
enable could be done at probe-time, and then all subsequent ASPM
configuration could done via the standard PCIe registers?

I.e., does the chip work correctly if chip-specific ASPM is enabled,
but standard PCIe ASPM config is *disabled*?

The ASPM sysfs controls [1] assume that L0s, L1, L1.1, L1.2 can all be
controlled simply by using the standard PCIe registers.  If that's not
the case for r8169, things will break when people use the sysfs knobs.

Bjorn

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/ABI/testing/sysfs-bus-pci?id=v6.2#n420

> Signed-off-by: Kai-Heng Feng <kai.heng.feng@canonical.com>
> ---
> v9:
>  - No change.
> 
> v8:
>  - Enable chip-side ASPM only when PCIe ASPM is already available.
>  - Wording.
> 
> v7:
>  - No change.
> 
> v6:
>  - Unconditionally enable chip-specific ASPM.
> 
> v5:
>  - New patch.
> 
>  drivers/net/ethernet/realtek/r8169_main.c | 22 ++++++++++++++++++----
>  1 file changed, 18 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c
> index 45147a1016bec..a857650c2e82b 100644
> --- a/drivers/net/ethernet/realtek/r8169_main.c
> +++ b/drivers/net/ethernet/realtek/r8169_main.c
> @@ -2675,8 +2675,11 @@ static void rtl_disable_exit_l1(struct rtl8169_private *tp)
>  
>  static void rtl_hw_aspm_clkreq_enable(struct rtl8169_private *tp, bool enable)
>  {
> -	/* Don't enable ASPM in the chip if OS can't control ASPM */
> -	if (enable && tp->aspm_manageable) {
> +	/* Skip if PCIe ASPM isn't possible */
> +	if (!tp->aspm_manageable)
> +		return;
> +
> +	if (enable) {
>  		RTL_W8(tp, Config5, RTL_R8(tp, Config5) | ASPM_en);
>  		RTL_W8(tp, Config2, RTL_R8(tp, Config2) | ClkReqEn);
>  
> @@ -4545,8 +4548,13 @@ static void rtl_task(struct work_struct *work)
>  		/* ASPM compatibility issues are a typical reason for tx timeouts */
>  		ret = pci_disable_link_state(tp->pci_dev, PCIE_LINK_STATE_L1 |
>  							  PCIE_LINK_STATE_L0S);
> +
> +		/* OS may not be granted to control PCIe ASPM, prevent the driver from using it */
> +		tp->aspm_manageable = 0;
> +
>  		if (!ret)
>  			netdev_warn_once(tp->dev, "ASPM disabled on Tx timeout\n");
> +
>  		goto reset;
>  	}
>  
> @@ -5227,13 +5235,19 @@ static int rtl_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
>  	 * Chips from RTL8168h partially have issues with L1.2, but seem
>  	 * to work fine with L1 and L1.1.
>  	 */
> -	if (rtl_aspm_is_safe(tp))
> +	if (!pcie_aspm_support_enabled() || !pcie_aspm_capable(pdev))
> +		rc = -EINVAL;
> +	else if (rtl_aspm_is_safe(tp))
>  		rc = 0;
>  	else if (tp->mac_version >= RTL_GIGA_MAC_VER_46)
>  		rc = pci_disable_link_state(pdev, PCIE_LINK_STATE_L1_2);
>  	else
>  		rc = pci_disable_link_state(pdev, PCIE_LINK_STATE_L1);
> -	tp->aspm_manageable = !rc;
> +
> +	/* -EPERM means BIOS doesn't grant OS ASPM control, ASPM should be use
> +	 * as is. Honor it.
> +	 */
> +	tp->aspm_manageable = (rc == -EPERM) ? 1 : !rc;
>  
>  	tp->dash_type = rtl_check_dash(tp);
>  
> -- 
> 2.34.1
> 

Re: [PATCH net-next v9 3/5] r8169: Consider chip-specific ASPM can be enabled on more cases

[Heiner Kallweit]

On 09.03.2023 21:17, Bjorn Helgaas wrote:
> On Sat, Feb 25, 2023 at 11:46:33AM +0800, Kai-Heng Feng wrote:
>> To really enable ASPM on r8169 NICs, both standard PCIe ASPM and
>> chip-specific ASPM have to be enabled at the same time.
>>
>> Before enabling ASPM at chip side, make sure the following conditions
>> are met:
>> 1) Use pcie_aspm_support_enabled() to check if ASPM is disabled by
>>    kernel parameter.
>> 2) Use pcie_aspm_capable() to see if the device is capable to perform
>>    PCIe ASPM.
>> 3) Check the return value of pci_disable_link_state(). If it's -EPERM,
>>    it means BIOS doesn't grant ASPM control to OS, and device should use
>>    the ASPM setting as is.
>>
>> Consider ASPM is manageable when those conditions are met.
>>
>> While at it, disable ASPM at chip-side for TX timeout reset, since
>> pci_disable_link_state() doesn't have any effect when OS isn't granted
>> with ASPM control.
> 
> 1) "While at it, ..." is always a hint that maybe this part could be
> split to a separate patch.
> 
> 2) The mix of chip-specific and standard PCIe ASPM configuration is a
> mess.  Does it *have* to be intermixed at run-time, or could all the
> chip-specific stuff be done once, e.g., maybe chip-specific ASPM
> enable could be done at probe-time, and then all subsequent ASPM
> configuration could done via the standard PCIe registers?
> 
> I.e., does the chip work correctly if chip-specific ASPM is enabled,
> but standard PCIe ASPM config is *disabled*?
> 
> The ASPM sysfs controls [1] assume that L0s, L1, L1.1, L1.2 can all be
> controlled simply by using the standard PCIe registers.  If that's not
> the case for r8169, things will break when people use the sysfs knobs.
> 
This series has been superseded meanwhile and what is being discussed
here has become obsolete.

> Bjorn
> 
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/ABI/testing/sysfs-bus-pci?id=v6.2#n420
> 
>> Signed-off-by: Kai-Heng Feng <kai.heng.feng@canonical.com>
>> ---
>> v9:
>>  - No change.
>>
>> v8:
>>  - Enable chip-side ASPM only when PCIe ASPM is already available.
>>  - Wording.
>>
>> v7:
>>  - No change.
>>
>> v6:
>>  - Unconditionally enable chip-specific ASPM.
>>
>> v5:
>>  - New patch.
>>
>>  drivers/net/ethernet/realtek/r8169_main.c | 22 ++++++++++++++++++----
>>  1 file changed, 18 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c
>> index 45147a1016bec..a857650c2e82b 100644
>> --- a/drivers/net/ethernet/realtek/r8169_main.c
>> +++ b/drivers/net/ethernet/realtek/r8169_main.c
>> @@ -2675,8 +2675,11 @@ static void rtl_disable_exit_l1(struct rtl8169_private *tp)
>>  
>>  static void rtl_hw_aspm_clkreq_enable(struct rtl8169_private *tp, bool enable)
>>  {
>> -	/* Don't enable ASPM in the chip if OS can't control ASPM */
>> -	if (enable && tp->aspm_manageable) {
>> +	/* Skip if PCIe ASPM isn't possible */
>> +	if (!tp->aspm_manageable)
>> +		return;
>> +
>> +	if (enable) {
>>  		RTL_W8(tp, Config5, RTL_R8(tp, Config5) | ASPM_en);
>>  		RTL_W8(tp, Config2, RTL_R8(tp, Config2) | ClkReqEn);
>>  
>> @@ -4545,8 +4548,13 @@ static void rtl_task(struct work_struct *work)
>>  		/* ASPM compatibility issues are a typical reason for tx timeouts */
>>  		ret = pci_disable_link_state(tp->pci_dev, PCIE_LINK_STATE_L1 |
>>  							  PCIE_LINK_STATE_L0S);
>> +
>> +		/* OS may not be granted to control PCIe ASPM, prevent the driver from using it */
>> +		tp->aspm_manageable = 0;
>> +
>>  		if (!ret)
>>  			netdev_warn_once(tp->dev, "ASPM disabled on Tx timeout\n");
>> +
>>  		goto reset;
>>  	}
>>  
>> @@ -5227,13 +5235,19 @@ static int rtl_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
>>  	 * Chips from RTL8168h partially have issues with L1.2, but seem
>>  	 * to work fine with L1 and L1.1.
>>  	 */
>> -	if (rtl_aspm_is_safe(tp))
>> +	if (!pcie_aspm_support_enabled() || !pcie_aspm_capable(pdev))
>> +		rc = -EINVAL;
>> +	else if (rtl_aspm_is_safe(tp))
>>  		rc = 0;
>>  	else if (tp->mac_version >= RTL_GIGA_MAC_VER_46)
>>  		rc = pci_disable_link_state(pdev, PCIE_LINK_STATE_L1_2);
>>  	else
>>  		rc = pci_disable_link_state(pdev, PCIE_LINK_STATE_L1);
>> -	tp->aspm_manageable = !rc;
>> +
>> +	/* -EPERM means BIOS doesn't grant OS ASPM control, ASPM should be use
>> +	 * as is. Honor it.
>> +	 */
>> +	tp->aspm_manageable = (rc == -EPERM) ? 1 : !rc;
>>  
>>  	tp->dash_type = rtl_check_dash(tp);
>>  
>> -- 
>> 2.34.1
>>

Re: [PATCH net-next v9 3/5] r8169: Consider chip-specific ASPM can be enabled on more cases

[Bjorn Helgaas]

On Fri, Mar 10, 2023 at 09:42:04PM +0100, Heiner Kallweit wrote:
> On 09.03.2023 21:17, Bjorn Helgaas wrote:
> > On Sat, Feb 25, 2023 at 11:46:33AM +0800, Kai-Heng Feng wrote:
> >> To really enable ASPM on r8169 NICs, both standard PCIe ASPM and
> >> chip-specific ASPM have to be enabled at the same time.
> >>
> >> Before enabling ASPM at chip side, make sure the following conditions
> >> are met:
> >> 1) Use pcie_aspm_support_enabled() to check if ASPM is disabled by
> >>    kernel parameter.
> >> 2) Use pcie_aspm_capable() to see if the device is capable to perform
> >>    PCIe ASPM.
> >> 3) Check the return value of pci_disable_link_state(). If it's -EPERM,
> >>    it means BIOS doesn't grant ASPM control to OS, and device should use
> >>    the ASPM setting as is.
> >>
> >> Consider ASPM is manageable when those conditions are met.
> >>
> >> While at it, disable ASPM at chip-side for TX timeout reset, since
> >> pci_disable_link_state() doesn't have any effect when OS isn't granted
> >> with ASPM control.
> > 
> > 1) "While at it, ..." is always a hint that maybe this part could be
> > split to a separate patch.
> > 
> > 2) The mix of chip-specific and standard PCIe ASPM configuration is a
> > mess.  Does it *have* to be intermixed at run-time, or could all the
> > chip-specific stuff be done once, e.g., maybe chip-specific ASPM
> > enable could be done at probe-time, and then all subsequent ASPM
> > configuration could done via the standard PCIe registers?
> > 
> > I.e., does the chip work correctly if chip-specific ASPM is enabled,
> > but standard PCIe ASPM config is *disabled*?
> > 
> > The ASPM sysfs controls [1] assume that L0s, L1, L1.1, L1.2 can all be
> > controlled simply by using the standard PCIe registers.  If that's not
> > the case for r8169, things will break when people use the sysfs knobs.
> > 
> This series has been superseded meanwhile and what is being discussed
> here has become obsolete.

For completeness, I guess the replacement of this series is:

  https://lore.kernel.org/all/af076f1f-a034-82e5-8f76-f3ec32a14eaa@gmail.com/

[PATCH net-next v9 4/5] r8169: Use spinlock to guard config register locking

[Kai-Heng Feng]

Right now r8169 doesn't have parallel access to its config register, but
the next patch makes the driver access config register at anytime.

So add a spinlock to protect the config register from any potential race.

Signed-off-by: Kai-Heng Feng <kai.heng.feng@canonical.com>
---
v9:
 - Replace mutex with spinlock so it can be used in softirq context.

v8:
 - Swap the place when using the mutex. Protect when config register is
   unlocked.

v7:
 - This is a new patch.

 drivers/net/ethernet/realtek/r8169_main.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c
index a857650c2e82b..fb73b5386701f 100644
--- a/drivers/net/ethernet/realtek/r8169_main.c
+++ b/drivers/net/ethernet/realtek/r8169_main.c
@@ -613,6 +613,8 @@ struct rtl8169_private {
 		struct work_struct work;
 	} wk;
 
+	spinlock_t config_lock;
+
 	unsigned supports_gmii:1;
 	unsigned aspm_manageable:1;
 	dma_addr_t counters_phys_addr;
@@ -662,10 +664,12 @@ static inline struct device *tp_to_dev(struct rtl8169_private *tp)
 static void rtl_lock_config_regs(struct rtl8169_private *tp)
 {
 	RTL_W8(tp, Cfg9346, Cfg9346_Lock);
+	spin_unlock_bh(&tp->config_lock);
 }
 
 static void rtl_unlock_config_regs(struct rtl8169_private *tp)
 {
+	spin_lock_bh(&tp->config_lock);
 	RTL_W8(tp, Cfg9346, Cfg9346_Unlock);
 }
 
@@ -5217,6 +5221,8 @@ static int rtl_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 		return rc;
 	}
 
+	spin_lock_init(&tp->config_lock);
+
 	tp->mmio_addr = pcim_iomap_table(pdev)[region];
 
 	xid = (RTL_R32(tp, TxConfig) >> 20) & 0xfcf;
-- 
2.34.1

Re: [PATCH net-next v9 4/5] r8169: Use spinlock to guard config register locking

[Sathyanarayanan Kuppuswamy]

On 2/24/23 7:46 PM, Kai-Heng Feng wrote:
> Right now r8169 doesn't have parallel access to its config register, but
> the next patch makes the driver access config register at anytime.
> 
> So add a spinlock to protect the config register from any potential race.
> 
> Signed-off-by: Kai-Heng Feng <kai.heng.feng@canonical.com>
> ---
> v9:
>  - Replace mutex with spinlock so it can be used in softirq context.
> 
> v8:
>  - Swap the place when using the mutex. Protect when config register is
>    unlocked.
> 
> v7:
>  - This is a new patch.
> 
>  drivers/net/ethernet/realtek/r8169_main.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c
> index a857650c2e82b..fb73b5386701f 100644
> --- a/drivers/net/ethernet/realtek/r8169_main.c
> +++ b/drivers/net/ethernet/realtek/r8169_main.c
> @@ -613,6 +613,8 @@ struct rtl8169_private {
>  		struct work_struct work;
>  	} wk;
>  
> +	spinlock_t config_lock;
> +
>  	unsigned supports_gmii:1;
>  	unsigned aspm_manageable:1;
>  	dma_addr_t counters_phys_addr;
> @@ -662,10 +664,12 @@ static inline struct device *tp_to_dev(struct rtl8169_private *tp)
>  static void rtl_lock_config_regs(struct rtl8169_private *tp)
>  {
>  	RTL_W8(tp, Cfg9346, Cfg9346_Lock);
> +	spin_unlock_bh(&tp->config_lock);

You mean spin_lock_bh() here right?

>  }
>  
>  static void rtl_unlock_config_regs(struct rtl8169_private *tp)
>  {
> +	spin_lock_bh(&tp->config_lock);
>  	RTL_W8(tp, Cfg9346, Cfg9346_Unlock);
>  }
>  
> @@ -5217,6 +5221,8 @@ static int rtl_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
>  		return rc;
>  	}
>  
> +	spin_lock_init(&tp->config_lock);
> +
>  	tp->mmio_addr = pcim_iomap_table(pdev)[region];
>  
>  	xid = (RTL_R32(tp, TxConfig) >> 20) & 0xfcf;

-- 
Sathyanarayanan Kuppuswamy
Linux Kernel Developer

[PATCH net-next v9 5/5] r8169: Disable ASPM while doing NAPI poll

[Kai-Heng Feng]

NAPI poll of Realtek NICs don't seem to perform well ASPM is enabled.
The vendor driver uses a mechanism called "dynamic ASPM" to toggle ASPM
based on the packet number in given time period.

Instead of implementing "dynamic ASPM", use a more straightforward way
by disabling ASPM during NAPI poll, as a similar approach was
implemented to solve slow performance on Realtek wireless NIC, see
commit 24f5e38a13b5 ("rtw88: Disable PCIe ASPM while doing NAPI poll on
8821CE").

Since NAPI poll should be handled as fast as possible, also remove the
delay in rtl_hw_aspm_clkreq_enable() which was added by commit
94235460f9eaef ("r8169: Align ASPM/CLKREQ setting function with vendor
driver").

Signed-off-by: Kai-Heng Feng <kai.heng.feng@canonical.com>
---
v9:
 - No change.

v8:
 - New patch.

 drivers/net/ethernet/realtek/r8169_main.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c
index fb73b5386701f..4e874fa661852 100644
--- a/drivers/net/ethernet/realtek/r8169_main.c
+++ b/drivers/net/ethernet/realtek/r8169_main.c
@@ -2711,8 +2711,6 @@ static void rtl_hw_aspm_clkreq_enable(struct rtl8169_private *tp, bool enable)
 		RTL_W8(tp, Config2, RTL_R8(tp, Config2) & ~ClkReqEn);
 		RTL_W8(tp, Config5, RTL_R8(tp, Config5) & ~ASPM_en);
 	}
-
-	udelay(10);
 }
 
 static void rtl_set_fifo_size(struct rtl8169_private *tp, u16 rx_stat,
@@ -4577,6 +4575,12 @@ static int rtl8169_poll(struct napi_struct *napi, int budget)
 	struct net_device *dev = tp->dev;
 	int work_done;
 
+	if (tp->aspm_manageable) {
+		rtl_unlock_config_regs(tp);
+		rtl_hw_aspm_clkreq_enable(tp, false);
+		rtl_lock_config_regs(tp);
+	}
+
 	rtl_tx(dev, tp, budget);
 
 	work_done = rtl_rx(dev, tp, budget);
@@ -4584,6 +4588,12 @@ static int rtl8169_poll(struct napi_struct *napi, int budget)
 	if (work_done < budget && napi_complete_done(napi, work_done))
 		rtl_irq_enable(tp);
 
+	if (tp->aspm_manageable) {
+		rtl_unlock_config_regs(tp);
+		rtl_hw_aspm_clkreq_enable(tp, true);
+		rtl_lock_config_regs(tp);
+	}
+
 	return work_done;
 }
 
-- 
2.34.1