[PATCH net-next resubmit v2] r8169: disable ASPM in case of tx timeout

[PATCH net-next resubmit v2] r8169: disable ASPM in case of tx timeout

[Heiner Kallweit]

There are still single reports of systems where ASPM incompatibilities
cause tx timeouts. It's not clear whom to blame, so let's disable
ASPM in case of a tx timeout.

v2:
- add one-time warning for informing the user

Signed-off-by: Heiner Kallweit <hkallweit1@gmail.com>
---
 drivers/net/ethernet/realtek/r8169_main.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c
index a9dcc98b6..49c124d8e 100644
--- a/drivers/net/ethernet/realtek/r8169_main.c
+++ b/drivers/net/ethernet/realtek/r8169_main.c
@@ -576,6 +576,7 @@ struct rtl8169_tc_offsets {
 enum rtl_flag {
 	RTL_FLAG_TASK_ENABLED = 0,
 	RTL_FLAG_TASK_RESET_PENDING,
+	RTL_FLAG_TASK_TX_TIMEOUT,
 	RTL_FLAG_MAX
 };
 
@@ -3931,7 +3932,7 @@ static void rtl8169_tx_timeout(struct net_device *dev, unsigned int txqueue)
 {
 	struct rtl8169_private *tp = netdev_priv(dev);
 
-	rtl_schedule_task(tp, RTL_FLAG_TASK_RESET_PENDING);
+	rtl_schedule_task(tp, RTL_FLAG_TASK_TX_TIMEOUT);
 }
 
 static int rtl8169_tx_map(struct rtl8169_private *tp, const u32 *opts, u32 len,
@@ -4525,6 +4526,7 @@ static void rtl_task(struct work_struct *work)
 {
 	struct rtl8169_private *tp =
 		container_of(work, struct rtl8169_private, wk.work);
+	int ret;
 
 	rtnl_lock();
 
@@ -4532,7 +4534,17 @@ static void rtl_task(struct work_struct *work)
 	    !test_bit(RTL_FLAG_TASK_ENABLED, tp->wk.flags))
 		goto out_unlock;
 
+	if (test_and_clear_bit(RTL_FLAG_TASK_TX_TIMEOUT, tp->wk.flags)) {
+		/* ASPM compatibility issues are a typical reason for tx timeouts */
+		ret = pci_disable_link_state(tp->pci_dev, PCIE_LINK_STATE_L1 |
+							  PCIE_LINK_STATE_L0S);
+		if (!ret)
+			netdev_warn_once(tp->dev, "ASPM disabled on Tx timeout\n");
+		goto reset;
+	}
+
 	if (test_and_clear_bit(RTL_FLAG_TASK_RESET_PENDING, tp->wk.flags)) {
+reset:
 		rtl_reset_work(tp);
 		netif_wake_queue(tp->dev);
 	}
-- 
2.39.0

Re: [PATCH net-next resubmit v2] r8169: disable ASPM in case of tx timeout

[Alexander H Duyck]

On Tue, 2023-01-10 at 23:03 +0100, Heiner Kallweit wrote:
> There are still single reports of systems where ASPM incompatibilities
> cause tx timeouts. It's not clear whom to blame, so let's disable
> ASPM in case of a tx timeout.
> 
> v2:
> - add one-time warning for informing the user
> 
> Signed-off-by: Heiner Kallweit <hkallweit1@gmail.com>

From past experience I have seen ASPM issues cause the device to
disappear from the bus after failing to come out of L1. If that occurs
this won't be able to recover after the timeout without resetting the
bus itself. As such it may be necessary to disable the link states
prior to using the device rather than waiting until after the error.
That can be addressed in a follow-on patch if this doesn't resolve the
issue.

As for the code it looks fine to me.

Reviewed-by: Alexander Duyck <alexanderduyck@fb.com>

Re: [PATCH net-next resubmit v2] r8169: disable ASPM in case of tx timeout

[Heiner Kallweit]

On 11.01.2023 17:16, Alexander H Duyck wrote:
> On Tue, 2023-01-10 at 23:03 +0100, Heiner Kallweit wrote:
>> There are still single reports of systems where ASPM incompatibilities
>> cause tx timeouts. It's not clear whom to blame, so let's disable
>> ASPM in case of a tx timeout.
>>
>> v2:
>> - add one-time warning for informing the user
>>
>> Signed-off-by: Heiner Kallweit <hkallweit1@gmail.com>
> 
>>From past experience I have seen ASPM issues cause the device to
> disappear from the bus after failing to come out of L1. If that occurs
> this won't be able to recover after the timeout without resetting the
> bus itself. As such it may be necessary to disable the link states
> prior to using the device rather than waiting until after the error.
> That can be addressed in a follow-on patch if this doesn't resolve the
> issue.
> 

Interesting, reports about disappearing devices I haven't seen yet.
Symptoms I've seen differ, based on combination of more or less faulty
NIC chipset version, BIOS bugs, PCIe mainboard chipset.
Typically users experienced missed rx packets, tx timeouts or NIC lockups.
Disabling ASPM resulted in complaints of notebook users about reduced
system runtime on battery.
Meanwhile we found a good balance and reports about ASPM issues
became quite rare.
Just L1.2 still causes issues under load even with newer chipset versions,
therefore L1.2 is disabled per default.

> As for the code it looks fine to me.
> 
> Reviewed-by: Alexander Duyck <alexanderduyck@fb.com>

Re: [PATCH net-next resubmit v2] r8169: disable ASPM in case of tx timeout

[Alexander Duyck]

On Wed, Jan 11, 2023 at 12:17 PM Heiner Kallweit <hkallweit1@gmail.com> wrote:
>
> On 11.01.2023 17:16, Alexander H Duyck wrote:
> > On Tue, 2023-01-10 at 23:03 +0100, Heiner Kallweit wrote:
> >> There are still single reports of systems where ASPM incompatibilities
> >> cause tx timeouts. It's not clear whom to blame, so let's disable
> >> ASPM in case of a tx timeout.
> >>
> >> v2:
> >> - add one-time warning for informing the user
> >>
> >> Signed-off-by: Heiner Kallweit <hkallweit1@gmail.com>
> >
> >>From past experience I have seen ASPM issues cause the device to
> > disappear from the bus after failing to come out of L1. If that occurs
> > this won't be able to recover after the timeout without resetting the
> > bus itself. As such it may be necessary to disable the link states
> > prior to using the device rather than waiting until after the error.
> > That can be addressed in a follow-on patch if this doesn't resolve the
> > issue.
> >
>
> Interesting, reports about disappearing devices I haven't seen yet.
> Symptoms I've seen differ, based on combination of more or less faulty
> NIC chipset version, BIOS bugs, PCIe mainboard chipset.
> Typically users experienced missed rx packets, tx timeouts or NIC lockups.
> Disabling ASPM resulted in complaints of notebook users about reduced
> system runtime on battery.
> Meanwhile we found a good balance and reports about ASPM issues
> became quite rare.
> Just L1.2 still causes issues under load even with newer chipset versions,
> therefore L1.2 is disabled per default.

Does your driver do any checking for MMIO failures on reads? Basically
when the device disappears it should start returning ~0 on mmio reads.
The device itself doesn't disappear, but it doesn't respond to
requests anymore so it might be the "NIC lockups" case you mentioned.
The Intel parts would disappear as they would trigger their "surprise
removal" logic which would detach the netdevice. I have seen that
issue on some platforms. It is kind of interesting when you can
actually watch it happen as the issue was essentially a marginal PCIe
connection so it would start out at x4, then renegotiate down with
each ASPM L1 link bounce, and eventually it would end up at x1 before
just dropping off the bus.

I agree pro-actively disabling ASPM is bad for power savings. So if
this approach can resolve it then I am more than willing to give it a
try. My main concern is if MMIO is already borked, updating the ASPM
settings may not be enough to bring it back and it may require a
secondary bus reset.

Re: [PATCH net-next resubmit v2] r8169: disable ASPM in case of tx timeout

[Heiner Kallweit]

On 11.01.2023 23:38, Alexander Duyck wrote:
> On Wed, Jan 11, 2023 at 12:17 PM Heiner Kallweit <hkallweit1@gmail.com> wrote:
>>
>> On 11.01.2023 17:16, Alexander H Duyck wrote:
>>> On Tue, 2023-01-10 at 23:03 +0100, Heiner Kallweit wrote:
>>>> There are still single reports of systems where ASPM incompatibilities
>>>> cause tx timeouts. It's not clear whom to blame, so let's disable
>>>> ASPM in case of a tx timeout.
>>>>
>>>> v2:
>>>> - add one-time warning for informing the user
>>>>
>>>> Signed-off-by: Heiner Kallweit <hkallweit1@gmail.com>
>>>
>>> >From past experience I have seen ASPM issues cause the device to
>>> disappear from the bus after failing to come out of L1. If that occurs
>>> this won't be able to recover after the timeout without resetting the
>>> bus itself. As such it may be necessary to disable the link states
>>> prior to using the device rather than waiting until after the error.
>>> That can be addressed in a follow-on patch if this doesn't resolve the
>>> issue.
>>>
>>
>> Interesting, reports about disappearing devices I haven't seen yet.
>> Symptoms I've seen differ, based on combination of more or less faulty
>> NIC chipset version, BIOS bugs, PCIe mainboard chipset.
>> Typically users experienced missed rx packets, tx timeouts or NIC lockups.
>> Disabling ASPM resulted in complaints of notebook users about reduced
>> system runtime on battery.
>> Meanwhile we found a good balance and reports about ASPM issues
>> became quite rare.
>> Just L1.2 still causes issues under load even with newer chipset versions,
>> therefore L1.2 is disabled per default.
> 
> Does your driver do any checking for MMIO failures on reads? Basically
> when the device disappears it should start returning ~0 on mmio reads.

Not yet, good idea.

> The device itself doesn't disappear, but it doesn't respond to
> requests anymore so it might be the "NIC lockups" case you mentioned.
> The Intel parts would disappear as they would trigger their "surprise
> removal" logic which would detach the netdevice. I have seen that
> issue on some platforms. It is kind of interesting when you can
> actually watch it happen as the issue was essentially a marginal PCIe
> connection so it would start out at x4, then renegotiate down with
> each ASPM L1 link bounce, and eventually it would end up at x1 before
> just dropping off the bus.
> 
> I agree pro-actively disabling ASPM is bad for power savings. So if
> this approach can resolve it then I am more than willing to give it a
> try. My main concern is if MMIO is already borked, updating the ASPM
> settings may not be enough to bring it back and it may require a
> secondary bus reset.

I'll think about how to further improve recovery in case of ASPM issues.

Re: [PATCH net-next resubmit v2] r8169: disable ASPM in case of tx timeout

[patchwork-bot+netdevbpf]

Hello:

This patch was applied to netdev/net-next.git (master)
by Jakub Kicinski <kuba@kernel.org>:

On Tue, 10 Jan 2023 23:03:18 +0100 you wrote:
> There are still single reports of systems where ASPM incompatibilities
> cause tx timeouts. It's not clear whom to blame, so let's disable
> ASPM in case of a tx timeout.
> 
> v2:
> - add one-time warning for informing the user
> 
> [...]

Here is the summary with links:
  - [net-next,resubmit,v2] r8169: disable ASPM in case of tx timeout
    https://git.kernel.org/netdev/net-next/c/80c0576ef179

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html