Re: [WITHDRAW PATCH v1 1/1] net: r8169: fix the pci setup so the Realtek RTL8111/8168/8411 ethernet speeds up

[Heiner Kallweit <hkallweit1@gmail.com>]

On 30.04.2023 01:09, Mirsad Goran Todorovac wrote:
> On 25. 04. 2023. 13:44, Mirsad Goran Todorovac wrote:
>> It was noticed that Ookla Speedtest had shown only 250 Mbps download and
>> 310 Mbps upload where Windows 10 on the same box showed 440/310 Mbps, which
>> is the link capacity.
>>
>> This article: https://www.phoronix.com/news/Intel-i219-LM-Linux-60p-Fix
>> inspired to check our speeds. (Previously I used to think it was a network
>> congestion, or reduction on our ISP, but now each time Windows 10 downlink
>> speed is 440 compared to 250 Mbps in Linuxes Linux is performing at 60% of
>> the speed.)
>>
>> The latest 6.3 kernel shows 95% speed up with this patch as compared to the
>> same commit without it:
>>
>> ::::::::::::::
>> speedtest/6.3.0-00436-g173ea743bf7a-dirty-1
>> ::::::::::::::
>> [marvin@pc-mtodorov ~]$ speedtest -s 41437
>>
>>    Speedtest by Ookla
>>
>>       Server: A1 Hrvatska d.o.o. - Zagreb (id: 41437)
>>          ISP: Croatian Academic and Research Network
>> Idle Latency:     1.53 ms   (jitter: 0.15ms, low: 1.30ms, high: 1.71ms)
>>     Download:   225.13 Mbps (data used: 199.3 MB)
>>                   1.65 ms   (jitter: 20.15ms, low: 0.81ms, high: 418.27ms)
>>       Upload:   350.00 Mbps (data used: 157.9 MB)
>>                   3.35 ms   (jitter: 19.46ms, low: 1.61ms, high: 474.55ms)
>>  Packet Loss:     0.0%
>>   Result URL: https://www.speedtest.net/result/c/a0084fd8-c275-4019-899a-a1590e49a34b
>> [marvin@pc-mtodorov ~]$ speedtest -s 41437
>>
>>    Speedtest by Ookla
>>
>>       Server: A1 Hrvatska d.o.o. - Zagreb (id: 41437)
>>          ISP: Croatian Academic and Research Network
>> Idle Latency:     1.54 ms   (jitter: 0.28ms, low: 1.17ms, high: 1.64ms)
>>     Download:   222.88 Mbps (data used: 207.9 MB)
>>                  10.23 ms   (jitter: 31.76ms, low: 0.75ms, high: 353.79ms)
>>       Upload:   349.91 Mbps (data used: 157.7 MB)
>>                   3.27 ms   (jitter: 13.05ms, low: 1.67ms, high: 236.76ms)
>>  Packet Loss:     0.0%
>>   Result URL: https://www.speedtest.net/result/c/f4c663ba-830d-44c6-8033-ce3b3b818c42
>> [marvin@pc-mtodorov ~]$
>> ::::::::::::::
>> speedtest/6.3.0-r8169-00437-g323fe5352af6-dirty-2
>> ::::::::::::::
>> [marvin@pc-mtodorov ~]$ speedtest -s 41437
>>
>>    Speedtest by Ookla
>>
>>       Server: A1 Hrvatska d.o.o. - Zagreb (id: 41437)
>>          ISP: Croatian Academic and Research Network
>> Idle Latency:     0.84 ms   (jitter: 0.05ms, low: 0.82ms, high: 0.93ms)
>>     Download:   432.37 Mbps (data used: 360.5 MB)
>>                 142.43 ms   (jitter: 76.45ms, low: 1.02ms, high: 1105.19ms)
>>       Upload:   346.29 Mbps (data used: 164.6 MB)
>>                   7.72 ms   (jitter: 29.80ms, low: 0.92ms, high: 283.48ms)
>>  Packet Loss:    12.8%
>>   Result URL: https://www.speedtest.net/result/c/e473359e-c37e-4f29-aa9f-4b008210cf7c
>> [marvin@pc-mtodorov ~]$ speedtest -s 41437
>>
>>    Speedtest by Ookla
>>
>>       Server: A1 Hrvatska d.o.o. - Zagreb (id: 41437)
>>          ISP: Croatian Academic and Research Network
>> Idle Latency:     0.82 ms   (jitter: 0.16ms, low: 0.75ms, high: 1.05ms)
>>     Download:   440.97 Mbps (data used: 427.5 MB)
>>                  72.50 ms   (jitter: 52.89ms, low: 0.91ms, high: 865.08ms)
>>       Upload:   342.75 Mbps (data used: 166.6 MB)
>>                   3.26 ms   (jitter: 22.93ms, low: 1.07ms, high: 239.41ms)
>>  Packet Loss:    13.4%
>>   Result URL: https://www.speedtest.net/result/c/f393e149-38d4-4a34-acc4-5cf81ff13708
>>
>> 440 Mbps is the speed achieved in Windows 10, and Linux 6.3 with
>> the patch, while 225 Mbps without this patch is running at 51% of
>> the nominal speed with the same hardware and Linux kernel commit.
>>
>> Cc: David S. Miller <davem@davemloft.net>
>> Cc: Eric Dumazet <edumazet@google.com>
>> Cc: Jakub Kicinski <kuba@kernel.org>
>> Cc: Paolo Abeni <pabeni@redhat.com>
>> Cc: nic_swsd@realtek.com
>> Cc: netdev@vger.kernel.org
>> Link: https://bugzilla.redhat.com/show_bug.cgi?id=1671958#c60
>> Suggested-by: Heiner Kallweit <hkallweit1@gmail.com>
>> Signed-off-by: Mirsad Goran Todorovac <mirsad.todorovac@alu.unizg.hr>
>> ---
>>  drivers/net/ethernet/realtek/r8169_main.c | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c
>> index 45147a1016be..b8a04301d130 100644
>> --- a/drivers/net/ethernet/realtek/r8169_main.c
>> +++ b/drivers/net/ethernet/realtek/r8169_main.c
>> @@ -3239,6 +3239,7 @@ static void rtl_hw_start_8168h_1(struct rtl8169_private *tp)
>>  	r8168_mac_ocp_write(tp, 0xc094, 0x0000);
>>  	r8168_mac_ocp_write(tp, 0xc09e, 0x0000);
>>  
>> +	pci_disable_link_state(tp->pci_dev, PCIE_LINK_STATE_CLKPM);
>>  	rtl_hw_aspm_clkreq_enable(tp, true);
>>  }
>>  
> 
> After some additional research, I came to the obvious realisation, reading more
> thoroughly the discussion at the link - that the above patch did not work for
> all Realtek RTL819x cards back then.
> 
> My version, the RTL8168h/8111h indeed works 95% faster on the 6.3 Linux kernel,
> but I cannot speak for the people with the power management problems and
> battery life issues ... and the concerns explained here: https://github.com/KastB/r8169
> 
> [root@pc-mtodorov marvin]# dmesg | grep RTL
> [    7.304130] r8169 0000:01:00.0 eth0: RTL8168h/8111h, f4:93:9f:f0:a5:f5, XID 541, IRQ 123
> 
> Currently there seem to be  at least 43 revisions of the RTL816x cards and firmware,
> and I really cannot test on all of them.
> 
> I will test the other Heiner's experimental patch, but it seems to disable ASPM completely,
> while for my Lenovo desktop with RTL8168h/8111h disabling CLKPM alone.
> 
> However, further homework revealed that the kernel patch is unnecessary, as the same
> effect can be achieved in runtime by the sysfs parm introduced with THIS PATCH:
> https://patchwork.kernel.org/project/linux-pci/patch/b1c83f8a-9bf6-eac5-82d0-cf5b90128fbf@gmail.com/
> which was solved 3 1/2 years ago, but the default on my AlmaLinux 8.7 and Lenovo desktop
> box 10TX000VCR was the 53% of the link capacity and speed.
> 
> (I don't know if the card would operate with 220 Mbps on a Gigabit link, it was
> not tested.)
> 
> [marvin@pc-mtodorov ~]$ speedtest -s 41437
> 
>    Speedtest by Ookla
> 
>       Server: A1 Hrvatska d.o.o. - Zagreb (id: 41437)
>          ISP: Croatian Academic and Research Network
> Idle Latency:     1.44 ms   (jitter: 0.23ms, low: 1.20ms, high: 1.65ms)
>     Download:   220.62 Mbps (data used: 214.2 MB)                                                   
>                  22.01 ms   (jitter: 36.04ms, low: 0.84ms, high: 817.47ms)
>       Upload:   346.86 Mbps (data used: 169.1 MB)                                                   
>                   3.32 ms   (jitter: 12.12ms, low: 0.87ms, high: 221.69ms)
>  Packet Loss:     0.6%
>   Result URL: https://www.speedtest.net/result/c/20c546e7-0b8f-4a2e-a669-a597bb5aee36
> [marvin@pc-mtodorov ~]$ sudo bash
> [sudo] password for marvin: 
> [root@pc-mtodorov marvin]# cat /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/link/clkpm
> 1
> [root@pc-mtodorov marvin]# echo 0 > /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/link/clkpm
> [root@pc-mtodorov marvin]# cat /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/link/clkpm
> 0
> [root@pc-mtodorov marvin]# speedtest -s 41437
> 
>    Speedtest by Ookla
> 
>       Server: A1 Hrvatska d.o.o. - Zagreb (id: 41437)
>          ISP: Croatian Academic and Research Network
> Idle Latency:     0.85 ms   (jitter: 0.06ms, low: 0.78ms, high: 0.92ms)
>     Download:   431.13 Mbps (data used: 341.0 MB)                                                   
>                 157.40 ms   (jitter: 68.09ms, low: 0.88ms, high: 823.19ms)
>       Upload:   351.36 Mbps (data used: 158.3 MB)                                                   
>                   2.88 ms   (jitter: 6.24ms, low: 1.41ms, high: 209.74ms)
>  Packet Loss:    13.4%
>   Result URL: https://www.speedtest.net/result/c/ff695466-3ac7-405e-8cae-0a85c2c3d5cd
> [root@pc-mtodorov marvin]# 
> 
> The clkpm setting can be reversed back to 1, causing the RTL speed to drop again.
> 
> So, the patch is withdrawn as unnecessary, even when the majority of RTL8168h/8111h
> and possibly other Realtek Gigabit cards will by default run at sub-Gigabit speeds.
> 

RTL8168h doesn't need the CLKPM quirk in general. E.g. my test system runs fine w/o it
at 950Mbps. Seems that ASPM is broken on your system.
Alternatively you can test with latest linux-next, it disables ASPM during NAPI poll.

> Thank you for your time.
> 
> Best regards,
> Mirsad
> 
Heiner