[BUG] MTK SoC Ethernet throughput TX only ~620Mbit/s since 6.2-rc1

[BUG] MTK SoC Ethernet throughput TX only ~620Mbit/s since 6.2-rc1

[Frank Wunderlich]

Hi,

i've noticed that beginning on 6.2-rc1 the throughput on my Bananapi-R2 and Bananapi-R3 goes from 940Mbit/s down do 620Mbit/s since 6.2-rc1.
Only TX (from SBC PoV) is affected, RX is still 940Mbit/s.

i bisected this to this commit:

f63959c7eec3151c30a2ee0d351827b62e742dcb ("net: ethernet: mtk_eth_soc: implement multi-queue support for per-port queues")

Daniel reported me that this is known so far and they need assistance from MTK and i should report it officially.

As far as i understand it the commit should fix problems with clients using non-GBE speeds (10/100 Mbit/s) on the Gbit-capable dsa
interfaces (mt753x connected) behind the mac, but now the Gigabit speed is no more reached.
I see no CRC/dropped packets, retransmitts or similar.

after reverting the commit above i get 940Mbit like in rx direction, but this will introduce the problems mentioned above so this not a complete fix.

example output before revert on mt7623/bpi.r2:

root@bpi-r2:~# iperf3 -c 192.168.0.21
Connecting to host 192.168.0.21, port 5201
[ 5] local 192.168.0.11 port 48882 connected to 192.168.0.21 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 75.3 MBytes 632 Mbits/sec 0 396 KBytes
[ 5] 1.00-2.00 sec 74.3 MBytes 623 Mbits/sec 0 396 KBytes
[ 5] 2.00-3.00 sec 74.6 MBytes 625 Mbits/sec 0 396 KBytes
[ 5] 3.00-4.00 sec 73.9 MBytes 620 Mbits/sec 0 396 KBytes
[ 5] 4.00-5.00 sec 74.6 MBytes 626 Mbits/sec 0 396 KBytes
[ 5] 5.00-6.00 sec 74.4 MBytes 624 Mbits/sec 0 396 KBytes
[ 5] 6.00-7.00 sec 74.4 MBytes 624 Mbits/sec 0 396 KBytes
[ 5] 7.00-8.00 sec 74.4 MBytes 624 Mbits/sec 0 396 KBytes
[ 5] 8.00-9.00 sec 73.9 MBytes 620 Mbits/sec 0 396 KBytes
[ 5] 9.00-10.00 sec 74.6 MBytes 626 Mbits/sec 0 396 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 745 MBytes 625 Mbits/sec 0 sender
[ 5] 0.00-10.05 sec 744 MBytes 621 Mbits/sec receiver

iperf Done.
root@bpi-r2:~# ethtool -S eth0 | grep -v ': 0'
NIC statistics:
tx_bytes: 819999267
tx_packets: 538815
rx_bytes: 18338089
rx_packets: 261984
p06_TxUnicast: 261974
p06_TxMulticast: 10
p06_TxPktSz65To127: 261983
p06_TxPktSz256To511: 1
p06_TxBytes: 19386025
p06_RxFiltering: 13
p06_RxUnicast: 538783
p06_RxMulticast: 31
p06_RxBroadcast: 1
p06_RxPktSz64: 3
p06_RxPktSz65To127: 47
p06_RxPktSz128To255: 2
p06_RxPktSz256To511: 1
p06_RxPktSz512To1023: 2
p06_RxPktSz1024ToMax: 538760
p06_RxBytes: 819999267

Hope i've forgot no information for now :)

regards Frank

Re: [BUG] MTK SoC Ethernet throughput TX only ~620Mbit/s since 6.2-rc1

[Linux regression tracking #adding (Thorsten Leemhuis)]

[CCing the regression list, as it should be in the loop for regressions:
https://docs.kernel.org/admin-guide/reporting-regressions.html]

[TLDR: I'm adding this report to the list of tracked Linux kernel
regressions; the text you find below is based on a few templates
paragraphs you might have encountered already in similar form.
See link in footer if these mails annoy you.]

On 07.03.23 19:32, Frank Wunderlich wrote:
> 
> i've noticed that beginning on 6.2-rc1 the throughput on my Bananapi-R2 and Bananapi-R3 goes from 940Mbit/s down do 620Mbit/s since 6.2-rc1.
> Only TX (from SBC PoV) is affected, RX is still 940Mbit/s.
> 
> i bisected this to this commit:
> 
> f63959c7eec3151c30a2ee0d351827b62e742dcb ("net: ethernet: mtk_eth_soc: implement multi-queue support for per-port queues")
> [...]

Thanks for the report. To be sure the issue doesn't fall through the
cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression
tracking bot:

#regzbot ^introduced f63959c7eec315
#regzbot title net: MTK SoC Ethernet throughput TX only ~620Mbit/s since
6.2-rc1
#regzbot ignore-activity

This isn't a regression? This issue or a fix for it are already
discussed somewhere else? It was fixed already? You want to clarify when
the regression started to happen? Or point out I got the title or
something else totally wrong? Then just reply and tell me -- ideally
while also telling regzbot about it, as explained by the page listed in
the footer of this mail.

Developers: When fixing the issue, remember to add 'Link:' tags pointing
to the report (the parent of this mail). See page linked in footer for
details.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

Re: [BUG] MTK SoC Ethernet throughput TX only ~620Mbit/s since 6.2-rc1

[Felix Fietkau]

On 07.03.23 19:32, Frank Wunderlich wrote:
> Hi,
> 
> i've noticed that beginning on 6.2-rc1 the throughput on my Bananapi-R2 and Bananapi-R3 goes from 940Mbit/s down do 620Mbit/s since 6.2-rc1.
> Only TX (from SBC PoV) is affected, RX is still 940Mbit/s.
> 
> i bisected this to this commit:
> 
> f63959c7eec3151c30a2ee0d351827b62e742dcb ("net: ethernet: mtk_eth_soc: implement multi-queue support for per-port queues")
> 
> Daniel reported me that this is known so far and they need assistance from MTK and i should report it officially.
> 
> As far as i understand it the commit should fix problems with clients using non-GBE speeds (10/100 Mbit/s) on the Gbit-capable dsa
> interfaces (mt753x connected) behind the mac, but now the Gigabit speed is no more reached.
> I see no CRC/dropped packets, retransmitts or similar.
> 
> after reverting the commit above i get 940Mbit like in rx direction, but this will introduce the problems mentioned above so this not a complete fix.
I don't have a BPI-R2, but I tested on BPI-R3 and can't reproduce this 
issue. Do you see it on all ports, or only on specific ones?

- Felix

Re: [BUG] MTK SoC Ethernet throughput TX only ~620Mbit/s since 6.2-rc1

[Daniel Golle]

Hi Felix,

On Tue, Mar 14, 2023 at 11:30:53AM +0100, Felix Fietkau wrote:
> On 07.03.23 19:32, Frank Wunderlich wrote:
> > Hi,
> > 
> > i've noticed that beginning on 6.2-rc1 the throughput on my Bananapi-R2 and Bananapi-R3 goes from 940Mbit/s down do 620Mbit/s since 6.2-rc1.
> > Only TX (from SBC PoV) is affected, RX is still 940Mbit/s.
> > 
> > i bisected this to this commit:
> > 
> > f63959c7eec3151c30a2ee0d351827b62e742dcb ("net: ethernet: mtk_eth_soc: implement multi-queue support for per-port queues")
> > 
> > Daniel reported me that this is known so far and they need assistance from MTK and i should report it officially.
> > 
> > As far as i understand it the commit should fix problems with clients using non-GBE speeds (10/100 Mbit/s) on the Gbit-capable dsa
> > interfaces (mt753x connected) behind the mac, but now the Gigabit speed is no more reached.
> > I see no CRC/dropped packets, retransmitts or similar.
> > 
> > after reverting the commit above i get 940Mbit like in rx direction, but this will introduce the problems mentioned above so this not a complete fix.
> I don't have a BPI-R2, but I tested on BPI-R3 and can't reproduce this
> issue. Do you see it on all ports, or only on specific ones?

I also can't reproduce this if unsing any of the gigE ports wired via
MT7531 on the R3. However, I can reproduce the issue if using a 1 GBit/s
SFP module in slot SFP1 of the R3 (connected directly to GMAC2/eth1).

Users have reported this also to be a problem also on MT7622 on devices
directly connecting a PHY (and not using MT7531).

In all cases, reverting the commit above fixes the issue.


Cheers


Daniel

Aw: Re: [BUG] MTK SoC Ethernet throughput TX only ~620Mbit/s since 6.2-rc1

[Frank Wunderlich]

Hi,
> Gesendet: Dienstag, 14. März 2023 um 13:47 Uhr
> Von: "Daniel Golle" <daniel@makrotopia.org>
> An: "Felix Fietkau" <nbd@nbd.name>
> Cc: "Frank Wunderlich" <frank-w@public-files.de>, "Mark Lee" <Mark-MC.Lee@mediatek.com>, "Sean Wang" <sean.wang@mediatek.com>, "Lorenzo Bianconi" <lorenzo@kernel.org>, "David S. Miller" <davem@davemloft.net>, "Eric Dumazet" <edumazet@google.com>, "Jakub Kicinski" <kuba@kernel.org>, "Paolo Abeni" <pabeni@redhat.com>, "Matthias Brugger" <matthias.bgg@gmail.com>, "John Crispin" <john@phrozen.org>, "AngeloGioacchino Del Regno" <angelogioacchino.delregno@collabora.com>, netdev@vger.kernel.org, linux-mediatek@lists.infradead.org, linux-arm-kernel@lists.infradead.org
> Betreff: Re: [BUG] MTK SoC Ethernet throughput TX only ~620Mbit/s since 6.2-rc1
>
> Hi Felix,
> 
> On Tue, Mar 14, 2023 at 11:30:53AM +0100, Felix Fietkau wrote:
> > On 07.03.23 19:32, Frank Wunderlich wrote:
> > > Hi,
> > > 
> > > i've noticed that beginning on 6.2-rc1 the throughput on my Bananapi-R2 and Bananapi-R3 goes from 940Mbit/s down do 620Mbit/s since 6.2-rc1.
> > > Only TX (from SBC PoV) is affected, RX is still 940Mbit/s.
> > > 
> > > i bisected this to this commit:
> > > 
> > > f63959c7eec3151c30a2ee0d351827b62e742dcb ("net: ethernet: mtk_eth_soc: implement multi-queue support for per-port queues")
> > > 
> > > Daniel reported me that this is known so far and they need assistance from MTK and i should report it officially.
> > > 
> > > As far as i understand it the commit should fix problems with clients using non-GBE speeds (10/100 Mbit/s) on the Gbit-capable dsa
> > > interfaces (mt753x connected) behind the mac, but now the Gigabit speed is no more reached.
> > > I see no CRC/dropped packets, retransmitts or similar.
> > > 
> > > after reverting the commit above i get 940Mbit like in rx direction, but this will introduce the problems mentioned above so this not a complete fix.
> > I don't have a BPI-R2, but I tested on BPI-R3 and can't reproduce this
> > issue. Do you see it on all ports, or only on specific ones?
> 
> I also can't reproduce this if unsing any of the gigE ports wired via
> MT7531 on the R3. However, I can reproduce the issue if using a 1 GBit/s
> SFP module in slot SFP1 of the R3 (connected directly to GMAC2/eth1).
> 
> Users have reported this also to be a problem also on MT7622 on devices
> directly connecting a PHY (and not using MT7531).
> 
> In all cases, reverting the commit above fixes the issue.


made quick test with 6.3-rc1 on r3 without reverting the patch above and can confirm daniels test

it seems the problem is no more on switch-ports, but on eth1 i have massive packet loss...seems this caused by the same patch because i tested with reverted version and have no issue there.

regards Frank

Aw: Re: [BUG] MTK SoC Ethernet throughput TX only ~620Mbit/s since 6.2-rc1

[Frank Wunderlich]

> Gesendet: Freitag, 17. März 2023 um 19:23 Uhr
> Von: "Frank Wunderlich" <frank-w@public-files.de>
> An: "Daniel Golle" <daniel@makrotopia.org>
> Cc: "Felix Fietkau" <nbd@nbd.name>, "Mark Lee" <Mark-MC.Lee@mediatek.com>, "Sean Wang" <sean.wang@mediatek.com>, "Lorenzo Bianconi" <lorenzo@kernel.org>, "David S. Miller" <davem@davemloft.net>, "Eric Dumazet" <edumazet@google.com>, "Jakub Kicinski" <kuba@kernel.org>, "Paolo Abeni" <pabeni@redhat.com>, "Matthias Brugger" <matthias.bgg@gmail.com>, "John Crispin" <john@phrozen.org>, "AngeloGioacchino Del Regno" <angelogioacchino.delregno@collabora.com>, netdev@vger.kernel.org, linux-mediatek@lists.infradead.org, linux-arm-kernel@lists.infradead.org
> Betreff: Aw: Re: [BUG] MTK SoC Ethernet throughput TX only ~620Mbit/s since 6.2-rc1
>
> Hi,
> > Gesendet: Dienstag, 14. März 2023 um 13:47 Uhr
> > Von: "Daniel Golle" <daniel@makrotopia.org>
> > An: "Felix Fietkau" <nbd@nbd.name>
> > Cc: "Frank Wunderlich" <frank-w@public-files.de>, "Mark Lee" <Mark-MC.Lee@mediatek.com>, "Sean Wang" <sean.wang@mediatek.com>, "Lorenzo Bianconi" <lorenzo@kernel.org>, "David S. Miller" <davem@davemloft.net>, "Eric Dumazet" <edumazet@google.com>, "Jakub Kicinski" <kuba@kernel.org>, "Paolo Abeni" <pabeni@redhat.com>, "Matthias Brugger" <matthias.bgg@gmail.com>, "John Crispin" <john@phrozen.org>, "AngeloGioacchino Del Regno" <angelogioacchino.delregno@collabora.com>, netdev@vger.kernel.org, linux-mediatek@lists.infradead.org, linux-arm-kernel@lists.infradead.org
> > Betreff: Re: [BUG] MTK SoC Ethernet throughput TX only ~620Mbit/s since 6.2-rc1
> >
> > Hi Felix,
> > 
> > On Tue, Mar 14, 2023 at 11:30:53AM +0100, Felix Fietkau wrote:
> > > On 07.03.23 19:32, Frank Wunderlich wrote:
> > > > Hi,
> > > > 
> > > > i've noticed that beginning on 6.2-rc1 the throughput on my Bananapi-R2 and Bananapi-R3 goes from 940Mbit/s down do 620Mbit/s since 6.2-rc1.
> > > > Only TX (from SBC PoV) is affected, RX is still 940Mbit/s.
> > > > 
> > > > i bisected this to this commit:
> > > > 
> > > > f63959c7eec3151c30a2ee0d351827b62e742dcb ("net: ethernet: mtk_eth_soc: implement multi-queue support for per-port queues")
> > > > 
> > > > Daniel reported me that this is known so far and they need assistance from MTK and i should report it officially.
> > > > 
> > > > As far as i understand it the commit should fix problems with clients using non-GBE speeds (10/100 Mbit/s) on the Gbit-capable dsa
> > > > interfaces (mt753x connected) behind the mac, but now the Gigabit speed is no more reached.
> > > > I see no CRC/dropped packets, retransmitts or similar.
> > > > 
> > > > after reverting the commit above i get 940Mbit like in rx direction, but this will introduce the problems mentioned above so this not a complete fix.
> > > I don't have a BPI-R2, but I tested on BPI-R3 and can't reproduce this
> > > issue. Do you see it on all ports, or only on specific ones?
> > 
> > I also can't reproduce this if unsing any of the gigE ports wired via
> > MT7531 on the R3. However, I can reproduce the issue if using a 1 GBit/s
> > SFP module in slot SFP1 of the R3 (connected directly to GMAC2/eth1).
> > 
> > Users have reported this also to be a problem also on MT7622 on devices
> > directly connecting a PHY (and not using MT7531).
> > 
> > In all cases, reverting the commit above fixes the issue.
> 
> 
> made quick test with 6.3-rc1 on r3 without reverting the patch above and can confirm daniels test
> 
> it seems the problem is no more on switch-ports, but on eth1 i have massive packet loss...seems this caused by the same patch because i tested with reverted version and have no issue there.

on BPI-R2 the eth0/gmac0 (tested with wan-port) is affected. here i have in TX-Direction only 620Mbit.

I have no idea yet why there the gmac0 is affected and on r3 only gmac1.

But it looks differently...on r3 the gmac1 is nearly completely broken.

regards Frank

Re: Aw: Re: [BUG] MTK SoC Ethernet throughput TX only ~620Mbit/s since 6.2-rc1

[Felix Fietkau]

On 18.03.23 14:07, Frank Wunderlich wrote:
>> Gesendet: Freitag, 17. März 2023 um 19:23 Uhr
>> Von: "Frank Wunderlich" <frank-w@public-files.de>
>> An: "Daniel Golle" <daniel@makrotopia.org>
>> Cc: "Felix Fietkau" <nbd@nbd.name>, "Mark Lee" <Mark-MC.Lee@mediatek.com>, "Sean Wang" <sean.wang@mediatek.com>, "Lorenzo Bianconi" <lorenzo@kernel.org>, "David S. Miller" <davem@davemloft.net>, "Eric Dumazet" <edumazet@google.com>, "Jakub Kicinski" <kuba@kernel.org>, "Paolo Abeni" <pabeni@redhat.com>, "Matthias Brugger" <matthias.bgg@gmail.com>, "John Crispin" <john@phrozen.org>, "AngeloGioacchino Del Regno" <angelogioacchino.delregno@collabora.com>, netdev@vger.kernel.org, linux-mediatek@lists.infradead.org, linux-arm-kernel@lists.infradead.org
>> Betreff: Aw: Re: [BUG] MTK SoC Ethernet throughput TX only ~620Mbit/s since 6.2-rc1
>>
>> Hi,
>> > Gesendet: Dienstag, 14. März 2023 um 13:47 Uhr
>> > Von: "Daniel Golle" <daniel@makrotopia.org>
>> > An: "Felix Fietkau" <nbd@nbd.name>
>> > Cc: "Frank Wunderlich" <frank-w@public-files.de>, "Mark Lee" <Mark-MC.Lee@mediatek.com>, "Sean Wang" <sean.wang@mediatek.com>, "Lorenzo Bianconi" <lorenzo@kernel.org>, "David S. Miller" <davem@davemloft.net>, "Eric Dumazet" <edumazet@google.com>, "Jakub Kicinski" <kuba@kernel.org>, "Paolo Abeni" <pabeni@redhat.com>, "Matthias Brugger" <matthias.bgg@gmail.com>, "John Crispin" <john@phrozen.org>, "AngeloGioacchino Del Regno" <angelogioacchino.delregno@collabora.com>, netdev@vger.kernel.org, linux-mediatek@lists.infradead.org, linux-arm-kernel@lists.infradead.org
>> > Betreff: Re: [BUG] MTK SoC Ethernet throughput TX only ~620Mbit/s since 6.2-rc1
>> >
>> > Hi Felix,
>> > 
>> > On Tue, Mar 14, 2023 at 11:30:53AM +0100, Felix Fietkau wrote:
>> > > On 07.03.23 19:32, Frank Wunderlich wrote:
>> > > > Hi,
>> > > > 
>> > > > i've noticed that beginning on 6.2-rc1 the throughput on my Bananapi-R2 and Bananapi-R3 goes from 940Mbit/s down do 620Mbit/s since 6.2-rc1.
>> > > > Only TX (from SBC PoV) is affected, RX is still 940Mbit/s.
>> > > > 
>> > > > i bisected this to this commit:
>> > > > 
>> > > > f63959c7eec3151c30a2ee0d351827b62e742dcb ("net: ethernet: mtk_eth_soc: implement multi-queue support for per-port queues")
>> > > > 
>> > > > Daniel reported me that this is known so far and they need assistance from MTK and i should report it officially.
>> > > > 
>> > > > As far as i understand it the commit should fix problems with clients using non-GBE speeds (10/100 Mbit/s) on the Gbit-capable dsa
>> > > > interfaces (mt753x connected) behind the mac, but now the Gigabit speed is no more reached.
>> > > > I see no CRC/dropped packets, retransmitts or similar.
>> > > > 
>> > > > after reverting the commit above i get 940Mbit like in rx direction, but this will introduce the problems mentioned above so this not a complete fix.
>> > > I don't have a BPI-R2, but I tested on BPI-R3 and can't reproduce this
>> > > issue. Do you see it on all ports, or only on specific ones?
>> > 
>> > I also can't reproduce this if unsing any of the gigE ports wired via
>> > MT7531 on the R3. However, I can reproduce the issue if using a 1 GBit/s
>> > SFP module in slot SFP1 of the R3 (connected directly to GMAC2/eth1).
>> > 
>> > Users have reported this also to be a problem also on MT7622 on devices
>> > directly connecting a PHY (and not using MT7531).
>> > 
>> > In all cases, reverting the commit above fixes the issue.
>> 
>> 
>> made quick test with 6.3-rc1 on r3 without reverting the patch above and can confirm daniels test
>> 
>> it seems the problem is no more on switch-ports, but on eth1 i have massive packet loss...seems this caused by the same patch because i tested with reverted version and have no issue there.
> 
> on BPI-R2 the eth0/gmac0 (tested with wan-port) is affected. here i have in TX-Direction only 620Mbit.
> 
> I have no idea yet why there the gmac0 is affected and on r3 only gmac1.
> 
> But it looks differently...on r3 the gmac1 is nearly completely broken.
Please try this patch and let me know if it resolves the regression:

---
--- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c
+++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
@@ -718,8 +718,6 @@ static void mtk_mac_link_up(struct phyli
  		break;
  	}
  
-	mtk_set_queue_speed(mac->hw, mac->id, speed);
-
  	/* Configure duplex */
  	if (duplex == DUPLEX_FULL)
  		mcr |= MAC_MCR_FORCE_DPX;

Re: Aw: Re: [BUG] MTK SoC Ethernet throughput TX only ~620Mbit/s since 6.2-rc1

[Daniel Golle]

Hi Felix,
Hi Frank,

On Fri, Mar 24, 2023 at 12:52:09PM +0100, Felix Fietkau wrote:
> On 18.03.23 14:07, Frank Wunderlich wrote:
> > on BPI-R2 the eth0/gmac0 (tested with wan-port) is affected. here i have in TX-Direction only 620Mbit.
> > 
> > I have no idea yet why there the gmac0 is affected and on r3 only gmac1.
> > 
> > But it looks differently...on r3 the gmac1 is nearly completely broken.

If it looks "completely broken" as in you see only kilobits or even
complete starvation of RX, then you probably miss this fix:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=193250ace270fecd586dd2d0dfbd9cbd2ade977f

> Please try this patch and let me know if it resolves the regression:
> 
> ---
> --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c
> +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
> @@ -718,8 +718,6 @@ static void mtk_mac_link_up(struct phyli
>  		break;
>  	}
> -	mtk_set_queue_speed(mac->hw, mac->id, speed);
> -
>  	/* Configure duplex */
>  	if (duplex == DUPLEX_FULL)
>  		mcr |= MAC_MCR_FORCE_DPX;
> 
> 

I was about to suggest exactly this fix, and yes, it resolves the issue
and now I see full 1 Gbit/s speed on both, switch ports (even while 100M
connection is also used) and eth1 (via 1000Base-T SFP module).

Thank you for taking care of this!