[PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links

[PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links

[Felix Fietkau]

Using the QDMA tx scheduler to throttle tx to line speed works fine for
switch ports, but apparently caused a regression on non-switch ports.

Based on a number of tests, it seems that this throttling can be safely
dropped without re-introducing the issues on switch ports that the
tx scheduling changes resolved.

Link: https://lore.kernel.org/netdev/trinity-92c3826f-c2c8-40af-8339-bc6d0d3ffea4-1678213958520@3c-app-gmx-bs16/
Fixes: f63959c7eec3 ("net: ethernet: mtk_eth_soc: implement multi-queue support for per-port queues")
Reported-by: Frank Wunderlich <frank-w@public-files.de>
Reported-by: Daniel Golle <daniel@makrotopia.org>
Tested-by: Daniel Golle <daniel@makrotopia.org>
Signed-off-by: Felix Fietkau <nbd@nbd.name>
---
 drivers/net/ethernet/mediatek/mtk_eth_soc.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.c b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
index a94aa08515af..282f9435d5ff 100644
--- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c
+++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
@@ -763,8 +763,6 @@ static void mtk_mac_link_up(struct phylink_config *config,
 		break;
 	}
 
-	mtk_set_queue_speed(mac->hw, mac->id, speed);
-
 	/* Configure duplex */
 	if (duplex == DUPLEX_FULL)
 		mcr |= MAC_MCR_FORCE_DPX;
-- 
2.39.0

Aw: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links

[Frank Wunderlich]

> Gesendet: Freitag, 24. März 2023 um 15:04 Uhr
> Von: "Felix Fietkau" <nbd@nbd.name>
> An: netdev@vger.kernel.org
> Cc: "Frank Wunderlich" <frank-w@public-files.de>, "Daniel Golle" <daniel@makrotopia.org>
> Betreff: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links
>
> Using the QDMA tx scheduler to throttle tx to line speed works fine for
> switch ports, but apparently caused a regression on non-switch ports.
> 
> Based on a number of tests, it seems that this throttling can be safely
> dropped without re-introducing the issues on switch ports that the
> tx scheduling changes resolved.
> 
> Link: https://lore.kernel.org/netdev/trinity-92c3826f-c2c8-40af-8339-bc6d0d3ffea4-1678213958520@3c-app-gmx-bs16/
> Fixes: f63959c7eec3 ("net: ethernet: mtk_eth_soc: implement multi-queue support for per-port queues")
> Reported-by: Frank Wunderlich <frank-w@public-files.de>
> Reported-by: Daniel Golle <daniel@makrotopia.org>
> Tested-by: Daniel Golle <daniel@makrotopia.org>
> Signed-off-by: Felix Fietkau <nbd@nbd.name>
> ---
>  drivers/net/ethernet/mediatek/mtk_eth_soc.c | 2 --
>  1 file changed, 2 deletions(-)
> 
> diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.c b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
> index a94aa08515af..282f9435d5ff 100644
> --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c
> +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
> @@ -763,8 +763,6 @@ static void mtk_mac_link_up(struct phylink_config *config,
>  		break;
>  	}
>  
> -	mtk_set_queue_speed(mac->hw, mac->id, speed);
> -
>  	/* Configure duplex */
>  	if (duplex == DUPLEX_FULL)
>  		mcr |= MAC_MCR_FORCE_DPX;

thx for the fix, as daniel already checked it on mt7986/bpi-r3 i tested bpi-r2/mt7623

but unfortunately it does not fix issue on bpi-r2 where the gmac0/mt7530 part is affected.

maybe it needs a special handling like you do for mt7621? maybe it is because the trgmii mode used on this path?

regards Frank

Re: Aw: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links

[Felix Fietkau]

On 25.03.23 10:28, Frank Wunderlich wrote:
>> Gesendet: Freitag, 24. März 2023 um 15:04 Uhr
>> Von: "Felix Fietkau" <nbd@nbd.name>
>> An: netdev@vger.kernel.org
>> Cc: "Frank Wunderlich" <frank-w@public-files.de>, "Daniel Golle" <daniel@makrotopia.org>
>> Betreff: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links
>>
>> Using the QDMA tx scheduler to throttle tx to line speed works fine for
>> switch ports, but apparently caused a regression on non-switch ports.
>> 
>> Based on a number of tests, it seems that this throttling can be safely
>> dropped without re-introducing the issues on switch ports that the
>> tx scheduling changes resolved.
>> 
>> Link: https://lore.kernel.org/netdev/trinity-92c3826f-c2c8-40af-8339-bc6d0d3ffea4-1678213958520@3c-app-gmx-bs16/
>> Fixes: f63959c7eec3 ("net: ethernet: mtk_eth_soc: implement multi-queue support for per-port queues")
>> Reported-by: Frank Wunderlich <frank-w@public-files.de>
>> Reported-by: Daniel Golle <daniel@makrotopia.org>
>> Tested-by: Daniel Golle <daniel@makrotopia.org>
>> Signed-off-by: Felix Fietkau <nbd@nbd.name>
>> ---
>>  drivers/net/ethernet/mediatek/mtk_eth_soc.c | 2 --
>>  1 file changed, 2 deletions(-)
>> 
>> diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.c b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
>> index a94aa08515af..282f9435d5ff 100644
>> --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c
>> +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
>> @@ -763,8 +763,6 @@ static void mtk_mac_link_up(struct phylink_config *config,
>>  		break;
>>  	}
>>  
>> -	mtk_set_queue_speed(mac->hw, mac->id, speed);
>> -
>>  	/* Configure duplex */
>>  	if (duplex == DUPLEX_FULL)
>>  		mcr |= MAC_MCR_FORCE_DPX;
> 
> thx for the fix, as daniel already checked it on mt7986/bpi-r3 i tested bpi-r2/mt7623
> 
> but unfortunately it does not fix issue on bpi-r2 where the gmac0/mt7530 part is affected.
> 
> maybe it needs a special handling like you do for mt7621? maybe it is because the trgmii mode used on this path?
Could you please test if making it use the MT7621 codepath brings back 
performance? I don't have any MT7623 hardware for testing right now.

- Felix

Aw: Re: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links

[Frank Wunderlich]

> Gesendet: Sonntag, 26. März 2023 um 17:56 Uhr
> Von: "Felix Fietkau" <nbd@nbd.name>
> An: "Frank Wunderlich" <frank-w@public-files.de>
> Cc: netdev@vger.kernel.org, "Daniel Golle" <daniel@makrotopia.org>
> Betreff: Re: Aw: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links
>
> On 25.03.23 10:28, Frank Wunderlich wrote:
> >> Gesendet: Freitag, 24. März 2023 um 15:04 Uhr
> >> Von: "Felix Fietkau" <nbd@nbd.name>
> >> An: netdev@vger.kernel.org
> >> Cc: "Frank Wunderlich" <frank-w@public-files.de>, "Daniel Golle" <daniel@makrotopia.org>
> >> Betreff: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links
> >>
> >> Using the QDMA tx scheduler to throttle tx to line speed works fine for
> >> switch ports, but apparently caused a regression on non-switch ports.
> >> 
> >> Based on a number of tests, it seems that this throttling can be safely
> >> dropped without re-introducing the issues on switch ports that the
> >> tx scheduling changes resolved.
> >> 
> >> Link: https://lore.kernel.org/netdev/trinity-92c3826f-c2c8-40af-8339-bc6d0d3ffea4-1678213958520@3c-app-gmx-bs16/
> >> Fixes: f63959c7eec3 ("net: ethernet: mtk_eth_soc: implement multi-queue support for per-port queues")
> >> Reported-by: Frank Wunderlich <frank-w@public-files.de>
> >> Reported-by: Daniel Golle <daniel@makrotopia.org>
> >> Tested-by: Daniel Golle <daniel@makrotopia.org>
> >> Signed-off-by: Felix Fietkau <nbd@nbd.name>
> >> ---
> >>  drivers/net/ethernet/mediatek/mtk_eth_soc.c | 2 --
> >>  1 file changed, 2 deletions(-)
> >> 
> >> diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.c b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
> >> index a94aa08515af..282f9435d5ff 100644
> >> --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c
> >> +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
> >> @@ -763,8 +763,6 @@ static void mtk_mac_link_up(struct phylink_config *config,
> >>  		break;
> >>  	}
> >>  
> >> -	mtk_set_queue_speed(mac->hw, mac->id, speed);
> >> -
> >>  	/* Configure duplex */
> >>  	if (duplex == DUPLEX_FULL)
> >>  		mcr |= MAC_MCR_FORCE_DPX;
> > 
> > thx for the fix, as daniel already checked it on mt7986/bpi-r3 i tested bpi-r2/mt7623
> > 
> > but unfortunately it does not fix issue on bpi-r2 where the gmac0/mt7530 part is affected.
> > 
> > maybe it needs a special handling like you do for mt7621? maybe it is because the trgmii mode used on this path?
> Could you please test if making it use the MT7621 codepath brings back 
> performance? I don't have any MT7623 hardware for testing right now.

Hi,

this seems to make the CPU stall (after kernel is loaded completely when userspace begins to start):

-       if (IS_ENABLED(CONFIG_SOC_MT7621)) {
+       if (IS_ENABLED(CONFIG_SOC_MT7621) || IS_ENABLED(CONFIG_SOC_MT7623)) {

[   27.252672] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:              
[   27.258618] rcu:     2-...0: (0 ticks this GP) idle=54c4/1/0x40000000 softir8
[   27.266973] rcu:     (detected by 1, t=2102 jiffies, g=-891, q=7 ncpus=4)    
[   27.273499] Sending NMI from CPU 1 to CPUs 2:                                
                                                                                
[USBD] USB PRB0 LineState: 0                                                    

wonder why this happens...i expected some kind of tranmit queue erros with trace...

full log here
https://pastebin.com/de4dZDt4

but i've found no error there (sorry for cutting on the right side...my terminal window was to small)

regards Frank

Re: Aw: Re: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links

[Felix Fietkau]

On 26.03.23 19:10, Frank Wunderlich wrote:
>> Gesendet: Sonntag, 26. März 2023 um 17:56 Uhr
>> Von: "Felix Fietkau" <nbd@nbd.name>
>> An: "Frank Wunderlich" <frank-w@public-files.de>
>> Cc: netdev@vger.kernel.org, "Daniel Golle" <daniel@makrotopia.org>
>> Betreff: Re: Aw: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links
>>
>> On 25.03.23 10:28, Frank Wunderlich wrote:
>> >> Gesendet: Freitag, 24. März 2023 um 15:04 Uhr
>> >> Von: "Felix Fietkau" <nbd@nbd.name>
>> >> An: netdev@vger.kernel.org
>> >> Cc: "Frank Wunderlich" <frank-w@public-files.de>, "Daniel Golle" <daniel@makrotopia.org>
>> >> Betreff: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links
>> >>
>> >> Using the QDMA tx scheduler to throttle tx to line speed works fine for
>> >> switch ports, but apparently caused a regression on non-switch ports.
>> >> 
>> >> Based on a number of tests, it seems that this throttling can be safely
>> >> dropped without re-introducing the issues on switch ports that the
>> >> tx scheduling changes resolved.
>> >> 
>> >> Link: https://lore.kernel.org/netdev/trinity-92c3826f-c2c8-40af-8339-bc6d0d3ffea4-1678213958520@3c-app-gmx-bs16/
>> >> Fixes: f63959c7eec3 ("net: ethernet: mtk_eth_soc: implement multi-queue support for per-port queues")
>> >> Reported-by: Frank Wunderlich <frank-w@public-files.de>
>> >> Reported-by: Daniel Golle <daniel@makrotopia.org>
>> >> Tested-by: Daniel Golle <daniel@makrotopia.org>
>> >> Signed-off-by: Felix Fietkau <nbd@nbd.name>
>> >> ---
>> >>  drivers/net/ethernet/mediatek/mtk_eth_soc.c | 2 --
>> >>  1 file changed, 2 deletions(-)
>> >> 
>> >> diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.c b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
>> >> index a94aa08515af..282f9435d5ff 100644
>> >> --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c
>> >> +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
>> >> @@ -763,8 +763,6 @@ static void mtk_mac_link_up(struct phylink_config *config,
>> >>  		break;
>> >>  	}
>> >>  
>> >> -	mtk_set_queue_speed(mac->hw, mac->id, speed);
>> >> -
>> >>  	/* Configure duplex */
>> >>  	if (duplex == DUPLEX_FULL)
>> >>  		mcr |= MAC_MCR_FORCE_DPX;
>> > 
>> > thx for the fix, as daniel already checked it on mt7986/bpi-r3 i tested bpi-r2/mt7623
>> > 
>> > but unfortunately it does not fix issue on bpi-r2 where the gmac0/mt7530 part is affected.
>> > 
>> > maybe it needs a special handling like you do for mt7621? maybe it is because the trgmii mode used on this path?
>> Could you please test if making it use the MT7621 codepath brings back 
>> performance? I don't have any MT7623 hardware for testing right now.
> 
> Hi,
> 
> this seems to make the CPU stall (after kernel is loaded completely when userspace begins to start):
> 
> -       if (IS_ENABLED(CONFIG_SOC_MT7621)) {
> +       if (IS_ENABLED(CONFIG_SOC_MT7621) || IS_ENABLED(CONFIG_SOC_MT7623)) {
> 
> [   27.252672] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
> [   27.258618] rcu:     2-...0: (0 ticks this GP) idle=54c4/1/0x40000000 softir8
> [   27.266973] rcu:     (detected by 1, t=2102 jiffies, g=-891, q=7 ncpus=4)
> [   27.273499] Sending NMI from CPU 1 to CPUs 2:
>                                                                                  
> [USBD] USB PRB0 LineState: 0
> 
> wonder why this happens...i expected some kind of tranmit queue erros with trace...
> 
> full log here
> https://pastebin.com/de4dZDt4
> 
> but i've found no error there (sorry for cutting on the right side...my terminal window was to small)
The change you made has no effect, because CONFIG_SOC_MT7623 does not 
exist, so there must be something else that broke your system.
Try CONFIG_MACH_MT7623 instead, once you've resolved the system hang.

- Felix

Aw: Re: Re: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links

[Frank Wunderlich]

> Gesendet: Sonntag, 26. März 2023 um 19:27 Uhr
> Von: "Felix Fietkau" <nbd@nbd.name>
> An: "Frank Wunderlich" <frank-w@public-files.de>
> Cc: netdev@vger.kernel.org, "Daniel Golle" <daniel@makrotopia.org>
> Betreff: Re: Aw: Re: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links
>
> On 26.03.23 19:10, Frank Wunderlich wrote:
> >> Gesendet: Sonntag, 26. März 2023 um 17:56 Uhr
> >> Von: "Felix Fietkau" <nbd@nbd.name>
> >> An: "Frank Wunderlich" <frank-w@public-files.de>
> >> Cc: netdev@vger.kernel.org, "Daniel Golle" <daniel@makrotopia.org>
> >> Betreff: Re: Aw: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links
> >>
> >> On 25.03.23 10:28, Frank Wunderlich wrote:
> >> >> Gesendet: Freitag, 24. März 2023 um 15:04 Uhr
> >> >> Von: "Felix Fietkau" <nbd@nbd.name>
> >> >> An: netdev@vger.kernel.org
> >> >> Cc: "Frank Wunderlich" <frank-w@public-files.de>, "Daniel Golle" <daniel@makrotopia.org>
> >> >> Betreff: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links
> >> >>
> >> >> Using the QDMA tx scheduler to throttle tx to line speed works fine for
> >> >> switch ports, but apparently caused a regression on non-switch ports.
> >> >> 
> >> >> Based on a number of tests, it seems that this throttling can be safely
> >> >> dropped without re-introducing the issues on switch ports that the
> >> >> tx scheduling changes resolved.
> >> >> 
> >> >> Link: https://lore.kernel.org/netdev/trinity-92c3826f-c2c8-40af-8339-bc6d0d3ffea4-1678213958520@3c-app-gmx-bs16/
> >> >> Fixes: f63959c7eec3 ("net: ethernet: mtk_eth_soc: implement multi-queue support for per-port queues")
> >> >> Reported-by: Frank Wunderlich <frank-w@public-files.de>
> >> >> Reported-by: Daniel Golle <daniel@makrotopia.org>
> >> >> Tested-by: Daniel Golle <daniel@makrotopia.org>
> >> >> Signed-off-by: Felix Fietkau <nbd@nbd.name>
> >> >> ---
> >> >>  drivers/net/ethernet/mediatek/mtk_eth_soc.c | 2 --
> >> >>  1 file changed, 2 deletions(-)
> >> >> 
> >> >> diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.c b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
> >> >> index a94aa08515af..282f9435d5ff 100644
> >> >> --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c
> >> >> +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
> >> >> @@ -763,8 +763,6 @@ static void mtk_mac_link_up(struct phylink_config *config,
> >> >>  		break;
> >> >>  	}
> >> >>  
> >> >> -	mtk_set_queue_speed(mac->hw, mac->id, speed);
> >> >> -
> >> >>  	/* Configure duplex */
> >> >>  	if (duplex == DUPLEX_FULL)
> >> >>  		mcr |= MAC_MCR_FORCE_DPX;
> >> > 
> >> > thx for the fix, as daniel already checked it on mt7986/bpi-r3 i tested bpi-r2/mt7623
> >> > 
> >> > but unfortunately it does not fix issue on bpi-r2 where the gmac0/mt7530 part is affected.
> >> > 
> >> > maybe it needs a special handling like you do for mt7621? maybe it is because the trgmii mode used on this path?
> >> Could you please test if making it use the MT7621 codepath brings back 
> >> performance? I don't have any MT7623 hardware for testing right now.
> > 
> > Hi,
> > 
> > this seems to make the CPU stall (after kernel is loaded completely when userspace begins to start):
> > 
> > -       if (IS_ENABLED(CONFIG_SOC_MT7621)) {
> > +       if (IS_ENABLED(CONFIG_SOC_MT7621) || IS_ENABLED(CONFIG_SOC_MT7623)) {
> > 
> > [   27.252672] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
> > [   27.258618] rcu:     2-...0: (0 ticks this GP) idle=54c4/1/0x40000000 softir8
> > [   27.266973] rcu:     (detected by 1, t=2102 jiffies, g=-891, q=7 ncpus=4)
> > [   27.273499] Sending NMI from CPU 1 to CPUs 2:
> >                                                                                  
> > [USBD] USB PRB0 LineState: 0
> > 
> > wonder why this happens...i expected some kind of tranmit queue erros with trace...
> > 
> > full log here
> > https://pastebin.com/de4dZDt4
> > 
> > but i've found no error there (sorry for cutting on the right side...my terminal window was to small)
> The change you made has no effect, because CONFIG_SOC_MT7623 does not 
> exist, so there must be something else that broke your system.
> Try CONFIG_MACH_MT7623 instead, once you've resolved the system hang.

it was exactly this change above...dropping it fixed it

do not know why i did no compile-error, looked into implementation of IS_ENABLED which is passed through different defines till

include/linux/kconfig.h:42:#define ___is_defined(val)		____is_defined(__ARG_PLACEHOLDER_##val)
include/linux/kconfig.h:43:#define ____is_defined(arg1_or_junk)	__take_second_arg(arg1_or_junk 1, 0)
include/linux/kconfig.h:14:#define __take_second_arg(__ignored, val, ...) val

i do not expect that there is anything wrong, just wonder why this stalls...

used the CONFIG_MACH_MT7623 (which is set in my config) boots up fine, but did not fix the 622Mbit-tx-issue

and i'm not sure i have tested it before...all ports of mt7531 are affected, not only wan (i remembered you asked for this)

regards Frank

Re: Aw: Re: Re: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links

[Felix Fietkau]

On 26.03.23 19:49, Frank Wunderlich wrote:
>> Gesendet: Sonntag, 26. März 2023 um 19:27 Uhr
>> Von: "Felix Fietkau" <nbd@nbd.name>
>> An: "Frank Wunderlich" <frank-w@public-files.de>
>> Cc: netdev@vger.kernel.org, "Daniel Golle" <daniel@makrotopia.org>
>> Betreff: Re: Aw: Re: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links
>>
>> On 26.03.23 19:10, Frank Wunderlich wrote:
>> >> Gesendet: Sonntag, 26. März 2023 um 17:56 Uhr
>> >> Von: "Felix Fietkau" <nbd@nbd.name>
>> >> An: "Frank Wunderlich" <frank-w@public-files.de>
>> >> Cc: netdev@vger.kernel.org, "Daniel Golle" <daniel@makrotopia.org>
>> >> Betreff: Re: Aw: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links
>> >>
>> >> On 25.03.23 10:28, Frank Wunderlich wrote:
>> >> >> Gesendet: Freitag, 24. März 2023 um 15:04 Uhr
>> >> >> Von: "Felix Fietkau" <nbd@nbd.name>
>> >> >> An: netdev@vger.kernel.org
>> >> >> Cc: "Frank Wunderlich" <frank-w@public-files.de>, "Daniel Golle" <daniel@makrotopia.org>
>> >> >> Betreff: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links
>> >> >>
>> >> >> Using the QDMA tx scheduler to throttle tx to line speed works fine for
>> >> >> switch ports, but apparently caused a regression on non-switch ports.
>> >> >> 
>> >> >> Based on a number of tests, it seems that this throttling can be safely
>> >> >> dropped without re-introducing the issues on switch ports that the
>> >> >> tx scheduling changes resolved.
>> >> >> 
>> >> >> Link: https://lore.kernel.org/netdev/trinity-92c3826f-c2c8-40af-8339-bc6d0d3ffea4-1678213958520@3c-app-gmx-bs16/
>> >> >> Fixes: f63959c7eec3 ("net: ethernet: mtk_eth_soc: implement multi-queue support for per-port queues")
>> >> >> Reported-by: Frank Wunderlich <frank-w@public-files.de>
>> >> >> Reported-by: Daniel Golle <daniel@makrotopia.org>
>> >> >> Tested-by: Daniel Golle <daniel@makrotopia.org>
>> >> >> Signed-off-by: Felix Fietkau <nbd@nbd.name>
>> >> >> ---
>> >> >>  drivers/net/ethernet/mediatek/mtk_eth_soc.c | 2 --
>> >> >>  1 file changed, 2 deletions(-)
>> >> >> 
>> >> >> diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.c b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
>> >> >> index a94aa08515af..282f9435d5ff 100644
>> >> >> --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c
>> >> >> +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
>> >> >> @@ -763,8 +763,6 @@ static void mtk_mac_link_up(struct phylink_config *config,
>> >> >>  		break;
>> >> >>  	}
>> >> >>  
>> >> >> -	mtk_set_queue_speed(mac->hw, mac->id, speed);
>> >> >> -
>> >> >>  	/* Configure duplex */
>> >> >>  	if (duplex == DUPLEX_FULL)
>> >> >>  		mcr |= MAC_MCR_FORCE_DPX;
>> >> > 
>> >> > thx for the fix, as daniel already checked it on mt7986/bpi-r3 i tested bpi-r2/mt7623
>> >> > 
>> >> > but unfortunately it does not fix issue on bpi-r2 where the gmac0/mt7530 part is affected.
>> >> > 
>> >> > maybe it needs a special handling like you do for mt7621? maybe it is because the trgmii mode used on this path?
>> >> Could you please test if making it use the MT7621 codepath brings back 
>> >> performance? I don't have any MT7623 hardware for testing right now.
>> > 
>> > Hi,
>> > 
>> > this seems to make the CPU stall (after kernel is loaded completely when userspace begins to start):
>> > 
>> > -       if (IS_ENABLED(CONFIG_SOC_MT7621)) {
>> > +       if (IS_ENABLED(CONFIG_SOC_MT7621) || IS_ENABLED(CONFIG_SOC_MT7623)) {
>> > 
>> > [   27.252672] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
>> > [   27.258618] rcu:     2-...0: (0 ticks this GP) idle=54c4/1/0x40000000 softir8
>> > [   27.266973] rcu:     (detected by 1, t=2102 jiffies, g=-891, q=7 ncpus=4)
>> > [   27.273499] Sending NMI from CPU 1 to CPUs 2:
>> >                                                                                  
>> > [USBD] USB PRB0 LineState: 0
>> > 
>> > wonder why this happens...i expected some kind of tranmit queue erros with trace...
>> > 
>> > full log here
>> > https://pastebin.com/de4dZDt4
>> > 
>> > but i've found no error there (sorry for cutting on the right side...my terminal window was to small)
>> The change you made has no effect, because CONFIG_SOC_MT7623 does not 
>> exist, so there must be something else that broke your system.
>> Try CONFIG_MACH_MT7623 instead, once you've resolved the system hang.
> 
> it was exactly this change above...dropping it fixed it
> 
> do not know why i did no compile-error, looked into implementation of IS_ENABLED which is passed through different defines till
> 
> include/linux/kconfig.h:42:#define ___is_defined(val)		____is_defined(__ARG_PLACEHOLDER_##val)
> include/linux/kconfig.h:43:#define ____is_defined(arg1_or_junk)	__take_second_arg(arg1_or_junk 1, 0)
> include/linux/kconfig.h:14:#define __take_second_arg(__ignored, val, ...) val
> 
> i do not expect that there is anything wrong, just wonder why this stalls...
> 
> used the CONFIG_MACH_MT7623 (which is set in my config) boots up fine, but did not fix the 622Mbit-tx-issue
> 
> and i'm not sure i have tested it before...all ports of mt7531 are affected, not only wan (i remembered you asked for this)
Does the MAC that's connected to the switch use flow control? Can you 
test if changing that makes a difference?

- Felix

Aw: Re: Re: Re: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links

[Frank Wunderlich]

> Gesendet: Sonntag, 26. März 2023 um 22:09 Uhr
> Von: "Felix Fietkau" <nbd@nbd.name>
> On 26.03.23 19:49, Frank Wunderlich wrote:
> >> Gesendet: Sonntag, 26. März 2023 um 19:27 Uhr
> >> Von: "Felix Fietkau" <nbd@nbd.name>

> >> On 26.03.23 19:10, Frank Wunderlich wrote:
> >> >> Gesendet: Sonntag, 26. März 2023 um 17:56 Uhr
> >> >> Von: "Felix Fietkau" <nbd@nbd.name>

> >> >> On 25.03.23 10:28, Frank Wunderlich wrote:
> >> >> >> Gesendet: Freitag, 24. März 2023 um 15:04 Uhr
> >> >> >> Von: "Felix Fietkau" <nbd@nbd.name>

> >> >> > thx for the fix, as daniel already checked it on mt7986/bpi-r3 i tested bpi-r2/mt7623
> >> >> > 
> >> >> > but unfortunately it does not fix issue on bpi-r2 where the gmac0/mt7530 part is affected.
> >> >> > 
> >> >> > maybe it needs a special handling like you do for mt7621? maybe it is because the trgmii mode used on this path?
> >> >> Could you please test if making it use the MT7621 codepath brings back 
> >> >> performance? I don't have any MT7623 hardware for testing right now.

> > used the CONFIG_MACH_MT7623 (which is set in my config) boots up fine, but did not fix the 622Mbit-tx-issue
> > 
> > and i'm not sure i have tested it before...all ports of mt7531 are affected, not only wan (i remembered you asked for this)
> Does the MAC that's connected to the switch use flow control? Can you 
> test if changing that makes a difference?

it does use flow control/pause on mac and switch-port, disabled it, but it does not change anything (still ~620Mbit on tx)

+++ b/arch/arm/boot/dts/mt7623n-bananapi-bpi-r2.dts
@@ -182,7 +182,7 @@ gmac0: mac@0 {
                fixed-link {
                        speed = <1000>;
                        full-duplex;
-                       pause;
+                       //pause;
                };
        };
 
@@ -235,7 +235,7 @@ port@6 {
                                        fixed-link {
                                                speed = <1000>;
                                                full-duplex;
-                                               pause;
+                                               //pause;
                                        };
                                };
                        };

regards Frank

Re: Aw: Re: Re: Re: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links

[Felix Fietkau]

On 27.03.23 19:28, Frank Wunderlich wrote:
> 
>> Gesendet: Sonntag, 26. März 2023 um 22:09 Uhr
>> Von: "Felix Fietkau" <nbd@nbd.name>
>> On 26.03.23 19:49, Frank Wunderlich wrote:
>> >> Gesendet: Sonntag, 26. März 2023 um 19:27 Uhr
>> >> Von: "Felix Fietkau" <nbd@nbd.name>
> 
>> >> On 26.03.23 19:10, Frank Wunderlich wrote:
>> >> >> Gesendet: Sonntag, 26. März 2023 um 17:56 Uhr
>> >> >> Von: "Felix Fietkau" <nbd@nbd.name>
> 
>> >> >> On 25.03.23 10:28, Frank Wunderlich wrote:
>> >> >> >> Gesendet: Freitag, 24. März 2023 um 15:04 Uhr
>> >> >> >> Von: "Felix Fietkau" <nbd@nbd.name>
> 
>> >> >> > thx for the fix, as daniel already checked it on mt7986/bpi-r3 i tested bpi-r2/mt7623
>> >> >> > 
>> >> >> > but unfortunately it does not fix issue on bpi-r2 where the gmac0/mt7530 part is affected.
>> >> >> > 
>> >> >> > maybe it needs a special handling like you do for mt7621? maybe it is because the trgmii mode used on this path?
>> >> >> Could you please test if making it use the MT7621 codepath brings back 
>> >> >> performance? I don't have any MT7623 hardware for testing right now.
> 
>> > used the CONFIG_MACH_MT7623 (which is set in my config) boots up fine, but did not fix the 622Mbit-tx-issue
>> > 
>> > and i'm not sure i have tested it before...all ports of mt7531 are affected, not only wan (i remembered you asked for this)
>> Does the MAC that's connected to the switch use flow control? Can you 
>> test if changing that makes a difference?
> 
> it does use flow control/pause on mac and switch-port, disabled it, but it does not change anything (still ~620Mbit on tx)
> 
> +++ b/arch/arm/boot/dts/mt7623n-bananapi-bpi-r2.dts
> @@ -182,7 +182,7 @@ gmac0: mac@0 {
>                  fixed-link {
>                          speed = <1000>;
>                          full-duplex;
> -                       pause;
> +                       //pause;
>                  };
>          };
>   
> @@ -235,7 +235,7 @@ port@6 {
>                                          fixed-link {
>                                                  speed = <1000>;
>                                                  full-duplex;
> -                                               pause;
> +                                               //pause;
>                                          };
>                                  };
>                          };
> 
> regards Frank
I think I have an idea on how to properly deal with this issue now. I 
had overlooked the fact that on MT7623 the link to the switch is only 
1000M, so any QDMA shaping to that speed does not make any sense, since 
throughput is already limited by MAC speed.
I will make a patch that enables shaping only if port_speed < mac_speed.

- Felix

Re: Aw: Re: Re: Re: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links

[Felix Fietkau]

On 27.03.23 19:28, Frank Wunderlich wrote:
> 
>> Gesendet: Sonntag, 26. März 2023 um 22:09 Uhr
>> Von: "Felix Fietkau" <nbd@nbd.name>
>> On 26.03.23 19:49, Frank Wunderlich wrote:
>> >> Gesendet: Sonntag, 26. März 2023 um 19:27 Uhr
>> >> Von: "Felix Fietkau" <nbd@nbd.name>
> 
>> >> On 26.03.23 19:10, Frank Wunderlich wrote:
>> >> >> Gesendet: Sonntag, 26. März 2023 um 17:56 Uhr
>> >> >> Von: "Felix Fietkau" <nbd@nbd.name>
> 
>> >> >> On 25.03.23 10:28, Frank Wunderlich wrote:
>> >> >> >> Gesendet: Freitag, 24. März 2023 um 15:04 Uhr
>> >> >> >> Von: "Felix Fietkau" <nbd@nbd.name>
> 
>> >> >> > thx for the fix, as daniel already checked it on mt7986/bpi-r3 i tested bpi-r2/mt7623
>> >> >> > 
>> >> >> > but unfortunately it does not fix issue on bpi-r2 where the gmac0/mt7530 part is affected.
>> >> >> > 
>> >> >> > maybe it needs a special handling like you do for mt7621? maybe it is because the trgmii mode used on this path?
>> >> >> Could you please test if making it use the MT7621 codepath brings back 
>> >> >> performance? I don't have any MT7623 hardware for testing right now.
> 
>> > used the CONFIG_MACH_MT7623 (which is set in my config) boots up fine, but did not fix the 622Mbit-tx-issue
>> > 
>> > and i'm not sure i have tested it before...all ports of mt7531 are affected, not only wan (i remembered you asked for this)
>> Does the MAC that's connected to the switch use flow control? Can you 
>> test if changing that makes a difference?
> 
> it does use flow control/pause on mac and switch-port, disabled it, but it does not change anything (still ~620Mbit on tx)

I finally found a MT7623 device in my stash, so I could run some
experiments with it. For some reason I could only reproduce your tx
throughput values after switching off TRGMII. With TRGMII enabled, I got
around 864 Mbit/s, which is of course still lower than what I get with
shaping disabled.
I also experimented with bumping the shaping rate to higher values, but
got no changes in throughput at all.
Based on that, I'm beginning to think that the shaper simply can't
handle rates close to the MAC rate and runs into a fundamental limit
somehow.
Now that I think about it, I do remember that shaping to 2.5 Gbps on
MT7622 also reduced link throughput.

I think we should simply use this patch to deal with it. Do you agree?

- Felix

---
--- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c
+++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
@@ -753,6 +753,7 @@ static void mtk_mac_link_up(struct phylink_config *config,
  		 MAC_MCR_FORCE_RX_FC);
  
  	/* Configure speed */
+	mac->speed = speed;
  	switch (speed) {
  	case SPEED_2500:
  	case SPEED_1000:
@@ -3235,6 +3236,9 @@ static int mtk_device_event(struct notifier_block *n, unsigned long event, void
  	if (dp->index >= MTK_QDMA_NUM_QUEUES)
  		return NOTIFY_DONE;
  
+	if (mac->speed > 0 && mac->speed <= s.base.speed)
+		s.base.speed = 0;
+
  	mtk_set_queue_speed(eth, dp->index + 3, s.base.speed);
  
  	return NOTIFY_DONE;

Aw: Re: Re: Re: Re: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links

[Frank Wunderlich]

something ist still strange...i get a rcu stall again with this patch...reverted it and my r2 boots again.

[   29.772755] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[   29.778689] rcu:     2-...0: (1 GPs behind) idle=547c/1/0x40000000 softirq=251/258 fqs=427
[   29.786697] rcu:     (detected by 1, t=2104 jiffies, g=-875, q=29 ncpus=4)
[   29.793308] Sending NMI from CPU 1 to CPUs 2:
[   34.492968] vusb: disabling
[   34.495828] vmc: disabling
[   34.498587] vmch: disabling
[   34.501433] vgp1: disabling
[   34.504426] vcamaf: disabling
[   39.797579] rcu: rcu_sched kthread timer wakeup didn't happen for 994 jiffies! g-875 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
[   39.808619] rcu:     Possible timer handling issue on cpu=1 timer-softirq=493
[   39.815487] rcu: rcu_sched kthread starved for 995 jiffies! g-875 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1
[   39.825571] rcu:     Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior.
[   39.834520] rcu: RCU grace-period kthread stack dump:
[   39.839564] task:rcu_sched       state:I stack:0     pid:14    ppid:2      flags:0x00000000
[   39.847928]  __schedule from schedule+0x54/0xe8
[   39.852472]  schedule from schedule_timeout+0x94/0x158
[   39.857619]  schedule_timeout from rcu_gp_fqs_loop+0x12c/0x50c
[   39.863467]  rcu_gp_fqs_loop from rcu_gp_kthread+0x194/0x21c
[   39.869135]  rcu_gp_kthread from kthread+0xc8/0xcc
[   39.873931]  kthread from ret_from_fork+0x14/0x2c
[   39.878639] Exception stack(0xf0859fb0 to 0xf0859ff8)
[   39.883690] 9fa0:                                     00000000 00000000 00000000 00000000
[   39.891864] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   39.900037] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[   39.906645] rcu: Stack dump where RCU GP kthread last ran:
[   39.912125] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.3.0-rc1-bpi-r2-rc-net #2
[   39.919518] Hardware name: Mediatek Cortex-A7 (Device Tree)
[   39.925082] PC is at default_idle_call+0x1c/0xb0
[   39.929698] LR is at ct_kernel_enter.constprop.0+0x48/0x11c
[   39.935267] pc : [<c0d105ec>]    lr : [<c0d0ffa4>]    psr: 600e0013
[   39.941527] sp : f0861fb0  ip : c15721e0  fp : 00000000
[   39.946746] r10: 00000000  r9 : 410fc073  r8 : 8000406a
[   39.951964] r7 : c1404f74  r6 : c19e0900  r5 : c15727e0  r4 : c19e0900
[   39.958486] r3 : 00000000  r2 : 2da0a000  r1 : 00000001  r0 : 00008cfc
[   39.965007] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[   39.972138] Control: 10c5387d  Table: 84f4806a  DAC: 00000051
[   39.977878]  default_idle_call from cpuidle_idle_call+0x24/0x68
[   39.983805]  cpuidle_idle_call from do_idle+0x9c/0xd0
[   39.988863]  do_idle from cpu_startup_entry+0x20/0x24
[   39.993921]  cpu_startup_entry from secondary_start_kernel+0x118/0x138
[   40.000457]  secondary_start_kernel from 0x801017a0

maybe i need additional patch or did anything else wrong?

still working on 6.3-rc1
https://github.com/frank-w/BPI-Router-Linux/commits/6.3-rc-net

regards Frank


> Gesendet: Mittwoch, 29. März 2023 um 14:04 Uhr
> Von: "Felix Fietkau" <nbd@nbd.name>
> An: "Frank Wunderlich" <frank-w@public-files.de>
> Cc: netdev@vger.kernel.org, "Daniel Golle" <daniel@makrotopia.org>
> Betreff: Re: Aw: Re: Re: Re: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links
>
> On 27.03.23 19:28, Frank Wunderlich wrote:
> > 
> >> Gesendet: Sonntag, 26. März 2023 um 22:09 Uhr
> >> Von: "Felix Fietkau" <nbd@nbd.name>
> >> On 26.03.23 19:49, Frank Wunderlich wrote:
> >> >> Gesendet: Sonntag, 26. März 2023 um 19:27 Uhr
> >> >> Von: "Felix Fietkau" <nbd@nbd.name>
> > 
> >> >> On 26.03.23 19:10, Frank Wunderlich wrote:
> >> >> >> Gesendet: Sonntag, 26. März 2023 um 17:56 Uhr
> >> >> >> Von: "Felix Fietkau" <nbd@nbd.name>
> > 
> >> >> >> On 25.03.23 10:28, Frank Wunderlich wrote:
> >> >> >> >> Gesendet: Freitag, 24. März 2023 um 15:04 Uhr
> >> >> >> >> Von: "Felix Fietkau" <nbd@nbd.name>
> > 
> >> >> >> > thx for the fix, as daniel already checked it on mt7986/bpi-r3 i tested bpi-r2/mt7623
> >> >> >> > 
> >> >> >> > but unfortunately it does not fix issue on bpi-r2 where the gmac0/mt7530 part is affected.
> >> >> >> > 
> >> >> >> > maybe it needs a special handling like you do for mt7621? maybe it is because the trgmii mode used on this path?
> >> >> >> Could you please test if making it use the MT7621 codepath brings back 
> >> >> >> performance? I don't have any MT7623 hardware for testing right now.
> > 
> >> > used the CONFIG_MACH_MT7623 (which is set in my config) boots up fine, but did not fix the 622Mbit-tx-issue
> >> > 
> >> > and i'm not sure i have tested it before...all ports of mt7531 are affected, not only wan (i remembered you asked for this)
> >> Does the MAC that's connected to the switch use flow control? Can you 
> >> test if changing that makes a difference?
> > 
> > it does use flow control/pause on mac and switch-port, disabled it, but it does not change anything (still ~620Mbit on tx)
> 
> I finally found a MT7623 device in my stash, so I could run some
> experiments with it. For some reason I could only reproduce your tx
> throughput values after switching off TRGMII. With TRGMII enabled, I got
> around 864 Mbit/s, which is of course still lower than what I get with
> shaping disabled.
> I also experimented with bumping the shaping rate to higher values, but
> got no changes in throughput at all.
> Based on that, I'm beginning to think that the shaper simply can't
> handle rates close to the MAC rate and runs into a fundamental limit
> somehow.
> Now that I think about it, I do remember that shaping to 2.5 Gbps on
> MT7622 also reduced link throughput.
> 
> I think we should simply use this patch to deal with it. Do you agree?
> 
> - Felix
> 
> ---
> --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c
> +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
> @@ -753,6 +753,7 @@ static void mtk_mac_link_up(struct phylink_config *config,
>   		 MAC_MCR_FORCE_RX_FC);
>   
>   	/* Configure speed */
> +	mac->speed = speed;
>   	switch (speed) {
>   	case SPEED_2500:
>   	case SPEED_1000:
> @@ -3235,6 +3236,9 @@ static int mtk_device_event(struct notifier_block *n, unsigned long event, void
>   	if (dp->index >= MTK_QDMA_NUM_QUEUES)
>   		return NOTIFY_DONE;
>   
> +	if (mac->speed > 0 && mac->speed <= s.base.speed)
> +		s.base.speed = 0;
> +
>   	mtk_set_queue_speed(eth, dp->index + 3, s.base.speed);
>   
>   	return NOTIFY_DONE;
> 
> 
> 
>

Re: Aw: Re: Re: Re: Re: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links

[Felix Fietkau]

On 30.03.23 15:58, Frank Wunderlich wrote:
> something ist still strange...i get a rcu stall again with this patch...reverted it and my r2 boots again.
> 
> [   29.772755] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
> [   29.778689] rcu:     2-...0: (1 GPs behind) idle=547c/1/0x40000000 softirq=251/258 fqs=427
> [   29.786697] rcu:     (detected by 1, t=2104 jiffies, g=-875, q=29 ncpus=4)
> [   29.793308] Sending NMI from CPU 1 to CPUs 2:
> [   34.492968] vusb: disabling
> [   34.495828] vmc: disabling
> [   34.498587] vmch: disabling
> [   34.501433] vgp1: disabling
> [   34.504426] vcamaf: disabling
> [   39.797579] rcu: rcu_sched kthread timer wakeup didn't happen for 994 jiffies! g-875 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
> [   39.808619] rcu:     Possible timer handling issue on cpu=1 timer-softirq=493
> [   39.815487] rcu: rcu_sched kthread starved for 995 jiffies! g-875 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1
> [   39.825571] rcu:     Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior.
> [   39.834520] rcu: RCU grace-period kthread stack dump:
> [   39.839564] task:rcu_sched       state:I stack:0     pid:14    ppid:2      flags:0x00000000
> [   39.847928]  __schedule from schedule+0x54/0xe8
> [   39.852472]  schedule from schedule_timeout+0x94/0x158
> [   39.857619]  schedule_timeout from rcu_gp_fqs_loop+0x12c/0x50c
> [   39.863467]  rcu_gp_fqs_loop from rcu_gp_kthread+0x194/0x21c
> [   39.869135]  rcu_gp_kthread from kthread+0xc8/0xcc
> [   39.873931]  kthread from ret_from_fork+0x14/0x2c
> [   39.878639] Exception stack(0xf0859fb0 to 0xf0859ff8)
> [   39.883690] 9fa0:                                     00000000 00000000 00000000 00000000
> [   39.891864] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> [   39.900037] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
> [   39.906645] rcu: Stack dump where RCU GP kthread last ran:
> [   39.912125] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 6.3.0-rc1-bpi-r2-rc-net #2
> [   39.919518] Hardware name: Mediatek Cortex-A7 (Device Tree)
> [   39.925082] PC is at default_idle_call+0x1c/0xb0
> [   39.929698] LR is at ct_kernel_enter.constprop.0+0x48/0x11c
> [   39.935267] pc : [<c0d105ec>]    lr : [<c0d0ffa4>]    psr: 600e0013
> [   39.941527] sp : f0861fb0  ip : c15721e0  fp : 00000000
> [   39.946746] r10: 00000000  r9 : 410fc073  r8 : 8000406a
> [   39.951964] r7 : c1404f74  r6 : c19e0900  r5 : c15727e0  r4 : c19e0900
> [   39.958486] r3 : 00000000  r2 : 2da0a000  r1 : 00000001  r0 : 00008cfc
> [   39.965007] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
> [   39.972138] Control: 10c5387d  Table: 84f4806a  DAC: 00000051
> [   39.977878]  default_idle_call from cpuidle_idle_call+0x24/0x68
> [   39.983805]  cpuidle_idle_call from do_idle+0x9c/0xd0
> [   39.988863]  do_idle from cpu_startup_entry+0x20/0x24
> [   39.993921]  cpu_startup_entry from secondary_start_kernel+0x118/0x138
> [   40.000457]  secondary_start_kernel from 0x801017a0
> 
> maybe i need additional patch or did anything else wrong?
> 
> still working on 6.3-rc1
> https://github.com/frank-w/BPI-Router-Linux/commits/6.3-rc-net
Can you try applying this patch to a stable kernel instead? These hangs 
don't make any sense to me, especially the one triggered by an earlier 
patch that should definitely have been a no-op because of the wrong 
config symbol.
It really looks to me like you have an issue in that kernel triggered by 
spurious code changes.

- Felix

Aw: Re: Re: Re: Re: Re: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links

[Frank Wunderlich]

regards Frank


> Gesendet: Donnerstag, 30. März 2023 um 19:06 Uhr
> Von: "Felix Fietkau" <nbd@nbd.name>
> An: "Frank Wunderlich" <frank-w@public-files.de>
> Cc: netdev@vger.kernel.org, "Daniel Golle" <daniel@makrotopia.org>
> Betreff: Re: Aw: Re: Re: Re: Re: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links
>
> On 30.03.23 15:58, Frank Wunderlich wrote:
> > something ist still strange...i get a rcu stall again with this patch...reverted it and my r2 boots again.
> > 
> > [   29.772755] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
> > [   29.778689] rcu:     2-...0: (1 GPs behind) idle=547c/1/0x40000000 softirq=251/258 fqs=427
> > [   29.786697] rcu:     (detected by 1, t=2104 jiffies, g=-875, q=29 ncpus=4)
> > [   29.793308] Sending NMI from CPU 1 to CPUs 2:
> > 
> > maybe i need additional patch or did anything else wrong?
> > 
> > still working on 6.3-rc1
> > https://github.com/frank-w/BPI-Router-Linux/commits/6.3-rc-net
> Can you try applying this patch to a stable kernel instead? These hangs 
> don't make any sense to me, especially the one triggered by an earlier 
> patch that should definitely have been a no-op because of the wrong 
> config symbol.
> It really looks to me like you have an issue in that kernel triggered by 
> spurious code changes.

Hi,

have applied it on top of 6.2.0 which has the "implement multi-queue support for per-port queues" in, and indeed it fixes the problem on mt7623.

thx for the fix, you can send it with my tested-tag

Tested-By: Frank Wunderlich <frank-w@public-files.de>

regards Frank

Re: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links

[Jakub Kicinski]

On Fri, 24 Mar 2023 15:04:04 +0100 Felix Fietkau wrote:
> Using the QDMA tx scheduler to throttle tx to line speed works fine for
> switch ports, but apparently caused a regression on non-switch ports.
> 
> Based on a number of tests, it seems that this throttling can be safely
> dropped without re-introducing the issues on switch ports that the
> tx scheduling changes resolved.
> 
> Link: https://lore.kernel.org/netdev/trinity-92c3826f-c2c8-40af-8339-bc6d0d3ffea4-1678213958520@3c-app-gmx-bs16/
> Fixes: f63959c7eec3 ("net: ethernet: mtk_eth_soc: implement multi-queue support for per-port queues")
> Reported-by: Frank Wunderlich <frank-w@public-files.de>
> Reported-by: Daniel Golle <daniel@makrotopia.org>
> Tested-by: Daniel Golle <daniel@makrotopia.org>
> Signed-off-by: Felix Fietkau <nbd@nbd.name>

My reading of the discussion was that this patch is good, even if it
doesn't fix all the models, but it's marked as Changes Requested in PW.
Could you confirm that we should apply this one as is, just to be sure?

Re: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links

[Felix Fietkau]

On 29.03.23 04:01, Jakub Kicinski wrote:
> On Fri, 24 Mar 2023 15:04:04 +0100 Felix Fietkau wrote:
>> Using the QDMA tx scheduler to throttle tx to line speed works fine for
>> switch ports, but apparently caused a regression on non-switch ports.
>> 
>> Based on a number of tests, it seems that this throttling can be safely
>> dropped without re-introducing the issues on switch ports that the
>> tx scheduling changes resolved.
>> 
>> Link: https://lore.kernel.org/netdev/trinity-92c3826f-c2c8-40af-8339-bc6d0d3ffea4-1678213958520@3c-app-gmx-bs16/
>> Fixes: f63959c7eec3 ("net: ethernet: mtk_eth_soc: implement multi-queue support for per-port queues")
>> Reported-by: Frank Wunderlich <frank-w@public-files.de>
>> Reported-by: Daniel Golle <daniel@makrotopia.org>
>> Tested-by: Daniel Golle <daniel@makrotopia.org>
>> Signed-off-by: Felix Fietkau <nbd@nbd.name>
> 
> My reading of the discussion was that this patch is good, even if it
> doesn't fix all the models, but it's marked as Changes Requested in PW.
> Could you confirm that we should apply this one as is, just to be sure?
Yes, please apply this one. I will send another patch to take care of 
MT7623 soon.

- Felix

Re: [PATCH net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links

[patchwork-bot+netdevbpf]

Hello:

This patch was applied to netdev/net.git (main)
by Jakub Kicinski <kuba@kernel.org>:

On Fri, 24 Mar 2023 15:04:04 +0100 you wrote:
> Using the QDMA tx scheduler to throttle tx to line speed works fine for
> switch ports, but apparently caused a regression on non-switch ports.
> 
> Based on a number of tests, it seems that this throttling can be safely
> dropped without re-introducing the issues on switch ports that the
> tx scheduling changes resolved.
> 
> [...]

Here is the summary with links:
  - [net] net: ethernet: mtk_eth_soc: fix tx throughput regression with direct 1G links
    https://git.kernel.org/netdev/net/c/07b3af42d8d5

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html