All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Théo Lebrun" <theo.lebrun@bootlin.com>
To: "Andrea della Porta" <andrea.porta@suse.com>,
	<netdev@vger.kernel.org>, "Andrew Lunn" <andrew+netdev@lunn.ch>,
	"David S . Miller" <davem@davemloft.net>,
	"Eric Dumazet" <edumazet@google.com>,
	"Jakub Kicinski" <kuba@kernel.org>,
	"Paolo Abeni" <pabeni@redhat.com>, <linux-kernel@vger.kernel.org>,
	<linux-arm-kernel@lists.infradead.org>,
	<linux-rpi-kernel@lists.infradead.org>,
	"Nicolai Buchwitz" <nb@tipi-net.de>,
	"Conor Dooley" <conor.dooley@microchip.com>
Cc: "Lukasz Raczylo" <lukasz@raczylo.com>,
	"Steffen Jaeckel" <sjaeckel@suse.de>,
	"Drew Streib" <dtype@dtype.org>
Subject: Re: [PATCH v2] net: macb: add TX stall timeout callback to recover from lost TSTART write
Date: Fri, 18 Sep 2026 17:22:49 +0200	[thread overview]
Message-ID: <DLIJN4CDS11K.RVJGK3UALMJE@bootlin.com> (raw)
In-Reply-To: <468f480454a314303bac6a54780b153f689f2267.1781598350.git.andrea.porta@suse.com>

Hello all,

On Tue Jun 16, 2026 at 3:23 PM CEST, Andrea della Porta wrote:
> From: Lukasz Raczylo <lukasz@raczylo.com>
>
> The MACB found in the Raspberry Pi RP1 suffers from sporadic stalls on
> the TX queue.
> While the exact root cause is not yet fully understood, it is likely
> related to a hardware issue where a TSTART write to the NCR register
> is missed, preventing the transmission from being kicked off.
>
> Implement a timeout callback to handle TX queue stalls, triggering the
> existing restart mechanism to recover.

Any news on this topic?

There was a guess that a "flush PCIe posted write after TSTART
doorbell" patch [0] could solve it but I was sceptical [1]. We never
landed that and only took the Tx timeout implementation. If it ever
triggers then the kernel log gets a "NETDEV WATCHDOG: ..." critical
line appended [2].

I got reminded because I came across on this wiki page [3] about the
issue, whose author is in Cc. As an aside, I landed there by testing out
the Marginalia search engine and queried "macb driver" (as one does).

Main discovery of the day: it happens on EyeQ5 & latest net/main
(46bc52d13594) as well. Thanks to Andrea for the quick reproducer.

	# udhcpc -i eth1
	...
	# iperf3 -c $IP -P10 -t3000
	...
	^C
	# dmesg | grep eth1
	[    2.182191] macb 2b00000.ethernet eth1: Cadence GEM rev 0x00070200 at 0x02b00000 irq 35 (00:28:f8:94:24:69)
	[   23.570335] macb 2b00000.ethernet eth1: PHY [2b00000.ethernet-ffffffff:0e] driver [Marvell 88E1510] (irq=POLL)
	[   23.570981] macb 2b00000.ethernet eth1: configuring for phy/rgmii-id link mode
	[   27.783260] macb 2b00000.ethernet eth1: Link is Up - 1Gbps/Full - flow control tx
	[   42.182370] macb 2b00000.ethernet eth1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 5088 ms
	[   51.158331] macb 2b00000.ethernet eth1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 5004 ms
	[   56.262381] macb 2b00000.ethernet eth1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 10108 ms
	[   61.126317] macb 2b00000.ethernet eth1: NETDEV WATCHDOG: CPU: 3: transmit queue 0 timed out 14972 ms
	[   67.206322] macb 2b00000.ethernet eth1: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 5004 ms

Note: the initial reproducer was `iperf3 -c $IP -P10 -t3000 -w4M`.
I can reproduce without -w4M but -P10 is required even though we reach
line rate with a single stream. This points to a race/mb issue in
macb_start_xmit?

I'm posting to see if anyone has theories. I'll be posting some race
fixes soon but they don't fix it (not surprising as they are unrelated).
I have many ideas but I need more time for testing; it'll be much easier
now with a reproduction setup.

[0]: https://lore.kernel.org/netdev/20260514215459.36109-2-lukasz@raczylo.com/
[1]: https://lore.kernel.org/netdev/DIK002QFFNBY.31C3KUX2SQC6W@bootlin.com/
[2]: https://elixir.bootlin.com/linux/v7.2.5/source/net/sched/sch_generic.c#L563-L569
[3]: https://dtype.org/wiki/Cm5_macb_network_hang

Thanks,

--
Théo Lebrun, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


      parent reply	other threads:[~2026-09-18 15:23 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-06-16 13:23 [PATCH v2] net: macb: add TX stall timeout callback to recover from lost TSTART write Andrea della Porta
2026-06-16 13:37 ` Nicolai Buchwitz
2026-06-16 15:07 ` Théo Lebrun
2026-06-19  7:17   ` Andrea della Porta
2026-06-19  7:39     ` Nicolai Buchwitz
2026-06-19  1:20 ` patchwork-bot+netdevbpf
2026-09-18 15:22 ` Théo Lebrun [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DLIJN4CDS11K.RVJGK3UALMJE@bootlin.com \
    --to=theo.lebrun@bootlin.com \
    --cc=andrea.porta@suse.com \
    --cc=andrew+netdev@lunn.ch \
    --cc=conor.dooley@microchip.com \
    --cc=davem@davemloft.net \
    --cc=dtype@dtype.org \
    --cc=edumazet@google.com \
    --cc=kuba@kernel.org \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-rpi-kernel@lists.infradead.org \
    --cc=lukasz@raczylo.com \
    --cc=nb@tipi-net.de \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=sjaeckel@suse.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.