Netdev List
 help / color / mirror / Atom feed
From: Nicolai Buchwitz <nb@tipi-net.de>
To: Chris Babroski <cbabroski@nvidia.com>
Cc: Raju.Lakkaraju@microchip.com, UNGLinuxDriver@microchip.com,
	andrew+netdev@lunn.ch, bryan.whitehead@microchip.com,
	davem@davemloft.net, davthompson@nvidia.com, edumazet@kernel.org,
	kuba@kernel.org, linux-kernel@vger.kernel.org,
	linux@armlinux.org.uk, maxime.chevallier@bootlin.com,
	netdev@vger.kernel.org, pabeni@redhat.com,
	richardcochran@gmail.com
Subject: Re: [PATCH net-next v1 0/5] net: lan743x: automatically recover from TX hang
Date: Wed, 21 Oct 2026 10:06:57 +0200	[thread overview]
Message-ID: <c0117e4e987c3bd349e56a979cbbdbc5@tipi-net.de> (raw)
In-Reply-To: <20261020203641.335809-1-cbabroski@nvidia.com>

Hi Chris

On 20.10.2026 22:36, Chris Babroski wrote:
> [...]

> Interestingly, we are consistently seeing FCT_TX_USED=0x34 when
> reproducing the TX hang. We have FCT_TX_FIFO_END=0x27, default of 20kB,
> so it is mostly empty when we reproduce. Is there anything special 
> about
> the 52 byte usage?

Not that I know of. It looks like the head of a scatter-gather frame
whose fragments never came in. Does it match the buffer_length of the
first active descriptor at tx->last_head?

> 
>> Is the MAC in LPI, or paused by the link partner? Does it still happen
>> with EEE and rx pause turned off?
> 
> In all repro cases so far the MAC does not appear to be in LPI and EEE
> is disabled (MAC_CR=0x200C and EEE_EN is clear). I also tested with
> "ethtool -A <if> autoneg off rx off tx off" and could still reproduce
> the tx hang.
> 
> We have a PCIe switch connecting the NIC to our SoC and on the switch I
> can see that the NIC's read requests drop to ~0 implying that it stops
> fetching TX data and descriptors rather than the MAC stopping queue
> drain.

Which one? Ours sit behind a Pericom PI7C9X2G404.

> 
>> The other place to look is the PCIe link. We had a LAN7430 stall on a
>> BCM2712 root complex that was also platform specific. The RC driver
>> cleared an undocumented bit in its CLKREQ#/L1 control register. That
>> one took the whole MMIO window down, so it is probably a different
>> failure, but it took days of uptime to show up too. Do the Device 
>> Status
>> or AER registers on the hung device show anything (UnsupReq, CmpltTO)?
>> Is the link running with ASPM L1 or L1 substates on that platform?
> 
> I do think the PCIe link could be related; our current best guess is
> that there is some issue between the NIC and PCIe switch and we have
> tickets open with both vendors to help debug (along with setting up 
> EVBs
> to run comparison tests on).
> 
> UnsupReq and CmpltTO appear clean. ASPM and L1 substates are disabled;
> the link stays in L0 at 2.5 GT/s x1.
> 
> Recently we found that after every reproduced hang we see that the NIC
> sent a NAK to the switch, potentially in response to a CplD from switch
> while reading TX data and descriptors, but I am not yet able to see the
> rejected TLP or NAK reason with my setup.

Then BadTLP/BadDLLP on the NIC and Rollover/Timeout on the switch port
should tell what went wrong with the replay. Are those clean too?

My guess: the replay hands the NIC a completion it had already counted,
and the DMA engine then waits for data that never comes, with nothing
in the error registers.

Lowering MaxReadReq to 128 bytes with setpci would be a cheap way to 
test that?

> [...]

Regards,
Nicolai

  reply	other threads:[~2026-10-21  8:07 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-15 15:00 [PATCH net-next v1 0/5] net: lan743x: automatically recover from TX hang Chris Babroski
2026-10-15 15:00 ` [PATCH net-next v1 1/5] net: lan743x: reset DMA channels that fail to stop Chris Babroski
2026-10-19 10:18   ` Nicolai Buchwitz
2026-10-15 15:00 ` [PATCH net-next v1 2/5] net: lan743x: start phylink last and stop it first Chris Babroski
2026-10-19 10:13   ` Nicolai Buchwitz
2026-10-15 15:00 ` [PATCH net-next v1 3/5] net: lan743x: quiesce TX before freeing the rings Chris Babroski
2026-10-19 10:14   ` Nicolai Buchwitz
2026-10-21 18:20   ` Woojung.Huh
2026-10-15 15:00 ` [PATCH net-next v1 4/5] net: lan743x: reset the device on TX timeout Chris Babroski
2026-10-19 10:23   ` Nicolai Buchwitz
2026-10-15 15:00 ` [PATCH net-next v1 5/5] net: lan743x: add byte queue limits support Chris Babroski
2026-10-19 10:14   ` Nicolai Buchwitz
2026-10-19  9:47 ` [PATCH net-next v1 0/5] net: lan743x: automatically recover from TX hang Nicolai Buchwitz
2026-10-20 20:36   ` Chris Babroski
2026-10-21  8:06     ` Nicolai Buchwitz [this message]
2026-10-19 19:49 ` Woojung.Huh
2026-10-21  1:07 ` Jakub Kicinski
2026-10-21 14:03   ` Chris Babroski

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=c0117e4e987c3bd349e56a979cbbdbc5@tipi-net.de \
    --to=nb@tipi-net.de \
    --cc=Raju.Lakkaraju@microchip.com \
    --cc=UNGLinuxDriver@microchip.com \
    --cc=andrew+netdev@lunn.ch \
    --cc=bryan.whitehead@microchip.com \
    --cc=cbabroski@nvidia.com \
    --cc=davem@davemloft.net \
    --cc=davthompson@nvidia.com \
    --cc=edumazet@kernel.org \
    --cc=kuba@kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux@armlinux.org.uk \
    --cc=maxime.chevallier@bootlin.com \
    --cc=netdev@vger.kernel.org \
    --cc=pabeni@redhat.com \
    --cc=richardcochran@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox