Re: [BUG] ixgbe: Detected Tx Unit Hang (XDP)

[Michal Kubiak <michal.kubiak@intel.com>]

On Thu, Apr 10, 2025 at 04:54:35PM +0200, Marcus Wichelmann wrote:
> Am 10.04.25 um 16:30 schrieb Michal Kubiak:
> > On Wed, Apr 09, 2025 at 05:17:49PM +0200, Marcus Wichelmann wrote:
> >> Hi,
> >>
> >> in a setup where I use native XDP to redirect packets to a bonding interface
> >> that's backed by two ixgbe slaves, I noticed that the ixgbe driver constantly
> >> resets the NIC with the following kernel output:
> >>
> >>   ixgbe 0000:01:00.1 ixgbe-x520-2: Detected Tx Unit Hang (XDP)
> >>     Tx Queue             <4>
> >>     TDH, TDT             <17e>, <17e>
> >>     next_to_use          <181>
> >>     next_to_clean        <17e>
> >>   tx_buffer_info[next_to_clean]
> >>     time_stamp           <0>
> >>     jiffies              <10025c380>
> >>   ixgbe 0000:01:00.1 ixgbe-x520-2: tx hang 19 detected on queue 4, resetting adapter
> >>   ixgbe 0000:01:00.1 ixgbe-x520-2: initiating reset due to tx timeout
> >>   ixgbe 0000:01:00.1 ixgbe-x520-2: Reset adapter
> >>
> >> This only occurs in combination with a bonding interface and XDP, so I don't
> >> know if this is an issue with ixgbe or the bonding driver.
> >> I first discovered this with Linux 6.8.0-57, but kernel 6.14.0 and 6.15.0-rc1
> >> show the same issue.
> >>
> >>
> >> I managed to reproduce this bug in a lab environment. Here are some details
> >> about my setup and the steps to reproduce the bug:
> >>
> >> [...]
> >>
> >> Do you have any ideas what may be causing this issue or what I can do to
> >> diagnose this further?
> >>
> >> Please let me know when I should provide any more information.
> >>
> >>
> >> Thanks!
> >> Marcus
> >>
> > 
> > Hi Marcus,
> 
> Hi Michal,
> 
> thank you for looking into it. And not even 24 hours after my report, I'm
> very impressed! ;)
> 
> > I have just successfully reproduced the problem on our lab machine. What
> > is interesting is that I do not seem to have to use a bonding interface
> > to get the "Tx timeout" that causes the adapter to reset.
> 
> Interesting. I just tried again but had no luck yet with reproducing it
> without a bonding interface. May I ask how your setup looks like?
> 
> > I will try to debug the problem more closely and let you know of any
> > updates.
> > 
> > Thanks,
> > Michal
> 
> Great!
> 
> Marcus
>

Hi Marcus,

> thank you for looking into it. And not even 24 hours after my report, I'm
> very impressed! ;)

Thanks! :-)

> Interesting. I just tried again but had no luck yet with reproducing it
> without a bonding interface. May I ask how your setup looks like?

For now, I've just grabbed the first available system with the HW
controlled by the "ixgbe" driver. In my case it was:

  Ethernet controller: Intel Corporation Ethernet Controller X550

Also, for my first attempt, I didn't use the upstream kernel - I just tried
the kernel installed on that system. It was the Fedora kernel:

  6.12.8-200.fc41.x86_64


I think that may be the "beauty" of timing issues - sometimes you can change
just one piece in your system and get a completely different replication ratio.
Anyway, the higher the repro probability, the easier it is to debug
the timing problem. :-)

Thanks,
Michal