From: Ben Greear <greearb@candelatech.com>
To: NetDev <netdev@vger.kernel.org>
Subject: Detected Tx Unit Hang in ixgbe, kernel 2.6.25
Date: Tue, 06 May 2008 10:04:29 -0700 [thread overview]
Message-ID: <48208F9D.1080608@candelatech.com> (raw)
I'm using a 10Gbps copper(CX4) dual-port NIC from silicomusa.com.
It uses the Intel chipset and ixgbe driver. I'm using
kernel 2.6.25 plus some hacks (no patches to ixgbe).
This particular test case was to create 500 mac-vlans on
each of the two ports and generate UDP traffic between
them (I have a version of the send-to-self patch applied
to my kernel and enabled.)
During the setup for this test, the interfaces would have
been bounced (effectively ifdown, ifup), so that is the
reason for the link going up and down.
I noticed 90%+ drop rate when I first started the test,
and then after maybe 1-2 minutes, things calmed down and
started working. I checked /var/log/messages and saw the
messages below.
I previously ran 5Gbps of traffic through the two ports
with them acting like a bridge for more than 24-hours without
any obvious problems, so I think the hardware is probably OK.
May 6 09:51:41 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:51:41 simech-ice kernel: TDH <1e>
May 6 09:51:41 simech-ice kernel: TDT <3ff>
May 6 09:51:41 simech-ice kernel: next_to_use <3ff>
May 6 09:51:41 simech-ice kernel: next_to_clean <1a>
May 6 09:51:41 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:51:41 simech-ice kernel: time_stamp <11e035210>
May 6 09:51:41 simech-ice kernel: next_to_watch <1b>
May 6 09:51:41 simech-ice kernel: jiffies <11e035862>
May 6 09:51:41 simech-ice kernel: next_to_watch.status <17a8209>
May 6 09:51:41 simech-ice kernel: ixgbe: eth2: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:51:41 simech-ice kernel: TDH <3d6>
May 6 09:51:41 simech-ice kernel: TDT <3b0>
May 6 09:51:41 simech-ice kernel: next_to_use <3b0>
May 6 09:51:41 simech-ice kernel: next_to_clean <3d2>
May 6 09:51:41 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:51:41 simech-ice kernel: time_stamp <11e035211>
May 6 09:51:41 simech-ice kernel: next_to_watch <3d3>
May 6 09:51:41 simech-ice kernel: jiffies <11e035887>
May 6 09:51:41 simech-ice kernel: next_to_watch.status <17a8209>
May 6 09:51:46 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:51:46 simech-ice kernel: TDH <28d>
May 6 09:51:46 simech-ice kernel: TDT <26c>
May 6 09:51:46 simech-ice kernel: next_to_use <26c>
May 6 09:51:46 simech-ice kernel: next_to_clean <289>
May 6 09:51:46 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:51:46 simech-ice kernel: time_stamp <11e0363e0>
May 6 09:51:46 simech-ice kernel: next_to_watch <28a>
May 6 09:51:46 simech-ice kernel: jiffies <11e036e8e>
May 6 09:51:46 simech-ice kernel: next_to_watch.status <17a8209>
May 6 09:51:46 simech-ice kernel: ixgbe: eth2: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:51:46 simech-ice kernel: TDH <1bd>
May 6 09:51:46 simech-ice kernel: TDT <19c>
May 6 09:51:46 simech-ice kernel: next_to_use <19c>
May 6 09:51:46 simech-ice kernel: next_to_clean <1b9>
May 6 09:51:46 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:51:46 simech-ice kernel: time_stamp <11e036346>
May 6 09:51:46 simech-ice kernel: next_to_watch <1ba>
May 6 09:51:46 simech-ice kernel: jiffies <11e036e9a>
May 6 09:51:46 simech-ice kernel: next_to_watch.status <17a8209>
May 6 09:51:47 simech-ice kernel: ixgbe: eth2: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:51:47 simech-ice kernel: TDH <29e>
May 6 09:51:47 simech-ice kernel: TDT <27c>
May 6 09:51:47 simech-ice kernel: next_to_use <27c>
May 6 09:51:47 simech-ice kernel: next_to_clean <29a>
May 6 09:51:47 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:51:47 simech-ice kernel: time_stamp <11e0363e0>
May 6 09:51:47 simech-ice kernel: next_to_watch <29b>
May 6 09:51:47 simech-ice kernel: jiffies <11e036fee>
May 6 09:51:47 simech-ice kernel: next_to_watch.status <17a8209>
May 6 09:51:47 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:51:47 simech-ice kernel: TDH <33f>
May 6 09:51:47 simech-ice kernel: TDT <321>
May 6 09:51:47 simech-ice kernel: next_to_use <321>
May 6 09:51:47 simech-ice kernel: next_to_clean <33b>
May 6 09:51:47 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:51:47 simech-ice kernel: time_stamp <11e0363e2>
May 6 09:51:47 simech-ice kernel: next_to_watch <33c>
May 6 09:51:47 simech-ice kernel: jiffies <11e036ff5>
May 6 09:51:47 simech-ice kernel: next_to_watch.status <17a8209>
May 6 09:51:51 simech-ice kernel: ixgbe: eth2: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:51:51 simech-ice kernel: TDH <398>
May 6 09:51:51 simech-ice kernel: TDT <374>
May 6 09:51:51 simech-ice kernel: next_to_use <374>
May 6 09:51:51 simech-ice kernel: next_to_clean <394>
May 6 09:51:51 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:51:51 simech-ice kernel: time_stamp <11e037748>
May 6 09:51:51 simech-ice kernel: next_to_watch <395>
May 6 09:51:51 simech-ice kernel: jiffies <11e038251>
May 6 09:51:51 simech-ice kernel: next_to_watch.status <17a8209>
May 6 09:51:51 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:51:51 simech-ice kernel: TDH <101>
May 6 09:51:51 simech-ice kernel: TDT <dd>
May 6 09:51:51 simech-ice kernel: next_to_use <dd>
May 6 09:51:51 simech-ice kernel: next_to_clean <fd>
May 6 09:51:51 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:51:51 simech-ice kernel: time_stamp <11e037743>
May 6 09:51:51 simech-ice kernel: next_to_watch <fe>
May 6 09:51:51 simech-ice kernel: jiffies <11e03825c>
May 6 09:51:51 simech-ice kernel: next_to_watch.status <17a8209>
May 6 09:52:00 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:52:00 simech-ice kernel: TDH <2b5>
May 6 09:52:00 simech-ice kernel: TDT <292>
May 6 09:52:00 simech-ice kernel: next_to_use <292>
May 6 09:52:00 simech-ice kernel: next_to_clean <2b1>
May 6 09:52:00 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:52:00 simech-ice kernel: time_stamp <11e038937>
May 6 09:52:00 simech-ice kernel: next_to_watch <2b2>
May 6 09:52:00 simech-ice kernel: jiffies <11e03a29c>
May 6 09:52:00 simech-ice kernel: next_to_watch.status <17a8209>
May 6 09:52:00 simech-ice kernel: ixgbe: eth2: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:52:00 simech-ice kernel: TDH <8>
May 6 09:52:00 simech-ice kernel: TDT <3e6>
May 6 09:52:00 simech-ice kernel: next_to_use <3e6>
May 6 09:52:00 simech-ice kernel: next_to_clean <4>
May 6 09:52:00 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:52:00 simech-ice kernel: time_stamp <11e038957>
May 6 09:52:00 simech-ice kernel: next_to_watch <5>
May 6 09:52:00 simech-ice kernel: jiffies <11e03a2d5>
May 6 09:52:00 simech-ice kernel: next_to_watch.status <17a8209>
May 6 09:52:11 simech-ice kernel: NETDEV WATCHDOG: eth3: transmit timed out
May 6 09:52:11 simech-ice kernel: NETDEV WATCHDOG: eth2: transmit timed out
May 6 09:52:11 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Down
May 6 09:52:11 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:52:11 simech-ice kernel: TDH <18c>
May 6 09:52:11 simech-ice kernel: TDT <12a>
May 6 09:52:11 simech-ice kernel: next_to_use <12a>
May 6 09:52:11 simech-ice kernel: next_to_clean <188>
May 6 09:52:11 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:52:11 simech-ice kernel: time_stamp <11e03aa83>
May 6 09:52:11 simech-ice kernel: next_to_watch <189>
May 6 09:52:11 simech-ice kernel: jiffies <11e03cde1>
May 6 09:52:11 simech-ice kernel: next_to_watch.status <17a8209>
May 6 09:52:11 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 6 09:52:11 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Down
May 6 09:52:11 simech-ice kernel: ADDRCONF(NETDEV_UP): eth3#435: link is not ready
May 6 09:52:11 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 6 09:52:11 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 6 09:52:11 simech-ice kernel: ADDRCONF(NETDEV_CHANGE): eth3#435: link becomes ready
May 6 09:52:22 simech-ice kernel: NETDEV WATCHDOG: eth3: transmit timed out
May 6 09:52:22 simech-ice kernel: NETDEV WATCHDOG: eth2: transmit timed out
May 6 09:52:23 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Down
May 6 09:52:23 simech-ice kernel: ixgbe: eth2: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:52:23 simech-ice kernel: TDH <19b>
May 6 09:52:23 simech-ice kernel: TDT <173>
May 6 09:52:23 simech-ice kernel: next_to_use <173>
May 6 09:52:23 simech-ice kernel: next_to_clean <197>
May 6 09:52:23 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:52:23 simech-ice kernel: time_stamp <11e03d200>
May 6 09:52:23 simech-ice kernel: next_to_watch <198>
May 6 09:52:23 simech-ice kernel: jiffies <11e03fcd1>
May 6 09:52:23 simech-ice kernel: next_to_watch.status <17a8209>
May 6 09:52:23 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 6 09:52:23 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Down
May 6 09:52:23 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 6 09:52:23 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Down
May 6 09:52:23 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 6 09:52:23 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Down
May 6 09:52:23 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 6 09:52:23 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 6 09:52:27 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:52:27 simech-ice kernel: TDH <6>
May 6 09:52:27 simech-ice kernel: TDT <3e4>
May 6 09:52:27 simech-ice kernel: next_to_use <3e4>
May 6 09:52:27 simech-ice kernel: next_to_clean <2>
May 6 09:52:27 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:52:27 simech-ice kernel: time_stamp <11e0400bb>
May 6 09:52:27 simech-ice kernel: next_to_watch <3>
May 6 09:52:27 simech-ice kernel: jiffies <11e040d75>
May 6 09:52:27 simech-ice kernel: next_to_watch.status <17a8209>
May 6 09:52:34 simech-ice kernel: NETDEV WATCHDOG: eth3: transmit timed out
May 6 09:52:34 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Down
May 6 09:52:34 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 6 09:52:34 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Down
May 6 09:52:34 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 6 09:52:34 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 6 09:52:42 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:52:42 simech-ice kernel: TDH <189>
May 6 09:52:42 simech-ice kernel: TDT <159>
May 6 09:52:42 simech-ice kernel: next_to_use <159>
May 6 09:52:42 simech-ice kernel: next_to_clean <184>
May 6 09:52:42 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:52:42 simech-ice kernel: time_stamp <11e042edb>
May 6 09:52:42 simech-ice kernel: next_to_watch <185>
May 6 09:52:42 simech-ice kernel: jiffies <11e0449ec>
May 6 09:52:42 simech-ice kernel: next_to_watch.status <17a8209>
May 6 09:52:45 simech-ice kernel: NETDEV WATCHDOG: eth2: transmit timed out
May 6 09:52:48 simech-ice kernel: ixgbe: eth2: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:52:48 simech-ice kernel: TDH <d>
May 6 09:52:48 simech-ice kernel: TDT <3e6>
May 6 09:52:48 simech-ice kernel: next_to_use <3e6>
May 6 09:52:48 simech-ice kernel: next_to_clean <9>
May 6 09:52:48 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:52:48 simech-ice kernel: time_stamp <11e042de5>
May 6 09:52:48 simech-ice kernel: next_to_watch <a>
May 6 09:52:48 simech-ice kernel: jiffies <11e045e0b>
May 6 09:52:48 simech-ice kernel: next_to_watch.status <17a8209>
May 6 09:52:48 simech-ice kernel: ixgbe: eth2: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:52:48 simech-ice kernel: TDH <78>
May 6 09:52:48 simech-ice kernel: TDT <52>
May 6 09:52:48 simech-ice kernel: next_to_use <52>
May 6 09:52:48 simech-ice kernel: next_to_clean <73>
May 6 09:52:48 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:52:48 simech-ice kernel: time_stamp <11e042e11>
May 6 09:52:48 simech-ice kernel: next_to_watch <74>
May 6 09:52:48 simech-ice kernel: jiffies <11e045e7c>
May 6 09:52:48 simech-ice kernel: next_to_watch.status <17a8209>
May 6 09:52:48 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Down
May 6 09:52:48 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 6 09:52:48 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Down
May 6 09:52:48 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 6 09:52:48 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 6 09:52:59 simech-ice kernel: NETDEV WATCHDOG: eth3: transmit timed out
May 6 09:52:59 simech-ice kernel: NETDEV WATCHDOG: eth2: transmit timed out
May 6 09:52:59 simech-ice kernel: ixgbe: eth2: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 6 09:52:59 simech-ice kernel: ixgbe: eth3: ixgbe_watchdog: NIC Link is Up 10 Gbps, Flow Control: RX/TX
May 6 09:53:07 simech-ice kernel: ixgbe: eth2: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:53:07 simech-ice kernel: TDH <28>
May 6 09:53:07 simech-ice kernel: TDT <3>
May 6 09:53:07 simech-ice kernel: next_to_use <3>
May 6 09:53:07 simech-ice kernel: next_to_clean <23>
May 6 09:53:07 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:53:07 simech-ice kernel: time_stamp <11e049a4d>
May 6 09:53:07 simech-ice kernel: next_to_watch <24>
May 6 09:53:07 simech-ice kernel: jiffies <11e04a866>
May 6 09:53:07 simech-ice kernel: next_to_watch.status <17a8209>
May 6 09:53:07 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:53:07 simech-ice kernel: TDH <2ad>
May 6 09:53:07 simech-ice kernel: TDT <28c>
May 6 09:53:07 simech-ice kernel: next_to_use <28c>
May 6 09:53:07 simech-ice kernel: next_to_clean <2a7>
May 6 09:53:07 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:53:07 simech-ice kernel: time_stamp <11e04979e>
May 6 09:53:07 simech-ice kernel: next_to_watch <2a8>
May 6 09:53:07 simech-ice kernel: jiffies <11e04a880>
May 6 09:53:07 simech-ice kernel: next_to_watch.status <17a8209>
May 6 09:53:10 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:53:10 simech-ice kernel: TDH <129>
May 6 09:53:10 simech-ice kernel: TDT <103>
May 6 09:53:10 simech-ice kernel: next_to_use <103>
May 6 09:53:10 simech-ice kernel: next_to_clean <125>
May 6 09:53:10 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:53:10 simech-ice kernel: time_stamp <11e04b236>
May 6 09:53:10 simech-ice kernel: next_to_watch <126>
May 6 09:53:10 simech-ice kernel: jiffies <11e04b61f>
May 6 09:53:10 simech-ice kernel: next_to_watch.status <17a8209>
May 6 09:53:14 simech-ice kernel: ixgbe: eth3: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:53:14 simech-ice kernel: TDH <18e>
May 6 09:53:14 simech-ice kernel: TDT <165>
May 6 09:53:14 simech-ice kernel: next_to_use <165>
May 6 09:53:14 simech-ice kernel: next_to_clean <189>
May 6 09:53:14 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:53:14 simech-ice kernel: time_stamp <11e04b24c>
May 6 09:53:14 simech-ice kernel: next_to_watch <18a>
May 6 09:53:14 simech-ice kernel: jiffies <11e04c4e4>
May 6 09:53:14 simech-ice kernel: next_to_watch.status <17a8209>
May 6 09:53:14 simech-ice kernel: ixgbe: eth2: ixgbe_check_tx_hang: Detected Tx Unit Hang
May 6 09:53:14 simech-ice kernel: TDH <3b>
May 6 09:53:14 simech-ice kernel: TDT <16>
May 6 09:53:14 simech-ice kernel: next_to_use <16>
May 6 09:53:14 simech-ice kernel: next_to_clean <37>
May 6 09:53:14 simech-ice kernel: tx_buffer_info[next_to_clean]
May 6 09:53:14 simech-ice kernel: time_stamp <11e04b1d7>
May 6 09:53:14 simech-ice kernel: next_to_watch <38>
May 6 09:53:14 simech-ice kernel: jiffies <11e04c6e3>
May 6 09:53:14 simech-ice kernel: next_to_watch.status <17a8209>
Thanks,
Ben
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
next reply other threads:[~2008-05-06 17:04 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-05-06 17:04 Ben Greear [this message]
2008-05-06 20:42 ` Detected Tx Unit Hang in ixgbe, kernel 2.6.25 Brandeburg, Jesse
2008-05-06 20:58 ` Ben Greear
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=48208F9D.1080608@candelatech.com \
--to=greearb@candelatech.com \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).