NETDEV WATCHDOG: eth1 (r8169): transmit queue 0 timed out

NETDEV WATCHDOG: eth1 (r8169): transmit queue 0 timed out

[Dave Jones]

We continue to see warnings like this reported against the Fedora kernel
for a number of different NICs. I just hit this one myself for the first time
on that hardware iirc.

Anything else I can provide ?

	Dave

[22347.960097] ------------[ cut here ]------------
[22347.965388] WARNING: at net/sched/sch_generic.c:255 dev_watchdog+0x258/0x270()
[22347.973764] Hardware name:         
[22347.977748] NETDEV WATCHDOG: eth1 (r8169): transmit queue 0 timed out
[22347.985212] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_LOG xt_limit ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 ip6table_filter xt_conntrack ip6_tables nf_conntrack microcode pcspkr r8169 mii nfsd auth_rpcgss nfs_acl lockd sunrpc i915 video backlight i2c_algo_bit drm_kms_helper drm
[22348.020948] Pid: 0, comm: swapper/3 Not tainted 3.7.0-rc8+ #5
[22348.027483] Call Trace:
[22348.030420]  <IRQ>  [<ffffffff8104928f>] warn_slowpath_common+0x7f/0xc0
[22348.038029]  [<ffffffff81049386>] warn_slowpath_fmt+0x46/0x50
[22348.044710]  [<ffffffff814d35a8>] dev_watchdog+0x258/0x270
[22348.051096]  [<ffffffff814d3350>] ? dev_deactivate_queue.constprop.30+0x80/0x80
[22348.059408]  [<ffffffff8105afea>] call_timer_fn+0x8a/0x330
[22348.065791]  [<ffffffff8105af65>] ? call_timer_fn+0x5/0x330
[22348.072276]  [<ffffffff814d3350>] ? dev_deactivate_queue.constprop.30+0x80/0x80
[22348.080721]  [<ffffffff814d3350>] ? dev_deactivate_queue.constprop.30+0x80/0x80
[22348.089032]  [<ffffffff8105b4d4>] run_timer_softirq+0x244/0x380
[22348.095913]  [<ffffffff81052a00>] __do_softirq+0xe0/0x3c0
[22348.102204]  [<ffffffff810a71d4>] ? tick_program_event+0x24/0x30
[22348.109045]  [<ffffffff815e36cc>] call_softirq+0x1c/0x26
[22348.115236]  [<ffffffff810043cd>] do_softirq+0x8d/0xc0
[22348.121226]  [<ffffffff81052eb5>] irq_exit+0xd5/0xe0
[22348.126882]  [<ffffffff815e381b>] smp_apic_timer_interrupt+0x6b/0x98
[22348.134253]  [<ffffffff815e302f>] apic_timer_interrupt+0x6f/0x80
[22348.141293]  <EOI>  [<ffffffff8108bad5>] ? sched_clock_local+0x25/0xa0
[22348.148819]  [<ffffffff81343d3d>] ? intel_idle+0xfd/0x160
[22348.155110]  [<ffffffff81343d36>] ? intel_idle+0xf6/0x160
[22348.161399]  [<ffffffff81474dc9>] cpuidle_enter+0x19/0x20
[22348.167562]  [<ffffffff81475462>] cpuidle_idle_call+0xa2/0x5f0
[22348.174346]  [<ffffffff8100c59a>] cpu_idle+0xca/0x160
[22348.180235]  [<ffffffff81ceead4>] start_secondary+0x251/0x258
[22348.186776] ---[ end trace 62a349a839e1b608 ]---
[22348.261360] r8169 0000:05:00.0 eth1: link up

Re: NETDEV WATCHDOG: eth1 (r8169): transmit queue 0 timed out

[Ben Hutchings]

On Tue, 2012-12-04 at 20:19 -0500, Dave Jones wrote:
> We continue to see warnings like this reported against the Fedora kernel
> for a number of different NICs. I just hit this one myself for the first time
> on that hardware iirc.
> 
> Anything else I can provide ?
[...]

In general, useful information might include:
- was this preceded by any interface reconfiguration or link changes?
- extended network stats (ethtool -S)
- MDIO register dump (mii-tool -vv) (if the interface has an MDIO PHY)

Having seen this error many times with different causes, I wrote a short
summary for the support team here, which (with some references removed)
may be generally useful:

---
The watchdog will fire if all these conditions are met:
1. The interface is up
2. A TX queue is stopped (normally because it is full)
3. No packets have been added to the queue in the last 5 seconds
4. The driver has not told the kernel that the device is unable to
transmit now (e.g. link is down).

Conditions 2 and 3 together normally mean that the TX queue has been
stopped for 5 seconds and therefore that few packets (not necessarily
none at all) have been completed in that time.  The time taken for
individual packets to be completed is *not* considered.

This can happen due to:
a. Driver bug causing conditions 2 and 4 to be true during
reconfiguration
b. MAC blocked by a pause frame flood
c. IRQ handling is delayed by a long time (can happen due to excessive
serial logging)
d. Firmware bug causes driver to see link as up when it's not
e. Hardware fault (always a possibility)
---

Item d should really be expanded to hardware/firmware/software bug.

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: NETDEV WATCHDOG: eth1 (r8169): transmit queue 0 timed out

[Francois Romieu]

Dave Jones <davej@redhat.com> :
> Mezigues : 
> > Dave Jones <davej@redhat.com> :
> > [...]
> > > Anything else I can provide ?
> > Please send the XID of the device.
> 
> [   55.576012] r8169 0000:01:00.0 eth0: RTL8168d/8111d at 0xffffc9000037a000, 70:71:bc:63:19:05, XID 081000c0 IRQ 46
> [   55.805446] r8169 0000:05:00.0 eth1: RTL8110s at 0xffffc9000037e000, 00:0f:b5:fb:c3:a6, XID 04000000 IRQ 21
> 
> The one that seems to be complaining is the 2nd one.

I'll plug one and see if it went more touchy than usual. 

I do not see anything obvious at first sight. Reverting
aee77e4accbeb2c86b1d294cd84fec4a12dde3bd maybe.

-- 
Ueimor