niu lock-up (Transmit timed out, resetting) and NETDEV WATCHDOG

niu lock-up (Transmit timed out, resetting) and NETDEV WATCHDOG

[Andrew Brooks]

Hello

Using niu driver for this card: Oracle/SUN Multithreaded 10-Gigabit
Ethernet Network Controller
after a period (often less than 24 hours) the interface will hang with
errors every 5 seconds
"niu: xxx: eth2: Transmit timed out, resetting"

Sometimes also in syslog are messages
WARNING: at sch_generic:255 dev_watchdog
NETDEV WATCHDOG: eth2 (niu): transmit queue 10 timed out

I've seen this in kernel 3.5.0-26-generic #42~precise1-Ubuntu SMP
but I've not seen it in kernel 3.2.0-38-generic #61-Ubuntu SMP

Is there some change between kernels which has broken the driver
or is the difference elsewhere?

Thanks

Andrew

Re: niu lock-up (Transmit timed out, resetting) and NETDEV WATCHDOG

[Andrew Brooks]

Hi folks

This is a show-stopper. Does anyone have any ideas?

Thanks!

On 26 March 2013 13:46, Andrew Brooks <arb@sat.dundee.ac.uk> wrote:
>
> Hello
>
> Using niu driver for this card: Oracle/SUN Multithreaded 10-Gigabit
> Ethernet Network Controller
> after a period (often less than 24 hours) the interface will hang with
> errors every 5 seconds
> "niu: xxx: eth2: Transmit timed out, resetting"
>
> Sometimes also in syslog are messages
> WARNING: at sch_generic:255 dev_watchdog
> NETDEV WATCHDOG: eth2 (niu): transmit queue 10 timed out
>
> I've seen this in kernel 3.5.0-26-generic #42~precise1-Ubuntu SMP
> but I've not seen it in kernel 3.2.0-38-generic #61-Ubuntu SMP
>
> Is there some change between kernels which has broken the driver
> or is the difference elsewhere?

Re: niu lock-up (Transmit timed out, resetting) and NETDEV WATCHDOG

[Andrew Brooks]

Hi

This bug occurs in kernel 3.2.0-39 too.

Is anyone able to offer any advice?  Thanks.



> On 26 March 2013 13:46, Andrew Brooks <arb@sat.dundee.ac.uk> wrote:
> >
> > Using niu driver for this card: Oracle/SUN Multithreaded 10-Gigabit
> > Ethernet Network Controller
> > after a period (often less than 24 hours) the interface will hang with
> > errors every 5 seconds
> > "niu: xxx: eth2: Transmit timed out, resetting"
> >
> > Sometimes also in syslog are messages
> > WARNING: at sch_generic:255 dev_watchdog
> > NETDEV WATCHDOG: eth2 (niu): transmit queue 10 timed out
> >
> > I've seen this in kernel 3.5.0-26-generic #42~precise1-Ubuntu SMP
> > but I've not seen it in kernel 3.2.0-38-generic #61-Ubuntu SMP
> >
> > Is there some change between kernels which has broken the driver
> > or is the difference elsewhere?

Re: niu lock-up (Transmit timed out, resetting) and NETDEV WATCHDOG

[Andrew Brooks]

On 26 March 2013 13:44, Andrew Brooks <arb@sat.dundee.ac.uk> wrote:
>
> Using niu driver for this card: Oracle/SUN Multithreaded 10-Gigabit
> Ethernet Network Controller and after a period the interface will hang
> with errors every 5 seconds
> "niu: xxx: eth2: Transmit timed out, resetting"
>
> Sometimes also in syslog are messages
> WARNING: at sch_generic:255 dev_watchdog
> NETDEV WATCHDOG: eth2 (niu): transmit queue 10 timed out

Do you think this could be caused by a problem I've seen reported
by other machines on the network
"received unsolicited ack for DL_UNITDATA_REQ on nxge0" ?
Is there some bad packet flying around that causes the
niu driver to lock up the kernel?

Re: niu lock-up (Transmit timed out, resetting) and NETDEV WATCHDOG

[Andrew Brooks]

Hi

> On 26 March 2013 13:44, Andrew Brooks <arb@sat.dundee.ac.uk> wrote:
>> Using niu driver for this card: Oracle/SUN Multithreaded 10-Gigabit
>> Ethernet Network Controller and after a period the interface will hang
>> with errors every 5 seconds
>> "niu: xxx: eth2: Transmit timed out, resetting"

Here's more information about the problem:
When the interface hangs we see these messages from the driver:

[3408740.816032] niu: niu_interrupt() ldg[ffff8807141d16d0](18)
v0[8000000000] v1[0] v2[0]
[3408740.816036] niu 0000:09:00.0: eth2: niu_txchan_intr() cs[b860b860000c000]
[3408740.816038] niu 0000:09:00.0: eth2: niu_poll_core() v0[0000008000000000]
[3408740.816040] niu 0000:09:00.0: eth2: niu_tx_work() pkt_cnt[0] cons[119]
[3408740.816042] niu: niu_interrupt() ldg[ffff8807141d16d0](18)
v0[8000000000] v1[0] v2[0]
[3408740.820004] [sched_delayed] sched: RT throttling activated
[3408740.824021] niu 0000:09:00.0: eth2: Disable interrupts
[3408740.824044] niu 0000:09:00.0: eth2: Disable RX MAC
[3408740.824048] niu 0000:09:00.0: eth2: Disable IPP
[3408740.824054] niu 0000:09:00.0: eth2: Stop TX channels
[3408740.824641] niu 0000:09:00.0: eth2: Stop RX channels
[3408740.824652] niu 0000:09:00.0: eth2: Reset TX channels
[3408740.825212] niu 0000:09:00.0: eth2: Reset RX channels
[3408740.825999] niu 0000:09:00.0: eth2: Initialize TXC
[3408740.826002] niu 0000:09:00.0: eth2: Initialize TX channels

However the interface doesn't recover :-(
Are there any clues there?

Thanks!