Kernel v3.0.8 igb driver dies when pulling network cable

Kernel v3.0.8 igb driver dies when pulling network cable

[Stefan Priebe]

Hi List,

i triggered this bug nultiple times by just pulling out the network 
cable while running v3.0.8.

[  838.740689] igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow 
Control: RX/TX
[  838.761489] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  848.654167] ------------[ cut here ]------------
[  848.673476] WARNING: at net/sched/sch_generic.c:255 
dev_watchdog+0x25d/0x270()
[  848.692857] Hardware name: X8DTN
[  848.712062] NETDEV WATCHDOG: eth0 (igb): transmit queue 0 timed out
[  848.731613] Modules linked in: xt_tcpudp iptable_filter ip_tables 
x_tables coretemp k8temp ipv6 dm_snapshot dm_mod
[  848.752265] Pid: 0, comm: swapper Not tainted 2.6.40.8intel #3
[  848.773156] Call Trace:
[  848.793730]  <IRQ>  [<ffffffff8153be3d>] ? dev_watchdog+0x25d/0x270
[  848.793740]  [<ffffffff81045eb0>] warn_slowpath_common+0x80/0xc0
[  848.793744]  [<ffffffff81045fe9>] warn_slowpath_fmt+0x69/0x70
[  848.793749]  [<ffffffff81012288>] ? intel_pmu_nhm_enable_all+0x158/0x190
[  848.793755]  [<ffffffff81469917>] ? igb_poll+0x807/0xde0
[  848.793758]  [<ffffffff810125e4>] ? x86_pmu_enable+0x1f4/0x270
[  848.793763]  [<ffffffff810b1666>] ? perf_event_task_tick+0xa6/0x2e0
[  848.793767]  [<ffffffff8153be3d>] dev_watchdog+0x25d/0x270
[  848.793772]  [<ffffffff8105511c>] run_timer_softirq+0x1cc/0x320
[  848.793776]  [<ffffffff8153bbe0>] ? __netdev_watchdog_up+0x80/0x80
[  848.793782]  [<ffffffff8101d0d8>] ? lapic_next_event+0x18/0x20
[  848.793787]  [<ffffffff8104c390>] __do_softirq+0xd0/0x1c0
[  848.793793]  [<ffffffff815e288c>] call_softirq+0x1c/0x30
[  848.793798]  [<ffffffff810047b5>] do_softirq+0x55/0x90
[  848.793801]  [<ffffffff8104c16d>] irq_exit+0xad/0xe0
[  848.793805]  [<ffffffff8101ddc9>] smp_apic_timer_interrupt+0x69/0xa0
[  848.793809]  [<ffffffff815e2053>] apic_timer_interrupt+0x13/0x20
[  848.793811]  <EOI>  [<ffffffff81317093>] ? intel_idle+0xb3/0x120
[  848.793820]  [<ffffffff81317075>] ? intel_idle+0x95/0x120
[  848.793824]  [<ffffffff814f7d5c>] cpuidle_idle_call+0xdc/0x1a0
[  848.793829]  [<ffffffff81002091>] cpu_idle+0xb1/0x110
[  848.793833]  [<ffffffff815c54dd>] rest_init+0x6d/0x80
[  848.793837]  [<ffffffff81acbc5d>] start_kernel+0x3a2/0x44c
[  848.793847]  [<ffffffff81acb295>] x86_64_start_reservations+0xa5/0xc9
[  848.793851]  [<ffffffff81acb3a2>] x86_64_start_kernel+0xe9/0xfb
[  848.793853] ---[ end trace 737c145f3289e2ce ]---
[  848.794085] igb 0000:0a:00.0: eth0: Reset adapter
[  849.019407] eth0: no IPv6 routers present
[  851.626949] igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow 
Control: RX/TX
[  861.637255] igb 0000:0a:00.0: eth0: Reset adapter
[  864.479252] igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow 
Control: RX/TX

Greet Stefan

Re: Kernel v3.0.8 igb driver dies when pulling network cable

[Alexander Duyck]

Stefan,

You are going to need to clarify some things since as far as I know we
never released a v3.0.8 driver, and the kernel in the bug below is a
version 2.6.40.8 which sounds more like a modified and possibly
backported Linux kernel since as far as I know Linus never released a
2.6.40 kernel version.

In order to be able to assist you we will need some additional
information.  What Linux distro are your running?  Do you have the
latest updates from them?  Also have you tried running the latest
out-of-tree driver we provide at e1000.sf.net in order to verify if the
same issue occurs with that driver?

Thanks,

Alex

On 11/21/2011 04:26 AM, Stefan Priebe wrote:
> Hi List,
>
> i triggered this bug nultiple times by just pulling out the network
> cable while running v3.0.8.
>
> [  838.740689] igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow
> Control: RX/TX
> [  838.761489] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [  848.654167] ------------[ cut here ]------------
> [  848.673476] WARNING: at net/sched/sch_generic.c:255
> dev_watchdog+0x25d/0x270()
> [  848.692857] Hardware name: X8DTN
> [  848.712062] NETDEV WATCHDOG: eth0 (igb): transmit queue 0 timed out
> [  848.731613] Modules linked in: xt_tcpudp iptable_filter ip_tables
> x_tables coretemp k8temp ipv6 dm_snapshot dm_mod
> [  848.752265] Pid: 0, comm: swapper Not tainted 2.6.40.8intel #3
> [  848.773156] Call Trace:
> [  848.793730]  <IRQ>  [<ffffffff8153be3d>] ? dev_watchdog+0x25d/0x270
> [  848.793740]  [<ffffffff81045eb0>] warn_slowpath_common+0x80/0xc0
> [  848.793744]  [<ffffffff81045fe9>] warn_slowpath_fmt+0x69/0x70
> [  848.793749]  [<ffffffff81012288>] ?
> intel_pmu_nhm_enable_all+0x158/0x190
> [  848.793755]  [<ffffffff81469917>] ? igb_poll+0x807/0xde0
> [  848.793758]  [<ffffffff810125e4>] ? x86_pmu_enable+0x1f4/0x270
> [  848.793763]  [<ffffffff810b1666>] ? perf_event_task_tick+0xa6/0x2e0
> [  848.793767]  [<ffffffff8153be3d>] dev_watchdog+0x25d/0x270
> [  848.793772]  [<ffffffff8105511c>] run_timer_softirq+0x1cc/0x320
> [  848.793776]  [<ffffffff8153bbe0>] ? __netdev_watchdog_up+0x80/0x80
> [  848.793782]  [<ffffffff8101d0d8>] ? lapic_next_event+0x18/0x20
> [  848.793787]  [<ffffffff8104c390>] __do_softirq+0xd0/0x1c0
> [  848.793793]  [<ffffffff815e288c>] call_softirq+0x1c/0x30
> [  848.793798]  [<ffffffff810047b5>] do_softirq+0x55/0x90
> [  848.793801]  [<ffffffff8104c16d>] irq_exit+0xad/0xe0
> [  848.793805]  [<ffffffff8101ddc9>] smp_apic_timer_interrupt+0x69/0xa0
> [  848.793809]  [<ffffffff815e2053>] apic_timer_interrupt+0x13/0x20
> [  848.793811]  <EOI>  [<ffffffff81317093>] ? intel_idle+0xb3/0x120
> [  848.793820]  [<ffffffff81317075>] ? intel_idle+0x95/0x120
> [  848.793824]  [<ffffffff814f7d5c>] cpuidle_idle_call+0xdc/0x1a0
> [  848.793829]  [<ffffffff81002091>] cpu_idle+0xb1/0x110
> [  848.793833]  [<ffffffff815c54dd>] rest_init+0x6d/0x80
> [  848.793837]  [<ffffffff81acbc5d>] start_kernel+0x3a2/0x44c
> [  848.793847]  [<ffffffff81acb295>] x86_64_start_reservations+0xa5/0xc9
> [  848.793851]  [<ffffffff81acb3a2>] x86_64_start_kernel+0xe9/0xfb
> [  848.793853] ---[ end trace 737c145f3289e2ce ]---
> [  848.794085] igb 0000:0a:00.0: eth0: Reset adapter
> [  849.019407] eth0: no IPv6 routers present
> [  851.626949] igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow
> Control: RX/TX
> [  861.637255] igb 0000:0a:00.0: eth0: Reset adapter
> [  864.479252] igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow
> Control: RX/TX
>
> Greet Stefan

Re: Kernel v3.0.8 igb driver dies when pulling network cable

[Stefan Priebe]

> You are going to need to clarify some things since as far as I know we
> never released a v3.0.8 driver, and the kernel in the bug below is a
> version 2.6.40.8 which sounds more like a modified and possibly
> backported Linux kernel since as far as I know Linus never released a
> 2.6.40 kernel version.
Sorry that my description was so inaccurate. I'm talking about a vanilla 
v3.0.8 linux kernel. Just renamed to 2.6.40.8.

> In order to be able to assist you we will need some additional
> information.  What Linux distro are your running?  Do you have the
> latest updates from them?  Also have you tried running the latest
> out-of-tree driver we provide at e1000.sf.net in order to verify if the
> same issue occurs with that driver?
Linux Distri is Debian Lenny but we're running a self compiled vanilla 
3.0.X kernel. I haven't tried to use the up2date e1000.sf.net driver as 
3.0 is a long time supported kernel by greg - so all needed fixes should 
be backported or at least send to stable@ so that greg can have a look 
at them.

Thanks!

Stefan

Re: Kernel v3.0.8 igb driver dies when pulling network cable

[Alexander Duyck]

On 11/21/2011 10:07 AM, Stefan Priebe wrote:
>> You are going to need to clarify some things since as far as I know we
>> never released a v3.0.8 driver, and the kernel in the bug below is a
>> version 2.6.40.8 which sounds more like a modified and possibly
>> backported Linux kernel since as far as I know Linus never released a
>> 2.6.40 kernel version.
> Sorry that my description was so inaccurate. I'm talking about a
> vanilla v3.0.8 linux kernel. Just renamed to 2.6.40.8.
>
>> In order to be able to assist you we will need some additional
>> information.  What Linux distro are your running?  Do you have the
>> latest updates from them?  Also have you tried running the latest
>> out-of-tree driver we provide at e1000.sf.net in order to verify if the
>> same issue occurs with that driver?
> Linux Distri is Debian Lenny but we're running a self compiled vanilla
> 3.0.X kernel. I haven't tried to use the up2date e1000.sf.net driver
> as 3.0 is a long time supported kernel by greg - so all needed fixes
> should be backported or at least send to stable@ so that greg can have
> a look at them.
>
> Thanks!
>
> Stefan
Stefan,

It would be useful if you could try the latest driver from e1000.sf.net
just to verify  if this is a bug in the upstream kernel or if it is also
present in our e1000.sf.net.driver.  This way we can figure out if this
is an issue where a patch wasn't pushed into the stable kernel or if it
is an issue that still exists in our latest release.

Also could you provide us with the part number you are currently using. 
If you could provide us with the device ID for the part via lspci we can
start narrowing down the root cause for the issue as currently we don't
have any information about what hardware you are experiencing this issue on.

Thanks,

Alex

Re: Kernel v3.0.8 igb driver dies when pulling network cable

[Stefan Priebe - Profihost AG]

> It would be useful if you could try the latest driver from e1000.sf.net
> just to verify  if this is a bug in the upstream kernel or if it is also
> present in our e1000.sf.net.driver.  This way we can figure out if this
> is an issue where a patch wasn't pushed into the stable kernel or if it
> is an issue that still exists in our latest release.
>
> Also could you provide us with the part number you are currently using.
> If you could provide us with the device ID for the part via lspci we can
> start narrowing down the root cause for the issue as currently we don't
> have any information about what hardware you are experiencing this issue on.

Ok another note i missed last time. Ethernet and Server stays fully 
functional - it just prints the message and resets the adapter.

OK let's start with lspci:
a:00.0 Ethernet controller: Intel Corporation Device 10c9 (rev 01)
         Subsystem: Super Micro Computer Inc Device 10c9
         Flags: bus master, fast devsel, latency 0, IRQ 28
         Memory at fbe60000 (32-bit, non-prefetchable) [size=128K]
         Memory at fbe40000 (32-bit, non-prefetchable) [size=128K]
         I/O ports at e880 [size=32]
         Memory at fbe1c000 (32-bit, non-prefetchable) [size=16K]
         Expansion ROM at fbe20000 [disabled] [size=128K]
         Capabilities: [40] Power Management version 3
         Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+ 
Queue=0/0 Enable-
         Capabilities: [70] MSI-X: Enable+ Mask- TabSize=10
         Capabilities: [a0] Express Endpoint, MSI 00
         Capabilities: [100] Advanced Error Reporting <?>
         Capabilities: [140] Device Serial Number ce-5a-2b-ff-ff-90-25-00
         Capabilities: [150] #0e
         Capabilities: [160] #10
         Kernel driver in use: igb

0a:00.1 Ethernet controller: Intel Corporation Device 10c9 (rev 01)
         Subsystem: Super Micro Computer Inc Device 10c9
         Flags: bus master, fast devsel, latency 0, IRQ 40
         Memory at fbee0000 (32-bit, non-prefetchable) [size=128K]
         Memory at fbec0000 (32-bit, non-prefetchable) [size=128K]
         I/O ports at ec00 [size=32]
         Memory at fbe9c000 (32-bit, non-prefetchable) [size=16K]
         Expansion ROM at fbea0000 [disabled] [size=128K]
         Capabilities: [40] Power Management version 3
         Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+ 
Queue=0/0 Enable-
         Capabilities: [70] MSI-X: Enable+ Mask- TabSize=10
         Capabilities: [a0] Express Endpoint, MSI 00
         Capabilities: [100] Advanced Error Reporting <?>
         Capabilities: [140] Device Serial Number ce-5a-2b-ff-ff-90-25-00
         Capabilities: [150] #0e
         Capabilities: [160] #10
         Kernel driver in use: igb

Using the latest stable igb driver from e1000.sf.net works fine without 
any message.

Thanks,

Stefan

Re: Kernel v3.0.8 igb driver dies when pulling network cable

[Alexander Duyck]

On 11/22/2011 01:36 AM, Stefan Priebe - Profihost AG wrote:
>> It would be useful if you could try the latest driver from e1000.sf.net
>> just to verify  if this is a bug in the upstream kernel or if it is also
>> present in our e1000.sf.net.driver.  This way we can figure out if this
>> is an issue where a patch wasn't pushed into the stable kernel or if it
>> is an issue that still exists in our latest release.
>>
>> Also could you provide us with the part number you are currently using.
>> If you could provide us with the device ID for the part via lspci we can
>> start narrowing down the root cause for the issue as currently we don't
>> have any information about what hardware you are experiencing this
>> issue on.
>
> Ok another note i missed last time. Ethernet and Server stays fully
> functional - it just prints the message and resets the adapter.
>
> OK let's start with lspci:
> a:00.0 Ethernet controller: Intel Corporation Device 10c9 (rev 01)
>         Subsystem: Super Micro Computer Inc Device 10c9
>         Flags: bus master, fast devsel, latency 0, IRQ 28
>         Memory at fbe60000 (32-bit, non-prefetchable) [size=128K]
>         Memory at fbe40000 (32-bit, non-prefetchable) [size=128K]
>         I/O ports at e880 [size=32]
>         Memory at fbe1c000 (32-bit, non-prefetchable) [size=16K]
>         Expansion ROM at fbe20000 [disabled] [size=128K]
>         Capabilities: [40] Power Management version 3
>         Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+
> Queue=0/0 Enable-
>         Capabilities: [70] MSI-X: Enable+ Mask- TabSize=10
>         Capabilities: [a0] Express Endpoint, MSI 00
>         Capabilities: [100] Advanced Error Reporting <?>
>         Capabilities: [140] Device Serial Number ce-5a-2b-ff-ff-90-25-00
>         Capabilities: [150] #0e
>         Capabilities: [160] #10
>         Kernel driver in use: igb
>
> 0a:00.1 Ethernet controller: Intel Corporation Device 10c9 (rev 01)
>         Subsystem: Super Micro Computer Inc Device 10c9
>         Flags: bus master, fast devsel, latency 0, IRQ 40
>         Memory at fbee0000 (32-bit, non-prefetchable) [size=128K]
>         Memory at fbec0000 (32-bit, non-prefetchable) [size=128K]
>         I/O ports at ec00 [size=32]
>         Memory at fbe9c000 (32-bit, non-prefetchable) [size=16K]
>         Expansion ROM at fbea0000 [disabled] [size=128K]
>         Capabilities: [40] Power Management version 3
>         Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+
> Queue=0/0 Enable-
>         Capabilities: [70] MSI-X: Enable+ Mask- TabSize=10
>         Capabilities: [a0] Express Endpoint, MSI 00
>         Capabilities: [100] Advanced Error Reporting <?>
>         Capabilities: [140] Device Serial Number ce-5a-2b-ff-ff-90-25-00
>         Capabilities: [150] #0e
>         Capabilities: [160] #10
>         Kernel driver in use: igb
>
> Using the latest stable igb driver from e1000.sf.net works fine
> without any message.
>
> Thanks,
>
> Stefan
Hi Stefan,

It seems like there might be an issue with something specific to your
board since I tried reproducing the issue here on an 82576 based adapter
and the stable 3.0.9 kernel I have and I have not had much success.

I'm assuming the device that is failing is eth0.  I was wondering if you
could send me the output of the following three commands so that I can
do some further work to try and isolate the root cause for this issue:
ethtool eth0
ethtool -e eth0
grep eth0 /proc/interrupts

The issue seems to be that your adapter is not detecting that the cable
was unplugged.  This in turn is leaving stale packets on the Tx ring and
is what is resulting in the dev_watchdog message you are seeing. 
Typically this is due to one of two possible causes.  Either the device
is not detecting that the link went down, or the interrupt for the link
down event was never delivered.  Once we can isolate which of these two
events is occurring we will be much closer to having the root cause.

Thanks,

Alex

Re: Kernel v3.0.8 igb driver dies when pulling network cable

[Stefan Priebe - Profihost AG]

Hi Alex,

> It seems like there might be an issue with something specific to your
> board since I tried reproducing the issue here on an 82576 based adapter
> and the stable 3.0.9 kernel I have and I have not had much success.
but as it is working fine with the up2date igb module - it still was 
fixed ;-)

To reproduce try the following (gave me 100% success rate). Configure 
eth0 to DHCP, leave eth1 unconfigured.
Boot the system with lan cable plugged into eth1 !!
When the boot has finished switch the LAN cable from eth1 to eth0.

> I'm assuming the device that is failing is eth0.  I was wondering if you
> could send me the output of the following three commands so that I can
> do some further work to try and isolate the root cause for this issue:
> ethtool eth0
> ethtool -e eth0
> grep eth0 /proc/interrupts

Will send them as soon i have access to the machine again.

> The issue seems to be that your adapter is not detecting that the cable
> was unplugged.  This in turn is leaving stale packets on the Tx ring and
> is what is resulting in the dev_watchdog message you are seeing.
Are you sure? Cause when you look at the dmesg you'll see this:

igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX

ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

STACKTRACE!

igb 0000:0a:00.0: eth0: Reset adapter

eth0: no IPv6 routers present

igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX

igb 0000:0a:00.0: eth0: Reset adapter

igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX

So it happens when the cable is plugged in and NOT when i unplug it.

Stefan