Re: forcedeth doesn't pass traffic (3.0.x kernel and earliers)

["Arkadiusz Miśkiewicz" <a.miskiewicz@gmail.com>]

On Saturday 10 of December 2011, Arkadiusz Miśkiewicz wrote:
> On Saturday 10 of December 2011, Arkadiusz Miśkiewicz wrote:
> > Hello,
> > 
> > I have few supermicro machines connected to edge-core switch using 1Gbps
> > uplink. The problem is that after boot I have to restart net to get
> > network card pass traffic. The problem is only with forcedeth network
> > cards. It happens on different supermicro servers and it happens for as
> > far as I can remember (so, 2.6.3x kernels, too). Currently I'm on 3.0.13.
> > 
> > Logs below show state after fresh boot where network traffic doesn't get
> > through. Note that it shows that link is up but still - traffic doesn't
> > get passed. tcpdump on eth0 interface sees only traffic initiated from
> > this server and no traffic from outside.
> > 
> > Tried to set every offload (via ethtool -K) to off but that didn't help.
> > 
> > I have to make:
> > ip link set eth0 down; ip link set eth0 up
> > wait few second and then traffic starts to get passed. When I do that
> > this shows in dmesg: [ 1058.266480] forcedeth 0000:00:08.0: irq 40 for
> > MSI/MSI-X
> > [ 1069.163238] eth0: no IPv6 routers present
> 
> One more note. This doesn't always happens like that. Sometimes I boot
> and I get link working, traffic gets passed fine:

rmmod forcedeth; modprobe forcedeth; ifup doesn't help

Counters on switch side after forcedeth machine reboot, no traffic being 
passed, switch counters reset.

#show interfaces counters ethernet 1/19
Ethernet 1/19
 Iftable Stats:
  Octets Input: 0, Octets Output: 11116437
  Unicast Input: 0, Unicast Output: 59
  Discard Input: 0, Discard Output: 0
  Error Input: 0, Error Output: 0
  Unknown Protos Input: 0, QLen Output: 0
 Extended Iftable Stats:
  Multi-cast Input: 0, Multi-cast Output: 24742
  Broadcast Input: 0, Broadcast Output: 790
 Ether-like Stats:
  Alignment Errors: 0, FCS Errors: 0
  Single Collision Frames: 0, Multiple Collision Frames: 0
  SQE Test Errors: 0, Deferred Transmissions: 0
  Late Collisions: 0, Excessive Collisions: 0
  Internal Mac Transmit Errors: 0, Internal Mac Receive Errors: 0
  Frames Too Long: 0, Carrier Sense Errors: 0
  Symbol Errors: 0
 RMON Stats:
  Drop Events: 0, Octets: 11075170, Packets: 25549
  Broadcast PKTS: 790, Multi-cast PKTS: 24742
  Undersize PKTS: 0, Oversize PKTS: 0
  Fragments: 0, Jabbers: 0
  CRC Align Errors: 0, Collisions: 0
  Packet Size <= 64 Octets: 2060, Packet Size 65 to 127 Octets: 12764
  Packet Size 128 to 255 Octets: 364, Packet Size 256 to 511 Octets: 292
  Packet Size 512 to 1023 Octets: 8755, Packet Size 1024 to 1518 Octets: 1314

No errors here.

# dmesg|grep eth0
[   33.432266] forcedeth 0000:00:08.0: ifname eth0, PHY OUI 0x5043 @ 2, addr 
00:30:48:7a:57:a6
[   33.960287] Loading kernel module for a network device with CAP_SYS_MODULE 
(deprecated).  Use CAP_NET_ADMIN and alias netdev-eth0 instead
[   33.967925] forcedeth 0000:00:08.0: eth0: no link during initialization
[   33.970995] ADDRCONF(NETDEV_UP): eth0: link is not ready
[   36.380311] forcedeth 0000:00:08.0: eth0: link up
[   36.384016] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   47.150014] eth0: no IPv6 routers present

forcedeth side:

# ethtool -S eth0
NIC statistics:
     tx_bytes: 69926
     tx_zero_rexmt: 1011
     tx_one_rexmt: 0
     tx_many_rexmt: 0
     tx_late_collision: 0
     tx_fifo_errors: 0
     tx_carrier_errors: 0
     tx_excess_deferral: 0
     tx_retry_error: 0
     rx_frame_error: 0
     rx_extra_byte: 0
     rx_late_collision: 0
     rx_runt: 0
     rx_frame_too_long: 0
     rx_over_errors: 0
     rx_crc_errors: 0
     rx_frame_align_error: 0
     rx_length_error: 0
     rx_unicast: 3047
     rx_multicast: 0
     rx_broadcast: 2113
     rx_packets: 5160
     rx_errors_total: 0
     tx_errors_total: 0
     tx_deferral: 0
     tx_packets: 1011
     rx_bytes: 382259
     tx_pause: 0
     rx_pause: 0
     rx_drop_frame: 0


> 
> [   34.038476] forcedeth: Reverse Engineered nForce ethernet driver.
> Version 0.64. [   34.038698] ACPI: PCI Interrupt Link [LMAC] enabled at
> IRQ 22
> [   34.038705] forcedeth 0000:00:08.0: PCI INT A -> Link[LMAC] -> GSI 22
> (level, low) -> IRQ 22 [   34.038710] forcedeth 0000:00:08.0: setting
> latency timer to 64 [   34.565309] forcedeth 0000:00:08.0: ifname eth0,
> PHY OUI 0x5043 @ 2, addr 00:30:48:7a:b4:88 [   34.565314] forcedeth
> 0000:00:08.0: highdma csum vlan pwrctl mgmt gbit lnktim msi desc-v3 [  
> 34.566823] ACPI: PCI Interrupt Link [LMAD] enabled at IRQ 21
> [   34.566831] forcedeth 0000:00:09.0: PCI INT A -> Link[LMAD] -> GSI 21
> (level, low) -> IRQ 21 [   34.566836] forcedeth 0000:00:09.0: setting
> latency timer to 64 [   35.091341] forcedeth 0000:00:09.0: ifname eth1,
> PHY OUI 0x5043 @ 3, addr 00:30:48:7a:b4:89 [   35.091346] forcedeth
> 0000:00:09.0: highdma csum vlan pwrctl mgmt gbit lnktim msi desc-v3 [  
> 35.093492] Loading kernel module for a network device with CAP_SYS_MODULE
> (deprecated).  Use CAP_NET_ADMIN and alias netdev-eth0 instead [  
> 35.101074] forcedeth 0000:00:08.0: irq 40 for MSI/MSI-X
> [   35.101267] forcedeth 0000:00:08.0: eth0: no link during initialization
> [   35.102322] ADDRCONF(NETDEV_UP): eth0: link is not ready
> [   35.663785] ip_tables: (C) 2000-2006 Netfilter Core Team
> [   35.756083] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
> [   37.147869] forcedeth 0000:00:08.0: eth0: link up
> [   37.149108] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> [   47.176677] eth0: no IPv6 routers present
> 
> > (CC to few people shown by get_maintainer.pl)
> > 
> > LOGS/OUTPUTs from non working state after fresh boot.
> > # dmesg|grep eth
> > [   34.528326] forcedeth: Reverse Engineered nForce ethernet driver.
> > Version 0.64. [   34.528542] forcedeth 0000:00:08.0: PCI INT A ->
> > Link[LMAC] -> GSI 22 (level, low) -> IRQ 22 [   34.528547] forcedeth
> > 0000:00:08.0: setting latency timer to 64 [   35.055319] forcedeth
> > 0000:00:08.0: ifname eth0, PHY OUI 0x5043 @ 2, addr 00:30:48:7a:b4:88 [
> > 35.055324] forcedeth 0000:00:08.0: highdma csum vlan pwrctl mgmt gbit
> > lnktim msi desc-v3 [   35.056480] forcedeth 0000:00:09.0: PCI INT A ->
> > Link[LMAD] -> GSI 21 (level, low) -> IRQ 21 [   35.056485] forcedeth
> > 0000:00:09.0: setting latency timer to 64 [   35.581960] forcedeth
> > 0000:00:09.0: ifname eth1, PHY OUI 0x5043 @ 3, addr 00:30:48:7a:b4:89 [
> > 35.581965] forcedeth 0000:00:09.0: highdma csum vlan pwrctl mgmt gbit
> > lnktim msi desc-v3 [   35.584211] Loading kernel module for a network
> > device with CAP_SYS_MODULE (deprecated).  Use CAP_NET_ADMIN and alias
> > netdev-eth0 instead [   35.591778] forcedeth 0000:00:08.0: irq 40 for
> > MSI/MSI-X
> > [   35.591970] forcedeth 0000:00:08.0: eth0: no link during
> > initialization [   35.593030] ADDRCONF(NETDEV_UP): eth0: link is not
> > ready
> > [   37.666757] forcedeth 0000:00:08.0: eth0: link up
> > [   37.668624] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
> > [   48.150006] eth0: no IPv6 routers present
> > 
> > lspci -vv:
> > 00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a3)
> > 
> >         Subsystem: Super Micro Computer Inc Device 1611
> >         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> >         ParErr-
> > 
> > Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B+
> > ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> > Latency: 0 (250ns min, 5000ns max)
> > 
> >         Interrupt: pin A routed to IRQ 21
> >         Region 0: Memory at feaf5000 (32-bit, non-prefetchable) [size=4K]
> >         Region 1: I/O ports at b080 [size=8]
> >         Region 2: Memory at feafa000 (32-bit, non-prefetchable)
> >         [size=256] Region 3: Memory at feaf4c00 (32-bit,
> >         non-prefetchable) [size=16] Capabilities: [44] Power Management
> >         version 2
> >         
> >                 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
> > 
> > PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0
> > DScale=0 PME- Capabilities: [70] MSI-X: Enable- Count=8 Masked-
> > 
> >                 Vector table: BAR=2 offset=00000000
> >                 PBA: BAR=3 offset=00000000
> >         
> >         Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit+
> >         
> >                 Address: 0000000000000000  Data: 0000
> >                 Masking: 00000000  Pending: 00000000
> >         
> >         Capabilities: [6c] HyperTransport: MSI Mapping Enable- Fixed+
> >         Kernel driver in use: forcedeth
> > 
> > # ethtool eth0
> > 
> > Settings for eth0:
> >         Supported ports: [ MII ]
> >         Supported link modes:   10baseT/Half 10baseT/Full
> >         
> >                                 100baseT/Half 100baseT/Full
> >                                 1000baseT/Full
> >         
> >         Supported pause frame use: No
> >         Supports auto-negotiation: Yes
> >         Advertised link modes:  10baseT/Half 10baseT/Full
> >         
> >                                 100baseT/Half 100baseT/Full
> >                                 1000baseT/Full
> >         
> >         Advertised pause frame use: No
> >         Advertised auto-negotiation: Yes
> >         Speed: 1000Mb/s
> >         Duplex: Full
> >         Port: MII
> >         PHYAD: 2
> >         Transceiver: external
> >         Auto-negotiation: on
> >         Supports Wake-on: g
> >         Wake-on: d
> >         Link detected: yes
> > 
> > # ethtool -S eth0
> > 
> > NIC statistics:
> >      tx_bytes: 46352
> >      tx_zero_rexmt: 704
> >      tx_one_rexmt: 0
> >      tx_many_rexmt: 0
> >      tx_late_collision: 0
> >      tx_fifo_errors: 0
> >      tx_carrier_errors: 0
> >      tx_excess_deferral: 0
> >      tx_retry_error: 0
> >      rx_frame_error: 0
> >      rx_extra_byte: 0
> >      rx_late_collision: 0
> >      rx_runt: 0
> >      rx_frame_too_long: 0
> >      rx_over_errors: 1
> >      rx_crc_errors: 0
> >      rx_frame_align_error: 0
> >      rx_length_error: 0
> >      rx_unicast: 31
> >      rx_multicast: 0
> >      rx_broadcast: 436
> >      rx_packets: 467
> >      rx_errors_total: 1
> >      tx_errors_total: 0
> >      tx_deferral: 0
> >      tx_packets: 704
> >      rx_bytes: 35872
> >      tx_pause: 0
> >      rx_pause: 0
> >      rx_drop_frame: 170
> > 
> > # ethtool -k eth0
> > Offload parameters for eth0:
> > rx-checksumming: on
> > tx-checksumming: on
> > scatter-gather: on
> > tcp-segmentation-offload: on
> > udp-fragmentation-offload: off
> > generic-segmentation-offload: on
> > generic-receive-offload: on
> > large-receive-offload: off
> > rx-vlan-offload: on
> > tx-vlan-offload: on
> > ntuple-filters: off
> > receive-hashing: off
> > 
> > 
> > 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc sfq state UP
> > qlen 1000 link/ether 00:30:48:7a:b4:88 brd ff:ff:ff:ff:ff:ff


-- 
Arkadiusz Miśkiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/