* r8169 performance?
@ 2011-12-04 21:10 Chris Adams
2011-12-04 21:50 ` Eric Dumazet
2011-12-04 23:40 ` Francois Romieu
0 siblings, 2 replies; 9+ messages in thread
From: Chris Adams @ 2011-12-04 21:10 UTC (permalink / raw)
To: netdev
I have a system with an on-board RealTek gigabit network interface that
is giving me poor performance. I'm using a simple test, running "nc -l"
on one system and "dd if=/dev/zero | nc" on another. I see about 280
Mbps transmit and 511 Mbps receive. I checked the obvious things
(cable, switch port) but they didn't seem to be the source of the
problem (and nothing is showing any errors).
I have another system with a different motherboard (different
manufacturer) but with an onboard NIC that appears the same chip from
lspci and dmesg (and they both appear to be wired up to PCI-E). It gets
over 900 Mbps in the same type test.
I'm running Fedora (64 bit) on my systems; I upgraded the "problem"
system to F16 with kernel 3.1.2.
Are there any suggestions for what I might be able to do to improve
throughput? Is there a driver issue, or is it in how the motherboard
implemented the NIC?
Here's the lspci -vv output for the "problem" card.
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 03)
Subsystem: ASRock Incorporation Motherboard (one of many)
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 42
Region 0: I/O ports at e800 [size=256]
Region 2: Memory at fdfff000 (64-bit, prefetchable) [size=4K]
Region 4: Memory at fdff8000 (64-bit, prefetchable) [size=16K]
Expansion ROM at febe0000 [disabled] [size=128K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee0100c Data: 4171
Capabilities: [70] Express (v2) Endpoint, MSI 01
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <64us
ClockPM+ Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB
Capabilities: [ac] MSI-X: Enable- Count=4 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00000800
Capabilities: [cc] Vital Product Data
Unknown small resource type 00, will not decode more.
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [160 v1] Device Serial Number 03-00-00-00-68-4c-e0-00
Kernel driver in use: r8169
Kernel modules: r8169
--
Chris Adams <cmadams@hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: r8169 performance?
2011-12-04 21:10 r8169 performance? Chris Adams
@ 2011-12-04 21:50 ` Eric Dumazet
2011-12-04 22:02 ` Chris Adams
2011-12-04 23:40 ` Francois Romieu
1 sibling, 1 reply; 9+ messages in thread
From: Eric Dumazet @ 2011-12-04 21:50 UTC (permalink / raw)
To: Chris Adams; +Cc: netdev
Le dimanche 04 décembre 2011 à 15:10 -0600, Chris Adams a écrit :
> I have a system with an on-board RealTek gigabit network interface that
> is giving me poor performance. I'm using a simple test, running "nc -l"
> on one system and "dd if=/dev/zero | nc" on another. I see about 280
> Mbps transmit and 511 Mbps receive.
Do you mean that if this problematic NIC is the receiver, it receives
280Mbps, and if it is the sender, speed is 511 Mbps ?
Could you check counters ?
netstat -s
ethtool -S ethX
cat /proc/net/softnet_stat
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: r8169 performance?
2011-12-04 21:50 ` Eric Dumazet
@ 2011-12-04 22:02 ` Chris Adams
0 siblings, 0 replies; 9+ messages in thread
From: Chris Adams @ 2011-12-04 22:02 UTC (permalink / raw)
To: Eric Dumazet; +Cc: netdev
Once upon a time, Eric Dumazet <eric.dumazet@gmail.com> said:
> Le dimanche 04 décembre 2011 à 15:10 -0600, Chris Adams a écrit :
> > I have a system with an on-board RealTek gigabit network interface that
> > is giving me poor performance. I'm using a simple test, running "nc -l"
> > on one system and "dd if=/dev/zero | nc" on another. I see about 280
> > Mbps transmit and 511 Mbps receive.
>
> Do you mean that if this problematic NIC is the receiver, it receives
> 280Mbps, and if it is the sender, speed is 511 Mbps ?
No, the other way around (the problem NIC transmits at 280Mbps and
receives at 511Mbps).
> Could you check counters ?
Is there anything specific to look at? I'm not seeing anything that
looks like a problem to me (for example, the "144 dropped because of
missing route" all appear to be from during boot; that number isn't
changing).
> netstat -s
Ip:
818382 total packets received
0 forwarded
0 incoming packets discarded
818382 incoming packets delivered
1190521 requests sent out
144 dropped because of missing route
Icmp:
2 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
destination unreachable: 1
echo requests: 1
1 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
echo replies: 1
IcmpMsg:
InType3: 1
InType8: 1
OutType0: 1
Tcp:
11 active connections openings
15 passive connection openings
5 failed connection attempts
1 connection resets received
5 connections established
821140 segments received
1193282 segments send out
7 segments retransmited
0 bad segments received.
11 resets sent
Udp:
318 packets received
0 packets to unknown port received.
0 packet receive errors
318 packets sent
0 receive buffer errors
0 send buffer errors
UdpLite:
TcpExt:
4 TCP sockets finished time wait in fast timer
23 delayed acks sent
7 packets directly queued to recvmsg prequeue.
410916 packets header predicted
105061 acknowledgments not containing data received
304005 predicted acknowledgments
0 TCP data loss events
2 retransmits in slow start
6 other TCP timeouts
TCPSackShiftFallback: 1
TCPBacklogDrop: 75
IpExt:
InMcastPkts: 21
InBcastPkts: 43
InOctets: 1653475944
OutOctets: 1201322384
InMcastOctets: 672
InBcastOctets: 7813
> ethtool -S ethX
NIC statistics:
tx_packets: 1192416
rx_packets: 1522908
tx_errors: 0
rx_errors: 0
rx_missed: 0
align_errors: 0
tx_single_collisions: 0
tx_multi_collisions: 0
unicast: 1522391
broadcast: 474
multicast: 43
tx_aborted: 0
tx_underrun: 0
> cat /proc/net/softnet_stat
000c8dc9 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
--
Chris Adams <cmadams@hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: r8169 performance?
2011-12-04 21:10 r8169 performance? Chris Adams
2011-12-04 21:50 ` Eric Dumazet
@ 2011-12-04 23:40 ` Francois Romieu
2011-12-05 0:46 ` Chris Adams
1 sibling, 1 reply; 9+ messages in thread
From: Francois Romieu @ 2011-12-04 23:40 UTC (permalink / raw)
To: Chris Adams, netdev; +Cc: hayeswang
Chris Adams <cmadams@hiwaay.net> :
[...]
> Are there any suggestions for what I might be able to do to improve
> throughput? Is there a driver issue, or is it in how the motherboard
> implemented the NIC ?
Can you grep for the r8169 lines in the dmesg of both computers and
send the XID lines ?
It should show if the nics are the same or not.
--
Ueimor
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: r8169 performance?
2011-12-04 23:40 ` Francois Romieu
@ 2011-12-05 0:46 ` Chris Adams
2011-12-05 6:44 ` Francois Romieu
0 siblings, 1 reply; 9+ messages in thread
From: Chris Adams @ 2011-12-05 0:46 UTC (permalink / raw)
To: Francois Romieu; +Cc: netdev, hayeswang
Once upon a time, Francois Romieu <romieu@fr.zoreil.com> said:
> Chris Adams <cmadams@hiwaay.net> :
> [...]
> > Are there any suggestions for what I might be able to do to improve
> > throughput? Is there a driver issue, or is it in how the motherboard
> > implemented the NIC ?
>
> Can you grep for the r8169 lines in the dmesg of both computers and
> send the XID lines ?
>
> It should show if the nics are the same or not.
On the "problem" computer (on Fedora 16, kernel 3.1.2):
[ 7.101106] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[ 7.102665] r8169 0000:02:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
[ 7.107308] r8169 0000:02:00.0: setting latency timer to 64
[ 7.107370] r8169 0000:02:00.0: irq 42 for MSI/MSI-X
[ 7.107703] r8169 0000:02:00.0: eth0: RTL8168d/8111d at 0xffffc900112fc000, 00:19:66:f2:dc:0b, XID 081000c0 IRQ 42
On the "okay" computer (on Fedora 14, kernel 2.6.35.14):
[ 9.053574] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[ 9.053594] r8169 0000:06:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
[ 9.053689] r8169 0000:06:00.0: setting latency timer to 64
[ 9.053740] r8169 0000:06:00.0: irq 47 for MSI/MSI-X
[ 9.053830] r8169 0000:06:00.0: eth0: RTL8168d/8111d at 0xffffc90012998000, 6c:f0:49:b7:67:91, XID 083000c0 IRQ 47
Both are configured with static IPv4/v6 addresses (no NetworkManager).
The only other config difference I can think of is that the "okay"
computer is using bridging (eth0 in br0, IP config on br0) so I can
bridge virtual machines to the LAN.
--
Chris Adams <cmadams@hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: r8169 performance?
2011-12-05 0:46 ` Chris Adams
@ 2011-12-05 6:44 ` Francois Romieu
2011-12-05 14:54 ` Chris Adams
0 siblings, 1 reply; 9+ messages in thread
From: Francois Romieu @ 2011-12-05 6:44 UTC (permalink / raw)
To: Chris Adams, netdev, hayeswang
Chris Adams <cmadams@hiwaay.net> :
[...]
> On the "problem" computer (on Fedora 16, kernel 3.1.2):
>
> [ 7.101106] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
> [ 7.102665] r8169 0000:02:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
> [ 7.107308] r8169 0000:02:00.0: setting latency timer to 64
> [ 7.107370] r8169 0000:02:00.0: irq 42 for MSI/MSI-X
> [ 7.107703] r8169 0000:02:00.0: eth0: RTL8168d/8111d at 0xffffc900112fc000, 00:19:66:f2:dc:0b, XID 081000c0 IRQ 42
>
> On the "okay" computer (on Fedora 14, kernel 2.6.35.14):
>
> [ 9.053574] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
> [ 9.053594] r8169 0000:06:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
> [ 9.053689] r8169 0000:06:00.0: setting latency timer to 64
> [ 9.053740] r8169 0000:06:00.0: irq 47 for MSI/MSI-X
> [ 9.053830] r8169 0000:06:00.0: eth0: RTL8168d/8111d at 0xffffc90012998000, 6c:f0:49:b7:67:91, XID 083000c0 IRQ 47
Almost the same: RTL_GIGA_MAC_VER_25 versus RTL_GIGA_MAC_VER_26 (faster).
The only difference in the driver is the hw_phy_config part.
--
Ueimor
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: r8169 performance?
2011-12-05 6:44 ` Francois Romieu
@ 2011-12-05 14:54 ` Chris Adams
2011-12-06 8:22 ` Francois Romieu
0 siblings, 1 reply; 9+ messages in thread
From: Chris Adams @ 2011-12-05 14:54 UTC (permalink / raw)
To: Francois Romieu; +Cc: netdev, hayeswang
Once upon a time, Francois Romieu <romieu@fr.zoreil.com> said:
> Chris Adams <cmadams@hiwaay.net> :
> [...]
> > On the "problem" computer (on Fedora 16, kernel 3.1.2):
> >
> > [ 7.101106] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
> > [ 7.102665] r8169 0000:02:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
> > [ 7.107308] r8169 0000:02:00.0: setting latency timer to 64
> > [ 7.107370] r8169 0000:02:00.0: irq 42 for MSI/MSI-X
> > [ 7.107703] r8169 0000:02:00.0: eth0: RTL8168d/8111d at 0xffffc900112fc000, 00:19:66:f2:dc:0b, XID 081000c0 IRQ 42
> >
> > On the "okay" computer (on Fedora 14, kernel 2.6.35.14):
> >
> > [ 9.053574] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
> > [ 9.053594] r8169 0000:06:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
> > [ 9.053689] r8169 0000:06:00.0: setting latency timer to 64
> > [ 9.053740] r8169 0000:06:00.0: irq 47 for MSI/MSI-X
> > [ 9.053830] r8169 0000:06:00.0: eth0: RTL8168d/8111d at 0xffffc90012998000, 6c:f0:49:b7:67:91, XID 083000c0 IRQ 47
>
> Almost the same: RTL_GIGA_MAC_VER_25 versus RTL_GIGA_MAC_VER_26 (faster).
Ahh, my eyes didn't see that one bit difference.
> The only difference in the driver is the hw_phy_config part.
There's also a difference in the interrupt handling (looks like only in
the receive FIFO overflow case). Also, they load different firmware (so
I guess my problem could be firmware related, which I guess only RealTek
could find/fix).
I guess my question is this: is there a chance this can be solved with
driver changes, or should I just accept that the on-board NIC has poor
performance and install an add-in NIC?
--
Chris Adams <cmadams@hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: r8169 performance?
2011-12-05 14:54 ` Chris Adams
@ 2011-12-06 8:22 ` Francois Romieu
2011-12-07 4:19 ` Chris Adams
0 siblings, 1 reply; 9+ messages in thread
From: Francois Romieu @ 2011-12-06 8:22 UTC (permalink / raw)
To: Chris Adams, netdev, hayeswang
Chris Adams <cmadams@hiwaay.net> :
> Once upon a time, Francois Romieu <romieu@fr.zoreil.com> said:
[...]
> > Almost the same: RTL_GIGA_MAC_VER_25 versus RTL_GIGA_MAC_VER_26 (faster).
>
> Ahh, my eyes didn't see that one bit difference.
>
> > The only difference in the driver is the hw_phy_config part.
>
> There's also a difference in the interrupt handling (looks like only in
> the receive FIFO overflow case).
Ahh, I looked at the patched driver.
It will disappear once http://marc.info/?l=linux-netdev&m=132306718323579
is applied. You may apply http://marc.info/?l=linux-netdev&m=132306715823570
too but I doubt it will make much difference for you.
> Also, they load different firmware (so I guess my problem could be firmware
> related, which I guess only RealTek could find/fix).
>
> I guess my question is this: is there a chance this can be solved with
> driver changes, or should I just accept that the on-board NIC has poor
> performance and install an add-in NIC?
(I understand that the systems (CPU, bus...) are not necessary the same.)
Did you check if Tx checksumming is enabled ?
It could make a noticeable difference.
--
Ueimor
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: r8169 performance?
2011-12-06 8:22 ` Francois Romieu
@ 2011-12-07 4:19 ` Chris Adams
0 siblings, 0 replies; 9+ messages in thread
From: Chris Adams @ 2011-12-07 4:19 UTC (permalink / raw)
To: Francois Romieu; +Cc: netdev, hayeswang
Once upon a time, Francois Romieu <romieu@fr.zoreil.com> said:
> It will disappear once http://marc.info/?l=linux-netdev&m=132306718323579
> is applied. You may apply http://marc.info/?l=linux-netdev&m=132306715823570
> too but I doubt it will make much difference for you.
I'll give it a try to double-check (it'll probably be a few days before
I can get to it).
> > Also, they load different firmware (so I guess my problem could be firmware
> > related, which I guess only RealTek could find/fix).
> >
> > I guess my question is this: is there a chance this can be solved with
> > driver changes, or should I just accept that the on-board NIC has poor
> > performance and install an add-in NIC?
>
> (I understand that the systems (CPU, bus...) are not necessary the same.)
>
> Did you check if Tx checksumming is enabled ?
>
> It could make a noticeable difference.
That is off on both of my r8169 systems. Enabling it on the "good"
system killed networking (even my existing SSH session was hung) until I
re-disabled it. Enabling it on the "problem" system didn't appear to
make any significant difference.
The otherwise "good" system is running an older kernel (Fedora 14,
2.6.35.14); maybe some changes have been made so that "tx on" works on a
newer driver? Or maybe the slightly different chip revision is the
difference.
--
Chris Adams <cmadams@hiwaay.net>
Systems and Network Administrator - HiWAAY Internet Services
I don't speak for anybody but myself - that's enough trouble.
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2011-12-07 4:19 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-12-04 21:10 r8169 performance? Chris Adams
2011-12-04 21:50 ` Eric Dumazet
2011-12-04 22:02 ` Chris Adams
2011-12-04 23:40 ` Francois Romieu
2011-12-05 0:46 ` Chris Adams
2011-12-05 6:44 ` Francois Romieu
2011-12-05 14:54 ` Chris Adams
2011-12-06 8:22 ` Francois Romieu
2011-12-07 4:19 ` Chris Adams
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).