* Re: [Bugme-new] [Bug 11752] New: Extremely low netperf UDP_RR throughput for nvidia MCP65 [not found] <bug-11752-10286@http.bugzilla.kernel.org/> @ 2008-10-16 20:49 ` Andrew Morton 2008-10-16 21:07 ` Rick Jones 0 siblings, 1 reply; 9+ messages in thread From: Andrew Morton @ 2008-10-16 20:49 UTC (permalink / raw) To: netdev; +Cc: bugme-daemon, arno, Ayaz Abdulla (switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface). On Mon, 13 Oct 2008 13:41:19 -0700 (PDT) bugme-daemon@bugzilla.kernel.org wrote: > http://bugzilla.kernel.org/show_bug.cgi?id=11752 > > Summary: Extremely low netperf UDP_RR throughput for nvidia MCP65 > Product: Drivers > Version: 2.5 > KernelVersion: from F10-Beta-x86_64-Live-KDE.iso > Platform: All > OS/Version: Linux > Tree: Mainline > Status: NEW > Severity: normal > Priority: P1 > Component: Network > AssignedTo: jgarzik@pobox.com > ReportedBy: arno@heho.snv.jussieu.fr > > > Latest working kernel version: - > Earliest failing kernel version: > Distribution: F10-Beta-x86_64 > Hardware Environment: HP Pavillon dv6820ef > Software Environment: > Problem Description: when running at 1Gbps netperf shows good performance for > TCP_STREAM and UDP_STREAM tests, but extremely bad performance for UDP_RR test > (less then 1 ping-pong a second whereas at 100Mbps performance easily reaches > 10-20K a second) > > A friend figured out that it looks like small packets at 1Gbps are dropped > as being falsely considered crc-errored. > > > > Steps to reproduce: Which driver is this using? forcedeth? ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bugme-new] [Bug 11752] New: Extremely low netperf UDP_RR throughput for nvidia MCP65 2008-10-16 20:49 ` [Bugme-new] [Bug 11752] New: Extremely low netperf UDP_RR throughput for nvidia MCP65 Andrew Morton @ 2008-10-16 21:07 ` Rick Jones 2008-10-17 14:28 ` Arno J. Klaassen 2008-10-17 20:49 ` Arno J. Klaassen 0 siblings, 2 replies; 9+ messages in thread From: Rick Jones @ 2008-10-16 21:07 UTC (permalink / raw) To: Andrew Morton; +Cc: netdev, bugme-daemon, arno, Ayaz Abdulla Andrew Morton wrote: > (switched to email. Please respond via emailed reply-to-all, not via the > bugzilla web interface). > > On Mon, 13 Oct 2008 13:41:19 -0700 (PDT) > bugme-daemon@bugzilla.kernel.org wrote: > > >>http://bugzilla.kernel.org/show_bug.cgi?id=11752 >> >> Summary: Extremely low netperf UDP_RR throughput for nvidia MCP65 >> Product: Drivers >> Version: 2.5 >> KernelVersion: from F10-Beta-x86_64-Live-KDE.iso >> Platform: All >> OS/Version: Linux >> Tree: Mainline >> Status: NEW >> Severity: normal >> Priority: P1 >> Component: Network >> AssignedTo: jgarzik@pobox.com >> ReportedBy: arno@heho.snv.jussieu.fr >> >> >>Latest working kernel version: - >>Earliest failing kernel version: >>Distribution: F10-Beta-x86_64 >>Hardware Environment: HP Pavillon dv6820ef >>Software Environment: >>Problem Description: when running at 1Gbps netperf shows good performance for >>TCP_STREAM and UDP_STREAM tests, but extremely bad performance for UDP_RR test >>(less then 1 ping-pong a second whereas at 100Mbps performance easily reaches >>10-20K a second) >> >>A friend figured out that it looks like small packets at 1Gbps are dropped >>as being falsely considered crc-errored. Given how netperf UDP_RR has _no_ recovery from lost datagrams, it makes sense that performance on that test would be very low - the first lost datagram the transactions come to a screeching halt until the end-of-test timer expires. Are netstat stats showing retransmissions during a TCP_STREAM test? How about a TCP_RR test? TCP_RR might be low too, it just wouldn't necessarily be as low since TCP will get things started again after a loss. UDP_STREAM would just go blasting along without a care in the world... rick jones ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bugme-new] [Bug 11752] New: Extremely low netperf UDP_RR throughput for nvidia MCP65 2008-10-16 21:07 ` Rick Jones @ 2008-10-17 14:28 ` Arno J. Klaassen 2008-10-17 20:49 ` Arno J. Klaassen 1 sibling, 0 replies; 9+ messages in thread From: Arno J. Klaassen @ 2008-10-17 14:28 UTC (permalink / raw) To: Rick Jones, Andrew Morton; +Cc: netdev, bugme-daemon, arno, Ayaz Abdulla Hello all, thanx for your help. Rick Jones <rick.jones2@hp.com> writes: > Andrew Morton wrote: > > (switched to email. Please respond via emailed reply-to-all, not via the > > bugzilla web interface). > > On Mon, 13 Oct 2008 13:41:19 -0700 (PDT) > > bugme-daemon@bugzilla.kernel.org wrote: > > > >>http://bugzilla.kernel.org/show_bug.cgi?id=11752 > >> > >> Summary: Extremely low netperf UDP_RR throughput for nvidia MCP65 > >> Product: Drivers > >> Version: 2.5 > >> KernelVersion: from F10-Beta-x86_64-Live-KDE.iso > >> Platform: All > >> OS/Version: Linux > >> Tree: Mainline > >> Status: NEW > >> Severity: normal > >> Priority: P1 > >> Component: Network > >> AssignedTo: jgarzik@pobox.com > >> ReportedBy: arno@heho.snv.jussieu.fr > >> > >> > >>Latest working kernel version: - > >>Earliest failing kernel version: > >>Distribution: F10-Beta-x86_64 > >>Hardware Environment: HP Pavillon dv6820ef > >>Software Environment: > >>Problem Description: when running at 1Gbps netperf shows good performance for > >>TCP_STREAM and UDP_STREAM tests, but extremely bad performance for UDP_RR test > >>(less then 1 ping-pong a second whereas at 100Mbps performance easily reaches > >>10-20K a second) > >> > >>A friend figured out that it looks like small packets at 1Gbps are dropped > >>as being falsely considered crc-errored. > > Given how netperf UDP_RR has _no_ recovery from lost datagrams, it > makes sense that performance on that test would be very low - the > first lost datagram the transactions come to a screeching halt until > the end-of-test timer expires. > > Are netstat stats showing retransmissions during a TCP_STREAM test? I will check that later tonoght and/or this WE > How about a TCP_RR test? TCP_RR might be low too, it just wouldn't > necessarily be as low since TCP will get things started again after a > loss. UDP_STREAM would just go blasting along without a care in the > world... A summary of all tests is (REFERENCE being a freebsd6 box on same LAN against same server running netserver; it's pretty clear that the *STREAM tests perform OK and the *RR test poor to very poor) : TCP_STREAM Throughput 10^6bits/sec REFERENCE 349.57 fc10-x64 138.48 UDP_STREAM Throughput 10^6bits/sec REFERENCE 388.45 fc10-x64 365.10 TCP_RR Trans. Rate per sec REFERENCE 9801.58 fc10-x64 86.87 TCP_CRR Trans. Rate per sec REFERENCE 4520.98 fc10-x64 5.60 UDP_RR Trans. Rate per sec REFERENCE 9473.20 fc10-x64 0.80 Arno ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bugme-new] [Bug 11752] New: Extremely low netperf UDP_RR throughput for nvidia MCP65 2008-10-16 21:07 ` Rick Jones 2008-10-17 14:28 ` Arno J. Klaassen @ 2008-10-17 20:49 ` Arno J. Klaassen 2008-10-17 21:05 ` Rick Jones 1 sibling, 1 reply; 9+ messages in thread From: Arno J. Klaassen @ 2008-10-17 20:49 UTC (permalink / raw) To: Rick Jones, Andrew Morton; +Cc: netdev, bugme-daemon, arno, Ayaz Abdulla Hello, > Are netstat stats showing retransmissions during a TCP_STREAM test? some more info : [root@localhost mcp65]# uname -a Linux localhost.localdomain 2.6.27-0.352.rc7.git1.fc10.x86_64 #1 SMP Tue Sep 23 21:13:29 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux [root@localhost ~]# ifconfig eth0 Link encap:Ethernet HWaddr 00:1E:68:XXX inet addr:172.16.1.31 Bcast:172.16.1.255 Mask:255.255.255.0 inet6 addr: XXX/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1731 errors:149 dropped:0 overruns:0 frame:149 TX packets:1628 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2549871 (2.4 MiB) TX bytes:125378 (122.4 KiB) Interrupt:20 Base address:0x6000 After some fiddling (essentially installing a netperf-rpm) : [root@localhost mcp65]# netstat -Ieth0 Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 6347 508 0 0 5838 0 0 0 BMRU [root@localhost mcp65]# ethtool eth0 Settings for eth0: Supported ports: [ MII ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised auto-negotiation: Yes Speed: 1000Mb/s Duplex: Full Port: MII PHYAD: 1 Transceiver: external Auto-negotiation: on Supports Wake-on: g Wake-on: d Link detected: yes [root@localhost mcp65]# netperf -v -t TCP_STREAM -H 172.16.1.7 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.1.7 (172.16.1.7) port 0 AF_INET 132.64 [root@localhost mcp65]# netstat -Ieth0 Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVRFlg eth0 1500 0 84682 2933 0 0 127298 0 0 0BMRU I hope this is what you asked for ... Arno ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bugme-new] [Bug 11752] New: Extremely low netperf UDP_RR throughput for nvidia MCP65 2008-10-17 20:49 ` Arno J. Klaassen @ 2008-10-17 21:05 ` Rick Jones 2008-10-18 11:36 ` Arno J. Klaassen 0 siblings, 1 reply; 9+ messages in thread From: Rick Jones @ 2008-10-17 21:05 UTC (permalink / raw) To: Arno J. Klaassen; +Cc: Andrew Morton, netdev, bugme-daemon, Ayaz Abdulla Arno J. Klaassen wrote: > Hello, > > >>Are netstat stats showing retransmissions during a TCP_STREAM test? > > > some more info : > > [root@localhost mcp65]# uname -a > Linux localhost.localdomain 2.6.27-0.352.rc7.git1.fc10.x86_64 #1 SMP Tue Sep 23 21:13:29 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux > > > [root@localhost ~]# ifconfig > eth0 Link encap:Ethernet HWaddr 00:1E:68:XXX > inet addr:172.16.1.31 Bcast:172.16.1.255 Mask:255.255.255.0 > inet6 addr: XXX/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:1731 errors:149 dropped:0 overruns:0 frame:149 > TX packets:1628 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:2549871 (2.4 MiB) TX bytes:125378 (122.4 KiB) > Interrupt:20 Base address:0x6000 > > After some fiddling (essentially installing a netperf-rpm) : > > [root@localhost mcp65]# netstat -Ieth0 > Kernel Interface table > Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg > eth0 1500 0 6347 508 0 0 5838 0 0 0 BMRU > > > [root@localhost mcp65]# ethtool eth0 > Settings for eth0: > Supported ports: [ MII ] > Supported link modes: 10baseT/Half 10baseT/Full > 100baseT/Half 100baseT/Full > 1000baseT/Full > Supports auto-negotiation: Yes > Advertised link modes: 10baseT/Half 10baseT/Full > 100baseT/Half 100baseT/Full > 1000baseT/Full > Advertised auto-negotiation: Yes > Speed: 1000Mb/s > Duplex: Full > Port: MII > PHYAD: 1 > Transceiver: external > Auto-negotiation: on > Supports Wake-on: g > Wake-on: d > Link detected: yes > > [root@localhost mcp65]# netperf -v -t TCP_STREAM -H 172.16.1.7 > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.1.7 (172.16.1.7) port 0 AF_INET > 132.64 Hmm, I'm surprised that the lack of a value following the -v was successful - I'll have to go back and look at the code :) Still, I guess it gave you the desired "-v 0" behaviour. > [root@localhost mcp65]# netstat -Ieth0 > Kernel Interface table > Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVRFlg > eth0 1500 0 84682 2933 0 0 127298 0 0 0BMRU > > > I hope this is what you asked for ... Close enough. I suspect that if you were to snap netstat -s -t before and after the netperf you'd have seen retransmissions correlated to those "RX-ERR" stats. My history is such that I don't think of netstat for link-level stats and only think of it in the context of IP-layer and higher (eg tcp). Clearly something is fubar with the rx side (well duh :). The next set of stats I'd try to look at would be ethtool stats for the interface, eg ethtool -S eth0 and see if it shows someting more specific for the "RX-ERR" shown by netstat -I eth0. rick jones ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bugme-new] [Bug 11752] New: Extremely low netperf UDP_RR throughput for nvidia MCP65 2008-10-17 21:05 ` Rick Jones @ 2008-10-18 11:36 ` Arno J. Klaassen 2008-10-20 17:41 ` Rick Jones 0 siblings, 1 reply; 9+ messages in thread From: Arno J. Klaassen @ 2008-10-18 11:36 UTC (permalink / raw) To: Rick Jones; +Cc: Andrew Morton, netdev, bugme-daemon, Ayaz Abdulla Rick Jones <rick.jones2@hp.com> writes: > Arno J. Klaassen wrote: > > Hello, > > > >>Are netstat stats showing retransmissions during a TCP_STREAM test? > > some more info : > > [root@localhost mcp65]# uname -a > > Linux localhost.localdomain 2.6.27-0.352.rc7.git1.fc10.x86_64 #1 > > SMP Tue Sep 23 21:13:29 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux > > [root@localhost ~]# ifconfig > > eth0 Link encap:Ethernet HWaddr 00:1E:68:XXX > > inet addr:172.16.1.31 Bcast:172.16.1.255 Mask:255.255.255.0 > > inet6 addr: XXX/64 Scope:Link UP > > BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > RX packets:1731 errors:149 dropped:0 overruns:0 frame:149 > > TX packets:1628 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:1000 > > RX bytes:2549871 (2.4 MiB) TX bytes:125378 (122.4 KiB) > > Interrupt:20 Base address:0x6000 > > After some fiddling (essentially installing a netperf-rpm) : > > [root@localhost mcp65]# netstat -Ieth0 > > Kernel Interface table > > Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR > > TX-DRP TX-OVR Flg > > eth0 1500 0 6347 508 0 0 5838 0 > > 0 0 BMRU > > [root@localhost mcp65]# ethtool eth0 > > Settings for eth0: > > Supported ports: [ MII ] > > Supported link modes: 10baseT/Half 10baseT/Full > > 100baseT/Half 100baseT/Full > > 1000baseT/Full Supports > > auto-negotiation: Yes > > Advertised link modes: 10baseT/Half 10baseT/Full > > 100baseT/Half 100baseT/Full > > 1000baseT/Full Advertised > > auto-negotiation: Yes > > Speed: 1000Mb/s > > Duplex: Full > > Port: MII > > PHYAD: 1 > > Transceiver: external > > Auto-negotiation: on > > Supports Wake-on: g > > Wake-on: d > > Link detected: yes > > [root@localhost mcp65]# netperf -v -t TCP_STREAM -H 172.16.1.7 > > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.1.7 (172.16.1.7) port 0 AF_INET > > 132.64 > > Hmm, I'm surprised that the lack of a value following the -v was > successful - I'll have to go back and look at the code :) Still, I > guess it gave you the desired "-v 0" behaviour. > > > [root@localhost mcp65]# netstat -Ieth0 > > Kernel Interface table > > Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVRFlg > > eth0 1500 0 84682 2933 0 0 127298 0 0 0BMRU > > I hope this is what you asked for ... > > Close enough. I suspect that if you were to snap netstat -s -t before > and after the netperf you'd have seen retransmissions correlated to > those "RX-ERR" stats. My history is such that I don't think of > netstat for link-level stats and only think of it in the context of > IP-layer and higher (eg tcp). > > Clearly something is fubar with the rx side (well duh :). The next > set of stats I'd try to look at would be ethtool stats for the > interface, eg ethtool -S eth0 and see if it shows someting more > specific for the "RX-ERR" shown by netstat -I eth0. OK, here it is (rx_errors_total: 10049, rx_crc_errors: 10049) : (NB, let me know how to eventually test eventual patches/binary modules on a life-CD; I've just limited linux kernel skills) [root@localhost ~]# netstat -s -t Tcp: 16 active connections openings 0 passive connection openings 10 failed connection attempts 0 connection resets received 2 connections established 683 segments received 693 segments send out 4 segments retransmited 0 bad segments received. 14 resets sent UdpLite: TcpExt: 7 delayed acks sent Quick ack mode was activated 3 times 4 packets directly queued to recvmsg prequeue. 2 packets directly received from prequeue 230 packets header predicted 9 acknowledgments not containing data received 47 predicted acknowledgments 2 congestion windows recovered after partial ack 0 TCP data loss events 4 other TCP timeouts 3 DSACKs sent for old packets 4 DSACKs sent for out of order packets 1 connections reset due to unexpected data IpExt: InMcastPkts: 14 OutMcastPkts: 16 InBcastPkts: 22 [root@localhost ~]# ethtool -S eth0 NIC statistics: tx_bytes: 86812 tx_zero_rexmt: 1057 tx_one_rexmt: 0 tx_many_rexmt: 0 tx_late_collision: 0 tx_fifo_errors: 0 tx_carrier_errors: 0 tx_excess_deferral: 0 tx_retry_error: 0 rx_frame_error: 0 rx_extra_byte: 0 rx_late_collision: 0 rx_runt: 0 rx_frame_too_long: 0 rx_over_errors: 0 rx_crc_errors: 323 rx_frame_align_error: 0 rx_length_error: 0 rx_unicast: 1021 rx_multicast: 0 rx_broadcast: 29 rx_packets: 1050 rx_errors_total: 323 tx_errors_total: 0 tx_deferral: 0 tx_packets: 1057 rx_bytes: 1421807 tx_pause: 0 rx_pause: 0 rx_drop_frame: 0 [root@localhost ~]# netperf -v 1 -l 60 -H 172.16.1.7 -t TCP_STREAM TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.1.7 (172.16.1.7) port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 3217968 16384 16384 60.00 149.06 [root@localhost ~]# netstat -s -t Tcp: 18 active connections openings 0 passive connection openings 10 failed connection attempts 0 connection resets received 2 connections established 523674 segments received 820982 segments send out 4 segments retransmited 0 bad segments received. 14 resets sent UdpLite: TcpExt: 1 TCP sockets finished time wait in fast timer 7 delayed acks sent Quick ack mode was activated 15 times 5 packets directly queued to recvmsg prequeue. 2 packets directly received from prequeue 666 packets header predicted 498190 acknowledgments not containing data received 23392 predicted acknowledgments 2 congestion windows recovered after partial ack 0 TCP data loss events 4 other TCP timeouts 15 DSACKs sent for old packets 16 DSACKs sent for out of order packets 1 connections reset due to unexpected data IpExt: InMcastPkts: 14 OutMcastPkts: 16 InBcastPkts: 33 [root@localhost ~]# ethtool -S eth0 NIC statistics: tx_bytes: 1175487542 tx_zero_rexmt: 821009 tx_one_rexmt: 0 tx_many_rexmt: 0 tx_late_collision: 0 tx_fifo_errors: 0 tx_carrier_errors: 0 tx_excess_deferral: 0 tx_retry_error: 0 rx_frame_error: 0 rx_extra_byte: 0 rx_late_collision: 0 rx_runt: 0 rx_frame_too_long: 0 rx_over_errors: 0 rx_crc_errors: 9301 rx_frame_align_error: 0 rx_length_error: 0 rx_unicast: 523675 rx_multicast: 0 rx_broadcast: 41 rx_packets: 523716 rx_errors_total: 9301 tx_errors_total: 0 tx_deferral: 0 tx_packets: 821009 rx_bytes: 39624285 tx_pause: 0 rx_pause: 0 rx_drop_frame: 0 [root@localhost ~]# netperf -v 1 -l 60 -H 172.16.1.7 -t TCP_RR TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.1.7 (172.16.1.7) port 0 AF_INET Local /Remote Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec 16384 87380 1 1 60.00 350.52 3217968 3217968 [root@localhost ~]# netstat -s -t Tcp: 20 active connections openings 0 passive connection openings 10 failed connection attempts 1 connection resets received 2 connections established 545934 segments received 843030 segments send out 226 segments retransmited 0 bad segments received. 15 resets sent UdpLite: TcpExt: 3 TCP sockets finished time wait in fast timer 9 delayed acks sent Quick ack mode was activated 22 times 20941 packets directly queued to recvmsg prequeue. 190 packets directly received from backlog 20725 packets directly received from prequeue 1046 packets header predicted 20913 packets header predicted and directly queued to user 498407 acknowledgments not containing data received 44212 predicted acknowledgments 3 congestion windows recovered after partial ack 0 TCP data loss events 7 timeouts after SACK recovery 219 other TCP timeouts 22 DSACKs sent for old packets 23 DSACKs sent for out of order packets 1 connections reset due to unexpected data 1 connections reset due to early user close IpExt: InMcastPkts: 14 OutMcastPkts: 16 InBcastPkts: 40 [root@localhost ~]# ethtool -S eth0 NIC statistics: tx_bytes: 1177080452 tx_zero_rexmt: 843341 tx_one_rexmt: 0 tx_many_rexmt: 0 tx_late_collision: 0 tx_fifo_errors: 0 tx_carrier_errors: 0 tx_excess_deferral: 0 tx_retry_error: 0 rx_frame_error: 0 rx_extra_byte: 0 rx_late_collision: 0 rx_runt: 0 rx_frame_too_long: 0 rx_over_errors: 0 rx_crc_errors: 9750 rx_frame_align_error: 0 rx_length_error: 0 rx_unicast: 545997 rx_multicast: 0 rx_broadcast: 50 rx_packets: 546047 rx_errors_total: 9750 tx_errors_total: 0 tx_deferral: 0 tx_packets: 843341 rx_bytes: 42743567 tx_pause: 0 rx_pause: 0 rx_drop_frame: 0 [root@localhost ~]# netperf -v 1 -l 60 -H 172.16.1.7 -t UDP_RR UDP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.16.1.7 (172.16.1.7) port 0 AF_INET Local /Remote Socket Size Request Resp. Elapsed Trans. Send Recv Size Size Time Rate bytes Bytes bytes bytes secs. per sec 124928 124928 1 1 60.00 5.55 57344 524288 [root@localhost ~]# netstat -s -t Tcp: 21 active connections openings 0 passive connection openings 10 failed connection attempts 1 connection resets received 2 connections established 546873 segments received 843968 segments send out 226 segments retransmited 0 bad segments received. 15 resets sent UdpLite: TcpExt: 4 TCP sockets finished time wait in fast timer 10 delayed acks sent Quick ack mode was activated 27 times 20941 packets directly queued to recvmsg prequeue. 190 packets directly received from backlog 20725 packets directly received from prequeue 1323 packets header predicted 20913 packets header predicted and directly queued to user 498409 acknowledgments not containing data received 44213 predicted acknowledgments 3 congestion windows recovered after partial ack 0 TCP data loss events 7 timeouts after SACK recovery 219 other TCP timeouts 27 DSACKs sent for old packets 29 DSACKs sent for out of order packets 1 connections reset due to unexpected data 1 connections reset due to early user close IpExt: InMcastPkts: 14 OutMcastPkts: 16 InBcastPkts: 47 [root@localhost ~]# ethtool -S eth0 NIC statistics: tx_bytes: 1177171368 tx_zero_rexmt: 844569 tx_one_rexmt: 0 tx_many_rexmt: 0 tx_late_collision: 0 tx_fifo_errors: 0 tx_carrier_errors: 0 tx_excess_deferral: 0 tx_retry_error: 0 rx_frame_error: 0 rx_extra_byte: 0 rx_late_collision: 0 rx_runt: 0 rx_frame_too_long: 0 rx_over_errors: 0 rx_crc_errors: 10049 rx_frame_align_error: 0 rx_length_error: 0 rx_unicast: 547225 rx_multicast: 0 rx_broadcast: 56 rx_packets: 547281 rx_errors_total: 10049 tx_errors_total: 0 tx_deferral: 0 tx_packets: 844569 rx_bytes: 44106473 tx_pause: 0 rx_pause: 0 rx_drop_frame: 0 [root@localhost ~]# Arno ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bugme-new] [Bug 11752] New: Extremely low netperf UDP_RR throughput for nvidia MCP65 2008-10-18 11:36 ` Arno J. Klaassen @ 2008-10-20 17:41 ` Rick Jones 2008-10-31 13:07 ` Arno J. Klaassen 2008-12-11 20:53 ` Arno J. Klaassen 0 siblings, 2 replies; 9+ messages in thread From: Rick Jones @ 2008-10-20 17:41 UTC (permalink / raw) To: Arno J. Klaassen; +Cc: Andrew Morton, netdev, bugme-daemon, Ayaz Abdulla >>Clearly something is fubar with the rx side (well duh :). The next >>set of stats I'd try to look at would be ethtool stats for the >>interface, eg ethtool -S eth0 and see if it shows someting more >>specific for the "RX-ERR" shown by netstat -I eth0. > > > OK, here it is (rx_errors_total: 10049, rx_crc_errors: 10049) : Well, that seems to confirm that it is CRC errors. Did your friend say why he thought they were false CRC errors? If indeed it is only small packets/frames getting the CRC errors, in _theory_ a TCP_RR test with a larger request/response size should show "good" performance because there should be few if any standalone ACKs. You could start with something like: netperf -H <remote> -t TCP_RR -- -r 1448 and work your way down, checking for those CRC errors as you go. I don't thnik folks need all the output, just some idea of if you still see CRC errors with full-size TCP_RR and if you don't at what size you start to see them. I picked 1448 to have the request and response both result in a "full" TCP segment assuming an MTU of 1500 bytes and timestamps being enabled. (net.ipv4.tcp_timestamps). Has the usual litany of cable swapping and such been done already? A cable *known* to be good at 1G swapped-in and such? If this is via a switch, just for completeness trying other switch ports etc etc. While I'd not expect it to be at 1Gig and autoneg, CRC errors can sometimes be a sign of a duplex mismatch, but I have a difficult time seeing that happening - unless there happens to be other traffic on the link a plain netperf TCP_RR or UDP_RR test should "never" have both sides trying to talk at the same time and so shouldn't trip-over a duplex mismatch like a TCP_STREAM test would. > (NB, let me know how to eventually test eventual patches/binary > modules on a life-CD; I've just limited linux kernel skills) I'm going to have to defer to others on that score. Meanwhile some additional information gathering: For grins and bugzilla posterity, ethtool -i <interface> would be goodness. What was the last "known good" configuration? What is running "on the other side?" etc etc. Does say some other or earlier distro (Fedora, Ubuntu whatnot) Live CD not exhibit this problem? If not, what are the kernel and ethtool -i information from that? rick jones ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bugme-new] [Bug 11752] New: Extremely low netperf UDP_RR throughput for nvidia MCP65 2008-10-20 17:41 ` Rick Jones @ 2008-10-31 13:07 ` Arno J. Klaassen 2008-12-11 20:53 ` Arno J. Klaassen 1 sibling, 0 replies; 9+ messages in thread From: Arno J. Klaassen @ 2008-10-31 13:07 UTC (permalink / raw) To: Rick Jones; +Cc: Andrew Morton, netdev, bugme-daemon, Ayaz Abdulla Hello, sorry for the late response Rick Jones <rick.jones2@hp.com> writes: > >>Clearly something is fubar with the rx side (well duh :). The next > >>set of stats I'd try to look at would be ethtool stats for the > >>interface, eg ethtool -S eth0 and see if it shows someting more > >>specific for the "RX-ERR" shown by netstat -I eth0. > > OK, here it is (rx_errors_total: 10049, rx_crc_errors: 10049) : > > Well, that seems to confirm that it is CRC errors. Did your friend > say why he thought they were false CRC errors? nope, but he gave me a patch for the freebsd driver which [would] "try to pass CRC errored frames to upper stack." And it improved the *RR tests by an order of magnitude (still leaving them another order of magnitude below reference values) > If indeed it is only small packets/frames getting the CRC errors, in > _theory_ a TCP_RR test with a larger request/response size should show > "good" performance because there should be few if any standalone > ACKs. You could start with something like: > > netperf -H <remote> -t TCP_RR -- -r 1448 > > and work your way down, checking for those CRC errors as you go. I > don't thnik folks need all the output, just some idea of if you still > see CRC errors with full-size TCP_RR and if you don't at what size you > start to see them. I picked 1448 to have the request and response > both result in a "full" TCP segment assuming an MTU of 1500 bytes and > timestamps being enabled. (net.ipv4.tcp_timestamps). packet/frame size does not seem to be of any influence; I tried a bunch of combinations and all give more or less the same performance > Has the usual litany of cable swapping and such been done already? A > cable *known* to be good at 1G swapped-in and such? If this is via a > switch, just for completeness trying other switch ports etc etc. yop; I tested with 3 differents switches and a couple of different cables, including the famous *known* good one as well as a brand new one; no difference > While I'd not expect it to be at 1Gig and autoneg, CRC errors can > sometimes be a sign of a duplex mismatch, but I have a difficult time > seeing that happening - unless there happens to be other traffic on > the link a plain netperf TCP_RR or UDP_RR test should "never" have > both sides trying to talk at the same time and so shouldn't trip-over > a duplex mismatch like a TCP_STREAM test would. > > > (NB, let me know how to eventually test eventual patches/binary > > modules on a life-CD; I've just limited linux kernel skills) > > I'm going to have to defer to others on that score. Meanwhile some > additional information gathering: > > For grins and bugzilla posterity, ethtool -i <interface> would be > goodness. [root@localhost mcp65]# ethtool -i eth0 driver: forcedeth version: 0.61 firmware-version: bus-info: 0000:00:06.0 > What was the last "known good" configuration? What is > running "on the other side?" etc etc. Does say some other or earlier > distro (Fedora, Ubuntu whatnot) Live CD not exhibit this problem? If > not, what are the kernel and ethtool -i information from that? I have this problem since I bought this notebook a month ago. I tried freebsd7 (nfe driver), opensolaris5.10 (nfo driver) and fc10 all with same result. It also runs vista, but I cannot find a netperf.exe for 2.4.4 ... if someone has a pointer (I found an earlier version but it makes netserver core dump when startiong the test) thanx for your help > rick jones Arno -- ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Bugme-new] [Bug 11752] New: Extremely low netperf UDP_RR throughput for nvidia MCP65 2008-10-20 17:41 ` Rick Jones 2008-10-31 13:07 ` Arno J. Klaassen @ 2008-12-11 20:53 ` Arno J. Klaassen 1 sibling, 0 replies; 9+ messages in thread From: Arno J. Klaassen @ 2008-12-11 20:53 UTC (permalink / raw) To: Rick Jones; +Cc: Andrew Morton, netdev, bugme-daemon, Ayaz Abdulla Hello, Rick Jones <rick.jones2@hp.com> writes: > >>Clearly something is fubar with the rx side (well duh :). The next > >>set of stats I'd try to look at would be ethtool stats for the > >>interface, eg ethtool -S eth0 and see if it shows someting more > >>specific for the "RX-ERR" shown by netstat -I eth0. > > OK, here it is (rx_errors_total: 10049, rx_crc_errors: 10049) : > > Well, that seems to confirm that it is CRC errors. [ .. stuff deleted .. ] well, just a quick 'knock knock' in case any of you has something new on this I can live with 100Mbps, but would be glad to help further to get it work at 1gbps (pavillon dv6700). Looks like a silicon bug to me, but alas; please beep me if I can test something. Best regards, Arno ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2008-12-11 22:05 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <bug-11752-10286@http.bugzilla.kernel.org/>
2008-10-16 20:49 ` [Bugme-new] [Bug 11752] New: Extremely low netperf UDP_RR throughput for nvidia MCP65 Andrew Morton
2008-10-16 21:07 ` Rick Jones
2008-10-17 14:28 ` Arno J. Klaassen
2008-10-17 20:49 ` Arno J. Klaassen
2008-10-17 21:05 ` Rick Jones
2008-10-18 11:36 ` Arno J. Klaassen
2008-10-20 17:41 ` Rick Jones
2008-10-31 13:07 ` Arno J. Klaassen
2008-12-11 20:53 ` Arno J. Klaassen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).