From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Dumazet Subject: [TG3]: About hw coalescing infrastructure. Date: Wed, 22 Jun 2005 17:25:08 +0200 Message-ID: <42B982D4.9040704@cosmosbay.com> References: <20050511.141530.57445142.davem@davemloft.net> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Cc: netdev@oss.sgi.com, mchan@broadcom.com Return-path: To: "David S. Miller" In-Reply-To: <20050511.141530.57445142.davem@davemloft.net> Sender: netdev-bounce@oss.sgi.com Errors-to: netdev-bounce@oss.sgi.com List-Id: netdev.vger.kernel.org David S. Miller a =E9crit : > Ok, now that we have the tagged status stuff sorted I began > to work on putting the hw mitigation bits back into the > driver. The discussion on the DMA rw-ctrl settings is still > ongoing, but I will get back to it shortly. >=20 > This is the first step, we cache the settings in the tg3 > struct and put those values into the chip via tg3_set_coalesce(). >=20 > ETHTOOL_GCOALESCE is supported, setting is not. >=20 Hi David I am using 2.6.12 now, but still experiment a high number of interrupts p= er second on my tg3 NIC, on an dual Opteron based machine. (about 7300 interrupts per second generated by one interface eth0, 100Mbi= t/s link) Is there anything I can try to tune the coalescing ? Being able to handle 100 packets each interrupt instead of one or two wou= ld certainly help. I dont mind about latency. But of course I would like not to drop packets= :) But maybe the BCM5702 is not able to delay an interrupt ? Thank you Eric Dumazet -------------------------------------------------------------------------= --------------- # lspci -v 02:03.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5702 Gigab= it Ethernet (rev 02) Subsystem: Broadcom Corporation BCM5702 1000Base-T Flags: bus master, 66Mhz, medium devsel, latency 64, IRQ 27 Memory at 00000000fe000000 (64-bit, non-prefetchable) [size=3D64= K] Expansion ROM at [disabled] [size=3D64K] Capabilities: [40] PCI-X non-bridge device. Capabilities: [48] Power Management version 2 Capabilities: [50] Vital Product Data Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=3D= 0/3 Enable- # ethtool -c eth0 Coalesce parameters for eth0: Adaptive RX: off TX: off stats-block-usecs: 1000000 sample-interval: 0 pkt-rate-low: 0 pkt-rate-high: 0 rx-usecs: 20 rx-frames: 5 rx-usecs-irq: 20 rx-frames-irq: 5 tx-usecs: 72 tx-frames: 53 tx-usecs-irq: 20 tx-frames-irq: 5 rx-usecs-low: 0 rx-frame-low: 0 tx-usecs-low: 0 tx-frame-low: 0 rx-usecs-high: 0 rx-frame-high: 0 tx-usecs-high: 0 tx-frame-high: 0 # ethtool -S eth0 NIC statistics: rx_octets: 104634072366 rx_fragments: 0 rx_ucast_packets: 852685070 rx_mcast_packets: 0 rx_bcast_packets: 20429 rx_fcs_errors: 0 rx_align_errors: 0 rx_xon_pause_rcvd: 0 rx_xoff_pause_rcvd: 0 rx_mac_ctrl_rcvd: 0 rx_xoff_entered: 0 rx_frame_too_long_errors: 0 rx_jabbers: 0 rx_undersize_packets: 0 rx_in_length_errors: 0 rx_out_length_errors: 0 rx_64_or_less_octet_packets: 451956709 rx_65_to_127_octet_packets: 272058231 rx_128_to_255_octet_packets: 63364655 rx_256_to_511_octet_packets: 35814973 rx_512_to_1023_octet_packets: 11867701 rx_1024_to_1522_octet_packets: 17643210 rx_1523_to_2047_octet_packets: 0 rx_2048_to_4095_octet_packets: 0 rx_4096_to_8191_octet_packets: 0 rx_8192_to_9022_octet_packets: 0 tx_octets: 134640205605 tx_collisions: 0 tx_xon_sent: 0 tx_xoff_sent: 0 tx_flow_control: 0 tx_mac_errors: 0 tx_single_collisions: 0 tx_mult_collisions: 0 tx_deferred: 0 tx_excessive_collisions: 0 tx_late_collisions: 0 tx_collide_2times: 0 tx_collide_3times: 0 tx_collide_4times: 0 tx_collide_5times: 0 tx_collide_6times: 0 tx_collide_7times: 0 tx_collide_8times: 0 tx_collide_9times: 0 tx_collide_10times: 0 tx_collide_11times: 0 tx_collide_12times: 0 tx_collide_13times: 0 tx_collide_14times: 0 tx_collide_15times: 0 tx_ucast_packets: 774312055 tx_mcast_packets: 13 tx_bcast_packets: 246 tx_carrier_sense_errors: 0 tx_discards: 0 tx_errors: 0 dma_writeq_full: 21375 dma_write_prioq_full: 0 rxbds_empty: 0 rx_discards: 2644 rx_errors: 0 rx_threshold_hit: 57384403 dma_readq_full: 27100189 dma_read_prioq_full: 1557267 tx_comp_queue_full: 35712755 ring_set_send_prod_index: 747986769 ring_status_update: 502110997 nic_irqs: 446148615 nic_avoided_irqs: 55962382 nic_tx_threshold_hit: 37282069 # ethtool -g eth0 Ring parameters for eth0: Pre-set maximums: RX: 511 RX Mini: 0 RX Jumbo: 255 TX: 0 Current hardware settings: RX: 200 RX Mini: 0 RX Jumbo: 100 TX: 511 # ethtool eth0 Settings for eth0: Supported ports: [ MII ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full Advertised auto-negotiation: Yes Speed: 100Mb/s Duplex: Full Port: Twisted Pair PHYAD: 1 Transceiver: internal Auto-negotiation: on Supports Wake-on: g Wake-on: d Current message level: 0x000000ff (255) Link detected: yes # cat /proc/interrupts (HZ=3D200) CPU0 CPU1 0: 164055 14038348 IO-APIC-edge timer 2: 0 0 XT-PIC cascade 8: 0 0 IO-APIC-edge rtc 14: 4073 368224 IO-APIC-edge ide0 15: 0 20 IO-APIC-edge ide1 27: 35985951 421578656 IO-APIC-level eth0, eth1 NMI: 874625217 905019517 (oprofile running) LOC: 14201857 14201976 ERR: 0 MIS: 0 oprofile data : # more /tmp/vmlinux.oprofile CPU: Hammer, speed 2205.08 MHz (estimated) Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a uni= t mask of 0x00 (No unit mask) count 100000 samples cum. samples % cum. % symbol name 20208503 20208503 7.7982 7.7982 ipt_do_table 8336463 28544966 3.2169 11.0151 tcp_v4_rcv 7746814 36291780 2.9894 14.0045 handle_IRQ_event 7117968 43409748 2.7467 16.7512 tg3_poll 6585377 49995125 2.5412 19.2924 memcpy 5184695 55179820 2.0007 21.2931 ip_route_input 4346890 59526710 1.6774 22.9705 kfree 4214007 63740717 1.6261 24.5967 copy_user_generic_c 4093885 67834602 1.5798 26.1764 tcp_ack 4006753 71841355 1.5462 27.7226 tg3_interrupt_tagged 3778976 75620331 1.4583 29.1809 tcp_rcv_established 3756498 79376829 1.4496 30.6304 ip_queue_xmit 3418999 82795828 1.3193 31.9498 schedule 3274459 86070287 1.2636 33.2134 try_to_wake_up 3034809 89105096 1.1711 34.3844 tcp_sendmsg 2846436 91951532 1.0984 35.4828 kmem_cache_alloc 2745147 94696679 1.0593 36.5422 free_block 2679056 97375735 1.0338 37.5760 kmem_cache_free 2595289 99971024 1.0015 38.5775 fn_hash_lookup 2582072 102553096 0.9964 39.5738 __memset 2576462 105129558 0.9942 40.5681 tcp_transmit_skb 2528313 107657871 0.9756 41.5437 tcp_recvmsg 2392370 110050241 0.9232 42.4669 timer_interrupt 2365615 112415856 0.9129 43.3797 system_call 2358666 114774522 0.9102 44.2899 sockfd_lookup 2357192 117131714 0.9096 45.1995 tcp_poll 2340568 119472282 0.9032 46.1027 ip_rcv 2315805 121788087 0.8936 46.9964 tcp_match 2276212 124064299 0.8784 47.8747 sys_epoll_wait 2260913 126325212 0.8725 48.7472 __mod_timer 2173905 128499117 0.8389 49.5861 tg3_start_xmit 2057738 130556855 0.7941 50.3801 __switch_to 2022435 132579290 0.7804 51.1605 ep_poll_callback 2020449 134599739 0.7797 51.9402 sock_wfree 1913008 136512747 0.7382 52.6784 find_busiest_group 1891578 138404325 0.7299 53.4083 local_bh_enable 1860130 140264455 0.7178 54.1261 ip_local_deliver 1793639 142058094 0.6921 54.8183 __ip_route_output_key 1789287 143847381 0.6905 55.5087 alloc_skb 1770972 145618353 0.6834 56.1921 tcp_write_timer 1727286 147345639 0.6665 56.8587 __wake_up 1634111 148979750 0.6306 57.4893 skb_release_data 1625157 150604907 0.6271 58.1164 __kmalloc 1567198 152172105 0.6048 58.7211 tcp_v4_do_rcv 1562495 153734600 0.6029 59.3241 __kfree_skb