[TG3]: About hw coalescing infrastructure.

[Eric Dumazet <dada1@cosmosbay.com>]

David S. Miller a écrit :
> Ok, now that we have the tagged status stuff sorted I began
> to work on putting the hw mitigation bits back into the
> driver.  The discussion on the DMA rw-ctrl settings is still
> ongoing, but I will get back to it shortly.
> 
> This is the first step, we cache the settings in the tg3
> struct and put those values into the chip via tg3_set_coalesce().
> 
> ETHTOOL_GCOALESCE is supported, setting is not.
> 
Hi David

I am using 2.6.12 now, but still experiment a high number of interrupts per second on
my tg3 NIC, on an dual Opteron based machine.

(about 7300 interrupts per second generated by one interface eth0, 100Mbit/s link)

Is there anything I can try to tune the coalescing ?
Being able to handle 100 packets each interrupt instead of one or two would certainly help.
I dont mind about latency. But of course I would like not to drop packets :)
But maybe the BCM5702 is not able to delay an interrupt ?

Thank you
Eric Dumazet

----------------------------------------------------------------------------------------
# lspci -v
02:03.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5702 Gigabit Ethernet (rev 02)
         Subsystem: Broadcom Corporation BCM5702 1000Base-T
         Flags: bus master, 66Mhz, medium devsel, latency 64, IRQ 27
         Memory at 00000000fe000000 (64-bit, non-prefetchable) [size=64K]
         Expansion ROM at <unassigned> [disabled] [size=64K]
         Capabilities: [40] PCI-X non-bridge device.
         Capabilities: [48] Power Management version 2
         Capabilities: [50] Vital Product Data
         Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-

# ethtool -c eth0
Coalesce parameters for eth0:
Adaptive RX: off  TX: off
stats-block-usecs: 1000000
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0

rx-usecs: 20
rx-frames: 5
rx-usecs-irq: 20
rx-frames-irq: 5

tx-usecs: 72
tx-frames: 53
tx-usecs-irq: 20
tx-frames-irq: 5

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0


# ethtool -S eth0
NIC statistics:
      rx_octets: 104634072366
      rx_fragments: 0
      rx_ucast_packets: 852685070
      rx_mcast_packets: 0
      rx_bcast_packets: 20429
      rx_fcs_errors: 0
      rx_align_errors: 0
      rx_xon_pause_rcvd: 0
      rx_xoff_pause_rcvd: 0
      rx_mac_ctrl_rcvd: 0
      rx_xoff_entered: 0
      rx_frame_too_long_errors: 0
      rx_jabbers: 0
      rx_undersize_packets: 0
      rx_in_length_errors: 0
      rx_out_length_errors: 0
      rx_64_or_less_octet_packets: 451956709
      rx_65_to_127_octet_packets: 272058231
      rx_128_to_255_octet_packets: 63364655
      rx_256_to_511_octet_packets: 35814973
      rx_512_to_1023_octet_packets: 11867701
      rx_1024_to_1522_octet_packets: 17643210
      rx_1523_to_2047_octet_packets: 0
      rx_2048_to_4095_octet_packets: 0
      rx_4096_to_8191_octet_packets: 0
      rx_8192_to_9022_octet_packets: 0
      tx_octets: 134640205605
      tx_collisions: 0
      tx_xon_sent: 0
      tx_xoff_sent: 0
      tx_flow_control: 0
      tx_mac_errors: 0
      tx_single_collisions: 0
      tx_mult_collisions: 0
      tx_deferred: 0
      tx_excessive_collisions: 0
      tx_late_collisions: 0
      tx_collide_2times: 0
      tx_collide_3times: 0
      tx_collide_4times: 0
      tx_collide_5times: 0
      tx_collide_6times: 0
      tx_collide_7times: 0
      tx_collide_8times: 0
      tx_collide_9times: 0
      tx_collide_10times: 0
      tx_collide_11times: 0
      tx_collide_12times: 0
      tx_collide_13times: 0
      tx_collide_14times: 0
      tx_collide_15times: 0
      tx_ucast_packets: 774312055
      tx_mcast_packets: 13
      tx_bcast_packets: 246
      tx_carrier_sense_errors: 0
      tx_discards: 0
      tx_errors: 0
      dma_writeq_full: 21375
      dma_write_prioq_full: 0
      rxbds_empty: 0
      rx_discards: 2644
      rx_errors: 0
      rx_threshold_hit: 57384403
      dma_readq_full: 27100189
      dma_read_prioq_full: 1557267
      tx_comp_queue_full: 35712755
      ring_set_send_prod_index: 747986769
      ring_status_update: 502110997
      nic_irqs: 446148615
      nic_avoided_irqs: 55962382
      nic_tx_threshold_hit: 37282069

# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:             511
RX Mini:        0
RX Jumbo:       255
TX:             0
Current hardware settings:
RX:             200
RX Mini:        0
RX Jumbo:       100
TX:             511


# ethtool eth0
Settings for eth0:
         Supported ports: [ MII ]
         Supported link modes:   10baseT/Half 10baseT/Full
                                 100baseT/Half 100baseT/Full
                                 1000baseT/Half 1000baseT/Full
         Supports auto-negotiation: Yes
         Advertised link modes:  10baseT/Half 10baseT/Full
                                 100baseT/Half 100baseT/Full
                                 1000baseT/Half 1000baseT/Full
         Advertised auto-negotiation: Yes
         Speed: 100Mb/s
         Duplex: Full
         Port: Twisted Pair
         PHYAD: 1
         Transceiver: internal
         Auto-negotiation: on
         Supports Wake-on: g
         Wake-on: d
         Current message level: 0x000000ff (255)
         Link detected: yes


# cat /proc/interrupts    (HZ=200)
            CPU0       CPU1
   0:     164055   14038348    IO-APIC-edge  timer
   2:          0          0          XT-PIC  cascade
   8:          0          0    IO-APIC-edge  rtc
  14:       4073     368224    IO-APIC-edge  ide0
  15:          0         20    IO-APIC-edge  ide1
  27:   35985951  421578656   IO-APIC-level  eth0, eth1
NMI:  874625217  905019517  (oprofile running)
LOC:   14201857   14201976
ERR:          0
MIS:          0

oprofile data :
# more /tmp/vmlinux.oprofile
CPU: Hammer, speed 2205.08 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit mask of 0x00 (No unit mask) count 100000
samples  cum. samples  %        cum. %     symbol name
20208503 20208503       7.7982   7.7982    ipt_do_table
8336463  28544966       3.2169  11.0151    tcp_v4_rcv
7746814  36291780       2.9894  14.0045    handle_IRQ_event
7117968  43409748       2.7467  16.7512    tg3_poll
6585377  49995125       2.5412  19.2924    memcpy
5184695  55179820       2.0007  21.2931    ip_route_input
4346890  59526710       1.6774  22.9705    kfree
4214007  63740717       1.6261  24.5967    copy_user_generic_c
4093885  67834602       1.5798  26.1764    tcp_ack
4006753  71841355       1.5462  27.7226    tg3_interrupt_tagged
3778976  75620331       1.4583  29.1809    tcp_rcv_established
3756498  79376829       1.4496  30.6304    ip_queue_xmit
3418999  82795828       1.3193  31.9498    schedule
3274459  86070287       1.2636  33.2134    try_to_wake_up
3034809  89105096       1.1711  34.3844    tcp_sendmsg
2846436  91951532       1.0984  35.4828    kmem_cache_alloc
2745147  94696679       1.0593  36.5422    free_block
2679056  97375735       1.0338  37.5760    kmem_cache_free
2595289  99971024       1.0015  38.5775    fn_hash_lookup
2582072  102553096      0.9964  39.5738    __memset
2576462  105129558      0.9942  40.5681    tcp_transmit_skb
2528313  107657871      0.9756  41.5437    tcp_recvmsg
2392370  110050241      0.9232  42.4669    timer_interrupt
2365615  112415856      0.9129  43.3797    system_call
2358666  114774522      0.9102  44.2899    sockfd_lookup
2357192  117131714      0.9096  45.1995    tcp_poll
2340568  119472282      0.9032  46.1027    ip_rcv
2315805  121788087      0.8936  46.9964    tcp_match
2276212  124064299      0.8784  47.8747    sys_epoll_wait
2260913  126325212      0.8725  48.7472    __mod_timer
2173905  128499117      0.8389  49.5861    tg3_start_xmit
2057738  130556855      0.7941  50.3801    __switch_to
2022435  132579290      0.7804  51.1605    ep_poll_callback
2020449  134599739      0.7797  51.9402    sock_wfree
1913008  136512747      0.7382  52.6784    find_busiest_group
1891578  138404325      0.7299  53.4083    local_bh_enable
1860130  140264455      0.7178  54.1261    ip_local_deliver
1793639  142058094      0.6921  54.8183    __ip_route_output_key
1789287  143847381      0.6905  55.5087    alloc_skb
1770972  145618353      0.6834  56.1921    tcp_write_timer
1727286  147345639      0.6665  56.8587    __wake_up
1634111  148979750      0.6306  57.4893    skb_release_data
1625157  150604907      0.6271  58.1164    __kmalloc
1567198  152172105      0.6048  58.7211    tcp_v4_do_rcv
1562495  153734600      0.6029  59.3241    __kfree_skb