tg3 driver and interrupt coalescence questions

tg3 driver and interrupt coalescence questions

[Chris A. Icide]

I've been digging around trying to get some information on the current status of interrupt mitigation features for a Braodcom 5704 interface.

Specifically I'm sending and receiving lots of VoIP packets (50 pps per stream, many streams).

What I can't seem to determine is this:

What version of the linux kernel & tg3 drivers are required to support both rx and tx mitigation?
What do the ethtool coalescence settings actually do (I've not been able to find actual descriptions of the different parameters in the -C section)
Is there anything special that needs to be done when compiling a kernel to enable this feature for both the kernel and the tg3 driver.

Just a warning, I'm not a C coder, so I've not had much luck digging around the code and looking for answers.

I've currently got a blade server with 10 blades I'm using 9 blades to generate this small packet high rate traffic to the 10th blade and trying to improve the ability of a blade to handle VoIP traffic.  I made some guesses at settings for the -C options in ethtool on both the test blade and the traffic generators.  Interestingly it seems to have had a very good effect on the test blade (%cpu for interrupt down from 99.9% to ~20%), but the same settings on the traffic generation servers seems to have had no effect.

Hardware is identical, kernel is identical.

Any help is GREATLY appreciated.

-Chris

05:01.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704S
Gigabit Ethernet (rev 10)
        Subsystem: IBM: Unknown device 02e8
        Flags: bus master, 66Mhz, medium devsel, latency 64, IRQ 201
        Memory at dcfe0000 (64-bit, non-prefetchable) [size=64K]
        Capabilities: [40] PCI-X non-bridge device.
        Capabilities: [48] Power Management version 2
        Capabilities: [50] Vital Product Data
        Capabilities: [58] Message Signalled Interrupts: 64bit+
Queue=0/3 Enable-

Linux version 2.6.9-34.ELsmp (buildcentos@nasha.karan.org) (gcc version
3.4.5 20051201 (Red Hat 3.4.5-2)) #1 SMP Thu Mar 9 06:23:23 GMT 2006

[root@db2 ~]# ethtool -c eth1
Coalesce parameters for eth1:
Adaptive RX: off  TX: off
stats-block-usecs: 1000000
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0

rx-usecs: 500
rx-frames: 30
rx-usecs-irq: 500
rx-frames-irq: 20

tx-usecs: 400
tx-frames: 53
tx-usecs-irq: 490
tx-frames-irq: 5

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0

[root@db2 ~]# ethtool -i eth1
driver: tg3
version: 3.43-rh
firmware-version:
bus-info: 0000:05:01.1

[root@db2 ~]# ethtool eth1
Settings for eth1:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Half 1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  1000baseT/Half 1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: g
        Wake-on: d
        Current message level: 0x000000ff (255)
        Link detected: yes

Re: tg3 driver and interrupt coalescence questions

[Michael Chan]

On Tue, 2006-06-27 at 10:16 -0700, Chris A. Icide wrote:

> What version of the linux kernel & tg3 drivers are required to support both rx and tx mitigation?

ethtool -C for tg3 was added around July of 2005. The version with this
change added was 3.33.

> What do the ethtool coalescence settings actually do (I've not been able to find actual descriptions of the different parameters in the -C section)

They set the delay between the tx and rx events and the generation of
interrupts for those events.

These are the only parameters that are relevant for tg3:

rx-frames[-irq]
rx-usecs[-irq]
tx-frames[-irq]
tx-usecs[-irq]

The frames parameters specify how many packets are received/transmitted
before generating an interrupt.  The usecs parameters specify how many
microseconds after at least 1 packet is received/transmitted before
generating an interrupt.  The [-irq] parameters are the corresponding
delays in updating the status when the interrupt is disabled.

> Is there anything special that needs to be done when compiling a kernel to enable this feature for both the kernel and the tg3 driver.

No.

> 05:01.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704S
> Gigabit Ethernet (rev 10)
>         Subsystem: IBM: Unknown device 02e8
>         Flags: bus master, 66Mhz, medium devsel, latency 64, IRQ 201
>         Memory at dcfe0000 (64-bit, non-prefetchable) [size=64K]
>         Capabilities: [40] PCI-X non-bridge device.
>         Capabilities: [48] Power Management version 2
>         Capabilities: [50] Vital Product Data
>         Capabilities: [58] Message Signalled Interrupts: 64bit+
> Queue=0/3 Enable-
> 
> Linux version 2.6.9-34.ELsmp (buildcentos@nasha.karan.org) (gcc version
> 3.4.5 20051201 (Red Hat 3.4.5-2)) #1 SMP Thu Mar 9 06:23:23 GMT 2006
> 
> [root@db2 ~]# ethtool -c eth1
> Coalesce parameters for eth1:
> Adaptive RX: off  TX: off
> stats-block-usecs: 1000000
> sample-interval: 0
> pkt-rate-low: 0
> pkt-rate-high: 0
> 
> rx-usecs: 500
> rx-frames: 30
> rx-usecs-irq: 500
> rx-frames-irq: 20
> 

This means that the first interrupt will be generated after 30 packets
are received or 500 microseconds after the nth packet is received (1 <=
n < 30). When irq is disabled, 20 packets instead of 30 before updating
status.

> tx-usecs: 400
> tx-frames: 53
> tx-usecs-irq: 490
> tx-frames-irq: 5

The first tx interrupt will be generated after 53 packets are
transmitted or 400 microseconds after the nth packet is transmitted (1
<= n < 53). When irq is disabled, 5 packets or 490 micosecs before
updating status.

If the condition for generating a tx or rx interrupt is met, you get all
the accumulated tx and rx status during the interrupt.

Hope this helps.

Re: tg3 driver and interrupt coalescence questions

[Rick Jones]

Chris A. Icide wrote:
> I've been digging around trying to get some information on the
> current status of interrupt mitigation features for a Braodcom 5704 interface.
> 
> Specifically I'm sending and receiving lots of VoIP packets (50 pps
 > per stream, many streams).
> 
> What I can't seem to determine is this:
> 
> What version of the linux kernel & tg3 drivers are required to
> support both rx and tx mitigation?
> What do the ethtool coalescence settings actually do (I've not been

Delay interrupts and increase individual packet latency with the 
intention being decreasing CPU utilization and allowing a higher 
aggregate packet per second limit.  IE bandwidth vs latency tradeoffs.

> able to find actual descriptions of the different parameters in the -C
> section)
> Is there anything special that needs to be done when compiling a
> kernel to enable this feature for both the kernel and the tg3 driver.

Are you looking to increase or decrease the settings?  I would think 
(initially at least) that for VOIP one might not want to increase them.

rick jones

Re: tg3 driver and interrupt coalescence questions

[Chris A. Icide]

Rick Jones wrote:
>
> Are you looking to increase or decrease the settings?  I would think
> (initially at least) that for VOIP one might not want to increase them.
>
> rick jones
I'm looking to decrease the interrupt load on the system.  During the
test I mentioned above I had some interesting and confusing results. 
The changes from the default settings to the settings I posted resulted
in a 100% performance increase (counted by the number of VoIP audio
streams the tested server could support).  With default settings one of
the two CPUs in the system maxed out at 99% cpu usage handling
interrupts, while the second CPU was not maxed out, but we started to
drop packets and the VoIP call setups started showing retransmits (which
is the measurement for failure in this test) at about 300 streams.  With
the new settings we were able to hit 600 streams.

So I definately recognized a significant improvement.  However I'd still
like to get more improvement.  At 600 streams and 20ms packets we are
looking at 30,000 pps.  The % of cpu (1 CPU as apparently the interrupts
can't be shared across multiple CPUs) used for interrupt handling at
this 600 stream limit was 88.0%.

Now what was interesting was on the test generation side (same hardware
exactly) of things, I was using the SIPP software to generate the VoIP
streams, and each blade in the blade server was only able to generate
~200 streams, with default settings in ethtool, one of the CPUs would
hit max usage for interrupt handling at that point.  So I modified the
ethtool settings to match those I listed above and there was no
discernable difference.  It was identical performance to the default
settings. 

Michael's response clarified for me what the actual parameters in the -C
section of ethtool do, thanks Michael.  However I';; be greatly
appreciative of any recommedations anyone might have for interrupt
mitigation settings for 100% UDP RTP traffic of 20ms packets (50 pps per
stream).

-Chris

Re: tg3 driver and interrupt coalescence questions

[Robert Iakobashvili]

Hi Cris,

> I'm looking to decrease the interrupt load on the system.  During the
> test I mentioned above I had some interesting and confusing results.
> The changes from the default settings to the settings I posted resulted
> in a 100% performance increase (counted by the number of VoIP audio
> streams the tested server could support).  With default settings one of
> the two CPUs in the system maxed out at 99% cpu usage handling
> interrupts, while the second CPU was not maxed out, but we started to
> drop packets and the VoIP call setups started showing retransmits (which
> is the measurement for failure in this test) at about 300 streams.  With
> the new settings we were able to hit 600 streams.
>
> So I definately recognized a significant improvement.  However I'd still
> like to get more improvement.  At 600 streams and 20ms packets we are
> looking at 30,000 pps.  The % of cpu (1 CPU as apparently the interrupts
> can't be shared across multiple CPUs) used for interrupt handling at
> this 600 stream limit was 88.0%.

interrupts can be balances across multiple CPUs or not.
It depends on 4 areas:
1. enabling/disabling such option in kernel upon compilation;
2. enabling/disabling of a user-space service for interrupt balancing,
"irqbalance" on redhat, nothing such on debian;
3. enabling of disabling cpu affinity for an irq;

Normally, irq-affinity for a nic interrupt is considered good, but if a CPU
is overloaded you may try irq balancing.

> Now what was interesting was on the test generation side (same hardware
> exactly) of things, I was using the SIPP software to generate the VoIP
> streams, and each blade in the blade server was only able to generate
> ~200 streams, with default settings in ethtool, one of the CPUs would
> hit max usage for interrupt handling at that point.  So I modified the
> ethtool settings to match those I listed above and there was no
> discernable difference.  It was identical performance to the default
> settings.

RTP streams generation can burn your CPU cycles as well as output
of them to network, thus balancing of the
load among the CPUs, irqbalancing may improve something.

-- 
Sincerely,
------------------------------------------------------------------
Robert Iakobashvili, coroberti at gmail dot com
Navigare necesse est, vivere non est necesse.
------------------------------------------------------------------