CPU usage of simple DROP rule

CPU usage of simple DROP rule

[Steve Fink]

(Oops; originally posted onto wrong list.)

I have a single OUTPUT rule (drop a particular UDP host:port) that
eats up a whole CPU core and a half (I have 8 cores total). It is the
only rule I have. It doesn't matter whether I do it in the raw table
or the filter table. This is when I'm just about maxing out 5 gigabit
NICs (outgoing traffic only). The rule matches nearly all of the
packets.

Is there any way to reduce the load, or at least spread it out over all CPUs?

My 8 cores are all at about 30% usage when I have no rules defined
(and the packets are going out to the switch). When I add that rule,
one of the cores shoots to 100%, another to 70% or so. The rest don't
really change.

I'm trying to figure out how to accomplish the same thing with a
blackhole route for comparison's sake, but I can't get it to work --
the packets are still going out.

Re: CPU usage of simple DROP rule

[Marek Kierdelewicz]

Hello

>I have a single OUTPUT rule (drop a particular UDP host:port) that
>...
>My 8 cores are all at about 30% usage when I have no rules defined
>(and the packets are going out to the switch). When I add that rule,
>one of the cores shoots to 100%, another to 70% or so. The rest don't
>really change.

Looks like two cores are being hit by ksoftirqd. There are some paths
you can explore to achieve lower cpu usage/better core-load
distribution:
- try using smp_affinity - bind different nic irqs to different cores;
  you can also use bonding to achieve better traffic distribution
  among nics;
- are you using intel i/o at dma support? it should lower network
  overhead for localy generated traffic;
- try adjusting nic coalesce parameters - it should lower network
  cpu overhead at the cost of higher latency;

Cheers
Marek Kierdelewicz

Re: CPU usage of simple DROP rule

[Steve Fink]

Well, I was originally having a problem with the CPU load distribution
independent of netfilter. Initially, one CPU was getting the bulk of
the IRQs, so I used smp_affinity to spread them out more evenly.
Oddly, that made no difference. However, when I created multiple
threads sending the packets out, and forced all packets coming from a
particular thread to go out on a specific interface, it fixed the
problem. (So I have one sending thread per NIC.) I still don't
entirely understand that -- with the smp_affinity method,
/proc/interrupts showed the interrupts spread out over CPU1..CPU5, and
yet CPU0 was getting hammered with soft irq processing. I think the
(single) sending thread was running on CPU0, which mattered more than
the smp_affinity of the irqs themselves.

How would I know if I'm using intel i/o dma or whatever? 4 of my 6
total gigabit NICs are using the e1000e driver. The other two are the
'igb' driver, whatever that is.  modinfo says "Intel(R) PRO/1000
Network Driver" for e1000e and "Intel(R) Gigabit Ethernet Network
Driver" for igb. lspci says

Intel Corporation 82576 Gigabit Network Connection (rev 01)
and
Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)

dmesg says:
igb 0000:05:00.0: eth0: PBA No: ffffff-0ff
igb 0000:05:00.0: Using MSI-X interrupts. 4 rx queue(s), 1 tx queue(s)
eth4: MAC: 0, PHY: 4, PBA No: d98771-007

(the latter is from e1000e).

By "nic coalesce parameters", do you mean TxIntDelay and
TxAbsIntDelay, or InterruptThrottleRate, or IntMode (all for e1000e)?
Or something else? How can I tell what the current settings are?
(Sorry, that's probably a basic question.)

Thanks!
Steve

On Tue, Sep 29, 2009 at 12:27 PM, Marek Kierdelewicz <marek@piasta.pl> wrote:
> Hello
>
>>I have a single OUTPUT rule (drop a particular UDP host:port) that
>>...
>>My 8 cores are all at about 30% usage when I have no rules defined
>>(and the packets are going out to the switch). When I add that rule,
>>one of the cores shoots to 100%, another to 70% or so. The rest don't
>>really change.
>
> Looks like two cores are being hit by ksoftirqd. There are some paths
> you can explore to achieve lower cpu usage/better core-load
> distribution:
> - try using smp_affinity - bind different nic irqs to different cores;
>  you can also use bonding to achieve better traffic distribution
>  among nics;
> - are you using intel i/o at dma support? it should lower network
>  overhead for localy generated traffic;
> - try adjusting nic coalesce parameters - it should lower network
>  cpu overhead at the cost of higher latency;
>
> Cheers
> Marek Kierdelewicz
>

Re: CPU usage of simple DROP rule

[Marek Kierdelewicz]

Hello,

>Well, I was originally having a problem with the CPU load distribution
>independent of netfilter. Initially, one CPU was getting the bulk of
>..
>(single) sending thread was running on CPU0, which mattered more than
>the smp_affinity of the irqs themselves.

Well, in my line of work I'm mostly dealing with routed/bridged traffic
not localy generated. That would explain different results of
setting smp_affinity.

>How would I know if I'm using intel i/o dma or whatever? 4 of my 6
>total gigabit NICs are using the e1000e driver. The other two are the
>'igb' driver, whatever that is.  modinfo says "Intel(R) PRO/1000
>Network Driver" for e1000e and "Intel(R) Gigabit Ethernet Network
>Driver" for igb. lspci says

There is some info about i/o at:
http://downloadmirror.intel.com/10959/eng/README.txt
... but it seems it can speed up receiving
not transimitting :(

>By "nic coalesce parameters", do you mean TxIntDelay and
>TxAbsIntDelay, or InterruptThrottleRate, or IntMode (all for e1000e)?
>Or something else? How can I tell what the current settings are?
>(Sorry, that's probably a basic question.)

I mean those settings: tx-usecs, tx-frames, tx-usecs-irq,
tx-frames-irq. They can be set by ethtool binary. AFAIK by default
there is no coalescence enabled (at least at my boxes I don't see any).
Meaning of parameters is well described here:
http://www.mail-archive.com/netdev@vger.kernel.org/msg14995.html

"> tx-usecs: 400
> tx-frames: 53
> tx-usecs-irq: 490
> tx-frames-irq: 5

The first tx interrupt will be generated after 53 packets are
transmitted or 400 microseconds after the nth packet is transmitted (1
<= n < 53). When irq is disabled, 5 packets or 490 micosecs before
updating status."

Cheers
Marek Kierdelewicz