netfilter performance dependent on arch

netfilter performance dependent on arch

[Marek Kierdelewicz]

Hi,

Can anyone point me to some performance comparison of netfilter on i686
and x86_64? I have a few linux routers doing a lot of firewalling and
QoS. Currently those routers use i686 arch on 64-bit hardware. Would I
notice any performance gain after moving to 64-bit kernel?

Next question. On some routers I don't need statefull firewall at all
and I have NOTRACT as a default rule in raw netfilter table. What is
the expected performance gain if I would fully disable conntrack
instead of using NOTRACK target? What would be the best approach to do
it on debian squeeze distribution kernel (nf_conntrack compiled as a
module)? Is blacklisting nf_conntrack module enough (to be safe in case
of accidental addition of statefull rule)?

best regards,
Marek Kierdelewicz

Re: netfilter performance dependent on arch

[Rick Jones]

On 02/07/2012 09:34 AM, Marek Kierdelewicz wrote:
> Hi,
>
> Can anyone point me to some performance comparison of netfilter on i686
> and x86_64? I have a few linux routers doing a lot of firewalling and
> QoS. Currently those routers use i686 arch on 64-bit hardware. Would I
> notice any performance gain after moving to 64-bit kernel?

Apart from the obvious "Try it on your workload and see." suggestion, 
you could I suppose make the assumption that a netfilter workload looks 
similar to one or more components of something like 
SPECint_rate_base2006 or SPECint_base2006 and mine the results on 
www.spec.org for some comparisons between 32 and 64 bit compiles on the 
same chip(s).  Probably want something with lots of control flow (term?) 
rather than loops and such.  A high-level description of the integer 
components can be found at http://www.spec.org/cpu2006/CINT2006/ . One 
of my first guesses would be 400.perlbench and 403.gcc but definitely 
peruse the descriptions of all of them.

rick jones

Re: netfilter performance dependent on arch

[Marek Kierdelewicz]

>> Can anyone point me to some performance comparison of netfilter on
>> i686 and x86_64? I have a few linux routers doing a lot of
>> firewalling and QoS. Currently those routers use i686 arch on 64-bit
>> hardware. Would I notice any performance gain after moving to 64-bit
>> kernel?

>Apart from the obvious "Try it on your workload and see." suggestion, 
>you could I suppose make the assumption that a netfilter workload
>looks similar to one or more components of something like 

Thanks for the response. Anyway moments ago I found answer to the first
question on vyatta forum here:
http://www.vyatta.org/forum/viewtopic.php?t=823&sid=3cec85f2cf7866ca26040ec491415ee1

shemminger "The researchers at Uppsla University who are measuring 10G
routing performance found that 64bit kernel is slower than 32bit kernel
for routing. Most likely the increased code size caused a larger number
of CPU cache misses. 64 bit would be an advantage if you are trying to
run lots of services."

DaveRoberts "This result is actually not too surprising. Networking code
is typically very cache sensitive. This gets worse with 64-bit. If
you're running code that uses data structures with lots of pointers
(which networking code often has) and you don't needed the expanded
address space, then you're wasting half your d-cache with 32 upper bits
of zeros. While the additional registers are a benefit, they aren't
enough to make up for the increased cache miss rate.

But as you say, without knowing the compiler, optimizer, and
instrumenting the heck out of the code with all the hardware analysis
counters, it's difficult to say what's really going on. All that is on
our list of things to do over time. The main takeaway is that it's
definitely not a slam dunk one way or another."

Best regards,
Marek Kierdelewicz

Re: netfilter performance dependent on arch

[Stephen Hemminger]

On Tue, 7 Feb 2012 19:54:11 +0100
Marek Kierdelewicz <marek@piasta.pl> wrote:

> >> Can anyone point me to some performance comparison of netfilter on
> >> i686 and x86_64? I have a few linux routers doing a lot of
> >> firewalling and QoS. Currently those routers use i686 arch on 64-bit
> >> hardware. Would I notice any performance gain after moving to 64-bit
> >> kernel?
> 
> >Apart from the obvious "Try it on your workload and see." suggestion, 
> >you could I suppose make the assumption that a netfilter workload
> >looks similar to one or more components of something like 
> 
> Thanks for the response. Anyway moments ago I found answer to the first
> question on vyatta forum here:
> http://www.vyatta.org/forum/viewtopic.php?t=823&sid=3cec85f2cf7866ca26040ec491415ee1

That information is out of date.
The bulk of the slow down in 64 bit was due to the fact that IOMMU
(CONFIG_DMAR) was enabled. The IOMMU requires programming DMA registers
on each packet which explains the extra overhead. After disabling
DMA remapping, the 64 bit kernel is faster because the compiler has
more registers and can generate better code.

Re: netfilter performance dependent on arch

[Marek Kierdelewicz]

>That information is out of date.
>The bulk of the slow down in 64 bit was due to the fact that IOMMU
>(CONFIG_DMAR) was enabled. The IOMMU requires programming DMA registers
>on each packet which explains the extra overhead. After disabling
>DMA remapping, the 64 bit kernel is faster because the compiler has
>more registers and can generate better code.

Thanks a lot. That's very valuable info.

Best regards,
Marek Kierdelewicz