fedora 14 kernel performance with ip forwarding workload

fedora 14 kernel performance with ip forwarding workload

[Jesse Brandeburg]

The other day I was running the stock fedora kernel on my ip
forwarding setup, to see what the performance was, and the performance
wasn't very good.

system is S5520HC dual socket 2.93GHz Xeon 5570 (Nehalem) with 3 quad
port 82580 adapters (12 ports).  Traffic is bidirectional 64 byte
packets being forwarded and received on each port, basically port to
port routing.  I am only using 12 flows currently.

The driver is igb, and I am using an affinity script that lines up
each pair of ports that are forwarding traffic into optimal
configurations for cache locality.  I am also disabling
remote_node_defrag_ratio to stop cross node traffic.

With the fedora default kernel from F14 it appears that
CONFIG_NETFILTER=y means that I cannot unload all of netfilter even if
I stop iptables service.

perf showed netfilter being prominent, and removing it gives me much
higher throughput.  Is there a reason CONFIG_NETFILTER=y ?  Isn't it a
good thing to be able to disable netfilter if you want to?

Jesse

Re: fedora 14 kernel performance with ip forwarding workload

[David Miller]

From: Jesse Brandeburg <jesse.brandeburg@gmail.com>
Date: Wed, 6 Apr 2011 11:51:06 -0700

> perf showed netfilter being prominent, and removing it gives me much
> higher throughput.  Is there a reason CONFIG_NETFILTER=y ?  Isn't it a
> good thing to be able to disable netfilter if you want to?

CONFIG_NETFILTER=y is the only possible non-disabled setting, as it's
a boolean.

All that is enabling is the hooks, and it's not possible to provide
netfilter at all without the hooks being compiled into the kernel
image.

Fedora by default is also probably installing several defauly
netfilter rules, make sure to remove them.

Re: fedora 14 kernel performance with ip forwarding workload

[Arnaldo Carvalho de Melo]

Em Wed, Apr 06, 2011 at 12:12:08PM -0700, David Miller escreveu:
> From: Jesse Brandeburg <jesse.brandeburg@gmail.com>
> Date: Wed, 6 Apr 2011 11:51:06 -0700
> > perf showed netfilter being prominent, and removing it gives me much
> > higher throughput.  Is there a reason CONFIG_NETFILTER=y ?  Isn't it a
> > good thing to be able to disable netfilter if you want to?
> 
> CONFIG_NETFILTER=y is the only possible non-disabled setting, as it's
> a boolean.
> 
> All that is enabling is the hooks, and it's not possible to provide
> netfilter at all without the hooks being compiled into the kernel
> image.
> 
> Fedora by default is also probably installing several defauly
> netfilter rules, make sure to remove them.

Was there ever any attempt to try to have netfilter built-in but not
causing the performance impact?

Something like ftrace code changing when the user inserts the first
rule?

People wanting top performance disable it in the build, but thos wanting
to stick to vendor provided kernels don't have that choice :)

- Arnaldo

Re: fedora 14 kernel performance with ip forwarding workload

[David Miller]

From: Arnaldo Carvalho de Melo <acme@ghostprotocols.net>
Date: Wed, 6 Apr 2011 16:57:19 -0300

> Something like ftrace code changing when the user inserts the first
> rule?
> 
> People wanting top performance disable it in the build, but thos wanting
> to stick to vendor provided kernels don't have that choice :)

Using ftrace-like stubs would be an interesting idea, and I highly encourage
people to work on something like that.

However I want to reiterate that I think that real rules are installed
in Jesse's case, and once he removes those the majority of the
overhead will disappear.  The FC14 workstation I'm using right now, on
which I've made no modifications to the installer's netfilter settings,
has the following rules:

--------------------
[root@ilbolle davem]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED 
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh 
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:ipp 
ACCEPT     udp  --  anywhere             224.0.0.251         state NEW udp dpt:mdns 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ipp 
ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:ipp 
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited 

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited 

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
[root@ilbolle davem]# 
--------------------

I suspect Jesse has something similar on his test box.

When no rules are loaded, all the stubs make happen is a function call
plus a list_empty() check.  Nothing more.  I really can't see that, all
by itself, obliterating routing performance.

In fact I've done udp flood tests, as recently as a month ago, with just
NETFILTER=y and no rules installed, and the impact was minimal.

And that was on sparc64 where function calls are expensive :)

Re: fedora 14 kernel performance with ip forwarding workload

[Arnaldo Carvalho de Melo]

Em Wed, Apr 06, 2011 at 01:02:39PM -0700, David Miller escreveu:
> From: Arnaldo Carvalho de Melo <acme-f8uhVLnGfZaxAyOMLChx1axOck334EZe@public.gmane.org>
> Date: Wed, 6 Apr 2011 16:57:19 -0300
> 
> > Something like ftrace code changing when the user inserts the first
> > rule?
> > 
> > People wanting top performance disable it in the build, but thos wanting
> > to stick to vendor provided kernels don't have that choice :)
> 
> Using ftrace-like stubs would be an interesting idea, and I highly encourage
> people to work on something like that.

cool, these code modification and JIT mechanizms open up a lot of
possibilities indeed ;-)
 
> However I want to reiterate that I think that real rules are installed
> in Jesse's case, and once he removes those the majority of the
> overhead will disappear.  The FC14 workstation I'm using right now, on
>
> which I've made no modifications to the installer's netfilter settings,
> has the following rules:

<SNIP>
 
> I suspect Jesse has something similar on his test box.
> 
> When no rules are loaded, all the stubs make happen is a function call
> plus a list_empty() check.  Nothing more.  I really can't see that, all
> by itself, obliterating routing performance.

Yeah, would be nice, since he is playing with it, for him to post
numbers about the overheads.
 
> In fact I've done udp flood tests, as recently as a month ago, with just
> NETFILTER=y and no rules installed, and the impact was minimal.
> 
> And that was on sparc64 where function calls are expensive :)

:-)

- Arnaldo

Re: fedora 14 kernel performance with ip forwarding workload

[Eric Dumazet]

Le mercredi 06 avril 2011 à 13:02 -0700, David Miller a écrit :
> From: Arnaldo Carvalho de Melo <acme@ghostprotocols.net>
> Date: Wed, 6 Apr 2011 16:57:19 -0300
> 
> > Something like ftrace code changing when the user inserts the first
> > rule?
> > 
> > People wanting top performance disable it in the build, but thos wanting
> > to stick to vendor provided kernels don't have that choice :)
> 
> Using ftrace-like stubs would be an interesting idea, and I highly encourage
> people to work on something like that.
> 
> However I want to reiterate that I think that real rules are installed
> in Jesse's case, and once he removes those the majority of the
> overhead will disappear.  The FC14 workstation I'm using right now, on
> which I've made no modifications to the installer's netfilter settings,
> has the following rules:
> 
> --------------------
> [root@ilbolle davem]# iptables -L
> Chain INPUT (policy ACCEPT)
> target     prot opt source               destination         
> ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED 
> ACCEPT     icmp --  anywhere             anywhere            
> ACCEPT     all  --  anywhere             anywhere            
> ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh 
> ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:ipp 
> ACCEPT     udp  --  anywhere             224.0.0.251         state NEW udp dpt:mdns 
> ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ipp 
> ACCEPT     udp  --  anywhere             anywhere            state NEW udp dpt:ipp 
> REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited 
> 
> Chain FORWARD (policy ACCEPT)
> target     prot opt source               destination         
> REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited 
> 
> Chain OUTPUT (policy ACCEPT)
> target     prot opt source               destination         
> [root@ilbolle davem]# 
> --------------------
> 
> I suspect Jesse has something similar on his test box.
> 

I suspect problem is worse than that.

I remember last time I work on a fedora kernel, it had conntrack enabled

And yes, conntrack can really slowdown a router, because of default
parameters.

cat /proc/sys/net/nf_conntrack_max

Re: fedora 14 kernel performance with ip forwarding workload

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Wed, 06 Apr 2011 22:18:56 +0200

> I remember last time I work on a fedora kernel, it had conntrack enabled
> 
> And yes, conntrack can really slowdown a router, because of default
> parameters.

Yes, if conntrack is enabled performance will indeed tank.

Re: fedora 14 kernel performance with ip forwarding workload

[Eric Dumazet]

Le mercredi 06 avril 2011 à 13:29 -0700, David Miller a écrit :
> From: Eric Dumazet <eric.dumazet@gmail.com>
> Date: Wed, 06 Apr 2011 22:18:56 +0200
> 
> > I remember last time I work on a fedora kernel, it had conntrack enabled
> > 
> > And yes, conntrack can really slowdown a router, because of default
> > parameters.
> 
> Yes, if conntrack is enabled performance will indeed tank.

I just check on latest fedora14 (yum updated) :

- conntrack is statically included in kernel, you cannot remove it.


Hmm...

Re: fedora 14 kernel performance with ip forwarding workload

[Brandeburg, Jesse]

On Wed, 6 Apr 2011, Eric Dumazet wrote:

> Le mercredi 06 avril 2011 à 13:29 -0700, David Miller a écrit :
> > From: Eric Dumazet <eric.dumazet@gmail.com>
> > Date: Wed, 06 Apr 2011 22:18:56 +0200
> > 
> > > I remember last time I work on a fedora kernel, it had conntrack enabled
> > > 
> > > And yes, conntrack can really slowdown a router, because of default
> > > parameters.
> > 
> > Yes, if conntrack is enabled performance will indeed tank.
> 
> I just check on latest fedora14 (yum updated) :
> 
> - conntrack is statically included in kernel, you cannot remove it.
> 
> 
> Hmm...

Thanks for the replies, 
Yes indeed, I bet that conntrack was the item that caused the grief I saw.  
In fact turning off CONFIG_NETFILTER disables conntrack and solved my 
problem. :-)

Now the question is how to get the netfiltery/conntracky goodness without 
impacting those who want to go fast, maybe without a helmet.

Re: fedora 14 kernel performance with ip forwarding workload

[Eric Dumazet]

Le mercredi 06 avril 2011 à 14:08 -0700, Brandeburg, Jesse a écrit :
> Thanks for the replies, 
> Yes indeed, I bet that conntrack was the item that caused the grief I saw.  
> In fact turning off CONFIG_NETFILTER disables conntrack and solved my 
> problem. :-)
> 
> Now the question is how to get the netfiltery/conntracky goodness without 
> impacting those who want to go fast, maybe without a helmet.

Thats pretty simple, just load netfilter modules if necessary.

If not loaded, cost is pretty small, being predicted branches.