why does netfilter make upload very slow? (was: Re: e1000 -> 82540EM on linux 2.6.0-test[45] very slow in one direction)

why does netfilter make upload very slow? (was: Re: e1000 -> 82540EM on linux 2.6.0-test[45] very slow in one direction)

[ookhoi]

Ookhoi wrote (ao):
# Ookhoi wrote (ao):
# > Florian Zwoch wrote (ao):
# > > issue seems to partly solved. the e1000 driver seems to be ok!
# > > i reconfigured my kernel and intentionally left out netfilter options. 
# > > after that my network performance was back to normal.
# > > 
# > > netfilter was only compiled in the kernel. it was not used with any rules!
# > > 
# > > so my wild guess would be that something with the netfilter code (i am 
# > > not 100% sure it was netfilter.. _maybe_ it was some small odd kernel 
# > > option i accidently enabled/disabled) is broken since test3 (again 
# > > uncertified. but i firstly noticed this switching from test3 to test4).
# 
# > I have netfilter enabled, and will try another -test6 kernel with
# > netfilter not compiled in to see if that indeed makes a difference.
# 
# I can confirm now that disabling netfilter in 2.6.0-test6 makes the nic
# perform oke wrt upload.
# I (just like Florian) had no iptables rules active in the former
# 2.6.0-test6 kernel, but netfilter was compiled in.

Would somebody like to explain why netfilter (in kernel, but not in use)
makes upload go very slow? I am by no means a network guru, but eager to
learn :-)

Re: why does netfilter make upload very slow? (was: Re: e1000 -> 82540EM on linux 2.6.0-test[45] very slow in one direction)

[David S. Miller]

On Wed, 8 Oct 2003 15:13:20 +0200
ookhoi@humilis.net wrote:

> Would somebody like to explain why netfilter (in kernel, but not in use)
> makes upload go very slow? I am by no means a network guru, but eager to
> learn :-)

It'll likely happen much quicker if you actually report this to
the netfilter lists, which is where the people who can help you
are paying attention.

Re: why does netfilter make upload very slow? (was: Re: e1000 -> 82540EM on linux 2.6.0-test[45] very slow in one direction)

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 1848 bytes --]

On Wed, Oct 08, 2003 at 03:13:20PM +0200, ookhoi@humilis.net wrote:
> # > I have netfilter enabled, and will try another -test6 kernel with
> # > netfilter not compiled in to see if that indeed makes a difference.
> # 
> # I can confirm now that disabling netfilter in 2.6.0-test6 makes the nic
> # perform oke wrt upload.
> # I (just like Florian) had no iptables rules active in the former
> # 2.6.0-test6 kernel, but netfilter was compiled in.
> 
> Would somebody like to explain why netfilter (in kernel, but not in use)
> makes upload go very slow? I am by no means a network guru, but eager to
> learn :-)

let's get this straight.  There are five possible cases

a) CONFIG_NETFILTER disabled.  you won't even have the netfilter hooks
   in the network stack (so certainly no netfilter-using modules loaded)
b) CONFIG_NETFILTER enabled, but _no_ modules (iptable_filter,
   ip_conntrack, ...) attached to the netfilter hook
c) CONFIG_NETFILTER enabled and iptable_filter.o (which pulls ip_tables.o)
   loaded, NO RULES in the table
d) CONFIG_NETFILTER enabled and iptable_filter.o (which pulls ip_tables.o)
   loaded, RULES in the table
e) CONFIG_NETFILTER enabled and ip_conntrack.o loaded, iptable_filter
   loaded or not, rules or not

So if you want to give us an idea about where the bottleneck might be,
please clearly indicate between which of the two cases you see this
performance penalty. 

This way we can isolate the culprit.

-- 
- Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: why does netfilter make upload very slow? (was: Re: e1000 -> 82540EM on linux 2.6.0-test[45] very slow in one direction)

[Florian Zwoch]

Harald Welte wrote:
>>Would somebody like to explain why netfilter (in kernel, but not in use)
>>makes upload go very slow? I am by no means a network guru, but eager to
>>learn :-)
> 
> 
> let's get this straight.  There are five possible cases
> 
> a) CONFIG_NETFILTER disabled.  you won't even have the netfilter hooks
>    in the network stack (so certainly no netfilter-using modules loaded)

no problem

> b) CONFIG_NETFILTER enabled, but _no_ modules (iptable_filter,
>    ip_conntrack, ...) attached to the netfilter hook

no problem

> c) CONFIG_NETFILTER enabled and iptable_filter.o (which pulls ip_tables.o)
>    loaded, NO RULES in the table

no problem

> d) CONFIG_NETFILTER enabled and iptable_filter.o (which pulls ip_tables.o)
>    loaded, RULES in the table

no problem (as long as i dont load any rules that require ip_conntrack)

> e) CONFIG_NETFILTER enabled and ip_conntrack.o loaded, iptable_filter
>    loaded or not, rules or not

*boink*

whenever i try to load ip_conntrack the nic performance drops from 5mb/s 
to 200k/s.

still using 2.6.0-test6.

regards,
Florian

Re: why does netfilter make upload very slow? (was: Re: e1000 -> 82540EM on linux 2.6.0-test[45] very slow in one direction)

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 2677 bytes --]

Hi Florian!

I'm Cc'ing all the mailinglists in order to keep them posted about the
question you've raised there.  All further discussion will move to
netfilter-devel, so for those interested: Please continue there.

On Wed, Oct 15, 2003 at 10:28:50AM +0200, Florian Zwoch wrote:
> >a) CONFIG_NETFILTER disabled.  you won't even have the netfilter hooks
> >   in the network stack (so certainly no netfilter-using modules loaded)
> no problem
> 
> >b) CONFIG_NETFILTER enabled, but _no_ modules (iptable_filter,
> >   ip_conntrack, ...) attached to the netfilter hook
> no problem
> 
> >c) CONFIG_NETFILTER enabled and iptable_filter.o (which pulls ip_tables.o)
> >   loaded, NO RULES in the table
> no problem
> 
> >d) CONFIG_NETFILTER enabled and iptable_filter.o (which pulls ip_tables.o)
> >   loaded, RULES in the table
> no problem (as long as i dont load any rules that require ip_conntrack)
> 
> >e) CONFIG_NETFILTER enabled and ip_conntrack.o loaded, iptable_filter
> >   loaded or not, rules or not
> *boink*

So It's clearly the connection tracking subsystem.  This is on one hand
good (because it means it's neither netfilter nor iptables).  

> whenever i try to load ip_conntrack the nic performance drops from 5mb/s 
> to 200k/s.

On the other hand, this is definitely way worse than you would expect.
Can you please tell me more information about:

- number of connections you have? (cat /proc/net/ip_conntrack | wc -l)
- number of buckets and ip_conntrack_max (printed at ip_conntrack
  loadtime
- your traffic pattern.  Are you spraying udp packets with random
  src/dst? What kind of connections (protocol, application) are you
  testing with?
- what about the hardware (cpu, memory, smp?)

Even the worst tests we've had so far (random UDP packets) 'only'
reduced the througput by about 50%.   Maybe we can do better than 50%
worst case behaviour, but you will always observe a visible impact as
soon as you start connection tracking for every single packet (which is
what 'insmod ip_conntrack' implies).

> still using 2.6.0-test6.

Have you observed this behaviour with other kernel versions?  Was there
a performance change between 2.4 and 2.6?  Or did you always observe
this grave performance loss?

> regards,
> Florian

-- 
- Harald Welte <laforge@netfilter.org>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]