Interesting problems

Interesting problems

[Peteris Krumins]

Hello,

 I am experiencing some trouble w/ linux-2.4.22-pre10-ac1 and
 iptables 1.2.8 w/ patch-o-matic-20030802.


 The problem is like this:

  - there is a computer PII 266, now with 192mb ram, it's running
    custom linux distribution with 1 user process - getty, as I log
    in, a bash is spawned.

    Only ESTABLISHED,RELATED outbound connections are allowed, NEW and
    INVALID inbound are REJECTed.

    All the traffic is routed via this computer - 8Mbit link
    constant, - lots of users more than 300 - everyone does something
    different.

    I set up max. 20000 conntrack entries unfortunately
    after a day there already are already 15000 connection tracking entires
    and the load of the computer each 5 minutes jumps up to 2 then goes
    down to 0.3 and jumps to 2 again etc., 15 minute avg load keeps at
    0.7. But unfortunately entries are growing and growing, more ram
    is used, computer load grows and suddenly any traffic routed via
    this computer is killed, nothing flows through.
    Kernel shouts that conntrack table is full but a
    `wc -l /proc/net/ip_conntrack' shows around 20000 entries.
    A reboot helps.
     
    If i increase it even more (50000), the load gets so high that
    establishing a new connection takes more than a second and then
    the box dies anyway with the same error that conntrack table is
    full.

    The question is what could i do to avoid the growthy of connection
    tracking entries? I guess the entries taking up space are already
    established tcp connections which were not properly terminated, so
    they are there to timeout.

    I noticed exactly the same on 2.4.22-pre6 and pom20030714
    
    An ascii for network setup is:
                         ____
           .---.   .---./    \
INTERNET---| 1 |---| 2 |-------corporate network
           `---'   `---'\____/

1 - my box
 (ESTAB., RELATED allowed to internet everything else denied)
2 - traffic accounting/shaping and main router
 (much much faster box than box 1)


P.Krumins

Re: Interesting problems

[Chris Wilson]

Hi Peteris,

>   - there is a computer PII 266, now with 192mb ram, it's running
>     custom linux distribution with 1 user process - getty, as I log
>     in, a bash is spawned.
> 
>     Only ESTABLISHED,RELATED outbound connections are allowed, NEW and
>     INVALID inbound are REJECTed.
> 
>     All the traffic is routed via this computer - 8Mbit link
>     constant, - lots of users more than 300 - everyone does something
>     different.
> 
>     I set up max. 20000 conntrack entries unfortunately
>     after a day there already are already 15000 connection tracking entires
>     and the load of the computer each 5 minutes jumps up to 2 then goes
>     down to 0.3 and jumps to 2 again etc., 15 minute avg load keeps at
>     0.7. But unfortunately entries are growing and growing, more ram
>     is used, computer load grows and suddenly any traffic routed via
>     this computer is killed, nothing flows through.
>     Kernel shouts that conntrack table is full but a
>     `wc -l /proc/net/ip_conntrack' shows around 20000 entries.
>     A reboot helps.

This does not surprise me at all. You have a very old computer handling an 
enormous amount of traffic and you complain that it's slow? =)

For reference, we have a 350MHz Cyrix machine handling iptables with
stateful inspection for a medium-loaded 2Mb line with maybe 50 users/boxes
behind it, and it has about 3000 connections right now and a load of 0.04.  
50% of CPU time is used by the System, indicating netfilter.  Perhaps your
users are very busy?

When `wc -l /proc/net/ip_conntrack' shows around 20000 entries, and you 
have set the max to 20000, then no more connections can be added and the 
kernel prints the warning you saw.

>     The question is what could i do to avoid the growthy of connection
>     tracking entries? I guess the entries taking up space are already
>     established tcp connections which were not properly terminated, so
>     they are there to timeout.

Terminate some of your users, or tell them not to use the Internet so 
much! But a better solution would be to get a real powerful box or not use 
stateful inspection (or maybe run nf-hipac or BSD's ipf instead of 
iptables, as apparently both are faster).

Cheers, Chris.
-- 
   ___ __     _
 / __// / ,__(_)_  | Chris Wilson -- UNIX Firewall Lead Developer |
/ (_ / ,\/ _/ /_ \ | NetServers.co.uk http://www.netservers.co.uk |
\ _//_/_/_//_/___/ | 21 Signet Court, Cambridge, UK. 01223 576516 |

Re: Interesting problems

[tsh]

>For reference, we have a 350MHz Cyrix machine handling iptables with
>stateful inspection for a medium-loaded 2Mb line with maybe 50 users/boxes
>behind it, and it has about 3000 connections right now and a load of 0.04.
>50% of CPU time is used by the System, indicating netfilter.  Perhaps your
>users are very busy?

Chris, I'm interested in why your 50 users generate 3000 connections.
We have about 1200 machines behind a stateful iptables box which has a 1GHz
cpu with 128Mb mem. Of these 1200, probably < two-thirds ever connect
to the outside world, (e.g. right now wc -l '/proc/net/ip_conntrack'
shows 1221 entries). I had no idea how to estimate the potential
load on such a box, but this box also does NAT and some routing
(we're gradually moving ourselves to private ip space) and I've
never seen it even blink in terms of load. I run an idle process
(a loopstop) at 'nice' 19, and top always shows this at 99% cpu.
We have a 100Mb link to Janet and regularly achieve wire-speed
transfers for things like ftp between us and fast nearby sites,
and again, cpu-load is essentially zero during these.

I realise that different iptables activities will present different
cpu loads (packet rate, number of entries in the tables, number of
active connections etc) but do you have any feel for which of these
is the most costly?

Cheers,
Terry



Terry Horsnell (tsh@mrc-lmb.cam.ac.uk)
I.T. Manager
Medical Research Council
Lab of Molecular Biology
Hills Road
CAMBRIDGE CB2 2QH
U.K.
Phone:	+44 (0)1223 248011
Fax:	+44 (0)1223 213556

Re: Interesting problems

[Chris Wilson]

Hi Terry,

> Chris, I'm interested in why your 50 users generate 3000 connections.

We have a number of customers' servers hosted here, about 40 boxes. There 
are probably about 30-40 desktop users as well, and one customer with 
maybe 10 servers of his own. Probably the servers are responsible for most 
of the connections, but I don't have an actual breakdown.

> I realise that different iptables activities will present different
> cpu loads (packet rate, number of entries in the tables, number of
> active connections etc) but do you have any feel for which of these
> is the most costly?

I think searching the connections table is the most costly thing. The 
Slammer worm brought some of our firewalls to their knees with IRQ load 
(crappy Realtek network cards) and connection lookups. Reducing the number 
of conntracks from 65,000 to 4096 made the systems usable again. But 
that's just my own experience, I suspect that the real Netfilter 
developers have a much better idea than me where the bottlenecks are.

Cheers, Chris.
-- 
   ___ __     _
 / __// / ,__(_)_  | Chris Wilson -- UNIX Firewall Lead Developer |
/ (_ / ,\/ _/ /_ \ | NetServers.co.uk http://www.netservers.co.uk |
\ _//_/_/_//_/___/ | 21 Signet Court, Cambridge, UK. 01223 576516 |

Re[2]: Interesting problems

[Peteris Krumins]

Tuesday, August 5, 2003, 12:38:13 PM, Chris Wilson wrote:

CW> Hi Peteris,

>>   - there is a computer PII 266, now with 192mb ram, it's running
[...]
>>     `wc -l /proc/net/ip_conntrack' shows around 20000 entries.
>>     A reboot helps.

CW> This does not surprise me at all. You have a very old computer handling an 
CW> enormous amount of traffic and you complain that it's slow? =)

266 isnt that slow, that's only connection tracking, is it (too slow)?

CW> For reference, we have a 350MHz Cyrix machine handling iptables with
CW> stateful inspection for a medium-loaded 2Mb line with maybe 50 users/boxes
CW> behind it, and it has about 3000 connections right now and a load of 0.04.  
CW> 50% of CPU time is used by the System, indicating netfilter.  Perhaps your
CW> users are very busy?

Simple internet users, some are lazy and just downloading stuff from
different edonkies and emules. Some just use net for google.

Most of the entries taking up conntrack table are:
tcp 6 419547 ESTABLISHED src=X dst=Y sport=A dport=B [UNREPLIED] src=Y
dst=X sport=B dport=A use=1 mark=0

They would expire only after 419547 secs that is 4 days, but during those 4
days the table grows 5 times that much entries..

Is there some trick to kill UNREPLIED tcp conntrack entires if there
is no reply after NN seconds.
Also could it be done in software w/o much trouble? Monitoring
conntrack table and removing dead entries like these.
If it's not much trouble i can instruct our programmers instantly to
work on such tool.

CW> When `wc -l /proc/net/ip_conntrack' shows around 20000 entries, and you 
CW> have set the max to 20000, then no more connections can be added and the 
CW> kernel prints the warning you saw.

Yes, i know.

>>     The question is what could i do to avoid the growthy of connection
>>     tracking entries? I guess the entries taking up space are already
>>     established tcp connections which were not properly terminated, so
>>     they are there to timeout.

CW> Terminate some of your users, or tell them not to use the Internet so 
CW> much!

:)

CW> But a better solution would be to get a real powerful box or not use
CW> stateful inspection (or maybe run nf-hipac or BSD's ipf instead of 
CW> iptables, as apparently both are faster).

Stateful inspection is needed to allow only outbound connections. and
inbound only ESTABLISHED,RELATED.

Unfortunately the software using iptables and Linux is being programmed
for half a year. 3 months left to finish the product. Can't change the
platform.

I can instruct our programmers (I am the team leader) to do some work
on iptables connection tracking code and after improvments i could
send it as a patch for current iptables. So connection tracking would
be as efficient as possible. I just need someone to tell me exactly
what is inefficient in conntrack code.


P.Krumins

Re[2]: Interesting problems

[Chris Wilson]

Hi Peteris,

> 266 isnt that slow, that's only connection tracking, is it (too slow)?

For 20000 connections, I'd say it was too slow.

> Most of the entries taking up conntrack table are:
> tcp 6 419547 ESTABLISHED src=X dst=Y sport=A dport=B [UNREPLIED] src=Y
> dst=X sport=B dport=A use=1 mark=0
> 
> They would expire only after 419547 secs that is 4 days, but during those 4
> days the table grows 5 times that much entries..
> 
> Is there some trick to kill UNREPLIED tcp conntrack entires if there
> is no reply after NN seconds.

In theory they should be killed automatically to make room for real 
(ESTABLISHED) connections. So you may be able to reduce the size of the 
conntrack table without impacting user experience, and make your system 
run faster that way.

You could also try the patch that Daniel Chemko suggested,

> Also could it be done in software w/o much trouble? Monitoring
> conntrack table and removing dead entries like these.
> If it's not much trouble i can instruct our programmers instantly to
> work on such tool.

I believe that Harald is working on nfnetlink/ctnetlink which will allow 
you to do this, but I think it's still experimental. 

> I can instruct our programmers (I am the team leader) to do some work
> on iptables connection tracking code and after improvments i could
> send it as a patch for current iptables. So connection tracking would
> be as efficient as possible. I just need someone to tell me exactly
> what is inefficient in conntrack code.

Sorry, I'm really not the best person to ask about that, but Harald and
Rusty have some profiling and optimisations on their TODO lists which they
might be willing to hand off to your team.

Cheers, Chris.
-- 
   ___ __     _
 / __// / ,__(_)_  | Chris Wilson -- UNIX Firewall Lead Developer |
/ (_ / ,\/ _/ /_ \ | NetServers.co.uk http://www.netservers.co.uk |
\ _//_/_/_//_/___/ | 21 Signet Court, Cambridge, UK. 01223 576516 |