filtering asym. routing without "ip_conntrack: table full"?

filtering asym. routing without "ip_conntrack: table full"?

[Christian Hammers]

Hello

I have a border router that does dynamic and asymetric routing.
Now, after upgrading from 2.4.19 to 2.4.20 yesterday I got the following 
message in my syslog twice this night:
	kernel: ip_conntrack: table full, dropping packet.
The /proc/net/ip_conntrack table has 36911 entries, mostly all [UNREPLIED].

I'm now wondering, why ip_conntrack comes into play at all - I'm
thought, I was using it only for my INPUT table but the connections in
/proc/net/ip_conntrack are definitely ones that are going from one ethX
to another ethX getting filtered only in FORWARD and only by source and 
destination IP.

/proc/sys/net/ipv4/ip_conntrack_max was at 32767 and has now temporarily 
raised to 65520 although I don't want to have conntrack for forwarding
at all. 
Oh, it's grown to 36911 while typing, so I be happy about answers :)

bye,

-christian-


# free
             total       used       free     shared    buffers cached
Mem:        516548     274016     242532          0      12500 128248
-/+ buffers/cache:     133268     383280

# lsmod 
Module                  Size  Used by    Not tainted
ipt_state                608   1  (autoclean)
ipt_LOG                 3200   2  (autoclean)
ip_nat_ftp              3424   0  (unused)
iptable_nat            19348   1  [ip_nat_ftp]
ip_conntrack_ftp        4096   1  [ip_nat_ftp]
iptable_filter          1760   1  (autoclean)
ip_tables              13184   6  [ipt_state ipt_LOG iptable_nat iptable_filter]
dummy                   1088   1 
eepro100               18444   3 
mii                     2320   0  [eepro100]
rtc                     6012   0  (autoclean)
unix                   13892  16  (autoclean)


relevant Kernel config:
#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_NETLINK_DEV=m
CONFIG_NETFILTER=y
CONFIG_NETFILTER_DEBUG=y
CONFIG_FILTER=y
CONFIG_UNIX=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_FWMARK=y
# CONFIG_IP_ROUTE_NAT is not set
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_TOS=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_IP_ROUTE_LARGE_TABLES=y
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
CONFIG_ARPD=y
# CONFIG_INET_ECN is not set
CONFIG_SYN_COOKIES=y

#
#   IP: Netfilter Configuration
#
CONFIG_IP_NF_CONNTRACK=y
CONFIG_IP_NF_FTP=m
CONFIG_IP_NF_IRC=m
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_LIMIT=m
CONFIG_IP_NF_MATCH_MAC=m
CONFIG_IP_NF_MATCH_PKTTYPE=m
CONFIG_IP_NF_MATCH_MARK=m
CONFIG_IP_NF_MATCH_MULTIPORT=m
CONFIG_IP_NF_MATCH_TOS=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_DSCP=m
# CONFIG_IP_NF_MATCH_AH_ESP is not set
CONFIG_IP_NF_MATCH_LENGTH=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_MATCH_TCPMSS=m
CONFIG_IP_NF_MATCH_HELPER=m
CONFIG_IP_NF_MATCH_STATE=m
CONFIG_IP_NF_MATCH_CONNTRACK=m
CONFIG_IP_NF_MATCH_UNCLEAN=m
CONFIG_IP_NF_MATCH_OWNER=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_MIRROR=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
# CONFIG_IP_NF_NAT_LOCAL is not set
CONFIG_IP_NF_NAT_SNMP_BASIC=m
CONFIG_IP_NF_NAT_IRC=m
CONFIG_IP_NF_NAT_FTP=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_TOS=m
# CONFIG_IP_NF_TARGET_ECN is not set
# CONFIG_IP_NF_TARGET_DSCP is not set
CONFIG_IP_NF_TARGET_MARK=m
CONFIG_IP_NF_TARGET_LOG=m
# CONFIG_IP_NF_TARGET_ULOG is not set
CONFIG_IP_NF_TARGET_TCPMSS=m
# CONFIG_IP_NF_ARPTABLES is not set
# CONFIG_IPV6 is not set
# CONFIG_KHTTPD is not set
# CONFIG_ATM is not set
# CONFIG_VLAN_8021Q is not set

#
#  
#
# CONFIG_IPX is not set
# CONFIG_ATALK is not set

#
# Appletalk devices
#
# CONFIG_DEV_APPLETALK is not set
# CONFIG_DECNET is not set
# CONFIG_BRIDGE is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_LLC is not set
# CONFIG_NET_DIVERT is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_NET_FASTROUTE is not set
# CONFIG_NET_HW_FLOWCONTROL is not set

#
# QoS and/or fair queueing
#
CONFIG_NET_SCHED=y
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_CSZ=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_QOS=y
CONFIG_NET_ESTIMATOR=y
CONFIG_NET_CLS=y
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
CONFIG_NET_CLS_POLICE=y

#
# Network testing
#
CONFIG_NET_PKTGEN=m

-- 
Christian Hammers             WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Lütticher Strasse 10     Tel 0241/701333-11
ch@westend.com                D-52064 Aachen              Fax 0241/911879

/proc/net/ip_conntrack filling without ipt_conntrack.o loaded?

[Christian Hammers]

Hello

I had ipt_conntrack.o loaded (see last mail) and then removed. But still
my /proc/net/ip_conntrack got filled up.
Then I did "echo '10000' > /proc/sys/net/ipv4/ip_conntrack_max" and it 
still raised.
Now, after waiting 10min or so the values are slightly falling (I had
fear that it crashed when reaching 0xffff)..

Are the first two events signs for a bug or is it expected behaviour
that somehow the conntrack code remains in the kernel even if the module
has been removed?

bye,

-christian-

-- 
Christian Hammers             WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Lütticher Strasse 10     Tel 0241/701333-11
ch@westend.com                D-52064 Aachen              Fax 0241/911879

Re: /proc/net/ip_conntrack filling without ipt_conntrack.o loaded?

[Filip Sneppe]

On Tue, 2003-01-14 at 13:12, Christian Hammers wrote:
> Hello
> 
> I had ipt_conntrack.o loaded (see last mail) and then removed. But still
> my /proc/net/ip_conntrack got filled up.
> Then I did "echo '10000' > /proc/sys/net/ipv4/ip_conntrack_max" and it 
> still raised.
> Now, after waiting 10min or so the values are slightly falling (I had
> fear that it crashed when reaching 0xffff)..
> 
> Are the first two events signs for a bug or is it expected behaviour
> that somehow the conntrack code remains in the kernel even if the module
> has been removed?

You sure it's not due to a typo ? It's ip_conntrack.o, not
ipt_conntrack. After an rmmod, what does lsmod say ?

About the high nuber of tracked connections, are you
talking about /proc/net/ip_conntrack ?

Before thinking of a bug, you should get a clear view of
the type of traffic filling your connection tracking table.
broadcasts ? Are these primarily ESTABLISHED connections,
or UNREPLIED connections ? Are nimda infected IIS boxes
scanning the whole ipv4 address range through your machine ?
It takes only a couple of infected machines to generate a
lot of traffic.

So, what's the nature of the entries in /proc/net/ip_conntrack ?

Regards,
Filip
 

Re: /proc/net/ip_conntrack filling without ipt_conntrack.o loaded?

[Christian Hammers]

Hello

On Tue, Jan 14, 2003 at 02:43:45PM +0100, Filip Sneppe wrote:
> On Tue, 2003-01-14 at 13:12, Christian Hammers wrote:
> > I had ipt_conntrack.o loaded (see last mail) and then removed. But still
> > my /proc/net/ip_conntrack got filled up.
> > Then I did "echo '10000' > /proc/sys/net/ipv4/ip_conntrack_max" and it 
> > still raised.
> > Now, after waiting 10min or so the values are slightly falling (I had
> > fear that it crashed when reaching 0xffff)..
> > 
> > Are the first two events signs for a bug or is it expected behaviour
> > that somehow the conntrack code remains in the kernel even if the module
> > has been removed?
> 
> You sure it's not due to a typo ? It's ip_conntrack.o, not
> ipt_conntrack. After an rmmod, what does lsmod say ?
Ok, that was just a typo while writing the mail. I always checked with
lsmod as user root. Currently:

/home/ch# lsmod
Module                  Size  Used by    Not tainted
ipt_LOG                 3200   2  (autoclean)
iptable_filter          1760   1  (autoclean)
ip_tables              13184   2  [ipt_LOG iptable_filter]
dummy                   1088   1 
eepro100               18444   3 
mii                     2320   0  [eepro100]
unix                   13892  14  (autoclean)

> About the high nuber of tracked connections, are you
> talking about /proc/net/ip_conntrack ?
Yes. As wrote in my previous mail (should have written it here, too),
this router does asymetric routing, i.e. the packets for a connection
come in over it but the answer packets go out via another router. 
So it will almost never see a real 3way tcp handshake or the like.

Fitting to this explanation, the kind of traffic in the
/proc/net/ip_conntrack is 99.9% "UNREPLIED" but apart from that
absolutely normal traffic (mostly port 80 and usual IPs).

> Regards,
> Filip
bye,

-christian-

-- 
Christian Hammers             WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Lütticher Strasse 10     Tel 0241/701333-11
ch@westend.com                D-52064 Aachen              Fax 0241/911879

Re: /proc/net/ip_conntrack filling without ipt_conntrack.o loaded?

[Filip Sneppe]

On Tue, 2003-01-14 at 16:06, Christian Hammers wrote:
> 
> On Tue, Jan 14, 2003 at 02:43:45PM +0100, Filip Sneppe wrote: 
> > About the high nuber of tracked connections, are you
> > talking about /proc/net/ip_conntrack ?
> Yes. As wrote in my previous mail (should have written it here, too),

Ah - that mail - I already deleted it from my mailbox.

> this router does asymetric routing, i.e. the packets for a connection
> come in over it but the answer packets go out via another router. 
> So it will almost never see a real 3way tcp handshake or the like.
> 

Wel, connection tracking was not really designed to handle asymetric
routing setups, so you're basically screwed. No stateful
packet filtering firewall will handle this decently, I guess.

On thing you can do, is apply the 'raw' patch from patch-o-matic
(written by Jozsef Kadlecsik), this allows you not to track
particular traffic. In your case, you will need to specify
rules for all (asymetric) traffic that should not be tracked by 
your firewall. If *all* your traffic is essentially asymetric in
nature, you'de be better off not using ip_conntrack at all...

Hope this helps...

Regards,
Filip

Re: /proc/net/ip_conntrack filling without ipt_conntrack.o loaded?

[Christian Hammers]

On Tue, Jan 14, 2003 at 04:49:14PM +0100, Filip Sneppe wrote:
> If *all* your traffic is essentially asymetric in
> nature, you'de be better off not using ip_conntrack at all...

Yes, thought so, too. - The question that I was trying to ask in this
thread was, why the /proc/net/ip_conntrack is filled by the kernel
although I *already did* remove the module!

I would have guessed that just after I removed the ipt_conntrack module
and all the sub modules (ipt_conntrack_ftp, nat etc) the
/proc/net/ip_conntrack would either vanish or at least return nothing
because the code at the other end of the virtual device has gone.

But apparently it did not go away so I suspected a kernel function that 
was fogotten to free or similar...

bye,

-christian-


-- 
Christian Hammers             WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Lütticher Strasse 10     Tel 0241/701333-11
ch@westend.com                D-52064 Aachen              Fax 0241/911879

Re: /proc/net/ip_conntrack filling without ipt_conntrack.o loaded?

[Filip Sneppe]

On Tue, 2003-01-14 at 17:01, Christian Hammers wrote:
> Yes, thought so, too. - The question that I was trying to ask in this
> thread was, why the /proc/net/ip_conntrack is filled by the kernel
> although I *already did* remove the module!

Well that's not supposed to happen :-)
What kernel version are you running ? modutils version ?
Is this reproducable upon every reboot ?
I am no expert on this, but part of the reason why
Rusty is rewriting the modules infrastructure in 2.5
is that module loading/unloading is inherently racy iirc.
You may have hit a race condition with one particular
chain of events.
 
> I would have guessed that just after I removed the ipt_conntrack module
> and all the sub modules (ipt_conntrack_ftp, nat etc) the
> /proc/net/ip_conntrack would either vanish or at least return nothing
> because the code at the other end of the virtual device has gone.

I've been giving it a few tries on my machine, and 
ip_conntrack disappears nicely from /proc/net upon
unloads/reloads of ip_conntrack, even
with unreplied connections pending.

Have you already rebooted the box (this is no
Windows-advise - if something went wrong with
the module unload, there really isn't much 
other choice :-) ) ?

Re: /proc/net/ip_conntrack filling without ipt_conntrack.o loaded?

[Christian Hammers]

On Tue, Jan 14, 2003 at 05:09:53PM +0100, Filip Sneppe wrote:
> On Tue, 2003-01-14 at 17:01, Christian Hammers wrote:
> > Yes, thought so, too. - The question that I was trying to ask in this
> > thread was, why the /proc/net/ip_conntrack is filled by the kernel
> > although I *already did* remove the module!
> 
> What kernel version are you running ? modutils version ?
Kernel-2.4.20. modprobe-2.4.15. Debian 3.0 woody distribution.

> Is this reproducable upon every reboot ?
I'm not allowed to reboot it :-) But it's still reproducible that 
after decreasing with about 1000 per minute the value of
/proc/net/ip_conntrack has now stabilized around the
/proc/sys/net/ipv4/ipt_conntrack_max value which is currently 10000 
(was 65520 and filled up to ca. 50000)

> I've been giving it a few tries on my machine, and 
> ip_conntrack disappears nicely from /proc/net upon
> unloads/reloads of ip_conntrack, even
> with unreplied connections pending.
Hmm :) 
Maybe you should set your machine unter a load of at least 4mbit/s 
with random IPs. This was the amount of traffic my router had when I
reloaded the firewall rule script with a "rmmod" at the beginning.

bye,

-christian-

-- 
Christian Hammers             WESTEND GmbH  |  Internet-Business-Provider
Technik                       CISCO Systems Partner - Authorized Reseller
                              Lütticher Strasse 10     Tel 0241/701333-11
ch@westend.com                D-52064 Aachen              Fax 0241/911879

Re: /proc/net/ip_conntrack filling without ipt_conntrack.o loaded?

[Filip Sneppe]

On Tue, 2003-01-14 at 17:37, Christian Hammers wrote:

> Kernel-2.4.20. modprobe-2.4.15. Debian 3.0 woody distribution.
>
> > Is this reproducable upon every reboot ?
> I'm not allowed to reboot it :-) But it's still reproducible that 
> after decreasing with about 1000 per minute the value of
> /proc/net/ip_conntrack has now stabilized around the
> /proc/sys/net/ipv4/ipt_conntrack_max value which is currently 10000 
> (was 65520 and filled up to ca. 50000)

Since you haven't rebooted it, you will continue to have this
problem as basically your running kernel+ip_conntrack is
basically screwed until you reboot the box. 

> Hmm :) 
> Maybe you should set your machine unter a load of at least 4mbit/s 
> with random IPs. This was the amount of traffic my router had when I
> reloaded the firewall rule script with a "rmmod" at the beginning.

I think you're absolutely right that by unloading ip_conntrack
while the box is handling packets gives you a greater chance
of triggering a problem. I remember having this kind of problem
occasionally with NIC drivers and ip_conntrack_ftp.

See also this item on the netfilter TODO list:

TO BE INVESTIGATED:
[...]
- ip_conntrack rmmod loop (sometimes, Yan's patch?)
   
Could be your problem, couldn't it ?
Not a lot I can say to help you any further though...
If you can reproduce this, you may inform netfilter-devel
of your workload and test scenario, which could help
developers. Is your firewall an SMP (multiprocessor)
machine, by any chance ?

Regards,
Filip

Re: filtering asym. routing without "ip_conntrack: table full"?

[ard-netfilter]

On Tue, Jan 14, 2003 at 10:37:11AM +0100, Christian Hammers wrote:
> I have a border router that does dynamic and asymetric routing.
> Now, after upgrading from 2.4.19 to 2.4.20 yesterday I got the following 
> message in my syslog twice this night:
> 	kernel: ip_conntrack: table full, dropping packet.
> The /proc/net/ip_conntrack table has 36911 entries, mostly all [UNREPLIED].
Heh,
Next to the other replies:
If you do massive routing, or better: massive firewalling (a lot
of connections going through), always load the ip_conntrack
module with hashsize= .
If you don't, most of the connections have to be sequentially
searched in a linked list.
Default max setting of hashsize is 8192, with a maximum of 58000
connections being tracked. The maximum connections to be tracked
can be increased on the fly, but upping your hashsize to begin
with gives you certainly an extra performance boost.
(Heh, it can make your cpu system time go from 100% down to 5 or
so... At least it will make your ethernet driver be the bottle
neck)

-- 
mail          up   65+19:29,     4 users,  load 0.00, 0.02, 0.27
mistar1       up   18+15:59,     9 users,  load 0.00, 0.00, 0.01
Let your government know you value your freedom: sign the petition:
http://petition.eurolinux.org

Re: filtering asym. routing without "ip_conntrack: table full"?

[Jakub Jakacki]

On Tue, Jan 21, 2003 at 07:16:15AM +0100, Ard van Breemen wrote:
> On Tue, Jan 14, 2003 at 10:37:11AM +0100, Christian Hammers wrote:
> > I have a border router that does dynamic and asymetric routing.
> > Now, after upgrading from 2.4.19 to 2.4.20 yesterday I got the following 
> > message in my syslog twice this night:
> > 	kernel: ip_conntrack: table full, dropping packet.
> > The /proc/net/ip_conntrack table has 36911 entries, mostly all [UNREPLIED].
> Heh,
> Next to the other replies:
> If you do massive routing, or better: massive firewalling (a lot
> of connections going through), always load the ip_conntrack
> module with hashsize= .
> If you don't, most of the connections have to be sequentially
> searched in a linked list.
> Default max setting of hashsize is 8192, with a maximum of 58000
> connections being tracked. The maximum connections to be tracked
> can be increased on the fly, but upping your hashsize to begin
> with gives you certainly an extra performance boost.
> (Heh, it can make your cpu system time go from 100% down to 5 or
> so... At least it will make your ethernet driver be the bottle
> neck)
> 
> -- 
> mail          up   65+19:29,     4 users,  load 0.00, 0.02, 0.27
> mistar1       up   18+15:59,     9 users,  load 0.00, 0.00, 0.01
> Let your government know you value your freedom: sign the petition:
> http://petition.eurolinux.org
> 
> 
> 

I have the same problem, and have found /proc/sys/net/ip_conntrack_max.
Is it contains the default max hashsize? May I only write:

cat 16384 > /proc/sys/net/ip_conntract_max

to solve the problem of "full table"?

Will it be the same as loading ip_conntrack module with hashsize= ?
Best regards
Jakub

Re: filtering asym. routing without "ip_conntrack: table full"?

[Arnt Karlsen]

On Tue, 21 Jan 2003 11:45:05 +0100, 
Jakub Jakacki <admin@md4.pl> wrote in message 
<20030121104505.GD603@mispa>:

> I have the same problem, and have found
> /proc/sys/net/ip_conntrack_max.
>  Is it contains the default max hashsize? May I only write:
> 
>  cat 16384 > /proc/sys/net/ip_conntract_max

..you meant 'echo "16384" > /proc/sys/net/ip_conntract_max' ?

>  to solve the problem of "full table"?


-- 
..med vennlig hilsen = with Kind Regards from Arnt... ;-)
...with a number of polar bear hunters in his ancestry...
  Scenarios always come in sets of three: 
  best case, worst case, and just in case.