System with really big statetable crashing while doing "cat /proc/net/ip_conntrack | wc -l"

System with really big statetable crashing while doing "cat /proc/net/ip_conntrack | wc -l"

[Jens Hektor]

-----BEGIN PGP SIGNED MESSAGE-----

Hi,

we are running a production packetfilter based on Fedora Core-1
actual vanilla  2.4.22-1.2179.nptl-smp.

This packetfilter is in front of two class-B's and it has a big
statetable, really. We guess that it's about 200k entries.

So we only can guess for if doing

	cat /proc/net/ip_conntrack | wc -l

the system reproducably crashes.

We had the same effect on non-smp and 2.6 version of Fedora kernels
and I guess that the same might happen with the vanilla kernel.

Here is what the kernel oops tells:

- ------------------------------------------------------------------------
EIP is at print_expect [ip_conntrack] 0x14 (kernel-name)
[two lines with registers: varying values]
ds: 0068    es:0068     ss:0068
[Process is always "cat", pid, stackpage & stack varying]
Calltrace:  [<f8a3c442>] list_conntracks [ip_conntrack] 0x122 (varying_#)
[<varying_#>] proc_file_read [kernel] varying_#
[<varying_#>] sys_read [kernel] varying_#
[<varying_#>] system_call [kernel] varying_#
- ------------------------------------------------------------------------

Is this known, has anyone ever seen this?

Best regards, Jens Hektor


- --
Jens Hektor, RWTH Aachen, Rechenzentrum, Seffenter Weg 23, 52074 Aachen
Center for Computing and Communication, RWTH Aachen University, Germany
mailto:hektor@RZ.RWTH-Aachen.DE, Tel.: +49 241 80 29206, Raum: 2.35
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQEVAwUBQIdezhsVN+J7zzuXAQGHJgf9FYr4wtTh6vQwg7uZuxkzb18Nfv4kPTbw
WhLO8vGQQrFSYuHvQ4zt/PokGGMyhjbALesiRFZmYT6Rhg3HEkf2Pmax+FC75NcQ
SMityI8CPE9JsnGKQGlei3/jQY8SFevjfa5GrFKVwv5GQIEqq97xKP2s6Lnv1zsG
LlkIUDcg55uiKQ+r3N8O0cMTPahuvUh5EbzG6IeD8oPzEgO+FfEoocdUkF1Hkj8Z
bPoA2DEnAXNXUwHCniBwdMZBpW4krWnV7JOG8m8bhARwuUvXC6JlsRx5InAA6XMh
+FHVnU0dQKef7nple7MUXlWx8dgxMW0t3vvhkv+EPXR1rMWvFjzYPw==
=vJ/G
-----END PGP SIGNATURE-----

Re: System with really big statetable crashing while doing "cat /proc/net/ip_conntrack | wc -l"

[Phil Oester]

Yes, this is a known bug.  I submitted a patch to fix it a few days ago
but have not heard anything from core developers.  See archives:

https://lists.netfilter.org/pipermail/netfilter-devel/2004-April/015054.html

Phil Oester


On Thu, Apr 22, 2004 at 07:57:34AM +0200, Jens Hektor wrote:
> So we only can guess for if doing
> 
> 	cat /proc/net/ip_conntrack | wc -l
> 
> the system reproducably crashes.

Re: System with really big statetable crashing while doing "cat /proc/net/ip_conntrack | wc -l"

[Chris Wilson]

Hi Jens,

> So we only can guess for if doing
> 
> 	cat /proc/net/ip_conntrack | wc -l
> 
> the system reproducably crashes.
>
> we are running a production packetfilter based on Fedora Core-1
> actual vanilla  2.4.22-1.2179.nptl-smp.

Do you have ip_conntrack_tftp loaded? If so, try unloading it. It's known 
to create corrupt conntrack entries that will crash the system eventually. 

Phil Oester <kernel@linuxace.com> posted a patch earlier week (subject: 
Orphaned Expectations) which fixes occasional problems with conntrack 
(probably only the TFTP helper, but it may affect others too, e.g. 
Amanda). You could try that patch to see if it helps.

Please let us know if that helps,

Cheers, Chris.
-- 
_  __ __     _
 / __/ / ,__(_)_  | Chris Wilson -- UNIX Firewall Lead Developer |
/ (_  ,\/ _/ /_ \ | NetServers.co.uk http://www.netservers.co.uk |
\__/_/_/_//_/___/ | 21 Signet Court, Cambridge, UK. 01223 576516 |

Re: System with really big statetable crashing while doing "cat /proc/net/ip_conntrack | wc -l"

[Herve Eychenne]

On Thu, Apr 22, 2004 at 07:57:34AM +0200, Jens Hektor wrote:

 Hi,

> we are running a production packetfilter based on Fedora Core-1
> actual vanilla  2.4.22-1.2179.nptl-smp.

> This packetfilter is in front of two class-B's and it has a big
> statetable, really. We guess that it's about 200k entries.

> So we only can guess for if doing
> cat /proc/net/ip_conntrack | wc -l
> the system reproducably crashes.

> We had the same effect on non-smp and 2.6 version of Fedora kernels
> and I guess that the same might happen with the vanilla kernel.

> Is this known, has anyone ever seen this?

Yes, alas! See
http://lists.netfilter.org/pipermail/netfilter-devel/2003-December/013557.html

But is wasn't reproduced since then because we didn't dare to touch
/proc/net/ip_conntrack after that ;-/, as it is a production machine...
No oops trace either, as we had to reboot it as soon as possible for
the same reason.

And let me tell you that tftp was absolutely not loaded, so it cannot
be the root of our problem.

 Herve

-- 
 _
(°=  Hervé Eychenne
//)
v_/_ WallFire project:  http://www.wallfire.org/

Re: System with really big statetable crashing while doing "cat /proc/net/ip_conntrack | wc -l"

[Chris Wilson]

Hi Herve,

> And let me tell you that tftp was absolutely not loaded, so it cannot
> be the root of our problem.

Did you have any other nat/conntrack helpers loaded? If it's a helper 
creating bad expectations then it might be fixed by Phil Oester's patch.

Cheers, Chris.
-- 
_  __ __     _
 / __/ / ,__(_)_  | Chris Wilson -- UNIX Firewall Lead Developer |
/ (_  ,\/ _/ /_ \ | NetServers.co.uk http://www.netservers.co.uk |
\__/_/_/_//_/___/ | 21 Signet Court, Cambridge, UK. 01223 576516 |