Possible huge iptables memory leakage

Possible huge iptables memory leakage

[Peteris Krumins]

Hello,

 I suspect there is a huge memory leakage in one or more of the
 iptables modules or iptables core.
 The scenario is simple, i set up a test-machine and let different
 traffic flow through it.
 It's located near the central backbone of my company, so different
 traffic flows through.
 Different iptables matches/targets are used.

 There are no applications running which could make memory leaks.
 The Linux is custom built and takes up 12MB of disk space.
 Only a single getty process is running and if i log in correctly a
 shell - bash. Verifying the memory usage w/ `free' i see no userland
 application is taking up that much memory alltogether, but the memory
 usage is still growing.

two hours before:
used memory: 26304KB

now:
used memory: 29204KB
 
and the usage grows every hour around 1.5 MB.
The data flow is around 8mbit/s at a constant rate.

If i flush all the rules, the memory is not freed but it is not
growing anymore.

The Linux is - 2.4.22-pre6 patched w/ patch-o-matic-20030714
Iptables are 1.2.8 patched w/ patch-o-matic-20030714

Have some memory leaks recently been fixed?

I am tomorrow to try 2.4.22-pre9 and the latest patch-o-matic.


here's a list of used iptable modules:
(is there a way to find out how much memory each module uses?)

ipt_mark                 440   2  (autoclean)
^- used to match some marked values
ipt_mport                696   4  (autoclean)
^- used to match some mports
ipt_state                568   4  (autoclean)
^- used to -j MARK only ESTABLISHED,RELATED traffic
ipt_psd                42824   2  (autoclean)
^- used to derect portscans
ipt_ttl                  600   1  (autoclean)
^- used to fake outgoing ttl
ipt_limit                952   6  (autoclean)
^- used to limit some dataflow
ipt_MARK                 824 316  (autoclean)
^- mark much packets for fun
iptable_nat            20728   0  (autoclean) (unused)
^- not used
ipt_REJECT              3192   1  (autoclean)
^- reject some traffic
iptable_filter          1668   1  (autoclean)
^- a must
ip_conntrack_ftp        4016   0  (unused)
ip_conntrack_irc        3184   0  (unused)
ip_conntrack           25736   4  [ipt_state iptable_nat ip_conntrack_ftp ip_conntrack_irc]
^- used to connection-track.
ip_queue                5020   1
ipt_XOR                 1208   2  (autoclean)
^- used to encrypt some of the traffic.
iptable_mangle          2192   1  (autoclean)
^- used to match some packets in mangle table
ip_tables              13720  15  [ipt_mark ipt_mport ipt_state ipt_psd ipt_ttl ipt_limit ipt_MARK iptable_nat ipt_REJECT iptable_filter ipt_XOR iptable_mangle]
^- a must

RE: Possible huge iptables memory leakage

[Aldo S. Lagana]

I run many different Linux systems ranging in physical memory from 128M to
5Gig (that's RAM!) both with and without iptables:

The funny thing about Linux is - if you give it RAM - it will use it....I am
sure someone more in tune with mem management will set me straight, but it
seems that Linux will always grab up 90% of all physical memory after it has
been running for a while...

To make a short story long - unless you are tracking all connections (and
there are many dynamic connections) I would not be too quick to blame
iptables - I mean the amount of ram you are using in 30Meg - is this the
80's???


-----Original Message-----
From: netfilter-admin@lists.netfilter.org
[mailto:netfilter-admin@lists.netfilter.org] On Behalf Of Peteris Krumins
Sent: Thursday, July 31, 2003 3:23 PM
To: netfilter@lists.netfilter.org

Hello,

 I suspect there is a huge memory leakage in one or more of the
 iptables modules or iptables core.
 The scenario is simple, i set up a test-machine and let different
 traffic flow through it.
 It's located near the central backbone of my company, so different
 traffic flows through.
 Different iptables matches/targets are used.

 There are no applications running which could make memory leaks.
 The Linux is custom built and takes up 12MB of disk space.
 Only a single getty process is running and if i log in correctly a
 shell - bash. Verifying the memory usage w/ `free' i see no userland
 application is taking up that much memory alltogether, but the memory
 usage is still growing.

two hours before:
used memory: 26304KB

now:
used memory: 29204KB
 
and the usage grows every hour around 1.5 MB.
The data flow is around 8mbit/s at a constant rate.

If i flush all the rules, the memory is not freed but it is not
growing anymore.

The Linux is - 2.4.22-pre6 patched w/ patch-o-matic-20030714
Iptables are 1.2.8 patched w/ patch-o-matic-20030714

Have some memory leaks recently been fixed?

I am tomorrow to try 2.4.22-pre9 and the latest patch-o-matic.


here's a list of used iptable modules:
(is there a way to find out how much memory each module uses?)

ipt_mark                 440   2  (autoclean)
^- used to match some marked values
ipt_mport                696   4  (autoclean)
^- used to match some mports
ipt_state                568   4  (autoclean)
^- used to -j MARK only ESTABLISHED,RELATED traffic
ipt_psd                42824   2  (autoclean)
^- used to derect portscans
ipt_ttl                  600   1  (autoclean)
^- used to fake outgoing ttl
ipt_limit                952   6  (autoclean)
^- used to limit some dataflow
ipt_MARK                 824 316  (autoclean)
^- mark much packets for fun
iptable_nat            20728   0  (autoclean) (unused)
^- not used
ipt_REJECT              3192   1  (autoclean)
^- reject some traffic
iptable_filter          1668   1  (autoclean)
^- a must
ip_conntrack_ftp        4016   0  (unused)
ip_conntrack_irc        3184   0  (unused)
ip_conntrack           25736   4  [ipt_state iptable_nat ip_conntrack_ftp
ip_conntrack_irc]
^- used to connection-track.
ip_queue                5020   1
ipt_XOR                 1208   2  (autoclean)
^- used to encrypt some of the traffic.
iptable_mangle          2192   1  (autoclean)
^- used to match some packets in mangle table
ip_tables              13720  15  [ipt_mark ipt_mport ipt_state ipt_psd
ipt_ttl ipt_limit ipt_MARK iptable_nat ipt_REJECT iptable_filter ipt_XOR
iptable_mangle]
^- a must

Re[2]: Possible huge iptables memory leakage

[Peteris Krumins]

Hello Aldo,

Thursday, July 31, 2003, 10:42:12 PM, you wrote:

ASL> I run many different Linux systems ranging in physical memory from 128M to
ASL> 5Gig (that's RAM!) both with and without iptables:

Yes, i have running lots of systems, but the most interesting part is
that i have a traffic accounting, shaping and real netfilter box.
w/ 2.4.20 base, iptables 1.2.7a - it eats only 190 of 512 not ever
more.

ASL> The funny thing about Linux is - if you give it RAM - it will use it....I am
ASL> sure someone more in tune with mem management will set me straight, but it
ASL> seems that Linux will always grab up 90% of all physical memory after it has
ASL> been running for a while...

Only if some interactive (not necessary user interacive) applications are running.

ASL> To make a short story long - unless you are tracking all connections (and
ASL> there are many dynamic connections) I would not be too quick to blame
ASL> iptables - I mean the amount of ram you are using in 30Meg - is this the
ASL> 80's???

Not 80's, but i have to get my project running w/ less than 59MB ram.

Yes, i am tracking connections - but its now night (23:30/11:30 pm),
the computer w/ connection tracking has 600 conntrack entries (its
1200 less than it was 5 hours ago) but it uses 10 more megabytes of
ram than then (5h ago) and still growing.

I hope it's just Linux - eating as much ram as possible and then
using. And it's not a memory leak..


P.Krumins

RE: Possible huge iptables memory leakage

[Daniel Chemko]

I am not sure what everyone else's environments are like, but I am
definitely getting memory infinitely swallowed into Netfilter somewhere.
I have userspace programs, but put together they take less than 5% of
the system memory.

I forced myself to reboot once the ACTIVE memory takes 95% of the CPU,
which was every 3 weeks with 196mb of RAM. This server averaged 200 -
250 connection tracks during office hours. The ramping active memory
usage went up in a very linear pattern until it took all the system, at
which point I would have to reboot due to performance degradation.

The last reboot, I decided to play with the machine's settings and add
some memory totaling 256MB. I also disabled my SWAP partition because I
have a few database servers that perform fantastically when kswapd isn't
used much. 

The result: Active memory 'seems' to have stabilized around 75MB at
least for the few days that it has been stable. I cannot tell if this
was totally kswap's fault, or even if the problem has been fixed at all.
It does look promising though.

Re[2]: Possible huge iptables memory leakage

[Peteris Krumins]

Thursday, July 31, 2003, 11:14:51 PM, you wrote:

DC> I am not sure what everyone else's environments are like, but I am
DC> definitely getting memory infinitely swallowed into Netfilter somewhere.
DC> I have userspace programs, but put together they take less than 5% of
DC> the system memory.

I am getting only on this particular 2.4.22-pre6 and ipt1.2.8
w/patch-o-matic-20030714

As Aldo S. Lagana noted it might be conntracking but it does not seem
logical - conntracking entries shrinking, ram usage growing.
I hope it hold the peak hours tomorrow.

DC> I forced myself to reboot once the ACTIVE memory takes 95% of the CPU,
DC> which was every 3 weeks with 196mb of RAM. This server averaged 200 -
DC> 250 connection tracks during office hours. The ramping active memory
DC> usage went up in a very linear pattern until it took all the system, at
DC> which point I would have to reboot due to performance degradation.

I hope i dont have to reboot tomorrow..

DC> The last reboot, I decided to play with the machine's settings and add
DC> some memory totaling 256MB. I also disabled my SWAP partition because I
DC> have a few database servers that perform fantastically when kswapd isn't
DC> used much. 

I am not swapping at all, since the machine loads everything from 16MB
ide flash.

DC> The result: Active memory 'seems' to have stabilized around 75MB at
DC> least for the few days that it has been stable. I cannot tell if this
DC> was totally kswap's fault, or even if the problem has been fixed at all.
DC> It does look promising though.

We'll see, thanks.


P.Krumins