iptables: memory allocation problem

iptables: memory allocation problem

[Herve Eychenne]

 Hi everyone,

I'm currently setting up a quiet big firewall, and I experience some
annoying problems. Here is what I get right now:
# iptables -A INPUT -j ACCEPT
iptables: Memory allocation problem

Well... something is going wrong, obviously.
But before that, I had some strange things happening. Here is my story.
I use a script which produces a rules file (in iptables-save/restore format),
and I apply it with iptables-restore.

But from time to time, I had the following error:
iptables-restore: line 2613 failed
where 2613 is the last line of the file (final filter COMMIT). As you can see,
the file is quite big, but anyway, it should work. And it worked, most of
the time, with the exactly same rules file.

So, to try to reproduce the problem, I ran :
# while iptables-restore < rules ; do : ; done
And could get no error. I had the intuition that the error showed up
more often when there was some traffic (and maybe such a heavy loop 
prevents connections from establishing properly).
So I ran:
# while iptables-restore < rules ; do sleep 10 ; done
and I finally got an error again (always the same: iptables-restore:
line 2613 failed), but only after hundreds of loops without any
problem.
Then I ran iptables-restore < rules manually a times by hand, and no
problem appeared. Then, I got the error again, each time, now.

After that, I can not even insert a single rule with the regular
iptables command.
Now I systematically get:
# iptables -A INPUT -j ACCEPT
iptables: Memory allocation problem

Syslog says nothing about this (if netfilter cannot allocate
enough kernel memory for rules, maybe it should report it with
a printk...), and userspace message is quite vague.

Note that the machine is still perfectly functionnal. I can log in
(remotely) and do whatever I want without any problem. I just seem to be
running out of kernel memory for the iptables rule insertion.

So I suspect there is a kind of memory leak somewhere.

Maybe I'll try to reproduce the leak on another machine by doing a
# while iptables-restore < rules ; do : ; done

What can I do to figure out where's the problem exactly, and what
caused it?

Here are things that may be relevant. I don't see any particular
problem... even if there is one, for sure. :-(
But I'm not a kernel guru, so please help me!

The machine is a SMP (4 processors) with Xeon 2.40GHz.

The host is running iptables-1.2.9 with 2.4.23 vanilla kernel (highmem
4Go activated, as there is 1.5G of RAM), and no real exotic
patch-o-matic module added (I'll provide the list on request if you
want) and no module support (I'll send .config on request as well).

# cat /proc/meminfo
        total:    used:    free:  shared: buffers:  cached:
Mem:  1586384896 607453184 978931712        0 104947712 296701952
Swap: 2055528448        0 2055528448
MemTotal:      1549204 kB
MemFree:        955988 kB
MemShared:           0 kB
Buffers:        102488 kB
Cached:         289748 kB
SwapCached:          0 kB
Active:         171476 kB
Inactive:       229480 kB
HighTotal:      655336 kB
HighFree:       269948 kB
LowTotal:       893868 kB
LowFree:        686040 kB
SwapTotal:     2007352 kB
SwapFree:      2007352 kB

# wc -l /proc/net/ip_conntrack
     700 /proc/net/ip_conntrack          # nothing particular

I tweaked the number of buckets (based on my document
http://www.wallfire.org/misc/netfilter_conntrack_perf.txt)
# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max
3579157
# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_buckets
3579157
Note: maybe it's a little too much (there's no real memory consuming
userspace command running, so I want to reserve as much as possible for
conntrack), but I think I can recall having an iptables memory
allocation only one time (I was surprised, but forgot
about it) a long time ago on the same machine before I had set these
parameters to something else than the default.

# cat /proc/slabinfo
slabinfo - version: 1.1 (SMP)
kmem_cache            96     96    244    6    6    1 :  252  126
ip_fib_hash          452    452     32    4    4    1 :  252  126
ip_conntrack         218  19500    320   56 1625    1 :  124   62
ip_mrt_cache           0      0     96    0    0    1 :  252  126
tcp_tw_bucket        160    160     96    4    4    1 :  252  126
tcp_bind_bucket      452    452     32    4    4    1 :  252  126
tcp_open_request     464    590     64    9   10    1 :  252  126
inet_peer_cache      338    590     64   10   10    1 :  252  126
ip_dst_cache         354  75576    160   70 3149    1 :  252  126
arp_cache            324    450    128   15   15    1 :  252  126
blkdev_requests     4120   4120     96  103  103    1 :  252  126
journal_head         894  29874     48   15  383    1 :  252  126
revoke_table         252    253     12    1    1    1 :  252  126
revoke_record        535   1921     32   16   17    1 :  252  126
dnotify_cache          0      0     20    0    0    1 :  252  126
file_lock_cache      160    160     96    4    4    1 :  252  126
fasync_cache           0      0     16    0    0    1 :  252  126
uid_cache            339    339     32    3    3    1 :  252  126
skbuff_head_cache   1584   2340    192  113  117    1 :  252  126
sock                 370    432    832   48   48    2 :  124   62
sigqueue             376    754    132   17   26    1 :  252  126
kiobuf                 0      0     64    0    0    1 :  252  126
cdev_cache          1121   1121     64   19   19    1 :  252  126
bdev_cache           177    177     64    3    3    1 :  252  126
mnt_cache            118    118     64    2    2    1 :  252  126
inode_cache        33794  45822    512 4828 6546    1 :  124   62
dentry_cache       25494  47670    128  879 1589    1 :  252  126
filp                 570    570    128   19   19    1 :  252  126
names_cache           19     19   4096   19   19    1 :   60   30
buffer_head       150510 308640     96 6204 7716    1 :  252  126
mm_struct            642    768    160   30   32    1 :  252  126
vm_area_struct      1902   2280     96   52   57    1 :  252  126
fs_cache             641    767     64   12   13    1 :  252  126
files_cache          201    387    416   38   43    1 :  124   62
signal_act           159    219   1312   55   73    1 :   60   30
size-131072(DMA)       0      0 131072    0    0   32 :    0    0
size-131072            0      0 131072    0    0   32 :    0    0
size-65536(DMA)        0      0  65536    0    0   16 :    0    0
size-65536             0      1  65536    0    1   16 :    0    0
size-32768(DMA)        0      0  32768    0    0    8 :    0    0
size-32768             1      2  32768    1    2    8 :    0    0
size-16384(DMA)        0      0  16384    0    0    4 :    0    0
size-16384             0      4  16384    0    4    4 :    0    0
size-8192(DMA)         0      0   8192    0    0    2 :    0    0
size-8192             12     18   8192   12   18    2 :    0    0
size-4096(DMA)         0      0   4096    0    0    1 :   60   30
size-4096           1190   1250   4096 1190 1250    1 :   60   30
size-2048(DMA)         0      0   2048    0    0    1 :   60   30
size-2048            316    376   2048  158  188    1 :   60   30
size-1024(DMA)         0      0   1024    0    0    1 :  124   62
size-1024            450    512   1024  119  128    1 :  124   62
size-512(DMA)          0      0    512    0    0    1 :  124   62
size-512             230    416    512   35   52    1 :  124   62
size-256(DMA)          0      0    256    0    0    1 :  252  126
size-256             417    795    256   34   53    1 :  252  126
size-128(DMA)          0      0    128    0    0    1 :  252  126
size-128            1314   1440    128   48   48    1 :  252  126
size-64(DMA)           0      0     64    0    0    1 :  252  126
size-64              877   1003     64   17   17    1 :  252  126
size-32(DMA)           0      0     32    0    0    1 :  252  126
size-32             4594   4972     32   42   44    1 :  252  126

 Herve

-- 
 _
(°=  Hervé Eychenne
//)
v_/_ WallFire project:  http://www.wallfire.org/

Re: iptables: memory allocation problem

[Herve Eychenne]

On Tue, Dec 16, 2003 at 07:29:59PM +0100, Herve Eychenne wrote:

 Hi again,

> I'm currently setting up a quiet big firewall, and I experience some
> annoying problems. Here is what I get right now:
> # iptables -A INPUT -j ACCEPT
> iptables: Memory allocation problem
> Well... something is going wrong, obviously.
> [...]
> Note that the machine is still perfectly functionnal. I can log in
> (remotely) and do whatever I want without any problem. I just seem to be
> running out of kernel memory for the iptables rule insertion.
> So I suspect there is a kind of memory leak somewhere.
> [...]
> The machine is a SMP (4 processors) with Xeon 2.40GHz.
> The host is running iptables-1.2.9 with 2.4.23 vanilla kernel (highmem
> 4Go activated, as there is 1.5G of RAM), and no real exotic
> patch-o-matic module added (I'll provide the list on request if you
> want) and no module support (I'll send .config on request as well).
> [...]
> I tweaked the number of buckets (based on my document
> http://www.wallfire.org/misc/netfilter_conntrack_perf.txt)
> # cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max
> 3579157
> # cat /proc/sys/net/ipv4/netfilter/ip_conntrack_buckets
> 3579157
> Note: maybe it's a little too much (there's no real memory consuming
> userspace command running, so I want to reserve as much as possible for
> conntrack), but I think I can recall having an iptables memory
> allocation only one time (I was surprised, but forgot
> about it) a long time ago on the same machine before I had set these
> parameters to something else than the default.

I know... nobody dared to reply... too bad... :-(
So here's your punishment. ;-)
Well... after a reboot and 2 days without problem, another issue
showed up.

I tried another "iptables-restore <myrules", which produced no
memory outage. Then, shortly after, I came up with the strange
idea of wanting to know how many entries there was in the conntrack.
What a foolish idea... as my
# wc -l /proc/net/ip_conntrack
simply locked the machine: my remote shell freezed, no traffic could
go through the firewall, and the console itself was completely
unresponsive.
After 2 or 3 minutes, I decided I could not wait any longer, as I
had to reboot the firewall as soon as possible to get the important
traffic back.

Do you think it is possible that a wc -l on a huge conntrack (there
could have been up to 3 million entries (as you can see in the
quoted text above), but I have absolutely no idea of how many there
were here) would take so much time (more that 3 minutes on a
4*2.4 Ghz) and would get the kernel so busy that it would even
freeze the console?
Or do you think it's clearly a bug?

Well, anyway, as conntrack listing locks conntrack state, don't you
think we should definitely provide a way to read ip_conntrack_count
through /proc/sys/net/ipv4/netfilter/ip_conntrack_count ? (aka
"Yet Another Oneliner Patch For Patch-O-Matic")

 Herve

-- 
 _
(°=  Hervé Eychenne
//)
v_/_ WallFire project:  http://www.wallfire.org/

Re: iptables: memory allocation problem

[Martin Josefsson]

[-- Attachment #1: Type: text/plain, Size: 504 bytes --]

On Sat, 2003-12-20 at 13:12, Herve Eychenne wrote:

> Well, anyway, as conntrack listing locks conntrack state, don't you
> think we should definitely provide a way to read ip_conntrack_count
> through /proc/sys/net/ipv4/netfilter/ip_conntrack_count ? (aka
> "Yet Another Oneliner Patch For Patch-O-Matic")

Just a quick response before I vanish for some christmas vacation.

Look at extra/ctstat.patch*
Apply it and download the util, it will give you what you want and more.

-- 
/Martin

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: iptables: memory allocation problem

[Herve Eychenne]

On Sat, Dec 20, 2003 at 01:30:29PM +0100, Martin Josefsson wrote:

> On Sat, 2003-12-20 at 13:12, Herve Eychenne wrote:

> > Well, anyway, as conntrack listing locks conntrack state, don't you
> > think we should definitely provide a way to read ip_conntrack_count
> > through /proc/sys/net/ipv4/netfilter/ip_conntrack_count ? (aka
> > "Yet Another Oneliner Patch For Patch-O-Matic")

> Just a quick response before I vanish for some christmas vacation.

> Look at extra/ctstat.patch*
> Apply it and download the util, it will give you what you want and more.

Ok, let me put it differently... :-)
This patch is interesting, and it adds so little overhead that I don't
understand why it isn't already in submitted.

If, for any reason, the coreteam doesn't want to include it as-is,
then we could consider adding ip_conntrack_count to /proc, and add a
kernel config option (defaulting to no) for conntrack stats.
What do you think?

 Herve

-- 
 _
(°=  Hervé Eychenne
//)
v_/_ WallFire project:  http://www.wallfire.org/