ipset memory usage

ipset memory usage

[tedemo]

Hello,

The ipset modules seems to eat a lot of memory :
with a test set of  :
- 150 000 networks in a nethash
- 10 000 hosts in a iphash
ipset sometimes fails with out of memory errors :
"kernel: allocation failed: out of vmalloc space - use vmalloc=<size> to 
increase size"

The VmAllocUsed variable from /proc/meminfo goes from 6Mo to 160Mo after 
loading sets.

Is this normal ? Is there some way to reduce memory usage ?

Thanks

Re: ipset memory usage

[Jozsef Kadlecsik]

On Sun, 25 Sep 2005, tedemo wrote:

> The ipset modules seems to eat a lot of memory :
> with a test set of  :
> - 150 000 networks in a nethash
> - 10 000 hosts in a iphash
> ipset sometimes fails with out of memory errors :
> "kernel: allocation failed: out of vmalloc space - use vmalloc=<size> to
> increase size"
>
> The VmAllocUsed variable from /proc/meminfo goes from 6Mo to 160Mo after
> loading sets.

Check out the real size of the hashes by searching the kernel log:0
"rehashing of set <foo> triggered: hashsize grows from <n> to <m>".

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipset memory usage

[tedemo]

Hello,

Jozsef Kadlecsik a écrit :

>Check out the real size of the hashes by searching the kernel log:0
>"rehashing of set <foo> triggered: hashsize grows from <n> to <m>".
>
>Best regards,
>Jozsef
>
The size for the nethash, reported in the log is arround 20mo the first 
time it I run :
ipset --restore < ip.sets

I deleted each set (using ipset -X on each other) and the recreated them,
the size of the nethash went to 40Mo

A few reloads later its size is 90Mo.

NB: The set definition file has been created manualy to avoid hundred 
thousand calls to "ipset -A"
(i.e. without calling ipset --save). Can this cause troubles ?

Re: ipset memory usage

[Jozsef Kadlecsik]

Hi,

On Mon, 26 Sep 2005, tedemo wrote:

> >Check out the real size of the hashes by searching the kernel log:0
> >"rehashing of set <foo> triggered: hashsize grows from <n> to <m>".

> The size for the nethash, reported in the log is arround 20mo the first
> time it I run :
> ipset --restore < ip.sets

When the hash size is about 20 million entries, that alone requires ~80MB
physical memory. If resizing is triggered, at least two times of that
amount of memory is needed temporarily.

> I deleted each set (using ipset -X on each other) and the recreated them,
> the size of the nethash went to 40Mo
>
> A few reloads later its size is 90Mo.

Please write it exactly what you do and what happens. Do you create, fill
up and delete the *same* sets with exactly the *same* content, in a loop?

> NB: The set definition file has been created manualy to avoid hundred
> thousand calls to "ipset -A"
> (i.e. without calling ipset --save). Can this cause troubles ?

No.

What is your kernel and ipset versions?

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipset memory usage

[tedemo]

Jozsef Kadlecsik a écrit :

>Please write it exactly what you do and what happens. Do you create, fill
>up and delete the *same* sets with exactly the *same* content, in a loop?
>
>  
>
Yes I execute those commands in a loop on the same set (a nethash of 150 
000 entries).

the commands in the loop are :
ipset -X neth
ipset --restore < ip.sets

The obtained results are :
iteration 1: neth size 25856860    VmallocUsed 108004 kB
iteration 2: neth size 25856860    VmallocUsed 108004 kB
iteration 3: neth size 38785290    VmallocUsed 158168 kB
iteration 6: neth size 25856860    VmallocUsed 108004 kB
iteration 9: neth size 58177935    VmallocUsed 235844 kB

This seems a lot for 0.15 million entries.
Are the 230Mo of Vmalloc'ed memory really used by the kernel ?

>What is your kernel and ipset versions?
>
>  
>
ipset v2.2.2 Protocol version 2
iptables v1.3.2
Linux 2.6.13 i686

Re: ipset memory usage

[Jozsef Kadlecsik]

On Mon, 26 Sep 2005, tedemo wrote:

> >Please write it exactly what you do and what happens. Do you create, fill
> >up and delete the *same* sets with exactly the *same* content, in a loop?
> >
> Yes I execute those commands in a loop on the same set (a nethash of 150
> 000 entries).
>
> the commands in the loop are :
> ipset -X neth
> ipset --restore < ip.sets
>
> The obtained results are :
> iteration 1: neth size 25856860    VmallocUsed 108004 kB
> iteration 2: neth size 25856860    VmallocUsed 108004 kB
> iteration 3: neth size 38785290    VmallocUsed 158168 kB
> iteration 6: neth size 25856860    VmallocUsed 108004 kB
> iteration 9: neth size 58177935    VmallocUsed 235844 kB
>
> This seems a lot for 0.15 million entries.
> Are the 230Mo of Vmalloc'ed memory really used by the kernel ?

25-58 million hash entries isn't really needed to store 150.000 IP
addresses/netblocks!

How do you create the set neth in ip.sets? What is the exact command?

Coul you send me the file ip.sets in private for testings?

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: ipset memory usage

[tedemo]

[-- Attachment #1: Type: text/plain, Size: 457 bytes --]

Jozsef Kadlecsik a écrit :

>25-58 million hash entries isn't really needed to store 150.000 IP
>addresses/netblocks!
>
>How do you create the set neth in ip.sets? What is the exact command?
>
>Coul you send me the file ip.sets in private for testings?
>  
>
Unfortunatly no, because it seems that your mail server banned the one 
of my isp  ...

Anyway, I could reproduce this with the set generated by the attached 
perl script.

Regards

[-- Attachment #2: ipset_gen.pl --]
[-- Type: text/plain, Size: 220 bytes --]

print "-N neth nethash --hashsize 1024 --probes 2 --resize 50\n"; 
for ($i = 1; $i < 3; $i++) { 
for ($j = 0; $j < 255; $j++) { 
for ($k = 0; $k < 255; $k++) { 
print "-A neth $i.$j.$k.0/24\n"; 
} } };
print "COMMIT\n";

Re: ipset memory usage

[tedemo]

Jozsef Kadlecsik a écrit :

>When the hash size is about 20 million entries, that alone requires ~80MB
>physical memory. If resizing is triggered, at least two times of that
>amount of memory is needed temporarily.
>  
>
The problem was comming from the --probes parameter which was set to 2 !
Increasing its value to 10, reduced the hashsize up to a factor 100.

Thanks !

Re: ipset memory usage

[Jozsef Kadlecsik]

On Tue, 27 Sep 2005, tedemo wrote:

> >When the hash size is about 20 million entries, that alone requires ~80MB
> >physical memory. If resizing is triggered, at least two times of that
> >amount of memory is needed temporarily.
> >
> The problem was comming from the --probes parameter which was set to 2 !
> Increasing its value to 10, reduced the hashsize up to a factor 100.

Thus you trade memory for speed: lookups in the set will be ten times
slower.

Still I don't get how does it come that the initial hash size was so huge.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

ipset memory usage

[Akshat Kakkar]

As per info in http://lists.netfilter.org/pipermail/netfilter/2005-September/062859.html,
ipset of iphash of 20Million IPs, takes around only 80MB. I understand
that this might be simply multiplication of 20Million with 4 bytes
which IP takes if stored normally.

But when I try to replicate this I am getting size in memory as
506,481,176 (around 483MB) for number of entries 16,581,375.

This comes to around 30Bytes per IP.

I have got this values from ipset -L command

Is this much size anticipated?

Re: ipset memory usage

[Akshat Kakkar]

Anybody?

On Tue, Aug 25, 2015 at 5:35 PM, Akshat Kakkar <akshat.1984@gmail.com> wrote:
> As per info in http://lists.netfilter.org/pipermail/netfilter/2005-September/062859.html,
> ipset of iphash of 20Million IPs, takes around only 80MB. I understand
> that this might be simply multiplication of 20Million with 4 bytes
> which IP takes if stored normally.
>
> But when I try to replicate this I am getting size in memory as
> 506,481,176 (around 483MB) for number of entries 16,581,375.
>
> This comes to around 30Bytes per IP.
>
> I have got this values from ipset -L command
>
> Is this much size anticipated?

Re: ipset memory usage

[Jozsef Kadlecsik]

On Tue, 25 Aug 2015, Akshat Kakkar wrote:

> As per info in 
> http://lists.netfilter.org/pipermail/netfilter/2005-September/062859.html, 
> ipset of iphash of 20Million IPs, takes around only 80MB. I understand 
> that this might be simply multiplication of 20Million with 4 bytes which 
> IP takes if stored normally.
> 
> But when I try to replicate this I am getting size in memory as
> 506,481,176 (around 483MB) for number of entries 16,581,375.
> 
> This comes to around 30Bytes per IP.
> 
> I have got this values from ipset -L command
> 
> Is this much size anticipated?

At that time we had ipset 2.x - now it's 6.x. The algorithm behind the 
hash types has been completely rewritten since then and memory had been 
sacrified for the sake of speed. So that number does not apply to the 
current ipset memory usage.

The memory requirement for the hash types depends on the hashsize (H) and 
the number of elements (N) and it is something like (not counting some 
basic fixed structures):

H * 40byte + (N/4 + N%4) * 4 * element size

The RCU support also increased significantly the memory requirement.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
GGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary