[xtables-addons] memory usage in module geoip (probably)

[xtables-addons] memory usage in module geoip (probably)

[Marcin Mirosław]

Hello!
I've noticed high size of kmalloc-512/2048 on my vps after a one, two
weeks of work. Example:
  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
 83440  83411  99%    0.57K   5960       14     47680K kmalloc-512
 21615  21604  99%    2.07K   1441       15     46112K kmalloc-2048
103776 103765  99%    0.31K   8648       12     34592K skbuff_head_cache

I've done a little investigation (trial by error method) to discover
which process keep data in kmalloc. It looks those kmalloc are used by
modules from xtables-addons-1.39. I'm using ipset6, geoip, sysrq,
tarpit. I greped sources for string "kmalloc", kmalloc is used by sysrq
and geoip (in some other modules too but i don't use them).

I'd like to ask is such usage of memory ok? Which module could take so
much precious ram?:) (I suspect geoip). How much memory geoip can use in
worse case (db with ipv4 and ipv6)? Can i freed this cache without
reboot? (I tried flush rules, rmmod but without succes).

Thank you for any answers!
Marcin

P.S. Is any other method to check which process/module/etc uses kmallocs?

Re: [xtables-addons] memory usage in module geoip (probably)

[Jan Engelhardt]

On Friday 2012-01-20 17:12, Marcin Mirosław wrote:

>Hello!
>I've noticed high size of kmalloc-512/2048 on my vps after a one, two
>weeks of work. Example:
>  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
> 83440  83411  99%    0.57K   5960       14     47680K kmalloc-512
> 21615  21604  99%    2.07K   1441       15     46112K kmalloc-2048
>103776 103765  99%    0.31K   8648       12     34592K skbuff_head_cache
>
>I've done a little investigation (trial by error method) to discover
>which process keep data in kmalloc. It looks those kmalloc are used by
>modules from xtables-addons-1.39. I'm using ipset6, geoip, sysrq,
>tarpit. I greped sources for string "kmalloc", kmalloc is used by sysrq
>and geoip (in some other modules too but i don't use them).

geoip uses vmalloc for its huge allocations, so that can't be it. The 
massive size of skbuff_head_cache would point towards there being a lot 
of dead skbs being held, which - in my nose - would smell of tarpit. Did 
you properly feed all packets that you tarpited also to -j CT --notrack?

>I'd like to ask is such usage of memory ok? Which module could take so
>much precious ram?:) (I suspect geoip). How much memory geoip can use in
>worse case (db with ipv4 and ipv6)?

The on-disk files for geoip are loaded verbatim into the kernel, so that 
would be only ~6 MB at worst for geoip, not 40+.

Re: [xtables-addons] memory usage in module geoip (probably)

[Marcin Mirosław]

W dniu 2012-01-20 18:05, Jan Engelhardt napisał(a):
> geoip uses vmalloc for its huge allocations, so that can't be it. The
> massive size of skbuff_head_cache would point towards there being a 
> lot
> of dead skbs being held, which - in my nose - would smell of tarpit. 
> Did
> you properly feed all packets that you tarpited also to -j CT 
> --notrack?

Surely i didn't use notrack;) I has problem how to use it, i'd like to 
tarpit packets
at the end of INPUT at filter table, notrack i can use only in raw 
table.
Have you got idea about kmallo-512 and kmalloc-2048? I'll try every 
single module and watch
kmalloc usage. It will take a couple of days for each one. Maybe i'll 
isolate which module
uses kmallocs. I suspect xtables because without them i didn't notice 
such situation.

>>I'd like to ask is such usage of memory ok? Which module could take 
>> so
>>much precious ram?:) (I suspect geoip). How much memory geoip can use 
>> in
>>worse case (db with ipv4 and ipv6)?
>
> The on-disk files for geoip are loaded verbatim into the kernel, so 
> that
> would be only ~6 MB at worst for geoip, not 40+.

Thank you for all information.
Marcin

Re: [xtables-addons] memory usage in module geoip (probably)

[Jan Engelhardt]

On Friday 2012-01-20 21:18, Marcin Mirosław wrote:

> W dniu 2012-01-20 18:05, Jan Engelhardt napisał(a):
>> geoip uses vmalloc for its huge allocations, so that can't be it. The
>> massive size of skbuff_head_cache would point towards there being a lot
>> of dead skbs being held, which - in my nose - would smell of tarpit. Did
>> you properly feed all packets that you tarpited also to -j CT --notrack?
>
>Surely i didn't use notrack;) I has problem how to use it, i'd like to tarpit
>packets
>at the end of INPUT at filter table, notrack i can use only in raw table.

TARPIT does not have much of a use for random ports without any services 
because it is specific to clients sending data. You should use DELUDE at 
the end of the chain, also because it does not keep any connections 
around like tarpit.

Re: [xtables-addons] memory usage in module geoip (probably)

[Marcin Mirosław]

W dniu 2012-01-20 22:00, Jan Engelhardt pisze:
> TARPIT does not have much of a use for random ports without any services
> because it is specific to clients sending data. You should use DELUDE at
> the end of the chain, also because it does not keep any connections
> around like tarpit.

Hello Jan.
Meseems TARPIT is what i need, i'd like to slow down (a little) bots 
looking for mssql and other MS specific soft.
I've made quick test, it looks when TARPIT is in chain then size of all 
three slabs (kmalloc-512/2048, skbuff_head_cache) slowly grow up. 
Without TARPIT its size is almost constant.
It appears on host:
domU, FV, 3.1.8-hardened (i have such situation since some time, with 
older kernels too) , x86_64, xtables-addons-1.37 and 1.39.
Other host doesn't suffer such problem:
bare metal, 3.1.5-hardened, i686, xtables-addons-1.37

How can i track down what is the reason of such situation?
Thank you.

Re: [xtables-addons] memory usage in module geoip (probably)

[Marcin Mirosław]

W dniu 2012-01-22 21:38, Marcin Miros³aw pisze:
> Other host doesn't suffer such problem:
> bare metal, 3.1.5-hardened, i686, xtables-addons-1.37

I forgot mention, those kernel has patch taken from 
http://www.ssi.bg/~ja/#routes

Re: [xtables-addons] memory usage in module geoip (probably)

[Jan Engelhardt]

On Sunday 2012-01-22 21:38, Marcin Mirosław wrote:

> W dniu 2012-01-20 22:00, Jan Engelhardt pisze:
>> TARPIT does not have much of a use for random ports without any services
>> because it is specific to clients sending data. You should use DELUDE at
>> the end of the chain, also because it does not keep any connections
>> around like tarpit.
>
> Hello Jan.
> Meseems TARPIT is what i need, i'd like to slow down (a little) bots looking
> for mssql and other MS specific soft.

Then you should use
 -t raw -A PREROUTING -p tcp --dport ms-sql-s -j CT --notrack
if your service is not exposed to the internet.

> How can i track down what is the reason of such situation?

Also compare output of `ss -at`.

Re: [xtables-addons][solved] memory usage in module geoip (probably)

[Marcin Mirosław]

W dniu 20.01.2012 17:12, Marcin Mirosław pisze:
> Hello!
> I've noticed high size of kmalloc-512/2048 on my vps after a one, two
> weeks of work. Example:
>   OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
>  83440  83411  99%    0.57K   5960       14     47680K kmalloc-512
>  21615  21604  99%    2.07K   1441       15     46112K kmalloc-2048
> 103776 103765  99%    0.31K   8648       12     34592K skbuff_head_cache
> 
> I've done a little investigation (trial by error method) to discover
> which process keep data in kmalloc. It looks those kmalloc are used by
> modules from xtables-addons-1.39. I'm using ipset6, geoip, sysrq,
> tarpit. I greped sources for string "kmalloc", kmalloc is used by sysrq
> and geoip (in some other modules too but i don't use them).
> 
> I'd like to ask is such usage of memory ok? Which module could take so
> much precious ram?:) (I suspect geoip). How much memory geoip can use in
> worse case (db with ipv4 and ipv6)? Can i freed this cache without
> reboot? (I tried flush rules, rmmod but without succes).

With xtables-addons-1.46 I don't have such issue, probably commit
4ff5a8fbf62b75b17194a5cd324c08b8fa0f08bf in TARPIT resolves my problem.
Thanks:)