iptables crashes server?

Linux Netfilter discussions
 help / color / mirror / Atom feed

* iptables crashes server?
@ 2005-04-04  8:16 Moritz Gartenmeister
  2005-04-04 11:47 ` Mohamed Eldesoky
  2005-04-04 14:59 ` iptables crashes server? Grant Taylor
  0 siblings, 2 replies; 9+ messages in thread
From: Moritz Gartenmeister @ 2005-04-04  8:16 UTC (permalink / raw)
  To: netfilter

hi all

i'm running linux 2.6.11.3 and iptables 1.3.1 with pom 20050321. i patched the kernel with ipp2p, 
and layer-7 patch.

the server is running as a bridge and is working absolutly fine. after a while (there is no specific 
time limit) the server crashes. the server is no more able to allocate new memory and even swapping 
doesn't help. in this state i am unable to log in, i have to push the power button.

i don't see heavy traffic before a crash and i don't see any flooding. is there a known memory leak 
problem?

i checked /proc/sys/net/ipv4/netfilter/ip_conntrack_count this number is in the range of 2'000 - 5'000.
i checked /proc/slabinfo <active_objs> is more or less similiar to ip_conntrack_count, <num_objs> is 
the maximum of ip_contrack_count.
i also was checking /proc/meminfo and there was no steady increase.

/var/log/messages shows no warning.
/var/log/syslog shows nothing
icmp is working.
imap is probably working (someone told me).
http is not working.
pop over ssl is working (sometimes).

does anyone had/have the same experience? or does anyone have some hints for further steps?

hardware: dell poweredge 2560 with 2gybte ram, 2 xenon dual cpus.

i was running the same setup wiht an older kernel 2.6.7/10 without much troubles.

regards
moritz

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: iptables crashes server?
  2005-04-04  8:16 iptables crashes server? Moritz Gartenmeister
@ 2005-04-04 11:47 ` Mohamed Eldesoky
  2005-04-05  6:47   ` Moritz Gartenmeister
  2005-04-04 14:59 ` iptables crashes server? Grant Taylor
  1 sibling, 1 reply; 9+ messages in thread
From: Mohamed Eldesoky @ 2005-04-04 11:47 UTC (permalink / raw)
  To: Moritz Gartenmeister, netfilter

Can you check the memory ??
May be it is a faulty RAM

On Apr 4, 2005 10:16 AM, Moritz Gartenmeister <moritz@uplink-verein.ch> wrote:
> hi all
> 
> i'm running linux 2.6.11.3 and iptables 1.3.1 with pom 20050321. i patched the kernel with ipp2p,
> and layer-7 patch.
> 
> the server is running as a bridge and is working absolutly fine. after a while (there is no specific
> time limit) the server crashes. the server is no more able to allocate new memory and even swapping
> doesn't help. in this state i am unable to log in, i have to push the power button.
> 
> i don't see heavy traffic before a crash and i don't see any flooding. is there a known memory leak
> problem?
> 
> i checked /proc/sys/net/ipv4/netfilter/ip_conntrack_count this number is in the range of 2'000 - 5'000.
> i checked /proc/slabinfo <active_objs> is more or less similiar to ip_conntrack_count, <num_objs> is
> the maximum of ip_contrack_count.
> i also was checking /proc/meminfo and there was no steady increase.
> 
> /var/log/messages shows no warning.
> /var/log/syslog shows nothing
> icmp is working.
> imap is probably working (someone told me).
> http is not working.
> pop over ssl is working (sometimes).
> 
> does anyone had/have the same experience? or does anyone have some hints for further steps?
> 
> hardware: dell poweredge 2560 with 2gybte ram, 2 xenon dual cpus.
> 
> i was running the same setup wiht an older kernel 2.6.7/10 without much troubles.
> 
> regards
> moritz
> 
> 



-- 
Mohamed Eldesoky
www.eldesoky.net
RHCE


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: iptables crashes server?
  2005-04-04 11:47 ` Mohamed Eldesoky
@ 2005-04-05  6:47   ` Moritz Gartenmeister
  2005-04-05 16:34     ` R. DuFresne
  2005-04-05 16:45     ` Mariusz Kruk
  0 siblings, 2 replies; 9+ messages in thread
From: Moritz Gartenmeister @ 2005-04-05  6:47 UTC (permalink / raw)
  To: netfilter

i checked the memory usage a little more in detail:

the free ram is decreasing.
the active use of ram is increasing.

ergo, something is filling up my ram. i run apache, irc-proxy, snort and argus besides iptables.

for me, there seems something buggy in the kernelspace of iptables.

but, if i am the only one with this problem, it's maybe not.

can someone tell me, which kernel and which iptables version is working?

regards
moritz

Mohamed Eldesoky wrote:
> Can you check the memory ??
> May be it is a faulty RAM
> 
> On Apr 4, 2005 10:16 AM, Moritz Gartenmeister <moritz@uplink-verein.ch> wrote:
> 
>>hi all
>>
>>i'm running linux 2.6.11.3 and iptables 1.3.1 with pom 20050321. i patched the kernel with ipp2p,
>>and layer-7 patch.
>>
>>the server is running as a bridge and is working absolutly fine. after a while (there is no specific
>>time limit) the server crashes. the server is no more able to allocate new memory and even swapping
>>doesn't help. in this state i am unable to log in, i have to push the power button.
>>
>>i don't see heavy traffic before a crash and i don't see any flooding. is there a known memory leak
>>problem?
>>
>>i checked /proc/sys/net/ipv4/netfilter/ip_conntrack_count this number is in the range of 2'000 - 5'000.
>>i checked /proc/slabinfo <active_objs> is more or less similiar to ip_conntrack_count, <num_objs> is
>>the maximum of ip_contrack_count.
>>i also was checking /proc/meminfo and there was no steady increase.
>>
>>/var/log/messages shows no warning.
>>/var/log/syslog shows nothing
>>icmp is working.
>>imap is probably working (someone told me).
>>http is not working.
>>pop over ssl is working (sometimes).
>>
>>does anyone had/have the same experience? or does anyone have some hints for further steps?
>>
>>hardware: dell poweredge 2560 with 2gybte ram, 2 xenon dual cpus.
>>
>>i was running the same setup wiht an older kernel 2.6.7/10 without much troubles.
>>
>>regards
>>moritz
>>
>>
> 
> 
> 
> 

-- 
Uplink student association
Moritz Gartenmeister
Bülachstrasse 1 F
8057 Zürich
Switzerland


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: iptables crashes server?
  2005-04-05  6:47   ` Moritz Gartenmeister
@ 2005-04-05 16:34     ` R. DuFresne
  2005-04-05 18:22       ` Moritz Gartenmeister
  2005-04-05 16:45     ` Mariusz Kruk
  1 sibling, 1 reply; 9+ messages in thread
From: R. DuFresne @ 2005-04-05 16:34 UTC (permalink / raw)
  To: Moritz Gartenmeister; +Cc: netfilter

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, 5 Apr 2005, Moritz Gartenmeister wrote:

> i checked the memory usage a little more in detail:
>
> the free ram is decreasing.
> the active use of ram is increasing.
>
> ergo, something is filling up my ram. i run apache, irc-proxy, snort and 
> argus besides iptables.
>
> for me, there seems something buggy in the kernelspace of iptables.
>
> but, if i am the only one with this problem, it's maybe not.
>

There are two things you might do to deal with memory over consumption 
issues;

Add more RAM to the system

or off load some of the services to other systems rather then directly 
running them all on the firewall server.  And this is likely the better 
way to deal with the problem in my mind.  firewalls afterall are 
traditionally a single purpose system.


Thanks,

Ron DuFresne
- -- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
         admin & senior security consultant:  sysinfo.com
                         http://sysinfo.com

...Love is the ultimate outlaw.  It just won't adhere to rules.
The most any of us can do is sign on as it's accomplice.  Instead
of vowing to honor and obey, maybe we should swear to aid and abet.
That would mean that security is out of the question.  The words
"make" and "stay" become inappropriate.  My love for you has no
strings attached.  I love you for free...
                         -Tom Robins <Still Life With Woodpecker>
Key fingerprint = 9401 4B13 B918 164C 647A  E838 B2DF AFCC 94B0 6629
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFCUr43st+vzJSwZikRAhB5AJsFNxy0VUPQDPbgV2g4vS/NWl+mZgCcDUui
kJNnt/dXPVl48WVqY/4CZP8=
=hgsV
-----END PGP SIGNATURE-----


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: iptables crashes server?
  2005-04-05 16:34     ` R. DuFresne
@ 2005-04-05 18:22       ` Moritz Gartenmeister
  0 siblings, 0 replies; 9+ messages in thread
From: Moritz Gartenmeister @ 2005-04-05 18:22 UTC (permalink / raw)
  To: netfilter

R. DuFresne wrote:
> 
> There are two things you might do to deal with memory over consumption 
> issues;
> 
> Add more RAM to the system
> 
> or off load some of the services to other systems rather then directly 
> running them all on the firewall server.  And this is likely the better 
> way to deal with the problem in my mind.  firewalls afterall are 
> traditionally a single purpose system.

more ram does not solve the problem, only delay it.
i don't run much services on the server. and i stopped them to look at the memory usage.

my newest idea: network card driver tg3.

cheers
moritz


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: iptables crashes server?
  2005-04-05  6:47   ` Moritz Gartenmeister
  2005-04-05 16:34     ` R. DuFresne
@ 2005-04-05 16:45     ` Mariusz Kruk
  2005-04-10 10:50       ` iptables crashes server? [OT] Moritz Gartenmeister
  1 sibling, 1 reply; 9+ messages in thread
From: Mariusz Kruk @ 2005-04-05 16:45 UTC (permalink / raw)
  To: netfilter

On Tue, Apr 05, 2005 at 08:47:34AM +0200, Moritz Gartenmeister wrote:
> ergo, something is filling up my ram. i run apache, irc-proxy, snort and 
> argus besides iptables.
> 
> for me, there seems something buggy in the kernelspace of iptables.
> 
> but, if i am the only one with this problem, it's maybe not.

To make sure it's not a memory leak in any of those services, have you
tried setting limits for process memory?
Also, have you checked the processes' memory usage? Did it rise?

-- 
/\-\/\-\/\-\/\-\/\-\/\-\/\ Nobody's perfect, but I'm not nobody
\  Kruk@epsilon.eu.org   / 
/ http://epsilon.eu.org/ \ 
\/-/\/-/\/-/\/-/\/-/\/-/\/ 


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: iptables crashes server? [OT]
  2005-04-05 16:45     ` Mariusz Kruk
@ 2005-04-10 10:50       ` Moritz Gartenmeister
  2005-04-11  9:15         ` Mariusz Kruk
  0 siblings, 1 reply; 9+ messages in thread
From: Moritz Gartenmeister @ 2005-04-10 10:50 UTC (permalink / raw)
  To: netfilter

hi

i was shutting down all services and also unloaded almost all of iptables modules. i didn't solve 
the problem.

i was using netconsole the get the latest messages before the crash. now i am pretty sure it is the 
networkcard driver maybe its a dos...:

IPP2P v0.7.2 unloaded
ip_conntrack version 2.1 (8192 buckets, 65536 max) - 268 bytes per conntrack
IPP2P v0.7.2 loading
NETDEV WATCHDOG: eth2: transmit timed out
tg3: eth2: transmit timed out, resetting
tg3: tg3_stop_block timed out, ofs=2c00 enable_bit=2
tg3: tg3_stop_block timed out, ofs=3400 enable_bit=2
tg3: tg3_stop_block timed out, ofs=2400 enable_bit=2
tg3: tg3_stop_block timed out, ofs=1800 enable_bit=2
tg3: tg3_stop_block timed out, ofs=4800 enable_bit=2
tg3: eth2: Link is down.
br0: port 2(eth2) entering disabled state
tg3: eth2: Link is up at 100 Mbps, half duplex.
tg3: eth2: Flow control is off for TX and off for RX.
br0: port 2(eth2) entering learning state
br0: topology change detected, propagating
br0: port 2(eth2) entering forwarding state
Ebtables v2.0 registered
HTB: quantum of class 10033 is small. Consider r2q change.
oom-killer: gfp_mask=0xd0
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1
cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16

Free pages:      433920kB (420416kB HighMem)
Active:186580 inactive:1496 dirty:38836 writeback:0 unstable:0 free:108480 slab:218645 mapped:4893 
pagetables:131
DMA free:3696kB min:176kB low:220kB high:264kB active:0kB inactive:32kB present:16384kB 
pages_scanned:399 all_unreclaimable? yes
lowmem_reserve[]: 0 880 2031
Normal free:9808kB min:9820kB low:12272kB high:14728kB active:4kB inactive:108kB present:901120kB 
pages_scanned:14744 all_unreclaimable? yes
lowmem_reserve[]: 0 0 9215
HighMem free:420416kB min:512kB low:640kB high:768kB active:746316kB inactive:5844kB 
present:1179520kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3696kB
Normal: 0*4kB 0*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9808kB
HighMem: 0*4kB 18*8kB 63*16kB 6*32kB 4*64kB 2*128kB 1*256kB 1*512kB 2*1024kB 21*2048kB 91*4096kB = 
420416kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 979924kB
Total swap = 979924kB
Out of Memory: Killed process 13539 (apache).
oom-killer: gfp_mask=0xd0
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1
cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16

Free pages:      433920kB (420480kB HighMem)
Active:186531 inactive:1495 dirty:38835 writeback:0 unstable:0 free:108480 slab:218665 mapped:4837 
pagetables:125
DMA free:3696kB min:176kB low:220kB high:264kB active:0kB inactive:24kB present:16384kB 
pages_scanned:2265 all_unreclaimable? yes
lowmem_reserve[]: 0 880 2031
Normal free:9744kB min:9820kB low:12272kB high:14728kB active:0kB inactive:112kB present:901120kB 
pages_scanned:24068 all_unreclaimable? yes
lowmem_reserve[]: 0 0 9215
HighMem free:420480kB min:512kB low:640kB high:768kB active:746124kB inactive:5844kB 
present:1179520kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3696kB
Normal: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9744kB
HighMem: 0*4kB 26*8kB 63*16kB 6*32kB 4*64kB 2*128kB 1*256kB 1*512kB 2*1024kB 21*2048kB 91*4096kB = 
420480kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 979924kB
Total swap = 979924kB
Out of Memory: Killed process 29873 (apache).
oom-killer: gfp_mask=0xd0
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1
cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16

Free pages:      434176kB (420736kB HighMem)
Active:186472 inactive:1489 dirty:38835 writeback:0 unstable:0 free:108544 slab:218665 mapped:4769 
pagetables:119
DMA free:3696kB min:176kB low:220kB high:264kB active:36kB inactive:0kB present:16384kB 
pages_scanned:2276 all_unreclaimable? yes
lowmem_reserve[]: 0 880 2031
Normal free:9744kB min:9820kB low:12272kB high:14728kB active:0kB inactive:112kB present:901120kB 
pages_scanned:24068 all_unreclaimable? yes
lowmem_reserve[]: 0 0 9215
HighMem free:420736kB min:512kB low:640kB high:768kB active:745852kB inactive:5844kB 
present:1179520kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3696kB
Normal: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9744kB
HighMem: 36*4kB 38*8kB 64*16kB 6*32kB 4*64kB 2*128kB 1*256kB 1*512kB 2*1024kB 21*2048kB 91*4096kB = 
420736kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 979924kB
Total swap = 979924kB
Out of Memory: Killed process 6467 (apache).
oom-killer: gfp_mask=0xd0
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1
cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16

Free pages:      434432kB (420992kB HighMem)
Active:186395 inactive:1498 dirty:38835 writeback:0 unstable:0 free:108608 slab:218665 mapped:4701 
pagetables:113
DMA free:3696kB min:176kB low:220kB high:264kB active:0kB inactive:36kB present:16384kB 
pages_scanned:2285 all_unreclaimable? yes
lowmem_reserve[]: 0 880 2031
Normal free:9744kB min:9820kB low:12272kB high:14728kB active:0kB inactive:112kB present:901120kB 
pages_scanned:24068 all_unreclaimable? yes
lowmem_reserve[]: 0 0 9215
HighMem free:420992kB min:512kB low:640kB high:768kB active:745580kB inactive:5844kB 
present:1179520kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3696kB
Normal: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9744kB
HighMem: 84*4kB 46*8kB 64*16kB 6*32kB 4*64kB 2*128kB 1*256kB 1*512kB 2*1024kB 21*2048kB 91*4096kB = 
420992kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 979924kB
Total swap = 979924kB
Out of Memory: Killed process 6468 (apache).
oom-killer: gfp_mask=0xd0
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1
cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16

Free pages:      434688kB (421248kB HighMem)
Active:186327 inactive:1498 dirty:38835 writeback:0 unstable:0 free:108672 slab:218665 mapped:4633 
pagetables:107
DMA free:3696kB min:176kB low:220kB high:264kB active:0kB inactive:36kB present:16384kB 
pages_scanned:2285 all_unreclaimable? yes
lowmem_reserve[]: 0 880 2031
Normal free:9744kB min:9820kB low:12272kB high:14728kB active:0kB inactive:112kB present:901120kB 
pages_scanned:24068 all_unreclaimable? yes
lowmem_reserve[]: 0 0 9215
HighMem free:421248kB min:512kB low:640kB high:768kB active:745308kB inactive:5844kB 
present:1179520kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3696kB
Normal: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9744kB
HighMem: 134*4kB 53*8kB 64*16kB 6*32kB 4*64kB 2*128kB 1*256kB 1*512kB 2*1024kB 21*2048kB 91*4096kB = 
421248kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 979924kB
Total swap = 979924kB
Out of Memory: Killed process 6478 (apache).
oom-killer: gfp_mask=0xd0
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1
cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
oom-killer: gfp_mask=0xd0
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1
cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16

Free pages:      434688kB (421248kB HighMem)
Active:186329 inactive:1470 dirty:38835 writeback:0 unstable:0 free:108672 slab:218665 mapped:4607 
pagetables:101
DMA free:3696kB min:176kB low:220kB high:264kB active:0kB inactive:36kB present:16384kB 
pages_scanned:2285 all_unreclaimable? yes
lowmem_reserve[]: 0 880 2031
Normal free:9744kB min:9820kB low:12272kB high:14728kB active:112kB inactive:0kB present:901120kB 
pages_scanned:24098 all_unreclaimable? yes
lowmem_reserve[]: 0 0 9215
HighMem free:421248kB min:512kB low:640kB high:768kB active:745204kB inactive:5844kB 
present:1179520kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3696kB
Normal: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9744kB
HighMem: 134*4kB 53*8kB 64*16kB 6*32kB 4*64kB 2*128kB 1*256kB 1*512kB 2*1024kB 21*2048kB 91*4096kB = 
421248kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 979924kB
Total swap = 979924kB
Out of Memory: Killed process 6479 (apache).
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16

Free pages:      434688kB (421248kB HighMem)
Active:186329 inactive:1470 dirty:38835 writeback:0 unstable:0 free:108672 slab:218665 mapped:4607 
pagetables:101
DMA free:3696kB min:176kB low:220kB high:264kB active:0kB inactive:36kB present:16384kB 
pages_scanned:2285 all_unreclaimable? yes
lowmem_reserve[]: 0 880 2031
Normal free:9744kB min:9820kB low:12272kB high:14728kB active:112kB inactive:0kB present:901120kB 
pages_scanned:24098 all_unreclaimable? yes
lowmem_reserve[]: 0 0 9215
HighMem free:421248kB min:512kB low:640kB high:768kB active:745204kB inactive:5844kB 
present:1179520kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3696kB
Normal: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9744kB
HighMem: 134*4kB 53*8kB 64*16kB 6*32kB 4*64kB 2*128kB 1*256kB 1*512kB 2*1024kB 21*2048kB 91*4096kB = 
421248kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 979924kB
Total swap = 979924kB
Out of Memory: Killed process 6479 (apache).
oom-killer: gfp_mask=0xd0
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
oom-killer: gfp_mask=0xd0
cpu 1 cold: low 0, high 2, batch 1
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1
cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16

Free pages:      434816kB (421376kB HighMem)
Active:186275 inactive:1498 dirty:38835 writeback:0 unstable:0 free:108704 slab:218666 mapped:4581 
pagetables:95
DMA free:3696kB min:176kB low:220kB high:264kB active:0kB inactive:36kB present:16384kB 
pages_scanned:2285 all_unreclaimable? yes
lowmem_reserve[]: 0 880 2031
Normal free:9744kB min:9820kB low:12272kB high:14728kB active:0kB inactive:112kB present:901120kB 
pages_scanned:24126 all_unreclaimable? yes
lowmem_reserve[]: 0 0 9215
HighMem free:421376kB min:512kB low:640kB high:768kB active:745100kB inactive:5844kB 
present:1179520kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3696kB
Normal: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9744kB
HighMem: 158*4kB 57*8kB 64*16kB 6*32kB 4*64kB 2*128kB 1*256kB 1*512kB 2*1024kB 21*2048kB 91*4096kB = 
421376kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 979924kB
Total swap = 979924kB
Out of Memory: Killed process 10546 (apache).
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1
cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16

Free pages:      435200kB (421760kB HighMem)
Active:186198 inactive:1490 dirty:38835 writeback:0 unstable:0 free:108800 slab:218666 mapped:4488 
pagetables:89
DMA free:3696kB min:176kB low:220kB high:264kB active:0kB inactive:36kB present:16384kB 
pages_scanned:2305 all_unreclaimable? yes
lowmem_reserve[]: 0 880 2031
Normal free:9744kB min:9820kB low:12272kB high:14728kB active:0kB inactive:112kB present:901120kB 
pages_scanned:24126 all_unreclaimable? yes
lowmem_reserve[]: 0 0 9215
HighMem free:421760kB min:512kB low:640kB high:768kB active:744792kB inactive:5812kB 
present:1179520kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3696kB
Normal: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9744kB
HighMem: 206*4kB 81*8kB 64*16kB 6*32kB 4*64kB 2*128kB 1*256kB 1*512kB 2*1024kB 21*2048kB 91*4096kB = 
421760kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 979924kB
Total swap = 979924kB
oom-killer: gfp_mask=0xd0
oom-killer: gfp_mask=0xd0
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1
cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16

Free pages:      435200kB (421760kB HighMem)
Active:186198 inactive:1490 dirty:38835 writeback:0 unstable:0 free:108800 slab:218666 mapped:4488 
pagetables:89
DMA free:3696kB min:176kB low:220kB high:264kB active:0kB inactive:36kB present:16384kB 
pages_scanned:2305 all_unreclaimable? yes
lowmem_reserve[]: 0 880 2031
Normal free:9744kB min:9820kB low:12272kB high:14728kB active:0kB inactive:112kB present:901120kB 
pages_scanned:24126 all_unreclaimable? yes
lowmem_reserve[]: 0 0 9215
HighMem free:421760kB min:512kB low:640kB high:768kB active:744792kB inactive:5812kB 
present:1179520kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3696kB
Normal: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9744kB
HighMem: 206*4kB 81*8kB 64*16kB 6*32kB 4*64kB 2*128kB 1*256kB 1*512kB 2*1024kB 21*2048kB 91*4096kB = 
421760kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 979924kB
Total swap = 979924kB
Out of Memory: Killed process 10679 (apache).
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1
cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16

Free pages:      435520kB (422080kB HighMem)
Active:186119 inactive:1490 dirty:38835 writeback:0 unstable:0 free:108880 slab:218667 mapped:4341 
pagetables:83
DMA free:3696kB min:176kB low:220kB high:264kB active:0kB inactive:36kB present:16384kB 
pages_scanned:2305 all_unreclaimable? yes
lowmem_reserve[]: 0 880 2031
Normal free:9744kB min:9820kB low:12272kB high:14728kB active:0kB inactive:112kB present:901120kB 
pages_scanned:24126 all_unreclaimable? yes
lowmem_reserve[]: 0 0 9215
HighMem free:422080kB min:512kB low:640kB high:768kB active:744476kB inactive:5812kB 
present:1179520kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3696kB
Normal: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9744kB
HighMem: 254*4kB 87*8kB 69*16kB 6*32kB 4*64kB 2*128kB 1*256kB 1*512kB 2*1024kB 21*2048kB 91*4096kB = 
422080kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 979924kB
Total swap = 979924kB
oom-killer: gfp_mask=0xd0
oom-killer: gfp_mask=0xd0
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
oom-killer: gfp_mask=0xd0
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1
cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1

Free pages:      435520kB (422080kB HighMem)
Active:186147 inactive:1462 dirty:38835 writeback:0 unstable:0 free:108880 slab:218667 mapped:4341 
pagetables:83
DMA free:3696kB min:176kB low:220kB high:264kB active:0kB inactive:36kB present:16384kB 
pages_scanned:2305 all_unreclaimable? yes
lowmem_reserve[]: 0 880 2031
Normal free:9744kB min:9820kB low:12272kB high:14728kB active:112kB inactive:0kB present:901120kB 
pages_scanned:24156 all_unreclaimable? yes
lowmem_reserve[]: 0 0 9215
HighMem free:422080kB min:512kB low:640kB high:768kB active:744476kB inactive:5812kB 
present:1179520kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3696kB
Normal: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9744kB
HighMem: cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
254*4kB 87*8kB 69*16kB 6*32kB 4*64kB 2*128kB Normal per-cpu:
1*256kB 1*512kB 2*1024kB 21*2048kB 91*4096kB = 422080kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 979924kB
Total swap = 979924kB
Out of Memory: Killed process 22754 (sh).
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16

Free pages:      435520kB (422080kB HighMem)
Active:186147 inactive:1462 dirty:38835 writeback:0 unstable:0 free:108880 slab:218667 mapped:4341 
pagetables:83
DMA free:3696kB min:176kB low:220kB high:264kB active:0kB inactive:36kB present:16384kB 
pages_scanned:2305 all_unreclaimable? yes
lowmem_reserve[]: 0 880 2031
Normal free:9744kB min:9820kB low:12272kB high:14728kB active:112kB inactive:0kB present:901120kB 
pages_scanned:24156 all_unreclaimable? yes
lowmem_reserve[]: 0 0 9215
HighMem free:422080kB min:512kB low:640kB high:768kB active:744476kB inactive:5812kB 
present:1179520kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3696kB
Normal: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9744kB
HighMem: 254*4kB 87*8kB 69*16kB 6*32kB 4*64kB 2*128kB 1*256kB 1*512kB 2*1024kB 21*2048kB 91*4096kB = 
422080kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 979924kB
Total swap = 979924kB
Out of Memory: Killed process 22754 (sh).
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1
cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16

Free pages:      435520kB (422080kB HighMem)
Active:186147 inactive:1462 dirty:38835 writeback:0 unstable:0 free:108880 slab:218673 mapped:4341 
pagetables:83
DMA free:3696kB min:176kB low:220kB high:264kB active:0kB inactive:36kB present:16384kB 
pages_scanned:2305 all_unreclaimable? yes
lowmem_reserve[]: 0 880 2031
Normal free:9744kB min:9820kB low:12272kB high:14728kB active:112kB inactive:0kB present:901120kB 
pages_scanned:24156 all_unreclaimable? yes
lowmem_reserve[]: 0 0 9215
HighMem free:422080kB min:512kB low:640kB high:768kB active:744476kB inactive:5812kB 
present:1179520kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0
DMA: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 3696kB
Normal: 0*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB 1*1024kB 0*2048kB 2*4096kB = 9744kB
HighMem: 254*4kB 87*8kB 69*16kB 6*32kB 4*64kB 2*128kB 1*256kB 1*512kB 2*1024kB 21*2048kB 91*4096kB = 
422080kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap  = 979924kB
Total swap = 979924kB
Out of Memory: Killed process 22754 (sh).
swapper: page allocation failure. order:0, mode:0x20
  [<c0139923>] __alloc_pages+0x2b3/0x420
  [<c013c861>] kmem_getpages+0x31/0xa0
  [<c013d5de>] cache_grow+0xae/0x160
  [<c013d80f>] cache_alloc_refill+0x17f/0x230
  [<c013db8a>] __kmalloc+0x8a/0xa0
  [<c02e99b7>] alloc_skb+0x47/0xf0
  [<c02398dc>] tg3_alloc_rx_skb+0x7c/0x130
  [<c0239efc>] tg3_rx+0x3ac/0x420
  [<c0239ffe>] tg3_poll+0x8e/0x130
  [<c02f0661>] net_rx_action+0x81/0x100
  [<c011cf95>] __do_softirq+0xb5/0xd0
  [<c011cfe1>] do_softirq+0x31/0x40
  [<c011d0cb>] irq_exit+0x3b/0x40
  [<c0104d98>] do_IRQ+0x28/0x40
  [<c010326e>] common_interrupt+0x1a/0x20
  [<c01006d3>] default_idle+0x23/0x30
  [<c010077f>] cpu_idle+0x5f/0x70
  [<c043898d>] start_kernel+0x15d/0x180
  [<c0438390>] unknown_bootoption+0x0/0x1e0
swapper: page allocation failure. order:0, mode:0x20
  [<c0139923>] __alloc_pages+0x2b3/0x420
  [<c013c861>] kmem_getpages+0x31/0xa0
  [<c013d5de>] cache_grow+0xae/0x160
  [<c013d80f>] cache_alloc_refill+0x17f/0x230
  [<c013db8a>] __kmalloc+0x8a/0xa0
  [<c02e99b7>] alloc_skb+0x47/0xf0
  [<c02398dc>] tg3_alloc_rx_skb+0x7c/0x130
  [<c0239efc>] tg3_rx+0x3ac/0x420
  [<c0239ffe>] tg3_poll+0x8e/0x130
  [<c02f0661>] net_rx_action+0x81/0x100
  [<c011cf95>] __do_softirq+0xb5/0xd0
  [<c011cfe1>] do_softirq+0x31/0x40
  [<c011d0cb>] irq_exit+0x3b/0x40
  [<c0104d98>] do_IRQ+0x28/0x40
  [<c010326e>]<4>swapper: page allocation failure. order:0, mode:0x20
  [<c0139923>] __alloc_pages+0x2b3/0x420
  [<c013c861>] kmem_getpages+0x31/0xa0
  [<c01181b7>] printk+0x17/0x20
  [<c013d5de>] cache_grow+0xae/0x160
  [<c013d80f>] cache_alloc_refill+0x17f/0x230
  [<c013db8a>] __kmalloc+0x8a/0xa0
  [<c02e99b7>] alloc_skb+0x47/0xf0
  [<c02fc0d2>] find_skb+0x32/0xa0
  [<c02fc231>] netpoll_send_udp+0x41/0x260
  [<f89b1040>] write_msg+0x40/0x50 [netconsole]
  [<c0117ee2>] __call_console_drivers+0x62/0x70
  [<c0117ff0>] call_console_drivers+0x70/0x130
  [<c0118401>] release_console_sem+0x51/0xc0
  [<c01182e0>] vprintk+0x120/0x170
  [<c010326e>] common_interrupt+0x1a/0x20
  [<c01181b7>] printk+0x17/0x20
  [<c010364f>] show_trace+0x4f/0x90
  [<c010326e>] common_interrupt+0x1a/0x20
  [<c010374c>] dump_stack+0x1c/0x20
  [<c0139923>] __alloc_pages+0x2b3/0x420
  [<c013c861>] kmem_getpages+0x31/0xa0
  [<c013d5de>] cache_grow+0xae/0x160
  [<c013d80f>] cache_alloc_refill+0x17f/0x230
  [<c013db8a>] __kmalloc+0x8a/0xa0
  [<c02e99b7>] alloc_skb+0x47/0xf0
  [<c02398dc>] tg3_alloc_rx_skb+0x7c/0x130
  [<c0239efc>] tg3_rx+0x3ac/0x420
  [<c0239ffe>] tg3_poll+0x8e/0x130
  [<c02f0661>] net_rx_action+0x81/0x100
  [<c011cf95>] __do_softirq+0xb5/0xd0


Mariusz Kruk wrote:
> On Tue, Apr 05, 2005 at 08:47:34AM +0200, Moritz Gartenmeister wrote:
> 
>>ergo, something is filling up my ram. i run apache, irc-proxy, snort and 
>>argus besides iptables.
>>
>>for me, there seems something buggy in the kernelspace of iptables.
>>
>>but, if i am the only one with this problem, it's maybe not.
> 
> 
> To make sure it's not a memory leak in any of those services, have you
> tried setting limits for process memory?
> Also, have you checked the processes' memory usage? Did it rise?
> 

-- 
Uplink student association
Moritz Gartenmeister
Bülachstrasse 1 F
8057 Zürich
Switzerland


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: iptables crashes server? [OT]
  2005-04-10 10:50       ` iptables crashes server? [OT] Moritz Gartenmeister
@ 2005-04-11  9:15         ` Mariusz Kruk
  0 siblings, 0 replies; 9+ messages in thread
From: Mariusz Kruk @ 2005-04-11  9:15 UTC (permalink / raw)
  To: netfilter

On Sun, Apr 10, 2005 at 12:50:11PM +0200, Moritz Gartenmeister wrote:
> i was shutting down all services and also unloaded almost all of iptables 
> modules. i didn't solve the problem.

Ooops. A kernel oops.
It's a 2.6.x series kernel, isn't it?
I had some problems of my own with this line of kernels, so I'd advice
to stay with the 2.4.x line.
I also had oopses when heavily manipulating iptables/packet scheduling
queues. Unfortunately, It was a production machine and I didn't have
time to debug. Just switched to latest 2.4.x and the router runs well
and has several weeks of uptime with no problems.

-- 
[------------------------] Bodies  in  motion tend to remain in motion.
[  Kruk@epsilon.eu.org   ] Bodies at rest tend to remain in bed
[ http://epsilon.eu.org/ ] 
[------------------------] 

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: iptables crashes server?
  2005-04-04  8:16 iptables crashes server? Moritz Gartenmeister
  2005-04-04 11:47 ` Mohamed Eldesoky
@ 2005-04-04 14:59 ` Grant Taylor
  1 sibling, 0 replies; 9+ messages in thread
From: Grant Taylor @ 2005-04-04 14:59 UTC (permalink / raw)
  To: Moritz Gartenmeister; +Cc: netfilter

I've had a similar situation on a system that was extremely complex (ECMP across 8 UML routers & CableModems, etc) running a lot of things both kernel space and user land.  We put 2 GB in the box for all the contracks that were going on for support of roughly 64000 possible contracks (8000 per UML with 8 UMLs) (there is a formula that I can find if needed).  The box would end up in a very similar situation after about 36 hours of operations.  We decided to set up a cron job to reboot the box daily.  Yes I know that this is a unix box that we are talking about, but given the nature of what the system was doing and the amount of time that we had to work on things this was the simple solution at the time.  Needless to say it's (sort of) working so my boss will not let me go back and work on it any more.  :(  Yes it pains me every time that I think about it.

Grant. . . .

Moritz Gartenmeister wrote:

> hi all
> 
> i'm running linux 2.6.11.3 and iptables 1.3.1 with pom 20050321. i 
> patched the kernel with ipp2p, and layer-7 patch.
> 
> the server is running as a bridge and is working absolutly fine. after a 
> while (there is no specific time limit) the server crashes. the server 
> is no more able to allocate new memory and even swapping doesn't help. 
> in this state i am unable to log in, i have to push the power button.
> 
> i don't see heavy traffic before a crash and i don't see any flooding. 
> is there a known memory leak problem?
> 
> i checked /proc/sys/net/ipv4/netfilter/ip_conntrack_count this number is 
> in the range of 2'000 - 5'000.
> i checked /proc/slabinfo <active_objs> is more or less similiar to 
> ip_conntrack_count, <num_objs> is the maximum of ip_contrack_count.
> i also was checking /proc/meminfo and there was no steady increase.
> 
> /var/log/messages shows no warning.
> /var/log/syslog shows nothing
> icmp is working.
> imap is probably working (someone told me).
> http is not working.
> pop over ssl is working (sometimes).
> 
> does anyone had/have the same experience? or does anyone have some hints 
> for further steps?
> 
> hardware: dell poweredge 2560 with 2gybte ram, 2 xenon dual cpus.
> 
> i was running the same setup wiht an older kernel 2.6.7/10 without much 
> troubles.
> 
> regards
> moritz
> 
> 
> 

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2005-04-11  9:15 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2005-04-04  8:16 iptables crashes server? Moritz Gartenmeister
2005-04-04 11:47 ` Mohamed Eldesoky
2005-04-05  6:47   ` Moritz Gartenmeister
2005-04-05 16:34     ` R. DuFresne
2005-04-05 18:22       ` Moritz Gartenmeister
2005-04-05 16:45     ` Mariusz Kruk
2005-04-10 10:50       ` iptables crashes server? [OT] Moritz Gartenmeister
2005-04-11  9:15         ` Mariusz Kruk
2005-04-04 14:59 ` iptables crashes server? Grant Taylor

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox