out of memory

out of memory

[Ingo Freund]

Hi,

on our database machine with 4 GB RAM and 2 XEON CPUs I got the
following kernel messages.
There are 2 GB RAM declared as shared memory for database usage.
Can anybody explain to me what happened and -may be- why?


Apr  2 10:55:09 widbrz01 kernel: oom-killer: gfp_mask=0xd0, order=0
Apr  2 10:55:09 widbrz01 kernel: Mem-info:
Apr  2 10:55:09 widbrz01 kernel: DMA per-cpu:
Apr  2 10:55:09 widbrz01 kernel: cpu 0 hot: low 2, high 6, batch 1 used:2
Apr  2 10:55:09 widbrz01 kernel: cpu 0 cold: low 0, high 2, batch 1 used:1
Apr  2 10:55:09 widbrz01 kernel: cpu 1 hot: low 2, high 6, batch 1 used:2
Apr  2 10:55:09 widbrz01 kernel: cpu 1 cold: low 0, high 2, batch 1 used:1
Apr  2 10:55:09 widbrz01 kernel: cpu 2 hot: low 2, high 6, batch 1 used:2
Apr  2 10:55:09 widbrz01 kernel: cpu 2 cold: low 0, high 2, batch 1 used:1
Apr  2 10:55:09 widbrz01 kernel: cpu 3 hot: low 2, high 6, batch 1 used:2
Apr  2 10:55:09 widbrz01 kernel: cpu 3 cold: low 0, high 2, batch 1 used:1
Apr  2 10:55:09 widbrz01 kernel: Normal per-cpu:
Apr  2 10:55:09 widbrz01 kernel: cpu 0 hot: low 62, high 186, batch 31 used:174
Apr  2 10:56:10 widbrz01 kernel: cpu 0 cold: low 0, high 62, batch 31 used:52
Apr  2 10:56:10 widbrz01 kernel: cpu 1 hot: low 62, high 186, batch 31 used:86
Apr  2 10:56:10 dbm kernel: cpu 1 cold: low 0, high 62, batch 31 used:47
Apr  2 10:56:10 dbm kernel: cpu 2 hot: low 62, high 186, batch 31 used:107
Apr  2 10:56:10 dbm kernel: cpu 2 cold: low 0, high 62, batch 31 used:31
Apr  2 10:56:10 dbm kernel: cpu 3 hot: low 62, high 186, batch 31 used:71
Apr  2 10:56:10 dbm kernel: cpu 3 cold: low 0, high 62, batch 31 used:39
Apr  2 10:56:10 dbm kernel: HighMem per-cpu:
Apr  2 10:56:10 dbm kernel: cpu 0 hot: low 62, high 186, batch 31 used:124
Apr  2 10:56:10 dbm kernel: cpu 0 cold: low 0, high 62, batch 31 used:46
Apr  2 10:56:10 dbm kernel: cpu 1 hot: low 62, high 186, batch 31 used:93
Apr  2 10:56:10 dbm kernel: cpu 1 cold: low 0, high 62, batch 31 used:47
Apr  2 10:56:10 dbm kernel: cpu 2 hot: low 62, high 186, batch 31 used:160
Apr  2 10:56:10 dbm kernel: cpu 2 cold: low 0, high 62, batch 31 used:50
Apr  2 10:56:10 dbm kernel: cpu 3 hot: low 62, high 186, batch 31 used:87
Apr  2 10:56:10 dbm kernel: cpu 3 cold: low 0, high 62, batch 31 used:31
Apr  2 10:56:10 dbm kernel: Free pages:        7608kB (868kB HighMem)
Apr  2 10:56:10 dbm kernel: Active:401296 inactive:416067 dirty:76506 writeback:14 unstable:0 free:1902 slab:8550 mapped:399275
pagetables:20137
Apr  2 10:56:10 dbm kernel: DMA free:3572kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16384kB pages_scanned:4
all_unreclaimable? yes
Apr  2 10:56:10 dbm kernel: lowmem_reserve[]: 0 880 6128
Apr  2 10:56:10 dbm kernel: Normal free:3168kB min:3756kB low:4692kB high:5632kB active:700kB inactive:548kB present:901120kB
pages_scanned:1488 all_unreclaimable? yes
Apr  2 10:56:10 dbm kernel: lowmem_reserve[]: 0 0 41984
Apr  2 10:56:10 dbm kernel: HighMem free:868kB min:512kB low:640kB high:768kB active:1604484kB inactive:1663720kB present:5373952kB
pages_scanned:0 all_unreclaimable? no
Apr  2 10:56:10 dbm kernel: lowmem_reserve[]: 0 0 0
Apr  2 10:56:10 dbm kernel: DMA: 1*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3572kB
Apr  2 10:56:10 dbm kernel: Normal: 88*4kB 4*8kB 0*16kB 1*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3168kB
Apr  2 10:56:10 dbm kernel: HighMem: 5*4kB 4*8kB 3*16kB 2*32kB 3*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 868kB
Apr  2 10:56:10 dbm kernel: Swap cache: add 1, delete 1, find 0/0, race 0+0
Apr  2 10:56:10 dbm kernel: Free swap  = 4209004kB
Apr  2 10:56:10 dbm kernel: Total swap = 4209008kB
Apr  2 10:56:10 dbm kernel: Free swap:       4209004kB
Apr  2 10:56:10 dbm kernel: klogd 1.4.1, ---------- state change ----------
Apr  2 10:56:11 dbm kernel: Inspecting /boot/System.map-2.6.13.2
Apr  2 10:56:11 dbm kernel: Loaded 28714 symbols from /boot/System.map-2.6.13.2.
Apr  2 10:56:11 dbm kernel: Symbols match kernel version 2.6.13.
Apr  2 10:56:11 dbm kernel: No module symbols loaded - kernel modules not enabled.
Apr  2 10:56:11 dbm kernel: 1572864 pages of RAM
Apr  2 10:56:11 dbm kernel: 1343488 pages of HIGHMEM
Apr  2 10:56:11 dbm kernel: 538120 reserved pages
Apr  2 10:56:11 dbm kernel: 10551624 pages shared
Apr  2 10:56:11 dbm kernel: 0 pages swap cached
Apr  2 10:56:11 dbm kernel: 63038 pages dirty
Apr  2 10:56:11 dbm kernel: 14 pages writeback
Apr  2 10:56:11 dbm kernel: 399275 pages mapped
Apr  2 10:56:11 dbm kernel: 8538 pages slab
Apr  2 10:56:11 dbm kernel: 20137 pages pagetables
Apr  2 10:56:11 dbm kernel: Out of Memory: Killed process 18250 (db:SBWI).


Thanks,
Ingo.

Re: out of memory

[Andrew Morton]

Ingo Freund <Ingo.Freund@e-dict.net> wrote:
>
> Hi,
> 
> on our database machine with 4 GB RAM and 2 XEON CPUs I got the
> following kernel messages.
> There are 2 GB RAM declared as shared memory for database usage.
> Can anybody explain to me what happened and -may be- why?
> 

You have a kernel memory leak.

> 
> Apr  2 10:55:09 widbrz01 kernel: oom-killer: gfp_mask=0xd0, order=0

It was a GFP_KERNEL allocation.

> Apr  2 10:56:10 dbm kernel: Normal free:3168kB min:3756kB low:4692kB high:5632kB active:700kB inactive:548kB present:901120kB

There's a grand total of 1.2MB of ZONE_NORMAL memory on the LRU.  The rest
(900MB-odd) is lost.

> Apr  2 10:56:11 dbm kernel: 8538 pages slab

and it's not in slab.

> Apr  2 10:56:11 dbm kernel: Symbols match kernel version 2.6.13.

Boy, 2.6.13 was a long time ago - I'm sure we fixed many leaks since then,
but I do not recall any particular patch which might fix this, sorry.

Your best option would be to seek a kernel upgrade.

Re: out of memory

[Ingo Freund]

Andrew Morton wrote:
> Ingo Freund <Ingo.Freund@e-dict.net> wrote:
>> Apr  2 10:56:11 dbm kernel: 8538 pages slab
> 
> and it's not in slab.
> 
>> Apr  2 10:56:11 dbm kernel: Symbols match kernel version 2.6.13.
> 
> Boy, 2.6.13 was a long time ago - I'm sure we fixed many leaks since then,
> but I do not recall any particular patch which might fix this, sorry.
> 
> Your best option would be to seek a kernel upgrade.

I'll give a try to the last kernel version

Is there a way to get those kernel/memory information from time
to time from the running system which I found in the syslog file?

Thank you
Ingo.

Re: out of memory

[Andrew Morton]

Ingo Freund <Ingo.Freund@e-dict.net> wrote:
>

Please don't edit Cc:'s when working with kernel developers - just do
reply-to-all.

> Andrew Morton wrote:
> > Ingo Freund <Ingo.Freund@e-dict.net> wrote:
> >> Apr  2 10:56:11 dbm kernel: 8538 pages slab
> > 
> > and it's not in slab.
> > 
> >> Apr  2 10:56:11 dbm kernel: Symbols match kernel version 2.6.13.
> > 
> > Boy, 2.6.13 was a long time ago - I'm sure we fixed many leaks since then,
> > but I do not recall any particular patch which might fix this, sorry.
> > 
> > Your best option would be to seek a kernel upgrade.
> 
> I'll give a try to the last kernel version

OK, 2.6.16.1 would suit.

> Is there a way to get those kernel/memory information from time
> to time from the running system which I found in the syslog file?

/proc/meminfo gives a decent summary.

To watch your ZONE_NORMAL disappearing it would be better to monitor
/proc/zoneinfo.  There you'll see `free', `active', `inactive' and
`present'.  In /proc/meminfo you'll find the slab utilisation.

It should be approximately true that free+active+inactive+slab = present.

Out of memory

[Luca Ferrari]

Sometimes on my mail server an error "Out of memory: kill process xxx 
(httpd)", and then the server seems to be frozen. The server is not loaded 
(in my opinion), and I cannot understand why this machine, that is more 
powerful than other, gives me these errors. I'm using FC2, with an HP server 
(penitum II, 192 MB ram). Is it possible it's an hardware fault (maybe a 
memory bank that does not work)?

Thanks,
Luca

P.S.
Merry xmas!
-- 
Luca Ferrari,
fluca1978@infinito.it

Re: Out of memory

[chuck gelm]

Luca Ferrari wrote:
> Sometimes on my mail server an error "Out of memory: kill process xxx 
> (httpd)", and then the server seems to be frozen. The server is not loaded 
> (in my opinion), and I cannot understand why this machine, that is more 
> powerful than other, gives me these errors. I'm using FC2, with an HP server 
> (penitum II, 192 MB ram). Is it possible it's an hardware fault (maybe a 
> memory bank that does not work)?
> 
> Thanks,
> Luca
> 
> P.S.
> Merry xmas!
Hi, Luca:

  It might be a hardware (memory) problem, try:

"Memtest-86 is a very thorough stand-alone memory test for x86 and 
Pentium systems (and compatibles). It currently does 11 tests to fully 
determine whether you have faulty RAM or not.

Homepage 	http://www.memtest86.com/
Download 	http://www.memtest86.com/memtest86-3.0.tar.gz
Author 	Chris Brady <cbrady_AT_sgi_DOT_C.O.M>
Version 	3.0
Licence 	GNU General Public License (GPL)
Source 	Yes
Environment 	Console
Status 	Stable"

Else:
  If you running 'httpd' on your mail server, why?

  Run 'top' from another terminal and when the mail server crashes you
will have a screenful of information within 5 seconds of the crash.

  The guru's will probably want to see the output of 'free' or 'top'
while the mail server is up and running.  Especially the instant
before it crashes!  ;-)

You can also run 'top', sort the display by descending memory usage,
and send the output to a file.  To set 'top' to display by memory
usage by default:
run 'top'
  press 'M' ; to set display to sort by memory
  press 'W' ; to save current settings
  press 'q' ; to quit
now:
  top -bn 1 > filename.ext
will save an instance of 'top's display to a file.

 From mail server; send the output of top to a file on another system:
( There is probably an 'at' command to run a command every # seconds.  )

"that is more powerful than other"
  is 'other' a mail server too?

HTH, Chuck

Re: out of memory

[qhwang]

> Do you have recursive calls?
> If you alloc all memory you need in the beginning, then the only problem
> that I can figure is that you have an infinite recursive call... or a loop
> between calls. It may be causing a stack overflow.

In my program there is a loop over 10k iteratives (image). In each loop
there are some kind of statistical learning algorithm running. It is killed
after it uses about 650 MB space (code + mainly data) with my laptop after
out of memory and swap.  I believe the problem lies inside the loop but
there should be no infinite loop because when I reduce the loop to over 1k
iteratives the program runs well. In this case it uses more 100 MB space.

I wonder where is this huge data space requirement coming from, since there
is no memory allocation inside the loop?



> BTW: You have problems with your e-mail:
It's very strange.

QingHua

Re: out of memory

[qhwang]

Hi, there

Below is the relevant piece of code. The main loop


 for(i=0;i<(rows-32);i++)
   for(j=0;j<(cols-32);j++) {
     GetSubWindow(image,win,i,j,cols,32,32);
     detectiontest(win, w, tol, nvar,MU,SI,PS,ES,MUT,SIT,PST,EST,32); //the
statistical learning algo
    ...
   }

in detectiontest
...
    do
    {
       /* EM algorithm */
       p1=1.0;
       p1*=emhht(w,P,MU,SI,PS,ES);

       /* check convergence */
       temp=computeerr(...);
       ...
      }
    while(temp > tol);
...

There are lot of memory allocation in function emhht but they are well
conducted. There should be no problem with it.

Any suggestion?

Re: out of memory

[ronkhu]

Hello,

hmmm... how about making use of software tools? like gdb? how about 
other profilers?
r u sure memory storage had been properly deallocated within emhht?

-Ron


qhwang wrote:

>Hi, there
>
>Below is the relevant piece of code. The main loop
>
>
> for(i=0;i<(rows-32);i++)
>   for(j=0;j<(cols-32);j++) {
>     GetSubWindow(image,win,i,j,cols,32,32);
>     detectiontest(win, w, tol, nvar,MU,SI,PS,ES,MUT,SIT,PST,EST,32); //the
>statistical learning algo
>    ...
>   }
>
>in detectiontest
>...
>    do
>    {
>       /* EM algorithm */
>       p1=1.0;
>       p1*=emhht(w,P,MU,SI,PS,ES);
>
>       /* check convergence */
>       temp=computeerr(...);
>       ...
>      }
>    while(temp > tol);
>...
>
>There are lot of memory allocation in function emhht but they are well
>conducted. There should be no problem with it.
>
>Any suggestion?
>
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>  
>

Re: out of memory

[qhwang]

> hmmm... how about making use of software tools? like gdb? how about
> other profilers?
> r u sure memory storage had been properly deallocated within emhht?

I have already used gdb for debugging but the problem is still there. In the
function emhht, there are 8 2D pointers to structure allocated and released.
Seven structures have two double type members, 1 has four double type
members. The factual sizes of pointers are 32x32. My computer reports that
the double type is 8 bytes. So, bascially the amount of memory each time
needed is (7*8*2+8*4)*32*32=144 KB. If there structure are not released,
there should be a big mess. But they are really released in my program. And
there is no reportedly illegal memory operation when I set MALLOC_CHECK_=1
for checking.

QingHua

Re: out of memory

[qhwang]

> Try using something like memprof

Many thanks.
I am trying to download and install it.

Re: out of memory

[qhwang]

I have found where the problem arises. It's a wavelet transformation
algorithem right before the do/while loop in detectiontest. I am now
figouring it out.

Many thanks for all your response.

QingHua

RE: out of memory

[Sandro Dangui]

Then you have to send us the piece of code that contains this hungry loop.


-----Original Message-----
From: qhwang [mailto:qhwang@ieeta.pt] 
Sent: quarta-feira, 8 de outubro de 2003 09:36
To: Dangui, Sandro [CMPS:RY11:EXCH]
Cc: linux-c-programming@vger.kernel.org
Subject: Re: out of memory




> Do you have recursive calls?
> If you alloc all memory you need in the beginning, then the only 
> problem that I can figure is that you have an infinite recursive 
> call... or a loop between calls. It may be causing a stack overflow.

In my program there is a loop over 10k iteratives (image). In each loop
there are some kind of statistical learning algorithm running. It is killed
after it uses about 650 MB space (code + mainly data) with my laptop after
out of memory and swap.  I believe the problem lies inside the loop but
there should be no infinite loop because when I reduce the loop to over 1k
iteratives the program runs well. In this case it uses more 100 MB space.

I wonder where is this huge data space requirement coming from, since there
is no memory allocation inside the loop?



> BTW: You have problems with your e-mail:
It's very strange.

QingHua

RE: out of memory

[Mariano Moreyra]

Yes, but don't send the code snippet as an attachment. It's seems like you
have problems with that.
Send your code snippet as part of the message body please


-----Mensaje original-----
De: linux-c-programming-owner@vger.kernel.org
[mailto:linux-c-programming-owner@vger.kernel.org]En nombre de Sandro
Dangui
Enviado el: Miércoles, 08 de Octubre de 2003 09:44
Para: qhwang
CC: linux-c-programming@vger.kernel.org
Asunto: RE: out of memory



Then you have to send us the piece of code that contains this hungry loop.


-----Original Message-----
From: qhwang [mailto:qhwang@ieeta.pt]
Sent: quarta-feira, 8 de outubro de 2003 09:36
To: Dangui, Sandro [CMPS:RY11:EXCH]
Cc: linux-c-programming@vger.kernel.org
Subject: Re: out of memory




> Do you have recursive calls?
> If you alloc all memory you need in the beginning, then the only
> problem that I can figure is that you have an infinite recursive
> call... or a loop between calls. It may be causing a stack overflow.

In my program there is a loop over 10k iteratives (image). In each loop
there are some kind of statistical learning algorithm running. It is killed
after it uses about 650 MB space (code + mainly data) with my laptop after
out of memory and swap.  I believe the problem lies inside the loop but
there should be no infinite loop because when I reduce the loop to over 1k
iteratives the program runs well. In this case it uses more 100 MB space.

I wonder where is this huge data space requirement coming from, since there
is no memory allocation inside the loop?



> BTW: You have problems with your e-mail:
It's very strange.

QingHua


-
To unsubscribe from this list: send the line "unsubscribe
linux-c-programming" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

-
To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: out of memory

[Piyush Jain]

plz. send the code relevant part only...
On Wed, 8 Oct 2003, Mariano Moreyra wrote:

> Yes, but don't send the code snippet as an attachment. It's seems like you
> have problems with that.
> Send your code snippet as part of the message body please
> 
> 
> -----Mensaje original-----
> De: linux-c-programming-owner@vger.kernel.org
> [mailto:linux-c-programming-owner@vger.kernel.org]En nombre de Sandro
> Dangui
> Enviado el: Miércoles, 08 de Octubre de 2003 09:44
> Para: qhwang
> CC: linux-c-programming@vger.kernel.org
> Asunto: RE: out of memory
> 
> 
> 
> Then you have to send us the piece of code that contains this hungry loop.
> 
> 
> -----Original Message-----
> From: qhwang [mailto:qhwang@ieeta.pt]
> Sent: quarta-feira, 8 de outubro de 2003 09:36
> To: Dangui, Sandro [CMPS:RY11:EXCH]
> Cc: linux-c-programming@vger.kernel.org
> Subject: Re: out of memory
> 
> 
> 
> 
> > Do you have recursive calls?
> > If you alloc all memory you need in the beginning, then the only
> > problem that I can figure is that you have an infinite recursive
> > call... or a loop between calls. It may be causing a stack overflow.
> 
> In my program there is a loop over 10k iteratives (image). In each loop
> there are some kind of statistical learning algorithm running. It is killed
> after it uses about 650 MB space (code + mainly data) with my laptop after
> out of memory and swap.  I believe the problem lies inside the loop but
> there should be no infinite loop because when I reduce the loop to over 1k
> iteratives the program runs well. In this case it uses more 100 MB space.
> 
> I wonder where is this huge data space requirement coming from, since there
> is no memory allocation inside the loop?
> 
> 
> 
> > BTW: You have problems with your e-mail:
> It's very strange.
> 
> QingHua
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe
> linux-c-programming" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 

"Anyone can do any amount of work provided 
	it isn't the work he is supposed to be doing at the moment."
-----------------
Piyush Jain
PhD Student (CSE)
G-108/Hall7
Mobile:9839079614


-
To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

out of memory

[qhwang]

Hi there,

Can anyone help me out of the mess? My program runs of out memory and then
is killed by the system. I use "top" to monitor its running and find out of
both RAM and SWAP. The "SIZE" option of "top" reaches more than 260 MB and
"RSS" more than 40 MB. Is this reasonable? The size of my program is just
over 100 KB. And the memory of my computer is just 64 MB and the swap is 256
MB. By the way, I find no memory leaks. Any suggestion will be greatly
appreaciated.

Bests,

QingHua.

P.S. I sent a post last night but I don't receive it this morning. So I post
it again.

out of memory

[Balram Adlakha]

Why does my system logs have:
kernel : out of memory, killed process (xyz)

I have 256mb ram, and a huge swap partition which never gets used!
I am currently on 2.5.60 but i've been having this problem for a long time...