out of memory

out of memory

[qhwang]

Hi there,

Can anyone help me out of the mess? My program runs of out memory and then
is killed by the system. I use "top" to monitor its running and find out of
both RAM and SWAP. The "SIZE" option of "top" reaches more than 260 MB and
"RSS" more than 40 MB. Is this reasonable? The size of my program is just
over 100 KB. And the memory of my computer is just 64 MB and the swap is 256
MB. By the way, I find no memory leaks. Any suggestion will be greatly
appreaciated.

Bests,

QingHua.

P.S. I sent a post last night but I don't receive it this morning. So I post
it again.

Re: out of memory

[qhwang]

> Do you have recursive calls?
> If you alloc all memory you need in the beginning, then the only problem
> that I can figure is that you have an infinite recursive call... or a loop
> between calls. It may be causing a stack overflow.

In my program there is a loop over 10k iteratives (image). In each loop
there are some kind of statistical learning algorithm running. It is killed
after it uses about 650 MB space (code + mainly data) with my laptop after
out of memory and swap.  I believe the problem lies inside the loop but
there should be no infinite loop because when I reduce the loop to over 1k
iteratives the program runs well. In this case it uses more 100 MB space.

I wonder where is this huge data space requirement coming from, since there
is no memory allocation inside the loop?



> BTW: You have problems with your e-mail:
It's very strange.

QingHua

RE: out of memory

[Sandro Dangui]

Then you have to send us the piece of code that contains this hungry loop.


-----Original Message-----
From: qhwang [mailto:qhwang@ieeta.pt] 
Sent: quarta-feira, 8 de outubro de 2003 09:36
To: Dangui, Sandro [CMPS:RY11:EXCH]
Cc: linux-c-programming@vger.kernel.org
Subject: Re: out of memory




> Do you have recursive calls?
> If you alloc all memory you need in the beginning, then the only 
> problem that I can figure is that you have an infinite recursive 
> call... or a loop between calls. It may be causing a stack overflow.

In my program there is a loop over 10k iteratives (image). In each loop
there are some kind of statistical learning algorithm running. It is killed
after it uses about 650 MB space (code + mainly data) with my laptop after
out of memory and swap.  I believe the problem lies inside the loop but
there should be no infinite loop because when I reduce the loop to over 1k
iteratives the program runs well. In this case it uses more 100 MB space.

I wonder where is this huge data space requirement coming from, since there
is no memory allocation inside the loop?



> BTW: You have problems with your e-mail:
It's very strange.

QingHua

RE: out of memory

[Mariano Moreyra]

Yes, but don't send the code snippet as an attachment. It's seems like you
have problems with that.
Send your code snippet as part of the message body please


-----Mensaje original-----
De: linux-c-programming-owner@vger.kernel.org
[mailto:linux-c-programming-owner@vger.kernel.org]En nombre de Sandro
Dangui
Enviado el: Miércoles, 08 de Octubre de 2003 09:44
Para: qhwang
CC: linux-c-programming@vger.kernel.org
Asunto: RE: out of memory



Then you have to send us the piece of code that contains this hungry loop.


-----Original Message-----
From: qhwang [mailto:qhwang@ieeta.pt]
Sent: quarta-feira, 8 de outubro de 2003 09:36
To: Dangui, Sandro [CMPS:RY11:EXCH]
Cc: linux-c-programming@vger.kernel.org
Subject: Re: out of memory




> Do you have recursive calls?
> If you alloc all memory you need in the beginning, then the only
> problem that I can figure is that you have an infinite recursive
> call... or a loop between calls. It may be causing a stack overflow.

In my program there is a loop over 10k iteratives (image). In each loop
there are some kind of statistical learning algorithm running. It is killed
after it uses about 650 MB space (code + mainly data) with my laptop after
out of memory and swap.  I believe the problem lies inside the loop but
there should be no infinite loop because when I reduce the loop to over 1k
iteratives the program runs well. In this case it uses more 100 MB space.

I wonder where is this huge data space requirement coming from, since there
is no memory allocation inside the loop?



> BTW: You have problems with your e-mail:
It's very strange.

QingHua


-
To unsubscribe from this list: send the line "unsubscribe
linux-c-programming" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

-
To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: out of memory

[Piyush Jain]

plz. send the code relevant part only...
On Wed, 8 Oct 2003, Mariano Moreyra wrote:

> Yes, but don't send the code snippet as an attachment. It's seems like you
> have problems with that.
> Send your code snippet as part of the message body please
> 
> 
> -----Mensaje original-----
> De: linux-c-programming-owner@vger.kernel.org
> [mailto:linux-c-programming-owner@vger.kernel.org]En nombre de Sandro
> Dangui
> Enviado el: Miércoles, 08 de Octubre de 2003 09:44
> Para: qhwang
> CC: linux-c-programming@vger.kernel.org
> Asunto: RE: out of memory
> 
> 
> 
> Then you have to send us the piece of code that contains this hungry loop.
> 
> 
> -----Original Message-----
> From: qhwang [mailto:qhwang@ieeta.pt]
> Sent: quarta-feira, 8 de outubro de 2003 09:36
> To: Dangui, Sandro [CMPS:RY11:EXCH]
> Cc: linux-c-programming@vger.kernel.org
> Subject: Re: out of memory
> 
> 
> 
> 
> > Do you have recursive calls?
> > If you alloc all memory you need in the beginning, then the only
> > problem that I can figure is that you have an infinite recursive
> > call... or a loop between calls. It may be causing a stack overflow.
> 
> In my program there is a loop over 10k iteratives (image). In each loop
> there are some kind of statistical learning algorithm running. It is killed
> after it uses about 650 MB space (code + mainly data) with my laptop after
> out of memory and swap.  I believe the problem lies inside the loop but
> there should be no infinite loop because when I reduce the loop to over 1k
> iteratives the program runs well. In this case it uses more 100 MB space.
> 
> I wonder where is this huge data space requirement coming from, since there
> is no memory allocation inside the loop?
> 
> 
> 
> > BTW: You have problems with your e-mail:
> It's very strange.
> 
> QingHua
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe
> linux-c-programming" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 

"Anyone can do any amount of work provided 
	it isn't the work he is supposed to be doing at the moment."
-----------------
Piyush Jain
PhD Student (CSE)
G-108/Hall7
Mobile:9839079614


-
To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: out of memory

[qhwang]

Hi, there

Below is the relevant piece of code. The main loop


 for(i=0;i<(rows-32);i++)
   for(j=0;j<(cols-32);j++) {
     GetSubWindow(image,win,i,j,cols,32,32);
     detectiontest(win, w, tol, nvar,MU,SI,PS,ES,MUT,SIT,PST,EST,32); //the
statistical learning algo
    ...
   }

in detectiontest
...
    do
    {
       /* EM algorithm */
       p1=1.0;
       p1*=emhht(w,P,MU,SI,PS,ES);

       /* check convergence */
       temp=computeerr(...);
       ...
      }
    while(temp > tol);
...

There are lot of memory allocation in function emhht but they are well
conducted. There should be no problem with it.

Any suggestion?

Re: out of memory

[ronkhu]

Hello,

hmmm... how about making use of software tools? like gdb? how about 
other profilers?
r u sure memory storage had been properly deallocated within emhht?

-Ron


qhwang wrote:

>Hi, there
>
>Below is the relevant piece of code. The main loop
>
>
> for(i=0;i<(rows-32);i++)
>   for(j=0;j<(cols-32);j++) {
>     GetSubWindow(image,win,i,j,cols,32,32);
>     detectiontest(win, w, tol, nvar,MU,SI,PS,ES,MUT,SIT,PST,EST,32); //the
>statistical learning algo
>    ...
>   }
>
>in detectiontest
>...
>    do
>    {
>       /* EM algorithm */
>       p1=1.0;
>       p1*=emhht(w,P,MU,SI,PS,ES);
>
>       /* check convergence */
>       temp=computeerr(...);
>       ...
>      }
>    while(temp > tol);
>...
>
>There are lot of memory allocation in function emhht but they are well
>conducted. There should be no problem with it.
>
>Any suggestion?
>
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>  
>

Re: out of memory

[qhwang]

> hmmm... how about making use of software tools? like gdb? how about
> other profilers?
> r u sure memory storage had been properly deallocated within emhht?

I have already used gdb for debugging but the problem is still there. In the
function emhht, there are 8 2D pointers to structure allocated and released.
Seven structures have two double type members, 1 has four double type
members. The factual sizes of pointers are 32x32. My computer reports that
the double type is 8 bytes. So, bascially the amount of memory each time
needed is (7*8*2+8*4)*32*32=144 KB. If there structure are not released,
there should be a big mess. But they are really released in my program. And
there is no reportedly illegal memory operation when I set MALLOC_CHECK_=1
for checking.

QingHua

Re: out of memory

[qhwang]

> Try using something like memprof

Many thanks.
I am trying to download and install it.

Re: out of memory

[qhwang]

I have found where the problem arises. It's a wavelet transformation
algorithem right before the do/while loop in detectiontest. I am now
figouring it out.

Many thanks for all your response.

QingHua