Memory leak

Memory leak

[Aleksandar Kacanski]

Hello,
I am running 2.4.18-3 version of the kernel on smp dual
processor and 1GB of RAM. My memory usage is increasing and
I can't find what exactly is eating memory. Top and proc
are reporting increases, but I would like to know of a
better way of tracing usage of memory and possible leak in
application(s).

Please reply to kacanski@yahoo.com
thanks                Sasha


=====
------
     *  "Last night I dreamed I ate a ten-pound marshmallow, and when I woke up the pillow was gone."

__________________________________________________
Do You Yahoo!?
Yahoo! Finance - Get real-time stock quotes
http://finance.yahoo.com

Re: Memory leak

[Richard B. Johnson]

On Mon, 26 Aug 2002, Aleksandar Kacanski wrote:

> Hello,
> I am running 2.4.18-3 version of the kernel on smp dual
> processor and 1GB of RAM. My memory usage is increasing and
> I can't find what exactly is eating memory. Top and proc
> are reporting increases, but I would like to know of a
> better way of tracing usage of memory and possible leak in
> application(s).
> 
> Please reply to kacanski@yahoo.com
> thanks                Sasha
> 
> 

Applications that use malloc() and friends, get more memory from
the kernel by resetting the break address. It's called "morecore()".
You can put a procedure, perhaps off SIGALRM, that periodically
checks the break address and writes it to a log. Applications
that end up with an ever-increasing break address have memory
leaks. Note that the break address is almost never set back.
This is not an error; malloc() assumes that if you used a lot
of memory once, you'll probably use it again. Check out sbrk()
and brk() in the man pages.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
The US military has given us many words, FUBAR, SNAFU, now ENRON.
Yes, top management were graduates of West Point and Annapolis.

Re: Memory leak

[george anzinger]

"Richard B. Johnson" wrote:
> 
> On Mon, 26 Aug 2002, Aleksandar Kacanski wrote:
> 
> > Hello,
> > I am running 2.4.18-3 version of the kernel on smp dual
> > processor and 1GB of RAM. My memory usage is increasing and
> > I can't find what exactly is eating memory. Top and proc
> > are reporting increases, but I would like to know of a
> > better way of tracing usage of memory and possible leak in
> > application(s).
> >
> > Please reply to kacanski@yahoo.com
> > thanks                Sasha
> >
> >
> 
> Applications that use malloc() and friends, get more memory from
> the kernel by resetting the break address. It's called "morecore()".
> You can put a procedure, perhaps off SIGALRM, that periodically
> checks the break address and writes it to a log. Applications
> that end up with an ever-increasing break address have memory
> leaks. Note that the break address is almost never set back.
> This is not an error; malloc() assumes that if you used a lot
> of memory once, you'll probably use it again. Check out sbrk()
> and brk() in the man pages.

But this all comes back when the application ends.  You
should be able to see the memory reappear when the app
terminates.

-- 
George Anzinger   george@mvista.com
High-res-timers: 
http://sourceforge.net/projects/high-res-timers/
Preemption patch:
http://www.kernel.org/pub/linux/kernel/people/rml

Re: Memory leak

[Richard B. Johnson]

On Mon, 26 Aug 2002, george anzinger wrote:

> "Richard B. Johnson" wrote:
> > 
> > On Mon, 26 Aug 2002, Aleksandar Kacanski wrote:
> > 
> > > Hello,
> > > I am running 2.4.18-3 version of the kernel on smp dual
> > > processor and 1GB of RAM. My memory usage is increasing and
> > > I can't find what exactly is eating memory. Top and proc
> > > are reporting increases, but I would like to know of a
> > > better way of tracing usage of memory and possible leak in
> > > application(s).
> > >
> > > Please reply to kacanski@yahoo.com
> > > thanks                Sasha
> > >
> > >
> > 
> > Applications that use malloc() and friends, get more memory from
> > the kernel by resetting the break address. It's called "morecore()".
> > You can put a procedure, perhaps off SIGALRM, that periodically
> > checks the break address and writes it to a log. Applications
> > that end up with an ever-increasing break address have memory
> > leaks. Note that the break address is almost never set back.
> > This is not an error; malloc() assumes that if you used a lot
> > of memory once, you'll probably use it again. Check out sbrk()
> > and brk() in the man pages.
> 
> But this all comes back when the application ends.  You
> should be able to see the memory reappear when the app
> terminates.
> 

Not by looking at /proc/meminfo. The fact that Linux redistributes
buffers, etc., during normal operation has often been misconceived
as a memory leak. Linux will try to use all available memory. This
is because unused memory is wasted memory. When Linux first starts
up, there is little to do, and very little history, so it hasn't
gotten a chance to use much memory. If you run a few apps, compile
the kernel, etc., you get to use all the memory. Then when the
system is quiescent, often persons look at /proc/meminfo, note that
a lot of memory is being used, and declare that there must be some
kind of memory leak.

There just might be some kind of memory leak, but, you can't tell
it from looking at /proc/meminfo. You would need to instrument each
procedure that allocates memory and verify that all execution paths,
including exceptions, result in that memory being freed. This would
be a good project for one of the college students that is looking
to get his/her hands dirty. It's a lot more useful than 'inventing'
some new file-system that never gets used, etc.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
The US military has given us many words, FUBAR, SNAFU, now ENRON.
Yes, top management were graduates of West Point and Annapolis.

Re: Memory leak

[Mike Galbraith]

At 04:05 PM 8/26/2002 -0400, Richard B. Johnson wrote:
>There just might be some kind of memory leak, but, you can't tell
>it from looking at /proc/meminfo. You would need to instrument each
>procedure that allocates memory [...]

Ingo did that a long time ago in his memleak patch.  Unfortunately, it's
laying around unmaintained, as I ran out of time.  If someone wants a
(kernel) memory leak detector, all they have to is rip it out of the IKD
patch and update it (trivial).

         -Mike

Re: Memory leak

[jw schultz]

On Mon, Aug 26, 2002 at 03:29:17PM -0400, Richard B. Johnson wrote:
> On Mon, 26 Aug 2002, Aleksandar Kacanski wrote:
> 
> > Hello,
> > I am running 2.4.18-3 version of the kernel on smp dual
> > processor and 1GB of RAM. My memory usage is increasing and
> > I can't find what exactly is eating memory. Top and proc
> > are reporting increases, but I would like to know of a
> > better way of tracing usage of memory and possible leak in
> > application(s).
> > 
> > Please reply to kacanski@yahoo.com
> > thanks                Sasha
> > 
> > 
> 
> Applications that use malloc() and friends, get more memory from
> the kernel by resetting the break address. It's called "morecore()".
> You can put a procedure, perhaps off SIGALRM, that periodically
> checks the break address and writes it to a log. Applications
> that end up with an ever-increasing break address have memory
> leaks. Note that the break address is almost never set back.
> This is not an error; malloc() assumes that if you used a lot
> of memory once, you'll probably use it again. Check out sbrk()
> and brk() in the man pages.

brk() isn't the only way applications get memory.  A
commonly used method of getting process memory is to mmap()
anonymous regions.  Some implementations of malloc() will
use mmap() for larger allocations.  If you use realloc() a
lot this can be particularly useful.

Releasing memory that has been allocated via brk() is
dependent on allocation order so it is seldom done.
Realloc() and co. used poorly can produce lots of
fragmentation exacerbating things.  Allocations done via
mmap() tend to be independent of one another so they can
released more readily.

If an application or library is using mmap() just watching
brk won't do much good.  Actually detecting leakage is very
difficult especially from outside the code.  It requires
tracking the modifications of all the pointers to allocated
memory.

The thing to keep in mind for the common case is that
applications that leak memory will release it on exit.
Most of the time any sizable amount of application memory is 
in disuse it will be in sizable chunks so under memory
pressure will be paged out until exit so will have minimal
impact on the system.

Having said all that i'll now echo Linus et al and tell you
that you want the system to use all of your memory.  Unused
memory is wasted memory and wasted money.  That is why
otherwise free memory is used as cache and disused memory
will be paged out so that it too can be used as cache to
speed up the system as a whole.

And much thanks to all the good, and heartless bastard,
developers who have and continue to work on improving the
paging algorithms and VM system as a whole.

-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw@pegasys.ws

		Remember Cernan and Schmitt

Memory leak

[Skip Gaede]

Folks,

I'm looking for a recipe on how to track where my memory is 
disappearing to. I am running kernel 2.4.19-pre10-ben0, 
patched for the nubus-pmac platform. I have 40 MB RAM and a 
100 MB swap file on the local hard disk.

I am using the MkLinux booter and the kernel with an initrd. 
During the first init, I get an IP address and the path to 
the NFS root. I then do a pivot-root, and run the XFree86 
Server -query 192.168.0.1, where I do an autologon, and 
start up KDE 3.0.1 and Mozilla 1.0 running Choffman's 
browser buster. I also am running the snmp daemon, so I can 
monitor memory use from my server with a perl script.

When first booted, the kernel takes about 16 MB of memory, 
and after starting the X server, I have about 2000k 
available physical memory, and no swap file useage. Over 
the next 7 hours, I eat up about 20 MB of swap file, and at 
some point the available physical memory drops down to 
about 150k for about an hour and then the system locks up.
During the same period of time, I can monitor memory use on 
the client on another console, and the amount of memory 
allocated to each process remains fairly stable.

I'd like to understand what's going on. Can someone suggest 
what data I ought to be collecting, or perhaps some 
parameters I can tweak to modify the behavior?

Thanks,
Skip