Question on Memory Leak: very tough problem

Question on Memory Leak: very tough problem

[Sri Ram Vemulpali]

Hi All,

I have very difficult problem I am working on to figure out where the
memory leak is in my system. Sometimes it shows up and sometimes it
reveals very slowly (I mean increments very slowly). This is dependent
on system load, if the system is in very high load, then leak is very
fast. We tried solving this problem from many dimensions, but no
success.

After quite sometime, we started using valgrind to capture the memory
leak. The funny thing is the leak disappears once we start running our
application on top of valgrind.
So, we cannot capture leak using valgrind. We came to conclusion that,
since valgrind serializes the allocations (leak disappears), we
thought this might be result of race conditions, that
resulted from system high load (by the way ours is multi threaded
system). We run system without valgrind we see the memory leak.

The question is, is it true that valgrind serializes the mem
allocations. Does this happen to anyone disappearing the leak when
used valgrind.

Any thoughts and input that can help me solve this problem. Thanks to
all in advance.


-- 
Regards,
Sri.

Re: Question on Memory Leak: very tough problem

[Dave Hylands]

Hi Sri,

On Wed, Oct 26, 2011 at 8:20 PM, Sri Ram Vemulpali
<sri.ram.gmu06@gmail.com> wrote:
>
> Hi All,
>
> I have very difficult problem I am working on to figure out where the
> memory leak is in my system. Sometimes it shows up and sometimes it
> reveals very slowly (I mean increments very slowly). This is dependent
> on system load, if the system is in very high load, then leak is very
> fast. We tried solving this problem from many dimensions, but no
> success.
>
> After quite sometime, we started using valgrind to capture the memory
> leak. The funny thing is the leak disappears once we start running our
> application on top of valgrind.
> So, we cannot capture leak using valgrind. We came to conclusion that,
> since valgrind serializes the allocations (leak disappears), we
> thought this might be result of race conditions, that
> resulted from system high load (by the way ours is multi threaded
> system). We run system without valgrind we see the memory leak.
>
> The question is, is it true that valgrind serializes the mem
> allocations. Does this happen to anyone disappearing the leak when
> used valgrind.
>
> Any thoughts and input that can help me solve this problem. Thanks to
> all in advance.

A couple of possibilities spring to mind.

It might be that the leak is being caused by some uninitialized
variables/memory. Running under valgrind may be causing the
unitialized variables to change, which in turn causes the behavior to
change.

It also may be that you don't have a leak, but in fact have heap
fragmentation, which can look like a leak.

--
Dave Hylands
Shuswap, BC, Canada
http://www.davehylands.com

Re: Question on Memory Leak: very tough problem

[Bernd Petrovitsch]

Hi!

On Mit, 2011-10-26 at 23:20 -0400, Sri Ram Vemulpali wrote:
> I have very difficult problem I am working on to figure out where the
> memory leak is in my system. Sometimes it shows up and sometimes it
> reveals very slowly (I mean increments very slowly). This is dependent
> on system load, if the system is in very high load, then leak is very
> fast. We tried solving this problem from many dimensions, but no
> success.
> 
> After quite sometime, we started using valgrind to capture the memory
> leak. The funny thing is the leak disappears once we start running our
> application on top of valgrind.

You do speak of an user-space application?

> So, we cannot capture leak using valgrind. We came to conclusion that,
> since valgrind serializes the allocations (leak disappears), we
> thought this might be result of race conditions, that
> resulted from system high load (by the way ours is multi threaded
> system). We run system without valgrind we see the memory leak.

That really smells like a race condition. So you should make sure that
you got the synchronization between the threads correct and also the
pointer handling in your application. The memory leak seems to be either
a forgotten free() or an overwritten already allocated pointer as in
----  snip ----
void whatever(....)
{
	static void *p = NULL;
	if (p == NULL) {
		p = malloc(12345);
	}
}
----  snip ----
If that function is called in 2 threads in parallel and a task switch
occurs after the check for NULL and before the assignment afterwards,
you've got a memory leak.

That pattern works (at least) also for global pointers and pointers in
shared data structures.

> The question is, is it true that valgrind serializes the mem
> allocations. Does this happen to anyone disappearing the leak when
> used valgrind.

The manual page of malloc() doesn't mentions anything on threads so I
assume that malloc() is synchronized with itself (read: it is
thread-safe) and free(),realloc(),calloc(),strdup(), and all similar
functions.

Apart from valgrind, there are other memory debugging libs out there
(e.g. dmalloc) which may have far less impact on the systems
performance.
Depending on the skill, time and desperation level, building an
(somewhat old) gcc with an applied "bounds-checking" patch may help more
(but you may also run in Heisenberg-problems ....).

	Bernd
-- 
Bernd Petrovitsch                  Email : bernd@petrovitsch.priv.at
                     LUGA : http://www.luga.at