kernel memory leak?

kernel memory leak?

[Dow, Benjamin]

I (read: my company) am working on a semi-embedded system (no swap, booting
onto a ramdisk), and I recently changed linux kernels from 2.4.9 to 2.4.19.
Nearly everything works fine, except that whenever I read a file, it
allocates 4k of memory (cat /proc/meminfo reports 4k less memory, and
/proc/slabinfo reports that buffer_head increases by 4k, but those are the
only changes).  I found a thread from back in July or so on LKML where
somebody had a similar problem on 2.4.19-rc3, and Rik said that this was
normal, and the vm would release this memory under pressure, except when I
start a bunch of processes to use up memory and repeatedly cat
/proc/meminfo, the OOM killer is eventually triggered.

I'm running 2.4.19 with both the low-latency and kernel preemption patches
applied (though I recompiled the kernel with those disabled and the problem
still occurred), as well as a few changes of our own (though none of those
seem to touch the filesystem/vm layers).  It's a powerpc chip with 64 megs
of ram (roughly 28 of which are free after we load the kernel/ramdisk and
all our apps).  I tried to reproduce the problem on my PC (booted up without
X, specified mem=8M on command line, and turned off swap), which is also
2.4.19 and also has the kernel preemption and low-latency patches applied,
but is not booting from a ramdisk and is, obviously, a different
architecture, and it did allocate 4k blocks for a while, but then stopped
(or, more accurately, it would allocate 4k and then the next time I did a
cat /proc/meminfo it would be back to its original value).

So I guess my question is:
1) What is eating up memory and not allowing it to be freed, and
2) Why does it allocate 4k of memory every single time I access a file,
anyway?

Any help would be greatly appreciated.

RE: kernel memory leak?

[Dow, Benjamin]

I sent this out yesterday, and never got a reply.  Normally, I'd be far more
patient, but I'm afraid that by now people have forgotten about it, and I'm
under a LOT of pressure by management to get this fixed soon.

To recap my previous e-mail, every time I access a file, 4k of memory gets
allocated, and never gets freed, to the point of eventually triggering the
OOM killer.  I don't know nearly enough about the VM to debug this myself,
so even a pointer to where to start looking would be helpful.

Thanks in advance

Ben




-----Original Message-----
From: Dow, Benjamin [mailto:bdow@itouchcom.com]
Sent: Monday, October 07, 2002 5:59 PM
To: 'linux-kernel@vger.kernel.org'
Subject: kernel memory leak?


I (read: my company) am working on a semi-embedded system (no swap, booting
onto a ramdisk), and I recently changed linux kernels from 2.4.9 to 2.4.19.
Nearly everything works fine, except that whenever I read a file, it
allocates 4k of memory (cat /proc/meminfo reports 4k less memory, and
/proc/slabinfo reports that buffer_head increases by 4k, but those are the
only changes).  I found a thread from back in July or so on LKML where
somebody had a similar problem on 2.4.19-rc3, and Rik said that this was
normal, and the vm would release this memory under pressure, except when I
start a bunch of processes to use up memory and repeatedly cat
/proc/meminfo, the OOM killer is eventually triggered.

I'm running 2.4.19 with both the low-latency and kernel preemption patches
applied (though I recompiled the kernel with those disabled and the problem
still occurred), as well as a few changes of our own (though none of those
seem to touch the filesystem/vm layers).  It's a powerpc chip with 64 megs
of ram (roughly 28 of which are free after we load the kernel/ramdisk and
all our apps).  I tried to reproduce the problem on my PC (booted up without
X, specified mem=8M on command line, and turned off swap), which is also
2.4.19 and also has the kernel preemption and low-latency patches applied,
but is not booting from a ramdisk and is, obviously, a different
architecture, and it did allocate 4k blocks for a while, but then stopped
(or, more accurately, it would allocate 4k and then the next time I did a
cat /proc/meminfo it would be back to its original value).

So I guess my question is:
1) What is eating up memory and not allowing it to be freed, and
2) Why does it allocate 4k of memory every single time I access a file,
anyway?

Any help would be greatly appreciated.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: kernel memory leak?

[Pavel Machek]

Hi!

> I sent this out yesterday, and never got a reply.  Normally, I'd be far more
> patient, but I'm afraid that by now people have forgotten about it, and I'm
> under a LOT of pressure by management to get this fixed soon.
> 
> To recap my previous e-mail, every time I access a file, 4k of memory gets
> allocated, and never gets freed, to the point of eventually triggering the
> OOM killer.  I don't know nearly enough about the VM to debug this myself,
> so even a pointer to where to start looking would be helpful.

Write C code to reproduce this on normal machine, and post it to
bugtraq (its DoS, after all). Pretty aggresive but sure to get fixed
*fast*.

The same without going bugtraq should suffice, through.
									Pavel

-- 
I'm pavel@ucw.cz. "In my country we have almost anarchy and I don't care."
Panos Katsaloulis describing me w.r.t. patents at discuss@linmodems.org

RE: kernel memory leak?

[Dow, Benjamin]

<snip>

> Write C code to reproduce this on normal machine, and post it to
> bugtraq (its DoS, after all). Pretty aggresive but sure to get fixed
> *fast*.
> 
> The same without going bugtraq should suffice, through.
> 								
> 	Pavel
> 
> -- 
> I'm pavel@ucw.cz. "In my country we have almost anarchy and I 
> don't care."
> Panos Katsaloulis describing me w.r.t. patents at 
> discuss@linmodems.org
> 

That's the problem; I haven't been able to reproduce it yet on a "normal"
machine... I'm beginning to think it was a change to 2.4.9 that had some
unexpected side-effect in 2.4.19, though that doesn't really sit well with
me either.  I've been wracking my brain over this for days now, and
nothing's making any sense... I even tried applying different patches from
Andrea and Rik to change the VM behavior (since that was one of the biggest
changes between 2.4.9 and 19), but nothing's made a bit of difference.
Sorry, I'm just a bit frustrated.

Ben



 The information contained in this electronic mail is privileged and
confidential, intended only for the use of the individual or entity named
above. If the reader of this message is not the intended recipient, you are
hereby notified that any dissemination, distribution, copying or other use
of this communication is strictly prohibited. 

RE: kernel memory leak?

[Manfred Spraul]

 >
 > /proc/slabinfo reports that buffer_head increases by 4k, but
 > those are the only changes)
 >
4k or one object? the first column ist the number of objects, not 4k.

--
	Manfred

RE: kernel memory leak?

[Dow, Benjamin]

>  >
>  > /proc/slabinfo reports that buffer_head increases by 4k, but
>  > those are the only changes)
>  >
> 4k or one object? the first column ist the number of objects, not 4k.
> 

My mistake; I didn't understand slabinfo very well at the time... and
(embarassingly enough) that was related to my piping the output to a file
(in order to do a diff later) rather than a regularly-occurring phenomenon.

As for my memory leak problem, I did finally track it down to the
kernel-preemption patch... my best guess is that it interacts poorly with
some of our platform-specific modifications... though what I can't figure
out is why even with preemption disabled (in menuconfig), I still have the
problem.  I'm looking through the patch again to try to find additions that
aren't #ifdef'd out when it's disabled, but I have a feeling that since I've
found a way to fix the problem, I'll be moving on to other things soon.
<sigh>

Anyway, thanks to everyone who offered advice, and if any kernel preemption
experts want to set me straight, feel free!

Ben

(oh, and I apologize for the signature that the mail server apparently just
started appending to my messages... I can't do anything about it)
(how about that, a disclaimer for a disclaimer?)



 The information contained in this electronic mail is privileged and
confidential, intended only for the use of the individual or entity named
above. If the reader of this message is not the intended recipient, you are
hereby notified that any dissemination, distribution, copying or other use
of this communication is strictly prohibited. 

Re: kernel memory leak?

[Nicholas Hockey]

i am having a similar problem, i'm thinking somthing in the XFS software is
malloc()ing ram and not letting it go, by any chance are you using XFS ?
what i did was write this lil things to recover my ram,it eats ram till it's
killed effectivly recovering lost ram (i just had to include this)
---ayrabtu.c---
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[]) {
        char *allyourram;
        char arebelongtous[] = "all your ram are belong to us";

        while (1) {
                allyourram = malloc(30);
                sprintf(allyourram, "%s", arebelongtous);
        }

        return(0);
}
---EOF---

Re: kernel memory leak?

[Nicholas Hockey]

i am having a similar problem, i'm thinking somthing in the XFS software is
malloc()ing ram and not letting it go, by any chance are you using XFS ?
what i did was write this lil things to recover my ram,it eats ram till it's
killed effectivly recovering lost ram (i just had to include this)
---ayrabtu.c---
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[]) {
        char *allyourram;
        char arebelongtous[] = "all your ram are belong to us";

        while (1) {
                allyourram = malloc(30);
                sprintf(allyourram, "%s", arebelongtous);
        }

        return(0);
}
---EOF---
(sorry if this posts twice)

Kernel memory leak?

[Andreas Rieke]

Hi,

after booting a machine, it runs well using about 300 M of 1 G physical
RAM. However, the remaining RAM decreases day by day, and after 2 or 3
weeks, the machine crashes because swapping takes too much time.
However, all processes together take about 250 MBytes according to ps,
thus I assume that the kernel takes the rest. free tells me in fact that
much swap space is used an nearly no physical RAM is left.

This behaviour has been seen on Red Hat Enterprise Linux 3 with a 2.4
kernel and on SuSE Linux 10 with a 2.6.13-15-default kernel. There are
no unusual things running on the machine, the main application is an
apache web server with a PostgreSQL database.

Is there any kernel support to detect where the memory has gone?
Is any kind of memory eating virus or worm known?
Is it possible that processes request memory which is NOT considered in
/proc or in the procps tools?
Is it possible that processes are invisible in /proc or in the procps tools?

Thanks in advance,

Andreas

Re: Kernel memory leak?

[Michal Piotrowski]

Hi Andreas,

On 17/07/06, Andreas Rieke <andreas.rieke@isl.de> wrote:
> Hi,
>
> after booting a machine, it runs well using about 300 M of 1 G physical
> RAM. However, the remaining RAM decreases day by day, and after 2 or 3
> weeks, the machine crashes because swapping takes too much time.
> However, all processes together take about 250 MBytes according to ps,
> thus I assume that the kernel takes the rest. free tells me in fact that
> much swap space is used an nearly no physical RAM is left.
>
> This behaviour has been seen on Red Hat Enterprise Linux 3 with a 2.4
> kernel and on SuSE Linux 10 with a 2.6.13-15-default kernel. There are
> no unusual things running on the machine, the main application is an
> apache web server with a PostgreSQL database.
>
> Is there any kernel support to detect where the memory has gone?

Yes, the kmemleak patches http://homepage.ntlworld.com/cmarinas/kmemleak/

> Is any kind of memory eating virus or worm known?
> Is it possible that processes request memory which is NOT considered in
> /proc or in the procps tools?
> Is it possible that processes are invisible in /proc or in the procps tools?

Yes - if you have a rootkit.

>
> Thanks in advance,
>
> Andreas

Regards,
Michal

-- 
Michal K. K. Piotrowski
LTG - Linux Testers Group
(http://www.stardust.webpages.pl/ltg/wiki/)