SMP-related kernel memory leak

SMP-related kernel memory leak

[Bart Van Assche]

Hello,

I noticed that the amount of memory used by the Linux kernel steadily
increases over time on SMP systems (x86 architecture, 32-bit kernel).
This problem disappears when I add maxcpus=1 to the kernel command
line. I have observed this behavior both on the 2.6.22.18 and 2.6.24.2
kernels. Did anyone notice anything similar ?

See also: http://bugzilla.kernel.org/show_bug.cgi?id=9991

Bart Van Assche.

Re: SMP-related kernel memory leak

[Oliver Pinter]

Helló!

this patch fixed them http://lkml.org/lkml/2008/2/18/405 ?

On 2/19/08, Bart Van Assche <bart.vanassche@gmail.com> wrote:
> Hello,
>
> I noticed that the amount of memory used by the Linux kernel steadily
> increases over time on SMP systems (x86 architecture, 32-bit kernel).
> This problem disappears when I add maxcpus=1 to the kernel command
> line. I have observed this behavior both on the 2.6.22.18 and 2.6.24.2
> kernels. Did anyone notice anything similar ?
>
> See also: http://bugzilla.kernel.org/show_bug.cgi?id=9991
>
> Bart Van Assche.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>


-- 
Thanks,
Oliver

Re: SMP-related kernel memory leak

[Bart Van Assche]

On Feb 19, 2008 7:18 PM, Oliver Pinter <oliver.pntr@gmail.com> wrote:
> On 2/19/08, Bart Van Assche <bart.vanassche@gmail.com> wrote:
> > I noticed that the amount of memory used by the Linux kernel steadily
> > increases over time on SMP systems (x86 architecture, 32-bit kernel).
> > This problem disappears when I add maxcpus=1 to the kernel command
> > line. I have observed this behavior both on the 2.6.22.18 and 2.6.24.2
> > kernels. Did anyone notice anything similar ?
> >
> > See also: http://bugzilla.kernel.org/show_bug.cgi?id=9991
>
> this patch fixed them http://lkml.org/lkml/2008/2/18/405 ?

Thanks for the hint. If I interpreted the 2.6.24 changelog correctly
this patch is already included with 2.6.24 ? The problem still occurs
with 2.6.24.2. I am currently trying to find the minimal kernel config
which still triggers this problem. Any other hints for finding the
cause of this issue are welcome of course.

Bart Van Assche.

Re: SMP-related kernel memory leak

[Oliver Pinter]

and add plus CC's
On 2/21/08, Oliver Pinter <oliver.pntr@gmail.com> wrote:
> it is reproductable with SLUB?
> /* sorry for the bad english, but i not learned it .. */
> On 2/21/08, Bart Van Assche <bart.vanassche@gmail.com> wrote:
> > On Thu, Feb 21, 2008 at 9:48 AM, Bart Van Assche
> > <bart.vanassche@gmail.com> wrote:
> > > On Wed, Feb 20, 2008 at 10:42 PM, Oliver Pinter <oliver.pntr@gmail.com>
> > wrote:
> > > >
> > > > hmm it is with slub or slab?
> > >
> > > All tests were performed with SLAB. Please note that it's not yet
> > > clear to me whether this is an issue with the SLAB allocator or
> > > another memory allocation mechanism. In the meantime I also noticed
> > > different behavior between the 2.6.22.18 and 2.6.24.2 kernel: with
> > > 2.6.22.18 I see unbounded growth of the memory used, while with
> > > 2.6.24.2 memory usage increases from about 30 MB to about 70 MB and
> > > then keeps at the same level. I am still performing more tests (a.o.
> > > minimizing the kernel config). I will add the results of these tests
> > > to the kernel bugzilla entry.
> >
> > I have added a new graph to
> > http://bugzilla.kernel.org/show_bug.cgi?id=9991, namely a graph
> > showing memory usage for a PAE-kernel booted with mem=1G and with a
> > minimized kernel config. The graph shows that memory usage increases
> > to a certain limit. Other tests have shown that this limit is
> > proportional to the amount of memory specified in mem=... This is not
> > a SLAB leak: as the numbers show, slab usage remains constant during
> > all tests.
> >
> > I'm puzzled by these results ...
> >
> > Bart.
> > --
> > Met vriendelijke groeten,
> >
> > Bart Van Assche.
> >
> --
> Thanks,
> Oliver
>
--
Thanks,
Oliver

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: SMP-related kernel memory leak

[H. Peter Anvin]

Oliver Pinter wrote:
>>> I have added a new graph to
>>> http://bugzilla.kernel.org/show_bug.cgi?id=9991, namely a graph
>>> showing memory usage for a PAE-kernel booted with mem=1G and with a
>>> minimized kernel config. The graph shows that memory usage increases
>>> to a certain limit. Other tests have shown that this limit is
>>> proportional to the amount of memory specified in mem=... This is not
>>> a SLAB leak: as the numbers show, slab usage remains constant during
>>> all tests.
>>>
>>> I'm puzzled by these results ...
>>>

This sounds to me a lot like the quicklist PUD leak we had, which I 
thought had been fixed in recent kernels...

It would be useful to know: does this happen with UP at all?

	-hpa

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: SMP-related kernel memory leak

[Bart Van Assche]

On Thu, Feb 21, 2008 at 10:40 PM, H. Peter Anvin <hpa@zytor.com> wrote:
> Oliver Pinter wrote:
>  >>> I have added a new graph to
>  >>> http://bugzilla.kernel.org/show_bug.cgi?id=9991, namely a graph
>  >>> showing memory usage for a PAE-kernel booted with mem=1G and with a
>  >>> minimized kernel config. The graph shows that memory usage increases
>  >>> to a certain limit. Other tests have shown that this limit is
>  >>> proportional to the amount of memory specified in mem=... This is not
>  >>> a SLAB leak: as the numbers show, slab usage remains constant during
>  >>> all tests.
>  >>>
>  >>> I'm puzzled by these results ...
>
>  This sounds to me a lot like the quicklist PUD leak we had, which I
>  thought had been fixed in recent kernels...
>
>  It would be useful to know: does this happen with UP at all?

The behavior I see is consistent for both the 2.6.22.18 and the
2.6.24.2 kernels. The problem does not occur when I boot with
maxcpus=1 added to the kernel command line.

Bart Van Assche.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: SMP-related kernel memory leak

[Oliver Pinter]

Hi!

what is the patch name or git ID?

On 2/21/08, H. Peter Anvin <hpa@zytor.com> wrote:
> Oliver Pinter wrote:
> >>> I have added a new graph to
> >>> http://bugzilla.kernel.org/show_bug.cgi?id=9991, namely a graph
> >>> showing memory usage for a PAE-kernel booted with mem=1G and with a
> >>> minimized kernel config. The graph shows that memory usage increases
> >>> to a certain limit. Other tests have shown that this limit is
> >>> proportional to the amount of memory specified in mem=... This is not
> >>> a SLAB leak: as the numbers show, slab usage remains constant during
> >>> all tests.
> >>>
> >>> I'm puzzled by these results ...
> >>>
>
> This sounds to me a lot like the quicklist PUD leak we had, which I
> thought had been fixed in recent kernels...
>
> It would be useful to know: does this happen with UP at all?
>
> -hpa
>
--
Thanks,
Oliver

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: SMP-related kernel memory leak

[H. Peter Anvin]

Oliver Pinter wrote:
> Hi!
> 
> what is the patch name or git ID?

96990a4ae979df9e235d01097d6175759331e88c

However, there was a second portion, 
421d99193537a6522aac2148286f08792167d5fd, which was then reverted at 
49eaaa1a6c950e7a92c4386c199b8ec950f840b9.

The fact that it doesn't happen on a single processor makes me believe 
it's still a problem with the quicklists not getting freed properly.  It 
would be nice if someone could go in with system tap or just plain
"gdb vmlinux /proc/kcore" and verify if there is a large number of pages 
queued up on the quicklists on some of the CPUs, while at least one of 
them is zero.

(It would be nice to have quicklist statistics exported somewhere, too.)

	-hpa

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: SMP-related kernel memory leak

[Oliver Pinter]

hi thanks,
but 421d99193537a6522aac2148286f08792167d5fd is never in 2.6.22.y  and
nor stable-queue-2.6.22.y ...

On 2/22/08, H. Peter Anvin <hpa@zytor.com> wrote:
> Oliver Pinter wrote:
> > Hi!
> >
> > what is the patch name or git ID?
>
> 96990a4ae979df9e235d01097d6175759331e88c
>
> However, there was a second portion,
> 421d99193537a6522aac2148286f08792167d5fd, which was then reverted at
> 49eaaa1a6c950e7a92c4386c199b8ec950f840b9.
>
> The fact that it doesn't happen on a single processor makes me believe
> it's still a problem with the quicklists not getting freed properly.  It
> would be nice if someone could go in with system tap or just plain
> "gdb vmlinux /proc/kcore" and verify if there is a large number of pages
> queued up on the quicklists on some of the CPUs, while at least one of
> them is zero.
>
> (It would be nice to have quicklist statistics exported somewhere, too.)
>
> 	-hpa
>


-- 
Thanks,
Oliver

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: SMP-related kernel memory leak

[H. Peter Anvin]

Oliver Pinter wrote:
> hi thanks,
> but 421d99193537a6522aac2148286f08792167d5fd is never in 2.6.22.y  and
> nor stable-queue-2.6.22.y ...

That's a serious problem.  This is a critical bug.

	-hpa

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: SMP-related kernel memory leak

[Bart Van Assche]

On Sat, Feb 23, 2008 at 12:22 AM, H. Peter Anvin <hpa@zytor.com> wrote:
> Oliver Pinter wrote:
>  > hi thanks,
>  > but 421d99193537a6522aac2148286f08792167d5fd is never in 2.6.22.y  and
>  > nor stable-queue-2.6.22.y ...
>
>  That's a serious problem.  This is a critical bug.

The patch referenced above modifies a single file:
include/asm-generic/tlb.h. I had a look at the git history of Linus'
2.6.24 tree, and noticed that the cited patch was applied on December
18, 2007 to Linus' tree by Christoph Lameter. Two weeks later, on
December 27, the change was reverted by Christoph. Christoph, can you
provide us some more background information about why the patch was
reverted ?

See also:
http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.24.y.git;a=history;f=include/asm-generic/tlb.h;h=75f2bfab614f40639090702a2a6268f34864df75;hb=HEAD

Bart Van Assche.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: SMP-related kernel memory leak

[Christoph Lameter]

On Sat, 23 Feb 2008, Bart Van Assche wrote:

> The patch referenced above modifies a single file:
> include/asm-generic/tlb.h. I had a look at the git history of Linus'
> 2.6.24 tree, and noticed that the cited patch was applied on December
> 18, 2007 to Linus' tree by Christoph Lameter. Two weeks later, on
> December 27, the change was reverted by Christoph. Christoph, can you
> provide us some more background information about why the patch was
> reverted ?

It did not fix the problem that was reported. Commit 
96990a4ae979df9e235d01097d6175759331e88c took its place.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: SMP-related kernel memory leak

[Bart Van Assche]

On Wed, Feb 27, 2008 at 8:40 PM, Christoph Lameter <clameter@sgi.com> wrote:
> On Sat, 23 Feb 2008, Bart Van Assche wrote:
>
>  > The patch referenced above modifies a single file:
>  > include/asm-generic/tlb.h. I had a look at the git history of Linus'
>  > 2.6.24 tree, and noticed that the cited patch was applied on December
>  > 18, 2007 to Linus' tree by Christoph Lameter. Two weeks later, on
>  > December 27, the change was reverted by Christoph. Christoph, can you
>  > provide us some more background information about why the patch was
>  > reverted ?
>
>  It did not fix the problem that was reported. Commit
>  96990a4ae979df9e235d01097d6175759331e88c took its place.

I did a grep through the kernel changelogs for this commit:

$ grep 96990a4ae979df9e235d01097d6175759331e88c *
ChangeLog-2.6.22.19:    patch 96990a4ae979df9e235d01097d6175759331e88c
in mainline.
ChangeLog-2.6.23.15:    patch 96990a4ae979df9e235d01097d6175759331e88c
in mainline.
ChangeLog-2.6.24:commit 96990a4ae979df9e235d01097d6175759331e88c

Or: this commit should have been included in kernel 2.6.24. But the
PAE memory leak I observed still occurs with 2.6.24.2. So the question
remains: is there already a fix available for the problem I observed ?

Bart Van Assche.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: SMP-related kernel memory leak

[Christoph Lameter]

On Thu, 21 Feb 2008, H. Peter Anvin wrote:

> This sounds to me a lot like the quicklist PUD leak we had, which I thought
> had been fixed in recent kernels...

This was a pgd leak only AFAICT.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: SMP-related kernel memory leak

[H. Peter Anvin]

Christoph Lameter wrote:
> On Thu, 21 Feb 2008, H. Peter Anvin wrote:
> 
>> This sounds to me a lot like the quicklist PUD leak we had, which I thought
>> had been fixed in recent kernels...
> 
> This was a pgd leak only AFAICT.
> 

This was for 3-level page tables, so PGD == PUD.

	-hpa

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: SMP-related kernel memory leak

[Bart Van Assche]

On Thu, Feb 21, 2008 at 5:21 PM, Oliver Pinter <oliver.pntr@gmail.com> wrote:
> it is reproductable with SLUB?
>  /* sorry for the bad english, but i not learned it .. */

The behavior with SLUB is similar but slightly different than the SLAB
behavior: with SLUB I see that memory usage increases faster and the
upper limit is a little bit lower. I will add the updated graph to the
bugzilla entry.

Bart Van Assche.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>