weird memory related problems, negative memory usage or fake memory usage?

weird memory related problems, negative memory usage or fake memory usage?

[Olivier Sessink]

Hi all,

after upgrade from 2.4.10pre8 to 2.4.10 I have weird problems, Xfree
sometimes shows up with 99.9% memory in top (on a box with 512 mb), and in
ps axl it has 4294989036 in the RSS column. When this happens the box starts
to kill some processes, starts heavily swapping (top reports > 400MB in the
cache, but the machine is heavily swapping!!!) and is completely unusable.

The problems is triggered when I start edonkey, some filesharing program
that creates checksums of large files (a couple of > 700Mb files).

Since this makes the machine completely unusable, and since it is not
happening on 2.4.10pre8 I guess it is a bug ;-)

This is a top snapshot:

46 processes: 43 sleeping, 3 running, 0 zombie, 0 stopped
CPU states:   3.8% user,   1.2% system,  94.6% nice,   0.4% idle
Mem:    514032K total,   511436K used,     2596K free,     1244K buffers
Swap:   358808K total,    10928K used,   347880K free,   466072K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
  253 olivier   19  18   300  196   124 R N  52.9  0.0 119:50 dnetc
 1274 root      14   0  5940 5940   872 R    33.7  1.1   0:00 xsetbg
 1262 root       5 -10 50764  -1M  1320 S <   2.7 99.9   0:01 XFree86
 1263 root       9   0  1528 1492  1204 S     2.7  0.2   0:00 xdm
 1267 root      11   0   980  976   776 R     2.7  0.1   0:00 top
 1270 root       9   0  1016 1012   820 S     0.9  0.1   0:00 Xsetup_0
    1 root       8   0   104   56    36 S     0.0  0.0   0:04 init
    2 root       9   0     0    0     0 SW    0.0  0.0   0:00 keventd
    3 root       9   0     0    0     0 SW    0.0  0.0   0:00 kapm-idled
    4 root      19  19     0    0     0 SWN   0.0  0.0   0:00 ksoftirqd_CPU0
    5 root       9   0     0    0     0 SW    0.0  0.0   0:01 kswapd
    6 root       9   0     0    0     0 SW    0.0  0.0   0:00 bdflush
    7 root       9   0     0    0     0 SW    0.0  0.0   0:00 kupdated
    8 root       9   0     0    0     0 SW    0.0  0.0   0:00 kreiserfsd

the ps axl for X then is

100     0  1262  1260   5 -10 52408 4294965424 select S< ?      0:01
/usr/X11R6/bin/X vt7 -dpi 100 -nolisten tcp -auth /var/lib/xdm/authdir

regards,
	Olivier

Re: weird memory related problems, negative memory usage or fake memory usage?

[Rik van Riel]

[Andrea, please read this bugreport ...]

On Mon, 24 Sep 2001, Olivier Sessink wrote:

> after upgrade from 2.4.10pre8 to 2.4.10 I have weird problems,
> Xfree sometimes shows up with 99.9% memory in top (on a box with
> 512 mb), and in ps axl it has 4294989036 in the RSS column. When
> this happens the box starts to kill some processes, starts
> heavily swapping (top reports > 400MB in the cache, but the
> machine is heavily swapping!!!) and is completely unusable.

> Since this makes the machine completely unusable, and since it is not
> happening on 2.4.10pre8 I guess it is a bug ;-)

>   PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
>  1262 root       5 -10 50764  -1M  1320 S <   2.7 99.9   0:01 XFree86


It seems Andrea wasn't careful with the merge and
backed out some of the locking wrt mm->rss.

Andrea, you may want to spend some time auditing
your VM like has been done with the other 2.4 VM.

cheers,

Rik
--
IA64: a worthy successor to the i860.

		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com/

Re: weird memory related problems, negative memory usage or fake memory usage?

[Andrea Arcangeli]

On Mon, Sep 24, 2001 at 07:03:20PM -0300, Rik van Riel wrote:
> [Andrea, please read this bugreport ...]
> 
> On Mon, 24 Sep 2001, Olivier Sessink wrote:
> 
> > after upgrade from 2.4.10pre8 to 2.4.10 I have weird problems,
> > Xfree sometimes shows up with 99.9% memory in top (on a box with
> > 512 mb), and in ps axl it has 4294989036 in the RSS column. When
> > this happens the box starts to kill some processes, starts
> > heavily swapping (top reports > 400MB in the cache, but the
> > machine is heavily swapping!!!) and is completely unusable.
> 
> > Since this makes the machine completely unusable, and since it is not
> > happening on 2.4.10pre8 I guess it is a bug ;-)
> 
> >   PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
> >  1262 root       5 -10 50764  -1M  1320 S <   2.7 99.9   0:01 XFree86
> 
> 
> It seems Andrea wasn't careful with the merge and
> backed out some of the locking wrt mm->rss.

thanks for forwarding this report, actually I just noticed this here and
that's good so I can reproduce :)

it is possible it is my mistake, but I don't think so, infact I don't
recall to have changed rss stuff or locking around it. Incidentally the
first time I reproduced it here was after the tlb shootdown patch from
Ben was introduced, never reproduced it here previously with only my
changes.  However it is possibly just a coincidence.

> 
> Andrea, you may want to spend some time auditing
> your VM like has been done with the other 2.4 VM.
> 
> cheers,
> 
> Rik
> --
> IA64: a worthy successor to the i860.
> 
> 		http://www.surriel.com/
> http://www.conectiva.com/	http://distro.conectiva.com/
> 
> 


Andrea

Re: weird memory related problems, negative memory usage or fake memory usage?

[Rik van Riel]

On Wed, 26 Sep 2001, Andrea Arcangeli wrote:
> On Mon, Sep 24, 2001 at 07:03:20PM -0300, Rik van Riel wrote:
> > On Mon, 24 Sep 2001, Olivier Sessink wrote:

> > >   PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
> > >  1262 root       5 -10 50764  -1M  1320 S <   2.7 99.9   0:01 XFree86
> >
> > It seems Andrea wasn't careful with the merge and
> > backed out some of the locking wrt mm->rss.
>
> thanks for forwarding this report, actually I just noticed this
> here and that's good so I can reproduce :)
>
> it is possible it is my mistake, but I don't think so, infact I
> don't recall to have changed rss stuff or locking around it.

Mmm, then it could also be one of the bugs which got
fixed in -ac but where Linus never reacted to the
patch, IIRC the RSS thing was indeed fixed around the
time where Linus was in the habbit of silently dropping
half of the patches sent to him...

regards,

Rik
--
IA64: a worthy successor to the i860.

		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com/

Re: weird memory related problems, negative memory usage or fake memory usage?

[Andrea Arcangeli]

On Tue, Sep 25, 2001 at 07:54:07PM -0300, Rik van Riel wrote:
> On Wed, 26 Sep 2001, Andrea Arcangeli wrote:
> > On Mon, Sep 24, 2001 at 07:03:20PM -0300, Rik van Riel wrote:
> > > On Mon, 24 Sep 2001, Olivier Sessink wrote:
> 
> > > >   PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME COMMAND
> > > >  1262 root       5 -10 50764  -1M  1320 S <   2.7 99.9   0:01 XFree86
> > >
> > > It seems Andrea wasn't careful with the merge and
> > > backed out some of the locking wrt mm->rss.
> >
> > thanks for forwarding this report, actually I just noticed this
> > here and that's good so I can reproduce :)
> >
> > it is possible it is my mistake, but I don't think so, infact I
> > don't recall to have changed rss stuff or locking around it.
> 
> Mmm, then it could also be one of the bugs which got
> fixed in -ac but where Linus never reacted to the

possibly yes but maybe not, dunno right now or I would be just sending
the fix inline in this email :).  As said I never seen it before Ben's
tlb shootdown was merged into mainline, but again I repeat it can
_really_ be just an unlucky coincidence. But I guess because of this
coincidence the tlb shootdown will be the first things I will audit
tomorrow.

Andrea

[patch] Re: weird memory related problems, negative memory usage or fake memory usage?

[Benjamin LaHaise]

On Wed, Sep 26, 2001 at 01:11:16AM +0200, Andrea Arcangeli wrote:
> possibly yes but maybe not, dunno right now or I would be just sending
> the fix inline in this email :).  As said I never seen it before Ben's
> tlb shootdown was merged into mainline, but again I repeat it can
> _really_ be just an unlucky coincidence. But I guess because of this
> coincidence the tlb shootdown will be the first things I will audit
> tomorrow.

This should fix it.

		-ben

... v2.4.10-rss.diff ...
diff -urN v2.4.10/mm/memory.c foo/mm/memory.c
--- v2.4.10/mm/memory.c	Mon Sep 24 02:16:05 2001
+++ foo/mm/memory.c	Tue Sep 25 20:03:04 2001
@@ -319,7 +319,9 @@
 		if (pte_none(pte))
 			continue;
 		if (pte_present(pte)) {
-			freed ++;
+			struct page *page = pte_page(pte);
+			if (!PageReserved(page) && VALID_PAGE(page))
+				freed ++;
 			/* This will eventually call __free_pte on the pte. */
 			tlb_remove_page(tlb, ptep, address + offset);
 		} else {

Re: [patch] Re: weird memory related problems, negative memory usage or fake memory usage?

[Hugh Dickins]

On Tue, 25 Sep 2001, Benjamin LaHaise wrote:
> 
> This should fix it.
> 
> -			freed ++;
> +			struct page *page = pte_page(pte);
> +			if (!PageReserved(page) && VALID_PAGE(page))
> +				freed ++;

NO!  I expect that fixes the rss, but crashes my system: 
must test VALID_PAGE(page) _before_ !PageReserved(page).

Hugh

--- linux-2.4.10/mm/memory.c	Sun Sep 23 04:36:50 2001
+++ linux/mm/memory.c	Thu Sep 27 14:29:28 2001
@@ -319,7 +319,9 @@
 		if (pte_none(pte))
 			continue;
 		if (pte_present(pte)) {
-			freed ++;
+			struct page *page = pte_page(pte);
+			if (VALID_PAGE(page) && !PageReserved(page))
+				freed ++;
 			/* This will eventually call __free_pte on the pte. */
 			tlb_remove_page(tlb, ptep, address + offset);
 		} else {