buffer/memory strangeness in 2.4.16

buffer/memory strangeness in 2.4.16

[space-00002]

Hi,

I am experiencing a bit of strange system behaviour in a vanilla 2.4.16 
kernel (2.95.3, very stable machine etc.)

I noticed, that after running for a while (day) I had significantly less 
memory available for my simulation program than right after booting. Looking 
at the problem using 'xosview' (or 'free'), I noticed that there was a large 
number of MBs filled with 'buffers' that did not get wiped when other 
programs need the memory. The system seems to rather kill an 'offender' than 
clean out buffers.

Right after booting, I can allocate about 650MBs memory using the little 
program attached below. After a day (or after running updatedb), under the 
same conditions, even in single user mode with only a shell running (!) this 
is not possible anymore and the program (below), trying to allocate only 
300-400MBs, gets killed by the system after making it unresponsive for many 
seconds.

Apparently this problem occurs after running 'updatedb', which fills 'free 
memory' and generates lots of filled cache and buffers on my system.

This sort of behaviour must have been introduced after 2.4.13, which does not 
show these problems.

Please tell me if somebody needs more information to debug this, or if this 
behaviour is normal or expected. Please cc: me as I am only on lkml-digest.

Cheers
	Jan


P.S. All RAM slots are full, so please don't suggest buying more memory as a 
solution :^)

-------------------%<-----------------------

#include <stdio.h>
#define ONE_MEG 1024 * 1024

main ()
{
        long mem_avail = ONE_MEG;
        char *buf;
        char userchar = '@';
        int howmany;

        while (1)
        {
                printf ("Number of MBs to allocate? ");
                scanf ("%d", &howmany);
                printf ("Trying to allocate %ld bytes: ", mem_avail*howmany);

                getchar ();
                if ((buf = (char *) malloc ((size_t) mem_avail*howmany))){
                        printf (" success!\n");
                        printf ("Now filling it up...\n");
                        memset (buf, userchar, mem_avail * howmany);
                        printf ("Hit ENTER to free the memory.\n");
                        getchar ();
                        free (buf);
                } else {
                        printf (" failed :(\n");
                }
        }
}

-------------------%<-----------------------

Re: buffer/memory strangeness in 2.4.16

[Andrew Morton]

space-00002@vortex.physik.uni-konstanz.de wrote:
> 
> Hi,
> 
> I am experiencing a bit of strange system behaviour in a vanilla 2.4.16
> kernel (2.95.3, very stable machine etc.)
> 
> I noticed, that after running for a while (day) I had significantly less
> memory available for my simulation program than right after booting. Looking
> at the problem using 'xosview' (or 'free'), I noticed that there was a large
> number of MBs filled with 'buffers' that did not get wiped when other
> programs need the memory. The system seems to rather kill an 'offender' than
> clean out buffers.
> 
>

Seconded.   After an updatedb run, my 768 megabyte 2.5.1-pre4 machine
shows:

             total       used       free     shared    buffers     cached
Mem:        770668     384460     386208          0     138548      17744
-/+ buffers/cache:     228168     542500

and, after malloc/memset of 700 megs:

             total       used       free     shared    buffers     cached
Mem:        770668      73340     697328          0      41160       5960
-/+ buffers/cache:      26220     744448
Swap:       499928      18628     481300

I repeated the malloc/memset a few times, wrote a gigabyte file
and was unable to make the 40 megabytes of buffermem go away.

MemTotal:       770668 kB
MemFree:        698008 kB
MemShared:           0 kB
Buffers:         42092 kB
Cached:           6088 kB
SwapCached:       9808 kB
Active:          48064 kB
Inactive:        10112 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       770668 kB
LowFree:        698008 kB
SwapTotal:      499928 kB
SwapFree:       484512 kB

After running an extremely memory-intensive test program for
two minutes, buffermem fell to 38 megabytes.

Seems broken to me.

-

Re: buffer/memory strangeness in 2.4.16 / 2.4.17pre2

[space-00002]

It is still the same in 2.4.17pre2 :-(

On Friday 30 November 2001 10:29, Andrew Morton wrote:
> space-00002@vortex.physik.uni-konstanz.de wrote:
> > Hi,
> >
> > I am experiencing a bit of strange system behaviour in a vanilla 2.4.16
> > kernel (2.95.3, very stable machine etc.)
> >
> > I noticed, that after running for a while (day) I had significantly less
> > memory available for my simulation program than right after booting.
> > Looking at the problem using 'xosview' (or 'free'), I noticed that there
> > was a large number of MBs filled with 'buffers' that did not get wiped
> > when other programs need the memory. The system seems to rather kill an
> > 'offender' than clean out buffers.
>
> Seconded.   After an updatedb run, my 768 megabyte 2.5.1-pre4 machine
> shows:
>
>              total       used       free     shared    buffers     cached
> Mem:        770668     384460     386208          0     138548      17744
> -/+ buffers/cache:     228168     542500
>
> and, after malloc/memset of 700 megs:
>
>              total       used       free     shared    buffers     cached
> Mem:        770668      73340     697328          0      41160       5960
> -/+ buffers/cache:      26220     744448
> Swap:       499928      18628     481300
>
> I repeated the malloc/memset a few times, wrote a gigabyte file
> and was unable to make the 40 megabytes of buffermem go away.
>
> MemTotal:       770668 kB
> MemFree:        698008 kB
> MemShared:           0 kB
> Buffers:         42092 kB
> Cached:           6088 kB
> SwapCached:       9808 kB
> Active:          48064 kB
> Inactive:        10112 kB
> HighTotal:           0 kB
> HighFree:            0 kB
> LowTotal:       770668 kB
> LowFree:        698008 kB
> SwapTotal:      499928 kB
> SwapFree:       484512 kB
>
> After running an extremely memory-intensive test program for
> two minutes, buffermem fell to 38 megabytes.
>
> Seems broken to me.

Re: buffer/memory strangeness in 2.4.16 / 2.4.17pre2

[Andrew Morton]

space-00002@vortex.physik.uni-konstanz.de wrote:
> 
> It is still the same in 2.4.17pre2 :-(
> 

Yes, well, it would be.

It seems that what's happening is that all your buffercache memory
is on the active list, and all your anonymous memory is on the
inactive list.   Consequently the logic in shrink_caches() which
is supposed to move pages onto the inactive list doesn't do anything - it
just sits there calling refill_inactive() with a zero argument all
the time.

You'll find that if you push the machine really hard - allocate
1.5x physical memory and touch it all then the VM will, eventually,
with great reluctance and much swapping, relinquish the 30 megabytes
of buffercache memory.  But it's out of whack.

I tested the pre7aa1 patches and it seemed possibly a little better,
but not right.

If we put anon pages on the active list instead, then shrink_caches()
and refill_inactive() start to do something, and they move that stale
old buffercache memory onto the inactive list where it can be freed.

This is just a random hack.  I don't understand what's going on in
the VM, let alone what's *supposed* to be going on.  And given the
state of documentation available to us,  I never will.


--- linux-2.4.17-pre2/mm/memory.c	Thu Nov 22 23:02:59 2001
+++ linux-akpm/mm/memory.c	Sat Dec  1 20:14:28 2001
@@ -1174,6 +1174,7 @@ static int do_anonymous_page(struct mm_s
 		flush_page_to_ram(page);
 		entry = pte_mkwrite(pte_mkdirty(mk_pte(page, vma->vm_page_prot)));
 		lru_cache_add(page);
+		activate_page(page);
 	}
 
 	set_pte(page_table, entry);

-

Re: buffer/memory strangeness in 2.4.16 / 2.4.17pre2

[Rik van Riel]

On Sat, 1 Dec 2001, Andrew Morton wrote:

> You'll find that if you push the machine really hard - allocate
> 1.5x physical memory and touch it all then the VM will, eventually,
> with great reluctance and much swapping, relinquish the 30 megabytes
> of buffercache memory.  But it's out of whack.

This is an expected (and very bad) side effect of use-once.

> If we put anon pages on the active list instead, then shrink_caches()
> and refill_inactive() start to do something, and they move that stale
> old buffercache memory onto the inactive list where it can be freed.

This would fix the problem of not being able to evict stale
active pages, but I have no idea if it would unbalance
something else.

> This is just a random hack.  I don't understand what's going on in
> the VM, let alone what's *supposed* to be going on.  And given the
> state of documentation available to us,  I never will.

The balancing in Andrea's VM is just too subtle to understand
without docs, that is, if there is any particular idea behind
it and it isn't just experimentation.

regards,

Rik
-- 
Shortwave goes a long way:  irc.starchat.net  #swl

http://www.surriel.com/		http://distro.conectiva.com/

Re: buffer/memory strangeness in 2.4.16 / 2.4.17pre2

[space-00002]

I verified this by throwing in 1GB of swap. It does lose the buffers 
eventually, but it does so under great pains. However, since I do have 
*plenty* memory and slow disks, I prefer running without swap. These settings 
appear to be broken (2.4.13 worked!).

Without swap, I would have to reboot every day to make my simulation happy.- 
But it needs to allocate and use *only* about 300 of 768MB which *should* be 
available, only because the night before a cron job kicked off 'updatdb' 
filling the buffers.

Without swap, it really is *this* bad in 2.4.16 and the latest pre-releases. 
RAM is full of buffers that just won't disappear to make room for more 
important stuff. My simulation gets killed *every time* until I reboot to 
free 'buffers' or add swap. (This makes everything slower in total.)

:-(

	Jan

On Sunday 02 December 2001 13:17, Rik wrote:
> On Sat, 1 Dec 2001, Andrew Morton wrote:
> > You'll find that if you push the machine really hard - allocate
> > 1.5x physical memory and touch it all then the VM will, eventually,
> > with great reluctance and much swapping, relinquish the 30 megabytes
> > of buffercache memory.  But it's out of whack.
>
> This is an expected (and very bad) side effect of use-once.
>
> > If we put anon pages on the active list instead, then shrink_caches()
> > and refill_inactive() start to do something, and they move that stale
> > old buffercache memory onto the inactive list where it can be freed.
>
> This would fix the problem of not being able to evict stale
> active pages, but I have no idea if it would unbalance
> something else.
>
> > This is just a random hack.  I don't understand what's going on in
> > the VM, let alone what's *supposed* to be going on.  And given the
> > state of documentation available to us,  I never will.
>
> The balancing in Andrea's VM is just too subtle to understand
> without docs, that is, if there is any particular idea behind
> it and it isn't just experimentation.
>
> regards,
>
> Rik

Re: buffer/memory strangeness in 2.4.16; fixed in 2.4.17-rc1

[space-00002]

Hi,

just to let you know the problem described below seems to have been fixed in 
2.4.17-rc1. It behaves as it should. Great work!

Cheers,
	Jan

On Thursday 29 November 2001 21:39, you wrote:
> Hi,
>
> I am experiencing a bit of strange system behaviour in a vanilla 2.4.16
> kernel (2.95.3, very stable machine etc.)
>
> I noticed, that after running for a while (day) I had significantly less
> memory available for my simulation program than right after booting.
> Looking at the problem using 'xosview' (or 'free'), I noticed that there
> was a large number of MBs filled with 'buffers' that did not get wiped when
> other programs need the memory. The system seems to rather kill an
> 'offender' than clean out buffers.
>
> Right after booting, I can allocate about 650MBs memory using the little
> program attached below. After a day (or after running updatedb), under the
> same conditions, even in single user mode with only a shell running (!)
> this is not possible anymore and the program (below), trying to allocate
> only 300-400MBs, gets killed by the system after making it unresponsive for
> many seconds.
>
> Apparently this problem occurs after running 'updatedb', which fills 'free
> memory' and generates lots of filled cache and buffers on my system.
>
> This sort of behaviour must have been introduced after 2.4.13, which does
> not show these problems.