"Cached" grows and grows and grows...

"Cached" grows and grows and grows...

[Bob McElrath]

[-- Attachment #1: Type: text/plain, Size: 1128 bytes --]

This is probably closely related to the vm work going on...and you're
probably aware of it, but...

The "Cached" field in /proc/meminfo grows and grows and grows.  (kernel
2.4.7, 2.4.9, 2.4.10pre4aa1)  The kernel seems to be favoring buffer
cache for the filesystem over programs.  I recently purchased 256MB more
memory for my machine, only to find that Linux is using 200-300MB to
cache the filesystem.  Over time it swaps out everything to disk, and
"Cached" grows as large as 415MB on a 512MB machine.  Every time I come
back to my machine after not using the console for a while, it has to
swap everything back into memory in order to be usable.  (Note this
machine is basically unloaded except for setiathome while I'm away)

Let me place my vote that the vm subsystem should place pages with
program code, and their data at a higher priority to be in-memory than
filesystem cache buffers.  (Is the problem that the filesystem cache
buffers never expire, and stay in memory forever?)

Cheers,
-- Bob

Bob McElrath (rsmcelrath@students.wisc.edu) 
Univ. of Wisconsin at Madison, Department of Physics

[-- Attachment #2: Type: application/pgp-signature, Size: 240 bytes --]

Re: "Cached" grows and grows and grows...

[Stephan von Krawczynski]

On Fri, 7 Sep 2001 11:08:36 -0500 Bob McElrath
<mcelrath@draal.physics.wisc.edu> wrote:

> This is probably closely related to the vm work going on...and you're
> probably aware of it, but...
> 
> The "Cached" field in /proc/meminfo grows and grows and grows.  (kernel
> 2.4.7, 2.4.9, 2.4.10pre4aa1)  The kernel seems to be favoring buffer
> cache for the filesystem over programs.  I recently purchased 256MB more
> memory for my machine, only to find that Linux is using 200-300MB to
> cache the filesystem.  Over time it swaps out everything to disk, and
> "Cached" grows as large as 415MB on a 512MB machine.  Every time I come
> back to my machine after not using the console for a while, it has to
> swap everything back into memory in order to be usable.  (Note this
> machine is basically unloaded except for setiathome while I'm away)

To tell you the honest truth: you are not alone in cosmos (with this problem)
;-)
To give you that explicit hint for saving money: do not buy mem, it will be
eaten up by recent kernels without any performance gain or other positive
impact whatsoever. 
Try using 2.4.4, if it doesn't succeed, forget 2.4 and use 2.2.19. That works.
Unfortunately you may have to completely reinstall your system when going back
to 2.2.

Regards,
Stephan

Re: "Cached" grows and grows and grows...

[Alan Cox]

> To tell you the honest truth: you are not alone in cosmos (with this problem)
> ;-)
> To give you that explicit hint for saving money: do not buy mem, it will be
> eaten up by recent kernels without any performance gain or other positive
> impact whatsoever. 

Pick up a 2.4.9-ac kernel, and you shouldnt be seeing the problem (I say
shouldnt, I'm not 100% convinced its all under control)

> Try using 2.4.4, if it doesn't succeed, forget 2.4 and use 2.2.19. That works.
> Unfortunately you may have to completely reinstall your system when going back
> to 2.2.

That should not be needed at all. 

Re: "Cached" grows and grows and grows...

[Bob McElrath]

[-- Attachment #1: Type: text/plain, Size: 725 bytes --]

Alan Cox [alan@lxorguk.ukuu.org.uk] wrote:
> > To tell you the honest truth: you are not alone in cosmos (with this problem)
> > ;-)
> > To give you that explicit hint for saving money: do not buy mem, it will be
> > eaten up by recent kernels without any performance gain or other positive
> > impact whatsoever. 
> 
> Pick up a 2.4.9-ac kernel, and you shouldnt be seeing the problem (I say
> shouldnt, I'm not 100% convinced its all under control)

Hmmm...that will make my kernel 2.4.10pre4aa1acX-xfs.  Ack!  ;)

Maybe I can wait 'til 2.4.10.  Then it will only be 2.4.10aa1acX-xfs.
;)

Cheers,
-- Bob

Bob McElrath (rsmcelrath@students.wisc.edu) 
Univ. of Wisconsin at Madison, Department of Physics

[-- Attachment #2: Type: application/pgp-signature, Size: 240 bytes --]

VM improvement in -ac [was: "Cached" grows and grows and grows...]

[Mike Fedyk]

On Fri, Sep 07, 2001 at 11:15:36PM +0100, Alan Cox wrote:
> > To tell you the honest truth: you are not alone in cosmos (with this problem)
> > ;-)
> > To give you that explicit hint for saving money: do not buy mem, it will be
> > eaten up by recent kernels without any performance gain or other positive
> > impact whatsoever. 
> 
> Pick up a 2.4.9-ac kernel, and you shouldnt be seeing the problem (I say
> shouldnt, I'm not 100% convinced its all under control)
> 

I have to agree for the most part.  My system used to use a lot of cache,
and swapped all of the time with 300+MBs of ram on a Linus 2.4 kernel, now with
-ac my swap is under 1kB with the same apps and load.  I can even run ext3
now without another patch :).

One thing I have noticed is that I still see my system accessing the drive
softly for about 30 second time periods a few times a day.  I didn't see
this on 2.2, and vmstat doesn't show any paging traffic, but it does show
about 48 avg pages/sec going out to the disk.  I haven't really looked very
hard to find the culprit.  Has anyone else noticed this?

[OT] Hmm, why isn't ext3 in the Linus kernel yet?  It seems more mature than
reiserfs was when it was included back in 2.4.1...

Re: "Cached" grows and grows and grows...

[Stephan von Krawczynski]

On Fri, 7 Sep 2001 23:15:36 +0100 (BST) Alan Cox <alan@lxorguk.ukuu.org.uk>
wrote:

> > To tell you the honest truth: you are not alone in cosmos (with this
problem)
> > ;-)
> > To give you that explicit hint for saving money: do not buy mem, it will be
> > eaten up by recent kernels without any performance gain or other positive
> > impact whatsoever. 
> 
> Pick up a 2.4.9-ac kernel, and you shouldnt be seeing the problem (I say
> shouldnt, I'm not 100% convinced its all under control)

VERY FUNNY, Alan!

2.4.9-ac9: __alloc_pages:

        /* No luck.. */
//      printk(KERN_ERR "__alloc_pages: %lu-order allocation failed.\n", order)
        return NULL;

If there is no printk, you will obviously not notice the problem. You can bet
your car on not "seeing the problem".
 
> > Try using 2.4.4, if it doesn't succeed, forget 2.4 and use 2.2.19. That
works.
> > Unfortunately you may have to completely reinstall your system when going
back
> > to 2.2.
> 
> That should not be needed at all. 

Well, as long as you do not use any features that made you install 2.4 before,
e.g. files > 2GB and some others. Of course, if you do not use these, you might
be better of with 2.2 anyway.

That was not a very convincing comment, Alan.
But I must admit one thing: 2.4.9-ac9 runs smoother in my test. There are no
delays experienced during which the system desperately seeks mem. In fact I can
see a lot of inact_clean nearly all the time (a lot means 200-600 MB).
Nevertheless there _is_ a problem, because nfs still fails on low mem situation
when option "no_subtree_check" is _off_/not used.

I will have some closer looks on ac tree.

Regards,
Stephan

Re: "Cached" grows and grows and grows...

[Alan Cox]

> //      printk(KERN_ERR "__alloc_pages: %lu-order allocation failed.\n", order)
>         return NULL;
> 
> If there is no printk, you will obviously not notice the problem. You can bet
> your car on not "seeing the problem".

That printk is commented out because its pointless and bogus noise. It just
causes confused bug reporting. The stuff that matters is cache sizes
shrinking back when we need memory not slowly eating the computer alive.

> Nevertheless there _is_ a problem, because nfs still fails on low mem situation
> when option "no_subtree_check" is _off_/not used.

In certain cases NFS will struggle. It isnt as bad in -ac as we use
GFP_KERNEL for nfs level reassembly. It happens in 2.2 as well tho.

Alan

Re: "Cached" grows and grows and grows...

[Stephan von Krawczynski]

On Sat, 8 Sep 2001 18:14:46 +0100 (BST) Alan Cox <alan@lxorguk.ukuu.org.uk>
wrote:

> > //      printk(KERN_ERR "__alloc_pages: %lu-order allocation failed.\n",
order)
> >         return NULL;
> > 
> > If there is no printk, you will obviously not notice the problem. You can
bet
> > your car on not "seeing the problem".
> 
> That printk is commented out because its pointless and bogus noise. It just
> causes confused bug reporting. The stuff that matters is cache sizes
> shrinking back when we need memory not slowly eating the computer alive.

Hm, I guess I had to find out, that you've the same problem than linus' tree,
only on "the other side of the street". This is a meminfo from a 2.4.9-ac9
kernel working one day, and then going crazy on too low mem. "Going crazy" here
means that kswapd took virtually over the cpu(s) and swapped the hell out the
machine:
(BTW you can see that swap did not really grow, but seems to get in and out
permanently)

        total:    used:    free:  shared: buffers:  cached:
Mem:  921706496 894324736 27381760     4096 783917056 29102080
Swap: 271392768 13185024 258207744
MemTotal:       900104 kB
MemFree:         26740 kB
MemShared:           4 kB
Buffers:        765544 kB
Cached:          15744 kB
SwapCached:      12676 kB
Active:         502132 kB
Inact_dirty:    291836 kB
Inact_clean:         0 kB
Inact_target:       80 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       900104 kB
LowFree:         26740 kB
SwapTotal:      265032 kB
SwapFree:       252156 kB

You are right: page cache shrunk.
You are wrong: it does _not_ work, because now buffers increased and obviously
cannot be shrunk to allow "normal" applications to run. I could not even
shutdown the machine correctly. It looks like a deadlock in vm to me.

I switched back to Linus' tree, because it does have problems, but is not dead
within one day.

Regards,
Stephan

Re: "Cached" grows and grows and grows...

[Alan Cox]

> only on "the other side of the street". This is a meminfo from a 2.4.9-ac9
> kernel working one day, and then going crazy on too low mem. "Going crazy" here
> means that kswapd took virtually over the cpu(s) and swapped the hell out the
> machine:
> (BTW you can see that swap did not really grow, but seems to get in and out
> permanently)

That buffers count looks like you have something leaking. Probably a file
system. When kswapd is busy like that it is trying to free buffer cache.
The lack of freeing implies something is still holding those buffer cache
entries in memory. 

> You are right: page cache shrunk.
> You are wrong: it does _not_ work, because now buffers increased and obviously
> cannot be shrunk to allow "normal" applications to run. I could not even
> shutdown the machine correctly. It looks like a deadlock in vm to me.

Nope. The VM 

> I switched back to Linus' tree, because it does have problems, but is not dead
> within one day.

Thats good, Linus tree currently lasts about 30 minutes for me. I'd be
interested to know what your workload is and what drivers/filesystem you
are using so I can look for leaks.

Alan