Re: 2.4.16 memory badness (fixed?)

Re: 2.4.16 memory badness (fixed?)

[Leigh Orf]

In a personal email, Mike Galbraith wrote to me:

|   On Sat, 8 Dec 2001, Leigh Orf wrote:
|   
|   > inode_cache       439584 439586    512 62798 62798    1
|   > dentry_cache      454136 454200    128 15140 15140    1
|   
|   I'd try moving shrink_[id]cache_memory to the very top of vmscan.c::shrink_caches.
|   
|   	-Mike

Mike,

I tried what you suggested starting with a stock 2.4.16 kernel, and it
did fix the problem with 2.4.16 ENOMEM being returned.

Now with that change and after updatedb runs, here's what the memory
situation looks like. Note inode_cache and dentry_cache are almost
nothing. Dunno if that's a good thing or not, but I'd definitely
consider this for a patch.

home[1002]:/home/orf% free
             total       used       free     shared    buffers     cached
Mem:       1029828    1020316       9512          0     719596     145768
-/+ buffers/cache:     154952     874876
Swap:      2064344          0    2064344

home[1003]:/home/orf% cat /proc/meminfo
        total:    used:    free:  shared: buffers:  cached:
Mem:  1054543872 1044979712  9564160        0 736870400 149282816
Swap: 2113888256        0 2113888256
MemTotal:      1029828 kB
MemFree:          9340 kB
MemShared:           0 kB
Buffers:        719600 kB
Cached:         145784 kB
SwapCached:          0 kB
Active:         462276 kB
Inactive:       481572 kB
HighTotal:      130992 kB
HighFree:         2044 kB
LowTotal:       898836 kB
LowFree:          7296 kB
SwapTotal:     2064344 kB
SwapFree:      2064344 kB

home[1001]:/home/orf% cat /proc/slabinfo
slabinfo - version: 1.1
kmem_cache            65     68    112    2    2    1
ip_conntrack          13     30    384    3    3    1
nfs_write_data         0      0    384    0    0    1
nfs_read_data          0      0    384    0    0    1
nfs_page               0      0    128    0    0    1
ip_fib_hash           10    112     32    1    1    1
urb_priv               0      0     64    0    0    1
clip_arp_cache         0      0    128    0    0    1
ip_mrt_cache           0      0    128    0    0    1
tcp_tw_bucket          0     30    128    0    1    1
tcp_bind_bucket        8    112     32    1    1    1
tcp_open_request       0      0    128    0    0    1
inet_peer_cache        1     59     64    1    1    1
ip_dst_cache          21     40    192    2    2    1
arp_cache              3     30    128    1    1    1
blkdev_requests      640    660    128   22   22    1
journal_head           0      0     48    0    0    1
revoke_table           0      0     12    0    0    1
revoke_record          0      0     32    0    0    1
dnotify cache          0      0     20    0    0    1
file lock cache        2     42     92    1    1    1
fasync cache           2    202     16    1    1    1
uid_cache              5    112     32    1    1    1
skbuff_head_cache    293    300    192   15   15    1
sock                 209    213   1280   70   71    1
sigqueue               2     29    132    1    1    1
cdev_cache            21    295     64    3    5    1
bdev_cache             8     59     64    1    1    1
mnt_cache             19     59     64    1    1    1
inode_cache         7151  16905    512 2412 2415    1
dentry_cache        3043  12630    128  421  421    1
dquot                  0      0    128    0    0    1
filp                1990   2010    128   67   67    1
names_cache            0      2   4096    0    2    1
buffer_head       220278 220350    128 7344 7345    1
mm_struct             64     80    192    4    4    1
vm_area_struct      2779   3180    128   93  106    1
fs_cache              63    118     64    2    2    1
files_cache           63     72    448    7    8    1
signal_act            71     72   1344   24   24    1
size-131072(DMA)       0      0 131072    0    0   32
size-131072            0      0 131072    0    0   32
size-65536(DMA)        0      0  65536    0    0   16
size-65536             1      1  65536    1    1   16
size-32768(DMA)        0      0  32768    0    0    8
size-32768             1      1  32768    1    1    8
size-16384(DMA)        0      0  16384    0    0    4
size-16384             1      5  16384    1    5    4
size-8192(DMA)         0      0   8192    0    0    2
size-8192              4      5   8192    4    5    2
size-4096(DMA)         0      0   4096    0    0    1
size-4096             72     78   4096   72   78    1
size-2048(DMA)         0      0   2048    0    0    1
size-2048             61     96   2048   32   48    1
size-1024(DMA)         0      0   1024    0    0    1
size-1024           1475   2988   1024  532  747    1
size-512(DMA)          0      0    512    0    0    1
size-512            1650   4168    512  351  521    1
size-256(DMA)          0      0    256    0    0    1
size-256             486   2340    256   45  156    1
size-128(DMA)          2     30    128    1    1    1
size-128            3483  10050    128  230  335    1
size-64(DMA)           0      0     64    0    0    1
size-64             1792   3304     64   56   56    1
size-32(DMA)          34     59     64    1    1    1
size-32             4159  11092     64  159  188    1

Leigh Orf

Re: 2.4.16 memory badness (fixed?)

[Mike Galbraith]

On Sun, 9 Dec 2001, Leigh Orf wrote:

> In a personal email, Mike Galbraith wrote to me:
>
> |   On Sat, 8 Dec 2001, Leigh Orf wrote:
> |
> |   > inode_cache       439584 439586    512 62798 62798    1
> |   > dentry_cache      454136 454200    128 15140 15140    1
> |
> |   I'd try moving shrink_[id]cache_memory to the very top of vmscan.c::shrink_caches.
> |
> |   	-Mike
>
> Mike,
>
> I tried what you suggested starting with a stock 2.4.16 kernel, and it
> did fix the problem with 2.4.16 ENOMEM being returned.
>
> Now with that change and after updatedb runs, here's what the memory
> situation looks like. Note inode_cache and dentry_cache are almost
> nothing. Dunno if that's a good thing or not, but I'd definitely

Almost nothing isn't particularly good after updatedb ;-)

> consider this for a patch.

No, but those do need faster pruning imho.  The growth rate can be
really really fast at times.

	-Mike

Re: 2.4.16 memory badness (fixed?)

[Mike Galbraith]

On Sun, 9 Dec 2001, Leigh Orf wrote:

> buffer_head       220278 220350    128 7344 7345    1

P.S.

These don't bother my box much.. a little though.

	-Mike

Re: 2.4.16 memory badness (fixed?)

[Ken Brownfield]

What about moving the calls to shrink_[di]cache_memory() after the
nr_pages check after the call to kmem_cache_reap?  Or perhaps keep it at
the beginning, but only call it after priority has gone a number of
notches down from DEF_PRIORITY?

Something like that seems like the only obvious way to balance how soon
these caches are flushed without over- or under-kill.

Also, shouldn't shrink_dqcache_memory use priority rather than
DEF_PRIORITY?  I'm also not sure what the reasoning is behind the
nr_pages=chunk_size reset.

In the case that Leigh and I are seeing (and my readprofile runs) it
sounds like shrink_cache is getting called a ton, while the bloated d/i
caches are flushed too little, too late.

Just $0.02 from a newby. ;)

Thanks for the tip, Mike,
-- 
Ken.
brownfld@irridia.com


On Sun, Dec 09, 2001 at 06:17:11PM +0100, Mike Galbraith wrote:
| On Sun, 9 Dec 2001, Leigh Orf wrote:
| 
| > In a personal email, Mike Galbraith wrote to me:
| >
| > |   On Sat, 8 Dec 2001, Leigh Orf wrote:
| > |
| > |   > inode_cache       439584 439586    512 62798 62798    1
| > |   > dentry_cache      454136 454200    128 15140 15140    1
| > |
| > |   I'd try moving shrink_[id]cache_memory to the very top of vmscan.c::shrink_caches.
| > |
| > |   	-Mike
| >
| > Mike,
| >
| > I tried what you suggested starting with a stock 2.4.16 kernel, and it
| > did fix the problem with 2.4.16 ENOMEM being returned.
| >
| > Now with that change and after updatedb runs, here's what the memory
| > situation looks like. Note inode_cache and dentry_cache are almost
| > nothing. Dunno if that's a good thing or not, but I'd definitely
| 
| Almost nothing isn't particularly good after updatedb ;-)
| 
| > consider this for a patch.
| 
| No, but those do need faster pruning imho.  The growth rate can be
| really really fast at times.
| 
| 	-Mike

Re: 2.4.16 memory badness (fixed?)

[Rik van Riel]

On Mon, 10 Dec 2001, Ken Brownfield wrote:

> What about moving the calls to shrink_[di]cache_memory() after the
> nr_pages check after the call to kmem_cache_reap?  Or perhaps keep it at
> the beginning, but only call it after priority has gone a number of
> notches down from DEF_PRIORITY?
>
> Something like that seems like the only obvious way to balance how soon
> these caches are flushed without over- or under-kill.

So obvious that it's been re-introduced 3 times now even though
it broke each time. ;)

The only way to get stuff balanced somewhat is to call the
shrink functions unconditionally. It's not optimally balanced,
but at least the cache will stay reasonably small while still
being able to grow under load.

Rik
-- 
DMCA, SSSCA, W3C?  Who cares?  http://thefreeworld.net/

http://www.surriel.com/		http://distro.conectiva.com/

Re: 2.4.16 memory badness (fixed?)

[Leigh Orf]

Rik van Riel wrote:

|   On Mon, 10 Dec 2001, Ken Brownfield wrote:
|
|   > What about moving the calls to shrink_[di]cache_memory()
|   > after the nr_pages check after the call to kmem_cache_reap?
|   > Or perhaps keep it at the beginning, but only call it
|   > after priority has gone a number of notches down from
|   > DEF_PRIORITY?
|   >
|   > Something like that seems like the only obvious way to
|   > balance how soon these caches are flushed without over- or
|   > under-kill.
|
|   So obvious that it's been re-introduced 3 times now even
|   though it broke each time. ;)

And in fact, after furthur playing around with the "fixed" version
(moving shrink_[id]cache_memory to the top of vmscan.c::shrink_caches)
I find that I still will get ENOMEM after updatedb occasionally. Less
often than before, but it still happens.

|   The only way to get stuff balanced somewhat is to call
|   the shrink functions unconditionally. It's not optimally
|   balanced, but at least the cache will stay reasonably small
|   while still being able to grow under load.

I just can't understand why the kernel wouldn't tag application memory
as being more important tan buff/cache and free up some of that stuff
when an application calls for it. I mean, it won't even use the gobs of
swap I have. That just seems to be a plain ol' bug to me.

Leigh Orf

RE: 2.4.16 memory badness (fixed?)

[M. Edward Borasky]

> -----Original Message-----
> I just can't understand why the kernel wouldn't tag application memory
> as being more important than buff/cache and free up some of that stuff
> when an application calls for it. I mean, it won't even use the gobs of
> swap I have. That just seems to be a plain ol' bug to me.

It's not strictly a bug ... it's a design decision that has unfortunate
consequences. A simple fix would be to allow the system administrator to set
an upper limit on the size of the page cache.
--
M. Edward Borasky
znmeb@@borasky-research.net
http://www.borasky-research.net

Re: 2.4.16 memory badness (fixed?)

[Helge Hafting]

"M. Edward Borasky" wrote:
> 
> > -----Original Message-----
> > I just can't understand why the kernel wouldn't tag application memory
> > as being more important than buff/cache and free up some of that stuff
> > when an application calls for it. I mean, it won't even use the gobs of
> > swap I have. That just seems to be a plain ol' bug to me.
> 
> It's not strictly a bug ... it's a design decision that has unfortunate
> consequences. A simple fix would be to allow the system administrator to set
> an upper limit on the size of the page cache.

I'd say he has found a bug.  Merely prioritizing cache over apps
so apps go to swap is a design desicion.  Killing the app
for OOM reasons when there is free swap and/or cache
that can be freed up _is_ a bug. 

Helge Hafting

Re: 2.4.16 memory badness (fixed?)

[Mike Galbraith]

On Mon, 10 Dec 2001, Leigh Orf wrote:

> And in fact, after furthur playing around with the "fixed" version
> (moving shrink_[id]cache_memory to the top of vmscan.c::shrink_caches)
> I find that I still will get ENOMEM after updatedb occasionally. Less
> often than before, but it still happens.

Yes.. reasonable.

	-Mike