* Re: 2.4.16 memory badness (fixed?) [not found] <Pine.LNX.4.33.0112090808250.6883-100000@mikeg.weiden.de> @ 2001-12-09 16:07 ` Leigh Orf 2001-12-09 17:17 ` Mike Galbraith 2001-12-09 17:32 ` Mike Galbraith 0 siblings, 2 replies; 9+ messages in thread From: Leigh Orf @ 2001-12-09 16:07 UTC (permalink / raw) To: Mike Galbraith Cc: M.H.VanLeeuwen, Mark Hahn, Andrew Morton, Ken Brownfield, linux-kernel In a personal email, Mike Galbraith wrote to me: | On Sat, 8 Dec 2001, Leigh Orf wrote: | | > inode_cache 439584 439586 512 62798 62798 1 | > dentry_cache 454136 454200 128 15140 15140 1 | | I'd try moving shrink_[id]cache_memory to the very top of vmscan.c::shrink_caches. | | -Mike Mike, I tried what you suggested starting with a stock 2.4.16 kernel, and it did fix the problem with 2.4.16 ENOMEM being returned. Now with that change and after updatedb runs, here's what the memory situation looks like. Note inode_cache and dentry_cache are almost nothing. Dunno if that's a good thing or not, but I'd definitely consider this for a patch. home[1002]:/home/orf% free total used free shared buffers cached Mem: 1029828 1020316 9512 0 719596 145768 -/+ buffers/cache: 154952 874876 Swap: 2064344 0 2064344 home[1003]:/home/orf% cat /proc/meminfo total: used: free: shared: buffers: cached: Mem: 1054543872 1044979712 9564160 0 736870400 149282816 Swap: 2113888256 0 2113888256 MemTotal: 1029828 kB MemFree: 9340 kB MemShared: 0 kB Buffers: 719600 kB Cached: 145784 kB SwapCached: 0 kB Active: 462276 kB Inactive: 481572 kB HighTotal: 130992 kB HighFree: 2044 kB LowTotal: 898836 kB LowFree: 7296 kB SwapTotal: 2064344 kB SwapFree: 2064344 kB home[1001]:/home/orf% cat /proc/slabinfo slabinfo - version: 1.1 kmem_cache 65 68 112 2 2 1 ip_conntrack 13 30 384 3 3 1 nfs_write_data 0 0 384 0 0 1 nfs_read_data 0 0 384 0 0 1 nfs_page 0 0 128 0 0 1 ip_fib_hash 10 112 32 1 1 1 urb_priv 0 0 64 0 0 1 clip_arp_cache 0 0 128 0 0 1 ip_mrt_cache 0 0 128 0 0 1 tcp_tw_bucket 0 30 128 0 1 1 tcp_bind_bucket 8 112 32 1 1 1 tcp_open_request 0 0 128 0 0 1 inet_peer_cache 1 59 64 1 1 1 ip_dst_cache 21 40 192 2 2 1 arp_cache 3 30 128 1 1 1 blkdev_requests 640 660 128 22 22 1 journal_head 0 0 48 0 0 1 revoke_table 0 0 12 0 0 1 revoke_record 0 0 32 0 0 1 dnotify cache 0 0 20 0 0 1 file lock cache 2 42 92 1 1 1 fasync cache 2 202 16 1 1 1 uid_cache 5 112 32 1 1 1 skbuff_head_cache 293 300 192 15 15 1 sock 209 213 1280 70 71 1 sigqueue 2 29 132 1 1 1 cdev_cache 21 295 64 3 5 1 bdev_cache 8 59 64 1 1 1 mnt_cache 19 59 64 1 1 1 inode_cache 7151 16905 512 2412 2415 1 dentry_cache 3043 12630 128 421 421 1 dquot 0 0 128 0 0 1 filp 1990 2010 128 67 67 1 names_cache 0 2 4096 0 2 1 buffer_head 220278 220350 128 7344 7345 1 mm_struct 64 80 192 4 4 1 vm_area_struct 2779 3180 128 93 106 1 fs_cache 63 118 64 2 2 1 files_cache 63 72 448 7 8 1 signal_act 71 72 1344 24 24 1 size-131072(DMA) 0 0 131072 0 0 32 size-131072 0 0 131072 0 0 32 size-65536(DMA) 0 0 65536 0 0 16 size-65536 1 1 65536 1 1 16 size-32768(DMA) 0 0 32768 0 0 8 size-32768 1 1 32768 1 1 8 size-16384(DMA) 0 0 16384 0 0 4 size-16384 1 5 16384 1 5 4 size-8192(DMA) 0 0 8192 0 0 2 size-8192 4 5 8192 4 5 2 size-4096(DMA) 0 0 4096 0 0 1 size-4096 72 78 4096 72 78 1 size-2048(DMA) 0 0 2048 0 0 1 size-2048 61 96 2048 32 48 1 size-1024(DMA) 0 0 1024 0 0 1 size-1024 1475 2988 1024 532 747 1 size-512(DMA) 0 0 512 0 0 1 size-512 1650 4168 512 351 521 1 size-256(DMA) 0 0 256 0 0 1 size-256 486 2340 256 45 156 1 size-128(DMA) 2 30 128 1 1 1 size-128 3483 10050 128 230 335 1 size-64(DMA) 0 0 64 0 0 1 size-64 1792 3304 64 56 56 1 size-32(DMA) 34 59 64 1 1 1 size-32 4159 11092 64 159 188 1 Leigh Orf ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.4.16 memory badness (fixed?) 2001-12-09 16:07 ` 2.4.16 memory badness (fixed?) Leigh Orf @ 2001-12-09 17:17 ` Mike Galbraith 2001-12-10 7:24 ` Ken Brownfield 2001-12-09 17:32 ` Mike Galbraith 1 sibling, 1 reply; 9+ messages in thread From: Mike Galbraith @ 2001-12-09 17:17 UTC (permalink / raw) To: Leigh Orf Cc: M.H.VanLeeuwen, Mark Hahn, Andrew Morton, Ken Brownfield, linux-kernel On Sun, 9 Dec 2001, Leigh Orf wrote: > In a personal email, Mike Galbraith wrote to me: > > | On Sat, 8 Dec 2001, Leigh Orf wrote: > | > | > inode_cache 439584 439586 512 62798 62798 1 > | > dentry_cache 454136 454200 128 15140 15140 1 > | > | I'd try moving shrink_[id]cache_memory to the very top of vmscan.c::shrink_caches. > | > | -Mike > > Mike, > > I tried what you suggested starting with a stock 2.4.16 kernel, and it > did fix the problem with 2.4.16 ENOMEM being returned. > > Now with that change and after updatedb runs, here's what the memory > situation looks like. Note inode_cache and dentry_cache are almost > nothing. Dunno if that's a good thing or not, but I'd definitely Almost nothing isn't particularly good after updatedb ;-) > consider this for a patch. No, but those do need faster pruning imho. The growth rate can be really really fast at times. -Mike ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.4.16 memory badness (fixed?) 2001-12-09 17:17 ` Mike Galbraith @ 2001-12-10 7:24 ` Ken Brownfield 2001-12-10 11:10 ` Rik van Riel 0 siblings, 1 reply; 9+ messages in thread From: Ken Brownfield @ 2001-12-10 7:24 UTC (permalink / raw) To: Mike Galbraith Cc: Leigh Orf, M.H.VanLeeuwen, Mark Hahn, Andrew Morton, linux-kernel What about moving the calls to shrink_[di]cache_memory() after the nr_pages check after the call to kmem_cache_reap? Or perhaps keep it at the beginning, but only call it after priority has gone a number of notches down from DEF_PRIORITY? Something like that seems like the only obvious way to balance how soon these caches are flushed without over- or under-kill. Also, shouldn't shrink_dqcache_memory use priority rather than DEF_PRIORITY? I'm also not sure what the reasoning is behind the nr_pages=chunk_size reset. In the case that Leigh and I are seeing (and my readprofile runs) it sounds like shrink_cache is getting called a ton, while the bloated d/i caches are flushed too little, too late. Just $0.02 from a newby. ;) Thanks for the tip, Mike, -- Ken. brownfld@irridia.com On Sun, Dec 09, 2001 at 06:17:11PM +0100, Mike Galbraith wrote: | On Sun, 9 Dec 2001, Leigh Orf wrote: | | > In a personal email, Mike Galbraith wrote to me: | > | > | On Sat, 8 Dec 2001, Leigh Orf wrote: | > | | > | > inode_cache 439584 439586 512 62798 62798 1 | > | > dentry_cache 454136 454200 128 15140 15140 1 | > | | > | I'd try moving shrink_[id]cache_memory to the very top of vmscan.c::shrink_caches. | > | | > | -Mike | > | > Mike, | > | > I tried what you suggested starting with a stock 2.4.16 kernel, and it | > did fix the problem with 2.4.16 ENOMEM being returned. | > | > Now with that change and after updatedb runs, here's what the memory | > situation looks like. Note inode_cache and dentry_cache are almost | > nothing. Dunno if that's a good thing or not, but I'd definitely | | Almost nothing isn't particularly good after updatedb ;-) | | > consider this for a patch. | | No, but those do need faster pruning imho. The growth rate can be | really really fast at times. | | -Mike ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.4.16 memory badness (fixed?) 2001-12-10 7:24 ` Ken Brownfield @ 2001-12-10 11:10 ` Rik van Riel 2001-12-10 15:49 ` Leigh Orf 0 siblings, 1 reply; 9+ messages in thread From: Rik van Riel @ 2001-12-10 11:10 UTC (permalink / raw) To: Ken Brownfield Cc: Mike Galbraith, Leigh Orf, M.H.VanLeeuwen, Mark Hahn, Andrew Morton, linux-kernel On Mon, 10 Dec 2001, Ken Brownfield wrote: > What about moving the calls to shrink_[di]cache_memory() after the > nr_pages check after the call to kmem_cache_reap? Or perhaps keep it at > the beginning, but only call it after priority has gone a number of > notches down from DEF_PRIORITY? > > Something like that seems like the only obvious way to balance how soon > these caches are flushed without over- or under-kill. So obvious that it's been re-introduced 3 times now even though it broke each time. ;) The only way to get stuff balanced somewhat is to call the shrink functions unconditionally. It's not optimally balanced, but at least the cache will stay reasonably small while still being able to grow under load. Rik -- DMCA, SSSCA, W3C? Who cares? http://thefreeworld.net/ http://www.surriel.com/ http://distro.conectiva.com/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.4.16 memory badness (fixed?) 2001-12-10 11:10 ` Rik van Riel @ 2001-12-10 15:49 ` Leigh Orf 2001-12-10 16:29 ` M. Edward Borasky 2001-12-14 22:54 ` Mike Galbraith 0 siblings, 2 replies; 9+ messages in thread From: Leigh Orf @ 2001-12-10 15:49 UTC (permalink / raw) To: Rik van Riel Cc: Ken Brownfield, Mike Galbraith, M.H.VanLeeuwen, Mark Hahn, Andrew Morton, linux-kernel Rik van Riel wrote: | On Mon, 10 Dec 2001, Ken Brownfield wrote: | | > What about moving the calls to shrink_[di]cache_memory() | > after the nr_pages check after the call to kmem_cache_reap? | > Or perhaps keep it at the beginning, but only call it | > after priority has gone a number of notches down from | > DEF_PRIORITY? | > | > Something like that seems like the only obvious way to | > balance how soon these caches are flushed without over- or | > under-kill. | | So obvious that it's been re-introduced 3 times now even | though it broke each time. ;) And in fact, after furthur playing around with the "fixed" version (moving shrink_[id]cache_memory to the top of vmscan.c::shrink_caches) I find that I still will get ENOMEM after updatedb occasionally. Less often than before, but it still happens. | The only way to get stuff balanced somewhat is to call | the shrink functions unconditionally. It's not optimally | balanced, but at least the cache will stay reasonably small | while still being able to grow under load. I just can't understand why the kernel wouldn't tag application memory as being more important tan buff/cache and free up some of that stuff when an application calls for it. I mean, it won't even use the gobs of swap I have. That just seems to be a plain ol' bug to me. Leigh Orf ^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: 2.4.16 memory badness (fixed?) 2001-12-10 15:49 ` Leigh Orf @ 2001-12-10 16:29 ` M. Edward Borasky 2001-12-12 9:04 ` Helge Hafting 2001-12-14 22:54 ` Mike Galbraith 1 sibling, 1 reply; 9+ messages in thread From: M. Edward Borasky @ 2001-12-10 16:29 UTC (permalink / raw) To: linux-kernel > -----Original Message----- > I just can't understand why the kernel wouldn't tag application memory > as being more important than buff/cache and free up some of that stuff > when an application calls for it. I mean, it won't even use the gobs of > swap I have. That just seems to be a plain ol' bug to me. It's not strictly a bug ... it's a design decision that has unfortunate consequences. A simple fix would be to allow the system administrator to set an upper limit on the size of the page cache. -- M. Edward Borasky znmeb@@borasky-research.net http://www.borasky-research.net ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.4.16 memory badness (fixed?) 2001-12-10 16:29 ` M. Edward Borasky @ 2001-12-12 9:04 ` Helge Hafting 0 siblings, 0 replies; 9+ messages in thread From: Helge Hafting @ 2001-12-12 9:04 UTC (permalink / raw) To: M. Edward Borasky, linux-kernel "M. Edward Borasky" wrote: > > > -----Original Message----- > > I just can't understand why the kernel wouldn't tag application memory > > as being more important than buff/cache and free up some of that stuff > > when an application calls for it. I mean, it won't even use the gobs of > > swap I have. That just seems to be a plain ol' bug to me. > > It's not strictly a bug ... it's a design decision that has unfortunate > consequences. A simple fix would be to allow the system administrator to set > an upper limit on the size of the page cache. I'd say he has found a bug. Merely prioritizing cache over apps so apps go to swap is a design desicion. Killing the app for OOM reasons when there is free swap and/or cache that can be freed up _is_ a bug. Helge Hafting ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.4.16 memory badness (fixed?) 2001-12-10 15:49 ` Leigh Orf 2001-12-10 16:29 ` M. Edward Borasky @ 2001-12-14 22:54 ` Mike Galbraith 1 sibling, 0 replies; 9+ messages in thread From: Mike Galbraith @ 2001-12-14 22:54 UTC (permalink / raw) To: Leigh Orf Cc: Rik van Riel, Ken Brownfield, M.H.VanLeeuwen, Mark Hahn, Andrew Morton, linux-kernel On Mon, 10 Dec 2001, Leigh Orf wrote: > And in fact, after furthur playing around with the "fixed" version > (moving shrink_[id]cache_memory to the top of vmscan.c::shrink_caches) > I find that I still will get ENOMEM after updatedb occasionally. Less > often than before, but it still happens. Yes.. reasonable. -Mike ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 2.4.16 memory badness (fixed?) 2001-12-09 16:07 ` 2.4.16 memory badness (fixed?) Leigh Orf 2001-12-09 17:17 ` Mike Galbraith @ 2001-12-09 17:32 ` Mike Galbraith 1 sibling, 0 replies; 9+ messages in thread From: Mike Galbraith @ 2001-12-09 17:32 UTC (permalink / raw) To: Leigh Orf Cc: M.H.VanLeeuwen, Mark Hahn, Andrew Morton, Ken Brownfield, linux-kernel On Sun, 9 Dec 2001, Leigh Orf wrote: > buffer_head 220278 220350 128 7344 7345 1 P.S. These don't bother my box much.. a little though. -Mike ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2001-12-14 22:52 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <Pine.LNX.4.33.0112090808250.6883-100000@mikeg.weiden.de>
2001-12-09 16:07 ` 2.4.16 memory badness (fixed?) Leigh Orf
2001-12-09 17:17 ` Mike Galbraith
2001-12-10 7:24 ` Ken Brownfield
2001-12-10 11:10 ` Rik van Riel
2001-12-10 15:49 ` Leigh Orf
2001-12-10 16:29 ` M. Edward Borasky
2001-12-12 9:04 ` Helge Hafting
2001-12-14 22:54 ` Mike Galbraith
2001-12-09 17:32 ` Mike Galbraith
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.