From: "Theodore Ts'o" <tytso@mit.edu>
To: linux-mm@kvack.org, linux-kernel@vger.kernel.org
Subject: VM balancing issues on 2.6.13: dentry cache not getting shrunk enough
Date: Sun, 11 Sep 2005 06:57:09 -0400 [thread overview]
Message-ID: <20050911105709.GA16369@thunk.org> (raw)
[-- Attachment #1: Type: text/plain, Size: 2089 bytes --]
I've been noticing this for a while (probably since at least 2.6.11 or
so, but I haven't been keeping close attention), but I haven't had the
time to get some proof that this was the cause, and to write it up
until now.
I have a T40 laptop (Pentium M processor) with 2 gigs of memory, and
from time to time, after the system has been up for a while, the
dentry cache grows huge, as does the ext3_inode_cache:
slabinfo - version: 2.1
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
dentry_cache 434515 514112 136 29 1 : tunables 120 60 0 : slabdata 17728 17728 0
ext3_inode_cache 587635 589992 464 8 1 : tunables 54 27 0 : slabdata 73748 73749 0
Leading to an impending shortage in low memory:
LowFree: 9268 kB
... and if I don't take corrective measures, very shortly thereafter
the system will become unresponsive and will end up thrashing itself
to death, with symptoms that are identical to a case of 2.4 lowmem
exhaustion --- except this is on a 2.6.13 kernel, where all of these
problems were supposed to be solved.
It turns out I can head off the system lockup by requesting the
formation of hugepages, which will immediately cause a dramatic
reduction of memory usage in both high- and low- memory as various
caches and flushed:
echo 100 > /proc/sys/vm/nr_hugepages
echo 0 > /proc/sys/vm/nr_hugepages
The question is why isn't the kernel able to figure out how to do
release dentry cache entries automatically when it starts thrashing due
to a lack of low memory? Clearly it can, since requesting hugepages
does shrink the dentry cache:
dentry_cache 20097 20097 136 29 1 : tunables 120 60 0 : slabdata 693 693 0
ext3_inode_cache 17782 17784 464 8 1 : tunables 54 27 0 : slabdata 2223 2223 0
LowFree: 835916 kB
Has anyone else seen this, or have some ideas about how to fix it?
Thanks, regards,
- Ted
[-- Attachment #2: slabinfo --]
[-- Type: text/plain, Size: 13055 bytes --]
slabinfo - version: 2.1
# name <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
nfs_write_data 36 36 448 9 1 : tunables 54 27 0 : slabdata 4 4 0
nfs_read_data 32 36 448 9 1 : tunables 54 27 0 : slabdata 4 4 0
nfs_inode_cache 69 72 592 6 1 : tunables 54 27 0 : slabdata 12 12 0
nfs_page 0 0 64 61 1 : tunables 120 60 0 : slabdata 0 0 0
rpc_buffers 8 8 2048 2 1 : tunables 24 12 0 : slabdata 4 4 0
rpc_tasks 8 20 192 20 1 : tunables 120 60 0 : slabdata 1 1 0
rpc_inode_cache 8 9 448 9 1 : tunables 54 27 0 : slabdata 1 1 0
uhci_urb_priv 0 0 44 88 1 : tunables 120 60 0 : slabdata 0 0 0
fib6_nodes 7 119 32 119 1 : tunables 120 60 0 : slabdata 1 1 0
ip6_dst_cache 7 15 256 15 1 : tunables 120 60 0 : slabdata 1 1 0
ndisc_cache 1 20 192 20 1 : tunables 120 60 0 : slabdata 1 1 0
RAWv6 3 6 640 6 1 : tunables 54 27 0 : slabdata 1 1 0
UDPv6 2 7 576 7 1 : tunables 54 27 0 : slabdata 1 1 0
request_sock_TCPv6 0 0 128 31 1 : tunables 120 60 0 : slabdata 0 0 0
TCPv6 12 14 1088 7 2 : tunables 24 12 0 : slabdata 2 2 0
ip_fib_alias 11 226 16 226 1 : tunables 120 60 0 : slabdata 1 1 0
ip_fib_hash 11 119 32 119 1 : tunables 120 60 0 : slabdata 1 1 0
UNIX 343 350 384 10 1 : tunables 54 27 0 : slabdata 35 35 0
tcp_tw_bucket 0 0 128 31 1 : tunables 120 60 0 : slabdata 0 0 0
tcp_bind_bucket 29 226 16 226 1 : tunables 120 60 0 : slabdata 1 1 0
inet_peer_cache 0 0 64 61 1 : tunables 120 60 0 : slabdata 0 0 0
secpath_cache 0 0 128 31 1 : tunables 120 60 0 : slabdata 0 0 0
xfrm_dst_cache 0 0 320 12 1 : tunables 54 27 0 : slabdata 0 0 0
ip_dst_cache 29 45 256 15 1 : tunables 120 60 0 : slabdata 3 3 0
arp_cache 4 31 128 31 1 : tunables 120 60 0 : slabdata 1 1 0
RAW 2 9 448 9 1 : tunables 54 27 0 : slabdata 1 1 0
UDP 28 28 512 7 1 : tunables 54 27 0 : slabdata 4 4 0
request_sock_TCP 0 0 64 61 1 : tunables 120 60 0 : slabdata 0 0 0
TCP 144 148 960 4 1 : tunables 54 27 0 : slabdata 37 37 0
flow_cache 0 0 128 31 1 : tunables 120 60 0 : slabdata 0 0 0
cfq_ioc_pool 0 0 48 81 1 : tunables 120 60 0 : slabdata 0 0 0
cfq_pool 0 0 96 41 1 : tunables 120 60 0 : slabdata 0 0 0
crq_pool 0 0 44 88 1 : tunables 120 60 0 : slabdata 0 0 0
deadline_drq 0 0 48 81 1 : tunables 120 60 0 : slabdata 0 0 0
as_arq 65 130 60 65 1 : tunables 120 60 0 : slabdata 2 2 0
mqueue_inode_cache 1 7 512 7 1 : tunables 54 27 0 : slabdata 1 1 0
hugetlbfs_inode_cache 1 12 316 12 1 : tunables 54 27 0 : slabdata 1 1 0
ext2_inode_cache 0 0 444 9 1 : tunables 54 27 0 : slabdata 0 0 0
ext2_xattr 0 0 44 88 1 : tunables 120 60 0 : slabdata 0 0 0
journal_handle 8 185 20 185 1 : tunables 120 60 0 : slabdata 1 1 0
journal_head 2985 3000 52 75 1 : tunables 120 60 0 : slabdata 40 40 0
revoke_table 6 290 12 290 1 : tunables 120 60 0 : slabdata 1 1 0
revoke_record 0 0 16 226 1 : tunables 120 60 0 : slabdata 0 0 0
ext3_inode_cache 587635 589992 464 8 1 : tunables 54 27 0 : slabdata 73748 73749 0
ext3_xattr 0 0 44 88 1 : tunables 120 60 0 : slabdata 0 0 0
dnotify_cache 5 185 20 185 1 : tunables 120 60 0 : slabdata 1 1 0
eventpoll_pwq 0 0 36 107 1 : tunables 120 60 0 : slabdata 0 0 0
eventpoll_epi 0 0 128 31 1 : tunables 120 60 0 : slabdata 0 0 0
inotify_event_cache 0 0 28 135 1 : tunables 120 60 0 : slabdata 0 0 0
inotify_watch_cache 0 0 36 107 1 : tunables 120 60 0 : slabdata 0 0 0
kioctx 0 0 192 20 1 : tunables 120 60 0 : slabdata 0 0 0
kiocb 0 0 128 31 1 : tunables 120 60 0 : slabdata 0 0 0
fasync_cache 3 226 16 226 1 : tunables 120 60 0 : slabdata 1 1 0
shmem_inode_cache 963 963 408 9 1 : tunables 54 27 0 : slabdata 107 107 0
posix_timers_cache 0 0 96 41 1 : tunables 120 60 0 : slabdata 0 0 0
uid_cache 10 61 64 61 1 : tunables 120 60 0 : slabdata 1 1 0
blkdev_ioc 95 135 28 135 1 : tunables 120 60 0 : slabdata 1 1 0
blkdev_queue 25 30 380 10 1 : tunables 54 27 0 : slabdata 3 3 0
blkdev_requests 78 78 152 26 1 : tunables 120 60 0 : slabdata 3 3 0
biovec-(256) 256 256 3072 2 2 : tunables 24 12 0 : slabdata 128 128 0
biovec-128 256 260 1536 5 2 : tunables 24 12 0 : slabdata 52 52 0
biovec-64 256 260 768 5 1 : tunables 54 27 0 : slabdata 52 52 0
biovec-16 256 260 192 20 1 : tunables 120 60 0 : slabdata 13 13 0
biovec-4 258 305 64 61 1 : tunables 120 60 0 : slabdata 5 5 0
biovec-1 340 904 16 226 1 : tunables 120 60 0 : slabdata 4 4 0
bio 374 465 128 31 1 : tunables 120 60 0 : slabdata 14 15 0
file_lock_cache 45 45 88 45 1 : tunables 120 60 0 : slabdata 1 1 0
sock_inode_cache 570 570 384 10 1 : tunables 54 27 0 : slabdata 57 57 0
skbuff_head_cache 880 1160 192 20 1 : tunables 120 60 0 : slabdata 58 58 0
proc_inode_cache 672 672 332 12 1 : tunables 54 27 0 : slabdata 56 56 0
sigqueue 75 108 148 27 1 : tunables 120 60 0 : slabdata 4 4 0
radix_tree_node 27827 29162 276 14 1 : tunables 54 27 0 : slabdata 2083 2083 0
bdev_cache 7 9 448 9 1 : tunables 54 27 0 : slabdata 1 1 0
sysfs_dir_cache 3540 3552 40 96 1 : tunables 120 60 0 : slabdata 37 37 0
mnt_cache 28 31 128 31 1 : tunables 120 60 0 : slabdata 1 1 0
inode_cache 1251 1404 316 12 1 : tunables 54 27 0 : slabdata 117 117 0
dentry_cache 434515 514112 136 29 1 : tunables 120 60 0 : slabdata 17728 17728 0
filp 4500 4660 192 20 1 : tunables 120 60 0 : slabdata 233 233 0
names_cache 7 7 4096 1 1 : tunables 24 12 0 : slabdata 7 7 0
key_jar 20 31 128 31 1 : tunables 120 60 0 : slabdata 1 1 0
idr_layer_cache 91 116 136 29 1 : tunables 120 60 0 : slabdata 4 4 0
buffer_head 153510 162891 48 81 1 : tunables 120 60 0 : slabdata 2011 2011 0
mm_struct 119 119 576 7 1 : tunables 54 27 0 : slabdata 17 17 0
vm_area_struct 8115 8640 88 45 1 : tunables 120 60 0 : slabdata 192 192 0
fs_cache 113 119 32 119 1 : tunables 120 60 0 : slabdata 1 1 0
files_cache 114 117 448 9 1 : tunables 54 27 0 : slabdata 13 13 0
signal_cache 135 140 384 10 1 : tunables 54 27 0 : slabdata 14 14 0
sighand_cache 132 135 1344 3 1 : tunables 24 12 0 : slabdata 45 45 0
task_struct 150 153 1328 3 1 : tunables 24 12 0 : slabdata 51 51 0
anon_vma 3535 3663 8 407 1 : tunables 120 60 0 : slabdata 9 9 0
pgd 115 115 4096 1 1 : tunables 24 12 0 : slabdata 115 115 0
size-131072(DMA) 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0
size-131072 0 0 131072 1 32 : tunables 8 4 0 : slabdata 0 0 0
size-65536(DMA) 0 0 65536 1 16 : tunables 8 4 0 : slabdata 0 0 0
size-65536 0 0 65536 1 16 : tunables 8 4 0 : slabdata 0 0 0
size-32768(DMA) 0 0 32768 1 8 : tunables 8 4 0 : slabdata 0 0 0
size-32768 18 18 32768 1 8 : tunables 8 4 0 : slabdata 18 18 0
size-16384(DMA) 0 0 16384 1 4 : tunables 8 4 0 : slabdata 0 0 0
size-16384 1 1 16384 1 4 : tunables 8 4 0 : slabdata 1 1 0
size-8192(DMA) 0 0 8192 1 2 : tunables 8 4 0 : slabdata 0 0 0
size-8192 158 158 8192 1 2 : tunables 8 4 0 : slabdata 158 158 0
size-4096(DMA) 0 0 4096 1 1 : tunables 24 12 0 : slabdata 0 0 0
size-4096 385 387 4096 1 1 : tunables 24 12 0 : slabdata 385 387 0
size-2048(DMA) 0 0 2048 2 1 : tunables 24 12 0 : slabdata 0 0 0
size-2048 75 76 2048 2 1 : tunables 24 12 0 : slabdata 38 38 0
size-1024(DMA) 0 0 1024 4 1 : tunables 54 27 0 : slabdata 0 0 0
size-1024 212 212 1024 4 1 : tunables 54 27 0 : slabdata 53 53 0
size-512(DMA) 0 0 512 8 1 : tunables 54 27 0 : slabdata 0 0 0
size-512 375 456 512 8 1 : tunables 54 27 0 : slabdata 57 57 0
size-256(DMA) 0 0 256 15 1 : tunables 120 60 0 : slabdata 0 0 0
size-256 645 750 256 15 1 : tunables 120 60 0 : slabdata 50 50 0
size-192(DMA) 0 0 192 20 1 : tunables 120 60 0 : slabdata 0 0 0
size-192 100 100 192 20 1 : tunables 120 60 0 : slabdata 5 5 0
size-128(DMA) 0 0 128 31 1 : tunables 120 60 0 : slabdata 0 0 0
size-128 4259 4557 128 31 1 : tunables 120 60 0 : slabdata 147 147 0
size-64(DMA) 0 0 64 61 1 : tunables 120 60 0 : slabdata 0 0 0
size-64 150913 150914 64 61 1 : tunables 120 60 0 : slabdata 2474 2474 0
size-32(DMA) 0 0 32 119 1 : tunables 120 60 0 : slabdata 0 0 0
size-32 3273 3332 32 119 1 : tunables 120 60 0 : slabdata 28 28 0
kmem_cache 124 124 128 31 1 : tunables 120 60 0 : slabdata 4 4 0
[-- Attachment #3: meminfo --]
[-- Type: text/plain, Size: 670 bytes --]
MemTotal: 2074880 kB
MemFree: 15220 kB
Buffers: 339900 kB
Cached: 798368 kB
SwapCached: 18252 kB
Active: 1025436 kB
Inactive: 603900 kB
HighTotal: 1178944 kB
HighFree: 5952 kB
LowTotal: 895936 kB
LowFree: 9268 kB
SwapTotal: 2124352 kB
SwapFree: 2060040 kB
Dirty: 9356 kB
Writeback: 0 kB
Mapped: 691788 kB
Slab: 405400 kB
CommitLimit: 3161792 kB
Committed_AS: 1206060 kB
PageTables: 5276 kB
VmallocTotal: 114680 kB
VmallocUsed: 24256 kB
VmallocChunk: 89588 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 4096 kB
[-- Attachment #4: config.gz --]
[-- Type: application/octet-stream, Size: 11644 bytes --]
next reply other threads:[~2005-09-11 10:57 UTC|newest]
Thread overview: 64+ messages / expand[flat|nested] mbox.gz Atom feed top
2005-09-11 10:57 Theodore Ts'o [this message]
2005-09-11 12:00 ` VM balancing issues on 2.6.13: dentry cache not getting shrunk enough Dipankar Sarma
2005-09-11 12:00 ` Dipankar Sarma
2005-09-12 3:16 ` Theodore Ts'o
2005-09-12 3:16 ` Theodore Ts'o
2005-09-12 6:16 ` Martin J. Bligh
2005-09-12 6:16 ` Martin J. Bligh
2005-09-12 12:53 ` Bharata B Rao
2005-09-12 12:53 ` Bharata B Rao
2005-09-13 8:47 ` Bharata B Rao
2005-09-13 21:59 ` David Chinner
2005-09-13 21:59 ` David Chinner
2005-09-14 9:01 ` Andi Kleen
2005-09-14 9:01 ` Andi Kleen
2005-09-14 9:16 ` Manfred Spraul
2005-09-14 9:16 ` Manfred Spraul
2005-09-14 9:43 ` Andrew Morton
2005-09-14 9:43 ` Andrew Morton
2005-09-14 9:52 ` Dipankar Sarma
2005-09-14 9:52 ` Dipankar Sarma
2005-09-14 22:44 ` Theodore Ts'o
2005-09-14 22:44 ` Theodore Ts'o
2005-09-14 9:35 ` Andrew Morton
2005-09-14 9:35 ` Andrew Morton
2005-09-14 13:57 ` Martin J. Bligh
2005-09-14 13:57 ` Martin J. Bligh
2005-09-14 15:37 ` Sonny Rao
2005-09-14 15:37 ` Sonny Rao
2005-09-15 7:21 ` Helge Hafting
2005-09-15 7:21 ` Helge Hafting
2005-09-14 22:48 ` David Chinner
2005-09-14 22:48 ` David Chinner
2005-09-14 15:48 ` Sonny Rao
2005-09-14 15:48 ` Sonny Rao
2005-09-14 22:02 ` David Chinner
2005-09-14 22:02 ` David Chinner
2005-09-14 22:40 ` Sonny Rao
2005-09-14 22:40 ` Sonny Rao
2005-09-15 1:14 ` David Chinner
2005-09-15 1:14 ` David Chinner
2005-10-06 6:27 ` [PATCH] dcache: separate slab for directory dentries David Chinner, gnb
2005-10-06 12:28 ` Dave Kleikamp
2005-10-07 3:54 ` Greg Banks
2005-10-07 13:00 ` Dave Kleikamp
2005-09-14 21:34 ` VM balancing issues on 2.6.13: dentry cache not getting shrunk enough Marcelo Tosatti
2005-09-14 21:34 ` Marcelo Tosatti
2005-09-14 21:43 ` Dipankar Sarma
2005-09-14 21:43 ` Dipankar Sarma
2005-09-15 4:28 ` Bharata B Rao
2005-09-15 4:28 ` Bharata B Rao
2005-09-14 23:08 ` Marcelo Tosatti
2005-09-14 23:08 ` Marcelo Tosatti
2005-09-15 9:39 ` Bharata B Rao
2005-09-15 9:39 ` Bharata B Rao
2005-09-15 13:29 ` Marcelo Tosatti
2005-09-15 13:29 ` Marcelo Tosatti
2005-10-02 16:32 ` Bharata B Rao
2005-10-02 20:06 ` Marcelo
2005-10-02 20:06 ` Marcelo
2005-10-04 13:36 ` shrinkable cache statistics [was Re: VM balancing issues on 2.6.13: dentry cache not getting shrunk enough] Bharata B Rao
2005-10-05 21:25 ` Marcelo Tosatti
2005-10-05 21:25 ` Marcelo Tosatti
2005-10-07 8:12 ` Bharata B Rao
2005-10-07 8:12 ` Bharata B Rao
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20050911105709.GA16369@thunk.org \
--to=tytso@mit.edu \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.