Limit hash table size

Chen, Kenneth W
2004-01-08T23:12:35Z

[-- Attachment #1: Type: text/plain, Size: 1134 bytes --]

The issue of exceedingly large hash tables has been discussed on the
mailing list a while back, but seems to slip through the cracks.

What we found is it's not a problem for x86 (and most other
architectures) because __get_free_pages won't be able to get anything
beyond order MAX_ORDER-1 (10) which means at most those hash tables are
4MB each (assume 4K page size). However, on ia64, in order to support
larger hugeTLB page size, the MAX_ORDER is bumped up to 18, which now
means a 2GB upper limits enforced by the page allocator (assume 16K page
size). PPC64 is another example that bumps up MAX_ORDER.

Last time I checked, the tcp ehash table is taking a whooping (insane!)
2GB on one of our large machine. dentry and inode hash tables also take
considerable amount of memory.

This patch just enforces all the hash tables to have a max order of 10,
which limits them down to 16MB each on ia64. People can clean up other
part of table size calculation. But minimally, this patch doesn't
change any hash sizes already in use on x86.

Andrew, would you please apply?

- Ken Chen
<<hashtable.patch>>

[-- Attachment #2: hashtable.patch --]
[-- Type: application/octet-stream, Size: 2196 bytes --]

diff -Nurp linux-2.6.0.orig/fs/dcache.c linux-2.6.0/fs/dcache.c
--- linux-2.6.0.orig/fs/dcache.c 2003-12-17 18:58:15.000000000 -0800
+++ linux-2.6.0/fs/dcache.c 2004-01-08 14:59:58.000000000 -0800
@@ -1571,11 +1571,9 @@ static void __init dcache_init(unsigned

set_shrinker(DEFAULT_SEEKS, shrink_dcache_memory);

-#if PAGE_SHIFT < 13
- mempages >>= (13 - PAGE_SHIFT);
-#endif
+ mempages >>= 1;
mempages *= sizeof(struct hlist_head);
- for (order = 0; ((1UL << order) << PAGE_SHIFT) < mempages; order++)
+ for (order = 0; (order < 10) && (((1UL << order) << PAGE_SHIFT) < mempages); order++)
;

do {
diff -Nurp linux-2.6.0.orig/fs/inode.c linux-2.6.0/fs/inode.c
--- linux-2.6.0.orig/fs/inode.c 2003-12-17 18:59:55.000000000 -0800
+++ linux-2.6.0/fs/inode.c 2004-01-08 15:00:19.000000000 -0800
@@ -1340,9 +1340,9 @@ void __init inode_init(unsigned long mem
for (i = 0; i < ARRAY_SIZE(i_wait_queue_heads); i++)
init_waitqueue_head(&i_wait_queue_heads[i].wqh);

- mempages >>= (14 - PAGE_SHIFT);
+ mempages >>= 2;
mempages *= sizeof(struct hlist_head);
- for (order = 0; ((1UL << order) << PAGE_SHIFT) < mempages; order++)
+ for (order = 0; (order < 10) && (((1UL << order) << PAGE_SHIFT) < mempages); order++)
;

do {
diff -Nurp linux-2.6.0.orig/net/ipv4/route.c linux-2.6.0/net/ipv4/route.c
--- linux-2.6.0.orig/net/ipv4/route.c 2003-12-17 18:59:55.000000000 -0800
+++ linux-2.6.0/net/ipv4/route.c 2004-01-08 15:01:17.000000000 -0800
@@ -2747,7 +2747,7 @@ int __init ip_rt_init(void)

goal = num_physpages >> (26 - PAGE_SHIFT);

- for (order = 0; (1UL << order) < goal; order++)
+ for (order = 0; (order < 10) && ((1UL << order) < goal); order++)
/* NOTHING */;

do {
diff -Nurp linux-2.6.0.orig/net/ipv4/tcp.c linux-2.6.0/net/ipv4/tcp.c
--- linux-2.6.0.orig/net/ipv4/tcp.c 2003-12-17 18:58:38.000000000 -0800
+++ linux-2.6.0/net/ipv4/tcp.c 2004-01-08 15:00:42.000000000 -0800
@@ -2610,7 +2610,7 @@ void __init tcp_init(void)
else
goal = num_physpages >> (23 - PAGE_SHIFT);

- for (order = 0; (1UL << order) < goal; order++)
+ for (order = 0; (order < 10) && ((1UL << order) < goal); order++)
;
do {
tcp_ehash_size = (1UL << order) * PAGE_SIZE /