Re: dentry cache leak? Re: rsync out of memory 2.6.8-rc2

Linus Torvalds
2004-07-31T17:24:00Z

On Fri, 30 Jul 2004, Andrew Morton wrote:
>
> Seems that we reach a state where lowmem pagecache get reclaimed faster
> than dcache/icache. This causes the number of pages scanned for lowmem
> allocations to fall. This causes less scanning of the slab and the whole
> thing repeats. I expect changing nr_used_zone_pages() to ignore highmem
> will fix it, and might be the long-term fix, too.

Ouch. Indeed, looking at the usage of nr_used_zone_pages(), it looks
totally broken. However, I don't think the right thing to do is to ignore
the HIGHMEM zone altogether, I think it should take the list of zones
being shrunk into account.

Look at usage: one caller of shrink_slab() is try_to_free_pages(), and if
try_to_free_pages() is called for a HIGHMEM zone, then it is _fine_ to
count the HIGHMEM zone as pages, and shrink the slab memory much less
aggressively. The _problem_ is when slab competes against itself (lowmem)
or some other thing (inodes) that want lowmem-only memory, and then the
slab shrinking really should ignore the fact that there are tons of
highmem pages left.

Something like this (totally untested, may not compile, you get the idea)
might work. Or not. Since the _rest_ of "shrink_slab()" doesn't know about
zonelists, just making the "how many pages does this zone have free" take
the zonelist into account might cause other problems.

Linus
-----
===== include/linux/mm.h 1.179 vs edited =====
--- 1.179/include/linux/mm.h 2004-06-27 00:19:35 -07:00
+++ edited/include/linux/mm.h 2004-07-31 10:19:25 -07:00
@@ -706,7 +706,7 @@

extern struct vm_area_struct *find_extend_vma(struct mm_struct *mm, unsigned long addr);

-extern unsigned int nr_used_zone_pages(void);
+extern unsigned int nr_used_zone_pages(struct zone **);

extern struct page * vmalloc_to_page(void *addr);
extern struct page * follow_page(struct mm_struct *mm, unsigned long address,
===== mm/page_alloc.c 1.221 vs edited =====
--- 1.221/mm/page_alloc.c 2004-07-19 08:44:36 -07:00
+++ edited/mm/page_alloc.c 2004-07-31 10:18:37 -07:00
@@ -825,13 +825,15 @@

EXPORT_SYMBOL(nr_free_pages);

-unsigned int nr_used_zone_pages(void)
+unsigned int nr_used_zone_pages(struct zone **zones)
{
unsigned int pages = 0;
- struct zone *zone;

- for_each_zone(zone)
+ while (*zones) {
+ struct zone * zone = *zones;
+ zones++;
pages += zone->nr_active + zone->nr_inactive;
+ }

return pages;
}
===== mm/vmscan.c 1.224 vs edited =====
--- 1.224/mm/vmscan.c 2004-06-24 01:56:14 -07:00
+++ edited/mm/vmscan.c 2004-07-31 10:20:17 -07:00
@@ -170,7 +170,7 @@
*
* We do weird things to avoid (scanned*seeks*entries) overflowing 32 bits.
*/
-static int shrink_slab(unsigned long scanned, unsigned int gfp_mask)
+static int shrink_slab(unsigned long scanned, unsigned int gfp_mask, struct zone **zones)
{
struct shrinker *shrinker;
long pages;
@@ -178,7 +178,7 @@
if (down_trylock(&shrinker_sem))
return 0;

- pages = nr_used_zone_pages();
+ pages = nr_used_zone_pages(zones);
list_for_each_entry(shrinker, &shrinker_list, list) {
unsigned long long delta;

@@ -912,7 +912,7 @@
sc.nr_reclaimed = 0;
sc.priority = priority;
shrink_caches(zones, &sc);
- shrink_slab(sc.nr_scanned, gfp_mask);
+ shrink_slab(sc.nr_scanned, gfp_mask, zones);
if (reclaim_state) {
sc.nr_reclaimed += reclaim_state->reclaimed_slab;
reclaim_state->reclaimed_slab = 0;
@@ -1032,6 +1032,7 @@
*/
for (i = 0; i <= end_zone; i++) {
struct zone *zone = pgdat->node_zones + i;
+ struct zone * zones[2] = { zone, NULL };

if (zone->all_unreclaimable && priority != DEF_PRIORITY)
continue;
@@ -1048,7 +1049,7 @@
sc.priority = priority;
shrink_zone(zone, &sc);
reclaim_state->reclaimed_slab = 0;
- shrink_slab(sc.nr_scanned, GFP_KERNEL);
+ shrink_slab(sc.nr_scanned, GFP_KERNEL, zones);
sc.nr_reclaimed += reclaim_state->reclaimed_slab;
total_reclaimed += sc.nr_reclaimed;
if (zone->all_unreclaimable)