Strange dcache memory pressure when highmem enabled

Strange dcache memory pressure when highmem enabled

[Neil Brown]

Greetings all.

 I have a fairly busy NFS server which has been having performance
 problems lately.  I have managed to work around the problems, but
 would really like to get the root problem fixed.

 The symptom is sluggish performance.
 A more useful symptom is that the number of entries in dentry_cache
 (as reported by /proc/slabinfo) is hanging around 1000, whereas on
 other fileservers it is typically more like 200,000. (I can
 understand how this would correlate directly with the perceived
 sluggishness)

 The key difference between the problematic fileserver and the happy
 fileservers seems to be that the problematic one has HIGHMEM.  It has
 4Gig or RAM where as the others have 512M or 1G.
 When I boot the problematic server with mem=900M the symptom goes
 away (there are plenty of entries in the dentry_cache).

 I noticed this in 2.4.18, and confirmed that it still happens with
 2.4.22. 

 So my question is:
   Why does 3Gig of highmem impose excessive memory pressure on
   dentry_cache (and inode_cache I think).

 
 I tried having a look and found "shrink_caches" in mm/vmscan.c, and
 "try_to_free_pages_zone" which calls it.
 I noticed that shrink_caches calls shrink_dcache_memory independant
 of the classzone that is being shrunk.  So if we are trying to
 shrink ZONE_HIGHMEM, the dentry_cache is shrunk, even though the
 dentry_cache doesn't live in highmem.  However I'm not sure if I have
 understood the classzones well enough for that observation even to
 make sense.

 Help appreciated.

NeilBrown

Re: Strange dcache memory pressure when highmem enabled

[Bryan O'Sullivan]

On Tue, 2003-10-14 at 21:33, Neil Brown wrote:

>  I have a fairly busy NFS server which has been having performance
>  problems lately.  I have managed to work around the problems, but
>  would really like to get the root problem fixed.

Funny, I have seen exactly the same problem, though with a Red Hat
2.4.20 kernel, rather than a vanilla-ish 2.4.

I have a few thousand more entries in my dentry cache on a 2G machine,
but it's still a pitiful number.

What workarounds did you find?

	<b

Re: Strange dcache memory pressure when highmem enabled

[Neil Brown]

On Tuesday October 14, bos@serpentine.com wrote:
> On Tue, 2003-10-14 at 21:33, Neil Brown wrote:
> 
> >  I have a fairly busy NFS server which has been having performance
> >  problems lately.  I have managed to work around the problems, but
> >  would really like to get the root problem fixed.
> 
> Funny, I have seen exactly the same problem, though with a Red Hat
> 2.4.20 kernel, rather than a vanilla-ish 2.4.
> 
> I have a few thousand more entries in my dentry cache on a 2G machine,
> but it's still a pitiful number.
> 
> What workarounds did you find?

 I just boot with "mem=900M" and effectively removed the highmem.
 It's not ideal, but it is the best I have found.

NeilBrown

Re: Strange dcache memory pressure when highmem enabled

[Andrew Morton]

Neil Brown <neilb@cse.unsw.edu.au> wrote:
>
> I noticed that shrink_caches calls shrink_dcache_memory independant
>   of the classzone that is being shrunk.  So if we are trying to
>   shrink ZONE_HIGHMEM, the dentry_cache is shrunk, even though the
>   dentry_cache doesn't live in highmem.  However I'm not sure if I have
>   understood the classzones well enough for that observation even to
>   make sense.

Makes heaps of sense.  Here's an instabackport of what we did in 2.6:

 mm/vmscan.c |   12 +++++++++---
 1 files changed, 9 insertions(+), 3 deletions(-)

diff -puN mm/vmscan.c~a mm/vmscan.c
--- 24/mm/vmscan.c~a	2003-10-14 22:41:34.000000000 -0700
+++ 24-akpm/mm/vmscan.c	2003-10-14 22:42:22.000000000 -0700
@@ -640,11 +640,17 @@ int try_to_free_pages_zone(zone_t *class
 			nr_pages = shrink_caches(classzone, gfp_mask, nr_pages, &failed_swapout);
 			if (nr_pages <= 0)
 				return 1;
-			shrink_dcache_memory(vm_vfs_scan_ratio, gfp_mask);
-			shrink_icache_memory(vm_vfs_scan_ratio, gfp_mask);
+			if (classzone - classzone->zone_pgdat->node_zones <
+						ZONE_HIGHMEM) {
+				shrink_dcache_memory(vm_vfs_scan_ratio,
+							gfp_mask);
+				shrink_icache_memory(vm_vfs_scan_ratio,
+							gfp_mask);
 #ifdef CONFIG_QUOTA
-			shrink_dqcache_memory(vm_vfs_scan_ratio, gfp_mask);
+				shrink_dqcache_memory(vm_vfs_scan_ratio,
+							gfp_mask);
 #endif
+			}
 			if (!failed_swapout)
 				failed_swapout = !swap_out(classzone);
 		} while (--tries);

_

Re: Strange dcache memory pressure when highmem enabled

[Neil Brown]

On Tuesday October 14, akpm@osdl.org wrote:
> Neil Brown <neilb@cse.unsw.edu.au> wrote:
> >
> > I noticed that shrink_caches calls shrink_dcache_memory independant
> >   of the classzone that is being shrunk.  So if we are trying to
> >   shrink ZONE_HIGHMEM, the dentry_cache is shrunk, even though the
> >   dentry_cache doesn't live in highmem.  However I'm not sure if I have
> >   understood the classzones well enough for that observation even to
> >   make sense.
> 
> Makes heaps of sense.  Here's an instabackport of what we did in
> 2.6:

Hey!!! That's what I call *Service*.

I'll give it a try tomorrow (let the poor students get a feeling of
stability first before I start changing things again :-)

> +			if (classzone - classzone->zone_pgdat->node_zones <
> +						ZONE_HIGHMEM) {

That's the bit I was missing.  I feel that once I fully understand
that, I will be a long way towards understanding the zones memory
management :-)

NeilBrown

Re: Strange dcache memory pressure when highmem enabled

[Stephan von Krawczynski]

On Wed, 15 Oct 2003 15:46:11 +1000
Neil Brown <neilb@cse.unsw.edu.au> wrote:

> On Tuesday October 14, akpm@osdl.org wrote:
> > Neil Brown <neilb@cse.unsw.edu.au> wrote:
> > >
> > > I noticed that shrink_caches calls shrink_dcache_memory independant
> > >   of the classzone that is being shrunk.  So if we are trying to
> > >   shrink ZONE_HIGHMEM, the dentry_cache is shrunk, even though the
> > >   dentry_cache doesn't live in highmem.  However I'm not sure if I have
> > >   understood the classzones well enough for that observation even to
> > >   make sense.
> > 
> > Makes heaps of sense.  Here's an instabackport of what we did in
> > 2.6:
> 
> Hey!!! That's what I call *Service*.
> 
> I'll give it a try tomorrow (let the poor students get a feeling of
> stability first before I start changing things again :-)
> 
> > +			if (classzone - classzone->zone_pgdat->node_zones <
> > +						ZONE_HIGHMEM) {
> 
> That's the bit I was missing.  I feel that once I fully understand
> that, I will be a long way towards understanding the zones memory
> management :-)
> 
> NeilBrown

I have another simple advice for you: use the latest 2.4.23-pre. I think I saw
your problem and it seems solved.

Regards,
Stephan

Re: Strange dcache memory pressure when highmem enabled

[Andrea Arcangeli]

On Tue, Oct 14, 2003 at 10:43:52PM -0700, Andrew Morton wrote:
> Neil Brown <neilb@cse.unsw.edu.au> wrote:
> >
> > I noticed that shrink_caches calls shrink_dcache_memory independant
> >   of the classzone that is being shrunk.  So if we are trying to
> >   shrink ZONE_HIGHMEM, the dentry_cache is shrunk, even though the
> >   dentry_cache doesn't live in highmem.  However I'm not sure if I have
> >   understood the classzones well enough for that observation even to
> >   make sense.
> 
> Makes heaps of sense.  Here's an instabackport of what we did in 2.6:
> 
>  mm/vmscan.c |   12 +++++++++---
>  1 files changed, 9 insertions(+), 3 deletions(-)
> 
> diff -puN mm/vmscan.c~a mm/vmscan.c
> --- 24/mm/vmscan.c~a	2003-10-14 22:41:34.000000000 -0700
> +++ 24-akpm/mm/vmscan.c	2003-10-14 22:42:22.000000000 -0700
> @@ -640,11 +640,17 @@ int try_to_free_pages_zone(zone_t *class
>  			nr_pages = shrink_caches(classzone, gfp_mask, nr_pages, &failed_swapout);
>  			if (nr_pages <= 0)
>  				return 1;
> -			shrink_dcache_memory(vm_vfs_scan_ratio, gfp_mask);
> -			shrink_icache_memory(vm_vfs_scan_ratio, gfp_mask);
> +			if (classzone - classzone->zone_pgdat->node_zones <
> +						ZONE_HIGHMEM) {
> +				shrink_dcache_memory(vm_vfs_scan_ratio,
> +							gfp_mask);
> +				shrink_icache_memory(vm_vfs_scan_ratio,
> +							gfp_mask);
>  #ifdef CONFIG_QUOTA
> -			shrink_dqcache_memory(vm_vfs_scan_ratio, gfp_mask);
> +				shrink_dqcache_memory(vm_vfs_scan_ratio,
> +							gfp_mask);
>  #endif
> +			}
>  			if (!failed_swapout)
>  				failed_swapout = !swap_out(classzone);
>  		} while (--tries);

An highmem user can make use of lowmem too. Think a 1G box, it looks
wrong to disallow highmem to shrink the 800M of dcache in the normal/dma
zones.

I wonder if what he's suffering from is a reduced normal zone due the
mem_map_t being larger. The reduced normal zone will trigger the dcache
shrinking more frequently. But he may want to try again with 2.4.23pre7
with a classzone aware refill_inactive that will ensure the inactive
list has enough lowmem pages before shrink_caches claims failure.

Andrea - If you prefer relying on open source software, check these links:
	    rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
	    http://www.cobite.com/cvsps/

Re: Strange dcache memory pressure when highmem enabled

[Neil Brown]

On Thursday October 16, andrea@suse.de wrote:
> 
> I wonder if what he's suffering from is a reduced normal zone due the
> mem_map_t being larger. The reduced normal zone will trigger the dcache
> shrinking more frequently. But he may want to try again with 2.4.23pre7
> with a classzone aware refill_inactive that will ensure the inactive
> list has enough lowmem pages before shrink_caches claims failure.
> 

I didn't end up trying Andrew's patch, but tried 2.4.23-pre7 instead.
It appears to be doing the right thing.
Free Highmem (grep HighFree /proc/meminfo) steadily dropped from 3Gig
to about 2-3 Meg and stayed there.
The dentry cache (grep dentry_cache /proc/slabinfo) climbed up to
about 500,000 and stayed there for 24 hours - much better than before
where it would often be only a few thousand and we had complaints every
night when the backups ran.

Thanks.

NeilBrown

Re: Strange dcache memory pressure when highmem enabled

[Andrea Arcangeli]

On Wed, Oct 22, 2003 at 08:57:52AM +1000, Neil Brown wrote:
> I didn't end up trying Andrew's patch, but tried 2.4.23-pre7 instead.
> It appears to be doing the right thing.
> Free Highmem (grep HighFree /proc/meminfo) steadily dropped from 3Gig
> to about 2-3 Meg and stayed there.
> The dentry cache (grep dentry_cache /proc/slabinfo) climbed up to
> about 500,000 and stayed there for 24 hours - much better than before
> where it would often be only a few thousand and we had complaints every
> night when the backups ran.

sounds good. And unless I misread something Andrew's patch shrinking
lowmem only for highmem allocation definitely looks wrong and it would
be DoSable as well.

As for 2.4 mainline there's some bit of refill_inactive still not fully
classzone aware, it's the half merge bit mentioned earlier on the list
that also introduced a typo kind of bug. My tree is fully aware instead
to avoid falling apart on the 32G boxes in production. Those secondary
bits in refill_inactive should be merged in mainline too over time.

Anyways for a 1G machine (or maybe with 2G too), those bits could hardly
make any difference, I assume your highmem/lowmem ratio isn't too high.

thanks,

Andrea - If you prefer relying on open source software, check these links:
	    rsync.kernel.org::pub/scm/linux/kernel/bkcvs/linux-2.[45]/
	    http://www.cobite.com/cvsps/