Re: fsstress causes memory leak in test6, test8

Re: fsstress causes memory leak in test6, test8

[Andrew Morton]

Burton Windle <bwindle@fint.org> wrote:
>
> There is a memory leak in the test6 - test8 kernels when doing heavy disk
>  IO. After running fsstress, there is a significant ammount of memory that
>  is unaccounted for.

OK, this is simple, but fixing it invalidates lots of testing of the
traditionally-tricky VFS cache shrinking code.

It is not a "leak" as such - the dentries will get shrunk in normal usage
(create enough non-dir dentries and the "leaked" directory dentries will
get reclaimed).  The really deep directories which fsstress creates
demonstrated the bug.


Given that it took a year for anyone to notice, it's probably best that
this not be included for 2.6.0.




This fixes the recently-reported "fsstress memory leak" problem.  It has been
there since November 2002.

shrink_dcache() has a heuristic to prevent the dcache (and hence icache) from
getting shrunk too far: it refuses to allow the dcache to shrink below
2*nr_used.

Problem is, _all_ non-leaf dentries (directories) count as used.  So when you
have really deep directory hierarchies (fsstress creates these), nr_used is
really high, and there is no upper bound to the amount of pinned dcache.

The patch just rips out the heuristic.  This means that dcache (and hence
icache (and hence pagecache)) will be shrunk more aggressively.  This could
be a problem, and tons of testing is needed - a new heuristic may be needed.

However I am not able to reproduce the problem which cause me to add this
heuristic in the first place:

   Simple testcase: run a huge `dd' while running a concurrent `watch -n1
   cat /proc/meminfo'.  The program text for `cat' gets loaded from disk once
   per second.




 fs/dcache.c |   21 +--------------------
 1 files changed, 1 insertion(+), 20 deletions(-)

diff -puN fs/dcache.c~dentry-bloat-fix-2 fs/dcache.c
--- 25/fs/dcache.c~dentry-bloat-fix-2	2003-10-26 04:00:13.000000000 -0800
+++ 25-akpm/fs/dcache.c	2003-10-26 16:31:05.000000000 -0800
@@ -639,24 +639,9 @@ void shrink_dcache_anon(struct hlist_hea
 
 /*
  * This is called from kswapd when we think we need some more memory.
- *
- * We don't want the VM to steal _all_ unused dcache.  Because that leads to
- * the VM stealing all unused inodes, which shoots down recently-used
- * pagecache.  So what we do is to tell fibs to the VM about how many reapable
- * objects there are in this cache.   If the number of unused dentries is
- * less than half of the total dentry count then return zero.  The net effect
- * is that the number of unused dentries will be, at a minimum, equal to the
- * number of used ones.
- *
- * If unused_ratio is set to 5, the number of unused dentries will not fall
- * below 5* the number of used ones.
  */
 static int shrink_dcache_memory(int nr, unsigned int gfp_mask)
 {
-	int nr_used;
-	int nr_unused;
-	const int unused_ratio = 1;
-
 	if (nr) {
 		/*
 		 * Nasty deadlock avoidance.
@@ -672,11 +657,7 @@ static int shrink_dcache_memory(int nr, 
 		if (gfp_mask & __GFP_FS)
 			prune_dcache(nr);
 	}
-	nr_unused = dentry_stat.nr_unused;
-	nr_used = dentry_stat.nr_dentry - nr_unused;
-	if (nr_unused < nr_used * unused_ratio)
-		return 0;
-	return nr_unused - nr_used * unused_ratio;
+	return dentry_stat.nr_unused;
 }
 
 #define NAME_ALLOC_LEN(len)	((len+16) & ~15)

_

Re: fsstress causes memory leak in test6, test8

[Linus Torvalds]

On Sun, 26 Oct 2003, Andrew Morton wrote:
> 
> Given that it took a year for anyone to notice, it's probably best that
> this not be included for 2.6.0.

I agree, let's see if we ever see this as a real problem. Removing the 
heuristic might be worth it at some point, but let's not do it yet.

		Linus

Re: fsstress causes memory leak in test6, test8

[Dave Jones]

On Sun, Oct 26, 2003 at 05:02:41PM -0800, Andrew Morton wrote:

 > It is not a "leak" as such - the dentries will get shrunk in normal usage
 > (create enough non-dir dentries and the "leaked" directory dentries will
 > get reclaimed).  The really deep directories which fsstress creates
 > demonstrated the bug.

This could explain the random reiserfs oopses/hangs I was seeing several
months back after running fsstress for a day or so. The reiser folks
were scratching their heads, and we even put it down to flaky hardware
or maybe even a CPU bug back then.

 > Given that it took a year for anyone to notice, it's probably best that
 > this not be included for 2.6.0.

I agree in a "lets get 2.6 out the door" sense, but once thats 'out
there' a user-level DoS should be fixed up pretty quickly.
The paranoid could always run 2.6-mm I guess 8-)

		Dave

-- 
 Dave Jones     http://www.codemonkey.org.uk

Re: fsstress causes memory leak in test6, test8

[Hans Reiser]

Dave Jones wrote:

>On Sun, Oct 26, 2003 at 05:02:41PM -0800, Andrew Morton wrote:
>
> > It is not a "leak" as such - the dentries will get shrunk in normal usage
> > (create enough non-dir dentries and the "leaked" directory dentries will
> > get reclaimed).  The really deep directories which fsstress creates
> > demonstrated the bug.
>
>This could explain the random reiserfs oopses/hangs I was seeing several
>months back after running fsstress for a day or so. The reiser folks
>were scratching their heads, and we even put it down to flaky hardware
>or maybe even a CPU bug back then.
>  
>
This means we failed to make a properly serious effort at replicating it 
on our hardware.  My apologies for that.  Who at Namessys was it that 
investigated your bug report?

-- 
Hans

Re: fsstress causes memory leak in test6, test8

[Dave Jones]

On Mon, Oct 27, 2003 at 03:31:16PM +0300, Hans Reiser wrote:
 > >This could explain the random reiserfs oopses/hangs I was seeing several
 > >months back after running fsstress for a day or so. The reiser folks
 > >were scratching their heads, and we even put it down to flaky hardware
 > >or maybe even a CPU bug back then.
 > > 
 > This means we failed to make a properly serious effort at replicating it 
 > on our hardware.  My apologies for that.

It happens.

 > Who at Namessys was it that 
 > investigated your bug report?

Can't remember, it's in the archives somewhere.

		Dave

-- 
 Dave Jones     http://www.codemonkey.org.uk

Re: fsstress causes memory leak in test6, test8

[Nikita Danilov]

Dave Jones writes:
 > On Sun, Oct 26, 2003 at 05:02:41PM -0800, Andrew Morton wrote:
 > 
 >  > It is not a "leak" as such - the dentries will get shrunk in normal usage
 >  > (create enough non-dir dentries and the "leaked" directory dentries will
 >  > get reclaimed).  The really deep directories which fsstress creates
 >  > demonstrated the bug.
 > 
 > This could explain the random reiserfs oopses/hangs I was seeing several
 > months back after running fsstress for a day or so. The reiser folks

This could explain hangs, but hardly oopses. System just freezes due to
out-of-memory.

 > were scratching their heads, and we even put it down to flaky hardware
 > or maybe even a CPU bug back then.

Of course we did, there are no bugs in reiserfs, you know. :)

 > 
 >  > Given that it took a year for anyone to notice, it's probably best that
 >  > this not be included for 2.6.0.
 > 
 > I agree in a "lets get 2.6 out the door" sense, but once thats 'out
 > there' a user-level DoS should be fixed up pretty quickly.
 > The paranoid could always run 2.6-mm I guess 8-)
 > 
 > 		Dave
 > 

Nikita.

Re: fsstress causes memory leak in test6, test8

[Hans Reiser]

Nikita Danilov wrote:

>Dave Jones writes:
> > On Sun, Oct 26, 2003 at 05:02:41PM -0800, Andrew Morton wrote:
> > 
> >  > It is not a "leak" as such - the dentries will get shrunk in normal usage
> >  > (create enough non-dir dentries and the "leaked" directory dentries will
> >  > get reclaimed).  The really deep directories which fsstress creates
> >  > demonstrated the bug.
> > 
> > This could explain the random reiserfs oopses/hangs I was seeing several
> > months back after running fsstress for a day or so. The reiser folks
>
>This could explain hangs, but hardly oopses. System just freezes due to
>out-of-memory.
>
Ah, thanks Nikita.

-- 
Hans