Re: strange behavior creating and deleting files

[Mpourtounis Dimitris <db@wless.gr>]

On Fri, 2004-09-24 at 21:40, Hugh Dickins wrote:
> On Fri, 24 Sep 2004, Mpourtounis Dimitris wrote:
> > 
> > there seems to be a strange behaviour in the way my system creates and
> > deletes files, as long as memory allocation is concerned.
> > 
> > running a simple script that continuously creates and deletes files on
> > tmpfs filesystem, a got the following results:
> > 
> > files created		free memory on system 
> > -------------		---------------------
> > 0			48180
> > +6000			47936
> > +6000			47372
> > +6000			47372
> > +6000			47936
> > +6000			47936
> > +6000			47936
> > +6000			47936		(seems stable)
> > +9000			46976		(what on earth?)
> > +30000			45084
> > +80000			45084		(again stable)
> > +70000			39156		(not again...:( )
> > 
> > and sometime in the morning 25000 MB free RAM, and my system running too
> > slow
> > 
> > I am sure these are a lot a files and under normal conditions, there
> > will never be made and deleted so many.
> > 
> > It is that misbehavior of being stable for a long time and then again
> > allocating memory that concerns me.
> 
> I'll admit that I don't _fully_ understand it either.  Another user
> noticed related behaviour a couple of months ago, below is the patch
> which I hope you find fixes yours too.  An equivalent fix (to fs/libfs.c)
> went into 2.6.9-rc1-bk1, I'd been waiting for it to get more 2.6 exposure
> (lest unwanted side-effects appeared) before sending Marcelo the 2.4 fix.
> If it works well for you, let me know and I'll send it in for 2.4.28...
> 
I seems to work fine. Having created-deleted about 50000 files there is
no leak. Hope it continues that way.
Great job, thanks !

DB

> A tmpfs user reported increasingly slow directory reads when repeatedly
> creating and unlinking in a mkstemp-like way.  The negative dentries
> accumulate alarmingly (until memory pressure finally frees them), and
> are just a hindrance to any in-memory filesystem.  shmem_lookup set
> d_op to arrange for negative dentries to be deleted immediately.
> 
> (But I failed to discover how it is that on-disk filesystems seem to
> keep their negative dentries within manageable bounds: this effect was
> gross with tmpfs or ramfs, but no problem at all with extN or reiser.)
> 
> Signed-off-by: Hugh Dickins <hugh@veritas.com>
> 
> --- 2.4.26/mm/shmem.c	2003-11-28 18:26:21.000000000 +0000
> +++ linux/mm/shmem.c	2004-07-30 12:48:31.324978800 +0100
> @@ -1161,13 +1161,27 @@ static int shmem_statfs(struct super_blo
>  }
>  
>  /*
> + * Retaining negative dentries for an in-memory filesystem just wastes
> + * memory and lookup time: arrange for them to be deleted immediately.
> + */
> +static int shmem_delete_dentry(struct dentry *dentry)
> +{
> +	return 1;
> +}
> +
> +/*
>   * Lookup the data. This is trivial - if the dentry didn't already
> - * exist, we know it is negative.
> + * exist, we know it is negative.  Set d_op to delete negative dentries.
>   */
>  static struct dentry *shmem_lookup(struct inode *dir, struct dentry *dentry)
>  {
> +	static struct dentry_operations shmem_dentry_operations = {
> +		.d_delete = shmem_delete_dentry,
> +	};
> +
>  	if (dentry->d_name.len > NAME_MAX)
>  		return ERR_PTR(-ENAMETOOLONG);
> +	dentry->d_op = &shmem_dentry_operations;
>  	d_add(dentry, NULL);
>  	return NULL;
>  }