strange behavior creating and deleting files

strange behavior creating and deleting files

[Mpourtounis Dimitris]

Hi all,

there seems to be a strange behaviour in the way my system creates and
deletes files, as long as memory allocation is concerned.

running a simple script that continuously creates and deletes files on
tmpfs filesystem, a got the following results:

files created		free memory on system 
-------------		---------------------
0			48180
+6000			47936
+6000			47372
+6000			47372
+6000			47936
+6000			47936
+6000			47936
+6000			47936		(seems stable)
+9000			46976		(what on earth?)
+30000			45084
+80000			45084		(again stable)
+70000			39156		(not again...:( )

and sometime in the morning 25000 MB free RAM, and my system running too
slow

I am sure these are a lot a files and under normal conditions, there
will never be made and deleted so many.

It is that misbehavior of being stable for a long time and then again
allocating memory that concerns me.

I am running linux 2.4.26 on an x86 platform (gcc 3.2.3 uclib 0.9.20) 

Simple sh file:
i=0
while [ 1 ] do
echo "dont allocate more memory please" > $i
rm $i
let i=$i+1
done

Any clue???

Re: strange behavior creating and deleting files

[Hugh Dickins]

On Fri, 24 Sep 2004, Mpourtounis Dimitris wrote:
> 
> there seems to be a strange behaviour in the way my system creates and
> deletes files, as long as memory allocation is concerned.
> 
> running a simple script that continuously creates and deletes files on
> tmpfs filesystem, a got the following results:
> 
> files created		free memory on system 
> -------------		---------------------
> 0			48180
> +6000			47936
> +6000			47372
> +6000			47372
> +6000			47936
> +6000			47936
> +6000			47936
> +6000			47936		(seems stable)
> +9000			46976		(what on earth?)
> +30000			45084
> +80000			45084		(again stable)
> +70000			39156		(not again...:( )
> 
> and sometime in the morning 25000 MB free RAM, and my system running too
> slow
> 
> I am sure these are a lot a files and under normal conditions, there
> will never be made and deleted so many.
> 
> It is that misbehavior of being stable for a long time and then again
> allocating memory that concerns me.

I'll admit that I don't _fully_ understand it either.  Another user
noticed related behaviour a couple of months ago, below is the patch
which I hope you find fixes yours too.  An equivalent fix (to fs/libfs.c)
went into 2.6.9-rc1-bk1, I'd been waiting for it to get more 2.6 exposure
(lest unwanted side-effects appeared) before sending Marcelo the 2.4 fix.
If it works well for you, let me know and I'll send it in for 2.4.28...

A tmpfs user reported increasingly slow directory reads when repeatedly
creating and unlinking in a mkstemp-like way.  The negative dentries
accumulate alarmingly (until memory pressure finally frees them), and
are just a hindrance to any in-memory filesystem.  shmem_lookup set
d_op to arrange for negative dentries to be deleted immediately.

(But I failed to discover how it is that on-disk filesystems seem to
keep their negative dentries within manageable bounds: this effect was
gross with tmpfs or ramfs, but no problem at all with extN or reiser.)

Signed-off-by: Hugh Dickins <hugh@veritas.com>

--- 2.4.26/mm/shmem.c	2003-11-28 18:26:21.000000000 +0000
+++ linux/mm/shmem.c	2004-07-30 12:48:31.324978800 +0100
@@ -1161,13 +1161,27 @@ static int shmem_statfs(struct super_blo
 }
 
 /*
+ * Retaining negative dentries for an in-memory filesystem just wastes
+ * memory and lookup time: arrange for them to be deleted immediately.
+ */
+static int shmem_delete_dentry(struct dentry *dentry)
+{
+	return 1;
+}
+
+/*
  * Lookup the data. This is trivial - if the dentry didn't already
- * exist, we know it is negative.
+ * exist, we know it is negative.  Set d_op to delete negative dentries.
  */
 static struct dentry *shmem_lookup(struct inode *dir, struct dentry *dentry)
 {
+	static struct dentry_operations shmem_dentry_operations = {
+		.d_delete = shmem_delete_dentry,
+	};
+
 	if (dentry->d_name.len > NAME_MAX)
 		return ERR_PTR(-ENAMETOOLONG);
+	dentry->d_op = &shmem_dentry_operations;
 	d_add(dentry, NULL);
 	return NULL;
 }

Re: strange behavior creating and deleting files

[Mpourtounis Dimitris]

On Fri, 2004-09-24 at 21:40, Hugh Dickins wrote:
> On Fri, 24 Sep 2004, Mpourtounis Dimitris wrote:
> > 
> > there seems to be a strange behaviour in the way my system creates and
> > deletes files, as long as memory allocation is concerned.
> > 
> > running a simple script that continuously creates and deletes files on
> > tmpfs filesystem, a got the following results:
> > 
> > files created		free memory on system 
> > -------------		---------------------
> > 0			48180
> > +6000			47936
> > +6000			47372
> > +6000			47372
> > +6000			47936
> > +6000			47936
> > +6000			47936
> > +6000			47936		(seems stable)
> > +9000			46976		(what on earth?)
> > +30000			45084
> > +80000			45084		(again stable)
> > +70000			39156		(not again...:( )
> > 
> > and sometime in the morning 25000 MB free RAM, and my system running too
> > slow
> > 
> > I am sure these are a lot a files and under normal conditions, there
> > will never be made and deleted so many.
> > 
> > It is that misbehavior of being stable for a long time and then again
> > allocating memory that concerns me.
> 
> I'll admit that I don't _fully_ understand it either.  Another user
> noticed related behaviour a couple of months ago, below is the patch
> which I hope you find fixes yours too.  An equivalent fix (to fs/libfs.c)
> went into 2.6.9-rc1-bk1, I'd been waiting for it to get more 2.6 exposure
> (lest unwanted side-effects appeared) before sending Marcelo the 2.4 fix.
> If it works well for you, let me know and I'll send it in for 2.4.28...
> 
I seems to work fine. Having created-deleted about 50000 files there is
no leak. Hope it continues that way.
Great job, thanks !

DB

> A tmpfs user reported increasingly slow directory reads when repeatedly
> creating and unlinking in a mkstemp-like way.  The negative dentries
> accumulate alarmingly (until memory pressure finally frees them), and
> are just a hindrance to any in-memory filesystem.  shmem_lookup set
> d_op to arrange for negative dentries to be deleted immediately.
> 
> (But I failed to discover how it is that on-disk filesystems seem to
> keep their negative dentries within manageable bounds: this effect was
> gross with tmpfs or ramfs, but no problem at all with extN or reiser.)
> 
> Signed-off-by: Hugh Dickins <hugh@veritas.com>
> 
> --- 2.4.26/mm/shmem.c	2003-11-28 18:26:21.000000000 +0000
> +++ linux/mm/shmem.c	2004-07-30 12:48:31.324978800 +0100
> @@ -1161,13 +1161,27 @@ static int shmem_statfs(struct super_blo
>  }
>  
>  /*
> + * Retaining negative dentries for an in-memory filesystem just wastes
> + * memory and lookup time: arrange for them to be deleted immediately.
> + */
> +static int shmem_delete_dentry(struct dentry *dentry)
> +{
> +	return 1;
> +}
> +
> +/*
>   * Lookup the data. This is trivial - if the dentry didn't already
> - * exist, we know it is negative.
> + * exist, we know it is negative.  Set d_op to delete negative dentries.
>   */
>  static struct dentry *shmem_lookup(struct inode *dir, struct dentry *dentry)
>  {
> +	static struct dentry_operations shmem_dentry_operations = {
> +		.d_delete = shmem_delete_dentry,
> +	};
> +
>  	if (dentry->d_name.len > NAME_MAX)
>  		return ERR_PTR(-ENAMETOOLONG);
> +	dentry->d_op = &shmem_dentry_operations;
>  	d_add(dentry, NULL);
>  	return NULL;
>  }