repeated failed open()'s results in lots of used memory [Was: [Fwd: memory consumption]]

repeated failed open()'s results in lots of used memory [Was: [Fwd: memory consumption]]

[Brian Ristuccia]

We've been experiencing a problem where an errant process would run in a 
tight loop trying to create files in a directory where it did not have 
access. While this errant process was running, we'd notice all of the 
available memory shift from buffers/cache (or free) to used and stay 
that way while the process was running. vmstat also reports heavy in/out 
traffic on the swap, but swap consumption does not grow past a few dozen 
megabytes. The memory used by the process itself does not grow.

Note that we increase the default values for certain FS parameters:

echo '16384' >/proc/sys/fs/super-max
echo '32768' >/proc/sys/fs/file-max
echo '65535' > /proc/sys/fs/inode-max

We've created the following test program, which excercises the problem 
on both 2.2.19 and 2.4.7. All of the machines have at least 512mb of 
memory. I'd appreciate any feedback about reproducing the problem and 
potential causes/fixes.

Thanks.

-------- Original Message --------
Subject: memory consumption
Date: Wed, 1 Aug 2001 16:33:18 -0400 (EDT)
From: Dave Johnson <djohnson@starentnetworks.com>
Reply-To: djohnson@starentnetworks.com
To: bristucc@sw.starentnetworks.com



$ mkdir testdir
$ chmod a-w testdir
$ ./open_test testdir/test


strace shows:

open("testdir/test.10001051", O_WRONLY|O_CREAT|O_EXCL, 0666) = -1 EACCES 
(Permission denied)
open("testdir/test.10001052", O_WRONLY|O_CREAT|O_EXCL, 0666) = -1 EACCES 
(Permission denied)
open("testdir/test.10001053", O_WRONLY|O_CREAT|O_EXCL, 0666) = -1 EACCES 
(Permission denied)
open("testdir/test.10001054", O_WRONLY|O_CREAT|O_EXCL, 0666) = -1 EACCES 
(Permission denied)


memory starts getting used until the system begins to swap like crazy....

interestingly.. the filename needs to change, and it wont happen for a
while if you start with 0.


Happens with both 2.4.7 and 2.2.19



-----



#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

int main (int argc, char *argv[])
{
   int i;
   int fd;
   char name[100];

   if (argc != 2)
     return 1;

   for (i=10000000; i>=0; i++)
     {

       snprintf(name,99,"%s.%d",argv[1],i);

       fd = open(name, O_WRONLY|O_CREAT|O_EXCL, 0666);
       if (fd >= 0) close(fd);

     }

   return 0;
}

--------

-- 
David Johnson                       Starent Networks, Corp.
978-851-1173                        30 International Place
djohnson@starentnetworks.com        Tewksbury, MA 01876

Re: repeated failed open()'s results in lots of used memory [Was: [Fwd: memory consumption]]

[Andreas Dilger]

You write:
> We've been experiencing a problem where an errant process would run in a 
> tight loop trying to create files in a directory where it did not have 
> access. While this errant process was running, we'd notice all of the 
> available memory shift from buffers/cache (or free) to used and stay 
> that way while the process was running. vmstat also reports heavy in/out 
> traffic on the swap, but swap consumption does not grow past a few dozen 
> megabytes. The memory used by the process itself does not grow.
> 
> Note that we increase the default values for certain FS parameters:
> 
> echo '16384' >/proc/sys/fs/super-max
> echo '32768' >/proc/sys/fs/file-max
> echo '65535' > /proc/sys/fs/inode-max

You are probably creating negative dentries.  Check /proc/slabinfo for
the number of dentries, and it will confirm this.  I'm not sure why
that would cause swapping, but then again I haven't checked the policy
for shrinking the dentry cache recently, and there have been a number
of changes in that area lately.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: repeated failed open()'s results in lots of used memory [Was: [Fwd: memory consumption]]

[Brian Ristuccia]

Andreas Dilger wrote:

> You write:
> 
>>We've been experiencing a problem where an errant process would run in a 
>>tight loop trying to create files in a directory where it did not have 
>>access. While this errant process was running, we'd notice all of the 
>>available memory shift from buffers/cache (or free) to used and stay 
>>that way while the process was running. vmstat also reports heavy in/out 
>>traffic on the swap, but swap consumption does not grow past a few dozen 
>>megabytes. The memory used by the process itself does not grow.
>>
>>Note that we increase the default values for certain FS parameters:
>>
>>echo '16384' >/proc/sys/fs/super-max
>>echo '32768' >/proc/sys/fs/file-max
>>echo '65535' > /proc/sys/fs/inode-max
>>
> 
> You are probably creating negative dentries.  Check /proc/slabinfo for
> the number of dentries, and it will confirm this.  I'm not sure why
> that would cause swapping, but then again I haven't checked the policy
> for shrinking the dentry cache recently, and there have been a number
> of changes in that area lately.
> 

Yow! Right on. On 2.2.19 and 2.4.7, the  line for dentry_cache in 
/proc/slabinfo skyrockets while the test program is running. Also, on 
2.2.19 but not 2.4.7 the line for size-32 climbs steadily at around the 
same pace as dentry_cache when the test program is running. After I stop 
the test program, the number slowly declines as other processes allocate 
memory.

-- 
Brian Ristuccia

Re: repeated failed open()'s results in lots of used memory [Was: [Fwd: memory consumption]]

[Andreas Dilger]

Brian Ristuccia writes:
> Andreas Dilger wrote:
> > You are probably creating negative dentries.  Check /proc/slabinfo for
> > the number of dentries, and it will confirm this.  I'm not sure why
> > that would cause swapping, but then again I haven't checked the policy
> > for shrinking the dentry cache recently, and there have been a number
> > of changes in that area lately.
> 
> Yow! Right on. On 2.2.19 and 2.4.7, the  line for dentry_cache in 
> /proc/slabinfo skyrockets while the test program is running. Also, on 
> 2.2.19 but not 2.4.7 the line for size-32 climbs steadily at around the 
> same pace as dentry_cache when the test program is running. After I stop 
> the test program, the number slowly declines as other processes allocate 
> memory.

So it is identified, but still probably needs to be fixed in some way.
Otherwise, you could potentially have DOS from someone trying to read
or create files, even if they don't have write permission _anywhere_
on the system.

Looking at the code, it appears that if we call shrink_dcache_memory()
from while trying to allocate memoty for a filesystem, it returns
without doing anything, to avoid a deadlock.  Al Viro and/or Marcelo
Tosatti probably know how to fix this.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: repeated failed open()'s results in lots of used memory [Was: [Fwd: memory consumption]]

[Linus Torvalds]

In article <3B686D73.6040602@starentnetworks.com> you write:
>
>Note that we increase the default values for certain FS parameters:
>
>echo '16384' >/proc/sys/fs/super-max
>echo '32768' >/proc/sys/fs/file-max
>echo '65535' > /proc/sys/fs/inode-max

Doesn't matter.

The thing that you end up hitting looks very much like just tons of
negative dentries lying around.

If you are even moderately confident about playing around with the
kernel, I would suggest testing the following approach (if you aren't
comforable with playing with the kernel, I hope somebody else is willing
to try this out, or I could try to cook up a patch to test later this
week)

 - add a new dentry list in addition to the current 'dentry_unused' list
   in linux/fs/dentry.c:

	static LIST_HEAD(dentry_unused_negative);

 - when 'dput()' adds a dentry to the old 'dentry_unused' list it would
   instead check whether the dentry is negative, and if so add it to the
   negative list instead:

	list = &dentry_unused;
	if (!dentry->d_inode)
		list = &dentry_unused_negative;
	list_add(&dentry->d_lru, list);

 - add a new "shrink_negative_dentries()" function that just does
   something like

	struct list_head *tmp;

	spin_lock(&dcache_lock);
	for (;;) {
		struct list_head * tmp = dentry_unused_negative.next;
		if (tmp == &dentry_unused_negative)
			break;
		struct dentry *dentry = list_entry(tmp, struct dentry, d_lru);
		tmp = tmp->next;
		if (atomic_read(&dentry->d_count))
			BUG();
		if (dentry->d_inode)
			BUG();
		prune_one_dentry(dentry);
		/* prune_one_dentry() releases the lock */
		spin_lock(&dcache_lock);
	}
	spin_unlock(&dcache_lock);

 - make all the things that shrink dentries (notably the
   shrink_dcache_memory() function) call the above function first. 

Does that fix the behaviour for you?

		Linus

Re: repeated failed open()'s results in lots of used memory [Was: [Fwd: memory consumption]]

[Alexander Viro]

On Wed, 1 Aug 2001, Linus Torvalds wrote:

>  - make all the things that shrink dentries (notably the
>    shrink_dcache_memory() function) call the above function first. 
> 
> Does that fix the behaviour for you?

That will kill _all_ negative dentries whenever we get any amount of
memory pressure. For stuff a-la $PATH it will get very ugly - currently
we have a lot of negative dentries in /usr/local/bin that prevent tons
of bogus lookups there,

Re: repeated failed open()'s results in lots of used memory [Was: [Fwd: memory consumption]]

[Linus Torvalds]

On Wed, 1 Aug 2001, Alexander Viro wrote:
>
>
> On Wed, 1 Aug 2001, Linus Torvalds wrote:
>
> >  - make all the things that shrink dentries (notably the
> >    shrink_dcache_memory() function) call the above function first.
> >
> > Does that fix the behaviour for you?
>
> That will kill _all_ negative dentries whenever we get any amount of
> memory pressure. For stuff a-la $PATH it will get very ugly - currently
> we have a lot of negative dentries in /usr/local/bin that prevent tons
> of bogus lookups there,

I agree. I'm a big fan of negative dentries myself.

However, I'd like to see what the patch does for the bad case first, and
then we can see whether there are less drastic methods (like only killing
half of the negative dentries or something).

The current behaviour obviously has some problems, and negative dentries
_are_ different from positive ones. Trond (or somebody - maybe it was
Neil) also mentioned that negative dentries tend to hurt some NFS
benchmarks by virtue of filling up memory a bit too much.

Btw, my fix description wasn't perfect - you need to get things like the
unused dentry counts right too, etc. The basic approach stands, though.

		Linus

Re: repeated failed open()'s results in lots of used memory [Was: [Fwd: memory consumption]]

[Linus Torvalds]

In article <200108012253.f71MrxGO007546@webber.adilger.int> you write:
>
>Looking at the code, it appears that if we call shrink_dcache_memory()
>from while trying to allocate memoty for a filesystem, it returns
>without doing anything, to avoid a deadlock.  Al Viro and/or Marcelo
>Tosatti probably know how to fix this.

Note that negative dentries are different, and can be happily thrown
away even from a filesystem context - so the suggested fix with special
handling for negative dentries (another email) should work fine. 

		Linus

Re: repeated failed open()'s results in lots of used memory [Was: [Fwd: memory consumption]]

[Alexander Viro]

On Wed, 1 Aug 2001, Linus Torvalds wrote:

> However, I'd like to see what the patch does for the bad case first, and
> then we can see whether there are less drastic methods (like only killing
> half of the negative dentries or something).

Removing the "second chance" logics for negative dentries might be a good
start...

Re: repeated failed open()'s results in lots of used memory [Was: [Fwd: memory consumption]]

[Rik van Riel]

On Wed, 1 Aug 2001, Alexander Viro wrote:
> On Wed, 1 Aug 2001, Linus Torvalds wrote:
>
> > However, I'd like to see what the patch does for the bad case first, and
> > then we can see whether there are less drastic methods (like only killing
> > half of the negative dentries or something).
>
> Removing the "second chance" logics for negative dentries might
> be a good start...

Both the "second chance" logic and pure fifo are bad.

Something like Daniel Phillips' "use once" logic would
be fine for dentries, possibly even simpler.

Dentries could start their live at the head of the
"active" list so they are the first to be moved to
the "reclaim me" list.

If they get referenced while on the second list,
we move them to the tail of the active list.

As a balancing rule, we could tune the system to
always keep half of the dentries in the "reclaim me"
list.

regards,

Rik
--
Executive summary of a recent Microsoft press release:
   "we are concerned about the GNU General Public License (GPL)"


		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com/