repeated failed open()'s results in lots of used memory [Was: [Fwd: memory consumption]]

[Brian Ristuccia <bristuccia@starentnetworks.com>]

We've been experiencing a problem where an errant process would run in a 
tight loop trying to create files in a directory where it did not have 
access. While this errant process was running, we'd notice all of the 
available memory shift from buffers/cache (or free) to used and stay 
that way while the process was running. vmstat also reports heavy in/out 
traffic on the swap, but swap consumption does not grow past a few dozen 
megabytes. The memory used by the process itself does not grow.

Note that we increase the default values for certain FS parameters:

echo '16384' >/proc/sys/fs/super-max
echo '32768' >/proc/sys/fs/file-max
echo '65535' > /proc/sys/fs/inode-max

We've created the following test program, which excercises the problem 
on both 2.2.19 and 2.4.7. All of the machines have at least 512mb of 
memory. I'd appreciate any feedback about reproducing the problem and 
potential causes/fixes.

Thanks.

-------- Original Message --------
Subject: memory consumption
Date: Wed, 1 Aug 2001 16:33:18 -0400 (EDT)
From: Dave Johnson <djohnson@starentnetworks.com>
Reply-To: djohnson@starentnetworks.com
To: bristucc@sw.starentnetworks.com



$ mkdir testdir
$ chmod a-w testdir
$ ./open_test testdir/test


strace shows:

open("testdir/test.10001051", O_WRONLY|O_CREAT|O_EXCL, 0666) = -1 EACCES 
(Permission denied)
open("testdir/test.10001052", O_WRONLY|O_CREAT|O_EXCL, 0666) = -1 EACCES 
(Permission denied)
open("testdir/test.10001053", O_WRONLY|O_CREAT|O_EXCL, 0666) = -1 EACCES 
(Permission denied)
open("testdir/test.10001054", O_WRONLY|O_CREAT|O_EXCL, 0666) = -1 EACCES 
(Permission denied)


memory starts getting used until the system begins to swap like crazy....

interestingly.. the filename needs to change, and it wont happen for a
while if you start with 0.


Happens with both 2.4.7 and 2.2.19



-----



#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

int main (int argc, char *argv[])
{
   int i;
   int fd;
   char name[100];

   if (argc != 2)
     return 1;

   for (i=10000000; i>=0; i++)
     {

       snprintf(name,99,"%s.%d",argv[1],i);

       fd = open(name, O_WRONLY|O_CREAT|O_EXCL, 0666);
       if (fd >= 0) close(fd);

     }

   return 0;
}

--------

-- 
David Johnson                       Starent Networks, Corp.
978-851-1173                        30 International Place
djohnson@starentnetworks.com        Tewksbury, MA 01876