[OT] memory issue with GNU find

[OT] memory issue with GNU find

[Philippe Gramoullé]

Hi,

Sorry to be a little off topic here but here is the issue:

For legal reasons we need to list all files on a filer (reiserfs FS so not
_that_ offtopic :o).

Problem is that the filer is about 400Go and number of files according to 
/proc/fs/reiserfs/*/oidmap is about 20 Million files.

From what we've read and seen , find is eating up all memory to the point that
the filer start to swap and then crashes.

Would anyone know how to do that without bring the filer to its knees in terms
of memory ?

thanks,

Philippe

Re: [OT] memory issue with GNU find

[Nikita Danilov]

Philippe Gramoullé writes:
 > Hi,
 > 
 > Sorry to be a little off topic here but here is the issue:
 > 
 > For legal reasons we need to list all files on a filer (reiserfs FS so not
 > _that_ offtopic :o).
 > 
 > Problem is that the filer is about 400Go and number of files according to 
 > /proc/fs/reiserfs/*/oidmap is about 20 Million files.
 > 
 > >From what we've read and seen , find is eating up all memory to the point that
 > the filer start to swap and then crashes.
 > 
 > Would anyone know how to do that without bring the filer to its knees in terms
 > of memory ?

ls -R ?

 > 
 > thanks,
 > 
 > Philippe

Nikita.

Re: [OT] memory issue with GNU find

[Chris Mason]

On Thu, 2002-10-24 at 11:43, Philippe Gramoullé wrote:
> Hi,
> 
> Sorry to be a little off topic here but here is the issue:
> 
> For legal reasons we need to list all files on a filer (reiserfs FS so not
> _that_ offtopic :o).
> 
> Problem is that the filer is about 400Go and number of files according to 
> /proc/fs/reiserfs/*/oidmap is about 20 Million files.
> 
> >From what we've read and seen , find is eating up all memory to the point that
> the filer start to swap and then crashes.
> 
> Would anyone know how to do that without bring the filer to its knees in terms
> of memory ?

Interesting, find should not do that unless you're trying to get it to
sort the results somehow.

Exactly how are you running the find?

-chris

Re: [OT] memory issue with GNU find

[Fabien Combernous]

Do you have test tree ?

Philippe Gramoullé wrote:
> Hi,
> 
> Sorry to be a little off topic here but here is the issue:
> 
> For legal reasons we need to list all files on a filer (reiserfs FS so not
> _that_ offtopic :o).
> 
> Problem is that the filer is about 400Go and number of files according to 
> /proc/fs/reiserfs/*/oidmap is about 20 Million files.
> 
> From what we've read and seen , find is eating up all memory to the point that
> the filer start to swap and then crashes.
> 
> Would anyone know how to do that without bring the filer to its knees in terms
> of memory ?
> 
> thanks,
> 
> Philippe
> 

-- 

Fabien COMBERNOUS - IT Engineer
eProcess - Parc Club du Millénaire Batiment n° 6
1025 rue Henri Becquerel - 34000 Montpellier FRANCE
http://www.eprocess.fr - +33 (0)4 67 13 84 50

Re: [OT] memory issue with GNU find

[Philippe Gramoullé]

On 24 Oct 2002 11:54:45 -0400
Chris Mason <mason@suse.com> wrote:

  |  
  |   Exactly how are you running the find?

Like this:

find /path-to-list -type f ! -user root

Philippe

Re: [OT] memory issue with GNU find

[Philippe Gramoullé]

On Thu, 24 Oct 2002 17:54:47 +0200
Fabien Combernous <fcombernous@eprocess.fr> wrote:

  |  Do you have test tree ?

Yes we have couple ones, which are actual old producation data.
So tests would be pretty accurate, to what we would have in production.

Philippe

Re: [OT] memory issue with GNU find

[Philippe Gramoullé]

Hi,

On Fri, 25 Oct 2002 19:00:33 +0200
Russell Coker <russell@coker.com.au> wrote:

  |  One issue I noticed with find is that the amount of memory required is 
  |   propertional to the length of the maximum path name.  If you have a very deep 
  |   tree of subdirectories then the amount of memory taken becomes significant.  

Are you aware of a find variant that would be memory limited ? :

If memory usage goes up , then at a certain point, commit everything in RAM to a file
, free up memory, and continue like this again and again.

  |   It's easy to create a set of nested directories that crash find (it's a good 
  |   local DOS attack).
  |  
  |   while /bin/true ; do
  |     mkdir foo
  |     cd foo
  |   done

Users don't have shell access but they could do it over FTP indeed. We already found cases of such very
nested directory.

thanks,

Philippe

Re: [OT] memory issue with GNU find

[Russell Coker]

On Fri, 25 Oct 2002 20:23, Philippe Gramoullé wrote:
>   |  One issue I noticed with find is that the amount of memory required is
>   |   propertional to the length of the maximum path name.  If you have a
>   | very deep tree of subdirectories then the amount of memory taken
>   | becomes significant.
>
> Are you aware of a find variant that would be memory limited ? :

No.

> If memory usage goes up , then at a certain point, commit everything in RAM
> to a file , free up memory, and continue like this again and again.

That could be done, then there's always the issue of disk space.  For your 
example of a NAS device it might not be such an issue.  For a situation of a 
small machine and a DOS attack then putting a long chain of directories and a 
large file at the end could make things really difficult for recovery!

Also this still wouldn't make the following work:
find /tmp/junk -exec rm {} \;

>   |   It's easy to create a set of nested directories that crash find (it's
>   | a good local DOS attack).
>   |
>   |   while /bin/true ; do
>   |     mkdir foo
>   |     cd foo
>   |   done
>
> Users don't have shell access but they could do it over FTP indeed. We
> already found cases of such very nested directory.

When I was running my SE Linux play machine (public root password and a 
challenge to the world to hack it) someone did that to it.  Since then I've 
been meaning to write a program to remove a directory tree of arbitary depth 
in O(N) time where N is the depth.

-- 
http://www.coker.com.au/bonnie++/     Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/       Postal SMTP/POP benchmark
http://www.coker.com.au/projects.html Projects I am working on
http://www.coker.com.au/~russell/     My home page