Re: Hashing and directories

[Bill Crawford <billc@netcomuk.co.uk>]

 Before I reply: I apologise for starting this argument, or at least
making it worse, and please let me say again that I really would like
to see improvements in directory searching etc. ... my original point
was simply a half-joking aside to the effect that we should not
encourage people to put thousands of files in a single directory ...

"H. Peter Anvin" wrote:

> >         * userland issues (what, you thought that limits on the
> > command size will go away?)

> Last I checked, the command line size limit wasn't a userland issue, but
> rather a limit of the kernel exec().  This might have changed.

 Actually this is also a serious problem.  We have a ticketing system
at work that stores all its data in files in large directories, and I
discovered recently that I can only pass about a tenth of the current
file names on a command line.  Yes, we have xargs, but ...

 Also (this happens to be Solaris on a 8 x 40MHz Sparc system) there
is a significant delay just to read the directory.

> >         * portability

 Also an issue.  Fortunately we store a lot of data on NetApps, so
the performance of the filesystem as such is less of an issue; but,
that means the size of the data transfer across the network involved 
in reading a directory can become an issue too.

> > The point being: applications and users _do_ know better what structure
> > is there. Kernel can try to second-guess them and be real good at that, but
> > inability to second-guess is the last of the reasons why aforementioned
> > strategies are used.

 All good points ...

> However, there are issues where users and applications don't want to
> structure their namespace for the convenience of the kernel, for good or
> bad reasons.

 But there are other reasons (besides the kernel's and filesystems'
handling of directories and name lookups).  I think you're spot on
about the performance issues and on-disk structures, though.

>         -hpa

-- 
/* Bill Crawford, Unix Systems Developer, ebOne, formerly GTS Netcom */
#include "stddiscl.h"