fast approximate file count

fast approximate file count

[Jeff Breidenbach]

Didn't see this one on the FAQ.

Now that I have millions of small files on my reiserfs partition, I'd
like to know if there is reiserfs specific way to count them very
quickly.

While "find $DIR | wc -l" works and is quite accurate, it can take an
enormously long time. While "df" and dividing by a guesstimate of
the average file size is fast, I wonder if there is a way to do
better. Is there a prgram that can take advantage of reiserfs
specific metadata and quickly compute an approximate file count?
Especially if the directory tree structure is very shallow?

Cheers,
Jeff

Re: fast approximate file count

[Oleg Drokin]

Hello!

On Fri, Sep 20, 2002 at 10:36:23PM -0700, Jeff Breidenbach wrote:
> While "find $DIR | wc -l" works and is quite accurate, it can take an
> enormously long time. While "df" and dividing by a guesstimate of
> the average file size is fast, I wonder if there is a way to do
> better. Is there a prgram that can take advantage of reiserfs
> specific metadata and quickly compute an approximate file count?
> Especially if the directory tree structure is very shallow?

If you never/very seldom delete files,
you can create new file (any size), execute ls -i on it, like this:
 549661 z

This number is OID, oid grows up in case that there were no deletions/gaps
from previous deletions in OID.
OID is total number of objects on FS in this case.

Counting is started from 2 for root dir. objects are files and directories.

Bye,
    Oleg

Re: fast approximate file count

[Oleg Drokin]

Hello!

On Sun, Sep 22, 2002 at 05:43:09AM +0200, Ragnar Kj?rstad wrote:
> > If you never/very seldom delete files,
> > you can create new file (any size), execute ls -i on it, like this:
> >  549661 z
> > This number is OID, oid grows up in case that there were no deletions/gaps
> > from previous deletions in OID.
> > OID is total number of objects on FS in this case.
> Hmm, why doesn't statfs return the correct number of inodes in use? I
> think that's what you would want to use here.

Well, something like that can be done even now, I'll think about it tomorrow.

Bye,
    Oleg

Re: fast approximate file count

[Jeff Breidenbach]

df -i does not work for reiserfs, at least under Debian 3.0.
For example, the following partition is actually more than 10% 
full.

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sdb1            4294967295       0 4294967295    0% /home/archive

From responses so far, my understanding is there may be
a partition-wide file counter possible in the future.

However, it also sounds like there is little to no chance of a
reiserfs-specific fast approximate file counter that will work on a
directory subtree, even if that subtree is very wide and shallow.

-Jeff


On Sun, 2002-09-22 at 19:07, Chris Wedgwood wrote:
> On Fri, Sep 20, 2002 at 10:36:23PM -0700, Jeff Breidenbach wrote:
> 
> > Now that I have millions of small files on my reiserfs partition,
> > I'd like to know if there is reiserfs specific way to count them
> > very quickly.
> 
> Does df -i work?
> 
> 
>   --cw
> 

Re: fast approximate file count

[Chris Mason]

On Sun, 2002-09-22 at 22:30, Jeff Breidenbach wrote:
> df -i does not work for reiserfs, at least under Debian 3.0.
> For example, the following partition is actually more than 10% 
> full.
> 
> Filesystem            Inodes   IUsed   IFree IUse% Mounted on
> /dev/sdb1            4294967295       0 4294967295    0% /home/archive
> 
> >From responses so far, my understanding is there may be
> a partition-wide file counter possible in the future.
> 
> However, it also sounds like there is little to no chance of a
> reiserfs-specific fast approximate file counter that will work on a
> directory subtree, even if that subtree is very wide and shallow.

Correct, that information isn't really stored.  Not very hard to deal
with from an application point of view though

ln -s 8,765,432 num_files

8765432 doesn't have to exist, you can just read the symbolic link.

Note that if each file name has a fixed size, you can deduce the number
of files in the directory from the directory size.

-chris