Creating large numbers of files

Creating large numbers of files

[Dan Labute]

Hi all,

I'm a developer trying to create large numbers of *empty* files 
(~100000).  What is the fastest way to perform such an operation other 
than a simple open().  Will using multiple threads to perform concurrent 
operations help significantly, or am I just awell-off using a single thread?

Are there any extensions that could be used?  What about writing my own 
filesystem plugin?

Similarly, I'd like to create and destroy large numbers of hard & soft 
links.  What are my options?

(Apologies if these questions have been asked in other thread.)

Thanks,

Dan

Re: Creating large numbers of files

[Andreas Dilger]

[-- Attachment #1: Type: text/plain, Size: 645 bytes --]

On Jan 10, 2005  17:34 -0500, Dan Labute wrote:
> I'm a developer trying to create large numbers of *empty* files 
> (~100000).  What is the fastest way to perform such an operation other 
> than a simple open().  Will using multiple threads to perform concurrent 
> operations help significantly, or am I just awell-off using a single thread?

If you call 'mknod("/path/to/file", S_IFREG | 0666, 0)' from a program
it avoids the overhead of doing both an open and a close.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://members.shaw.ca/adilger/             http://members.shaw.ca/golinux/


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Creating large numbers of files

[Dan Labute]

Andreas Dilger wrote:
> On Jan 10, 2005  17:34 -0500, Dan Labute wrote:
> 
>>I'm a developer trying to create large numbers of *empty* files 
>>(~100000).  What is the fastest way to perform such an operation other 
>>than a simple open().  Will using multiple threads to perform concurrent 
>>operations help significantly, or am I just awell-off using a single thread?
> 

> If you call 'mknod("/path/to/file", S_IFREG | 0666, 0)' from a program
> it avoids the overhead of doing both an open and a close.

Thanks Andreas.  I just the change and had minimal results.  Regardless, 
it's the right call to make.

I also tried to use multiple threads (from separate test programs), and 
found that the performance went downhill.  I'm assuming that this would 
indicate that multi-threading the mknod() operations within the same 
executable would yield similar results.  I'm surprised that the combined 
performance was actually worse.

As for link() and unlink() (and mknod), I see no options but to use 
filesystem-specific ioctl() calls.  Does ReiserFS provide any 
batch-operation calls that would yield better performance?
If not, would it be worthwhile to invest in a plug-in?

Any thoughts from the Reiser crew?

Regards,

Dan