Files per directory

Files per directory

[Beginner]

Hi,

Seeming as the list is teeming with activity at the moment, and no, 
in the UK we are not experiencing network issues, I thought I'd wade 
in.

I am trying to design a files system that is based around database 
records. Each record within the DB could have up to 9 files (possibly 
more). The best design I can come up with would mean that a directory 
would end up with 9,999 files in. 

Is that going to cause performance issues? The current file system 
ext3. Would anyone suggest a limit I should set for the maximum or 
say if they think 10K files is acceptable?

TIA,
Dp.

Re: Files per directory

[Yuri Csapo]

[-- Attachment #1: Type: text/plain, Size: 1266 bytes --]

Beginner wrote:
> 
> Is that going to cause performance issues? The current file system 
> ext3. Would anyone suggest a limit I should set for the maximum or 
> say if they think 10K files is acceptable?
> 

I'm no expert but the answer is probably: "depends on the application."

As far as I know there's no limit to the number of files in a directory 
currently in ext3. There IS a limit to the number of files (actually 
inodes) in the whole filesystem, which is a completely different thing. 
According to Wikipedia "If V is the volume size in bytes, then the 
default number of inodes is given by V/2^13 (or the number of blocks, 
whichever is less)." There's also a limit to the number of 
sub-directories in a directory, currently 32000.

What I can tell you is I have seen applications that use hundreds of 
thousands of files in one directory.

HTH

-- 
Yuri Csapo
Academic Computing & Networking
Colorado School of Mines
CT-256
Phone:  (303) 273-3503
Fax:      (303) 273-3475
Email:   ycsapo@mines.edu

Please use the following link to open a service request:
http://helpdesk.mines.edu
===========================================
With a PC, I always felt limited
by the software available.
On Unix, I am limited only by my knowledge.
--Peter J. Schoenster

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 2840 bytes --]

Re: Files per directory

[Glynn Clements]

Yuri Csapo wrote:

> > Is that going to cause performance issues? The current file system 
> > ext3. Would anyone suggest a limit I should set for the maximum or 
> > say if they think 10K files is acceptable?
> 
> I'm no expert but the answer is probably: "depends on the application."
> 
> As far as I know there's no limit to the number of files in a directory 
> currently in ext3. There IS a limit to the number of files (actually 
> inodes) in the whole filesystem, which is a completely different thing. 

ext3 also has a limit of 32000 hard links, which means that a
directory can't have more than 31998 subdirectories.

However, the original poster wasn't asking about hard limits, but
efficiency.

If the filesystem wasn't created with the dir_index option, then
having thousands of files in a directory will be a major performance
problem, as any lookups will scan the directory linearly.

Even with the dir_index option, large directories could be an issue. I
think that you would really need to conduct tests to see exactly how
much of an issue.

OTOH, even if you keep the directories small, a database consisting of
many small files will be much slower than e.g. BerkeleyDB or DBM.

-- 
Glynn Clements <glynn@gclements.plus.com>