File limit inside a single directory

File limit inside a single directory

[suntrop]

Hello, i’m using a CMS which puts many files inside a single directory. I’m wondering what the maximum number of files is which can reside inside said directory. Is there a number which should not be exceeded on account of performance reasons e.g. Is there a difference between ext3 and ext4?

Thanks!
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: File limit inside a single directory

[Eric Sandeen]

On 2/12/15 1:16 PM, suntrop@web.de wrote:
> Hello, i’m using a CMS which puts many files inside a single
> directory. I’m wondering what the maximum number of files is which
> can reside inside said directory. Is there a number which should not
> be exceeded on account of performance reasons e.g. Is there a
> difference between ext3 and ext4?
> 
> Thanks!

for htree directories, you will be limited by the max size of the tree, which currently is of limited depth.  That limit then depends on block size and filename lengths.

for 4k blocks, and 40 char names, you max out at around 20 million directory entries, if my old spreadsheet is still correct.

That limit is the same for ext3 and ext4.

performance-wise, to be honest, I'm not sure.  how many inodes are you talking about?

-Eric

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: File limit inside a single directory

[suntrop]

Cool, that sounds good.

Many thanks!

 
 
> Thanks for your reply. I’m using CMS ProcessWire which would create
> about 500.000 folders for 500 users and 1.000.000 folders for 1000
> users (only in my case and with my setup). All inside one directory.
> No Subfolders. The max filename length is 8 chars. I don’t know the
> actual block size my provider chose but assuming 2k or 4k blocks what
> limit would that sum to?

4k is the default...

but w/ 8 chars and even 2k blocks, you can still get around 8 million
entries.

But again: I'm not sure where that will fall off for you performance-wise.

-Eric
 
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: File limit inside a single directory

[Andreas Dilger]

We started to have problems in our testing around 15M entries in one directory (depends on hashing, add/remove cycles), but 10-12M entries is probably safe unless the filenames are long (e.g. above 32 chars). 

Cheers, Andreas

> On Feb 12, 2015, at 12:35, Eric Sandeen <sandeen@redhat.com> wrote:
> 
>> On 2/12/15 1:16 PM, suntrop@web.de wrote:
>> Hello, i’m using a CMS which puts many files inside a single
>> directory. I’m wondering what the maximum number of files is which
>> can reside inside said directory. Is there a number which should not
>> be exceeded on account of performance reasons e.g. Is there a
>> difference between ext3 and ext4?
>> 
>> Thanks!
> 
> for htree directories, you will be limited by the max size of the tree, which currently is of limited depth.  That limit then depends on block size and filename lengths.
> 
> for 4k blocks, and 40 char names, you max out at around 20 million directory entries, if my old spreadsheet is still correct.
> 
> That limit is the same for ext3 and ext4.
> 
> performance-wise, to be honest, I'm not sure.  how many inodes are you talking about?
> 
> -Eric
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Re: File limit inside a single directory

[suntrop]

Thanks guys. I was afraid of having a couple 100K (at most). The server support team told me not to have more than 10 to 20K. There seems to be a misconception (for me and people from the CMS) about the 32K subdirectory limit, but this ins't for files/folders within a single directory but rather nested directories like /1/2/3.../32000

Again, many thanks!
 

Gesendet: Freitag, 13. Februar 2015 um 01:55 Uhr
Von: "Andreas Dilger" <adilger@dilger.ca>
An: "Eric Sandeen" <sandeen@redhat.com>
Cc: "suntrop@web.de" <suntrop@web.de>, "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>
Betreff: Re: File limit inside a single directory
We started to have problems in our testing around 15M entries in one directory (depends on hashing, add/remove cycles), but 10-12M entries is probably safe unless the filenames are long (e.g. above 32 chars).

Cheers, Andreas

> On Feb 12, 2015, at 12:35, Eric Sandeen <sandeen@redhat.com> wrote:
>
>> On 2/12/15 1:16 PM, suntrop@web.de wrote:
>> Hello, i’m using a CMS which puts many files inside a single
>> directory. I’m wondering what the maximum number of files is which
>> can reside inside said directory. Is there a number which should not
>> be exceeded on account of performance reasons e.g. Is there a
>> difference between ext3 and ext4?
>>
>> Thanks!
>
> for htree directories, you will be limited by the max size of the tree, which currently is of limited depth. That limit then depends on block size and filename lengths.
>
> for 4k blocks, and 40 char names, you max out at around 20 million directory entries, if my old spreadsheet is still correct.
>
> That limit is the same for ext3 and ext4.
>
> performance-wise, to be honest, I'm not sure. how many inodes are you talking about?
>
> -Eric
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: File limit inside a single directory

[Eric Sandeen]

On 2/13/15 10:49 AM, suntrop@web.de wrote:
> Thanks guys. I was afraid of having a couple 100K (at most). The
> server support team told me not to have more than 10 to 20K. There
> seems to be a misconception (for me and people from the CMS) about
> the 32K subdirectory limit, but this ins't for files/folders within a
> single directory but rather nested directories like /1/2/3.../32000

You've confused things a bit here, FWIW.

The 32k (well, 32000 because, sure) limit on ext3 is max link count;
each subdirectory increases the link count on its parent, but only
its parent.  It's not about deep nesting, or about files in a dir.
It's only about subdirs in a parent dir.

# mkdir dir
# stat dir | grep Links
Device: fd06h/64774d    Inode: 2490391     Links: 2	// . and .. entries
# mkdir dir/subdir1 dir/subdir2 dir/subdir3
# stat dir | grep Links
Device: fd06h/64774d    Inode: 2490391     Links: 5
# mkdir dir/subdir1/subsubdir1 dir/subdir1/subsubdir2 dir/subdir1/subsubdir3
# stat dir | grep Links
Device: fd06h/64774d    Inode: 2490391     Links: 5

ext4 bumped that max to 64000, and just stops counting if that number
gets exceeded...

-Eric

Re: Re: File limit inside a single directory

[suntrop]

Sorry for confusion, but now I am confused :-) My setup will be like
/files/
/files/1/
/files/2/
/files/3/
(...)
/files/24244/
(...)
/files/113524/

Thus all folders reside within the /files/ directory (can't change that). 
What happens when the max link count reach 32,000 and I create more folders? Will this result in any errors or problems? I’m using ext3. Is the only consequence that the links aren’t counting up any more, like ext4?
 
 

Gesendet: Freitag, 13. Februar 2015 um 18:28 Uhr
Von: "Eric Sandeen" <sandeen@redhat.com>
An: suntrop@web.de, "Andreas Dilger" <adilger@dilger.ca>
Cc: "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>
Betreff: Re: File limit inside a single directory
On 2/13/15 10:49 AM, suntrop@web.de wrote:
> Thanks guys. I was afraid of having a couple 100K (at most). The
> server support team told me not to have more than 10 to 20K. There
> seems to be a misconception (for me and people from the CMS) about
> the 32K subdirectory limit, but this ins't for files/folders within a
> single directory but rather nested directories like /1/2/3.../32000

You've confused things a bit here, FWIW.

The 32k (well, 32000 because, sure) limit on ext3 is max link count;
each subdirectory increases the link count on its parent, but only
its parent. It's not about deep nesting, or about files in a dir.
It's only about subdirs in a parent dir.

# mkdir dir
# stat dir | grep Links
Device: fd06h/64774d Inode: 2490391 Links: 2 // . and .. entries
# mkdir dir/subdir1 dir/subdir2 dir/subdir3
# stat dir | grep Links
Device: fd06h/64774d Inode: 2490391 Links: 5
# mkdir dir/subdir1/subsubdir1 dir/subdir1/subsubdir2 dir/subdir1/subsubdir3
# stat dir | grep Links
Device: fd06h/64774d Inode: 2490391 Links: 5

ext4 bumped that max to 64000, and just stops counting if that number
gets exceeded...

-Eric
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: File limit inside a single directory

[Andreas Dilger]

If you are using ext3 then you will get an EMLINK error at 32000 subdirs. 
If you are using ext4 with the "dir_nlink" feature enabled then you
can create several million subdirs if you want.

As for performance, if you are not regularly creating and deleting the
directory I don't think you would even notice the performance difference
of a large directory.  The create and unlink rate get somewhat slower at
100k files (from 25k/sec to 10k/sec) and 1M entries or so (to 5k/sec).
For lookups the whole directory will easily fit into cache (at about 40MB
for 1M entries) so the lookup rate is not affected much by size. 

Cheers, Andreas

> On Feb 14, 2015, at 03:17, suntrop@web.de wrote:
> 
> Sorry for confusion, but now I am confused :-) My setup will be like
> /files/
> /files/1/
> /files/2/
> /files/3/
> (...)
> /files/24244/
> (...)
> /files/113524/
> 
> Thus all folders reside within the /files/ directory (can't change that). 
> What happens when the max link count reach 32,000 and I create more folders? Will this result in any errors or problems? I’m using ext3. Is the only consequence that the links aren’t counting up any more, like ext4?
>  
>  
> 
> Gesendet: Freitag, 13. Februar 2015 um 18:28 Uhr
> Von: "Eric Sandeen" <sandeen@redhat.com>
> An: suntrop@web.de, "Andreas Dilger" <adilger@dilger.ca>
> Cc: "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>
> Betreff: Re: File limit inside a single directory
>> On 2/13/15 10:49 AM, suntrop@web.de wrote:
>> Thanks guys. I was afraid of having a couple 100K (at most). The
>> server support team told me not to have more than 10 to 20K. There
>> seems to be a misconception (for me and people from the CMS) about
>> the 32K subdirectory limit, but this ins't for files/folders within a
>> single directory but rather nested directories like /1/2/3.../32000
> 
> You've confused things a bit here, FWIW.
> 
> The 32k (well, 32000 because, sure) limit on ext3 is max link count;
> each subdirectory increases the link count on its parent, but only
> its parent. It's not about deep nesting, or about files in a dir.
> It's only about subdirs in a parent dir.
> 
> # mkdir dir
> # stat dir | grep Links
> Device: fd06h/64774d Inode: 2490391 Links: 2 // . and .. entries
> # mkdir dir/subdir1 dir/subdir2 dir/subdir3
> # stat dir | grep Links
> Device: fd06h/64774d Inode: 2490391 Links: 5
> # mkdir dir/subdir1/subsubdir1 dir/subdir1/subsubdir2 dir/subdir1/subsubdir3
> # stat dir | grep Links
> Device: fd06h/64774d Inode: 2490391 Links: 5
> 
> ext4 bumped that max to 64000, and just stops counting if that number
> gets exceeded...
> 
> -Eric
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: File limit inside a single directory

[Theodore Ts'o]

On Fri, Feb 13, 2015 at 05:49:22PM +0100, suntrop@web.de wrote:
> Thanks guys. I was afraid of having a couple 100K (at most). The
> server support team told me not to have more than 10 to 20K. There
> seems to be a misconception (for me and people from the CMS) about
> the 32K subdirectory limit, but this ins't for files/folders within
> a single directory but rather nested directories like
> /1/2/3.../32000

In general the limitation you will hit first is one of performance,
rather than a hard limit.  Using nested directories, i.e.,
/home/t/y/t/tytso will generally be a performance win if you can do
it.

Cheers,

					- Ted