MDS configuration

MDS configuration

[Fyodor Ustinov]

Hi!

Which configuration would you recommended for cluster with 50-80 million 
files?

WBR,
     Fyodor.

Re: MDS configuration

[djlee064]

based on the empirical measurement, starting with a set of (real)
number of files, focusing mainly on small-files. and found some rough
relationship,
all filesets with same dist. i.e., ~80% <4MB file, mostly small, all 1x. replc.
For MDS RAM:

You need about:
0.6GB RAM to store 0.03million files (fileset vol. 1.2TB)
1.2GB to store  0.065million files (fileset vol. 2.4TB)
1.8GB to store 0.13million files  (fileset vol. 4.8TB)

ratio of no.files per GB fortunately increases, (i.e., 0.07million per
GB for the 0.13million files), hopefully per-GB will support
0.1million files as more files are stored.
then, 18GB = 1.8million files,
180GB = 18million
1800GB= 180million (this is about 6.64PB),

So to support that amount,, we need 100 MDS nodes with 18GB purely for
cmds, incl mem for OS, etc, maybe 20-24GB per node.

Cheers

On Sat, Jun 11, 2011 at 06:47, Fyodor Ustinov <ufm@ufm.su> wrote:
>
> Hi!
>
> Which configuration would you recommended for cluster with 50-80 million files?
>
> WBR,
>    Fyodor.
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: MDS configuration

[djlee064]

ah; previous message was an example of 'during the operation' i.e.,
randread entire sets;
sorry about that,
cmds uses only a tiny MB ~100MB once files stored.
for cosd, during the operation uses 150-200mb per cosd (both during write/read)

still, when considering performance, previous message stands or else
the performance decreases if the minimum amount of RAM isn't there.

cheers

On Sun, Jun 12, 2011 at 20:41, djlee064 <djlee064@gmail.com> wrote:
> based on the empirical measurement, starting with a set of (real)
> number of files, focusing mainly on small-files. and found some rough
> relationship,
> all filesets with same dist. i.e., ~80% <4MB file, mostly small, all 1x. replc.
> For MDS RAM:
>
> You need about:
> 0.6GB RAM to store 0.03million files (fileset vol. 1.2TB)
> 1.2GB to store  0.065million files (fileset vol. 2.4TB)
> 1.8GB to store 0.13million files  (fileset vol. 4.8TB)
>
> ratio of no.files per GB fortunately increases, (i.e., 0.07million per
> GB for the 0.13million files), hopefully per-GB will support
> 0.1million files as more files are stored.
> then, 18GB = 1.8million files,
> 180GB = 18million
> 1800GB= 180million (this is about 6.64PB),
>
> So to support that amount,, we need 100 MDS nodes with 18GB purely for
> cmds, incl mem for OS, etc, maybe 20-24GB per node.
>
> Cheers
>
> On Sat, Jun 11, 2011 at 06:47, Fyodor Ustinov <ufm@ufm.su> wrote:
>>
>> Hi!
>>
>> Which configuration would you recommended for cluster with 50-80 million files?
>>
>> WBR,
>>    Fyodor.
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: MDS configuration

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2400 bytes --]

On Sun, 12 Jun 2011, djlee064 wrote:
> ah; previous message was an example of 'during the operation' i.e.,
> randread entire sets;
> sorry about that,
> cmds uses only a tiny MB ~100MB once files stored.
> for cosd, during the operation uses 150-200mb per cosd (both during write/read)
> 
> still, when considering performance, previous message stands or else
> the performance decreases if the minimum amount of RAM isn't there.

Right.  The ~1k/file ratio Greg mentioned is also without any memory 
optimization work.  There is a huge amount of inode and dentry state 
associated with modifications that need not consume memory all of the 
time.  Fixing it up has been low on the priority list.

sage


> 
> cheers
> 
> On Sun, Jun 12, 2011 at 20:41, djlee064 <djlee064@gmail.com> wrote:
> > based on the empirical measurement, starting with a set of (real)
> > number of files, focusing mainly on small-files. and found some rough
> > relationship,
> > all filesets with same dist. i.e., ~80% <4MB file, mostly small, all 1x. replc.
> > For MDS RAM:
> >
> > You need about:
> > 0.6GB RAM to store 0.03million files (fileset vol. 1.2TB)
> > 1.2GB to store  0.065million files (fileset vol. 2.4TB)
> > 1.8GB to store 0.13million files  (fileset vol. 4.8TB)
> >
> > ratio of no.files per GB fortunately increases, (i.e., 0.07million per
> > GB for the 0.13million files), hopefully per-GB will support
> > 0.1million files as more files are stored.
> > then, 18GB = 1.8million files,
> > 180GB = 18million
> > 1800GB= 180million (this is about 6.64PB),
> >
> > So to support that amount,, we need 100 MDS nodes with 18GB purely for
> > cmds, incl mem for OS, etc, maybe 20-24GB per node.
> >
> > Cheers
> >
> > On Sat, Jun 11, 2011 at 06:47, Fyodor Ustinov <ufm@ufm.su> wrote:
> >>
> >> Hi!
> >>
> >> Which configuration would you recommended for cluster with 50-80 million files?
> >>
> >> WBR,
> >>    Fyodor.
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: MDS configuration

[Gregory Farnum]

This depends very much on how many active files you have at once and how beefy your MDS machines are. By default the MDS will cache 100k dentries, and each dentry takes ~1KB of RAM. If you have sufficient RAM you can increase the size of the cache arbitrarily, though!

There's actually no required relationship between total filesystem size and MDS memory, though, since all the metadata is stored in on-disk objects. We haven't done serious memory profiling in a while, though, so there may be other constraints acting up that we're missing out on here.
-Greg

On Jun 10, 2011, at 11:47 AM, Fyodor Ustinov wrote:

> Hi!
> 
> Which configuration would you recommended for cluster with 50-80 million files?
> 
> WBR,
>    Fyodor.
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html