Crush and Monitor questions

Crush and Monitor questions

[Bryant Ng]

Hi,

I'm pretty new to Ceph and am just learning about it.

Where are the CRUSH maps stored in Ceph? In the documentation I see you 
use the 'crushtool' to compile and decompile the crush map.  I 
understand that if a single monitor comes online, it can talk to the 
other existing monitors to get the cluster map but how does it work on 
initial startup?  Or if the entire Ceph clusters goes down b/c of power 
failure or something.

What is the recommended hardware configuration for monitors?  In the 
Hardware Recommendation page it says "A monitor requires approximately 
10GB of storage space per daemon instance."  per daemon instance is 
talking about the monitor daemon, not the osd daemons?

Also further down on that page, it list some hardware examples where it 
mentions a ligher configuration for monitors.  I am assuming that is the 
Dell PE R510 which contains 8 x 2 TB drives.  Why does the monitor need 
so much space if it's "10GB of storage space per daemon instance".

-Bryant

Re: Crush and Monitor questions

[Bryant Ng]

Sorry, I misread the Hardware Configuration section of the Ceph 
documentation.  I thought one of the Dell's was a configuration for the 
monitors but both of the Dell R510 and R515 are OSD configuration.

I had another question on the monitors though.  What kind of load 
(requests/second) can a monitor node handle?  My understanding is that 
it just returns the cluster map to the client requesting it?  The 
documentation mentions 3 to 5 monitors in a ceph cluster but what is the 
request rate expected on each of these monitors?

thanks.
Bryant


Bryant Ng wrote:
> Hi,
>
> I'm pretty new to Ceph and am just learning about it.
>
> Where are the CRUSH maps stored in Ceph? In the documentation I see you
> use the 'crushtool' to compile and decompile the crush map.  I
> understand that if a single monitor comes online, it can talk to the
> other existing monitors to get the cluster map but how does it work on
> initial startup?  Or if the entire Ceph clusters goes down b/c of power
> failure or something.
>
> What is the recommended hardware configuration for monitors?  In the
> Hardware Recommendation page it says "A monitor requires approximately
> 10GB of storage space per daemon instance."  per daemon instance is
> talking about the monitor daemon, not the osd daemons?
>
> Also further down on that page, it list some hardware examples where it
> mentions a ligher configuration for monitors.  I am assuming that is the
> Dell PE R510 which contains 8 x 2 TB drives.  Why does the monitor need
> so much space if it's "10GB of storage space per daemon instance".
>
> -Bryant
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Crush and Monitor questions

[Joao Eduardo Luis]

Hello Bryant,

On 12/11/2012 08:23 PM, Bryant Ng wrote:
> Hi,
>
> I'm pretty new to Ceph and am just learning about it.
>
> Where are the CRUSH maps stored in Ceph? In the documentation I see you
> use the 'crushtool' to compile and decompile the crush map.

The crushmap is kept alongside with the osdmap, and shared by the monitors.

> I understand that if a single monitor comes online, it can talk to the
> other existing monitors to get the cluster map but how does it work on
> initial startup?  Or if the entire Ceph clusters goes down b/c of power
> failure or something.

If you add a new monitor to an existing cluster, it will synchronize 
with the existing monitors and will obtain all their infos, including 
the crushmap. Updates to the crushmap will also be shared among the 
monitors in the quorum.

If you are starting up fresh, you will have to either add your custom 
crushmap to the monitors (using the ceph tool), or stick with the 
default crushmap (which only defines something along the lines of a 
'default' root, a 'defaultrack' rack and a 'localhost' host).

If the entire cluster goes down... well, if the monitors are not up they 
won't be able to share the crushmap. When they are brought back up, then 
they will pick up where they left. But I'm not sure if I understand what 
your question is.

   -Joao

Re: Crush and Monitor questions

[Bryant Ng]

I guess my question was where is the crushmap (and osdmap) persisted on 
the monitor node?

If the entire cluster goes down, I assume the monitor is reading the 
crushmap from some persistent file stored on disk or a db?  Is that why 
the minimum recommended storage for monitors is 10GB?  Is the crushmap 
and osdmap stored in those 10GB?

-Bryant

Joao Eduardo Luis wrote:
> Hello Bryant,
>
> On 12/11/2012 08:23 PM, Bryant Ng wrote:
>> Hi,
>>
>> I'm pretty new to Ceph and am just learning about it.
>>
>> Where are the CRUSH maps stored in Ceph? In the documentation I see you
>> use the 'crushtool' to compile and decompile the crush map.
>
> The crushmap is kept alongside with the osdmap, and shared by the monitors.
>
>> I understand that if a single monitor comes online, it can talk to the
>> other existing monitors to get the cluster map but how does it work on
>> initial startup?  Or if the entire Ceph clusters goes down b/c of power
>> failure or something.
>
> If you add a new monitor to an existing cluster, it will synchronize
> with the existing monitors and will obtain all their infos, including
> the crushmap. Updates to the crushmap will also be shared among the
> monitors in the quorum.
>
> If you are starting up fresh, you will have to either add your custom
> crushmap to the monitors (using the ceph tool), or stick with the
> default crushmap (which only defines something along the lines of a
> 'default' root, a 'defaultrack' rack and a 'localhost' host).
>
> If the entire cluster goes down... well, if the monitors are not up they
> won't be able to share the crushmap. When they are brought back up, then
> they will pick up where they left. But I'm not sure if I understand what
> your question is.
>
>    -Joao
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Crush and Monitor questions

[Joao Eduardo Luis]

On 12/12/2012 07:02 PM, Bryant Ng wrote:
> I guess my question was where is the crushmap (and osdmap) persisted on
> the monitor node?
>
> If the entire cluster goes down, I assume the monitor is reading the
> crushmap from some persistent file stored on disk or a db?  Is that why
> the minimum recommended storage for monitors is 10GB?  Is the crushmap
> and osdmap stored in those 10GB?
>
> -Bryant
>

The monitor maintains a 'store'. This is why you have to 'ceph-mon 
--mkfs' before you can run the monitor.

The monitor store needs to room to grow, given that the monitor will 
store pretty much every update to the osdmap, monmap, crushmap, keyring,...

Some of this info will also be pruned regularly though, but it's advised 
to keep enough space around.

Hope this clarifies things.

   -Joao

> Joao Eduardo Luis wrote:
>> Hello Bryant,
>>
>> On 12/11/2012 08:23 PM, Bryant Ng wrote:
>>> Hi,
>>>
>>> I'm pretty new to Ceph and am just learning about it.
>>>
>>> Where are the CRUSH maps stored in Ceph? In the documentation I see you
>>> use the 'crushtool' to compile and decompile the crush map.
>>
>> The crushmap is kept alongside with the osdmap, and shared by the
>> monitors.
>>
>>> I understand that if a single monitor comes online, it can talk to the
>>> other existing monitors to get the cluster map but how does it work on
>>> initial startup?  Or if the entire Ceph clusters goes down b/c of power
>>> failure or something.
>>
>> If you add a new monitor to an existing cluster, it will synchronize
>> with the existing monitors and will obtain all their infos, including
>> the crushmap. Updates to the crushmap will also be shared among the
>> monitors in the quorum.
>>
>> If you are starting up fresh, you will have to either add your custom
>> crushmap to the monitors (using the ceph tool), or stick with the
>> default crushmap (which only defines something along the lines of a
>> 'default' root, a 'defaultrack' rack and a 'localhost' host).
>>
>> If the entire cluster goes down... well, if the monitors are not up they
>> won't be able to share the crushmap. When they are brought back up, then
>> they will pick up where they left. But I'm not sure if I understand what
>> your question is.
>>
>>    -Joao
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Crush and Monitor questions

[Bryant Ng]

Thanks Joao.  Makes more sense now.

What are your thoughts on my other question about expected load a 
monitor can handle?

My understanding is that it just returns the cluster map to the client 
requesting it?  The documentation mentions 3 to 5 monitors in a ceph 
cluster but what is the request rate expected on each of these monitors?

If we are expecting a request rate of about 40 requests/second to the 
ceph cluster from clients, how many monitors would be needed to handle that?

-Bryant

Joao Eduardo Luis wrote:
> On 12/12/2012 07:02 PM, Bryant Ng wrote:
>> I guess my question was where is the crushmap (and osdmap) persisted on
>> the monitor node?
>>
>> If the entire cluster goes down, I assume the monitor is reading the
>> crushmap from some persistent file stored on disk or a db?  Is that why
>> the minimum recommended storage for monitors is 10GB?  Is the crushmap
>> and osdmap stored in those 10GB?
>>
>> -Bryant
>>
>
> The monitor maintains a 'store'. This is why you have to 'ceph-mon
> --mkfs' before you can run the monitor.
>
> The monitor store needs to room to grow, given that the monitor will
> store pretty much every update to the osdmap, monmap, crushmap, keyring,...
>
> Some of this info will also be pruned regularly though, but it's advised
> to keep enough space around.
>
> Hope this clarifies things.
>
>    -Joao
>
>> Joao Eduardo Luis wrote:
>>> Hello Bryant,
>>>
>>> On 12/11/2012 08:23 PM, Bryant Ng wrote:
>>>> Hi,
>>>>
>>>> I'm pretty new to Ceph and am just learning about it.
>>>>
>>>> Where are the CRUSH maps stored in Ceph? In the documentation I see you
>>>> use the 'crushtool' to compile and decompile the crush map.
>>>
>>> The crushmap is kept alongside with the osdmap, and shared by the
>>> monitors.
>>>
>>>> I understand that if a single monitor comes online, it can talk to the
>>>> other existing monitors to get the cluster map but how does it work on
>>>> initial startup?  Or if the entire Ceph clusters goes down b/c of power
>>>> failure or something.
>>>
>>> If you add a new monitor to an existing cluster, it will synchronize
>>> with the existing monitors and will obtain all their infos, including
>>> the crushmap. Updates to the crushmap will also be shared among the
>>> monitors in the quorum.
>>>
>>> If you are starting up fresh, you will have to either add your custom
>>> crushmap to the monitors (using the ceph tool), or stick with the
>>> default crushmap (which only defines something along the lines of a
>>> 'default' root, a 'defaultrack' rack and a 'localhost' host).
>>>
>>> If the entire cluster goes down... well, if the monitors are not up they
>>> won't be able to share the crushmap. When they are brought back up, then
>>> they will pick up where they left. But I'm not sure if I understand what
>>> your question is.
>>>
>>>    -Joao
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>