Rack Awareness

Rack Awareness

[Gandalf Corvotempesta]

Hi all,
is ceph able to distribute datas across multiple racks like MooseFS
does with Rack Awareness features?

For example, let's imagine a ceph cluster made with multiple OSDs
distributed in multiple server on multiple racks.
Is possible to tell ceph to distribute replicas across multiple racks
avoiding to have 3 copies of the same chunk on a single rack?

For example:

rack1.server1.osd1
rack2.server5.osd9
rack3.server9.osd11

Re: Rack Awareness

[Wido den Hollander]

Hi,

On 01/15/2013 11:17 AM, Gandalf Corvotempesta wrote:
> Hi all,
> is ceph able to distribute datas across multiple racks like MooseFS
> does with Rack Awareness features?
>
> For example, let's imagine a ceph cluster made with multiple OSDs
> distributed in multiple server on multiple racks.
> Is possible to tell ceph to distribute replicas across multiple racks
> avoiding to have 3 copies of the same chunk on a single rack?
>
> For example:
>
> rack1.server1.osd1
> rack2.server5.osd9
> rack3.server9.osd11

Yes, no problem at all! That's where the crushmap is for. This way you 
can tell Ceph exactly how to distribute your data.

You might want to take a look at these docs: 
http://ceph.com/docs/master/rados/operations/crush-map/

Wido

> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Rack Awareness

[Gandalf Corvotempesta]

2013/1/15 Wido den Hollander <wido@widodh.nl>:
> Yes, no problem at all! That's where the crushmap is for. This way you can
> tell Ceph exactly how to distribute your data.

Cool.
If I understood properly, I have to configure ceph with OSD in a
standard way and then
map these osd into a custom crushmap like this:

host node1 {
        id -1
        alg straw
        hash 0
        item osd.0 weight 1.00
        item osd.1 weight 1.00
}

host node2 {
        id -2
        alg straw
        hash 0
        item osd.2 weight 1.00
        item osd.3 weight 1.00
}

rack rack1 {
        id -3
        alg straw
        hash 0
        item node1 weight 2.00
}

rack rack2 {
        id -4
        alg straw
        hash 0
        item node2 weight 2.00
}

This one should allow me to have 4 OSDs, 2 servers and 2 racks. Each
rack with one server with 2 OSDs
Data will be automatically striped across both rack, right?

Re: Rack Awareness

[Wido den Hollander]

On 01/15/2013 11:34 AM, Gandalf Corvotempesta wrote:
> 2013/1/15 Wido den Hollander <wido@widodh.nl>:
>> Yes, no problem at all! That's where the crushmap is for. This way you can
>> tell Ceph exactly how to distribute your data.
>
> Cool.
> If I understood properly, I have to configure ceph with OSD in a
> standard way and then
> map these osd into a custom crushmap like this:
>
> host node1 {
>          id -1
>          alg straw
>          hash 0
>          item osd.0 weight 1.00
>          item osd.1 weight 1.00
> }
>
> host node2 {
>          id -2
>          alg straw
>          hash 0
>          item osd.2 weight 1.00
>          item osd.3 weight 1.00
> }
>
> rack rack1 {
>          id -3
>          alg straw
>          hash 0
>          item node1 weight 2.00
> }
>
> rack rack2 {
>          id -4
>          alg straw
>          hash 0
>          item node2 weight 2.00
> }
>
> This one should allow me to have 4 OSDs, 2 servers and 2 racks. Each
> rack with one server with 2 OSDs
> Data will be automatically striped across both rack, right?
>

Almost! A couple of things:

You don't have to sum the weight of a node in the rack, crush will sum 
the nodes automatically. If all nodes are equal there is no need to do so.


You need to add this as well:

root default {
         id -1
         alg straw
         hash 0
         item rack1
         item rack2
}

rule data {
         ruleset 0
         type replicated
         min_size 1
         max_size 10
         step take default
         step chooseleaf firstn 0 type rack
         step emit
}

You should know that you can never set replication higher then 2 with 
this setting since it always tries to pick a rack and you only have two.

Wido

Re: Rack Awareness

[Gandalf Corvotempesta]

2013/1/15 Wido den Hollander <wido@widodh.nl>:
> You should know that you can never set replication higher then 2 with this
> setting since it always tries to pick a rack and you only have two.

Good to know, but is not our case, we have 24 racks :)
That was just an example.