crushmap errors

crushmap errors

[Martin Mailand]

Hi,
I used in ceph v0.38 the host and rack feature in the conf during an 
mkcephfs. Now I have to problems with the crushmap

1. I cannot compile a ceph genearated crushmap.
crushtool -c file.txt -o file
file.txt:4 error: parse error at '.0'

# begin crush map

# devices
device 0 osd.0


2. Why are 2 racks are not enough for 2 failure domains?
 From the commit:
If there are >2 racks, separate across racks.

and in the src/osd/OSDMap.cc

    if (racks.size() > 3) {
       // spread replicas across hosts
       crush_rule_set_step(rule, 1, CRUSH_RULE_CHOOSE_LEAF_FIRSTN, 
CRUSH_CHOOSE_N, 2);

shouldn't that be

    if (racks.size() > 1) {
       // spread replicas across racks
       crush_rule_set_step(rule, 1, CRUSH_RULE_CHOOSE_LEAF_FIRSTN, 
CRUSH_CHOOSE_N, 2);

Best Regards,
  martin

Re: crushmap errors

[Sage Weil]

On Fri, 11 Nov 2011, Martin Mailand wrote:
> Hi,
> I used in ceph v0.38 the host and rack feature in the conf during an mkcephfs.
> Now I have to problems with the crushmap
> 
> 1. I cannot compile a ceph genearated crushmap.
> crushtool -c file.txt -o file
> file.txt:4 error: parse error at '.0'

Whoops, will push a patch to stable shortly.  The grammer wasn't 
recognizing '.' as a legal character.

> 2. Why are 2 racks are not enough for 2 failure domains?
> From the commit:
> If there are >2 racks, separate across racks.

Well, technically they are.  My worry is that it's more likely that racks 
will have significantly vary capacity (i.e. crush weight) due to, say, 1 
full rack and a second 1/2 rack.  If the policy forces replicas be placed 
across racks things won't balance well.

I suppose there should be an argument like --min-racks that controls that 
threshold?

sage

Re: crushmap errors

[Martin Mailand]

Hi Sage,
1. The crushtool grammer fix is working for me. Thanks.

2. I think if an admin puts the extra rack info into the ceph.conf file, 
than it should do what expected. I understand your worries but on the 
other end ceph is not an end user tool, and people should know what they 
do and balance there racks evenly.
Just my two cents.

-martin


Am 11.11.2011 23:51, schrieb Sage Weil:
> On Fri, 11 Nov 2011, Martin Mailand wrote:
>> Hi,
>> I used in ceph v0.38 the host and rack feature in the conf during an mkcephfs.
>> Now I have to problems with the crushmap
>>
>> 1. I cannot compile a ceph genearated crushmap.
>> crushtool -c file.txt -o file
>> file.txt:4 error: parse error at '.0'
>
> Whoops, will push a patch to stable shortly.  The grammer wasn't
> recognizing '.' as a legal character.
>
>> 2. Why are 2 racks are not enough for 2 failure domains?
>>  From the commit:
>> If there are>2 racks, separate across racks.
>
> Well, technically they are.  My worry is that it's more likely that racks
> will have significantly vary capacity (i.e. crush weight) due to, say, 1
> full rack and a second 1/2 rack.  If the policy forces replicas be placed
> across racks things won't balance well.
>
> I suppose there should be an argument like --min-racks that controls that
> threshold?
>
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: crushmap errors

[Kelly Kane]

On Fri, Nov 11, 2011 at 14:51, Sage Weil <sage@newdream.net> wrote:
>> 2. Why are 2 racks are not enough for 2 failure domains?
>> From the commit:
>> If there are >2 racks, separate across racks.
>
> Well, technically they are.  My worry is that it's more likely that racks
> will have significantly vary capacity (i.e. crush weight) due to, say, 1
> full rack and a second 1/2 rack.  If the policy forces replicas be placed
> across racks things won't balance well.
>
> I suppose there should be an argument like --min-racks that controls that
> threshold?

In theory the operator can shoot themselves in the foot if they so
please. It seems like a Ceph management console could warn about
"imbalanced crush weight" across racks. This would also allow the
cluster operator to check on their balance over time assuming hardware
gets replaced over time. This could introduce larger rotational hdds,
or smaller ssds, across the cluster.

Kelly
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html