Re: Very unbalanced storage

[Xiaopong Tran <xiaopong.tran@gmail.com>]

On 09/01/2012 12:39 AM, Gregory Farnum wrote:
> On Fri, Aug 31, 2012 at 9:24 AM, Andrew Thompson <andrewkt@aktzero.com> wrote:
>> On 8/31/2012 12:10 PM, Sage Weil wrote:
>>>
>>> On Fri, 31 Aug 2012, Andrew Thompson wrote:
>>>>
>>>> Have you been reweight-ing osds? I went round and round with my cluster a
>>>> few days ago reloading different crush maps only to find that it
>>>> re-injecting a crush map didn't seem to overwrite reweights. Take a look at
>>>> `ceph osd tree` to see if the reweight column matches the weight column.
>>>
>>> Note that the ideal situation is for reweight to be 1, regardless of what
>>> the crush weight is.  If you find the utilizations are skewed, I would
>>> look for other causes before resorting to reweight-by-utilization; it is
>>> meant to adjust the normal statistical variation you expect from a
>>> (pseudo)random placement, but if the variance is high there is likely
>>> another cause.
>>
>>
>> So if someone(me, guilty) had been messing with reweight, will setting them
>> all to 1 return it to a normal un-reweight-ed state?
>
> Yep!
> If you have OSDs with different sizes you'll want to adjust the CRUSH
> weights, not the reweight values:
> http://ceph.com/docs/master/ops/manage/crush/#adjusting-the-crush-weight

Thanks for the reply. Yes, this was what I did, we had 1TB and 2TB HD,
so using 1TB as the base line, with weight being 1.0, then I'd like that
the 2TB HD store 2x amount of data, so that the disks always have
roughly same relative amount of data.

Originally, every osd has weight of 1.0, and I did:

ceph osd crush reweight osd.30 2.0

and all the 2TB disks.

And that's probably what caused the skew afterward. The crush map
attached in my last message was fetched from the cluster, and

ceph osd tree

does show that the weight of the 2TB disks as 2, but reweight is 1.

Now I'm getting confused by the meaning of crush weight :)

Best,

Xiaopong