Data distribution

Data distribution

[Martin Wilderoth]

Hello

I have a ceph cluster of 6 osd 146gb each. I have copied a lot of data
filling to 87%. Between the osd's the data is not evenly distributed

host1
/dev/sdb              137G  119G   15G  90% /data/osd0
/dev/sdc              137G  126G  7.4G  95% /data/osd1

host2
/dev/sdc              137G  114G   21G  85% /data/osd2
/dev/sdd              137G  130G  3.6G  98% /data/osd3

host3
/dev/sdb              137G  107G   27G  81% /data/osd4
/dev/sdc              137G   98G   36G  74% /data/osd5

During the copy i got I/O error, but after restarting the cluster it seems fine.

By some reason osd3 seems to have much more data than osd5. Is there a way of geting the data distributed better ?.

Re: Data distribution

[Josh Durgin]

On 06/25/2011 08:48 PM, Martin Wilderoth wrote:
> Hello
>
> I have a ceph cluster of 6 osd 146gb each. I have copied a lot of data
> filling to 87%. Between the osd's the data is not evenly distributed
>
> host1
> /dev/sdb              137G  119G   15G  90% /data/osd0
> /dev/sdc              137G  126G  7.4G  95% /data/osd1
>
> host2
> /dev/sdc              137G  114G   21G  85% /data/osd2
> /dev/sdd              137G  130G  3.6G  98% /data/osd3
>
> host3
> /dev/sdb              137G  107G   27G  81% /data/osd4
> /dev/sdc              137G   98G   36G  74% /data/osd5
>
> During the copy i got I/O error, but after restarting the cluster it seems fine.
>
> By some reason osd3 seems to have much more data than osd5. Is there a way of geting the data distributed better ?.

Hi Martin,

Since the distribution is pseudo-random, you'll get some variance from 
an even split. You can reweight the osds manually with:

ceph osd reweight osd3 new_weight

or use the more automatic:

ceph osd reweight-by-utilization 110

This reduces the weight of all osds that have a utilization that is more 
than 110% of the average utilization.

Josh

Re: Data distribution

[Gregory Farnum]

On Jun 27, 2011, at 12:19 PM, Josh Durgin wrote:

> On 06/25/2011 08:48 PM, Martin Wilderoth wrote:
>> Hello
>> 
>> I have a ceph cluster of 6 osd 146gb each. I have copied a lot of data
>> filling to 87%. Between the osd's the data is not evenly distributed
>> 
>> host1
>> /dev/sdb              137G  119G   15G  90% /data/osd0
>> /dev/sdc              137G  126G  7.4G  95% /data/osd1
>> 
>> host2
>> /dev/sdc              137G  114G   21G  85% /data/osd2
>> /dev/sdd              137G  130G  3.6G  98% /data/osd3
>> 
>> host3
>> /dev/sdb              137G  107G   27G  81% /data/osd4
>> /dev/sdc              137G   98G   36G  74% /data/osd5
>> 
>> During the copy i got I/O error, but after restarting the cluster it seems fine.
>> 
>> By some reason osd3 seems to have much more data than osd5. Is there a way of geting the data distributed better ?.
> 
> Hi Martin,
> 
> Since the distribution is pseudo-random, you'll get some variance from an even split. You can reweight the osds manually with:
> 
> ceph osd reweight osd3 new_weight
> 
> or use the more automatic:
> 
> ceph osd reweight-by-utilization 110
> 
> This reduces the weight of all osds that have a utilization that is more than 110% of the average utilization.
> 
> Josh

That said if the data is this unevenly distributed something odd is going on. Are you using anything besides the filesystem on this cluster? If not we probably need to figure out if there's a problem with the hashing.
-Greg