* Data distribution [not found] <344660039.17770.1309060083991.JavaMail.root@mail.linserv.se> @ 2011-06-26 3:48 ` Martin Wilderoth 2011-06-27 19:19 ` Josh Durgin 0 siblings, 1 reply; 3+ messages in thread From: Martin Wilderoth @ 2011-06-26 3:48 UTC (permalink / raw) To: ceph-devel Hello I have a ceph cluster of 6 osd 146gb each. I have copied a lot of data filling to 87%. Between the osd's the data is not evenly distributed host1 /dev/sdb 137G 119G 15G 90% /data/osd0 /dev/sdc 137G 126G 7.4G 95% /data/osd1 host2 /dev/sdc 137G 114G 21G 85% /data/osd2 /dev/sdd 137G 130G 3.6G 98% /data/osd3 host3 /dev/sdb 137G 107G 27G 81% /data/osd4 /dev/sdc 137G 98G 36G 74% /data/osd5 During the copy i got I/O error, but after restarting the cluster it seems fine. By some reason osd3 seems to have much more data than osd5. Is there a way of geting the data distributed better ?. ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Data distribution 2011-06-26 3:48 ` Data distribution Martin Wilderoth @ 2011-06-27 19:19 ` Josh Durgin 2011-06-27 19:23 ` Gregory Farnum 0 siblings, 1 reply; 3+ messages in thread From: Josh Durgin @ 2011-06-27 19:19 UTC (permalink / raw) To: Martin Wilderoth; +Cc: ceph-devel On 06/25/2011 08:48 PM, Martin Wilderoth wrote: > Hello > > I have a ceph cluster of 6 osd 146gb each. I have copied a lot of data > filling to 87%. Between the osd's the data is not evenly distributed > > host1 > /dev/sdb 137G 119G 15G 90% /data/osd0 > /dev/sdc 137G 126G 7.4G 95% /data/osd1 > > host2 > /dev/sdc 137G 114G 21G 85% /data/osd2 > /dev/sdd 137G 130G 3.6G 98% /data/osd3 > > host3 > /dev/sdb 137G 107G 27G 81% /data/osd4 > /dev/sdc 137G 98G 36G 74% /data/osd5 > > During the copy i got I/O error, but after restarting the cluster it seems fine. > > By some reason osd3 seems to have much more data than osd5. Is there a way of geting the data distributed better ?. Hi Martin, Since the distribution is pseudo-random, you'll get some variance from an even split. You can reweight the osds manually with: ceph osd reweight osd3 new_weight or use the more automatic: ceph osd reweight-by-utilization 110 This reduces the weight of all osds that have a utilization that is more than 110% of the average utilization. Josh ^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Data distribution 2011-06-27 19:19 ` Josh Durgin @ 2011-06-27 19:23 ` Gregory Farnum 0 siblings, 0 replies; 3+ messages in thread From: Gregory Farnum @ 2011-06-27 19:23 UTC (permalink / raw) To: Josh Durgin, Martin Wilderoth; +Cc: ceph-devel On Jun 27, 2011, at 12:19 PM, Josh Durgin wrote: > On 06/25/2011 08:48 PM, Martin Wilderoth wrote: >> Hello >> >> I have a ceph cluster of 6 osd 146gb each. I have copied a lot of data >> filling to 87%. Between the osd's the data is not evenly distributed >> >> host1 >> /dev/sdb 137G 119G 15G 90% /data/osd0 >> /dev/sdc 137G 126G 7.4G 95% /data/osd1 >> >> host2 >> /dev/sdc 137G 114G 21G 85% /data/osd2 >> /dev/sdd 137G 130G 3.6G 98% /data/osd3 >> >> host3 >> /dev/sdb 137G 107G 27G 81% /data/osd4 >> /dev/sdc 137G 98G 36G 74% /data/osd5 >> >> During the copy i got I/O error, but after restarting the cluster it seems fine. >> >> By some reason osd3 seems to have much more data than osd5. Is there a way of geting the data distributed better ?. > > Hi Martin, > > Since the distribution is pseudo-random, you'll get some variance from an even split. You can reweight the osds manually with: > > ceph osd reweight osd3 new_weight > > or use the more automatic: > > ceph osd reweight-by-utilization 110 > > This reduces the weight of all osds that have a utilization that is more than 110% of the average utilization. > > Josh That said if the data is this unevenly distributed something odd is going on. Are you using anything besides the filesystem on this cluster? If not we probably need to figure out if there's a problem with the hashing. -Greg ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2011-06-27 19:23 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <344660039.17770.1309060083991.JavaMail.root@mail.linserv.se>
2011-06-26 3:48 ` Data distribution Martin Wilderoth
2011-06-27 19:19 ` Josh Durgin
2011-06-27 19:23 ` Gregory Farnum
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.