Ceph PG Number

Ceph PG Number

[Zhen Cao]

Hi,
I am reading the RADOS paper,but i could not understand why more PGs
per OSD result in a more balanced distribution.I konw the mapping
procedure between oid and osd，but it is neverless difficult to
understand for me.
Thanks for your help!

Re: Ceph PG Number

[Piotr Dałek]

On Wed, Nov 30, 2016 at 04:22:10PM +0800, Zhen Cao wrote:
> Hi,
> I am reading the RADOS paper,but i could not understand why more PGs
> per OSD result in a more balanced distribution.I konw the mapping
> procedure between oid and osd，but it is neverless difficult to
> understand for me.
> Thanks for your help!

In a nuthshell - because each PG corresponds to multiple objects.
And because of this, it is possible that some PGs will aggregate more
objects than other PGs, resulting in lack of acutal balance. Also see here:
http://docs.ceph.com/docs/jewel/rados/operations/placement-groups/ 

-- 
Piotr Dałek
branch@predictor.org.pl
http://blog.predictor.org.pl

Re: Ceph PG Number

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1089 bytes --]

On Wed, 30 Nov 2016, Piotr Dałek wrote:
> On Wed, Nov 30, 2016 at 04:22:10PM +0800, Zhen Cao wrote:
> > Hi,
> > I am reading the RADOS paper,but i could not understand why more PGs
> > per OSD result in a more balanced distribution.I konw the mapping
> > procedure between oid and osd，but it is neverless difficult to
> > understand for me.
> > Thanks for your help!
> 
> In a nuthshell - because each PG corresponds to multiple objects.
> And because of this, it is possible that some PGs will aggregate more
> objects than other PGs, resulting in lack of acutal balance. Also see here:
> http://docs.ceph.com/docs/jewel/rados/operations/placement-groups/ 

More generally, it's a [pseudo-]random process that puts objects in PGs 
and PGs on OSDs.  If you have a small number of objects, you'll have a 
small number of PG and a large variance (one PG may have 2 objects, 
another 4).  If it's a large number, things will converge toward an 
average (e.g., very close to ~1,000,000 objects per PG).  Same goes for 
PGs distributed to OSDs.  Larger numbers mean a tighter variance.

sage

Fwd: Ceph PG Number

[Zhen Cao]

---------- Forwarded message ----------
From: Zhen Cao <juehuiconst@gmail.com>
Date: 2016-11-30 17:01 GMT+08:00
Subject: Re: Ceph PG Number
To: Piotr Dałek <branch@predictor.org.pl>


Thanks! I understood it.

http://docs.ceph.com/docs/jewel/rados/operations/placement-groups/ said

Ideally objects are evenly distributed in each placement group. Since
CRUSH computes the placement group for each object, but does not
actually know how much data is stored in each OSD within this
placement group, the ratio between the number of placement groups and
the number of OSDs may influence the distribution of the data
significantly.

For instance, if there was single a placement group for ten OSDs in a
three replica pool, only three OSD would be used because CRUSH would
have no other choice. When more placement groups are available,
objects are more likely to be evenly spread among them. CRUSH also
makes every effort to evenly spread OSDs among all existing Placement
Groups.

I got it,thanks for your help!


2016-11-30 16:31 GMT+08:00 Piotr Dałek <branch@predictor.org.pl>:
> On Wed, Nov 30, 2016 at 04:22:10PM +0800, Zhen Cao wrote:
>> Hi,
>> I am reading the RADOS paper,but i could not understand why more PGs
>> per OSD result in a more balanced distribution.I konw the mapping
>> procedure between oid and osd，but it is neverless difficult to
>> understand for me.
>> Thanks for your help!
>
> In a nuthshell - because each PG corresponds to multiple objects.
> And because of this, it is possible that some PGs will aggregate more
> objects than other PGs, resulting in lack of acutal balance. Also see here:
> http://docs.ceph.com/docs/jewel/rados/operations/placement-groups/
>
> --
> Piotr Dałek
> branch@predictor.org.pl
> http://blog.predictor.org.pl