chaning pg_num / pgp_num after adding more osds

chaning pg_num / pgp_num after adding more osds

[Stefan Priebe - Profihost AG]

Hello list,

Is there a way to change the number of pg_num / pgp_num after adding 
more osds?

I mean i would like to start with 16 OSDs but i think i'll expand over 
time to up to 100 OSDs. So i think i need to tune pg_num / pgp_num.

Greets,
Stefan

Re: chaning pg_num / pgp_num after adding more osds

[Josh Durgin]

On 11/05/2012 06:14 AM, Stefan Priebe - Profihost AG wrote:
> Hello list,
>
> Is there a way to change the number of pg_num / pgp_num after adding
> more osds?

The pg_num/pgp_num settings are only used by mkcephfs at install time.

> I mean i would like to start with 16 OSDs but i think i'll expand over
> time to up to 100 OSDs. So i think i need to tune pg_num / pgp_num.

You can specify pg_num when creating a pool:

ceph osd pool create <name> <pgnum>

But you don't want to have too many (thousands per osd). Being able
to change the number of pgs in a pool (pg splitting/merging) is in
the works, but in the mean time you can create more pools after you add
a bunch of osds to keep your pg/osd ratio around 100.

Josh

Re: chaning pg_num / pgp_num after adding more osds

[Stefan Priebe - Profihost AG]

Am 06.11.2012 00:45, schrieb Josh Durgin:
> On 11/05/2012 06:14 AM, Stefan Priebe - Profihost AG wrote:
>> Hello list,
>>
>> Is there a way to change the number of pg_num / pgp_num after adding
>> more osds?
>
> The pg_num/pgp_num settings are only used by mkcephfs at install time.
>
>> I mean i would like to start with 16 OSDs but i think i'll expand over
>> time to up to 100 OSDs. So i think i need to tune pg_num / pgp_num.
>
> You can specify pg_num when creating a pool:
>
> ceph osd pool create <name> <pgnum>
>
> But you don't want to have too many (thousands per osd). Being able
> to change the number of pgs in a pool (pg splitting/merging) is in
> the works, but in the mean time you can create more pools after you add
> a bunch of osds to keep your pg/osd ratio around 100.

Thanks Josh for your explanation. I'm not sure if i already understood 
what pg is at all.

First i see that ceph is creating 832 pgs for 12 osds in my case per 
default. This is 69,3333333 per OSD. You're talking about 100 - is the 
default calculation broken or hardcoded?

When i have one pool with 800 pgs and i add 20 new OSDs how does a new 
pool help? I mean the old pools will stay with 800 pgs.

Thanks again!

Greets,
Stefan

Re: chaning pg_num / pgp_num after adding more osds

[Sage Weil]

On Tue, 6 Nov 2012, Stefan Priebe - Profihost AG wrote:
> Am 06.11.2012 00:45, schrieb Josh Durgin:
> > On 11/05/2012 06:14 AM, Stefan Priebe - Profihost AG wrote:
> > > Hello list,
> > > 
> > > Is there a way to change the number of pg_num / pgp_num after adding
> > > more osds?
> > 
> > The pg_num/pgp_num settings are only used by mkcephfs at install time.
> > 
> > > I mean i would like to start with 16 OSDs but i think i'll expand over
> > > time to up to 100 OSDs. So i think i need to tune pg_num / pgp_num.
> > 
> > You can specify pg_num when creating a pool:
> > 
> > ceph osd pool create <name> <pgnum>
> > 
> > But you don't want to have too many (thousands per osd). Being able
> > to change the number of pgs in a pool (pg splitting/merging) is in
> > the works, but in the mean time you can create more pools after you add
> > a bunch of osds to keep your pg/osd ratio around 100.
> 
> Thanks Josh for your explanation. I'm not sure if i already understood what pg
> is at all.
> 
> First i see that ceph is creating 832 pgs for 12 osds in my case per default.
> This is 69,3333333 per OSD. You're talking about 100 - is the default
> calculation broken or hardcoded?

Each PG has N copies, where N defaults to 2.  So that would be ~139 per 
osd.

The 100 per osd is a very rough guide; that is just a decent balance 
between variance in utilization (~10%) and pg overhead (too many PGs can 
use RAM on the ceph-osds and introduce more replication/syncrhonization 
related network traffic).

> When i have one pool with 800 pgs and i add 20 new OSDs how does a new pool
> help? I mean the old pools will stay with 800 pgs.

It make the distribution of existing data less coarse, but as the size of 
teh PGs for the new pool increases things will tend to level out.

My suggestion is to overshoot the PG count a little bit (not too much!), 
maybe ~200 pgs per osd.  If things get too unbalanced after a significant 
expansion you can put new data is new pools, or make fine-grainted 
adjustments in the CRUSH map.  

I suspect that this will be sufficient for just about everyone until the 
splitting functionality is in place... hopefully in 1-2 dev releases after 
bobtail.

sage

Re: chaning pg_num / pgp_num after adding more osds

[Stefan Kleijkers]

On 11/06/2012 11:59 AM, Sage Weil wrote:
>
>> When i have one pool with 800 pgs and i add 20 new OSDs how does a new pool
>> help? I mean the old pools will stay with 800 pgs.
> It make the distribution of existing data less coarse, but as the size of
> teh PGs for the new pool increases things will tend to level out.
>
> My suggestion is to overshoot the PG count a little bit (not too much!),
> maybe ~200 pgs per osd.  If things get too unbalanced after a significant
> expansion you can put new data is new pools, or make fine-grainted
> adjustments in the CRUSH map.
>
> I suspect that this will be sufficient for just about everyone until the
> splitting functionality is in place... hopefully in 1-2 dev releases after
> bobtail.
>
Maybe I missed something but do you mean about 100-200 PG's per pool per 
OSD or 100-200 PG's per OSD? Because when you create new pools, the PG's 
per OSD increases a lot.

Stefan