Re: placement group sizing

[Mark Nelson <mark.nelson@inktank.com>]

On 04/25/2013 07:39 AM, Anders Saaby wrote:
> Hi,
>
> We are working on prototype infrastructure for RADOS clusters, and are now ready to deploy the first production size storage pool. One question remains; How many placement groups will we need, balancing memory footprint and ability to level data placement and data reads. - And still keeping stuff within sane limits.
>
> Our initial plan is to deploy 4PB pools, based on 4TB drives with 3 replicas (One OSD/disk). So, 3.000 disks per pool.
>
> Acording to the documentation 1), we should have: 3.000 OSDs * 100 / 3 replicas == 100.000 placement groups.
>
>  From the maillist, 100.000 PG's is way more than I have seen, so, do you have any insights and advises on pg_num for a RADOS pool with these characteristics? Also, will it be a problem with a pg_num size this bit, if the pool is started out with only ~100 OSDs, and then grown to 3.000.

I pretty regularly test single-mon configurations with 64k PGs.  ~100k 
PGs tends to be starting to get a bit intense, but with a larger mon 
cluster and some tweaking it should be doable.

I don't mean to push anything on you, but if you guys are really 
thinking about deploying multiple 4PB pools, you might want to talk to 
us about some kind of support/consulting agreement.  That's a lot of 
storage!

Mark

>
>
> Thanks in advance,
> Anders
>
> 1: http://ceph.com/docs/master/rados/operations/placement-groups/
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>