Re: Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

[Alexandre DERUMIER <aderumier-U/x3PoR4x10AvxtiuMwx3w@public.gmane.org>]

>>I don't have really good insight yet into how tweaking these would 
>>affect single-osd performance. I know the PCIe SSDs do have multiple 
>>controllers on-board so perhaps increasing the number of shards would 
>>improve things, but I suspect that going too high could maybe start 
>>hurting performance as well. Have you done any testing here? It could 
>>be an interesting follow-up paper. 

I think it should be tunned regarding number of osds and number of cores you have.
I have done test in past with sommath values

osd_op_num_threads_per_shard = 1
osd_op_num_shards = 25
filestore_fd_cache_size = 64
filestore_fd_cache_shards = 32

But don't have take time to try differents values.
But I was to be able to reach 120000iops 4k read with 3osd if I remember. (But I was limited by client cpu)

I'm going to do big benchmark next month (3 nodes (20cores) with 6ssd each),
So I'll try to test different sharding values, with different number of osd.


----- Mail original -----
De: "Mark Nelson" <mnelson@redhat.com>
À: "aderumier" <aderumier@odiso.com>
Cc: "ceph-devel" <ceph-devel@vger.kernel.org>, "ceph-users" <ceph-users@lists.ceph.com>
Envoyé: Mercredi 18 Février 2015 15:56:44
Objet: Re: [ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

Hi Alex, 

Thanks! I didn't tweak the sharding settings at all, so they are just 
at the default values: 

OPTION(osd_op_num_threads_per_shard, OPT_INT, 2) 
OPTION(osd_op_num_shards, OPT_INT, 5) 

I don't have really good insight yet into how tweaking these would 
affect single-osd performance. I know the PCIe SSDs do have multiple 
controllers on-board so perhaps increasing the number of shards would 
improve things, but I suspect that going too high could maybe start 
hurting performance as well. Have you done any testing here? It could 
be an interesting follow-up paper. 

Mark 

On 02/18/2015 02:34 AM, Alexandre DERUMIER wrote: 
> Nice Work Mark ! 
> 
> I don't see any tuning about sharding in the config file sample 
> 
> (osd_op_num_threads_per_shard,osd_op_num_shards,...) 
> 
> as you only use 1 ssd for the bench, I think it should improve results for hammer ? 
> 
> 
> 
> ----- Mail original ----- 
> De: "Mark Nelson" <mnelson@redhat.com> 
> À: "ceph-devel" <ceph-devel@vger.kernel.org> 
> Cc: "ceph-users" <ceph-users@lists.ceph.com> 
> Envoyé: Mardi 17 Février 2015 18:37:01 
> Objet: [ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison 
> 
> Hi All, 
> 
> I wrote up a short document describing some tests I ran recently to look 
> at how SSD backed OSD performance has changed across our LTS releases. 
> This is just looking at RADOS performance and not RBD or RGW. It also 
> doesn't offer any real explanations regarding the results. It's just a 
> first high level step toward understanding some of the behaviors folks 
> on the mailing list have reported over the last couple of releases. I 
> hope you find it useful. 
> 
> Mark 
> 
> _______________________________________________ 
> ceph-users mailing list 
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
> -- 
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
> the body of a message to majordomo@vger.kernel.org 
> More majordomo info at http://vger.kernel.org/majordomo-info.html 
> 

_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com