* Re: EC backend benchmark [not found] <755F6B91B3BE364F9BCA11EA3F9E0C6F2CD8A41B@SACMBXIP01.sdcorp.global.sandisk.com> @ 2015-05-11 21:24 ` Loic Dachary [not found] ` <55511E0A.7010408-cLsNCMjd+0JAfugRpC6u6w@public.gmane.org> 2015-05-11 22:02 ` Loic Dachary 1 sibling, 1 reply; 4+ messages in thread From: Loic Dachary @ 2015-05-11 21:24 UTC (permalink / raw) To: Somnath Roy; +Cc: Ceph Development, ceph-users [-- Attachment #1: Type: text/plain, Size: 14630 bytes --] Hi, Thanks for sharing :-) Have you published the tools that you used to gather these results ? It would be great to have a way to reproduce the same measures in different contexts. Cheers On 11/05/2015 23:13, Somnath Roy wrote: > > > Hi Loic and community, > > > > I have gathered the following data on EC backend (all flash). I have decided to use Jerasure since space saving is the utmost priority. > > > > Setup: > > -------- > > > > 41 OSDs (each on 8 TB flash), 5 node Ceph cluster. 48 core HT enabled cpu/64 GB RAM. Tested with Rados Bench clients. > > > > > > EC plug-in > > > > EC ratio > > > > EC fault domain > > > > Workload > > > > Total clients > > > > Num client Host > > > > Runtime (Sec) > > > > QD > > (single client) > > > > Latency/client > > (avg/Max) > > > > BW (aggregated) > > > > Object_size > > > > Node Cpu usage % > > > > BW/HT core > > Jerasure > > > > 9,3 > > > > OSD > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.5/1.2 > > > > 1786 MB/S > > > > 4M > > > > 28% > > > > 132 MB/s > > Jerasure > > > > 9,3 > > > > OSD > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 0.9/2.1 > > > > 2174 MB/s > > > > 4M > > > > 35% > > > > 129 MB/s > > Jerasure > > > > 4,1 > > > > Host > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.5/2.3 > > > > 1737 MB/s > > > > 4M > > > > 14% > > > > 258 MB/s > > Jerasure > > > > 4,1 > > > > Host > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 1.0/25 (!) > > > > 1783 MB/s > > > > 4M > > > > 14% > > > > 265 MB/s > > Jerasure > > > > 15,3 > > > > OSD > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.6/1.4 > > > > 1530 MB/s > > > > 4M > > > > 40% > > > > 79 MB/s > > Jerasure > > > > 15,3 > > > > OSD > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 1.0/4.7 > > > > 1886 MB/s > > > > 4M > > > > 45% > > > > 87 MB/s > > Jerasure > > > > 6,2 > > > > OSD > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.5/1.2 > > > > 1917 MB/s > > > > 4M > > > > 24% > > > > 166 MB/s > > Jerasure > > > > 6,2 > > > > OSD > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 0.8/2.2 > > > > 2281 MB/s > > > > 4M > > > > 28% > > > > 170 MB/s > > Jerasure > > > > 6,2 (RS_r6_op) > > > > OSD > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.5/1.2 > > > > 1876 MB/s > > > > 4M > > > > 25% > > > > 156 MB/s > > Jerasure > > > > 6,2 (RS_r6_op) > > > > OSD > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 0.8/1.9 > > > > 2292 MB/s > > > > 4M > > > > 31% > > > > 154 MB/s > > *Jerasure* > > > > *6,2 (cauchy_orig)* > > > > *OSD* > > > > *PUT* > > > > *4* > > > > *1* > > > > *100* > > > > *64* > > > > *0.5/1.1* > > > > *2025 MB/s* > > > > *4M* > > > > *18%* > > > > *234 MB/s* > > *Jerasure* > > > > *6,2 (cauchy_orig)* > > > > *OSD* > > > > *PUT* > > > > *8* > > > > *2* > > > > *100* > > > > *64* > > > > *0.8/1.9* > > > > *2497 MB/s* > > > > *4M* > > > > *21%* > > > > *247 MB/s* > > Jerasure > > > > 6,2 (cauchy_good) > > > > OSD > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.5/1.3 > > > > 1947MB/s > > > > 4M > > > > 18% > > > > 225 MB/s > > Jerasure > > > > 6,2 (cauchy_good) > > > > OSD > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 0.9/8.5 > > > > 2336 MB/s > > > > 4M > > > > 21% > > > > 231 MB/s > > Jerasure > > > > 6,2 (liberation) > > > > OSD > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.6/1.6 > > > > 1806 MB/s > > > > 4M > > > > 16% > > > > 235 MB/s > > Jerasure > > > > 6,2 (liberation) > > > > OSD > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 1.1/12 > > > > 1969 MB/s > > > > 4M > > > > 17% > > > > 241 MB/s > > Jerasure > > > > 6,2 (blaum_roth) > > > > OSD > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.5/1.5 > > > > 1859 MB/s > > > > 4M > > > > 17% > > > > 227 MB/s > > Jerasure > > > > 6,2 (blaum_roth) > > > > OSD > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 1.0/5.8 > > > > 2042 MB/s > > > > 4M > > > > 19% > > > > 224 MB/s > > Jerasure > > > > 6,2 (liber8tion) > > > > OSD > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.5/1.3 > > > > 1809 MB/s > > > > 4M > > > > 17% > > > > 221 MB/s > > Jerasure > > > > 6,2 (liber8tion) > > > > OSD > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 1.1/15.7 > > > > 1749 MB/s > > > > 4M > > > > 16% > > > > 227 MB/s > > Jerasure > > > > 10,2 (cauchy_orig) > > > > OSD > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.5/1.3 > > > > 2066 MB/s > > > > 4M > > > > 20% > > > > 215 MB/s > > Jerasure > > > > 10,2 (cauchy_orig) > > > > OSD > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 0.9/6.2 > > > > 2019 MB/s > > > > 4M > > > > 24% > > > > 175 MB/s > > *Jerasure* > > > > *14,2 (cauchy_orig)* > > > > *OSD* > > > > *PUT* > > > > *4* > > > > *1* > > > > *100* > > > > *64* > > > > *0.5/1.5* > > > > *1872 MB/s* > > > > *4M* > > > > *18%* > > > > *216 MB/s* > > *Jerasure* > > > > *14,2 (cauchy_orig)* > > > > *OSD* > > > > *PUT* > > > > *8* > > > > *2* > > > > *100* > > > > *64* > > > > *1.0/7.4* > > > > *2043 Mb/s* > > > > *4M* > > > > *18%* > > > > *236 MB/s* > > Replication > > > > 2 replica > > > > Host > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.7/8.8 > > > > 1198 MB/s > > > > 4M > > > > 8% > > > > 311 MB/s > > Replication > > > > 2 replica > > > > Host > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 1.7/33 > > > > 1256 MB/s > > > > 4M > > > > 8% > > > > 327 MB/s > > Jerasure > > > > 9,3 > > > > OSD > > > > GET > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.2/0.6 > > > > 4338 MB/s > > > > 4M > > > > 24% > > > > 376 MB/s > > Jerasure > > > > 9,3 > > > > OSD > > > > GET > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 0.2/0.9 > > > > 8002 MB/s > > > > 4M > > > > 54% > > > > 308 MB/s > > Jerasure > > > > 4,1 > > > > Host > > > > GET > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.2/0.7 > > > > 4630 MB/s > > > > 4M > > > > 18% > > > > 535 MB/s > > Jerasure > > > > 4,1 > > > > Host > > > > GET > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 0.2/0.7 > > > > 8600 MB/s > > > > 4M > > > > 42% > > > > 426 MB/s > > *Jerasure* > > > > *14,2 (cauchy_orig)* > > > > *OSD* > > > > *GET* > > > > *4* > > > > *1* > > > > *100* > > > > *64* > > > > *0.2/0.7* > > > > *4329 MB/s* > > > > *4M* > > > > *24%* > > > > *375 MB/s* > > Jerasure > > > > 10,2 (cauchy_orig) > > > > OSD > > > > GET > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.2/0.6 > > > > 4366 MB/s > > > > 4M > > > > 19% > > > > 478 MB/s > > Jerasure > > > > 6,2 (cauchy_orig) > > > > OSD > > > > GET > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.2/0.7 > > > > 4370 MB/s > > > > 4M > > > > 16% > > > > 569 MB/s > > Jerasure > > > > 6,2 > > > > OSD > > > > GET > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.2/0.5 > > > > 4324 MB/s > > > > 4M > > > > 20% > > > > 450 MB/s > > Replication > > > > 2 replica > > > > Host > > > > GET > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.2/0.5 > > > > 4418 MB/s > > > > 4M > > > > 8% > > > > 1150 MB/s > > Replication > > > > 2 replica > > > > Host > > > > GET > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 0.2/0.9 > > > > 8935 MB/s > > > > 4M > > > > 18% > > > > 1034 MB/s > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Summary : > > ------------- > > > > 1. It is doing pretty good in Reads and 4 Rados Bench clients are saturating 40 GB network. With more physical server, it is scaling almost linearly and saturating 40 GbE on both the host. > > > > 2. As suspected with Ceph, problem is again with writes. Throughput wise it is beating replicated pools in significant numbers. But, it is not scaling with multiple clients and not saturating anything. > > > > > > So, my question is the following. > > > > 1. Probably, nothing to do with EC backend, we are suffering because of filestore inefficiencies. Do you think any tunable like EC stipe size (or anything else) will help here ? > > > > 2. I couldn’t make fault domain as ‘host’, because of HW limitation. Do you think will that play a role in performance for bigger k values ? > > > > 3. Even though it is not saturating 40 GbE for writes, do you think separating out public/private network will help in terms of performance ? > > > > > > Any feedback on this is much appreciated. > > > > > > Thanks & Regards > > Somnath > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > > PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). > -- Loïc Dachary, Artisan Logiciel Libre [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 4+ messages in thread
[parent not found: <55511E0A.7010408-cLsNCMjd+0JAfugRpC6u6w@public.gmane.org>]
* Re: EC backend benchmark [not found] ` <55511E0A.7010408-cLsNCMjd+0JAfugRpC6u6w@public.gmane.org> @ 2015-05-11 21:28 ` Somnath Roy 0 siblings, 0 replies; 4+ messages in thread From: Somnath Roy @ 2015-05-11 21:28 UTC (permalink / raw) To: Loic Dachary; +Cc: Ceph Development, ceph-users Loic, I thought this one didn't go through ! I have sent another mail with attached doc. This is the data with rados bench . In case you missed it, could you please share your thoughts on the questions I posted (way below in the mail, not sure how so many space came along!!) below ? Thanks & Regards Somnath -----Original Message----- From: Loic Dachary [mailto:loic-cLsNCMjd+0JAfugRpC6u6w@public.gmane.org] Sent: Monday, May 11, 2015 2:24 PM To: Somnath Roy Cc: Ceph Development; ceph-users Subject: Re: EC backend benchmark Hi, Thanks for sharing :-) Have you published the tools that you used to gather these results ? It would be great to have a way to reproduce the same measures in different contexts. Cheers On 11/05/2015 23:13, Somnath Roy wrote: > > > Hi Loic and community, > > > > I have gathered the following data on EC backend (all flash). I have decided to use Jerasure since space saving is the utmost priority. > > > > Setup: > > -------- > > > > 41 OSDs (each on 8 TB flash), 5 node Ceph cluster. 48 core HT enabled cpu/64 GB RAM. Tested with Rados Bench clients. > > > > > > EC plug-in > > > > EC ratio > > > > EC fault domain > > > > Workload > > > > Total clients > > > > Num client Host > > > > Runtime (Sec) > > > > QD > > (single client) > > > > Latency/client > > (avg/Max) > > > > BW (aggregated) > > > > Object_size > > > > Node Cpu usage % > > > > BW/HT core > > Jerasure > > > > 9,3 > > > > OSD > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.5/1.2 > > > > 1786 MB/S > > > > 4M > > > > 28% > > > > 132 MB/s > > Jerasure > > > > 9,3 > > > > OSD > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 0.9/2.1 > > > > 2174 MB/s > > > > 4M > > > > 35% > > > > 129 MB/s > > Jerasure > > > > 4,1 > > > > Host > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.5/2.3 > > > > 1737 MB/s > > > > 4M > > > > 14% > > > > 258 MB/s > > Jerasure > > > > 4,1 > > > > Host > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 1.0/25 (!) > > > > 1783 MB/s > > > > 4M > > > > 14% > > > > 265 MB/s > > Jerasure > > > > 15,3 > > > > OSD > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.6/1.4 > > > > 1530 MB/s > > > > 4M > > > > 40% > > > > 79 MB/s > > Jerasure > > > > 15,3 > > > > OSD > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 1.0/4.7 > > > > 1886 MB/s > > > > 4M > > > > 45% > > > > 87 MB/s > > Jerasure > > > > 6,2 > > > > OSD > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.5/1.2 > > > > 1917 MB/s > > > > 4M > > > > 24% > > > > 166 MB/s > > Jerasure > > > > 6,2 > > > > OSD > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 0.8/2.2 > > > > 2281 MB/s > > > > 4M > > > > 28% > > > > 170 MB/s > > Jerasure > > > > 6,2 (RS_r6_op) > > > > OSD > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.5/1.2 > > > > 1876 MB/s > > > > 4M > > > > 25% > > > > 156 MB/s > > Jerasure > > > > 6,2 (RS_r6_op) > > > > OSD > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 0.8/1.9 > > > > 2292 MB/s > > > > 4M > > > > 31% > > > > 154 MB/s > > *Jerasure* > > > > *6,2 (cauchy_orig)* > > > > *OSD* > > > > *PUT* > > > > *4* > > > > *1* > > > > *100* > > > > *64* > > > > *0.5/1.1* > > > > *2025 MB/s* > > > > *4M* > > > > *18%* > > > > *234 MB/s* > > *Jerasure* > > > > *6,2 (cauchy_orig)* > > > > *OSD* > > > > *PUT* > > > > *8* > > > > *2* > > > > *100* > > > > *64* > > > > *0.8/1.9* > > > > *2497 MB/s* > > > > *4M* > > > > *21%* > > > > *247 MB/s* > > Jerasure > > > > 6,2 (cauchy_good) > > > > OSD > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.5/1.3 > > > > 1947MB/s > > > > 4M > > > > 18% > > > > 225 MB/s > > Jerasure > > > > 6,2 (cauchy_good) > > > > OSD > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 0.9/8.5 > > > > 2336 MB/s > > > > 4M > > > > 21% > > > > 231 MB/s > > Jerasure > > > > 6,2 (liberation) > > > > OSD > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.6/1.6 > > > > 1806 MB/s > > > > 4M > > > > 16% > > > > 235 MB/s > > Jerasure > > > > 6,2 (liberation) > > > > OSD > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 1.1/12 > > > > 1969 MB/s > > > > 4M > > > > 17% > > > > 241 MB/s > > Jerasure > > > > 6,2 (blaum_roth) > > > > OSD > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.5/1.5 > > > > 1859 MB/s > > > > 4M > > > > 17% > > > > 227 MB/s > > Jerasure > > > > 6,2 (blaum_roth) > > > > OSD > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 1.0/5.8 > > > > 2042 MB/s > > > > 4M > > > > 19% > > > > 224 MB/s > > Jerasure > > > > 6,2 (liber8tion) > > > > OSD > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.5/1.3 > > > > 1809 MB/s > > > > 4M > > > > 17% > > > > 221 MB/s > > Jerasure > > > > 6,2 (liber8tion) > > > > OSD > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 1.1/15.7 > > > > 1749 MB/s > > > > 4M > > > > 16% > > > > 227 MB/s > > Jerasure > > > > 10,2 (cauchy_orig) > > > > OSD > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.5/1.3 > > > > 2066 MB/s > > > > 4M > > > > 20% > > > > 215 MB/s > > Jerasure > > > > 10,2 (cauchy_orig) > > > > OSD > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 0.9/6.2 > > > > 2019 MB/s > > > > 4M > > > > 24% > > > > 175 MB/s > > *Jerasure* > > > > *14,2 (cauchy_orig)* > > > > *OSD* > > > > *PUT* > > > > *4* > > > > *1* > > > > *100* > > > > *64* > > > > *0.5/1.5* > > > > *1872 MB/s* > > > > *4M* > > > > *18%* > > > > *216 MB/s* > > *Jerasure* > > > > *14,2 (cauchy_orig)* > > > > *OSD* > > > > *PUT* > > > > *8* > > > > *2* > > > > *100* > > > > *64* > > > > *1.0/7.4* > > > > *2043 Mb/s* > > > > *4M* > > > > *18%* > > > > *236 MB/s* > > Replication > > > > 2 replica > > > > Host > > > > PUT > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.7/8.8 > > > > 1198 MB/s > > > > 4M > > > > 8% > > > > 311 MB/s > > Replication > > > > 2 replica > > > > Host > > > > PUT > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 1.7/33 > > > > 1256 MB/s > > > > 4M > > > > 8% > > > > 327 MB/s > > Jerasure > > > > 9,3 > > > > OSD > > > > GET > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.2/0.6 > > > > 4338 MB/s > > > > 4M > > > > 24% > > > > 376 MB/s > > Jerasure > > > > 9,3 > > > > OSD > > > > GET > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 0.2/0.9 > > > > 8002 MB/s > > > > 4M > > > > 54% > > > > 308 MB/s > > Jerasure > > > > 4,1 > > > > Host > > > > GET > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.2/0.7 > > > > 4630 MB/s > > > > 4M > > > > 18% > > > > 535 MB/s > > Jerasure > > > > 4,1 > > > > Host > > > > GET > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 0.2/0.7 > > > > 8600 MB/s > > > > 4M > > > > 42% > > > > 426 MB/s > > *Jerasure* > > > > *14,2 (cauchy_orig)* > > > > *OSD* > > > > *GET* > > > > *4* > > > > *1* > > > > *100* > > > > *64* > > > > *0.2/0.7* > > > > *4329 MB/s* > > > > *4M* > > > > *24%* > > > > *375 MB/s* > > Jerasure > > > > 10,2 (cauchy_orig) > > > > OSD > > > > GET > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.2/0.6 > > > > 4366 MB/s > > > > 4M > > > > 19% > > > > 478 MB/s > > Jerasure > > > > 6,2 (cauchy_orig) > > > > OSD > > > > GET > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.2/0.7 > > > > 4370 MB/s > > > > 4M > > > > 16% > > > > 569 MB/s > > Jerasure > > > > 6,2 > > > > OSD > > > > GET > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.2/0.5 > > > > 4324 MB/s > > > > 4M > > > > 20% > > > > 450 MB/s > > Replication > > > > 2 replica > > > > Host > > > > GET > > > > 4 > > > > 1 > > > > 100 > > > > 64 > > > > 0.2/0.5 > > > > 4418 MB/s > > > > 4M > > > > 8% > > > > 1150 MB/s > > Replication > > > > 2 replica > > > > Host > > > > GET > > > > 8 > > > > 2 > > > > 100 > > > > 64 > > > > 0.2/0.9 > > > > 8935 MB/s > > > > 4M > > > > 18% > > > > 1034 MB/s > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Summary : > > ------------- > > > > 1. It is doing pretty good in Reads and 4 Rados Bench clients are saturating 40 GB network. With more physical server, it is scaling almost linearly and saturating 40 GbE on both the host. > > > > 2. As suspected with Ceph, problem is again with writes. Throughput wise it is beating replicated pools in significant numbers. But, it is not scaling with multiple clients and not saturating anything. > > > > > > So, my question is the following. > > > > 1. Probably, nothing to do with EC backend, we are suffering because of filestore inefficiencies. Do you think any tunable like EC stipe size (or anything else) will help here ? > > > > 2. I couldn't make fault domain as 'host', because of HW limitation. Do you think will that play a role in performance for bigger k values ? > > > > 3. Even though it is not saturating 40 GbE for writes, do you think separating out public/private network will help in terms of performance ? > > > > > > Any feedback on this is much appreciated. > > > > > > Thanks & Regards > > Somnath > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > > PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). > -- Loïc Dachary, Artisan Logiciel Libre ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: EC backend benchmark [not found] <755F6B91B3BE364F9BCA11EA3F9E0C6F2CD8A41B@SACMBXIP01.sdcorp.global.sandisk.com> 2015-05-11 21:24 ` EC backend benchmark Loic Dachary @ 2015-05-11 22:02 ` Loic Dachary 2015-05-11 22:11 ` Somnath Roy 1 sibling, 1 reply; 4+ messages in thread From: Loic Dachary @ 2015-05-11 22:02 UTC (permalink / raw) To: Somnath Roy; +Cc: ceph-users@lists.ceph.com, Ceph Development [-- Attachment #1: Type: text/plain, Size: 1529 bytes --] Hi, [Sorry I missed the body of your questions, here is my answer ;-] On 11/05/2015 23:13, Somnath Roy wrote:> Summary : > > ------------- > > > > 1. It is doing pretty good in Reads and 4 Rados Bench clients are saturating 40 GB network. With more physical server, it is scaling almost linearly and saturating 40 GbE on both the host. > > > > 2. As suspected with Ceph, problem is again with writes. Throughput wise it is beating replicated pools in significant numbers. But, it is not scaling with multiple clients and not saturating anything. > > > > > > So, my question is the following. > > > > 1. Probably, nothing to do with EC backend, we are suffering because of filestore inefficiencies. Do you think any tunable like EC stipe size (or anything else) will help here ? I think Mark Nelson would be in a better position that me to answer as he has conducted many experiments with erasure coded pools. > 2. I couldn’t make fault domain as ‘host’, because of HW limitation. Do you think will that play a role in performance for bigger k values ? I don't see a reason why there would be a direct relationship between the failure domain and the values of k. Do you have a specific example in mind ? > 3. Even though it is not saturating 40 GbE for writes, do you think separating out public/private network will help in terms of performance ? I don't think so. What is the bottleneck ? CPU or disk I/O ? Cheers -- Loïc Dachary, Artisan Logiciel Libre [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 198 bytes --] ^ permalink raw reply [flat|nested] 4+ messages in thread
* RE: EC backend benchmark 2015-05-11 22:02 ` Loic Dachary @ 2015-05-11 22:11 ` Somnath Roy 0 siblings, 0 replies; 4+ messages in thread From: Somnath Roy @ 2015-05-11 22:11 UTC (permalink / raw) To: Loic Dachary; +Cc: ceph-users@lists.ceph.com, Ceph Development Thanks Loic.. << inline Regards Somnath -----Original Message----- From: Loic Dachary [mailto:loic@dachary.org] Sent: Monday, May 11, 2015 3:02 PM To: Somnath Roy Cc: ceph-users@lists.ceph.com; Ceph Development Subject: Re: EC backend benchmark Hi, [Sorry I missed the body of your questions, here is my answer ;-] On 11/05/2015 23:13, Somnath Roy wrote:> Summary : > > ------------- > > > > 1. It is doing pretty good in Reads and 4 Rados Bench clients are saturating 40 GB network. With more physical server, it is scaling almost linearly and saturating 40 GbE on both the host. > > > > 2. As suspected with Ceph, problem is again with writes. Throughput wise it is beating replicated pools in significant numbers. But, it is not scaling with multiple clients and not saturating anything. > > > > > > So, my question is the following. > > > > 1. Probably, nothing to do with EC backend, we are suffering because of filestore inefficiencies. Do you think any tunable like EC stipe size (or anything else) will help here ? I think Mark Nelson would be in a better position that me to answer as he has conducted many experiments with erasure coded pools. [Somnath] Sure, Mark, any insight ? :-) > 2. I couldn't make fault domain as 'host', because of HW limitation. Do you think will that play a role in performance for bigger k values ? I don't see a reason why there would be a direct relationship between the failure domain and the values of k. Do you have a specific example in mind ? [Somnath] Nope, other than more network hops..If failure domain is OSD, more than one chunks could be within a host...But, since I have 40GbE and not saturating network BW (for bigger cluster that probability is less), IMO it shouldn't matter. I thought of checking with you. > 3. Even though it is not saturating 40 GbE for writes, do you think separating out public/private network will help in terms of performance ? I don't think so. What is the bottleneck ? CPU or disk I/O ? [Somnath] For write, no resources (cpu/network/disk) are saturating. Cheers -- Loïc Dachary, Artisan Logiciel Libre ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2015-05-11 22:13 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <755F6B91B3BE364F9BCA11EA3F9E0C6F2CD8A41B@SACMBXIP01.sdcorp.global.sandisk.com>
2015-05-11 21:24 ` EC backend benchmark Loic Dachary
[not found] ` <55511E0A.7010408-cLsNCMjd+0JAfugRpC6u6w@public.gmane.org>
2015-05-11 21:28 ` Somnath Roy
2015-05-11 22:02 ` Loic Dachary
2015-05-11 22:11 ` Somnath Roy
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.