* RadosGW performance and disk space usage @ 2013-01-24 15:27 Cesar Mello 2013-01-25 23:14 ` Sam Lang 0 siblings, 1 reply; 11+ messages in thread From: Cesar Mello @ 2013-01-24 15:27 UTC (permalink / raw) To: ceph-devel Hi! I have successfully prototyped read/write access to ceph from Windows using the S3 API, thanks so much for the help. Now I would like to do some prototypes targeting performance evaluation. My scenario typically requires parallel storage of data from tens of thousands of loggers, but scalability to hundreds of thousands is the main reason for investigating ceph. My tests using a single laptop running ceph with 2 local OSDs and local radosgw allows writing in average 2.5 small objects per second (100 objects in 40 seconds). Is this the expected performance? It seems to be I/O bound because the HDD led keeps on during the PutObject requests. Any suggestion or documentation pointers for profiling are very appreciated. I am afraid the S3 API is not good for my scenario, because there is no way to append data to existing objects (so I won't be able to model a single object for each data collector). If this is the case, then I would need to store billions of small objects. I would like to know how much disk space each object instance requires other than the object content length. If the S3 API is not well suited to my scenario, then my effort should be better directed to porting or writing a native ceph client for Windows. I just need an API to read and write/append blocks to files. Any comments are really appreciated. Thank you a lot for the attention! Best regards Mello ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RadosGW performance and disk space usage 2013-01-24 15:27 RadosGW performance and disk space usage Cesar Mello @ 2013-01-25 23:14 ` Sam Lang 2013-01-26 5:35 ` Dan Mick ` (2 more replies) 0 siblings, 3 replies; 11+ messages in thread From: Sam Lang @ 2013-01-25 23:14 UTC (permalink / raw) To: Cesar Mello; +Cc: ceph-devel@vger.kernel.org On Thu, Jan 24, 2013 at 9:27 AM, Cesar Mello <cmello@gmail.com> wrote: > Hi! > > I have successfully prototyped read/write access to ceph from Windows > using the S3 API, thanks so much for the help. > > Now I would like to do some prototypes targeting performance > evaluation. My scenario typically requires parallel storage of data > from tens of thousands of loggers, but scalability to hundreds of > thousands is the main reason for investigating ceph. > > My tests using a single laptop running ceph with 2 local OSDs and > local radosgw allows writing in average 2.5 small objects per second > (100 objects in 40 seconds). Is this the expected performance? It > seems to be I/O bound because the HDD led keeps on during the > PutObject requests. Any suggestion or documentation pointers for > profiling are very appreciated. Hi Mello, 2.5 objects/sec seems terribly slow, even on your laptop. How "small" are these objects? You might try to benchmark without the disk as a potential bottleneck, by putting your osd data and journals in /tmp (for benchmarking only of course) or create/mount a tmpfs and point your osd backends there. > > I am afraid the S3 API is not good for my scenario, because there is > no way to append data to existing objects (so I won't be able to model > a single object for each data collector). If this is the case, then I > would need to store billions of small objects. I would like to know > how much disk space each object instance requires other than the > object content length. > > If the S3 API is not well suited to my scenario, then my effort should > be better directed to porting or writing a native ceph client for > Windows. I just need an API to read and write/append blocks to files. > Any comments are really appreciated. Hopefully someone with more windows experience will give you better info/advice than I can. You could try to port the rados API to windows. Its purely userspace, but does rely on pthreads and other libc/gcc specifics. With something like cygwin a port might not be too hard though. If you decide to go that route, let us know how you progress! -sam > > Thank you a lot for the attention! > > Best regards > Mello > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RadosGW performance and disk space usage 2013-01-25 23:14 ` Sam Lang @ 2013-01-26 5:35 ` Dan Mick 2013-01-28 2:32 ` Dan Mick 2013-01-26 5:44 ` Marcus Sorensen [not found] ` <CALFpzo4u2PspjkvSig3LL821YY_XA2iHBjAh30JxgwWTFGsgcA@mail.gmail.com> 2 siblings, 1 reply; 11+ messages in thread From: Dan Mick @ 2013-01-26 5:35 UTC (permalink / raw) To: Sam Lang; +Cc: Cesar Mello, ceph-devel@vger.kernel.org >> If the S3 API is not well suited to my scenario, then my effort should >> be better directed to porting or writing a native ceph client for >> Windows. I just need an API to read and write/append blocks to files. >> Any comments are really appreciated. > > Hopefully someone with more windows experience will give you better > info/advice than I can. The python bindings should Just Work(tm). Just a thought. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RadosGW performance and disk space usage 2013-01-26 5:35 ` Dan Mick @ 2013-01-28 2:32 ` Dan Mick 0 siblings, 0 replies; 11+ messages in thread From: Dan Mick @ 2013-01-28 2:32 UTC (permalink / raw) To: Sam Lang; +Cc: Cesar Mello, ceph-devel@vger.kernel.org On 1/25/2013 9:35 PM, Dan Mick wrote: > >>> If the S3 API is not well suited to my scenario, then my effort should >>> be better directed to porting or writing a native ceph client for >>> Windows. I just need an API to read and write/append blocks to files. >>> Any comments are really appreciated. >> >> Hopefully someone with more windows experience will give you better >> info/advice than I can. > > The python bindings should Just Work(tm). Just a thought. ...but of course, as Sam points out, they rely on the C++ libs. Duh. Sorry to mislead. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RadosGW performance and disk space usage 2013-01-25 23:14 ` Sam Lang 2013-01-26 5:35 ` Dan Mick @ 2013-01-26 5:44 ` Marcus Sorensen [not found] ` <CALFpzo4u2PspjkvSig3LL821YY_XA2iHBjAh30JxgwWTFGsgcA@mail.gmail.com> 2 siblings, 0 replies; 11+ messages in thread From: Marcus Sorensen @ 2013-01-26 5:44 UTC (permalink / raw) To: Sam Lang; +Cc: Cesar Mello, ceph-devel@vger.kernel.org Have you tried rest-bench on localhost at the rados gateway? I was playing with the rados gateway in a VM the other day, and was getting up to 400/s on 4k objects. Above that I was getting connection failures, but I think it was just due to a default max connections setting somewhere or something. My VM is on SSD though. I was just thinking it may help isolate the issue. On Fri, Jan 25, 2013 at 4:14 PM, Sam Lang <sam.lang@inktank.com> wrote: > > On Thu, Jan 24, 2013 at 9:27 AM, Cesar Mello <cmello@gmail.com> wrote: > > Hi! > > > > I have successfully prototyped read/write access to ceph from Windows > > using the S3 API, thanks so much for the help. > > > > Now I would like to do some prototypes targeting performance > > evaluation. My scenario typically requires parallel storage of data > > from tens of thousands of loggers, but scalability to hundreds of > > thousands is the main reason for investigating ceph. > > > > My tests using a single laptop running ceph with 2 local OSDs and > > local radosgw allows writing in average 2.5 small objects per second > > (100 objects in 40 seconds). Is this the expected performance? It > > seems to be I/O bound because the HDD led keeps on during the > > PutObject requests. Any suggestion or documentation pointers for > > profiling are very appreciated. > > Hi Mello, > > 2.5 objects/sec seems terribly slow, even on your laptop. How "small" > are these objects? You might try to benchmark without the disk as a > potential bottleneck, by putting your osd data and journals in /tmp > (for benchmarking only of course) or create/mount a tmpfs and point > your osd backends there. > > > > > I am afraid the S3 API is not good for my scenario, because there is > > no way to append data to existing objects (so I won't be able to model > > a single object for each data collector). If this is the case, then I > > would need to store billions of small objects. I would like to know > > how much disk space each object instance requires other than the > > object content length. > > > > If the S3 API is not well suited to my scenario, then my effort should > > be better directed to porting or writing a native ceph client for > > Windows. I just need an API to read and write/append blocks to files. > > Any comments are really appreciated. > > Hopefully someone with more windows experience will give you better > info/advice than I can. > > You could try to port the rados API to windows. Its purely userspace, > but does rely on pthreads and other libc/gcc specifics. With > something like cygwin a port might not be too hard though. If you > decide to go that route, let us know how you progress! > > -sam > > > > > > Thank you a lot for the attention! > > > > Best regards > > Mello > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <CALFpzo4u2PspjkvSig3LL821YY_XA2iHBjAh30JxgwWTFGsgcA@mail.gmail.com>]
* Re: RadosGW performance and disk space usage [not found] ` <CALFpzo4u2PspjkvSig3LL821YY_XA2iHBjAh30JxgwWTFGsgcA@mail.gmail.com> @ 2013-01-26 14:20 ` Cesar Mello 2013-01-27 23:10 ` Cesar Mello 0 siblings, 1 reply; 11+ messages in thread From: Cesar Mello @ 2013-01-26 14:20 UTC (permalink / raw) To: Marcus Sorensen; +Cc: Sam Lang, ceph-devel@vger.kernel.org Dear Sam, Dan and Marcus, Thank you a lot for the replies. I'll do more tests today. The length of each object used in my test is just 20 bytes. I'm glad you got 400 objects/s! Iif I get that with a length of 8 KB using a 2-node cluster, then ceph with rados will be already faster than my current solution. And then I will be able to present it to my boss. :-) I'll try rest-bench later. Thanks for the help! Best regards Mello On Sat, Jan 26, 2013 at 3:43 AM, Marcus Sorensen <shadowsor@gmail.com> wrote: > Have you tried rest-bench on localhost at the rados gateway? I was playing > with the rados gateway in a VM the other day, and was getting up to 400/s on > 4k objects. Above that I was getting connection failures, but I think it was > just due to a default max connections setting somewhere or something. My VM > is on SSD though. I was just thinking it may help isolate the issue. > > > On Fri, Jan 25, 2013 at 4:14 PM, Sam Lang <sam.lang@inktank.com> wrote: >> >> On Thu, Jan 24, 2013 at 9:27 AM, Cesar Mello <cmello@gmail.com> wrote: >> > Hi! >> > >> > I have successfully prototyped read/write access to ceph from Windows >> > using the S3 API, thanks so much for the help. >> > >> > Now I would like to do some prototypes targeting performance >> > evaluation. My scenario typically requires parallel storage of data >> > from tens of thousands of loggers, but scalability to hundreds of >> > thousands is the main reason for investigating ceph. >> > >> > My tests using a single laptop running ceph with 2 local OSDs and >> > local radosgw allows writing in average 2.5 small objects per second >> > (100 objects in 40 seconds). Is this the expected performance? It >> > seems to be I/O bound because the HDD led keeps on during the >> > PutObject requests. Any suggestion or documentation pointers for >> > profiling are very appreciated. >> >> Hi Mello, >> >> 2.5 objects/sec seems terribly slow, even on your laptop. How "small" >> are these objects? You might try to benchmark without the disk as a >> potential bottleneck, by putting your osd data and journals in /tmp >> (for benchmarking only of course) or create/mount a tmpfs and point >> your osd backends there. >> >> > >> > I am afraid the S3 API is not good for my scenario, because there is >> > no way to append data to existing objects (so I won't be able to model >> > a single object for each data collector). If this is the case, then I >> > would need to store billions of small objects. I would like to know >> > how much disk space each object instance requires other than the >> > object content length. >> > >> > If the S3 API is not well suited to my scenario, then my effort should >> > be better directed to porting or writing a native ceph client for >> > Windows. I just need an API to read and write/append blocks to files. >> > Any comments are really appreciated. >> >> Hopefully someone with more windows experience will give you better >> info/advice than I can. >> >> You could try to port the rados API to windows. Its purely userspace, >> but does rely on pthreads and other libc/gcc specifics. With >> something like cygwin a port might not be too hard though. If you >> decide to go that route, let us know how you progress! >> >> -sam >> >> >> > >> > Thank you a lot for the attention! >> > >> > Best regards >> > Mello >> > -- >> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> > the body of a message to majordomo@vger.kernel.org >> > More majordomo info at http://vger.kernel.org/majordomo-info.html >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RadosGW performance and disk space usage 2013-01-26 14:20 ` Cesar Mello @ 2013-01-27 23:10 ` Cesar Mello 2013-01-28 14:28 ` Joao Eduardo Luis 2013-01-28 17:35 ` Yehuda Sadeh 0 siblings, 2 replies; 11+ messages in thread From: Cesar Mello @ 2013-01-27 23:10 UTC (permalink / raw) To: Marcus Sorensen; +Cc: Sam Lang, ceph-devel@vger.kernel.org Hi, Just tried rest-bench. This little tool is wonderful, thanks! I still have to learn lots of things. So please don't spend much time explaining me, but instead please give me any pointers to documentation or source code that can be useful. As a curiosity, I'm pasting the results from my laptop. I'll repeat the same tests using my desktop as the server. Notice there is an assert being triggered, so I guess I'm running a build with debugging code ?!. I compiled using ./configure --with-radosgw --with-rest-bench followed by make. Thanks a lot for the attention. Best regards! Mello root@l3:/etc/init.d# rest-bench --api-host=localhost --bucket=test --access-key=JJABVJ3AWBS1ZOCML7NS --secret=A+ecBz2+Sdxj4Y8Mo+u3akIewGvJPkwOhwRgPKkX --protocol=http --uri_style=path write host=localhost Maintaining 16 concurrent writes of 4194304 bytes for at least 60 seconds. Object prefix: benchmark_data_l3_4032 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 0 3 3 0 0 0 - 0 1 16 16 0 0 0 - 0 2 16 16 0 0 0 - 0 3 16 16 0 0 0 - 0 4 16 16 0 0 0 - 0 5 16 16 0 0 0 - 0 6 16 16 0 0 0 - 0 7 16 16 0 0 0 - 0 8 16 16 0 0 0 - 0 9 16 16 0 0 0 - 0 10 16 16 0 0 0 - 0 11 16 16 0 0 0 - 0 12 16 17 1 0.333265 0.333333 11.2761 11.2761 13 16 18 2 0.615257 4 12.5964 11.9363 14 16 20 4 1.14262 8 13.1392 12.5365 15 16 23 7 1.86628 12 14.2273 13.2594 16 16 27 11 2.74944 16 15.0222 13.8968 17 16 32 16 3.76394 20 16.2604 14.6301 18 16 32 16 3.55483 0 - 14.6301 19 16 34 18 3.7887 4 6.2274 13.7695 2013-01-27 20:49:29.703509min lat: 6.2274 max lat: 16.2604 avg lat: 13.7695 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 20 16 34 18 3.59927 0 - 13.7695 21 16 34 18 3.42787 0 - 13.7695 22 16 34 18 3.27205 0 - 13.7695 23 16 35 19 3.30367 1 9.09053 13.5233 24 16 36 20 3.33264 4 9.09667 13.302 25 16 36 20 3.19933 0 - 13.302 26 16 36 20 3.07628 0 - 13.302 27 16 37 21 3.11047 1.33333 11.2459 13.204 28 16 37 21 2.99938 0 - 13.204 29 16 37 21 2.89595 0 - 13.204 30 16 37 21 2.79942 0 - 13.204 31 16 37 21 2.70912 0 - 13.204 32 16 39 23 2.87441 1.6 14.9981 13.3602 33 16 39 23 2.7873 0 - 13.3602 34 16 39 23 2.70533 0 - 13.3602 35 16 40 24 2.74229 1.33333 21.5365 13.7009 36 16 40 24 2.66612 0 - 13.7009 37 16 42 26 2.81023 4 22.6059 14.3855 38 16 42 26 2.73628 0 - 14.3855 39 16 45 29 2.97374 6 23.2615 15.3025 2013-01-27 20:49:49.707740min lat: 6.2274 max lat: 23.4496 avg lat: 16.1307 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 40 16 51 35 3.49927 24 21.0123 16.1307 41 16 51 35 3.41392 0 - 16.1307 42 16 52 36 3.42786 2 19.0243 16.211 43 16 52 36 3.34814 0 - 16.211 44 16 52 36 3.27204 0 - 16.211 45 16 52 36 3.19933 0 - 16.211 46 16 53 37 3.21672 1 11.0793 16.0723 47 16 53 37 3.14828 0 - 16.0723 48 16 53 37 3.08269 0 - 16.0723 49 16 53 37 3.01978 0 - 16.0723 50 16 55 39 3.11935 2 23.4253 16.5245 51 16 55 39 3.05819 0 - 16.5245 52 16 58 42 3.2301 6 13.3052 16.2945 53 16 58 42 3.16915 0 - 16.2945 54 16 60 44 3.25858 4 14.2116 16.1999 55 16 60 44 3.19933 0 - 16.1999 56 16 63 47 3.35644 6 15.6276 16.3393 57 16 63 47 3.29756 0 - 16.3393 58 16 66 50 3.44755 6 18.0547 16.5454 59 16 66 50 3.38912 0 - 16.5454 2013-01-27 20:50:09.712027min lat: 6.2274 max lat: 26.3519 avg lat: 16.5454 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 60 16 66 50 3.33263 0 - 16.5454 61 16 66 50 3.278 0 - 16.5454 62 16 66 50 3.22513 0 - 16.5454 63 16 67 51 3.23741 0.8 10.9553 16.4358 64 16 67 51 3.18683 0 - 16.4358 65 16 67 51 3.1378 0 - 16.4358 66 16 67 51 3.09025 0 - 16.4358 67 16 67 51 3.04413 0 - 16.4358 68 16 67 51 2.99937 0 - 16.4358 69 16 67 51 2.9559 0 - 16.4358 70 16 67 51 2.91367 0 - 16.4358 71 14 67 53 2.98528 1 15.1841 16.2524 72 9 67 58 3.22154 20 26.2114 16.6243 Total time run: 72.403588 Total writes made: 67 Write size: 4194304 Bandwidth (MB/sec): 3.701 Stddev Bandwidth: 4.92554 Max bandwidth (MB/sec): 24 Min bandwidth (MB/sec): 0 Average Latency: 17.1886 Stddev Latency: 5.48181 Max latency: 32.7038 Min latency: 6.2274 common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()' thread 7f1211401780 time 2013-01-27 20:51:01.196525 common/WorkQueue.cc: 59: FAILED assert(_threads.empty()) ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f) 1: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c] 2: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021] 3: (main()+0x75b) [0x42521b] 4: (__libc_start_main()+0xed) [0x7f120f37576d] 5: rest-bench() [0x426079] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 2013-01-27 20:51:01.197253 7f1211401780 -1 common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()' thread 7f1211401780 time 2013-01-27 20:51:01.196525 common/WorkQueue.cc: 59: FAILED assert(_threads.empty()) ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f) 1: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c] 2: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021] 3: (main()+0x75b) [0x42521b] 4: (__libc_start_main()+0xed) [0x7f120f37576d] 5: rest-bench() [0x426079] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- -11> 2013-01-27 20:49:09.292227 7f1211401780 5 asok(0x29dc270) register_command perfcounters_dump hook 0x29dc440 -10> 2013-01-27 20:49:09.292259 7f1211401780 5 asok(0x29dc270) register_command 1 hook 0x29dc440 -9> 2013-01-27 20:49:09.292262 7f1211401780 5 asok(0x29dc270) register_command perf dump hook 0x29dc440 -8> 2013-01-27 20:49:09.292271 7f1211401780 5 asok(0x29dc270) register_command perfcounters_schema hook 0x29dc440 -7> 2013-01-27 20:49:09.292275 7f1211401780 5 asok(0x29dc270) register_command 2 hook 0x29dc440 -6> 2013-01-27 20:49:09.292278 7f1211401780 5 asok(0x29dc270) register_command perf schema hook 0x29dc440 -5> 2013-01-27 20:49:09.292285 7f1211401780 5 asok(0x29dc270) register_command config show hook 0x29dc440 -4> 2013-01-27 20:49:09.292290 7f1211401780 5 asok(0x29dc270) register_command config set hook 0x29dc440 -3> 2013-01-27 20:49:09.292292 7f1211401780 5 asok(0x29dc270) register_command log flush hook 0x29dc440 -2> 2013-01-27 20:49:09.292296 7f1211401780 5 asok(0x29dc270) register_command log dump hook 0x29dc440 -1> 2013-01-27 20:49:09.292300 7f1211401780 5 asok(0x29dc270) register_command log reopen hook 0x29dc440 0> 2013-01-27 20:51:01.197253 7f1211401780 -1 common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()' thread 7f1211401780 time 2013-01-27 20:51:01.196525 common/WorkQueue.cc: 59: FAILED assert(_threads.empty()) ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f) 1: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c] 2: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021] 3: (main()+0x75b) [0x42521b] 4: (__libc_start_main()+0xed) [0x7f120f37576d] 5: rest-bench() [0x426079] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 0/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 hadoop 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) 99/99 (stderr threshold) max_recent 100000 max_new 1000 log_file --- end dump of recent events --- terminate called after throwing an instance of 'ceph::FailedAssertion' *** Caught signal (Aborted) ** in thread 7f1211401780 ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f) 1: rest-bench() [0x436b40] 2: (()+0xfcb0) [0x7f1210ffccb0] 3: (gsignal()+0x35) [0x7f120f38a425] 4: (abort()+0x17b) [0x7f120f38db8b] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f120fc85e2d] 6: (()+0x5ef26) [0x7f120fc83f26] 7: (()+0x5ef53) [0x7f120fc83f53] 8: (()+0x5f17e) [0x7f120fc8417e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x43eced] 10: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c] 11: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021] 12: (main()+0x75b) [0x42521b] 13: (__libc_start_main()+0xed) [0x7f120f37576d] 14: rest-bench() [0x426079] 2013-01-27 20:51:01.200578 7f1211401780 -1 *** Caught signal (Aborted) ** in thread 7f1211401780 ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f) 1: rest-bench() [0x436b40] 2: (()+0xfcb0) [0x7f1210ffccb0] 3: (gsignal()+0x35) [0x7f120f38a425] 4: (abort()+0x17b) [0x7f120f38db8b] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f120fc85e2d] 6: (()+0x5ef26) [0x7f120fc83f26] 7: (()+0x5ef53) [0x7f120fc83f53] 8: (()+0x5f17e) [0x7f120fc8417e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x43eced] 10: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c] 11: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021] 12: (main()+0x75b) [0x42521b] 13: (__libc_start_main()+0xed) [0x7f120f37576d] 14: rest-bench() [0x426079] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- begin dump of recent events --- 0> 2013-01-27 20:51:01.200578 7f1211401780 -1 *** Caught signal (Aborted) ** in thread 7f1211401780 ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f) 1: rest-bench() [0x436b40] 2: (()+0xfcb0) [0x7f1210ffccb0] 3: (gsignal()+0x35) [0x7f120f38a425] 4: (abort()+0x17b) [0x7f120f38db8b] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f120fc85e2d] 6: (()+0x5ef26) [0x7f120fc83f26] 7: (()+0x5ef53) [0x7f120fc83f53] 8: (()+0x5f17e) [0x7f120fc8417e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x43eced] 10: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c] 11: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021] 12: (main()+0x75b) [0x42521b] 13: (__libc_start_main()+0xed) [0x7f120f37576d] 14: rest-bench() [0x426079] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 0/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 hadoop 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) 99/99 (stderr threshold) max_recent 100000 max_new 1000 log_file --- end dump of recent events --- Aborted (core dumped) root@l3:/etc/init.d# On Sat, Jan 26, 2013 at 12:20 PM, Cesar Mello <cmello@gmail.com> wrote: > Dear Sam, Dan and Marcus, > > Thank you a lot for the replies. I'll do more tests today. > > The length of each object used in my test is just 20 bytes. I'm glad > you got 400 objects/s! Iif I get that with a length of 8 KB using a > 2-node cluster, then ceph with rados will be already faster than my > current solution. And then I will be able to present it to my boss. > :-) > > I'll try rest-bench later. Thanks for the help! > > Best regards > Mello > > On Sat, Jan 26, 2013 at 3:43 AM, Marcus Sorensen <shadowsor@gmail.com> wrote: >> Have you tried rest-bench on localhost at the rados gateway? I was playing >> with the rados gateway in a VM the other day, and was getting up to 400/s on >> 4k objects. Above that I was getting connection failures, but I think it was >> just due to a default max connections setting somewhere or something. My VM >> is on SSD though. I was just thinking it may help isolate the issue. >> >> >> On Fri, Jan 25, 2013 at 4:14 PM, Sam Lang <sam.lang@inktank.com> wrote: >>> >>> On Thu, Jan 24, 2013 at 9:27 AM, Cesar Mello <cmello@gmail.com> wrote: >>> > Hi! >>> > >>> > I have successfully prototyped read/write access to ceph from Windows >>> > using the S3 API, thanks so much for the help. >>> > >>> > Now I would like to do some prototypes targeting performance >>> > evaluation. My scenario typically requires parallel storage of data >>> > from tens of thousands of loggers, but scalability to hundreds of >>> > thousands is the main reason for investigating ceph. >>> > >>> > My tests using a single laptop running ceph with 2 local OSDs and >>> > local radosgw allows writing in average 2.5 small objects per second >>> > (100 objects in 40 seconds). Is this the expected performance? It >>> > seems to be I/O bound because the HDD led keeps on during the >>> > PutObject requests. Any suggestion or documentation pointers for >>> > profiling are very appreciated. >>> >>> Hi Mello, >>> >>> 2.5 objects/sec seems terribly slow, even on your laptop. How "small" >>> are these objects? You might try to benchmark without the disk as a >>> potential bottleneck, by putting your osd data and journals in /tmp >>> (for benchmarking only of course) or create/mount a tmpfs and point >>> your osd backends there. >>> >>> > >>> > I am afraid the S3 API is not good for my scenario, because there is >>> > no way to append data to existing objects (so I won't be able to model >>> > a single object for each data collector). If this is the case, then I >>> > would need to store billions of small objects. I would like to know >>> > how much disk space each object instance requires other than the >>> > object content length. >>> > >>> > If the S3 API is not well suited to my scenario, then my effort should >>> > be better directed to porting or writing a native ceph client for >>> > Windows. I just need an API to read and write/append blocks to files. >>> > Any comments are really appreciated. >>> >>> Hopefully someone with more windows experience will give you better >>> info/advice than I can. >>> >>> You could try to port the rados API to windows. Its purely userspace, >>> but does rely on pthreads and other libc/gcc specifics. With >>> something like cygwin a port might not be too hard though. If you >>> decide to go that route, let us know how you progress! >>> >>> -sam >>> >>> >>> > >>> > Thank you a lot for the attention! >>> > >>> > Best regards >>> > Mello >>> > -- >>> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>> > the body of a message to majordomo@vger.kernel.org >>> > More majordomo info at http://vger.kernel.org/majordomo-info.html >>> -- >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> >> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RadosGW performance and disk space usage 2013-01-27 23:10 ` Cesar Mello @ 2013-01-28 14:28 ` Joao Eduardo Luis 2013-01-28 17:33 ` Yehuda Sadeh 2013-01-28 17:35 ` Yehuda Sadeh 1 sibling, 1 reply; 11+ messages in thread From: Joao Eduardo Luis @ 2013-01-28 14:28 UTC (permalink / raw) To: Cesar Mello; +Cc: Marcus Sorensen, Sam Lang, ceph-devel@vger.kernel.org, yehuda On 01/27/2013 11:10 PM, Cesar Mello wrote: > Hi, > > Just tried rest-bench. This little tool is wonderful, thanks! > > I still have to learn lots of things. So please don't spend much time > explaining me, but instead please give me any pointers to > documentation or source code that can be useful. As a curiosity, I'm > pasting the results from my laptop. I'll repeat the same tests using > my desktop as the server. > > Notice there is an assert being triggered, so I guess I'm running a > build with debugging code ?!. I compiled using ./configure > --with-radosgw --with-rest-bench followed by make. asserts are usually used to mark invariants on the code logic, and are always built, regardless debugging being enabled or disabled. Given you are hitting one, it probably means something is not quite right (might be a bug, or some invariant was broken for some reason). > common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()' > thread 7f1211401780 time 2013-01-27 20:51:01.196525 > common/WorkQueue.cc: 59: FAILED assert(_threads.empty()) > ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f) > 1: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c] > 2: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021] > 3: (main()+0x75b) [0x42521b] > 4: (__libc_start_main()+0xed) [0x7f120f37576d] > 5: rest-bench() [0x426079] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. Looks like http://tracker.newdream.net/issues/3896 Am not sure who should be made aware of this issue though. Maybe Yehuda (cc'ing)? -Joao > 2013-01-27 20:51:01.197253 7f1211401780 -1 common/WorkQueue.cc: In > function 'virtual ThreadPool::~ThreadPool()' thread 7f1211401780 time > 2013-01-27 20:51:01.196525 > common/WorkQueue.cc: 59: FAILED assert(_threads.empty()) > > ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f) > 1: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c] > 2: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021] > 3: (main()+0x75b) [0x42521b] > 4: (__libc_start_main()+0xed) [0x7f120f37576d] > 5: rest-bench() [0x426079] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > > --- begin dump of recent events --- > -11> 2013-01-27 20:49:09.292227 7f1211401780 5 asok(0x29dc270) > register_command perfcounters_dump hook 0x29dc440 > -10> 2013-01-27 20:49:09.292259 7f1211401780 5 asok(0x29dc270) > register_command 1 hook 0x29dc440 > -9> 2013-01-27 20:49:09.292262 7f1211401780 5 asok(0x29dc270) > register_command perf dump hook 0x29dc440 > -8> 2013-01-27 20:49:09.292271 7f1211401780 5 asok(0x29dc270) > register_command perfcounters_schema hook 0x29dc440 > -7> 2013-01-27 20:49:09.292275 7f1211401780 5 asok(0x29dc270) > register_command 2 hook 0x29dc440 > -6> 2013-01-27 20:49:09.292278 7f1211401780 5 asok(0x29dc270) > register_command perf schema hook 0x29dc440 > -5> 2013-01-27 20:49:09.292285 7f1211401780 5 asok(0x29dc270) > register_command config show hook 0x29dc440 > -4> 2013-01-27 20:49:09.292290 7f1211401780 5 asok(0x29dc270) > register_command config set hook 0x29dc440 > -3> 2013-01-27 20:49:09.292292 7f1211401780 5 asok(0x29dc270) > register_command log flush hook 0x29dc440 > -2> 2013-01-27 20:49:09.292296 7f1211401780 5 asok(0x29dc270) > register_command log dump hook 0x29dc440 > -1> 2013-01-27 20:49:09.292300 7f1211401780 5 asok(0x29dc270) > register_command log reopen hook 0x29dc440 > 0> 2013-01-27 20:51:01.197253 7f1211401780 -1 > common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()' > thread 7f1211401780 time 2013-01-27 20:51:01.196525 > common/WorkQueue.cc: 59: FAILED assert(_threads.empty()) > > ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f) > 1: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c] > 2: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021] > 3: (main()+0x75b) [0x42521b] > 4: (__libc_start_main()+0xed) [0x7f120f37576d] > 5: rest-bench() [0x426079] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > > --- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 0/ 5 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 journal > 0/ 5 ms > 1/ 5 mon > 0/10 monc > 0/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/ 5 hadoop > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > -2/-2 (syslog threshold) > 99/99 (stderr threshold) > max_recent 100000 > max_new 1000 > log_file > --- end dump of recent events --- > terminate called after throwing an instance of 'ceph::FailedAssertion' > *** Caught signal (Aborted) ** > in thread 7f1211401780 > ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f) > 1: rest-bench() [0x436b40] > 2: (()+0xfcb0) [0x7f1210ffccb0] > 3: (gsignal()+0x35) [0x7f120f38a425] > 4: (abort()+0x17b) [0x7f120f38db8b] > 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f120fc85e2d] > 6: (()+0x5ef26) [0x7f120fc83f26] > 7: (()+0x5ef53) [0x7f120fc83f53] > 8: (()+0x5f17e) [0x7f120fc8417e] > 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x43d) [0x43eced] > 10: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c] > 11: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021] > 12: (main()+0x75b) [0x42521b] > 13: (__libc_start_main()+0xed) [0x7f120f37576d] > 14: rest-bench() [0x426079] > 2013-01-27 20:51:01.200578 7f1211401780 -1 *** Caught signal (Aborted) ** > in thread 7f1211401780 > > ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f) > 1: rest-bench() [0x436b40] > 2: (()+0xfcb0) [0x7f1210ffccb0] > 3: (gsignal()+0x35) [0x7f120f38a425] > 4: (abort()+0x17b) [0x7f120f38db8b] > 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f120fc85e2d] > 6: (()+0x5ef26) [0x7f120fc83f26] > 7: (()+0x5ef53) [0x7f120fc83f53] > 8: (()+0x5f17e) [0x7f120fc8417e] > 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x43d) [0x43eced] > 10: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c] > 11: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021] > 12: (main()+0x75b) [0x42521b] > 13: (__libc_start_main()+0xed) [0x7f120f37576d] > 14: rest-bench() [0x426079] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > > --- begin dump of recent events --- > 0> 2013-01-27 20:51:01.200578 7f1211401780 -1 *** Caught signal > (Aborted) ** > in thread 7f1211401780 > > ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f) > 1: rest-bench() [0x436b40] > 2: (()+0xfcb0) [0x7f1210ffccb0] > 3: (gsignal()+0x35) [0x7f120f38a425] > 4: (abort()+0x17b) [0x7f120f38db8b] > 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f120fc85e2d] > 6: (()+0x5ef26) [0x7f120fc83f26] > 7: (()+0x5ef53) [0x7f120fc83f53] > 8: (()+0x5f17e) [0x7f120fc8417e] > 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x43d) [0x43eced] > 10: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c] > 11: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021] > 12: (main()+0x75b) [0x42521b] > 13: (__libc_start_main()+0xed) [0x7f120f37576d] > 14: rest-bench() [0x426079] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is > needed to interpret this. > > --- logging levels --- > 0/ 5 none > 0/ 1 lockdep > 0/ 1 context > 1/ 1 crush > 1/ 5 mds > 1/ 5 mds_balancer > 1/ 5 mds_locker > 1/ 5 mds_log > 1/ 5 mds_log_expire > 1/ 5 mds_migrator > 0/ 1 buffer > 0/ 1 timer > 0/ 1 filer > 0/ 1 striper > 0/ 1 objecter > 0/ 5 rados > 0/ 5 rbd > 0/ 5 journaler > 0/ 5 objectcacher > 0/ 5 client > 0/ 5 osd > 0/ 5 optracker > 0/ 5 objclass > 1/ 3 filestore > 1/ 3 journal > 0/ 5 ms > 1/ 5 mon > 0/10 monc > 0/ 5 paxos > 0/ 5 tp > 1/ 5 auth > 1/ 5 crypto > 1/ 1 finisher > 1/ 5 heartbeatmap > 1/ 5 perfcounter > 1/ 5 rgw > 1/ 5 hadoop > 1/ 5 javaclient > 1/ 5 asok > 1/ 1 throttle > -2/-2 (syslog threshold) > 99/99 (stderr threshold) > max_recent 100000 > max_new 1000 > log_file > --- end dump of recent events --- > Aborted (core dumped) > root@l3:/etc/init.d# > > On Sat, Jan 26, 2013 at 12:20 PM, Cesar Mello <cmello@gmail.com> wrote: >> Dear Sam, Dan and Marcus, >> >> Thank you a lot for the replies. I'll do more tests today. >> >> The length of each object used in my test is just 20 bytes. I'm glad >> you got 400 objects/s! Iif I get that with a length of 8 KB using a >> 2-node cluster, then ceph with rados will be already faster than my >> current solution. And then I will be able to present it to my boss. >> :-) >> >> I'll try rest-bench later. Thanks for the help! >> >> Best regards >> Mello >> >> On Sat, Jan 26, 2013 at 3:43 AM, Marcus Sorensen <shadowsor@gmail.com> wrote: >>> Have you tried rest-bench on localhost at the rados gateway? I was playing >>> with the rados gateway in a VM the other day, and was getting up to 400/s on >>> 4k objects. Above that I was getting connection failures, but I think it was >>> just due to a default max connections setting somewhere or something. My VM >>> is on SSD though. I was just thinking it may help isolate the issue. >>> >>> >>> On Fri, Jan 25, 2013 at 4:14 PM, Sam Lang <sam.lang@inktank.com> wrote: >>>> >>>> On Thu, Jan 24, 2013 at 9:27 AM, Cesar Mello <cmello@gmail.com> wrote: >>>>> Hi! >>>>> >>>>> I have successfully prototyped read/write access to ceph from Windows >>>>> using the S3 API, thanks so much for the help. >>>>> >>>>> Now I would like to do some prototypes targeting performance >>>>> evaluation. My scenario typically requires parallel storage of data >>>>> from tens of thousands of loggers, but scalability to hundreds of >>>>> thousands is the main reason for investigating ceph. >>>>> >>>>> My tests using a single laptop running ceph with 2 local OSDs and >>>>> local radosgw allows writing in average 2.5 small objects per second >>>>> (100 objects in 40 seconds). Is this the expected performance? It >>>>> seems to be I/O bound because the HDD led keeps on during the >>>>> PutObject requests. Any suggestion or documentation pointers for >>>>> profiling are very appreciated. >>>> >>>> Hi Mello, >>>> >>>> 2.5 objects/sec seems terribly slow, even on your laptop. How "small" >>>> are these objects? You might try to benchmark without the disk as a >>>> potential bottleneck, by putting your osd data and journals in /tmp >>>> (for benchmarking only of course) or create/mount a tmpfs and point >>>> your osd backends there. >>>> >>>>> >>>>> I am afraid the S3 API is not good for my scenario, because there is >>>>> no way to append data to existing objects (so I won't be able to model >>>>> a single object for each data collector). If this is the case, then I >>>>> would need to store billions of small objects. I would like to know >>>>> how much disk space each object instance requires other than the >>>>> object content length. >>>>> >>>>> If the S3 API is not well suited to my scenario, then my effort should >>>>> be better directed to porting or writing a native ceph client for >>>>> Windows. I just need an API to read and write/append blocks to files. >>>>> Any comments are really appreciated. >>>> >>>> Hopefully someone with more windows experience will give you better >>>> info/advice than I can. >>>> >>>> You could try to port the rados API to windows. Its purely userspace, >>>> but does rely on pthreads and other libc/gcc specifics. With >>>> something like cygwin a port might not be too hard though. If you >>>> decide to go that route, let us know how you progress! >>>> >>>> -sam >>>> >>>> >>>>> >>>>> Thank you a lot for the attention! >>>>> >>>>> Best regards >>>>> Mello >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>>> the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>> the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >>> > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RadosGW performance and disk space usage 2013-01-28 14:28 ` Joao Eduardo Luis @ 2013-01-28 17:33 ` Yehuda Sadeh 0 siblings, 0 replies; 11+ messages in thread From: Yehuda Sadeh @ 2013-01-28 17:33 UTC (permalink / raw) To: Joao Eduardo Luis Cc: Cesar Mello, Marcus Sorensen, Sam Lang, ceph-devel@vger.kernel.org On Mon, Jan 28, 2013 at 6:28 AM, Joao Eduardo Luis <joao.luis@inktank.com> wrote: > On 01/27/2013 11:10 PM, Cesar Mello wrote: >> >> Hi, >> >> Just tried rest-bench. This little tool is wonderful, thanks! >> >> I still have to learn lots of things. So please don't spend much time >> explaining me, but instead please give me any pointers to >> documentation or source code that can be useful. As a curiosity, I'm >> pasting the results from my laptop. I'll repeat the same tests using >> my desktop as the server. >> >> Notice there is an assert being triggered, so I guess I'm running a >> build with debugging code ?!. I compiled using ./configure >> --with-radosgw --with-rest-bench followed by make. > > > asserts are usually used to mark invariants on the code logic, and are > always built, regardless debugging being enabled or disabled. Given you are > hitting one, it probably means something is not quite right (might be a bug, > or some invariant was broken for some reason). > > >> common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()' >> thread 7f1211401780 time 2013-01-27 20:51:01.196525 >> common/WorkQueue.cc: 59: FAILED assert(_threads.empty()) >> ceph version 0.56-464-gfa421cf >> (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f) >> 1: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c] >> 2: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021] >> 3: (main()+0x75b) [0x42521b] >> 4: (__libc_start_main()+0xed) [0x7f120f37576d] >> 5: rest-bench() [0x426079] >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is >> needed to interpret this. > > > Looks like http://tracker.newdream.net/issues/3896 > Right, 3896. Probably some cleanup before shutdown issues. Yehuda ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RadosGW performance and disk space usage 2013-01-27 23:10 ` Cesar Mello 2013-01-28 14:28 ` Joao Eduardo Luis @ 2013-01-28 17:35 ` Yehuda Sadeh 2013-01-28 18:18 ` Cesar Mello 1 sibling, 1 reply; 11+ messages in thread From: Yehuda Sadeh @ 2013-01-28 17:35 UTC (permalink / raw) To: Cesar Mello; +Cc: Marcus Sorensen, Sam Lang, ceph-devel@vger.kernel.org On Sun, Jan 27, 2013 at 3:10 PM, Cesar Mello <cmello@gmail.com> wrote: > Hi, > > Just tried rest-bench. This little tool is wonderful, thanks! > > I still have to learn lots of things. So please don't spend much time > explaining me, but instead please give me any pointers to > documentation or source code that can be useful. As a curiosity, I'm > pasting the results from my laptop. I'll repeat the same tests using > my desktop as the server. > > Notice there is an assert being triggered, so I guess I'm running a > build with debugging code ?!. I compiled using ./configure > --with-radosgw --with-rest-bench followed by make. > > Thanks a lot for the attention. > > Best regards! > Mello > > root@l3:/etc/init.d# rest-bench --api-host=localhost --bucket=test > --access-key=JJABVJ3AWBS1ZOCML7NS > --secret=A+ecBz2+Sdxj4Y8Mo+u3akIewGvJPkwOhwRgPKkX --protocol=http > --uri_style=path write > host=localhost > Maintaining 16 concurrent writes of 4194304 bytes for at least 60 seconds. > Object prefix: benchmark_data_l3_4032 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 0 3 3 0 0 0 - 0 > 1 16 16 0 0 0 - 0 > 2 16 16 0 0 0 - 0 > 3 16 16 0 0 0 - 0 > 4 16 16 0 0 0 - 0 > 5 16 16 0 0 0 - 0 > 6 16 16 0 0 0 - 0 > 7 16 16 0 0 0 - 0 > 8 16 16 0 0 0 - 0 > 9 16 16 0 0 0 - 0 > 10 16 16 0 0 0 - 0 > 11 16 16 0 0 0 - 0 > 12 16 17 1 0.333265 0.333333 11.2761 11.2761 > 13 16 18 2 0.615257 4 12.5964 11.9363 > 14 16 20 4 1.14262 8 13.1392 12.5365 > 15 16 23 7 1.86628 12 14.2273 13.2594 > 16 16 27 11 2.74944 16 15.0222 13.8968 > 17 16 32 16 3.76394 20 16.2604 14.6301 > 18 16 32 16 3.55483 0 - 14.6301 > 19 16 34 18 3.7887 4 6.2274 13.7695 > 2013-01-27 20:49:29.703509min lat: 6.2274 max lat: 16.2604 avg lat: 13.7695 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 20 16 34 18 3.59927 0 - 13.7695 > 21 16 34 18 3.42787 0 - 13.7695 > 22 16 34 18 3.27205 0 - 13.7695 > 23 16 35 19 3.30367 1 9.09053 13.5233 > 24 16 36 20 3.33264 4 9.09667 13.302 > 25 16 36 20 3.19933 0 - 13.302 > 26 16 36 20 3.07628 0 - 13.302 > 27 16 37 21 3.11047 1.33333 11.2459 13.204 > 28 16 37 21 2.99938 0 - 13.204 > 29 16 37 21 2.89595 0 - 13.204 > 30 16 37 21 2.79942 0 - 13.204 > 31 16 37 21 2.70912 0 - 13.204 > 32 16 39 23 2.87441 1.6 14.9981 13.3602 > 33 16 39 23 2.7873 0 - 13.3602 > 34 16 39 23 2.70533 0 - 13.3602 > 35 16 40 24 2.74229 1.33333 21.5365 13.7009 > 36 16 40 24 2.66612 0 - 13.7009 > 37 16 42 26 2.81023 4 22.6059 14.3855 > 38 16 42 26 2.73628 0 - 14.3855 > 39 16 45 29 2.97374 6 23.2615 15.3025 > 2013-01-27 20:49:49.707740min lat: 6.2274 max lat: 23.4496 avg lat: 16.1307 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 40 16 51 35 3.49927 24 21.0123 16.1307 > 41 16 51 35 3.41392 0 - 16.1307 > 42 16 52 36 3.42786 2 19.0243 16.211 > 43 16 52 36 3.34814 0 - 16.211 > 44 16 52 36 3.27204 0 - 16.211 > 45 16 52 36 3.19933 0 - 16.211 > 46 16 53 37 3.21672 1 11.0793 16.0723 > 47 16 53 37 3.14828 0 - 16.0723 > 48 16 53 37 3.08269 0 - 16.0723 > 49 16 53 37 3.01978 0 - 16.0723 > 50 16 55 39 3.11935 2 23.4253 16.5245 > 51 16 55 39 3.05819 0 - 16.5245 > 52 16 58 42 3.2301 6 13.3052 16.2945 > 53 16 58 42 3.16915 0 - 16.2945 > 54 16 60 44 3.25858 4 14.2116 16.1999 > 55 16 60 44 3.19933 0 - 16.1999 > 56 16 63 47 3.35644 6 15.6276 16.3393 > 57 16 63 47 3.29756 0 - 16.3393 > 58 16 66 50 3.44755 6 18.0547 16.5454 > 59 16 66 50 3.38912 0 - 16.5454 > 2013-01-27 20:50:09.712027min lat: 6.2274 max lat: 26.3519 avg lat: 16.5454 > sec Cur ops started finished avg MB/s cur MB/s last lat avg lat > 60 16 66 50 3.33263 0 - 16.5454 > 61 16 66 50 3.278 0 - 16.5454 > 62 16 66 50 3.22513 0 - 16.5454 > 63 16 67 51 3.23741 0.8 10.9553 16.4358 > 64 16 67 51 3.18683 0 - 16.4358 > 65 16 67 51 3.1378 0 - 16.4358 > 66 16 67 51 3.09025 0 - 16.4358 > 67 16 67 51 3.04413 0 - 16.4358 > 68 16 67 51 2.99937 0 - 16.4358 > 69 16 67 51 2.9559 0 - 16.4358 > 70 16 67 51 2.91367 0 - 16.4358 > 71 14 67 53 2.98528 1 15.1841 16.2524 > 72 9 67 58 3.22154 20 26.2114 16.6243 > Total time run: 72.403588 > Total writes made: 67 > Write size: 4194304 > Bandwidth (MB/sec): 3.701 > > Stddev Bandwidth: 4.92554 > Max bandwidth (MB/sec): 24 > Min bandwidth (MB/sec): 0 > Average Latency: 17.1886 > Stddev Latency: 5.48181 > Max latency: 32.7038 > Min latency: 6.2274 Can you run 'rados bench' on the same setup (with the same params)? The numbers are really horrible. Yehuda ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RadosGW performance and disk space usage 2013-01-28 17:35 ` Yehuda Sadeh @ 2013-01-28 18:18 ` Cesar Mello 0 siblings, 0 replies; 11+ messages in thread From: Cesar Mello @ 2013-01-28 18:18 UTC (permalink / raw) To: Yehuda Sadeh; +Cc: Marcus Sorensen, Sam Lang, ceph-devel@vger.kernel.org Sure I can later when I arrive home. With the end of my vacation, I'll be able to devote a couple of hours after my 3-year-old sleeps. :-) I guess my laptop hard disk has horrible seek times. I'll repeat the tests in my desktop as soon as possible. Thanks a lot for the attention! Best regards Mello On Mon, Jan 28, 2013 at 3:35 PM, Yehuda Sadeh <yehuda@inktank.com> wrote: > On Sun, Jan 27, 2013 at 3:10 PM, Cesar Mello <cmello@gmail.com> wrote: >> Hi, >> >> Just tried rest-bench. This little tool is wonderful, thanks! >> >> I still have to learn lots of things. So please don't spend much time >> explaining me, but instead please give me any pointers to >> documentation or source code that can be useful. As a curiosity, I'm >> pasting the results from my laptop. I'll repeat the same tests using >> my desktop as the server. >> >> Notice there is an assert being triggered, so I guess I'm running a >> build with debugging code ?!. I compiled using ./configure >> --with-radosgw --with-rest-bench followed by make. >> >> Thanks a lot for the attention. >> >> Best regards! >> Mello >> >> root@l3:/etc/init.d# rest-bench --api-host=localhost --bucket=test >> --access-key=JJABVJ3AWBS1ZOCML7NS >> --secret=A+ecBz2+Sdxj4Y8Mo+u3akIewGvJPkwOhwRgPKkX --protocol=http >> --uri_style=path write >> host=localhost >> Maintaining 16 concurrent writes of 4194304 bytes for at least 60 seconds. >> Object prefix: benchmark_data_l3_4032 >> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat >> 0 3 3 0 0 0 - 0 >> 1 16 16 0 0 0 - 0 >> 2 16 16 0 0 0 - 0 >> 3 16 16 0 0 0 - 0 >> 4 16 16 0 0 0 - 0 >> 5 16 16 0 0 0 - 0 >> 6 16 16 0 0 0 - 0 >> 7 16 16 0 0 0 - 0 >> 8 16 16 0 0 0 - 0 >> 9 16 16 0 0 0 - 0 >> 10 16 16 0 0 0 - 0 >> 11 16 16 0 0 0 - 0 >> 12 16 17 1 0.333265 0.333333 11.2761 11.2761 >> 13 16 18 2 0.615257 4 12.5964 11.9363 >> 14 16 20 4 1.14262 8 13.1392 12.5365 >> 15 16 23 7 1.86628 12 14.2273 13.2594 >> 16 16 27 11 2.74944 16 15.0222 13.8968 >> 17 16 32 16 3.76394 20 16.2604 14.6301 >> 18 16 32 16 3.55483 0 - 14.6301 >> 19 16 34 18 3.7887 4 6.2274 13.7695 >> 2013-01-27 20:49:29.703509min lat: 6.2274 max lat: 16.2604 avg lat: 13.7695 >> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat >> 20 16 34 18 3.59927 0 - 13.7695 >> 21 16 34 18 3.42787 0 - 13.7695 >> 22 16 34 18 3.27205 0 - 13.7695 >> 23 16 35 19 3.30367 1 9.09053 13.5233 >> 24 16 36 20 3.33264 4 9.09667 13.302 >> 25 16 36 20 3.19933 0 - 13.302 >> 26 16 36 20 3.07628 0 - 13.302 >> 27 16 37 21 3.11047 1.33333 11.2459 13.204 >> 28 16 37 21 2.99938 0 - 13.204 >> 29 16 37 21 2.89595 0 - 13.204 >> 30 16 37 21 2.79942 0 - 13.204 >> 31 16 37 21 2.70912 0 - 13.204 >> 32 16 39 23 2.87441 1.6 14.9981 13.3602 >> 33 16 39 23 2.7873 0 - 13.3602 >> 34 16 39 23 2.70533 0 - 13.3602 >> 35 16 40 24 2.74229 1.33333 21.5365 13.7009 >> 36 16 40 24 2.66612 0 - 13.7009 >> 37 16 42 26 2.81023 4 22.6059 14.3855 >> 38 16 42 26 2.73628 0 - 14.3855 >> 39 16 45 29 2.97374 6 23.2615 15.3025 >> 2013-01-27 20:49:49.707740min lat: 6.2274 max lat: 23.4496 avg lat: 16.1307 >> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat >> 40 16 51 35 3.49927 24 21.0123 16.1307 >> 41 16 51 35 3.41392 0 - 16.1307 >> 42 16 52 36 3.42786 2 19.0243 16.211 >> 43 16 52 36 3.34814 0 - 16.211 >> 44 16 52 36 3.27204 0 - 16.211 >> 45 16 52 36 3.19933 0 - 16.211 >> 46 16 53 37 3.21672 1 11.0793 16.0723 >> 47 16 53 37 3.14828 0 - 16.0723 >> 48 16 53 37 3.08269 0 - 16.0723 >> 49 16 53 37 3.01978 0 - 16.0723 >> 50 16 55 39 3.11935 2 23.4253 16.5245 >> 51 16 55 39 3.05819 0 - 16.5245 >> 52 16 58 42 3.2301 6 13.3052 16.2945 >> 53 16 58 42 3.16915 0 - 16.2945 >> 54 16 60 44 3.25858 4 14.2116 16.1999 >> 55 16 60 44 3.19933 0 - 16.1999 >> 56 16 63 47 3.35644 6 15.6276 16.3393 >> 57 16 63 47 3.29756 0 - 16.3393 >> 58 16 66 50 3.44755 6 18.0547 16.5454 >> 59 16 66 50 3.38912 0 - 16.5454 >> 2013-01-27 20:50:09.712027min lat: 6.2274 max lat: 26.3519 avg lat: 16.5454 >> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat >> 60 16 66 50 3.33263 0 - 16.5454 >> 61 16 66 50 3.278 0 - 16.5454 >> 62 16 66 50 3.22513 0 - 16.5454 >> 63 16 67 51 3.23741 0.8 10.9553 16.4358 >> 64 16 67 51 3.18683 0 - 16.4358 >> 65 16 67 51 3.1378 0 - 16.4358 >> 66 16 67 51 3.09025 0 - 16.4358 >> 67 16 67 51 3.04413 0 - 16.4358 >> 68 16 67 51 2.99937 0 - 16.4358 >> 69 16 67 51 2.9559 0 - 16.4358 >> 70 16 67 51 2.91367 0 - 16.4358 >> 71 14 67 53 2.98528 1 15.1841 16.2524 >> 72 9 67 58 3.22154 20 26.2114 16.6243 >> Total time run: 72.403588 >> Total writes made: 67 >> Write size: 4194304 >> Bandwidth (MB/sec): 3.701 >> >> Stddev Bandwidth: 4.92554 >> Max bandwidth (MB/sec): 24 >> Min bandwidth (MB/sec): 0 >> Average Latency: 17.1886 >> Stddev Latency: 5.48181 >> Max latency: 32.7038 >> Min latency: 6.2274 > > Can you run 'rados bench' on the same setup (with the same params)? > The numbers are really horrible. > > Yehuda ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2013-01-28 18:18 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-01-24 15:27 RadosGW performance and disk space usage Cesar Mello
2013-01-25 23:14 ` Sam Lang
2013-01-26 5:35 ` Dan Mick
2013-01-28 2:32 ` Dan Mick
2013-01-26 5:44 ` Marcus Sorensen
[not found] ` <CALFpzo4u2PspjkvSig3LL821YY_XA2iHBjAh30JxgwWTFGsgcA@mail.gmail.com>
2013-01-26 14:20 ` Cesar Mello
2013-01-27 23:10 ` Cesar Mello
2013-01-28 14:28 ` Joao Eduardo Luis
2013-01-28 17:33 ` Yehuda Sadeh
2013-01-28 17:35 ` Yehuda Sadeh
2013-01-28 18:18 ` Cesar Mello
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.