* RadosGW performance and disk space usage
@ 2013-01-24 15:27 Cesar Mello
2013-01-25 23:14 ` Sam Lang
0 siblings, 1 reply; 11+ messages in thread
From: Cesar Mello @ 2013-01-24 15:27 UTC (permalink / raw)
To: ceph-devel
Hi!
I have successfully prototyped read/write access to ceph from Windows
using the S3 API, thanks so much for the help.
Now I would like to do some prototypes targeting performance
evaluation. My scenario typically requires parallel storage of data
from tens of thousands of loggers, but scalability to hundreds of
thousands is the main reason for investigating ceph.
My tests using a single laptop running ceph with 2 local OSDs and
local radosgw allows writing in average 2.5 small objects per second
(100 objects in 40 seconds). Is this the expected performance? It
seems to be I/O bound because the HDD led keeps on during the
PutObject requests. Any suggestion or documentation pointers for
profiling are very appreciated.
I am afraid the S3 API is not good for my scenario, because there is
no way to append data to existing objects (so I won't be able to model
a single object for each data collector). If this is the case, then I
would need to store billions of small objects. I would like to know
how much disk space each object instance requires other than the
object content length.
If the S3 API is not well suited to my scenario, then my effort should
be better directed to porting or writing a native ceph client for
Windows. I just need an API to read and write/append blocks to files.
Any comments are really appreciated.
Thank you a lot for the attention!
Best regards
Mello
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RadosGW performance and disk space usage
2013-01-24 15:27 RadosGW performance and disk space usage Cesar Mello
@ 2013-01-25 23:14 ` Sam Lang
2013-01-26 5:35 ` Dan Mick
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: Sam Lang @ 2013-01-25 23:14 UTC (permalink / raw)
To: Cesar Mello; +Cc: ceph-devel@vger.kernel.org
On Thu, Jan 24, 2013 at 9:27 AM, Cesar Mello <cmello@gmail.com> wrote:
> Hi!
>
> I have successfully prototyped read/write access to ceph from Windows
> using the S3 API, thanks so much for the help.
>
> Now I would like to do some prototypes targeting performance
> evaluation. My scenario typically requires parallel storage of data
> from tens of thousands of loggers, but scalability to hundreds of
> thousands is the main reason for investigating ceph.
>
> My tests using a single laptop running ceph with 2 local OSDs and
> local radosgw allows writing in average 2.5 small objects per second
> (100 objects in 40 seconds). Is this the expected performance? It
> seems to be I/O bound because the HDD led keeps on during the
> PutObject requests. Any suggestion or documentation pointers for
> profiling are very appreciated.
Hi Mello,
2.5 objects/sec seems terribly slow, even on your laptop. How "small"
are these objects? You might try to benchmark without the disk as a
potential bottleneck, by putting your osd data and journals in /tmp
(for benchmarking only of course) or create/mount a tmpfs and point
your osd backends there.
>
> I am afraid the S3 API is not good for my scenario, because there is
> no way to append data to existing objects (so I won't be able to model
> a single object for each data collector). If this is the case, then I
> would need to store billions of small objects. I would like to know
> how much disk space each object instance requires other than the
> object content length.
>
> If the S3 API is not well suited to my scenario, then my effort should
> be better directed to porting or writing a native ceph client for
> Windows. I just need an API to read and write/append blocks to files.
> Any comments are really appreciated.
Hopefully someone with more windows experience will give you better
info/advice than I can.
You could try to port the rados API to windows. Its purely userspace,
but does rely on pthreads and other libc/gcc specifics. With
something like cygwin a port might not be too hard though. If you
decide to go that route, let us know how you progress!
-sam
>
> Thank you a lot for the attention!
>
> Best regards
> Mello
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RadosGW performance and disk space usage
2013-01-25 23:14 ` Sam Lang
@ 2013-01-26 5:35 ` Dan Mick
2013-01-28 2:32 ` Dan Mick
2013-01-26 5:44 ` Marcus Sorensen
[not found] ` <CALFpzo4u2PspjkvSig3LL821YY_XA2iHBjAh30JxgwWTFGsgcA@mail.gmail.com>
2 siblings, 1 reply; 11+ messages in thread
From: Dan Mick @ 2013-01-26 5:35 UTC (permalink / raw)
To: Sam Lang; +Cc: Cesar Mello, ceph-devel@vger.kernel.org
>> If the S3 API is not well suited to my scenario, then my effort should
>> be better directed to porting or writing a native ceph client for
>> Windows. I just need an API to read and write/append blocks to files.
>> Any comments are really appreciated.
>
> Hopefully someone with more windows experience will give you better
> info/advice than I can.
The python bindings should Just Work(tm). Just a thought.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RadosGW performance and disk space usage
2013-01-25 23:14 ` Sam Lang
2013-01-26 5:35 ` Dan Mick
@ 2013-01-26 5:44 ` Marcus Sorensen
[not found] ` <CALFpzo4u2PspjkvSig3LL821YY_XA2iHBjAh30JxgwWTFGsgcA@mail.gmail.com>
2 siblings, 0 replies; 11+ messages in thread
From: Marcus Sorensen @ 2013-01-26 5:44 UTC (permalink / raw)
To: Sam Lang; +Cc: Cesar Mello, ceph-devel@vger.kernel.org
Have you tried rest-bench on localhost at the rados gateway? I was
playing with the rados gateway in a VM the other day, and was getting
up to 400/s on 4k objects. Above that I was getting connection
failures, but I think it was just due to a default max connections
setting somewhere or something. My VM is on SSD though. I was just
thinking it may help isolate the issue.
On Fri, Jan 25, 2013 at 4:14 PM, Sam Lang <sam.lang@inktank.com> wrote:
>
> On Thu, Jan 24, 2013 at 9:27 AM, Cesar Mello <cmello@gmail.com> wrote:
> > Hi!
> >
> > I have successfully prototyped read/write access to ceph from Windows
> > using the S3 API, thanks so much for the help.
> >
> > Now I would like to do some prototypes targeting performance
> > evaluation. My scenario typically requires parallel storage of data
> > from tens of thousands of loggers, but scalability to hundreds of
> > thousands is the main reason for investigating ceph.
> >
> > My tests using a single laptop running ceph with 2 local OSDs and
> > local radosgw allows writing in average 2.5 small objects per second
> > (100 objects in 40 seconds). Is this the expected performance? It
> > seems to be I/O bound because the HDD led keeps on during the
> > PutObject requests. Any suggestion or documentation pointers for
> > profiling are very appreciated.
>
> Hi Mello,
>
> 2.5 objects/sec seems terribly slow, even on your laptop. How "small"
> are these objects? You might try to benchmark without the disk as a
> potential bottleneck, by putting your osd data and journals in /tmp
> (for benchmarking only of course) or create/mount a tmpfs and point
> your osd backends there.
>
> >
> > I am afraid the S3 API is not good for my scenario, because there is
> > no way to append data to existing objects (so I won't be able to model
> > a single object for each data collector). If this is the case, then I
> > would need to store billions of small objects. I would like to know
> > how much disk space each object instance requires other than the
> > object content length.
> >
> > If the S3 API is not well suited to my scenario, then my effort should
> > be better directed to porting or writing a native ceph client for
> > Windows. I just need an API to read and write/append blocks to files.
> > Any comments are really appreciated.
>
> Hopefully someone with more windows experience will give you better
> info/advice than I can.
>
> You could try to port the rados API to windows. Its purely userspace,
> but does rely on pthreads and other libc/gcc specifics. With
> something like cygwin a port might not be too hard though. If you
> decide to go that route, let us know how you progress!
>
> -sam
>
>
> >
> > Thank you a lot for the attention!
> >
> > Best regards
> > Mello
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RadosGW performance and disk space usage
[not found] ` <CALFpzo4u2PspjkvSig3LL821YY_XA2iHBjAh30JxgwWTFGsgcA@mail.gmail.com>
@ 2013-01-26 14:20 ` Cesar Mello
2013-01-27 23:10 ` Cesar Mello
0 siblings, 1 reply; 11+ messages in thread
From: Cesar Mello @ 2013-01-26 14:20 UTC (permalink / raw)
To: Marcus Sorensen; +Cc: Sam Lang, ceph-devel@vger.kernel.org
Dear Sam, Dan and Marcus,
Thank you a lot for the replies. I'll do more tests today.
The length of each object used in my test is just 20 bytes. I'm glad
you got 400 objects/s! Iif I get that with a length of 8 KB using a
2-node cluster, then ceph with rados will be already faster than my
current solution. And then I will be able to present it to my boss.
:-)
I'll try rest-bench later. Thanks for the help!
Best regards
Mello
On Sat, Jan 26, 2013 at 3:43 AM, Marcus Sorensen <shadowsor@gmail.com> wrote:
> Have you tried rest-bench on localhost at the rados gateway? I was playing
> with the rados gateway in a VM the other day, and was getting up to 400/s on
> 4k objects. Above that I was getting connection failures, but I think it was
> just due to a default max connections setting somewhere or something. My VM
> is on SSD though. I was just thinking it may help isolate the issue.
>
>
> On Fri, Jan 25, 2013 at 4:14 PM, Sam Lang <sam.lang@inktank.com> wrote:
>>
>> On Thu, Jan 24, 2013 at 9:27 AM, Cesar Mello <cmello@gmail.com> wrote:
>> > Hi!
>> >
>> > I have successfully prototyped read/write access to ceph from Windows
>> > using the S3 API, thanks so much for the help.
>> >
>> > Now I would like to do some prototypes targeting performance
>> > evaluation. My scenario typically requires parallel storage of data
>> > from tens of thousands of loggers, but scalability to hundreds of
>> > thousands is the main reason for investigating ceph.
>> >
>> > My tests using a single laptop running ceph with 2 local OSDs and
>> > local radosgw allows writing in average 2.5 small objects per second
>> > (100 objects in 40 seconds). Is this the expected performance? It
>> > seems to be I/O bound because the HDD led keeps on during the
>> > PutObject requests. Any suggestion or documentation pointers for
>> > profiling are very appreciated.
>>
>> Hi Mello,
>>
>> 2.5 objects/sec seems terribly slow, even on your laptop. How "small"
>> are these objects? You might try to benchmark without the disk as a
>> potential bottleneck, by putting your osd data and journals in /tmp
>> (for benchmarking only of course) or create/mount a tmpfs and point
>> your osd backends there.
>>
>> >
>> > I am afraid the S3 API is not good for my scenario, because there is
>> > no way to append data to existing objects (so I won't be able to model
>> > a single object for each data collector). If this is the case, then I
>> > would need to store billions of small objects. I would like to know
>> > how much disk space each object instance requires other than the
>> > object content length.
>> >
>> > If the S3 API is not well suited to my scenario, then my effort should
>> > be better directed to porting or writing a native ceph client for
>> > Windows. I just need an API to read and write/append blocks to files.
>> > Any comments are really appreciated.
>>
>> Hopefully someone with more windows experience will give you better
>> info/advice than I can.
>>
>> You could try to port the rados API to windows. Its purely userspace,
>> but does rely on pthreads and other libc/gcc specifics. With
>> something like cygwin a port might not be too hard though. If you
>> decide to go that route, let us know how you progress!
>>
>> -sam
>>
>>
>> >
>> > Thank you a lot for the attention!
>> >
>> > Best regards
>> > Mello
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RadosGW performance and disk space usage
2013-01-26 14:20 ` Cesar Mello
@ 2013-01-27 23:10 ` Cesar Mello
2013-01-28 14:28 ` Joao Eduardo Luis
2013-01-28 17:35 ` Yehuda Sadeh
0 siblings, 2 replies; 11+ messages in thread
From: Cesar Mello @ 2013-01-27 23:10 UTC (permalink / raw)
To: Marcus Sorensen; +Cc: Sam Lang, ceph-devel@vger.kernel.org
Hi,
Just tried rest-bench. This little tool is wonderful, thanks!
I still have to learn lots of things. So please don't spend much time
explaining me, but instead please give me any pointers to
documentation or source code that can be useful. As a curiosity, I'm
pasting the results from my laptop. I'll repeat the same tests using
my desktop as the server.
Notice there is an assert being triggered, so I guess I'm running a
build with debugging code ?!. I compiled using ./configure
--with-radosgw --with-rest-bench followed by make.
Thanks a lot for the attention.
Best regards!
Mello
root@l3:/etc/init.d# rest-bench --api-host=localhost --bucket=test
--access-key=JJABVJ3AWBS1ZOCML7NS
--secret=A+ecBz2+Sdxj4Y8Mo+u3akIewGvJPkwOhwRgPKkX --protocol=http
--uri_style=path write
host=localhost
Maintaining 16 concurrent writes of 4194304 bytes for at least 60 seconds.
Object prefix: benchmark_data_l3_4032
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
0 3 3 0 0 0 - 0
1 16 16 0 0 0 - 0
2 16 16 0 0 0 - 0
3 16 16 0 0 0 - 0
4 16 16 0 0 0 - 0
5 16 16 0 0 0 - 0
6 16 16 0 0 0 - 0
7 16 16 0 0 0 - 0
8 16 16 0 0 0 - 0
9 16 16 0 0 0 - 0
10 16 16 0 0 0 - 0
11 16 16 0 0 0 - 0
12 16 17 1 0.333265 0.333333 11.2761 11.2761
13 16 18 2 0.615257 4 12.5964 11.9363
14 16 20 4 1.14262 8 13.1392 12.5365
15 16 23 7 1.86628 12 14.2273 13.2594
16 16 27 11 2.74944 16 15.0222 13.8968
17 16 32 16 3.76394 20 16.2604 14.6301
18 16 32 16 3.55483 0 - 14.6301
19 16 34 18 3.7887 4 6.2274 13.7695
2013-01-27 20:49:29.703509min lat: 6.2274 max lat: 16.2604 avg lat: 13.7695
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
20 16 34 18 3.59927 0 - 13.7695
21 16 34 18 3.42787 0 - 13.7695
22 16 34 18 3.27205 0 - 13.7695
23 16 35 19 3.30367 1 9.09053 13.5233
24 16 36 20 3.33264 4 9.09667 13.302
25 16 36 20 3.19933 0 - 13.302
26 16 36 20 3.07628 0 - 13.302
27 16 37 21 3.11047 1.33333 11.2459 13.204
28 16 37 21 2.99938 0 - 13.204
29 16 37 21 2.89595 0 - 13.204
30 16 37 21 2.79942 0 - 13.204
31 16 37 21 2.70912 0 - 13.204
32 16 39 23 2.87441 1.6 14.9981 13.3602
33 16 39 23 2.7873 0 - 13.3602
34 16 39 23 2.70533 0 - 13.3602
35 16 40 24 2.74229 1.33333 21.5365 13.7009
36 16 40 24 2.66612 0 - 13.7009
37 16 42 26 2.81023 4 22.6059 14.3855
38 16 42 26 2.73628 0 - 14.3855
39 16 45 29 2.97374 6 23.2615 15.3025
2013-01-27 20:49:49.707740min lat: 6.2274 max lat: 23.4496 avg lat: 16.1307
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
40 16 51 35 3.49927 24 21.0123 16.1307
41 16 51 35 3.41392 0 - 16.1307
42 16 52 36 3.42786 2 19.0243 16.211
43 16 52 36 3.34814 0 - 16.211
44 16 52 36 3.27204 0 - 16.211
45 16 52 36 3.19933 0 - 16.211
46 16 53 37 3.21672 1 11.0793 16.0723
47 16 53 37 3.14828 0 - 16.0723
48 16 53 37 3.08269 0 - 16.0723
49 16 53 37 3.01978 0 - 16.0723
50 16 55 39 3.11935 2 23.4253 16.5245
51 16 55 39 3.05819 0 - 16.5245
52 16 58 42 3.2301 6 13.3052 16.2945
53 16 58 42 3.16915 0 - 16.2945
54 16 60 44 3.25858 4 14.2116 16.1999
55 16 60 44 3.19933 0 - 16.1999
56 16 63 47 3.35644 6 15.6276 16.3393
57 16 63 47 3.29756 0 - 16.3393
58 16 66 50 3.44755 6 18.0547 16.5454
59 16 66 50 3.38912 0 - 16.5454
2013-01-27 20:50:09.712027min lat: 6.2274 max lat: 26.3519 avg lat: 16.5454
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
60 16 66 50 3.33263 0 - 16.5454
61 16 66 50 3.278 0 - 16.5454
62 16 66 50 3.22513 0 - 16.5454
63 16 67 51 3.23741 0.8 10.9553 16.4358
64 16 67 51 3.18683 0 - 16.4358
65 16 67 51 3.1378 0 - 16.4358
66 16 67 51 3.09025 0 - 16.4358
67 16 67 51 3.04413 0 - 16.4358
68 16 67 51 2.99937 0 - 16.4358
69 16 67 51 2.9559 0 - 16.4358
70 16 67 51 2.91367 0 - 16.4358
71 14 67 53 2.98528 1 15.1841 16.2524
72 9 67 58 3.22154 20 26.2114 16.6243
Total time run: 72.403588
Total writes made: 67
Write size: 4194304
Bandwidth (MB/sec): 3.701
Stddev Bandwidth: 4.92554
Max bandwidth (MB/sec): 24
Min bandwidth (MB/sec): 0
Average Latency: 17.1886
Stddev Latency: 5.48181
Max latency: 32.7038
Min latency: 6.2274
common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()'
thread 7f1211401780 time 2013-01-27 20:51:01.196525
common/WorkQueue.cc: 59: FAILED assert(_threads.empty())
ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f)
1: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c]
2: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021]
3: (main()+0x75b) [0x42521b]
4: (__libc_start_main()+0xed) [0x7f120f37576d]
5: rest-bench() [0x426079]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
2013-01-27 20:51:01.197253 7f1211401780 -1 common/WorkQueue.cc: In
function 'virtual ThreadPool::~ThreadPool()' thread 7f1211401780 time
2013-01-27 20:51:01.196525
common/WorkQueue.cc: 59: FAILED assert(_threads.empty())
ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f)
1: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c]
2: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021]
3: (main()+0x75b) [0x42521b]
4: (__libc_start_main()+0xed) [0x7f120f37576d]
5: rest-bench() [0x426079]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- begin dump of recent events ---
-11> 2013-01-27 20:49:09.292227 7f1211401780 5 asok(0x29dc270)
register_command perfcounters_dump hook 0x29dc440
-10> 2013-01-27 20:49:09.292259 7f1211401780 5 asok(0x29dc270)
register_command 1 hook 0x29dc440
-9> 2013-01-27 20:49:09.292262 7f1211401780 5 asok(0x29dc270)
register_command perf dump hook 0x29dc440
-8> 2013-01-27 20:49:09.292271 7f1211401780 5 asok(0x29dc270)
register_command perfcounters_schema hook 0x29dc440
-7> 2013-01-27 20:49:09.292275 7f1211401780 5 asok(0x29dc270)
register_command 2 hook 0x29dc440
-6> 2013-01-27 20:49:09.292278 7f1211401780 5 asok(0x29dc270)
register_command perf schema hook 0x29dc440
-5> 2013-01-27 20:49:09.292285 7f1211401780 5 asok(0x29dc270)
register_command config show hook 0x29dc440
-4> 2013-01-27 20:49:09.292290 7f1211401780 5 asok(0x29dc270)
register_command config set hook 0x29dc440
-3> 2013-01-27 20:49:09.292292 7f1211401780 5 asok(0x29dc270)
register_command log flush hook 0x29dc440
-2> 2013-01-27 20:49:09.292296 7f1211401780 5 asok(0x29dc270)
register_command log dump hook 0x29dc440
-1> 2013-01-27 20:49:09.292300 7f1211401780 5 asok(0x29dc270)
register_command log reopen hook 0x29dc440
0> 2013-01-27 20:51:01.197253 7f1211401780 -1
common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()'
thread 7f1211401780 time 2013-01-27 20:51:01.196525
common/WorkQueue.cc: 59: FAILED assert(_threads.empty())
ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f)
1: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c]
2: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021]
3: (main()+0x75b) [0x42521b]
4: (__libc_start_main()+0xed) [0x7f120f37576d]
5: rest-bench() [0x426079]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
0/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 hadoop
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 100000
max_new 1000
log_file
--- end dump of recent events ---
terminate called after throwing an instance of 'ceph::FailedAssertion'
*** Caught signal (Aborted) **
in thread 7f1211401780
ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f)
1: rest-bench() [0x436b40]
2: (()+0xfcb0) [0x7f1210ffccb0]
3: (gsignal()+0x35) [0x7f120f38a425]
4: (abort()+0x17b) [0x7f120f38db8b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f120fc85e2d]
6: (()+0x5ef26) [0x7f120fc83f26]
7: (()+0x5ef53) [0x7f120fc83f53]
8: (()+0x5f17e) [0x7f120fc8417e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x43d) [0x43eced]
10: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c]
11: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021]
12: (main()+0x75b) [0x42521b]
13: (__libc_start_main()+0xed) [0x7f120f37576d]
14: rest-bench() [0x426079]
2013-01-27 20:51:01.200578 7f1211401780 -1 *** Caught signal (Aborted) **
in thread 7f1211401780
ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f)
1: rest-bench() [0x436b40]
2: (()+0xfcb0) [0x7f1210ffccb0]
3: (gsignal()+0x35) [0x7f120f38a425]
4: (abort()+0x17b) [0x7f120f38db8b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f120fc85e2d]
6: (()+0x5ef26) [0x7f120fc83f26]
7: (()+0x5ef53) [0x7f120fc83f53]
8: (()+0x5f17e) [0x7f120fc8417e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x43d) [0x43eced]
10: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c]
11: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021]
12: (main()+0x75b) [0x42521b]
13: (__libc_start_main()+0xed) [0x7f120f37576d]
14: rest-bench() [0x426079]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- begin dump of recent events ---
0> 2013-01-27 20:51:01.200578 7f1211401780 -1 *** Caught signal
(Aborted) **
in thread 7f1211401780
ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f)
1: rest-bench() [0x436b40]
2: (()+0xfcb0) [0x7f1210ffccb0]
3: (gsignal()+0x35) [0x7f120f38a425]
4: (abort()+0x17b) [0x7f120f38db8b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f120fc85e2d]
6: (()+0x5ef26) [0x7f120fc83f26]
7: (()+0x5ef53) [0x7f120fc83f53]
8: (()+0x5f17e) [0x7f120fc8417e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x43d) [0x43eced]
10: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c]
11: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021]
12: (main()+0x75b) [0x42521b]
13: (__libc_start_main()+0xed) [0x7f120f37576d]
14: rest-bench() [0x426079]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
0/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 hadoop
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 100000
max_new 1000
log_file
--- end dump of recent events ---
Aborted (core dumped)
root@l3:/etc/init.d#
On Sat, Jan 26, 2013 at 12:20 PM, Cesar Mello <cmello@gmail.com> wrote:
> Dear Sam, Dan and Marcus,
>
> Thank you a lot for the replies. I'll do more tests today.
>
> The length of each object used in my test is just 20 bytes. I'm glad
> you got 400 objects/s! Iif I get that with a length of 8 KB using a
> 2-node cluster, then ceph with rados will be already faster than my
> current solution. And then I will be able to present it to my boss.
> :-)
>
> I'll try rest-bench later. Thanks for the help!
>
> Best regards
> Mello
>
> On Sat, Jan 26, 2013 at 3:43 AM, Marcus Sorensen <shadowsor@gmail.com> wrote:
>> Have you tried rest-bench on localhost at the rados gateway? I was playing
>> with the rados gateway in a VM the other day, and was getting up to 400/s on
>> 4k objects. Above that I was getting connection failures, but I think it was
>> just due to a default max connections setting somewhere or something. My VM
>> is on SSD though. I was just thinking it may help isolate the issue.
>>
>>
>> On Fri, Jan 25, 2013 at 4:14 PM, Sam Lang <sam.lang@inktank.com> wrote:
>>>
>>> On Thu, Jan 24, 2013 at 9:27 AM, Cesar Mello <cmello@gmail.com> wrote:
>>> > Hi!
>>> >
>>> > I have successfully prototyped read/write access to ceph from Windows
>>> > using the S3 API, thanks so much for the help.
>>> >
>>> > Now I would like to do some prototypes targeting performance
>>> > evaluation. My scenario typically requires parallel storage of data
>>> > from tens of thousands of loggers, but scalability to hundreds of
>>> > thousands is the main reason for investigating ceph.
>>> >
>>> > My tests using a single laptop running ceph with 2 local OSDs and
>>> > local radosgw allows writing in average 2.5 small objects per second
>>> > (100 objects in 40 seconds). Is this the expected performance? It
>>> > seems to be I/O bound because the HDD led keeps on during the
>>> > PutObject requests. Any suggestion or documentation pointers for
>>> > profiling are very appreciated.
>>>
>>> Hi Mello,
>>>
>>> 2.5 objects/sec seems terribly slow, even on your laptop. How "small"
>>> are these objects? You might try to benchmark without the disk as a
>>> potential bottleneck, by putting your osd data and journals in /tmp
>>> (for benchmarking only of course) or create/mount a tmpfs and point
>>> your osd backends there.
>>>
>>> >
>>> > I am afraid the S3 API is not good for my scenario, because there is
>>> > no way to append data to existing objects (so I won't be able to model
>>> > a single object for each data collector). If this is the case, then I
>>> > would need to store billions of small objects. I would like to know
>>> > how much disk space each object instance requires other than the
>>> > object content length.
>>> >
>>> > If the S3 API is not well suited to my scenario, then my effort should
>>> > be better directed to porting or writing a native ceph client for
>>> > Windows. I just need an API to read and write/append blocks to files.
>>> > Any comments are really appreciated.
>>>
>>> Hopefully someone with more windows experience will give you better
>>> info/advice than I can.
>>>
>>> You could try to port the rados API to windows. Its purely userspace,
>>> but does rely on pthreads and other libc/gcc specifics. With
>>> something like cygwin a port might not be too hard though. If you
>>> decide to go that route, let us know how you progress!
>>>
>>> -sam
>>>
>>>
>>> >
>>> > Thank you a lot for the attention!
>>> >
>>> > Best regards
>>> > Mello
>>> > --
>>> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> > the body of a message to majordomo@vger.kernel.org
>>> > More majordomo info at http://vger.kernel.org/majordomo-info.html
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RadosGW performance and disk space usage
2013-01-26 5:35 ` Dan Mick
@ 2013-01-28 2:32 ` Dan Mick
0 siblings, 0 replies; 11+ messages in thread
From: Dan Mick @ 2013-01-28 2:32 UTC (permalink / raw)
To: Sam Lang; +Cc: Cesar Mello, ceph-devel@vger.kernel.org
On 1/25/2013 9:35 PM, Dan Mick wrote:
>
>>> If the S3 API is not well suited to my scenario, then my effort should
>>> be better directed to porting or writing a native ceph client for
>>> Windows. I just need an API to read and write/append blocks to files.
>>> Any comments are really appreciated.
>>
>> Hopefully someone with more windows experience will give you better
>> info/advice than I can.
>
> The python bindings should Just Work(tm). Just a thought.
...but of course, as Sam points out, they rely on the C++ libs. Duh.
Sorry to mislead.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RadosGW performance and disk space usage
2013-01-27 23:10 ` Cesar Mello
@ 2013-01-28 14:28 ` Joao Eduardo Luis
2013-01-28 17:33 ` Yehuda Sadeh
2013-01-28 17:35 ` Yehuda Sadeh
1 sibling, 1 reply; 11+ messages in thread
From: Joao Eduardo Luis @ 2013-01-28 14:28 UTC (permalink / raw)
To: Cesar Mello; +Cc: Marcus Sorensen, Sam Lang, ceph-devel@vger.kernel.org, yehuda
On 01/27/2013 11:10 PM, Cesar Mello wrote:
> Hi,
>
> Just tried rest-bench. This little tool is wonderful, thanks!
>
> I still have to learn lots of things. So please don't spend much time
> explaining me, but instead please give me any pointers to
> documentation or source code that can be useful. As a curiosity, I'm
> pasting the results from my laptop. I'll repeat the same tests using
> my desktop as the server.
>
> Notice there is an assert being triggered, so I guess I'm running a
> build with debugging code ?!. I compiled using ./configure
> --with-radosgw --with-rest-bench followed by make.
asserts are usually used to mark invariants on the code logic, and are
always built, regardless debugging being enabled or disabled. Given you
are hitting one, it probably means something is not quite right (might
be a bug, or some invariant was broken for some reason).
> common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()'
> thread 7f1211401780 time 2013-01-27 20:51:01.196525
> common/WorkQueue.cc: 59: FAILED assert(_threads.empty())
> ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f)
> 1: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c]
> 2: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021]
> 3: (main()+0x75b) [0x42521b]
> 4: (__libc_start_main()+0xed) [0x7f120f37576d]
> 5: rest-bench() [0x426079]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
Looks like http://tracker.newdream.net/issues/3896
Am not sure who should be made aware of this issue though. Maybe Yehuda
(cc'ing)?
-Joao
> 2013-01-27 20:51:01.197253 7f1211401780 -1 common/WorkQueue.cc: In
> function 'virtual ThreadPool::~ThreadPool()' thread 7f1211401780 time
> 2013-01-27 20:51:01.196525
> common/WorkQueue.cc: 59: FAILED assert(_threads.empty())
>
> ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f)
> 1: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c]
> 2: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021]
> 3: (main()+0x75b) [0x42521b]
> 4: (__libc_start_main()+0xed) [0x7f120f37576d]
> 5: rest-bench() [0x426079]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
> --- begin dump of recent events ---
> -11> 2013-01-27 20:49:09.292227 7f1211401780 5 asok(0x29dc270)
> register_command perfcounters_dump hook 0x29dc440
> -10> 2013-01-27 20:49:09.292259 7f1211401780 5 asok(0x29dc270)
> register_command 1 hook 0x29dc440
> -9> 2013-01-27 20:49:09.292262 7f1211401780 5 asok(0x29dc270)
> register_command perf dump hook 0x29dc440
> -8> 2013-01-27 20:49:09.292271 7f1211401780 5 asok(0x29dc270)
> register_command perfcounters_schema hook 0x29dc440
> -7> 2013-01-27 20:49:09.292275 7f1211401780 5 asok(0x29dc270)
> register_command 2 hook 0x29dc440
> -6> 2013-01-27 20:49:09.292278 7f1211401780 5 asok(0x29dc270)
> register_command perf schema hook 0x29dc440
> -5> 2013-01-27 20:49:09.292285 7f1211401780 5 asok(0x29dc270)
> register_command config show hook 0x29dc440
> -4> 2013-01-27 20:49:09.292290 7f1211401780 5 asok(0x29dc270)
> register_command config set hook 0x29dc440
> -3> 2013-01-27 20:49:09.292292 7f1211401780 5 asok(0x29dc270)
> register_command log flush hook 0x29dc440
> -2> 2013-01-27 20:49:09.292296 7f1211401780 5 asok(0x29dc270)
> register_command log dump hook 0x29dc440
> -1> 2013-01-27 20:49:09.292300 7f1211401780 5 asok(0x29dc270)
> register_command log reopen hook 0x29dc440
> 0> 2013-01-27 20:51:01.197253 7f1211401780 -1
> common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()'
> thread 7f1211401780 time 2013-01-27 20:51:01.196525
> common/WorkQueue.cc: 59: FAILED assert(_threads.empty())
>
> ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f)
> 1: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c]
> 2: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021]
> 3: (main()+0x75b) [0x42521b]
> 4: (__libc_start_main()+0xed) [0x7f120f37576d]
> 5: rest-bench() [0x426079]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
> --- logging levels ---
> 0/ 5 none
> 0/ 1 lockdep
> 0/ 1 context
> 1/ 1 crush
> 1/ 5 mds
> 1/ 5 mds_balancer
> 1/ 5 mds_locker
> 1/ 5 mds_log
> 1/ 5 mds_log_expire
> 1/ 5 mds_migrator
> 0/ 1 buffer
> 0/ 1 timer
> 0/ 1 filer
> 0/ 1 striper
> 0/ 1 objecter
> 0/ 5 rados
> 0/ 5 rbd
> 0/ 5 journaler
> 0/ 5 objectcacher
> 0/ 5 client
> 0/ 5 osd
> 0/ 5 optracker
> 0/ 5 objclass
> 1/ 3 filestore
> 1/ 3 journal
> 0/ 5 ms
> 1/ 5 mon
> 0/10 monc
> 0/ 5 paxos
> 0/ 5 tp
> 1/ 5 auth
> 1/ 5 crypto
> 1/ 1 finisher
> 1/ 5 heartbeatmap
> 1/ 5 perfcounter
> 1/ 5 rgw
> 1/ 5 hadoop
> 1/ 5 javaclient
> 1/ 5 asok
> 1/ 1 throttle
> -2/-2 (syslog threshold)
> 99/99 (stderr threshold)
> max_recent 100000
> max_new 1000
> log_file
> --- end dump of recent events ---
> terminate called after throwing an instance of 'ceph::FailedAssertion'
> *** Caught signal (Aborted) **
> in thread 7f1211401780
> ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f)
> 1: rest-bench() [0x436b40]
> 2: (()+0xfcb0) [0x7f1210ffccb0]
> 3: (gsignal()+0x35) [0x7f120f38a425]
> 4: (abort()+0x17b) [0x7f120f38db8b]
> 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f120fc85e2d]
> 6: (()+0x5ef26) [0x7f120fc83f26]
> 7: (()+0x5ef53) [0x7f120fc83f53]
> 8: (()+0x5f17e) [0x7f120fc8417e]
> 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x43d) [0x43eced]
> 10: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c]
> 11: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021]
> 12: (main()+0x75b) [0x42521b]
> 13: (__libc_start_main()+0xed) [0x7f120f37576d]
> 14: rest-bench() [0x426079]
> 2013-01-27 20:51:01.200578 7f1211401780 -1 *** Caught signal (Aborted) **
> in thread 7f1211401780
>
> ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f)
> 1: rest-bench() [0x436b40]
> 2: (()+0xfcb0) [0x7f1210ffccb0]
> 3: (gsignal()+0x35) [0x7f120f38a425]
> 4: (abort()+0x17b) [0x7f120f38db8b]
> 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f120fc85e2d]
> 6: (()+0x5ef26) [0x7f120fc83f26]
> 7: (()+0x5ef53) [0x7f120fc83f53]
> 8: (()+0x5f17e) [0x7f120fc8417e]
> 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x43d) [0x43eced]
> 10: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c]
> 11: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021]
> 12: (main()+0x75b) [0x42521b]
> 13: (__libc_start_main()+0xed) [0x7f120f37576d]
> 14: rest-bench() [0x426079]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
> --- begin dump of recent events ---
> 0> 2013-01-27 20:51:01.200578 7f1211401780 -1 *** Caught signal
> (Aborted) **
> in thread 7f1211401780
>
> ceph version 0.56-464-gfa421cf (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f)
> 1: rest-bench() [0x436b40]
> 2: (()+0xfcb0) [0x7f1210ffccb0]
> 3: (gsignal()+0x35) [0x7f120f38a425]
> 4: (abort()+0x17b) [0x7f120f38db8b]
> 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f120fc85e2d]
> 6: (()+0x5ef26) [0x7f120fc83f26]
> 7: (()+0x5ef53) [0x7f120fc83f53]
> 8: (()+0x5f17e) [0x7f120fc8417e]
> 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x43d) [0x43eced]
> 10: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c]
> 11: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021]
> 12: (main()+0x75b) [0x42521b]
> 13: (__libc_start_main()+0xed) [0x7f120f37576d]
> 14: rest-bench() [0x426079]
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
> --- logging levels ---
> 0/ 5 none
> 0/ 1 lockdep
> 0/ 1 context
> 1/ 1 crush
> 1/ 5 mds
> 1/ 5 mds_balancer
> 1/ 5 mds_locker
> 1/ 5 mds_log
> 1/ 5 mds_log_expire
> 1/ 5 mds_migrator
> 0/ 1 buffer
> 0/ 1 timer
> 0/ 1 filer
> 0/ 1 striper
> 0/ 1 objecter
> 0/ 5 rados
> 0/ 5 rbd
> 0/ 5 journaler
> 0/ 5 objectcacher
> 0/ 5 client
> 0/ 5 osd
> 0/ 5 optracker
> 0/ 5 objclass
> 1/ 3 filestore
> 1/ 3 journal
> 0/ 5 ms
> 1/ 5 mon
> 0/10 monc
> 0/ 5 paxos
> 0/ 5 tp
> 1/ 5 auth
> 1/ 5 crypto
> 1/ 1 finisher
> 1/ 5 heartbeatmap
> 1/ 5 perfcounter
> 1/ 5 rgw
> 1/ 5 hadoop
> 1/ 5 javaclient
> 1/ 5 asok
> 1/ 1 throttle
> -2/-2 (syslog threshold)
> 99/99 (stderr threshold)
> max_recent 100000
> max_new 1000
> log_file
> --- end dump of recent events ---
> Aborted (core dumped)
> root@l3:/etc/init.d#
>
> On Sat, Jan 26, 2013 at 12:20 PM, Cesar Mello <cmello@gmail.com> wrote:
>> Dear Sam, Dan and Marcus,
>>
>> Thank you a lot for the replies. I'll do more tests today.
>>
>> The length of each object used in my test is just 20 bytes. I'm glad
>> you got 400 objects/s! Iif I get that with a length of 8 KB using a
>> 2-node cluster, then ceph with rados will be already faster than my
>> current solution. And then I will be able to present it to my boss.
>> :-)
>>
>> I'll try rest-bench later. Thanks for the help!
>>
>> Best regards
>> Mello
>>
>> On Sat, Jan 26, 2013 at 3:43 AM, Marcus Sorensen <shadowsor@gmail.com> wrote:
>>> Have you tried rest-bench on localhost at the rados gateway? I was playing
>>> with the rados gateway in a VM the other day, and was getting up to 400/s on
>>> 4k objects. Above that I was getting connection failures, but I think it was
>>> just due to a default max connections setting somewhere or something. My VM
>>> is on SSD though. I was just thinking it may help isolate the issue.
>>>
>>>
>>> On Fri, Jan 25, 2013 at 4:14 PM, Sam Lang <sam.lang@inktank.com> wrote:
>>>>
>>>> On Thu, Jan 24, 2013 at 9:27 AM, Cesar Mello <cmello@gmail.com> wrote:
>>>>> Hi!
>>>>>
>>>>> I have successfully prototyped read/write access to ceph from Windows
>>>>> using the S3 API, thanks so much for the help.
>>>>>
>>>>> Now I would like to do some prototypes targeting performance
>>>>> evaluation. My scenario typically requires parallel storage of data
>>>>> from tens of thousands of loggers, but scalability to hundreds of
>>>>> thousands is the main reason for investigating ceph.
>>>>>
>>>>> My tests using a single laptop running ceph with 2 local OSDs and
>>>>> local radosgw allows writing in average 2.5 small objects per second
>>>>> (100 objects in 40 seconds). Is this the expected performance? It
>>>>> seems to be I/O bound because the HDD led keeps on during the
>>>>> PutObject requests. Any suggestion or documentation pointers for
>>>>> profiling are very appreciated.
>>>>
>>>> Hi Mello,
>>>>
>>>> 2.5 objects/sec seems terribly slow, even on your laptop. How "small"
>>>> are these objects? You might try to benchmark without the disk as a
>>>> potential bottleneck, by putting your osd data and journals in /tmp
>>>> (for benchmarking only of course) or create/mount a tmpfs and point
>>>> your osd backends there.
>>>>
>>>>>
>>>>> I am afraid the S3 API is not good for my scenario, because there is
>>>>> no way to append data to existing objects (so I won't be able to model
>>>>> a single object for each data collector). If this is the case, then I
>>>>> would need to store billions of small objects. I would like to know
>>>>> how much disk space each object instance requires other than the
>>>>> object content length.
>>>>>
>>>>> If the S3 API is not well suited to my scenario, then my effort should
>>>>> be better directed to porting or writing a native ceph client for
>>>>> Windows. I just need an API to read and write/append blocks to files.
>>>>> Any comments are really appreciated.
>>>>
>>>> Hopefully someone with more windows experience will give you better
>>>> info/advice than I can.
>>>>
>>>> You could try to port the rados API to windows. Its purely userspace,
>>>> but does rely on pthreads and other libc/gcc specifics. With
>>>> something like cygwin a port might not be too hard though. If you
>>>> decide to go that route, let us know how you progress!
>>>>
>>>> -sam
>>>>
>>>>
>>>>>
>>>>> Thank you a lot for the attention!
>>>>>
>>>>> Best regards
>>>>> Mello
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RadosGW performance and disk space usage
2013-01-28 14:28 ` Joao Eduardo Luis
@ 2013-01-28 17:33 ` Yehuda Sadeh
0 siblings, 0 replies; 11+ messages in thread
From: Yehuda Sadeh @ 2013-01-28 17:33 UTC (permalink / raw)
To: Joao Eduardo Luis
Cc: Cesar Mello, Marcus Sorensen, Sam Lang,
ceph-devel@vger.kernel.org
On Mon, Jan 28, 2013 at 6:28 AM, Joao Eduardo Luis
<joao.luis@inktank.com> wrote:
> On 01/27/2013 11:10 PM, Cesar Mello wrote:
>>
>> Hi,
>>
>> Just tried rest-bench. This little tool is wonderful, thanks!
>>
>> I still have to learn lots of things. So please don't spend much time
>> explaining me, but instead please give me any pointers to
>> documentation or source code that can be useful. As a curiosity, I'm
>> pasting the results from my laptop. I'll repeat the same tests using
>> my desktop as the server.
>>
>> Notice there is an assert being triggered, so I guess I'm running a
>> build with debugging code ?!. I compiled using ./configure
>> --with-radosgw --with-rest-bench followed by make.
>
>
> asserts are usually used to mark invariants on the code logic, and are
> always built, regardless debugging being enabled or disabled. Given you are
> hitting one, it probably means something is not quite right (might be a bug,
> or some invariant was broken for some reason).
>
>
>> common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()'
>> thread 7f1211401780 time 2013-01-27 20:51:01.196525
>> common/WorkQueue.cc: 59: FAILED assert(_threads.empty())
>> ceph version 0.56-464-gfa421cf
>> (fa421cf5f52ca16fa1328dbea2f4bda85c56cd3f)
>> 1: (ThreadPool::~ThreadPool()+0x10c) [0x43bf9c]
>> 2: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x42a021]
>> 3: (main()+0x75b) [0x42521b]
>> 4: (__libc_start_main()+0xed) [0x7f120f37576d]
>> 5: rest-bench() [0x426079]
>> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
>> needed to interpret this.
>
>
> Looks like http://tracker.newdream.net/issues/3896
>
Right, 3896. Probably some cleanup before shutdown issues.
Yehuda
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RadosGW performance and disk space usage
2013-01-27 23:10 ` Cesar Mello
2013-01-28 14:28 ` Joao Eduardo Luis
@ 2013-01-28 17:35 ` Yehuda Sadeh
2013-01-28 18:18 ` Cesar Mello
1 sibling, 1 reply; 11+ messages in thread
From: Yehuda Sadeh @ 2013-01-28 17:35 UTC (permalink / raw)
To: Cesar Mello; +Cc: Marcus Sorensen, Sam Lang, ceph-devel@vger.kernel.org
On Sun, Jan 27, 2013 at 3:10 PM, Cesar Mello <cmello@gmail.com> wrote:
> Hi,
>
> Just tried rest-bench. This little tool is wonderful, thanks!
>
> I still have to learn lots of things. So please don't spend much time
> explaining me, but instead please give me any pointers to
> documentation or source code that can be useful. As a curiosity, I'm
> pasting the results from my laptop. I'll repeat the same tests using
> my desktop as the server.
>
> Notice there is an assert being triggered, so I guess I'm running a
> build with debugging code ?!. I compiled using ./configure
> --with-radosgw --with-rest-bench followed by make.
>
> Thanks a lot for the attention.
>
> Best regards!
> Mello
>
> root@l3:/etc/init.d# rest-bench --api-host=localhost --bucket=test
> --access-key=JJABVJ3AWBS1ZOCML7NS
> --secret=A+ecBz2+Sdxj4Y8Mo+u3akIewGvJPkwOhwRgPKkX --protocol=http
> --uri_style=path write
> host=localhost
> Maintaining 16 concurrent writes of 4194304 bytes for at least 60 seconds.
> Object prefix: benchmark_data_l3_4032
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 0 3 3 0 0 0 - 0
> 1 16 16 0 0 0 - 0
> 2 16 16 0 0 0 - 0
> 3 16 16 0 0 0 - 0
> 4 16 16 0 0 0 - 0
> 5 16 16 0 0 0 - 0
> 6 16 16 0 0 0 - 0
> 7 16 16 0 0 0 - 0
> 8 16 16 0 0 0 - 0
> 9 16 16 0 0 0 - 0
> 10 16 16 0 0 0 - 0
> 11 16 16 0 0 0 - 0
> 12 16 17 1 0.333265 0.333333 11.2761 11.2761
> 13 16 18 2 0.615257 4 12.5964 11.9363
> 14 16 20 4 1.14262 8 13.1392 12.5365
> 15 16 23 7 1.86628 12 14.2273 13.2594
> 16 16 27 11 2.74944 16 15.0222 13.8968
> 17 16 32 16 3.76394 20 16.2604 14.6301
> 18 16 32 16 3.55483 0 - 14.6301
> 19 16 34 18 3.7887 4 6.2274 13.7695
> 2013-01-27 20:49:29.703509min lat: 6.2274 max lat: 16.2604 avg lat: 13.7695
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 20 16 34 18 3.59927 0 - 13.7695
> 21 16 34 18 3.42787 0 - 13.7695
> 22 16 34 18 3.27205 0 - 13.7695
> 23 16 35 19 3.30367 1 9.09053 13.5233
> 24 16 36 20 3.33264 4 9.09667 13.302
> 25 16 36 20 3.19933 0 - 13.302
> 26 16 36 20 3.07628 0 - 13.302
> 27 16 37 21 3.11047 1.33333 11.2459 13.204
> 28 16 37 21 2.99938 0 - 13.204
> 29 16 37 21 2.89595 0 - 13.204
> 30 16 37 21 2.79942 0 - 13.204
> 31 16 37 21 2.70912 0 - 13.204
> 32 16 39 23 2.87441 1.6 14.9981 13.3602
> 33 16 39 23 2.7873 0 - 13.3602
> 34 16 39 23 2.70533 0 - 13.3602
> 35 16 40 24 2.74229 1.33333 21.5365 13.7009
> 36 16 40 24 2.66612 0 - 13.7009
> 37 16 42 26 2.81023 4 22.6059 14.3855
> 38 16 42 26 2.73628 0 - 14.3855
> 39 16 45 29 2.97374 6 23.2615 15.3025
> 2013-01-27 20:49:49.707740min lat: 6.2274 max lat: 23.4496 avg lat: 16.1307
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 40 16 51 35 3.49927 24 21.0123 16.1307
> 41 16 51 35 3.41392 0 - 16.1307
> 42 16 52 36 3.42786 2 19.0243 16.211
> 43 16 52 36 3.34814 0 - 16.211
> 44 16 52 36 3.27204 0 - 16.211
> 45 16 52 36 3.19933 0 - 16.211
> 46 16 53 37 3.21672 1 11.0793 16.0723
> 47 16 53 37 3.14828 0 - 16.0723
> 48 16 53 37 3.08269 0 - 16.0723
> 49 16 53 37 3.01978 0 - 16.0723
> 50 16 55 39 3.11935 2 23.4253 16.5245
> 51 16 55 39 3.05819 0 - 16.5245
> 52 16 58 42 3.2301 6 13.3052 16.2945
> 53 16 58 42 3.16915 0 - 16.2945
> 54 16 60 44 3.25858 4 14.2116 16.1999
> 55 16 60 44 3.19933 0 - 16.1999
> 56 16 63 47 3.35644 6 15.6276 16.3393
> 57 16 63 47 3.29756 0 - 16.3393
> 58 16 66 50 3.44755 6 18.0547 16.5454
> 59 16 66 50 3.38912 0 - 16.5454
> 2013-01-27 20:50:09.712027min lat: 6.2274 max lat: 26.3519 avg lat: 16.5454
> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
> 60 16 66 50 3.33263 0 - 16.5454
> 61 16 66 50 3.278 0 - 16.5454
> 62 16 66 50 3.22513 0 - 16.5454
> 63 16 67 51 3.23741 0.8 10.9553 16.4358
> 64 16 67 51 3.18683 0 - 16.4358
> 65 16 67 51 3.1378 0 - 16.4358
> 66 16 67 51 3.09025 0 - 16.4358
> 67 16 67 51 3.04413 0 - 16.4358
> 68 16 67 51 2.99937 0 - 16.4358
> 69 16 67 51 2.9559 0 - 16.4358
> 70 16 67 51 2.91367 0 - 16.4358
> 71 14 67 53 2.98528 1 15.1841 16.2524
> 72 9 67 58 3.22154 20 26.2114 16.6243
> Total time run: 72.403588
> Total writes made: 67
> Write size: 4194304
> Bandwidth (MB/sec): 3.701
>
> Stddev Bandwidth: 4.92554
> Max bandwidth (MB/sec): 24
> Min bandwidth (MB/sec): 0
> Average Latency: 17.1886
> Stddev Latency: 5.48181
> Max latency: 32.7038
> Min latency: 6.2274
Can you run 'rados bench' on the same setup (with the same params)?
The numbers are really horrible.
Yehuda
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: RadosGW performance and disk space usage
2013-01-28 17:35 ` Yehuda Sadeh
@ 2013-01-28 18:18 ` Cesar Mello
0 siblings, 0 replies; 11+ messages in thread
From: Cesar Mello @ 2013-01-28 18:18 UTC (permalink / raw)
To: Yehuda Sadeh; +Cc: Marcus Sorensen, Sam Lang, ceph-devel@vger.kernel.org
Sure I can later when I arrive home. With the end of my vacation, I'll
be able to devote a couple of hours after my 3-year-old sleeps. :-)
I guess my laptop hard disk has horrible seek times. I'll repeat the
tests in my desktop as soon as possible.
Thanks a lot for the attention!
Best regards
Mello
On Mon, Jan 28, 2013 at 3:35 PM, Yehuda Sadeh <yehuda@inktank.com> wrote:
> On Sun, Jan 27, 2013 at 3:10 PM, Cesar Mello <cmello@gmail.com> wrote:
>> Hi,
>>
>> Just tried rest-bench. This little tool is wonderful, thanks!
>>
>> I still have to learn lots of things. So please don't spend much time
>> explaining me, but instead please give me any pointers to
>> documentation or source code that can be useful. As a curiosity, I'm
>> pasting the results from my laptop. I'll repeat the same tests using
>> my desktop as the server.
>>
>> Notice there is an assert being triggered, so I guess I'm running a
>> build with debugging code ?!. I compiled using ./configure
>> --with-radosgw --with-rest-bench followed by make.
>>
>> Thanks a lot for the attention.
>>
>> Best regards!
>> Mello
>>
>> root@l3:/etc/init.d# rest-bench --api-host=localhost --bucket=test
>> --access-key=JJABVJ3AWBS1ZOCML7NS
>> --secret=A+ecBz2+Sdxj4Y8Mo+u3akIewGvJPkwOhwRgPKkX --protocol=http
>> --uri_style=path write
>> host=localhost
>> Maintaining 16 concurrent writes of 4194304 bytes for at least 60 seconds.
>> Object prefix: benchmark_data_l3_4032
>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>> 0 3 3 0 0 0 - 0
>> 1 16 16 0 0 0 - 0
>> 2 16 16 0 0 0 - 0
>> 3 16 16 0 0 0 - 0
>> 4 16 16 0 0 0 - 0
>> 5 16 16 0 0 0 - 0
>> 6 16 16 0 0 0 - 0
>> 7 16 16 0 0 0 - 0
>> 8 16 16 0 0 0 - 0
>> 9 16 16 0 0 0 - 0
>> 10 16 16 0 0 0 - 0
>> 11 16 16 0 0 0 - 0
>> 12 16 17 1 0.333265 0.333333 11.2761 11.2761
>> 13 16 18 2 0.615257 4 12.5964 11.9363
>> 14 16 20 4 1.14262 8 13.1392 12.5365
>> 15 16 23 7 1.86628 12 14.2273 13.2594
>> 16 16 27 11 2.74944 16 15.0222 13.8968
>> 17 16 32 16 3.76394 20 16.2604 14.6301
>> 18 16 32 16 3.55483 0 - 14.6301
>> 19 16 34 18 3.7887 4 6.2274 13.7695
>> 2013-01-27 20:49:29.703509min lat: 6.2274 max lat: 16.2604 avg lat: 13.7695
>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>> 20 16 34 18 3.59927 0 - 13.7695
>> 21 16 34 18 3.42787 0 - 13.7695
>> 22 16 34 18 3.27205 0 - 13.7695
>> 23 16 35 19 3.30367 1 9.09053 13.5233
>> 24 16 36 20 3.33264 4 9.09667 13.302
>> 25 16 36 20 3.19933 0 - 13.302
>> 26 16 36 20 3.07628 0 - 13.302
>> 27 16 37 21 3.11047 1.33333 11.2459 13.204
>> 28 16 37 21 2.99938 0 - 13.204
>> 29 16 37 21 2.89595 0 - 13.204
>> 30 16 37 21 2.79942 0 - 13.204
>> 31 16 37 21 2.70912 0 - 13.204
>> 32 16 39 23 2.87441 1.6 14.9981 13.3602
>> 33 16 39 23 2.7873 0 - 13.3602
>> 34 16 39 23 2.70533 0 - 13.3602
>> 35 16 40 24 2.74229 1.33333 21.5365 13.7009
>> 36 16 40 24 2.66612 0 - 13.7009
>> 37 16 42 26 2.81023 4 22.6059 14.3855
>> 38 16 42 26 2.73628 0 - 14.3855
>> 39 16 45 29 2.97374 6 23.2615 15.3025
>> 2013-01-27 20:49:49.707740min lat: 6.2274 max lat: 23.4496 avg lat: 16.1307
>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>> 40 16 51 35 3.49927 24 21.0123 16.1307
>> 41 16 51 35 3.41392 0 - 16.1307
>> 42 16 52 36 3.42786 2 19.0243 16.211
>> 43 16 52 36 3.34814 0 - 16.211
>> 44 16 52 36 3.27204 0 - 16.211
>> 45 16 52 36 3.19933 0 - 16.211
>> 46 16 53 37 3.21672 1 11.0793 16.0723
>> 47 16 53 37 3.14828 0 - 16.0723
>> 48 16 53 37 3.08269 0 - 16.0723
>> 49 16 53 37 3.01978 0 - 16.0723
>> 50 16 55 39 3.11935 2 23.4253 16.5245
>> 51 16 55 39 3.05819 0 - 16.5245
>> 52 16 58 42 3.2301 6 13.3052 16.2945
>> 53 16 58 42 3.16915 0 - 16.2945
>> 54 16 60 44 3.25858 4 14.2116 16.1999
>> 55 16 60 44 3.19933 0 - 16.1999
>> 56 16 63 47 3.35644 6 15.6276 16.3393
>> 57 16 63 47 3.29756 0 - 16.3393
>> 58 16 66 50 3.44755 6 18.0547 16.5454
>> 59 16 66 50 3.38912 0 - 16.5454
>> 2013-01-27 20:50:09.712027min lat: 6.2274 max lat: 26.3519 avg lat: 16.5454
>> sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
>> 60 16 66 50 3.33263 0 - 16.5454
>> 61 16 66 50 3.278 0 - 16.5454
>> 62 16 66 50 3.22513 0 - 16.5454
>> 63 16 67 51 3.23741 0.8 10.9553 16.4358
>> 64 16 67 51 3.18683 0 - 16.4358
>> 65 16 67 51 3.1378 0 - 16.4358
>> 66 16 67 51 3.09025 0 - 16.4358
>> 67 16 67 51 3.04413 0 - 16.4358
>> 68 16 67 51 2.99937 0 - 16.4358
>> 69 16 67 51 2.9559 0 - 16.4358
>> 70 16 67 51 2.91367 0 - 16.4358
>> 71 14 67 53 2.98528 1 15.1841 16.2524
>> 72 9 67 58 3.22154 20 26.2114 16.6243
>> Total time run: 72.403588
>> Total writes made: 67
>> Write size: 4194304
>> Bandwidth (MB/sec): 3.701
>>
>> Stddev Bandwidth: 4.92554
>> Max bandwidth (MB/sec): 24
>> Min bandwidth (MB/sec): 0
>> Average Latency: 17.1886
>> Stddev Latency: 5.48181
>> Max latency: 32.7038
>> Min latency: 6.2274
>
> Can you run 'rados bench' on the same setup (with the same params)?
> The numbers are really horrible.
>
> Yehuda
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2013-01-28 18:18 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-01-24 15:27 RadosGW performance and disk space usage Cesar Mello
2013-01-25 23:14 ` Sam Lang
2013-01-26 5:35 ` Dan Mick
2013-01-28 2:32 ` Dan Mick
2013-01-26 5:44 ` Marcus Sorensen
[not found] ` <CALFpzo4u2PspjkvSig3LL821YY_XA2iHBjAh30JxgwWTFGsgcA@mail.gmail.com>
2013-01-26 14:20 ` Cesar Mello
2013-01-27 23:10 ` Cesar Mello
2013-01-28 14:28 ` Joao Eduardo Luis
2013-01-28 17:33 ` Yehuda Sadeh
2013-01-28 17:35 ` Yehuda Sadeh
2013-01-28 18:18 ` Cesar Mello
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.