rbd create error with 0.26

rbd create error with 0.26

[Simon Tian]

Hi folks,

I met an error when creating a rbd image:

====================================================
[root@mon-00 ~]# ceph class list
2011-05-10 17:36:29.530759 mon <- [class,list]
2011-05-10 17:36:29.531491 mon0 -> 'no installed classes!' (0)
[root@mon-00 ~]# cclass -a
Loading class: /usr/local/lib/rados-classes/libcls_rbd.so.1.0.0: rbd 1.3 x86-64
2011-05-10 17:36:37.657254 7eff9cfc36f0 read 184650 bytes from
/usr/local/lib/rados-classes/libcls_rbd.so.1.0.0
2011-05-10 17:36:37.658820 mon <- [class,add,rbd,1.3,x86-64,changed]
2011-05-10 17:36:37.749791 mon0 -> 'updated' (0)
[root@mon-00 ~]# ceph class list
2011-05-10 17:36:45.176860 mon <- [class,list]
2011-05-10 17:36:45.177511 mon1 -> 'installed classes:
rbd (v1.3 [x86-64]) [active]
' (0)
[root@mon-00 ~]# rbd list
pool rbd doesn't contain rbd images
[root@mon-00 ~]# rbd list
pool rbd doesn't contain rbd images
[root@mon-00 ~]# rbd create foo --size 1024
2011-05-10 17:42:14.854907 7fcff83cb6f0 librbd: failed to assign a
block name for image
create error: Input/output error
[root@mon-00 ~]# ceph class activate rbd 1.3
2011-05-10 17:42:34.824676 mon <- [class,activate,rbd,1.3]
2011-05-10 17:42:34.944398 mon2 -> 'updated' (0)
[root@mon-00 ~]# rbd create foo --size 1024
2011-05-10 17:42:44.227398 7fa9411866f0 librbd: failed to assign a
block name for image
create error: Input/output error
====================================================
I just done as this tracker said:  http://tracker.newdream.net/issues/263

I found that the return value of io_ctx.exec function is bellow zero:

  r = io_ctx.exec(info_oid, "rbd", "assign_bid", bl, out);
  if (r < 0)
    return r;

Thx!

Re: rbd create error with 0.26

[Yehuda Sadeh Weinraub]

On Mon, May 9, 2011 at 7:30 PM, Simon Tian <aixt2006@gmail.com> wrote:
> [root@mon-00 ~]# rbd create foo --size 1024
> 2011-05-10 17:42:44.227398 7fa9411866f0 librbd: failed to assign a
> block name for image
> create error: Input/output error
> ====================================================
> I just done as this tracker said:  http://tracker.newdream.net/issues/263
>
> I found that the return value of io_ctx.exec function is bellow zero:
>
>  r = io_ctx.exec(info_oid, "rbd", "assign_bid", bl, out);
>  if (r < 0)
>    return r;
>

Can you verify that the osd has a /var/lib/ceph/tmp directory, which
is writable by the cosd daemon, and that it doesn't have a noexec
mount option?

Thanks,
Yehuda
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rbd create error with 0.26

[Fyodor Ustinov]

Simon Tian <aixt2006 <at> gmail.com> writes:

> [root <at> mon-00 ~]# cclass -a
> [root <at> mon-00 ~]# ceph class list

Do you forget "ceph class activate rbd 1.3" ?

Re: rbd create error with 0.26

[Simon Tian]

Sorry, I didn't learn the wiki carefullllllly..

There is another problem:
====================================================
[root@mon-00 ~]# rbd create foo --size 1024
2011-05-11 02:37:15.400313 42d6a940 -- 10.250.6.98:0/1010326 >>
10.250.6.30:6801/3051 pipe(0x63f140 sd=6 pgs=0 cs=0 l=0).fault first
fault
terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
  what():  buffer::end_of_buffer
*** Caught signal (Aborted) **
 in thread 0x7f9cbfc6c6f0
*** Caught signal (Aborted) **
 in thread 0x7f9cbfc6c6f0
 ceph version 0.26 (commit:9981ff90968398da43c63106694d661f5e3d07d5)
 1: /usr/local/lib/librados.so.2 [0x7f9cbfff1b98]
 2: /lib64/libpthread.so.0 [0x324ea0e7c0]
 3: (gsignal()+0x35) [0x324de30265]
 4: (abort()+0x110) [0x324de31d10]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x114) [0x32610bec44]
 6: /usr/lib64/libstdc++.so.6 [0x32610bcdb6]
 7: /usr/lib64/libstdc++.so.6 [0x32610bcde3]
 8: /usr/lib64/libstdc++.so.6 [0x32610bceca]
 9: (librbd::rbd_assign_bid(librados::IoCtx&, std::string&, unsigned
long*)+0x227) [0x7f9cc02726c7]
 10: (librbd::create(librados::IoCtx&, std::string&, char const*,
unsigned long, int*)+0x115) [0x7f9cc0272865]
 11: (librbd::RBD::create(librados::IoCtx&, char const*, unsigned
long, int*)+0x61) [0x7f9cc0273141]
 12: (__gxx_personality_v0()+0x489) [0x403e59]
 13: (main()+0x177c) [0x4063dc]
 14: (__libc_start_main()+0xf4) [0x324de1d994]
 15: (__gxx_personality_v0()+0xc9) [0x403a99]
Aborted
================================================================

cclass -a should run on all host of the cluster or any one host or the
host to run rbd create?

Thx very much!

Simon


2011/5/10 Yehuda Sadeh Weinraub <yehudasa@gmail.com>:
> On Mon, May 9, 2011 at 7:30 PM, Simon Tian <aixt2006@gmail.com> wrote:
>> [root@mon-00 ~]# rbd create foo --size 1024
>> 2011-05-10 17:42:44.227398 7fa9411866f0 librbd: failed to assign a
>> block name for image
>> create error: Input/output error
>> ====================================================
>> I just done as this tracker said:  http://tracker.newdream.net/issues/263
>>
>> I found that the return value of io_ctx.exec function is bellow zero:
>>
>>  r = io_ctx.exec(info_oid, "rbd", "assign_bid", bl, out);
>>  if (r < 0)
>>    return r;
>>
>
> Can you verify that the osd has a /var/lib/ceph/tmp directory, which
> is writable by the cosd daemon, and that it doesn't have a noexec
> mount option?
>
> Thanks,
> Yehuda
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rbd create error with 0.26

[Simon Tian]

2011/5/10 Fyodor Ustinov <ufm@ufm.su>:
> Simon Tian <aixt2006 <at> gmail.com> writes:
>
>> [root <at> mon-00 ~]# cclass -a
>> [root <at> mon-00 ~]# ceph class list
>
> Do you forget "ceph class activate rbd 1.3" ?
>

As the log show, the rbd 1.3 is already active:

[root@mon-00 ~]# rbd create foo --size 1024
2011-05-11 05:46:59.074470 42fb0940 -- 10.250.6.98:0/1014581 >>
10.250.6.30:6801/3240 pipe(0x63ee70 sd=5 pgs=0 cs=0 l=0).fault first
fault


Thx!

Simon


[root@mon-00 ~]# cclass -a
Loading class: /usr/local/lib/rados-classes/libcls_rbd.so.1.0.0: rbd 1.3 x86-64
2011-05-10 17:36:37.657254 7eff9cfc36f0 read 184650 bytes from
/usr/local/lib/rados-classes/libcls_rbd.so.1.0.0
2011-05-10 17:36:37.658820 mon <- [class,add,rbd,1.3,x86-64,changed]
2011-05-10 17:36:37.749791 mon0 -> 'updated' (0)
[root@mon-00 ~]# ceph class list
2011-05-10 17:36:45.176860 mon <- [class,list]
2011-05-10 17:36:45.177511 mon1 -> 'installed classes:
rbd (v1.3 [x86-64]) [active]
                            ^^^^^^^
' (0)

After I done "ceph class activate rbd 1.3" , It

Re: rbd create error with 0.26

[Yehuda Sadeh Weinraub]

On Tue, May 10, 2011 at 6:39 AM, Simon Tian <aixt2006@gmail.com> wrote:
> Sorry, I didn't learn the wiki carefullllllly..
>
> There is another problem:
> ====================================================
> [root@mon-00 ~]# rbd create foo --size 1024
> 2011-05-11 02:37:15.400313 42d6a940 -- 10.250.6.98:0/1010326 >>
> 10.250.6.30:6801/3051 pipe(0x63f140 sd=6 pgs=0 cs=0 l=0).fault first
> fault

This looks like your osd is not reachable, might have crashed?

> terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
>  what():  buffer::end_of_buffer
> *** Caught signal (Aborted) **

This shouldn't happen anyway, but probably bad error handling. Can you
verify whether one of the daemons went down?

Thanks,
Yehuda
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rbd create error with 0.26

[Simon Tian]

Hi,

    As you said, one of the osds crashed:
================= log ========================
2011-05-10 21:46:38.990311 4bc90940 osd2 8 pg[3.13a( v 8'1 (0'0,8'1]
n=1 ec=2 les=6 5/5/4) [2,3] r=0 mlcod 0'0 active+clean]
oi.user_version=8'2 is_modify=0
2011-05-10 21:46:38.990386 4bc90940 osd2 8 pg[3.13a( v 8'1 (0'0,8'1]
n=1 ec=2 les=6 5/5/4) [2,3] r=0 mlcod 0'0 active+clean]
oi.user_version=8'2 is_modify=1
*** Caught signal (Segmentation fault) **
 in thread 0x45382940
=========================================

I tried again, this time, i done "rbd create foo --size 1024"
successfully, but when I run the code of testlibrbd.c, one of the osds
crash again:
================= log ========================
2011-05-10 22:08:20.008871 4c115940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean
snaptrimq=[1~1]] dump_watchers testimg.rbd/head testimg.rbd/head(9'4
client4107.0:14 wrlock_by=unknown0.0:0)
2011-05-10 22:08:20.008903 4c115940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean
snaptrimq=[1~1]]  * obc->watcher: client4107 session=0xc80990
2011-05-10 22:08:20.008925 4c115940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean
snaptrimq=[1~1]]  * oi->watcher: client4107 cookie=2
2011-05-10 22:08:20.009232 4b914940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean]
oi.user_version=10'5 is_modify=1
2011-05-10 22:08:20.009267 4b914940 expires 2011-05-10 23:08:19.890032
now 2011-05-10 22:08:20.009260
2011-05-10 22:08:20.009284 napshots_list
2011-05-10 22:08:20.009307 4b914940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean]
oi.user_version=10'5 is_modify=0
2011-05-10 22:08:20.009375 4b914940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean]
oi.user_version=10'5 is_modify=1
*** Caught signal (Segmentation fault) **
 in thread 0x4eb1c940
=========================================

Thx!

Simon


2011/5/10 Yehuda Sadeh Weinraub <yehudasa@gmail.com>:
> On Tue, May 10, 2011 at 6:39 AM, Simon Tian <aixt2006@gmail.com> wrote:
>> Sorry, I didn't learn the wiki carefullllllly..
>>
>> There is another problem:
>> ====================================================
>> [root@mon-00 ~]# rbd create foo --size 1024
>> 2011-05-11 02:37:15.400313 42d6a940 -- 10.250.6.98:0/1010326 >>
>> 10.250.6.30:6801/3051 pipe(0x63f140 sd=6 pgs=0 cs=0 l=0).fault first
>> fault
>
> This looks like your osd is not reachable, might have crashed?
>
>> terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
>>  what():  buffer::end_of_buffer
>> *** Caught signal (Aborted) **
>
> This shouldn't happen anyway, but probably bad error handling. Can you
> verify whether one of the daemons went down?
>
> Thanks,
> Yehuda
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rbd create error with 0.26

[Simon Tian]

YES!! After try several times if I create a rbd, one of the cosd will coredump.


2011/5/10 Yehuda Sadeh Weinraub <yehudasa@gmail.com>:
> On Tue, May 10, 2011 at 6:39 AM, Simon Tian <aixt2006@gmail.com> wrote:
>> Sorry, I didn't learn the wiki carefullllllly..
>>
>> There is another problem:
>> ====================================================
>> [root@mon-00 ~]# rbd create foo --size 1024
>> 2011-05-11 02:37:15.400313 42d6a940 -- 10.250.6.98:0/1010326 >>
>> 10.250.6.30:6801/3051 pipe(0x63f140 sd=6 pgs=0 cs=0 l=0).fault first
>> fault
>
> This looks like your osd is not reachable, might have crashed?
>
>> terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
>>  what():  buffer::end_of_buffer
>> *** Caught signal (Aborted) **
>
> This shouldn't happen anyway, but probably bad error handling. Can you
> verify whether one of the daemons went down?
>
> Thanks,
> Yehuda
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rbd create error with 0.26

[Yehuda Sadeh Weinraub]

On Tue, May 10, 2011 at 7:15 AM, Simon Tian <aixt2006@gmail.com> wrote:
> Hi,
>
>    As you said, one of the osds crashed:
> ================= log ========================
> 2011-05-10 21:46:38.990311 4bc90940 osd2 8 pg[3.13a( v 8'1 (0'0,8'1]
> n=1 ec=2 les=6 5/5/4) [2,3] r=0 mlcod 0'0 active+clean]
> oi.user_version=8'2 is_modify=0
> 2011-05-10 21:46:38.990386 4bc90940 osd2 8 pg[3.13a( v 8'1 (0'0,8'1]
> n=1 ec=2 les=6 5/5/4) [2,3] r=0 mlcod 0'0 active+clean]
> oi.user_version=8'2 is_modify=1
> *** Caught signal (Segmentation fault) **
>  in thread 0x45382940
> =========================================
>
> I tried again, this time, i done "rbd create foo --size 1024"
> successfully, but when I run the code of testlibrbd.c, one of the osds
> crash again:
> ================= log ========================
> 2011-05-10 22:08:20.008871 4c115940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean
> snaptrimq=[1~1]] dump_watchers testimg.rbd/head testimg.rbd/head(9'4
> client4107.0:14 wrlock_by=unknown0.0:0)
> 2011-05-10 22:08:20.008903 4c115940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean
> snaptrimq=[1~1]]  * obc->watcher: client4107 session=0xc80990
> 2011-05-10 22:08:20.008925 4c115940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean
> snaptrimq=[1~1]]  * oi->watcher: client4107 cookie=2
> 2011-05-10 22:08:20.009232 4b914940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean]
> oi.user_version=10'5 is_modify=1
> 2011-05-10 22:08:20.009267 4b914940 expires 2011-05-10 23:08:19.890032
> now 2011-05-10 22:08:20.009260
> 2011-05-10 22:08:20.009284 napshots_list
> 2011-05-10 22:08:20.009307 4b914940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean]
> oi.user_version=10'5 is_modify=0
> 2011-05-10 22:08:20.009375 4b914940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean]
> oi.user_version=10'5 is_modify=1
> *** Caught signal (Segmentation fault) **
>  in thread 0x4eb1c940
> =========================================
>
Can you by any chance get backtrace for that crash (gdb cosd core;
bt)? You might need to have the debug packages installed.
Also, note that you're not running the latest version so you might be
hitting something that was already fixed (not that I remember anything
specific, but it might be worth a try).

Thanks,
Yehuda
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rbd create error with 0.26

[Simon Tian]

Hope this help:
===================================================
GNU gdb (GDB) Fedora (7.0.50.20100203-15.el5)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/bin/cosd...done.
[New Thread 15605]
[New Thread 15366]
[New Thread 15365]
[New Thread 15228]
[New Thread 15469]
[New Thread 15226]
[New Thread 15233]
[New Thread 15607]
[New Thread 15227]
[New Thread 15240]
[New Thread 15254]
[New Thread 15229]
[New Thread 15243]
[New Thread 15255]
[New Thread 15253]
[New Thread 15230]
[New Thread 15247]
[New Thread 15256]
[New Thread 15231]
[New Thread 15252]
[New Thread 15232]
[New Thread 15311]
[New Thread 15302]
[New Thread 15234]
[New Thread 15470]
[New Thread 17426]
[New Thread 15237]
[New Thread 15473]
[New Thread 15299]
[New Thread 15475]
[New Thread 15303]
[New Thread 15246]
[New Thread 15304]
[New Thread 15245]
[New Thread 15606]
[New Thread 15307]
[New Thread 15241]
[New Thread 15309]
[New Thread 15251]
[New Thread 15242]
[New Thread 15239]
[New Thread 15308]
[New Thread 15244]
[New Thread 15236]
[New Thread 15248]
[New Thread 17425]
[New Thread 15235]
[New Thread 15250]
[New Thread 15609]
[New Thread 15238]
[New Thread 15249]
Error while mapping shared library sections:
/opt/ceph_var/tmp/class-AVGRjS: No such file or directory.
Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libnspr4.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libnspr4.so
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /usr/lib64/libplc4.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libplc4.so
Reading symbols from /usr/lib64/libplds4.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libplds4.so
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging
symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libsoftokn3.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libsoftokn3.so
Reading symbols from /usr/lib64/libfreebl3.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libfreebl3.so
Symbol file not found for /opt/ceph_var/tmp/class-AVGRjS
Core was generated by `/usr/local/bin/cosd -i 2 -c /tmp/ceph.conf.23025'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000003681a075c8 in ?? () from /lib64/libgcc_s.so.1
(gdb) bt
#0  0x0000003681a075c8 in ?? () from /lib64/libgcc_s.so.1
#1  0x0000003681a0882b in _Unwind_Backtrace () from /lib64/libgcc_s.so.1
#2  0x000000367fae5eb8 in backtrace () from /lib64/libc.so.6
#3  0x0000000000661318 in BackTrace (signum=11) at ./common/BackTrace.h:19
#4  handle_fatal_signal (signum=11) at common/signal.cc:72
#5  <signal handler called>
#6  0x0000000000000031 in ?? ()
#7  0x00007fbda871e390 in ?? ()
#8  0x0000000000000000 in ?? ()
(gdb)
=====================================================

If helpless, I will got more trace info.
BTW, where could I get the debug packages?


2011/5/10 Yehuda Sadeh Weinraub <yehudasa@gmail.com>:
> On Tue, May 10, 2011 at 7:15 AM, Simon Tian <aixt2006@gmail.com> wrote:
>> Hi,
>>
>>    As you said, one of the osds crashed:
>> ================= log ========================
>> 2011-05-10 21:46:38.990311 4bc90940 osd2 8 pg[3.13a( v 8'1 (0'0,8'1]
>> n=1 ec=2 les=6 5/5/4) [2,3] r=0 mlcod 0'0 active+clean]
>> oi.user_version=8'2 is_modify=0
>> 2011-05-10 21:46:38.990386 4bc90940 osd2 8 pg[3.13a( v 8'1 (0'0,8'1]
>> n=1 ec=2 les=6 5/5/4) [2,3] r=0 mlcod 0'0 active+clean]
>> oi.user_version=8'2 is_modify=1
>> *** Caught signal (Segmentation fault) **
>>  in thread 0x45382940
>> =========================================
>>
>> I tried again, this time, i done "rbd create foo --size 1024"
>> successfully, but when I run the code of testlibrbd.c, one of the osds
>> crash again:
>> ================= log ========================
>> 2011-05-10 22:08:20.008871 4c115940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
>> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean
>> snaptrimq=[1~1]] dump_watchers testimg.rbd/head testimg.rbd/head(9'4
>> client4107.0:14 wrlock_by=unknown0.0:0)
>> 2011-05-10 22:08:20.008903 4c115940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
>> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean
>> snaptrimq=[1~1]]  * obc->watcher: client4107 session=0xc80990
>> 2011-05-10 22:08:20.008925 4c115940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
>> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean
>> snaptrimq=[1~1]]  * oi->watcher: client4107 cookie=2
>> 2011-05-10 22:08:20.009232 4b914940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
>> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean]
>> oi.user_version=10'5 is_modify=1
>> 2011-05-10 22:08:20.009267 4b914940 expires 2011-05-10 23:08:19.890032
>> now 2011-05-10 22:08:20.009260
>> 2011-05-10 22:08:20.009284 napshots_list
>> 2011-05-10 22:08:20.009307 4b914940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
>> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean]
>> oi.user_version=10'5 is_modify=0
>> 2011-05-10 22:08:20.009375 4b914940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
>> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean]
>> oi.user_version=10'5 is_modify=1
>> *** Caught signal (Segmentation fault) **
>>  in thread 0x4eb1c940
>> =========================================
>>
> Can you by any chance get backtrace for that crash (gdb cosd core;
> bt)? You might need to have the debug packages installed.
> Also, note that you're not running the latest version so you might be
> hitting something that was already fixed (not that I remember anything
> specific, but it might be worth a try).
>
> Thanks,
> Yehuda
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rbd create error with 0.26

[Josh Durgin]

On 05/10/2011 06:54 PM, Simon Tian wrote:
> If helpless, I will got more trace info.
> BTW, where could I get the debug packages?

Unfortunately, the backtrace isn't very useful without debugging 
symbols, and I don't see any Fedora packages that include them.

You can create a package with debugging symbols using the srpm by adding 
this to the ceph.spec file:

RPM_OPT_FLAGS="$RPM_OPT_FLAGS -g"
export RPM_OPT_FLAGS

It looks like the most recent srpm is here:

http://download.fedora.redhat.com/pub/fedora/linux/development/rawhide/source/SRPMS/ceph-0.26-2.fc16.src.rpm

It might be better to build 0.27 from source, in case this is a bug that 
was already fixed.

-Josh

> 2011/5/10 Yehuda Sadeh Weinraub<yehudasa@gmail.com>:
>> On Tue, May 10, 2011 at 7:15 AM, Simon Tian<aixt2006@gmail.com>  wrote:
>>> Hi,
>>>
>>>     As you said, one of the osds crashed:
>>> ================= log ========================
>>> 2011-05-10 21:46:38.990311 4bc90940 osd2 8 pg[3.13a( v 8'1 (0'0,8'1]
>>> n=1 ec=2 les=6 5/5/4) [2,3] r=0 mlcod 0'0 active+clean]
>>> oi.user_version=8'2 is_modify=0
>>> 2011-05-10 21:46:38.990386 4bc90940 osd2 8 pg[3.13a( v 8'1 (0'0,8'1]
>>> n=1 ec=2 les=6 5/5/4) [2,3] r=0 mlcod 0'0 active+clean]
>>> oi.user_version=8'2 is_modify=1
>>> *** Caught signal (Segmentation fault) **
>>>   in thread 0x45382940
>>> =========================================
>>>
>>> I tried again, this time, i done "rbd create foo --size 1024"
>>> successfully, but when I run the code of testlibrbd.c, one of the osds
>>> crash again:
>>> ================= log ========================
>>> 2011-05-10 22:08:20.008871 4c115940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
>>> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean
>>> snaptrimq=[1~1]] dump_watchers testimg.rbd/head testimg.rbd/head(9'4
>>> client4107.0:14 wrlock_by=unknown0.0:0)
>>> 2011-05-10 22:08:20.008903 4c115940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
>>> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean
>>> snaptrimq=[1~1]]  * obc->watcher: client4107 session=0xc80990
>>> 2011-05-10 22:08:20.008925 4c115940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
>>> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean
>>> snaptrimq=[1~1]]  * oi->watcher: client4107 cookie=2
>>> 2011-05-10 22:08:20.009232 4b914940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
>>> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean]
>>> oi.user_version=10'5 is_modify=1
>>> 2011-05-10 22:08:20.009267 4b914940 expires 2011-05-10 23:08:19.890032
>>> now 2011-05-10 22:08:20.009260
>>> 2011-05-10 22:08:20.009284 napshots_list
>>> 2011-05-10 22:08:20.009307 4b914940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
>>> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean]
>>> oi.user_version=10'5 is_modify=0
>>> 2011-05-10 22:08:20.009375 4b914940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
>>> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean]
>>> oi.user_version=10'5 is_modify=1
>>> *** Caught signal (Segmentation fault) **
>>>   in thread 0x4eb1c940
>>> =========================================
>>>
>> Can you by any chance get backtrace for that crash (gdb cosd core;
>> bt)? You might need to have the debug packages installed.
>> Also, note that you're not running the latest version so you might be
>> hitting something that was already fixed (not that I remember anything
>> specific, but it might be worth a try).
>>
>> Thanks,
>> Yehuda
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rbd create error with 0.26

[Simon Tian]

Hi guys,

     I did some test, 4 scenario:
(1) 5 osds in 2 hosts, ceph-0.26      one of the osds will core dump
when create several rbd.
(2) 4 osds in 4 hosts, ceph-0.26      OK
(3) 5 osds in 2 hosts, ceph-0.27.1   OK
(4) 4 osds in 4 hosts, ceph-0.27.1   OK


BTW, I have 2 question:
1.   In these scenario, after I execute "cclass -a" and "ceph class
activate rbd 1.3",  I need to wait for several second before create
rbd, otherwise, "librbd: failed to assign a block name for image" will
come out. Is this all right?

2.   I run some test with a modified testlibrbd.c, code add like:

gettimeofday(&tv1, NULL);
  for (i = 0; i < num_test; i++)
    write_test_data(image, test_data, TEST_IO_SIZE * i, TEST_IO_SIZE);
gettimeofday(&tv2, NULL);
t1 = tv2.tv_sec-tv1.tv_sec;
temp = (float)t1 + (tv2.tv_usec-tv1.tv_usec)/1000000.0;
speed = 1.0*TEST_IO_SIZE*num_test/temp/1024/1024;
printf("time used: temp=%.3f\n", temp);
printf("write speed: %.2f MB/s\n", speed);

The result I got is so slowly:
time used: temp=46.611
write speed: 0.21 MB/s
time used: temp=14.706
read speed: 0.68 MB/s
time used: temp=45.453
aio write speed: 0.22 MB/s
time used: temp=14.759
aio read speed: 0.68 MB/s

But while the test, some cosd process is running with high CPU usage.

Thx!
Simon


2011/5/10 Yehuda Sadeh Weinraub <yehudasa@gmail.com>:
> On Tue, May 10, 2011 at 7:15 AM, Simon Tian <aixt2006@gmail.com> wrote:
>> Hi,
>>
>>    As you said, one of the osds crashed:
>> ================= log ========================
>> 2011-05-10 21:46:38.990311 4bc90940 osd2 8 pg[3.13a( v 8'1 (0'0,8'1]
>> n=1 ec=2 les=6 5/5/4) [2,3] r=0 mlcod 0'0 active+clean]
>> oi.user_version=8'2 is_modify=0
>> 2011-05-10 21:46:38.990386 4bc90940 osd2 8 pg[3.13a( v 8'1 (0'0,8'1]
>> n=1 ec=2 les=6 5/5/4) [2,3] r=0 mlcod 0'0 active+clean]
>> oi.user_version=8'2 is_modify=1
>> *** Caught signal (Segmentation fault) **
>>  in thread 0x45382940
>> =========================================
>>
>> I tried again, this time, i done "rbd create foo --size 1024"
>> successfully, but when I run the code of testlibrbd.c, one of the osds
>> crash again:
>> ================= log ========================
>> 2011-05-10 22:08:20.008871 4c115940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
>> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean
>> snaptrimq=[1~1]] dump_watchers testimg.rbd/head testimg.rbd/head(9'4
>> client4107.0:14 wrlock_by=unknown0.0:0)
>> 2011-05-10 22:08:20.008903 4c115940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
>> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean
>> snaptrimq=[1~1]]  * obc->watcher: client4107 session=0xc80990
>> 2011-05-10 22:08:20.008925 4c115940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
>> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean
>> snaptrimq=[1~1]]  * oi->watcher: client4107 cookie=2
>> 2011-05-10 22:08:20.009232 4b914940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
>> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean]
>> oi.user_version=10'5 is_modify=1
>> 2011-05-10 22:08:20.009267 4b914940 expires 2011-05-10 23:08:19.890032
>> now 2011-05-10 22:08:20.009260
>> 2011-05-10 22:08:20.009284 napshots_list
>> 2011-05-10 22:08:20.009307 4b914940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
>> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean]
>> oi.user_version=10'5 is_modify=0
>> 2011-05-10 22:08:20.009375 4b914940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
>> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean]
>> oi.user_version=10'5 is_modify=1
>> *** Caught signal (Segmentation fault) **
>>  in thread 0x4eb1c940
>> =========================================
>>
> Can you by any chance get backtrace for that crash (gdb cosd core;
> bt)? You might need to have the debug packages installed.
> Also, note that you're not running the latest version so you might be
> hitting something that was already fixed (not that I remember anything
> specific, but it might be worth a try).
>
> Thanks,
> Yehuda
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rbd create error with 0.26

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 5133 bytes --]

On Fri, 13 May 2011, Simon Tian wrote:
> Hi guys,
> 
>      I did some test, 4 scenario:
> (1) 5 osds in 2 hosts, ceph-0.26      one of the osds will core dump
> when create several rbd.
> (2) 4 osds in 4 hosts, ceph-0.26      OK
> (3) 5 osds in 2 hosts, ceph-0.27.1   OK
> (4) 4 osds in 4 hosts, ceph-0.27.1   OK
> 
> 
> BTW, I have 2 question:
> 1.   In these scenario, after I execute "cclass -a" and "ceph class
> activate rbd 1.3",  I need to wait for several second before create
> rbd, otherwise, "librbd: failed to assign a block name for image" will
> come out. Is this all right?

For 0.27 and previous, yes.  The class distribution has removed in v0.28 
and it is now the administrator's (make install's, dpkg's, rpm's) 
responsibility to put the .so in the right directory.  This is much 
simpler all around and easier to debug.  If you are still having any class 
loading problems you should try the 'next' branch which will soon become 
v0.28.

> 2.   I run some test with a modified testlibrbd.c, code add like:
> 
> gettimeofday(&tv1, NULL);
>   for (i = 0; i < num_test; i++)
>     write_test_data(image, test_data, TEST_IO_SIZE * i, TEST_IO_SIZE);
> gettimeofday(&tv2, NULL);
> t1 = tv2.tv_sec-tv1.tv_sec;
> temp = (float)t1 + (tv2.tv_usec-tv1.tv_usec)/1000000.0;
> speed = 1.0*TEST_IO_SIZE*num_test/temp/1024/1024;
> printf("time used: temp=%.3f\n", temp);
> printf("write speed: %.2f MB/s\n", speed);
> 
> The result I got is so slowly:
> time used: temp=46.611
> write speed: 0.21 MB/s
> time used: temp=14.706
> read speed: 0.68 MB/s
> time used: temp=45.453
> aio write speed: 0.22 MB/s
> time used: temp=14.759
> aio read speed: 0.68 MB/s
> 
> But while the test, some cosd process is running with high CPU usage.

What is the IO size?  Is write_test_data synchronous?

For simple write benchmarking you can also use

	rados mkpool foo
	rados -p foo bench <seconds> write -b <blocksize> -t <threads>

and you'll see latency and throughput.  Blocksize defaults to 4M and 
"threads" (parallel IOs) default to 16, IIRC.

sage


> 
> Thx!
> Simon
> 
> 
> 2011/5/10 Yehuda Sadeh Weinraub <yehudasa@gmail.com>:
> > On Tue, May 10, 2011 at 7:15 AM, Simon Tian <aixt2006@gmail.com> wrote:
> >> Hi,
> >>
> >>    As you said, one of the osds crashed:
> >> ================= log ========================
> >> 2011-05-10 21:46:38.990311 4bc90940 osd2 8 pg[3.13a( v 8'1 (0'0,8'1]
> >> n=1 ec=2 les=6 5/5/4) [2,3] r=0 mlcod 0'0 active+clean]
> >> oi.user_version=8'2 is_modify=0
> >> 2011-05-10 21:46:38.990386 4bc90940 osd2 8 pg[3.13a( v 8'1 (0'0,8'1]
> >> n=1 ec=2 les=6 5/5/4) [2,3] r=0 mlcod 0'0 active+clean]
> >> oi.user_version=8'2 is_modify=1
> >> *** Caught signal (Segmentation fault) **
> >>  in thread 0x45382940
> >> =========================================
> >>
> >> I tried again, this time, i done "rbd create foo --size 1024"
> >> successfully, but when I run the code of testlibrbd.c, one of the osds
> >> crash again:
> >> ================= log ========================
> >> 2011-05-10 22:08:20.008871 4c115940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
> >> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean
> >> snaptrimq=[1~1]] dump_watchers testimg.rbd/head testimg.rbd/head(9'4
> >> client4107.0:14 wrlock_by=unknown0.0:0)
> >> 2011-05-10 22:08:20.008903 4c115940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
> >> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean
> >> snaptrimq=[1~1]]  * obc->watcher: client4107 session=0xc80990
> >> 2011-05-10 22:08:20.008925 4c115940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
> >> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean
> >> snaptrimq=[1~1]]  * oi->watcher: client4107 cookie=2
> >> 2011-05-10 22:08:20.009232 4b914940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
> >> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean]
> >> oi.user_version=10'5 is_modify=1
> >> 2011-05-10 22:08:20.009267 4b914940 expires 2011-05-10 23:08:19.890032
> >> now 2011-05-10 22:08:20.009260
> >> 2011-05-10 22:08:20.009284 napshots_list
> >> 2011-05-10 22:08:20.009307 4b914940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
> >> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean]
> >> oi.user_version=10'5 is_modify=0
> >> 2011-05-10 22:08:20.009375 4b914940 osd3 10 pg[4.1( v 9'4 (9'2,9'4]
> >> n=1 ec=9 les=9 9/9/9) [3,0] r=0 mlcod 9'3 active+clean]
> >> oi.user_version=10'5 is_modify=1
> >> *** Caught signal (Segmentation fault) **
> >>  in thread 0x4eb1c940
> >> =========================================
> >>
> > Can you by any chance get backtrace for that crash (gdb cosd core;
> > bt)? You might need to have the debug packages installed.
> > Also, note that you're not running the latest version so you might be
> > hitting something that was already fixed (not that I remember anything
> > specific, but it might be worth a try).
> >
> > Thanks,
> > Yehuda
> >
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: rbd create error with 0.26

[Simon Tian]

>> BTW, I have 2 question:
>> 1.   In these scenario, after I execute "cclass -a" and "ceph class
>> activate rbd 1.3",  I need to wait for several second before create
>> rbd, otherwise, "librbd: failed to assign a block name for image" will
>> come out. Is this all right?
>
> For 0.27 and previous, yes.  The class distribution has removed in v0.28
> and it is now the administrator's (make install's, dpkg's, rpm's)
> responsibility to put the .so in the right directory.  This is much
> simpler all around and easier to debug.  If you are still having any class
> loading problems you should try the 'next' branch which will soon become
> v0.28.

OK,  I'll try v0.28.
How to enable the debug of the libs (librbd, librados....)? I debug
testlibrbd in gdb to see the backtrace, but can't got the context in
the libs.


>> 2.   I run some test with a modified testlibrbd.c, code add like:
>>
>> gettimeofday(&tv1, NULL);
>>   for (i = 0; i < num_test; i++)
>>     write_test_data(image, test_data, TEST_IO_SIZE * i, TEST_IO_SIZE);
>> gettimeofday(&tv2, NULL);
>> t1 = tv2.tv_sec-tv1.tv_sec;
>> temp = (float)t1 + (tv2.tv_usec-tv1.tv_usec)/1000000.0;
>> speed = 1.0*TEST_IO_SIZE*num_test/temp/1024/1024;
>> printf("time used: temp=%.3f\n", temp);
>> printf("write speed: %.2f MB/s\n", speed);
>>
>> The result I got is so slowly:
>> time used: temp=46.611
>> write speed: 0.21 MB/s
>> time used: temp=14.706
>> read speed: 0.68 MB/s
>> time used: temp=45.453
>> aio write speed: 0.22 MB/s
>> time used: temp=14.759
>> aio read speed: 0.68 MB/s
>>
>> But while the test, some cosd process is running with high CPU usage.
>
> What is the IO size?  Is write_test_data synchronous?
>
> For simple write benchmarking you can also use
>
>        rados mkpool foo
>        rados -p foo bench <seconds> write -b <blocksize> -t <threads>
>
> and you'll see latency and throughput.  Blocksize defaults to 4M and
> "threads" (parallel IOs) default to 16, IIRC.

The IO size is TEST_IO_SIZE=512,  write_test_data is just synchronous,
it call the rbd_write( ) function.
aio write speed is 0.22 MB/s which called rbd_aio_write() function.
aio read speed is 0.68 MB/s which called  rbd_aio_read() function.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rbd create error with 0.26

[Simon Tian]

> What is the IO size?  Is write_test_data synchronous?
>
> For simple write benchmarking you can also use
>        rados mkpool foo
>        rados -p foo bench <seconds> write -b <blocksize> -t <threads>
>
> and you'll see latency and throughput.  Blocksize defaults to 4M and
> "threads" (parallel IOs) default to 16, IIRC.


Hi, Sage:

I just did the bench:
rados -p rbd bench 60 write -t 64   and    rados -p data bench 60 write -t 64
the avg throughput is about 46MB/s, one of the result is as follow.
But why it's slow with rbd api from <rbd/librbd.h>?
And I tried testlibrbdpp.cc, the result is more or less the same.
The attachments are the test codes. Could you run it on your platform please?

 sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
  40      63       482       419   41.8884        44   2.40044   2.40979
  41      63       494       431   42.0372        48   2.11044     2.406
  42      64       506       442   42.0837        44   2.11266   2.40229
  43      63       518       455   42.3139        52   2.33468    2.3982
  44      63       527       464   42.1703        36    2.4403   2.39559
  45      63       539       476   42.2995        48   2.19768   2.39413
  46      63       551       488   42.4232        48   2.51232    2.3928
  47      63       563       500   42.5416        48   2.18025   2.38958
  48      63       572       509   42.4051        36   2.27111   2.38791
  49      63       584       521    42.519        48   2.41684   2.38695
  50      63       596       533   42.6284        48   2.11087     2.384
  51      63       608       545   42.7335        48   2.18147   2.37925
  52      63       620       557   42.8345        48   2.45287   2.37787
  53      63       629       566   42.7054        36   2.45187   2.37801
  54      63       644       581   43.0255        60   2.22403   2.37477
  55      63       653       590   42.8976        36   2.22782   2.37157
  56      63       668       605   43.2026        60   2.20638   2.36597
  57      63       677       614   43.0761        36   2.19628   2.36209
  58      63       689       626   43.1608        48   2.18262   2.35762
  59      63       704       641   43.4459        60   2.27029   2.35352
min lat: 1.87981 max lat: 5.56194 avg lat: 2.34944
 sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
  60      63       716       653   43.5215        48   2.27835   2.34944
  61      64       717       653    42.808         0         -   2.34944
  62      63       717       654   42.1821         2   2.25694   2.34929
Total time run:        62.274719
Total writes made:     717
Write size:            4194304
Bandwidth (MB/sec):    46.054

Average Latency:       5.453
Max latency:           62.0339
Min latency:           1.87981

Re: rbd create error with 0.26

[Simon Tian]

[-- Attachment #1: Type: text/plain, Size: 3413 bytes --]

Here is the attachments..

2011/5/15 Simon Tian <aixt2006@gmail.com>:
>> What is the IO size?  Is write_test_data synchronous?
>>
>> For simple write benchmarking you can also use
>>        rados mkpool foo
>>        rados -p foo bench <seconds> write -b <blocksize> -t <threads>
>>
>> and you'll see latency and throughput.  Blocksize defaults to 4M and
>> "threads" (parallel IOs) default to 16, IIRC.
>
>
> Hi, Sage:
>
> I just did the bench:
> rados -p rbd bench 60 write -t 64   and    rados -p data bench 60 write -t 64
> the avg throughput is about 46MB/s, one of the result is as follow.
> But why it's slow with rbd api from <rbd/librbd.h>?
> And I tried testlibrbdpp.cc, the result is more or less the same.
> The attachments are the test codes. Could you run it on your platform please?
>
>  sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
>   40      63       482       419   41.8884        44   2.40044   2.40979
>   41      63       494       431   42.0372        48   2.11044     2.406
>   42      64       506       442   42.0837        44   2.11266   2.40229
>   43      63       518       455   42.3139        52   2.33468    2.3982
>   44      63       527       464   42.1703        36    2.4403   2.39559
>   45      63       539       476   42.2995        48   2.19768   2.39413
>   46      63       551       488   42.4232        48   2.51232    2.3928
>   47      63       563       500   42.5416        48   2.18025   2.38958
>   48      63       572       509   42.4051        36   2.27111   2.38791
>   49      63       584       521    42.519        48   2.41684   2.38695
>   50      63       596       533   42.6284        48   2.11087     2.384
>   51      63       608       545   42.7335        48   2.18147   2.37925
>   52      63       620       557   42.8345        48   2.45287   2.37787
>   53      63       629       566   42.7054        36   2.45187   2.37801
>   54      63       644       581   43.0255        60   2.22403   2.37477
>   55      63       653       590   42.8976        36   2.22782   2.37157
>   56      63       668       605   43.2026        60   2.20638   2.36597
>   57      63       677       614   43.0761        36   2.19628   2.36209
>   58      63       689       626   43.1608        48   2.18262   2.35762
>   59      63       704       641   43.4459        60   2.27029   2.35352
> min lat: 1.87981 max lat: 5.56194 avg lat: 2.34944
>  sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
>   60      63       716       653   43.5215        48   2.27835   2.34944
>   61      64       717       653    42.808         0         -   2.34944
>   62      63       717       654   42.1821         2   2.25694   2.34929
> Total time run:        62.274719
> Total writes made:     717
> Write size:            4194304
> Bandwidth (MB/sec):    46.054
>
> Average Latency:       5.453
> Max latency:           62.0339
> Min latency:           1.87981
>

[-- Attachment #2: testlibrbd.c --]
[-- Type: text/x-csrc, Size: 12723 bytes --]

// -*- mode:C; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- 
// vim: ts=8 sw=2 smarttab
/*
 * Ceph - scalable distributed file system
 *
 * Copyright (C) 2011 New Dream Network
 *
 * This is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public
 * License version 2, as published by the Free Software
 * Foundation.  See file COPYING.
 *
 */

#define __STDC_FORMAT_MACROS
#include <rados/librados.h>
#include <rbd/librbd.h>

#include <assert.h>
#include <errno.h>
#include <inttypes.h>
#include <stdarg.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <time.h>
#include <unistd.h>

#define TEST_IMAGE "testimg"
#define TEST_POOL "librbdtest"
#define TEST_SNAP "testsnap"
#define TEST_IO_SIZE 512
#define TEST_IO_TO_SNAP_SIZE 80
#define MB_BYTES(mb) (mb << 20)

void test_create_and_stat(rados_ioctx_t io_ctx, const char *name, size_t size)
{
  rbd_image_info_t info;
  rbd_image_t image;
  int order = 0;
  assert(rbd_create(io_ctx, name, size, &order) == 0);
  assert(rbd_open(io_ctx, name, &image, NULL) == 0);
  assert(rbd_stat(image, &info, sizeof(info)) == 0);
  printf("image has size %llu and order %d\n", (unsigned long long) info.size, info.order);
  assert(info.size == size);
  assert(info.order == order);
  assert(rbd_close(image) == 0);
}

void test_resize_and_stat(rbd_image_t image, size_t size)
{
  rbd_image_info_t info;
  assert(rbd_resize(image, size) == 0);
  assert(rbd_stat(image, &info, sizeof(info)) == 0);
  printf("image has size %llu and order %d\n", (unsigned long long) info.size, info.order);
  assert(info.size == size);
}

void test_ls(rados_ioctx_t io_ctx, size_t num_expected, ...)
{
  int num_images, i, j;
  char *expected, *names, *cur_name;
  va_list ap;
  size_t max_size = 1024;
  names = (char *) malloc(sizeof(char *) * 1024);
  printf("names is %p\n", names);
  num_images = rbd_list(io_ctx, names, &max_size);
  printf("names is %p\n", names);
  printf("num images is: %d\nexpected: %d\n", num_images, (int)num_expected);
  assert(num_images >= 0);
  assert(num_images == (int)num_expected);

  for (i = 0, cur_name = names; i < num_images; i++) {
    printf("image: %s\n", cur_name);
    cur_name += strlen(cur_name) + 1;
  }

  va_start(ap, num_expected);
  for (i = num_expected; i > 0; i--) {
    expected = va_arg(ap, char *);
    printf("expected = %s\n", expected);
    int found = 0;
    for (j = 0, cur_name = names; j < num_images; j++) {
      if (cur_name[0] == '_') {
	cur_name += strlen(cur_name) + 1;
	continue;
      }
      if (strcmp(cur_name, expected) == 0) {
	printf("found %s\n", cur_name);
	cur_name[0] = '_';
	found = 1;
	break;
      }
    }
    assert(found);
  }

  for (i = 0, cur_name = names; i < num_images; i++) {
    assert(cur_name[0] == '_');
    cur_name += strlen(cur_name) + 1;
  }
  free(names);
}

void test_delete(rados_ioctx_t io_ctx, const char *name)
{
  assert(rbd_remove(io_ctx, name) == 0);
}

void test_create_snap(rbd_image_t image, const char *name)
{
  assert(rbd_snap_create(image, name) == 0);
}

void test_ls_snaps(rbd_image_t image, int num_expected, ...)
{
  rbd_snap_info_t *snaps;
  int num_snaps, i, j, expected_size, max_size = 10;
  char *expected;
  va_list ap;
  snaps = (rbd_snap_info_t *) malloc(sizeof(rbd_snap_info_t *) * 10);
  num_snaps = rbd_snap_list(image, snaps, &max_size);
  printf("num snaps is: %d\nexpected: %d\n", num_snaps, num_expected);
  assert(num_snaps == num_expected);

  for (i = 0; i < num_snaps; i++) {
    printf("snap: %s\n", snaps[i].name);
  }

  va_start(ap, num_expected);
  for (i = num_expected; i > 0; i--) {
    expected = va_arg(ap, char *);
    expected_size = va_arg(ap, int);
    int found = 0;
    for (j = 0; j < num_snaps; j++) {
      if (snaps[j].name == NULL)
	continue;
      if (strcmp(snaps[j].name, expected) == 0) {
	printf("found %s with size %llu\n", snaps[j].name, (unsigned long long) snaps[j].size);
	assert((int)snaps[j].size == expected_size);
	free((void *) snaps[j].name);
	snaps[j].name = NULL;
	found = 1;
	break;
      }
    }
    assert(found);
  }

  for (i = 0; i < num_snaps; i++) {
    assert(snaps[i].name == NULL);
  }
  free(snaps);
}

void test_delete_snap(rbd_image_t image, const char *name)
{
  assert(rbd_snap_remove(image, name) == 0);
}

void simple_write_cb(rbd_completion_t cb, void *arg)
{
  //printf("write completion cb called!\n");
}

void simple_read_cb(rbd_completion_t cb, void *arg)
{
  //printf("read completion cb called!\n");
}

void aio_write_test_data(rbd_image_t image, const char *test_data, uint64_t off, size_t len)
{
  rbd_completion_t comp;
  rbd_aio_create_completion(NULL, (rbd_callback_t) simple_write_cb, &comp);
  //printf("created completion\n");
  rbd_aio_write(image, off, len, test_data, comp);
  //printf("started write\n");
  rbd_aio_wait_for_complete(comp);
  int r = rbd_aio_get_return_value(comp);
  //printf("return value is: %d\n", r);
  assert(r == 0);
  //printf("finished write\n");
  rbd_aio_release(comp);
}

void write_test_data(rbd_image_t image, const char *test_data, uint64_t off, size_t len)
{
  ssize_t written;
  written = rbd_write(image, off, len, test_data);
  //printf("wrote: %d\n", (int) written);
  assert(written == len);
}

void aio_read_test_data(rbd_image_t image, const char *expected, uint64_t off, size_t len)
{
  rbd_completion_t comp;
  char *result;
  assert((result = malloc(sizeof(result) * (len + 1))) != 0);

  rbd_aio_create_completion(NULL, (rbd_callback_t) simple_read_cb, &comp);
  //printf("created completion\n");
  rbd_aio_read(image, off, len, result, comp);
  //printf("started read\n");
  rbd_aio_wait_for_complete(comp);
  int r = rbd_aio_get_return_value(comp);
  //printf("return value is: %d\n", r);
  assert(r == len);
  rbd_aio_release(comp);
  //printf("read: %s\nexpected: %s\n", result, expected);
  assert(memcmp(result, expected, len) == 0);
  free(result);
}

void read_test_data(rbd_image_t image, const char *expected, uint64_t off, size_t len)
{
  ssize_t read;
  char *result;
  assert((result = malloc(sizeof(result) * (len + 1))) != 0);

  read = rbd_read(image, off, len, result);
  //printf("read: %d\n", (int) read);
  assert(read == len);
  result[len] = '\0';
  //printf("read: %s\nexpected: %s\n", result, expected);
  assert(memcmp(result, expected, len) == 0);
  free(result);
}


void test_io(rados_ioctx_t io, rbd_image_t image)
{
  char test_data[TEST_IO_SIZE + 1];
  int i;
	int num_test = 1024*20;

	
struct timeval tv1, tv2;
time_t t1;
float speed=2.0, temp=1.0;


  for (i = 0; i < TEST_IO_SIZE; ++i) {
    test_data[i] = (char) (rand() % (126 - 33) + 33);
  }
  test_data[TEST_IO_SIZE] = '\0';


gettimeofday(&tv1, NULL);
  for (i = 0; i < num_test; i++)
    write_test_data(image, test_data, TEST_IO_SIZE * i, TEST_IO_SIZE);
gettimeofday(&tv2, NULL);
t1 = tv2.tv_sec-tv1.tv_sec;
temp = (float)t1 + (tv2.tv_usec-tv1.tv_usec)/1000000.0;
speed = 1.0*TEST_IO_SIZE*num_test/temp/1024/1024;
printf("time used: temp=%.3f\n", temp);
printf("write speed: %.2f MB/s\n", speed);


gettimeofday(&tv1, NULL);
  for (i = 0; i < num_test; i++)
    read_test_data(image, test_data, TEST_IO_SIZE * i, TEST_IO_SIZE);
gettimeofday(&tv2, NULL);
t1 = tv2.tv_sec-tv1.tv_sec;
temp = (float)t1 + (tv2.tv_usec-tv1.tv_usec)/1000000.0;
speed = 1.0*TEST_IO_SIZE*num_test/temp/1024/1024;
printf("time used: temp=%.3f\n", temp);
printf("read speed: %.2f MB/s\n", speed);


gettimeofday(&tv1, NULL);
  for (i = num_test; i < 2*num_test; i++)
    aio_write_test_data(image, test_data, TEST_IO_SIZE * i, TEST_IO_SIZE);
gettimeofday(&tv2, NULL);
t1 = tv2.tv_sec-tv1.tv_sec;
temp = (float)t1 + (tv2.tv_usec-tv1.tv_usec)/1000000.0;
speed = 1.0*TEST_IO_SIZE*num_test/temp/1024/1024;
printf("time used: temp=%.3f\n", temp);
printf("aio write speed: %.2f MB/s\n", speed);


gettimeofday(&tv1, NULL);
  for (i = num_test; i < 2*num_test; i++)
    aio_read_test_data(image, test_data, TEST_IO_SIZE * i, TEST_IO_SIZE);
gettimeofday(&tv2, NULL);
t1 = tv2.tv_sec-tv1.tv_sec;
temp = (float)t1 + (tv2.tv_usec-tv1.tv_usec)/1000000.0;
speed = 1.0*TEST_IO_SIZE*num_test/temp/1024/1024;
printf("time used: temp=%.3f\n", temp);
printf("aio read speed: %.2f MB/s\n", speed);
}


void test_io_old(rados_ioctx_t io, rbd_image_t image)
{
  char test_data[TEST_IO_SIZE + 1];
  int i;

  for (i = 0; i < TEST_IO_SIZE; ++i) {
    test_data[i] = (char) (rand() % (126 - 33) + 33);
  }
  test_data[TEST_IO_SIZE] = '\0';

  for (i = 0; i < 5; ++i)
    write_test_data(image, test_data, TEST_IO_SIZE * i, TEST_IO_SIZE);

  for (i = 5; i < 10; ++i)
    aio_write_test_data(image, test_data, TEST_IO_SIZE * i, TEST_IO_SIZE);

  for (i = 0; i < 5; ++i)
    read_test_data(image, test_data, TEST_IO_SIZE * i, TEST_IO_SIZE);

  for (i = 5; i < 10; ++i)
    aio_read_test_data(image, test_data, TEST_IO_SIZE * i, TEST_IO_SIZE);
}

void test_io_to_snapshot(rados_ioctx_t io_ctx, rbd_image_t image, size_t isize)
{
  int i, r;
  rbd_image_t image_at_snap;
  char orig_data[TEST_IO_TO_SNAP_SIZE + 1];
  char test_data[TEST_IO_TO_SNAP_SIZE + 1];

  for (i = 0; i < TEST_IO_TO_SNAP_SIZE - 1; ++i)
    test_data[i] = (char) (i + 48);
  test_data[TEST_IO_TO_SNAP_SIZE] = '\0';

  r = rbd_read(image, 0, TEST_IO_TO_SNAP_SIZE, orig_data);
  assert(r == TEST_IO_TO_SNAP_SIZE);

  test_ls_snaps(image, 0);
  test_create_snap(image, "orig");
  test_ls_snaps(image, 1, "orig", isize);
  read_test_data(image, orig_data, 0, TEST_IO_TO_SNAP_SIZE);

  printf("write test data!\n");
  write_test_data(image, test_data, 0, TEST_IO_TO_SNAP_SIZE);
  test_create_snap(image, "written");
  test_ls_snaps(image, 2, "orig", isize, "written", isize);

  read_test_data(image, test_data, 0, TEST_IO_TO_SNAP_SIZE);

  rbd_snap_set(image, "orig");
  read_test_data(image, orig_data, 0, TEST_IO_TO_SNAP_SIZE);

  rbd_snap_set(image, "written");
  read_test_data(image, test_data, 0, TEST_IO_TO_SNAP_SIZE);

  rbd_snap_set(image, "orig");

  r = rbd_write(image, 0, TEST_IO_TO_SNAP_SIZE, test_data);
  printf("write to snapshot returned %d\n", r);
  assert(r < 0);
  printf("%s\n", strerror(-r));

  read_test_data(image, orig_data, 0, TEST_IO_TO_SNAP_SIZE);
  rbd_snap_set(image, "written");
  read_test_data(image, test_data, 0, TEST_IO_TO_SNAP_SIZE);

  r = rbd_snap_rollback(image, "orig");
  printf("rbd_snap_rollback returned %d\n", r);
  assert(r >= 0);

  r = rbd_snap_set(image, NULL);
  assert(r == 0);
  write_test_data(image, test_data, 0, TEST_IO_TO_SNAP_SIZE);

  printf("opening testimg@orig\n");
  assert(rbd_open(io_ctx, TEST_IMAGE, &image_at_snap, "orig") >= 0);
  read_test_data(image_at_snap, orig_data, 0, TEST_IO_TO_SNAP_SIZE);
  r = rbd_write(image_at_snap, 0, TEST_IO_TO_SNAP_SIZE, test_data);
  printf("write to snapshot returned %d\n", r);
  assert(r < 0);
  printf("%s\n", strerror(-r));
  assert(rbd_close(image_at_snap) == 0);

  test_ls_snaps(image, 2, "orig", isize, "written", isize);
  test_delete_snap(image, "written");
  test_ls_snaps(image, 1, "orig", isize);
  test_delete_snap(image, "orig");
  test_ls_snaps(image, 0);
}

int main(int argc, const char **argv) 
{
  rados_t cluster;
  rados_ioctx_t io_ctx;
  rbd_image_t image;

  srand(time(0));

  assert(rados_create(&cluster, NULL) == 0);
  assert(rados_conf_read_file(cluster, NULL) == 0);
  rados_reopen_log(cluster);
  assert(rados_connect(cluster) == 0);

  if (rados_pool_lookup(cluster, TEST_POOL) != -ENOENT) {
    int r = rados_pool_delete(cluster, TEST_POOL);
    printf("rados_pool_delete returned %d\n", r);
  }
  int r = rados_pool_create(cluster, TEST_POOL);
  printf("rados_pool_create returned %d\n", r);

  assert(rados_ioctx_create(cluster, "data", &io_ctx) == 0);
  test_ls(io_ctx, 0);

  test_create_and_stat(io_ctx, TEST_IMAGE, MB_BYTES(10));
  assert(rbd_open(io_ctx, TEST_IMAGE, &image, NULL) == 0);

  test_ls(io_ctx, 1, TEST_IMAGE);
  test_ls_snaps(image, 0);

  test_create_snap(image, TEST_SNAP);
  test_ls_snaps(image, 1, TEST_SNAP, MB_BYTES(10));
  test_resize_and_stat(image, MB_BYTES(20));
  test_io(io_ctx, image);

  test_create_snap(image, TEST_SNAP "1");
  test_ls_snaps(image, 2, TEST_SNAP, MB_BYTES(10), TEST_SNAP "1", MB_BYTES(20));

  test_delete_snap(image, TEST_SNAP);
  test_ls_snaps(image, 1, TEST_SNAP "1", MB_BYTES(20));

  test_delete_snap(image, TEST_SNAP "1");
  test_ls_snaps(image, 0);

  test_io_to_snapshot(io_ctx, image, MB_BYTES(20));
  assert(rbd_close(image) == 0);

  test_create_and_stat(io_ctx, TEST_IMAGE "1", MB_BYTES(20));
  test_ls(io_ctx, 2, TEST_IMAGE, TEST_IMAGE "1");

  test_delete(io_ctx, TEST_IMAGE);
  test_ls(io_ctx, 1, TEST_IMAGE "1");

  test_delete(io_ctx, TEST_IMAGE "1");
  test_ls(io_ctx, 0);

  rados_ioctx_destroy(io_ctx);
  rados_shutdown(cluster);

  return 0;
}

[-- Attachment #3: testlibrbdpp.cc --]
[-- Type: application/octet-stream, Size: 9665 bytes --]

// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- 
// vim: ts=8 sw=2 smarttab
/*
 * Ceph - scalable distributed file system
 *
 * Copyright (C) 2011 New Dream Network
 *
 * This is free software; you can redistribute it and/or
 * modify it under the terms of the GNU General Public
 * License version 2, as published by the Free Software
 * Foundation.  See file COPYING.
 *
 */

#define __STDC_FORMAT_MACROS
#include <rados/assert.h>
#include <rbd/librbd.hpp>
#include <rados/librados.hpp>
#include <rados/buffer.h>

#include <sys/time.h>
#include <time.h>

#include <algorithm>
#include <cstdarg>
#include <cstdio>
#include <cstdlib>
#include <ctime>
#include <errno.h>
#include <memory>
#include <sys/types.h>
#include <string>
#include <vector>

using namespace std;

	
#define TEST_IMAGE "testimg"
#define TEST_POOL "librbdtest"
#define TEST_SNAP "testsnap"
#define TEST_IO_SIZE 513
#define MB_BYTES(mb) (mb << 20)

librbd::RBD *rbd;

void test_create_and_stat(librados::IoCtx& io_ctx, const char *name, size_t size)
{
  librbd::image_info_t info;
  librbd::Image image;
  int order = 0;
  assert(rbd->create(io_ctx, name, size, &order) == 0);
  assert(rbd->open(io_ctx, image, name, NULL) == 0);
  assert(image.stat(info, sizeof(info)) == 0);
  cout << "image has size " << info.size << " and order " << info.order << endl;
  assert(info.size == size);
  assert(info.order == order);
}

void test_resize_and_stat(librbd::Image& image, size_t size)
{
  librbd::image_info_t info;
  assert(image.resize(size) == 0);
  assert(image.stat(info, sizeof(info)) == 0);
  cout << "image has size " << info.size << " and order " << info.order << endl;
  assert(info.size == size);
}

void test_ls(librados::IoCtx& io_ctx, size_t num_expected, ...)
{
  int r;
  size_t i;
  char *expected;
  va_list ap;
  vector<string> names;
  r = rbd->list(io_ctx, names);
  if (r == -ENOENT)
    r = 0;
  assert(r >= 0);
  cout << "num images is: " << names.size() << endl
       << "expected: " << num_expected << endl;
  assert(names.size() == num_expected);

  for (i = 0; i < names.size(); i++) {
    cout << "image: " << names[i] << endl;
  }

  va_start(ap, num_expected);
  for (i = num_expected; i > 0; i--) {
    expected = va_arg(ap, char *);
    cout << "expected = " << expected << endl;
    vector<string>::iterator listed_name = find(names.begin(), names.end(), string(expected));
    assert(listed_name != names.end());
    names.erase(listed_name);
  }
  assert(names.empty());
}

void test_delete(librados::IoCtx& io_ctx, const char *name)
{
  assert(rbd->remove(io_ctx, name) == 0);
}

void test_create_snap(librbd::Image& image, const char *name)
{
  assert(image.snap_create(name) == 0);
}

void test_ls_snaps(librbd::Image& image, size_t num_expected, ...)
{
  int r;
  size_t i, j, expected_size;
  char *expected;
  va_list ap;
  vector<librbd::snap_info_t> snaps;
  r = image.snap_list(snaps);
  assert(r >= 0);
  cout << "num snaps is: " << snaps.size() << endl
       << "expected: " << num_expected << endl;
  assert(snaps.size() == num_expected);

  for (i = 0; i < snaps.size(); i++) {
    cout << "snap: " << snaps[i].name << endl;
  }

  va_start(ap, num_expected);
  for (i = num_expected; i > 0; i--) {
    expected = va_arg(ap, char *);
    expected_size = va_arg(ap, int);
    int found = 0;
    for (j = 0; j < snaps.size(); j++) {
      if (snaps[j].name == "")
	continue;
      if (strcmp(snaps[j].name.c_str(), expected) == 0) {
	cout << "found " << snaps[j].name << " with size " << snaps[j].size << endl;
	assert(snaps[j].size == (size_t) expected_size);
	snaps[j].name = "";
	found = 1;
	break;
      }
    }
    assert(found);
  }

  for (i = 0; i < snaps.size(); i++) {
    assert(snaps[i].name == "");
  }
}

void test_delete_snap(librbd::Image& image, const char *name)
{
  assert(image.snap_remove(name) == 0);
}

void simple_write_cb(librbd::completion_t cb, void *arg)
{
  //cout << "write completion cb called!" << endl;
}

void simple_read_cb(librbd::completion_t cb, void *arg)
{
  //cout << "read completion cb called!" << endl;
}

void aio_write_test_data(librbd::Image& image, const char *test_data, off_t off)
{
  ceph::bufferlist bl;
  bl.append(test_data, strlen(test_data));
  librbd::RBD::AioCompletion *comp = new librbd::RBD::AioCompletion(NULL, (librbd::callback_t) simple_write_cb);
  //printf("created completion\n");
  image.aio_write(off, strlen(test_data), bl, comp);
  //printf("started write\n");
  comp->wait_for_complete();
  int r = comp->get_return_value();
  //printf("return value is: %d\n", r);
  assert(r == 0);
  //printf("finished write\n");
  comp->release();
}

void write_test_data(librbd::Image& image, const char *test_data, off_t off)
{
  int written;
  size_t len = strlen(test_data);
  ceph::bufferlist bl;
  bl.append(test_data, len);
  written = image.write(off, len, bl);
  assert(written >= 0);
  //printf("wrote: %u\n", (unsigned int) written);
}

void aio_read_test_data(librbd::Image& image, const char *expected, off_t off)
{
  librbd::RBD::AioCompletion *comp = new librbd::RBD::AioCompletion(NULL, (librbd::callback_t) simple_read_cb);
  ceph::bufferlist bl;
  //printf("created completion\n");
  image.aio_read(off, strlen(expected), bl, comp);
  //printf("started read\n");
  comp->wait_for_complete();
  int r = comp->get_return_value();
  //printf("return value is: %d\n", r);
  assert(r == TEST_IO_SIZE - 1);
  assert(strncmp(expected, bl.c_str(), TEST_IO_SIZE - 1) == 0);
  //printf("finished read\n");
  comp->release();
}

void read_test_data(librbd::Image& image, const char *expected, off_t off)
{
  int read, total_read = 0;
  size_t expected_len = strlen(expected);
  size_t len = expected_len;
  ceph::bufferlist bl;
  read = image.read(off + total_read, len, bl);
  assert(read >= 0);
  //printf("read: %u\n", (unsigned int) read);
  //printf("read: %s\nexpected: %s\n", bl.c_str(), expected);
  assert(strncmp(bl.c_str(), expected, expected_len) == 0);
}

void test_io(librados::IoCtx& io_ctx, librbd::Image& image)
{
  char test_data[TEST_IO_SIZE];
  int i;

	int num_test = 1024;
	struct timeval tv1, tv2;
	time_t t1;
	float speed=2.0, temp=1.0;


  srand(time(0));
  for (i = 0; i < TEST_IO_SIZE - 1; ++i) {
    test_data[i] = (char) (rand() % (126 - 33) + 33);
  }
  test_data[TEST_IO_SIZE - 1] = '\0';


/*
  for (i = 0; i < 5; ++i)
    write_test_data(image, test_data, strlen(test_data) * i);

  for (i = 5; i < 10; ++i)
    aio_write_test_data(image, test_data, strlen(test_data) * i);

  for (i = 0; i < 5; ++i)
    read_test_data(image, test_data, strlen(test_data) * i);

  for (i = 5; i < 10; ++i)
    aio_read_test_data(image, test_data, strlen(test_data) * i);

*/

gettimeofday(&tv1, NULL);
  for (i = 0; i < num_test; ++i)
  	write_test_data(image, test_data, strlen(test_data) * i);
gettimeofday(&tv2, NULL);
t1 = tv2.tv_sec-tv1.tv_sec;
temp = (float)t1 + (tv2.tv_usec-tv1.tv_usec)/1000000.0;
speed = 1.0*TEST_IO_SIZE*num_test/temp/1024/1024;
printf("time used: temp=%.3f\n", temp);
printf("write speed: %.2f MB/s\n", speed);


gettimeofday(&tv1, NULL);
  for (i = 0; i < num_test; ++i)
    read_test_data(image, test_data, strlen(test_data) * i);
gettimeofday(&tv2, NULL);
t1 = tv2.tv_sec-tv1.tv_sec;
temp = (float)t1 + (tv2.tv_usec-tv1.tv_usec)/1000000.0;
speed = 1.0*TEST_IO_SIZE*num_test/temp/1024/1024;
printf("time used: temp=%.3f\n", temp);
printf("read speed: %.2f MB/s\n", speed);


gettimeofday(&tv1, NULL);
  for (i = num_test; i < 2*num_test; ++i)
    aio_write_test_data(image, test_data, strlen(test_data) * i);
gettimeofday(&tv2, NULL);
t1 = tv2.tv_sec-tv1.tv_sec;
temp = (float)t1 + (tv2.tv_usec-tv1.tv_usec)/1000000.0;
speed = 1.0*TEST_IO_SIZE*num_test/temp/1024/1024;
printf("time used: temp=%.3f\n", temp);
printf("aio write speed: %.2f MB/s\n", speed);


gettimeofday(&tv1, NULL);
  for (i = num_test; i < 2*num_test; ++i)
    aio_read_test_data(image, test_data, strlen(test_data) * i);
gettimeofday(&tv2, NULL);
t1 = tv2.tv_sec-tv1.tv_sec;
temp = (float)t1 + (tv2.tv_usec-tv1.tv_usec)/1000000.0;
speed = 1.0*TEST_IO_SIZE*num_test/temp/1024/1024;
printf("time used: temp=%.3f\n", temp);
printf("aio read speed: %.2f MB/s\n", speed);

}

int main(int argc, const char **argv) 
{
  librados::Rados rados;
  librados::IoCtx io_ctx;
  librbd::Image image;
  rbd = new librbd::RBD();
  assert(rados.init(NULL) == 0);
  assert(rados.conf_read_file(NULL) == 0);
  rados.reopen_log();
  assert(rados.connect() == 0);
  if (rados.pool_lookup(TEST_POOL) != -ENOENT) {
    int r = rados.pool_delete(TEST_POOL);
    printf("rados.pool_delete returned %d\n", r);
  }
  int r = rados.pool_create(TEST_POOL);
  printf("rados.pool_create returned %d\n", r);
  assert(rados.ioctx_create(TEST_POOL, io_ctx) == 0);
  test_ls(io_ctx, 0);
  test_create_and_stat(io_ctx, TEST_IMAGE, MB_BYTES(1));
  assert(rbd->open(io_ctx, image, TEST_IMAGE, NULL) == 0);
  test_ls(io_ctx, 1, TEST_IMAGE);
  test_ls_snaps(image, 0);
  test_create_snap(image, TEST_SNAP);
  test_ls_snaps(image, 1, TEST_SNAP, MB_BYTES(1));
  test_resize_and_stat(image, MB_BYTES(2));
  test_io(io_ctx, image);
  test_create_snap(image, TEST_SNAP "1");
  test_ls_snaps(image, 2, TEST_SNAP, MB_BYTES(1), TEST_SNAP "1", MB_BYTES(2));
  test_delete_snap(image, TEST_SNAP);
  test_ls_snaps(image, 1, TEST_SNAP "1", MB_BYTES(2));
  test_delete_snap(image, TEST_SNAP "1");
  test_ls_snaps(image, 0);
  test_create_and_stat(io_ctx, TEST_IMAGE "1", MB_BYTES(2));
  test_ls(io_ctx, 2, TEST_IMAGE, TEST_IMAGE "1");
  test_delete(io_ctx, TEST_IMAGE);
  test_ls(io_ctx, 1, TEST_IMAGE "1");
  test_delete(io_ctx, TEST_IMAGE "1");
  test_ls(io_ctx, 0);
  delete rbd;
  return 0;
}

Re: rbd create error with 0.26

[Simon Tian]

[-- Attachment #1: Type: text/plain, Size: 4894 bytes --]

I have just test the performance with a modified src/testrados.c,
the read and write speed is very slow too:
=========================================================
root@houyi-vm16.dev.sd.aliyun.com # ./testrados
rados_ioctx_create = 0
rados_ioctx_create = 0, io_ctx = 0x7f52d00022b0
need buffer size of 42
begin pools.
 pool: 'data'
 pool: 'metadata'
 pool: 'casdata'
 pool: 'rbd'
 pool: 'librbdtest'
 pool: 'foo'
end pools.
rados_ioctx_pool_stat = 0, 0 KB, 0 objects
rados_ioctx_snap_create snap1 = 0
rados_ioctx_snap_list got snap 1 snap1
rados_ioctx_snap_lookup snap1 got 1, result 0
rados_ioctx_snap_remove snap1 = 0
rados_write = 1024
write time used: temp=20.400
write speed: 0.49 MB/s
                  ^^^^^^^^^^^^
rados_read = 1024
read time used: temp=7.110
read speed: 1.41 MB/s
                  ^^^^^^^^^^^^
rados_setxattr attr1=bar = 3
rados_getxattr attr1 = 3
rados_stat size = 10485760 mtime = 1305463463 = 0
rados_stat(does_not_exist) = -2
exec result=
read result=
size=128
a safe
b safe
rados_list_objects_open = 0, h = 0x7f52d0000c10
rados_list_objects_next got object '../b/bb_bb_bb\foo\bar'
rados_list_objects_next got object 'a'
rados_list_objects_next got object 'foo_object'
rados_stat_pool = 0, 10240 KB, 1 objects
rados_delete_pool = 0
rados_ioctx_pool_delete = 0
=========================================================


Hmm,  test with testceph.cc is all right.   3 replca, write speed:
25.17 MB/s     read speed: 53.46 MB/s
Attachments are the test code. FYI

Thx!
Simon


2011/5/15 Simon Tian <aixt2006@gmail.com>:
>> What is the IO size?  Is write_test_data synchronous?
>>
>> For simple write benchmarking you can also use
>>        rados mkpool foo
>>        rados -p foo bench <seconds> write -b <blocksize> -t <threads>
>>
>> and you'll see latency and throughput.  Blocksize defaults to 4M and
>> "threads" (parallel IOs) default to 16, IIRC.
>
>
> Hi, Sage:
>
> I just did the bench:
> rados -p rbd bench 60 write -t 64   and    rados -p data bench 60 write -t 64
> the avg throughput is about 46MB/s, one of the result is as follow.
> But why it's slow with rbd api from <rbd/librbd.h>?
> And I tried testlibrbdpp.cc, the result is more or less the same.
> The attachments are the test codes. Could you run it on your platform please?
>
>  sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
>   40      63       482       419   41.8884        44   2.40044   2.40979
>   41      63       494       431   42.0372        48   2.11044     2.406
>   42      64       506       442   42.0837        44   2.11266   2.40229
>   43      63       518       455   42.3139        52   2.33468    2.3982
>   44      63       527       464   42.1703        36    2.4403   2.39559
>   45      63       539       476   42.2995        48   2.19768   2.39413
>   46      63       551       488   42.4232        48   2.51232    2.3928
>   47      63       563       500   42.5416        48   2.18025   2.38958
>   48      63       572       509   42.4051        36   2.27111   2.38791
>   49      63       584       521    42.519        48   2.41684   2.38695
>   50      63       596       533   42.6284        48   2.11087     2.384
>   51      63       608       545   42.7335        48   2.18147   2.37925
>   52      63       620       557   42.8345        48   2.45287   2.37787
>   53      63       629       566   42.7054        36   2.45187   2.37801
>   54      63       644       581   43.0255        60   2.22403   2.37477
>   55      63       653       590   42.8976        36   2.22782   2.37157
>   56      63       668       605   43.2026        60   2.20638   2.36597
>   57      63       677       614   43.0761        36   2.19628   2.36209
>   58      63       689       626   43.1608        48   2.18262   2.35762
>   59      63       704       641   43.4459        60   2.27029   2.35352
> min lat: 1.87981 max lat: 5.56194 avg lat: 2.34944
>  sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
>   60      63       716       653   43.5215        48   2.27835   2.34944
>   61      64       717       653    42.808         0         -   2.34944
>   62      63       717       654   42.1821         2   2.25694   2.34929
> Total time run:        62.274719
> Total writes made:     717
> Write size:            4194304
> Bandwidth (MB/sec):    46.054
>
> Average Latency:       5.453
> Max latency:           62.0339
> Min latency:           1.87981
>

[-- Attachment #2: testrados.c --]
[-- Type: text/x-csrc, Size: 6636 bytes --]

// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- 
// vim: ts=8 sw=2 smarttab
/*
 * Ceph - scalable distributed file system
 *
 * Copyright (C) 2004-2006 Sage Weil <sage@newdream.net>
 *
 * This is free software; you can redistribute it and/or
 * modify it under the terms of the GNU Lesser General Public
 * License version 2.1, as published by the Free Software 
 * Foundation.  See file COPYING.
 * 
 */

#include <rados/librados.h>

#include <stdio.h>
#include <stdlib.h>
#include <time.h>

#define TEST_IO_SIZE	1024
#define TEST_IO_NUM		1024*10


int main(int argc, const char **argv) 
{
  char tmp[32];
  int i, r;
  rados_t cl;

  if (rados_create(&cl, NULL) < 0) {
    printf("error initializing\n");
    exit(1);
  }

  if (rados_conf_read_file(cl, NULL)) {
    printf("error reading configuration file\n");
    exit(1);
  }

  // Try to set a configuration option that doesn't exist.
  // This should fail.
  if (!rados_conf_set(cl, "config option that doesn't exist",
                     "some random value")) {
    printf("error: succeeded in setting nonexistent config option\n");
    exit(1);
  }

  if (rados_conf_get(cl, "log to stderr", tmp, sizeof(tmp))) {
    printf("error: failed to read log_to_stderr from config\n");
    exit(1);
  }

  // Can we change it? 
  if (rados_conf_set(cl, "log to stderr", "2")) {
    printf("error: error setting log_to_stderr\n");
    exit(1);
  }
  rados_reopen_log(cl);
  if (rados_conf_get(cl, "log to stderr", tmp, sizeof(tmp))) {
    printf("error: failed to read log_to_stderr from config\n");
    exit(1);
  }
  if (tmp[0] != '2') {
    printf("error: new setting for log_to_stderr failed to take effect.\n");
    exit(1);
  }

  if (rados_connect(cl)) {
    printf("error connecting\n");
    exit(1);
  }

  /* create an io_ctx */
  r = rados_pool_create(cl, "foo");
  printf("rados_ioctx_create = %d\n", r);

  rados_ioctx_t io_ctx;
  r = rados_ioctx_create(cl, "foo", &io_ctx);
  printf("rados_ioctx_create = %d, io_ctx = %p\n", r, io_ctx);

  /* list all pools */
  {
    int buf_sz = rados_pool_list(cl, NULL, 0);
    printf("need buffer size of %d\n", buf_sz);
    char buf[buf_sz];
    int r = rados_pool_list(cl, buf, buf_sz);
    if (r != buf_sz) {
      printf("buffer size mismatch: got %d the first time, but %d "
	     "the second.\n", buf_sz, r);
      exit(1);
    }
    const char *b = buf;
    printf("begin pools.\n");
    while (1) {
      if (b[0] == '\0')
        break;
      printf(" pool: '%s'\n", b);
      b += strlen(b) + 1;
    };
    printf("end pools.\n");
  }


  /* stat */
  struct rados_pool_stat_t st;
  r = rados_ioctx_pool_stat(io_ctx, &st);
  printf("rados_ioctx_pool_stat = %d, %lld KB, %lld objects\n", r, (long long)st.num_kb, (long long)st.num_objects);

  /* snapshots */
  r = rados_ioctx_snap_create(io_ctx, "snap1");
  printf("rados_ioctx_snap_create snap1 = %d\n", r);
  rados_snap_t snaps[10];
  r = rados_ioctx_snap_list(io_ctx, snaps, 10);
  for (i=0; i<r; i++) {
    char name[100];
    rados_ioctx_snap_get_name(io_ctx, snaps[i], name, sizeof(name));
    printf("rados_ioctx_snap_list got snap %lld %s\n", (long long)snaps[i], name);
  }
  rados_snap_t snapid;
  r = rados_ioctx_snap_lookup(io_ctx, "snap1", &snapid);
  printf("rados_ioctx_snap_lookup snap1 got %lld, result %d\n", (long long)snapid, r);
  r = rados_ioctx_snap_remove(io_ctx, "snap1");
  printf("rados_ioctx_snap_remove snap1 = %d\n", r);

  /* sync io */
  time_t tm;
  char buf[TEST_IO_SIZE], buf2[TEST_IO_SIZE];
  
  
	for(i=0; i<TEST_IO_SIZE; i++)
	{
		buf[i] = 0xFF&i;
	}
  
  const char *oid = "foo_object";
  
  
  
struct timeval tv1, tv2;
time_t t1;
gettimeofday(&tv1, NULL);
int offset = 0;
for(i=0; i<TEST_IO_NUM; i++)
{
  r = rados_write(io_ctx, oid, buf, TEST_IO_SIZE, offset);
	offset += TEST_IO_SIZE;
}
  printf("rados_write = %d\n", r);

gettimeofday(&tv2, NULL);
t1 = tv2.tv_sec-tv1.tv_sec;

float speed=0.1, temp=1.0;
temp = (float)t1 + (tv2.tv_usec-tv1.tv_usec)/1000000.0;
printf("write time used: temp=%.3f\n",temp);
speed = 1.0*TEST_IO_SIZE*TEST_IO_NUM/temp/1024/1024;
printf("write speed: %.2f MB/s\n", speed);

  
gettimeofday(&tv1, NULL);
offset = 0;
for(i=0; i<TEST_IO_NUM; i++)
{
  r = rados_read(io_ctx, oid, buf2, TEST_IO_SIZE, offset);
	offset += TEST_IO_SIZE;
}
  printf("rados_read = %d\n", r);
gettimeofday(&tv2, NULL);
t1 = tv2.tv_sec-tv1.tv_sec;
temp = (float)t1 + (tv2.tv_usec-tv1.tv_usec)/1000000.0;
printf("read time used: temp=%.3f\n",temp);
speed = 1.0*TEST_IO_SIZE*TEST_IO_NUM/temp/1024/1024;
printf("read speed: %.2f MB/s\n", speed);

  
  if (memcmp(buf, buf2, r))
    printf("*** content mismatch ***\n");

  /* attrs */
  r = rados_setxattr(io_ctx, oid, "attr1", "bar", 3);
  printf("rados_setxattr attr1=bar = %d\n", r);
  char val[10];
  r = rados_getxattr(io_ctx, oid, "attr1", val, sizeof(val));
  printf("rados_getxattr attr1 = %d\n", r);
  if (memcmp(val, "bar", 3))
    printf("*** attr value mismatch ***\n");

  uint64_t size;
  time_t mtime;
  r = rados_stat(io_ctx, oid, &size, &mtime);
  printf("rados_stat size = %lld mtime = %d = %d\n", (long long)size, (int)mtime, r);
  r = rados_stat(io_ctx, "does_not_exist", NULL, NULL);
  printf("rados_stat(does_not_exist) = %d\n", r);

  /* exec */
  rados_exec(io_ctx, oid, "crypto", "md5", buf, strlen(buf) + 1, buf, 128);
  printf("exec result=%s\n", buf);
  r = rados_read(io_ctx, oid, buf2, 128, 0);
  printf("read result=%s\n", buf2);
  printf("size=%d\n", r);

  /* aio */
  rados_completion_t a, b;
  rados_aio_create_completion(0, 0, 0, &a);
  rados_aio_create_completion(0, 0, 0, &b);
  rados_aio_write(io_ctx, "a", a, buf, 100, 0);
  rados_aio_write(io_ctx, "../b/bb_bb_bb\\foo\\bar", b, buf, 100, 0);
  rados_aio_wait_for_safe(a);
  printf("a safe\n");
  rados_aio_wait_for_safe(b);
  printf("b safe\n");
  rados_aio_release(a);
  rados_aio_release(b);

  rados_read(io_ctx, "../b/bb_bb_bb\\foo\\bar", buf2, 128, 0);

  /* list objects */
  rados_list_ctx_t h;
  r = rados_objects_list_open(io_ctx, &h);
  printf("rados_list_objects_open = %d, h = %p\n", r, h);
  const char *poolname;
  while (rados_objects_list_next(h, &poolname) == 0)
    printf("rados_list_objects_next got object '%s'\n", poolname);
  rados_objects_list_close(h);

  /* stat */
  r = rados_ioctx_pool_stat(io_ctx, &st);
  printf("rados_stat_pool = %d, %lld KB, %lld objects\n", r, (long long)st.num_kb, (long long)st.num_objects);

  /* delete a pool */
  printf("rados_delete_pool = %d\n", r);  
  rados_ioctx_destroy(io_ctx);

  r = rados_pool_delete(cl, "foo");
  printf("rados_ioctx_pool_delete = %d\n", r);

  rados_shutdown(cl);

  return 0;
}


[-- Attachment #3: testceph.cc --]
[-- Type: application/octet-stream, Size: 3879 bytes --]

// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- 
// vim: ts=8 sw=2 smarttab
/*
 * Ceph - scalable distributed file system
 *
 * Copyright (C) 2004-2006 Sage Weil <sage@newdream.net>
 *
 * This is free software; you can redistribute it and/or
 * modify it under the terms of the GNU Lesser General Public
 * License version 2.1, as published by the Free Software 
 * Foundation.  See file COPYING.
 * 
 */

#include <ceph/libceph.h>
#include <fcntl.h>
#include <sys/time.h>
#include <iostream>

using namespace std;

int main(int argc, const char **argv)
{
	int ret=0;
  if (ceph_initialize(argc, argv) < 0) {
    cerr << "error initializing\n" << std::endl;
    return(1);
  }
  cout << "Successfully initialized Ceph!" << std::endl;

  if(ceph_mount() < 0) {
    cerr << "error mounting\n" << std::endl;
    return(1);
  }
  cout << "Successfully mounted Ceph!" << std::endl;

/*
int local_osd_id = ceph_get_local_osd();
printf("local_osd_id=%d\n", local_osd_id);

if(ceph_set_default_preferred_pg(local_osd_id) < 0) {
	    cerr << "error ceph_set_default_preferred_pg\n" << std::endl;
    return(1);
}
*/

	const char test_path[] = "test_unfed";
	int len = 1024*1024, offset=0, num=1024;
	int i=0;
	char buf_wr[1024*1024];
	char buf_rd[1024*1024];


if ( ceph_mknod(test_path, 1, 0) < 0 )
{
    cerr << "error mknod\n" << std::endl;
}

int fd = ceph_open(test_path, O_RDWR|O_CREAT, 0);
if( fd<0 )
{
		 cerr << "error open\n" << std::endl;
}


for(i=0; i<len; i++)
{
	buf_wr[i] = 0xFF&i;
}

printf("before write once!\n");
ceph_write(fd, buf_wr, len, offset);
printf("after write once!\n");
ceph_read(fd, buf_rd, len, offset);

for(i=0; i<512; i++)
{
	printf("%d ",buf_rd[i]);
}
printf("END!!\n");










struct timeval tv1, tv2;
time_t t1;
gettimeofday(&tv1, NULL);

for(i=0; i<num; i++)
{
	ceph_write(fd, buf_wr, len, offset);
	offset += len;
}

gettimeofday(&tv2, NULL);
t1 = tv2.tv_sec-tv1.tv_sec;

float speed=0.1, temp=1.0;
temp = (float)t1 + (tv2.tv_usec-tv1.tv_usec)/1000000.0;
printf("write time used: temp=%.3f\n",temp);
speed = 1.0*len*num/temp/1024/1024;
printf("write speed: %.2f MB/s\n", speed);

struct stat_precise stat_p_info;
ret = ceph_lstat_precise(test_path, &stat_p_info);
if(ret<0)
{
		cerr << "error ceph_lstat_precise\n" << std::endl;
		return -1;
}
printf("st_size=%dM\n",stat_p_info.st_size/1024/1024);

offset = 0;
gettimeofday(&tv1, NULL);
for(i=0; i<num; i++)
{
	ceph_read(fd, buf_rd, len, offset);
	offset += len;
}

gettimeofday(&tv2, NULL);
t1 = tv2.tv_sec-tv1.tv_sec;
temp = (float)t1 + (tv2.tv_usec-tv1.tv_usec)/1000000.0;
printf("read time used: temp=%.3f\n",temp);
speed = 1.0*len*num/temp/1024/1024;
printf("read speed: %.2f MB/s\n", speed);








int de_pre = ceph_get_default_preferred_pg(fd);
printf("de_pre=%d\n", de_pre);

if(fd>=0)
{
		ceph_close(fd);
		printf("ceph_close: fd=%d\n",fd);
}


int sz_rep = ceph_get_file_replication(test_path);
printf("sz_rep=%d\n", sz_rep);

/*
/////////////////////////////////////////////////////
loff_t ceph_lseek(int fd, loff_t offset, int whence);


int ceph_get_file_stripe_unit(int fh);
int ceph_get_file_replication(const char *path);
int ceph_get_default_preferred_pg(int fd);
int ceph_get_file_stripe_address(int fd, loff_t offset, char *buf, int buflen);
*/

  if(ceph_umount() < 0) {
    cerr << "error umounting\n" << std::endl;
    return(1);
  }
  cout << "Successfully umounted Ceph!" << std::endl;

  ceph_deinitialize();
  cout << "Successfully deinitialized Ceph!" << std::endl;

/*
if(fd>=0)
{
		printf("222ceph_close: fd=%d\n",fd);
		ceph_close(fd);
		printf("222 after ceph_close: fd=%d\n",fd);
}*/
  if(ceph_umount() < 0) {
    cerr << "222error umounting\n" << std::endl;
    return(1);
  }
  cout << "222Successfully umounted Ceph!" << std::endl;

  ceph_deinitialize();
  cout << "222Successfully deinitialized Ceph!" << std::endl;

  return 0;
}

Re: rbd create error with 0.26

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3550 bytes --]

On Sun, 15 May 2011, Simon Tian wrote:
> > What is the IO size?  Is write_test_data synchronous?
> >
> > For simple write benchmarking you can also use
> >        rados mkpool foo
> >        rados -p foo bench <seconds> write -b <blocksize> -t <threads>
> >
> > and you'll see latency and throughput.  Blocksize defaults to 4M and
> > "threads" (parallel IOs) default to 16, IIRC.
> 
> 
> Hi, Sage:
> 
> I just did the bench:
> rados -p rbd bench 60 write -t 64   and    rados -p data bench 60 write -t 64
> the avg throughput is about 46MB/s, one of the result is as follow.
> But why it's slow with rbd api from <rbd/librbd.h>?

The problem is that your test is only doing a single IO at a time.  The 
request latency is relatively high because the data has to pass over the 
network to the OSD (and, for writes, do it again to be replicated), so 
the client node spends a lot of time waiting around.  The rados tool, by 
default, keeps 16 concurrent IOs in flight.

You'll want to look at the async (aio) read/write calls, or use multiple 
threads.

sage


> And I tried testlibrbdpp.cc, the result is more or less the same.
> The attachments are the test codes. Could you run it on your platform please?
> 
>  sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
>   40      63       482       419   41.8884        44   2.40044   2.40979
>   41      63       494       431   42.0372        48   2.11044     2.406
>   42      64       506       442   42.0837        44   2.11266   2.40229
>   43      63       518       455   42.3139        52   2.33468    2.3982
>   44      63       527       464   42.1703        36    2.4403   2.39559
>   45      63       539       476   42.2995        48   2.19768   2.39413
>   46      63       551       488   42.4232        48   2.51232    2.3928
>   47      63       563       500   42.5416        48   2.18025   2.38958
>   48      63       572       509   42.4051        36   2.27111   2.38791
>   49      63       584       521    42.519        48   2.41684   2.38695
>   50      63       596       533   42.6284        48   2.11087     2.384
>   51      63       608       545   42.7335        48   2.18147   2.37925
>   52      63       620       557   42.8345        48   2.45287   2.37787
>   53      63       629       566   42.7054        36   2.45187   2.37801
>   54      63       644       581   43.0255        60   2.22403   2.37477
>   55      63       653       590   42.8976        36   2.22782   2.37157
>   56      63       668       605   43.2026        60   2.20638   2.36597
>   57      63       677       614   43.0761        36   2.19628   2.36209
>   58      63       689       626   43.1608        48   2.18262   2.35762
>   59      63       704       641   43.4459        60   2.27029   2.35352
> min lat: 1.87981 max lat: 5.56194 avg lat: 2.34944
>  sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
>   60      63       716       653   43.5215        48   2.27835   2.34944
>   61      64       717       653    42.808         0         -   2.34944
>   62      63       717       654   42.1821         2   2.25694   2.34929
> Total time run:        62.274719
> Total writes made:     717
> Write size:            4194304
> Bandwidth (MB/sec):    46.054
> 
> Average Latency:       5.453
> Max latency:           62.0339
> Min latency:           1.87981
> N?????r??y???b?X??ÿÿv?^?)ÿÿ{.n?+???z?]z?{ay?\x1dÿÿÿÿ?,j\a??f???h???z?\x1e?w???\f???j:+v???w?j?m????\a????zZ+?????ÿÿj"??!?i

Re: rbd create error with 0.26

[Simon Tian]

Great!   Thx very much...
It's just that reason!!!
I just did "rados -p data bench 60 write -t 1 -b 1024", the speed is
more or less the same.

So I changed the IO_SIZE to 1MB and above in my test codes, the speed
became perfect.

There should be a buffer policy in the libceph api, so speed is not
hurt badly by the small IO_SIZE, is it right?  Hmm, where can I find
the buffer policy in the ceph code?

Thx!
Simon

2011/5/16 Sage Weil <sage@newdream.net>:
> On Sun, 15 May 2011, Simon Tian wrote:
>> > What is the IO size?  Is write_test_data synchronous?
>> >
>> > For simple write benchmarking you can also use
>> >        rados mkpool foo
>> >        rados -p foo bench <seconds> write -b <blocksize> -t <threads>
>> >
>> > and you'll see latency and throughput.  Blocksize defaults to 4M and
>> > "threads" (parallel IOs) default to 16, IIRC.
>>
>>
>> Hi, Sage:
>>
>> I just did the bench:
>> rados -p rbd bench 60 write -t 64   and    rados -p data bench 60 write -t 64
>> the avg throughput is about 46MB/s, one of the result is as follow.
>> But why it's slow with rbd api from <rbd/librbd.h>?
>
> The problem is that your test is only doing a single IO at a time.  The
> request latency is relatively high because the data has to pass over the
> network to the OSD (and, for writes, do it again to be replicated), so
> the client node spends a lot of time waiting around.  The rados tool, by
> default, keeps 16 concurrent IOs in flight.
>
> You'll want to look at the async (aio) read/write calls, or use multiple
> threads.
>
> sage
>
>
>> And I tried testlibrbdpp.cc, the result is more or less the same.
>> The attachments are the test codes. Could you run it on your platform please?
>>
>>  sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
>>   40      63       482       419   41.8884        44   2.40044   2.40979
>>   41      63       494       431   42.0372        48   2.11044     2.406
>>   42      64       506       442   42.0837        44   2.11266   2.40229
>>   43      63       518       455   42.3139        52   2.33468    2.3982
>>   44      63       527       464   42.1703        36    2.4403   2.39559
>>   45      63       539       476   42.2995        48   2.19768   2.39413
>>   46      63       551       488   42.4232        48   2.51232    2.3928
>>   47      63       563       500   42.5416        48   2.18025   2.38958
>>   48      63       572       509   42.4051        36   2.27111   2.38791
>>   49      63       584       521    42.519        48   2.41684   2.38695
>>   50      63       596       533   42.6284        48   2.11087     2.384
>>   51      63       608       545   42.7335        48   2.18147   2.37925
>>   52      63       620       557   42.8345        48   2.45287   2.37787
>>   53      63       629       566   42.7054        36   2.45187   2.37801
>>   54      63       644       581   43.0255        60   2.22403   2.37477
>>   55      63       653       590   42.8976        36   2.22782   2.37157
>>   56      63       668       605   43.2026        60   2.20638   2.36597
>>   57      63       677       614   43.0761        36   2.19628   2.36209
>>   58      63       689       626   43.1608        48   2.18262   2.35762
>>   59      63       704       641   43.4459        60   2.27029   2.35352
>> min lat: 1.87981 max lat: 5.56194 avg lat: 2.34944
>>  sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
>>   60      63       716       653   43.5215        48   2.27835   2.34944
>>   61      64       717       653    42.808         0         -   2.34944
>>   62      63       717       654   42.1821         2   2.25694   2.34929
>> Total time run:        62.274719
>> Total writes made:     717
>> Write size:            4194304
>> Bandwidth (MB/sec):    46.054
>>
>> Average Latency:       5.453
>> Max latency:           62.0339
>> Min latency:           1.87981
>> N?????r??y???b?X??ÿÿv?^?)ÿÿ{.n?+???z?]z?{ay? ÿÿÿÿ?,j ??f???h???z? ?w??? ???j:+v???w?j?m???? ????zZ+?????ÿÿj"??!?i

Re: rbd create error with 0.26

[Gregory Farnum]

On Sun, May 15, 2011 at 11:44 PM, Simon Tian <aixt2006@gmail.com> wrote:
> Great!   Thx very much...
> It's just that reason!!!
> I just did "rados -p data bench 60 write -t 1 -b 1024", the speed is
> more or less the same.
>
> So I changed the IO_SIZE to 1MB and above in my test codes, the speed
> became perfect.
>
> There should be a buffer policy in the libceph api, so speed is not
> hurt badly by the small IO_SIZE, is it right?  Hmm, where can I find
> the buffer policy in the ceph code?
>
> Thx!
> Simon

The Ceph code proper does do buffering, but the lower-level rados and
RBD apis don't have any at all -- applications that want it are
expected to implement that themselves. libceph uses the Client class
in the source code, and that does buffering and caching with the
ObjectCacher module. :)
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rbd create error with 0.26

[Simon Tian]

Hi folks,

      This assert failure appeared in ceph-0.31 when creating a pool,
could anyone help me?

Thx
Simon


2011/5/10 Simon Tian <aixt2006@gmail.com>:
> Hi folks,
>
> I met an error when creating a rbd image:
>
> ====================================================
> [root@mon-00 ~]# ceph class list
> 2011-05-10 17:36:29.530759 mon <- [class,list]
> 2011-05-10 17:36:29.531491 mon0 -> 'no installed classes!' (0)
> [root@mon-00 ~]# cclass -a
> Loading class: /usr/local/lib/rados-classes/libcls_rbd.so.1.0.0: rbd 1.3 x86-64
> 2011-05-10 17:36:37.657254 7eff9cfc36f0 read 184650 bytes from
> /usr/local/lib/rados-classes/libcls_rbd.so.1.0.0
> 2011-05-10 17:36:37.658820 mon <- [class,add,rbd,1.3,x86-64,changed]
> 2011-05-10 17:36:37.749791 mon0 -> 'updated' (0)
> [root@mon-00 ~]# ceph class list
> 2011-05-10 17:36:45.176860 mon <- [class,list]
> 2011-05-10 17:36:45.177511 mon1 -> 'installed classes:
> rbd (v1.3 [x86-64]) [active]
> ' (0)
> [root@mon-00 ~]# rbd list
> pool rbd doesn't contain rbd images
> [root@mon-00 ~]# rbd list
> pool rbd doesn't contain rbd images
> [root@mon-00 ~]# rbd create foo --size 1024
> 2011-05-10 17:42:14.854907 7fcff83cb6f0 librbd: failed to assign a
> block name for image
> create error: Input/output error
> [root@mon-00 ~]# ceph class activate rbd 1.3
> 2011-05-10 17:42:34.824676 mon <- [class,activate,rbd,1.3]
> 2011-05-10 17:42:34.944398 mon2 -> 'updated' (0)
> [root@mon-00 ~]# rbd create foo --size 1024
> 2011-05-10 17:42:44.227398 7fa9411866f0 librbd: failed to assign a
> block name for image
> create error: Input/output error
> ====================================================
> I just done as this tracker said:  http://tracker.newdream.net/issues/263
>
> I found that the return value of io_ctx.exec function is bellow zero:
>
>  r = io_ctx.exec(info_oid, "rbd", "assign_bid", bl, out);
>  if (r < 0)
>    return r;
>
> Thx!
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: rbd create error with 0.26

[Josh Durgin]

On Thu, 11 Aug 2011 22:49:11 +0800, Simon Tian <aixt2006@gmail.com>
wrote:
> Hi folks,
> 
>       This assert failure appeared in ceph-0.31 when creating a pool,
> could anyone help me?
> 
> Thx
> Simon

What's the error?

Re: rbd create error with 0.26

[Josh Durgin]

(bringing back to the list)

On 08/12/2011 12:33 AM, Simon Tian wrote:
> I have done all these:
> 1 cosd has permission to write /var/lib/ceph/tmp
> 2 The temp partition (/var/lib/ceph/tmp) has the execution permission
> ( no noexec mount option )
> 3 libcls_rbd.so is found in LD_LIBRARY_PATH or in the path specified
> in ceph.conf (osd class dir = /usr/lib64/rados-classes for distros
> like RHEL6)
>
> I am testing on RHEL5.4.
>
> The error didn't disappear....

Could you add 'debug osd = 20' to your ceph.conf, restart your osds, 
then try again? The osd log should tell us the cause of the error.

> 2011/8/12 Simon Tian<aixt2006@gmail.com>:
>> # rbd create foo --size 1024
>> 2011-08-12 10:13:13.180774 7f4c9a6656f0 librbd: failed to assign a
>> block name for image
>> create error: Input/output error
>>
>>
>> 2011/8/11 Josh Durgin<josh.durgin@dreamhost.com>:
>>> On Thu, 11 Aug 2011 22:49:11 +0800, Simon Tian<aixt2006@gmail.com>
>>> wrote:
>>>> Hi folks,
>>>>
>>>>        This assert failure appeared in ceph-0.31 when creating a pool,
>>>> could anyone help me?
>>>>
>>>> Thx
>>>> Simon
>>>
>>> What's the error?
>>>
>>

Re: rbd create error with 0.26

[Josh Durgin]

On 08/12/2011 10:56 AM, Josh Durgin wrote:
> (bringing back to the list)
>
> On 08/12/2011 12:33 AM, Simon Tian wrote:
>> I have done all these:
>> 1 cosd has permission to write /var/lib/ceph/tmp
>> 2 The temp partition (/var/lib/ceph/tmp) has the execution permission
>> ( no noexec mount option )
>> 3 libcls_rbd.so is found in LD_LIBRARY_PATH or in the path specified
>> in ceph.conf (osd class dir = /usr/lib64/rados-classes for distros
>> like RHEL6)
>>
>> I am testing on RHEL5.4.
>>
>> The error didn't disappear....
>
> Could you add 'debug osd = 20' to your ceph.conf, restart your osds,
> then try again? The osd log should tell us the cause of the error.
>

On further inspection, the wiki was wrong - LD_LIBRARY_PATH does not 
affect class loading. You probably just need to set 'osd class dir = 
/usr/lib64/rados-classes'.

>> 2011/8/12 Simon Tian<aixt2006@gmail.com>:
>>> # rbd create foo --size 1024
>>> 2011-08-12 10:13:13.180774 7f4c9a6656f0 librbd: failed to assign a
>>> block name for image
>>> create error: Input/output error
>>>
>>>
>>> 2011/8/11 Josh Durgin<josh.durgin@dreamhost.com>:
>>>> On Thu, 11 Aug 2011 22:49:11 +0800, Simon Tian<aixt2006@gmail.com>
>>>> wrote:
>>>>> Hi folks,
>>>>>
>>>>> This assert failure appeared in ceph-0.31 when creating a pool,
>>>>> could anyone help me?
>>>>>
>>>>> Thx
>>>>> Simon
>>>>
>>>> What's the error?
>>>>
>>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: rbd create error with 0.26

[Simon Tian]

Thx very much for your guidance, It's OK now!


2011/8/13 Josh Durgin <josh.durgin@dreamhost.com>:
> On 08/12/2011 10:56 AM, Josh Durgin wrote:
>>
>> (bringing back to the list)
>>
>> On 08/12/2011 12:33 AM, Simon Tian wrote:
>>>
>>> I have done all these:
>>> 1 cosd has permission to write /var/lib/ceph/tmp
>>> 2 The temp partition (/var/lib/ceph/tmp) has the execution permission
>>> ( no noexec mount option )
>>> 3 libcls_rbd.so is found in LD_LIBRARY_PATH or in the path specified
>>> in ceph.conf (osd class dir = /usr/lib64/rados-classes for distros
>>> like RHEL6)
>>>
>>> I am testing on RHEL5.4.
>>>
>>> The error didn't disappear....
>>
>> Could you add 'debug osd = 20' to your ceph.conf, restart your osds,
>> then try again? The osd log should tell us the cause of the error.
>>
>
> On further inspection, the wiki was wrong - LD_LIBRARY_PATH does not affect
> class loading. You probably just need to set 'osd class dir =
> /usr/lib64/rados-classes'.
>
>>> 2011/8/12 Simon Tian<aixt2006@gmail.com>:
>>>>
>>>> # rbd create foo --size 1024
>>>> 2011-08-12 10:13:13.180774 7f4c9a6656f0 librbd: failed to assign a
>>>> block name for image
>>>> create error: Input/output error
>>>>
>>>>
>>>> 2011/8/11 Josh Durgin<josh.durgin@dreamhost.com>:
>>>>>
>>>>> On Thu, 11 Aug 2011 22:49:11 +0800, Simon Tian<aixt2006@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Hi folks,
>>>>>>
>>>>>> This assert failure appeared in ceph-0.31 when creating a pool,
>>>>>> could anyone help me?
>>>>>>
>>>>>> Thx
>>>>>> Simon
>>>>>
>>>>> What's the error?
>>>>>
>>>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>