Question about librados

Question about librados

[Rutger ter Borg]

Hello,

I'm in the progress of evaluating librados as an object store. I'm using 
Debian's latest packages as of today, and noted that I need to define 
NO_ATOMIC_OPS to get something compiled that includes librados.hpp.

My actual question: is there a requirement/optimum on the relation 
between number of objects and number of pools? Or may this be chosen to 
be something completely arbitrary? I.e., is it a problem to have a 
couple of hundred IoCtxs active in one process? What is a 
reasonable/performance-wise-good object-size?

TIA,

Rutger

Re: Question about librados

[Gregory Farnum]

On Mon, Apr 4, 2011 at 4:33 AM, Rutger ter Borg <rutger@terborg.net> wrote:
>
> Hello,
>
> I'm in the progress of evaluating librados as an object store. I'm using
> Debian's latest packages as of today, and noted that I need to define
> NO_ATOMIC_OPS to get something compiled that includes librados.hpp.
Hmmm, that's not right! Can you elaborate? It just doesn't build
librados.hpp unless you define NO_ATOMIC_OPS? Is there a build error
at some point?

> My actual question: is there a requirement/optimum on the relation between
> number of objects and number of pools? Or may this be chosen to be something
> completely arbitrary?
There's no optimal relationship at all between the number of objects
and the number of pools.Pools are logical groupings that have very
little impact on performance.
Placement groups are rather more important to performance, but again
the number of objects/PG isn't terribly significant as long as
#objects >> #PGs (for optimal performance you want ~100 PGs/OSD, but
this number can vary significantly before it becomes a problem).

> I.e., is it a problem to have a couple of hundred
> IoCtxs active in one process?
Not at all!

> What is a reasonable/performance-wise-good object-size?
Hmm, that depends on your specific hardware configuration. Ceph itself
uses 4MB objects by default. In general the size of objects can vary a
great deal without issue. There may still be some issues with very
large objects (where object size is in the same order of magnitude as
the OSD's RAM) but I think these have all been dealt with.
In general the thing to be aware of is that objects are the atomic
unit as far as librados and CRUSH are concerned -- placement within
the cluster is pseudo-random by object, but if the size of your
objects varies dramatically then the disk utilization can vary a lot
without RADOS becoming aware of it. (There are mechanisms to deal with
overloaded OSDs so it's not a dealbreaker, just something to be aware
of.) Also, recovery and rebalancing and such, while logically handled
by placement group, are largely dependent on the objects -- so if you
have very large objects you may get strange behavior during recovery
since the throttling and such are designed for objects in the
several-MB range.
-Greg

Re: Question about librados

[Rutger ter Borg]

On 04/04/2011 06:09 PM, Gregory Farnum wrote:
> On Mon, Apr 4, 2011 at 4:33 AM, Rutger ter Borg<rutger@terborg.net>  wrote:
>>
>> Hello,
>>
>> I'm in the progress of evaluating librados as an object store. I'm using
>> Debian's latest packages as of today, and noted that I need to define
>> NO_ATOMIC_OPS to get something compiled that includes librados.hpp.
> Hmmm, that's not right! Can you elaborate? It just doesn't build
> librados.hpp unless you define NO_ATOMIC_OPS? Is there a build error
> at some point?

[snip]

Hello Greg,

thanks for the elaborate answer! With respect to the compile error, I 
got this in an empty file with just including rados/librados.hpp

$ g++ ./rados.cpp
In file included from /usr/include/rados/buffer.h:55:0,
                  from /usr/include/rados/librados.hpp:10,
                  from ./rados.cpp:6:
/usr/include/rados/atomic.h:25:24: fatal error: atomic_ops.h: No such 
file or directory 

compilation terminated.

Maybe it has to do with the .deb, apt pulled in librados2,

$ dpkg --status librados-dev
Version: 0.25.2-1 


Cheers,

Rutger

Re: Question about librados

[Gregory Farnum]

On Mon, Apr 4, 2011 at 11:44 PM, Rutger ter Borg <rutger@terborg.net> wrote:
> thanks for the elaborate answer! With respect to the compile error, I got
> this in an empty file with just including rados/librados.hpp
>
> $ g++ ./rados.cpp
> In file included from /usr/include/rados/buffer.h:55:0,
>                 from /usr/include/rados/librados.hpp:10,
>                 from ./rados.cpp:6:
> /usr/include/rados/atomic.h:25:24: fatal error: atomic_ops.h: No such file
> or directory
> compilation terminated.
Have you tried using the full make system (go to src dir,
./autogen.sh; ./configure; make)? All the required headers should get
pulled in when you install the package, but they might not be in your
default include paths.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Question about librados

[Rutger ter Borg]

On 04/05/2011 03:42 PM, Gregory Farnum wrote:
>
> Have you tried using the full make system (go to src dir,
> ./autogen.sh; ./configure; make)? All the required headers should get
> pulled in when you install the package, but they might not be in your
> default include paths.

No, I haven't tried compiling from source (yet). I did try the 
following, though

rutger@gonzo:~$ dpkg -S atomic_ops
dpkg-query: no path found matching pattern *atomic_ops*.

which made me conclude that I don't have the file. Probably a package 
thing; I'm using the

deb http://ceph.newdream.net/debian/ sid main

repository on Debian sid amd64.

Cheers,

Rutger

Re: Question about librados

[Wido den Hollander]

Hi,
----- Original message -----
> On 04/05/2011 03:42 PM, Gregory Farnum wrote:
> > 
> > Have you tried using the full make system (go to src dir,
> > ./autogen.sh; ./configure; make)? All the required headers should get
> > pulled in when you install the package, but they might not be in your
> > default include paths.
> 
> No, I haven't tried compiling from source (yet). I did try the 
> following, though
> 
> rutger@gonzo:~$ dpkg -S atomic_ops
> dpkg-query: no path found matching pattern *atomic_ops*.
> 
> which made me conclude that I don't have the file. Probably a package 
> thing; I'm using the
> 
> deb http://ceph.newdream.net/debian/ sid main
> 
> repository on Debian sid amd64.
> 

On Debian/Ubuntu that file is in libatomic-ops-dev, it might be useful if librados-dev depended on that package.

Wido

Re: Question about librados

[Tommi Virtanen]

On Tue, Apr 05, 2011 at 08:44:15AM +0200, Rutger ter Borg wrote:
> >>I'm in the progress of evaluating librados as an object store. I'm using
> >>Debian's latest packages as of today, and noted that I need to define
> >>NO_ATOMIC_OPS to get something compiled that includes librados.hpp.
> >Hmmm, that's not right! Can you elaborate? It just doesn't build
> >librados.hpp unless you define NO_ATOMIC_OPS? Is there a build error
> >at some point?
> thanks for the elaborate answer! With respect to the compile error,
> I got this in an empty file with just including rados/librados.hpp
> 
> $ g++ ./rados.cpp
> In file included from /usr/include/rados/buffer.h:55:0,
>                  from /usr/include/rados/librados.hpp:10,
>                  from ./rados.cpp:6:
> /usr/include/rados/atomic.h:25:24: fatal error: atomic_ops.h: No
> such file or directory
> 
> compilation terminated.

Ahhh, now I got it. You're not trying to compile Ceph, you're trying
to build software using librados.

The problem here is that whatever our autoconf setup detects, and
whatever libraries it decides to use, might not be available at the
time the external software using librados is compiles. That is, Ceph
compilation environment is different from the end user compilation
environment.

And, even more so, we don't use the autoconfig results at all in the
include files; nothing defines __CEPH__ in this case.

     18 #ifdef __CEPH__
     19 # include "acconfig.h"
     20 #endif

So when atomic.h says:

     23 #ifndef NO_ATOMIC_OPS
     24 //libatomic_ops implementation
     25 #include <atomic_ops.h>
     26 namespace ceph {
     27 

Yup, that'll fail if you don't have atomic-ops development libraries
available.

And even if it compiles, there's no quarantee it works right! Because
what if the actual Ceph compilation really was *without* atomic-ops.

I need to file a few more bug reports next, but after that I'll look
into this more and try to come up with a better recommendation.

Short term, your best bet is to try to match the environment where the
Deb was compiled; install the packages listed on the Build-Depends
line at
http://ceph.newdream.net/git/?p=ceph.git;a=blob;f=debian/control;hb=HEAD
-- you can do that quickly with "sudo apt-get build-dep ceph"

-- 
:(){ :|:&};: