Re: RBD Async request: When / How are the call back called ?

[Josh Durgin <josh.durgin@inktank.com>]

On 08/29/2012 07:57 AM, Sylvain Munaut wrote:
> Hi,
>
>
> It might be obvious for people knowing the API but somehow I can't
> figure it out:
>
> How and when will the call back specified in a rbd_completion_t be called ?
>
> Imagine I do a rbd_aio_write and then do while (1); ... I don't see
> how the library could call my callback unless there is threads
> involved (and then my cb would be called in another thread context
> which is not great ...)

It's called from another thread. There are several threads for
messaging started when you connect with a rados_cluster_t. When a
request is complete, it's put into a queue inside librados, and a
'finisher' thread reads from the queue and calls callbacks as
appropriate.

In the case of rbd, a single rbd_aio_write may translate to many rados
requests, but when the last is complete, the callback you provided is
called by the librados finisher thread. At this point the data is on
disk on all replicas.

> (Of course the 'while (1)' here is just a stub, in reality it's a call
> to a library beyond my control)
>
>
> My problem here is that I need to integrate inside the main loop of
> another software and I can't block, but I still need to be able to
> 'notify' when the requests are over. I can ask the scheduler of that
> main loop to call me on some even on a file descriptor (but I don't
> see any fd here), or to call me periodically (but I don't see any call
> to test for completion without waiting for it).

The way this is done in the qemu block driver (since qemu I/O
must be completed from the original qemu thread context) is to have a
pipe that the qemu thread reads from. When an rbd I/O completes, the
callback (called by a librados thread) tells the qemu thread that it's
done via the pipe. Here's the original discussion about this:

http://lists.gnu.org/archive/html/qemu-devel/2010-10/msg00420.html

and the code:

http://git.qemu.org/?p=qemu.git;a=blob;f=block/rbd.c;h=5a0f79fc8f38e90c5c635a86b60db4c0f2f20c2f;hb=HEAD

> Cheers,
>
>       Sylvain