RBD Async request: When / How are the call back called ?

RBD Async request: When / How are the call back called ?

[Sylvain Munaut]

Hi,


It might be obvious for people knowing the API but somehow I can't
figure it out:

How and when will the call back specified in a rbd_completion_t be called ?

Imagine I do a rbd_aio_write and then do while (1); ... I don't see
how the library could call my callback unless there is threads
involved (and then my cb would be called in another thread context
which is not great ...)

(Of course the 'while (1)' here is just a stub, in reality it's a call
to a library beyond my control)


My problem here is that I need to integrate inside the main loop of
another software and I can't block, but I still need to be able to
'notify' when the requests are over. I can ask the scheduler of that
main loop to call me on some even on a file descriptor (but I don't
see any fd here), or to call me periodically (but I don't see any call
to test for completion without waiting for it).


Cheers,

     Sylvain

Re: RBD Async request: When / How are the call back called ?

[Mike Ryan]

On Wed, Aug 29, 2012 at 04:57:39PM +0200, Sylvain Munaut wrote:
> How and when will the call back specified in a rbd_completion_t be called ?
> 
> Imagine I do a rbd_aio_write and then do while (1); ... I don't see
> how the library could call my callback unless there is threads
> involved (and then my cb would be called in another thread context
> which is not great ...)

I believe it's called from another thread.

> My problem here is that I need to integrate inside the main loop of
> another software and I can't block, but I still need to be able to
> 'notify' when the requests are over. I can ask the scheduler of that
> main loop to call me on some even on a file descriptor (but I don't
> see any fd here), or to call me periodically (but I don't see any call
> to test for completion without waiting for it).

from include/rados/librados.hpp:

  struct AioCompletion {
    ...
    bool is_complete();
    ...
  };

Is that sufficient?

Re: RBD Async request: When / How are the call back called ?

[Josh Durgin]

On 08/29/2012 07:57 AM, Sylvain Munaut wrote:
> Hi,
>
>
> It might be obvious for people knowing the API but somehow I can't
> figure it out:
>
> How and when will the call back specified in a rbd_completion_t be called ?
>
> Imagine I do a rbd_aio_write and then do while (1); ... I don't see
> how the library could call my callback unless there is threads
> involved (and then my cb would be called in another thread context
> which is not great ...)

It's called from another thread. There are several threads for
messaging started when you connect with a rados_cluster_t. When a
request is complete, it's put into a queue inside librados, and a
'finisher' thread reads from the queue and calls callbacks as
appropriate.

In the case of rbd, a single rbd_aio_write may translate to many rados
requests, but when the last is complete, the callback you provided is
called by the librados finisher thread. At this point the data is on
disk on all replicas.

> (Of course the 'while (1)' here is just a stub, in reality it's a call
> to a library beyond my control)
>
>
> My problem here is that I need to integrate inside the main loop of
> another software and I can't block, but I still need to be able to
> 'notify' when the requests are over. I can ask the scheduler of that
> main loop to call me on some even on a file descriptor (but I don't
> see any fd here), or to call me periodically (but I don't see any call
> to test for completion without waiting for it).

The way this is done in the qemu block driver (since qemu I/O
must be completed from the original qemu thread context) is to have a
pipe that the qemu thread reads from. When an rbd I/O completes, the
callback (called by a librados thread) tells the qemu thread that it's
done via the pipe. Here's the original discussion about this:

http://lists.gnu.org/archive/html/qemu-devel/2010-10/msg00420.html

and the code:

http://git.qemu.org/?p=qemu.git;a=blob;f=block/rbd.c;h=5a0f79fc8f38e90c5c635a86b60db4c0f2f20c2f;hb=HEAD

> Cheers,
>
>       Sylvain

Re: RBD Async request: When / How are the call back called ?

[Sylvain Munaut]

Hi,

Thanks Mike and Josh for the answer, I see how it works now.

Using a pipe seems like a good fit for me.

One more question: When I have async read pending, I would have
expected rbd_flush and/or rbd_close to wait for them to finish but
they dont. Is there a way to wait for all operation to be complete ?
Because if I shutdown everything before they are, the application
hangs ...

Cheers,

    Sylvain

Re: RBD Async request: When / How are the call back called ?

[Mike Ryan]

On Thu, Aug 30, 2012 at 02:16:36PM +0200, Sylvain Munaut wrote:
> One more question: When I have async read pending, I would have
> expected rbd_flush and/or rbd_close to wait for them to finish but
> they dont. Is there a way to wait for all operation to be complete ?
> Because if I shutdown everything before they are, the application
> hangs ...

There's nothing in the API for that. You have to manually iterate over
all your completion contexts calling wait_for_safe_and_cb() on each.

Note: even if you don't have a callback, you should use the _cb()
version. I just fixed a use-after-free bug the was caused by this.

Re: RBD Async request: When / How are the call back called ?

[Josh Durgin]

On 08/30/2012 07:52 AM, Mike Ryan wrote:
> On Thu, Aug 30, 2012 at 02:16:36PM +0200, Sylvain Munaut wrote:
>> One more question: When I have async read pending, I would have
>> expected rbd_flush and/or rbd_close to wait for them to finish but
>> they dont. Is there a way to wait for all operation to be complete ?
>> Because if I shutdown everything before they are, the application
>> hangs ...
>
> There's nothing in the API for that. You have to manually iterate over
> all your completion contexts calling wait_for_safe_and_cb() on each.
>
> Note: even if you don't have a callback, you should use the _cb()
> version. I just fixed a use-after-free bug the was caused by this.

That's in librados.

In librbd there's only wait_for_complete(), which
waits for the callback to complete as well.

Josh