Any librados C API users out there?

Any librados C API users out there?

[Piotr Dałek]

Hello,

As the subject says - are here any users/consumers of librados C API? I'm asking because we're researching if this PR: 
https://github.com/ceph/ceph/pull/12216 will be actually beneficial for larger group of users. This PR adds a bunch of new APIs that perform 
object writes without intermediate data copy, which will reduce cpu and memory load on clients. If you're using librados C API for object 
writes, feel free to comment here or in the pull request.


-- 
Piotr Dałek


_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Any librados C API users out there?

[Jason Dillaman]

+1

I'd be happy to tweak the internals of librbd to support pass-through
of C buffers all the way to librados. librbd clients like QEMU use the
C API and this currently results in several extra copies (in librbd
and librados).

On Wed, Jan 11, 2017 at 11:44 AM, Piotr Dałek <piotr.dalek@corp.ovh.com> wrote:
> Hello,
>
> As the subject says - are here any users/consumers of librados C API? I'm
> asking because we're researching if this PR:
> https://github.com/ceph/ceph/pull/12216 will be actually beneficial for
> larger group of users. This PR adds a bunch of new APIs that perform object
> writes without intermediate data copy, which will reduce cpu and memory load
> on clients. If you're using librados C API for object writes, feel free to
> comment here or in the pull request.
>
>
> --
> Piotr Dałek
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Jason
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Any librados C API users out there?

[Samuel Just]

Jason: librbd itself uses the librados C++ api though, right?
-Sam

On Wed, Jan 11, 2017 at 9:37 AM, Jason Dillaman <jdillama@redhat.com> wrote:
> +1
>
> I'd be happy to tweak the internals of librbd to support pass-through
> of C buffers all the way to librados. librbd clients like QEMU use the
> C API and this currently results in several extra copies (in librbd
> and librados).
>
> On Wed, Jan 11, 2017 at 11:44 AM, Piotr Dałek <piotr.dalek@corp.ovh.com> wrote:
>> Hello,
>>
>> As the subject says - are here any users/consumers of librados C API? I'm
>> asking because we're researching if this PR:
>> https://github.com/ceph/ceph/pull/12216 will be actually beneficial for
>> larger group of users. This PR adds a bunch of new APIs that perform object
>> writes without intermediate data copy, which will reduce cpu and memory load
>> on clients. If you're using librados C API for object writes, feel free to
>> comment here or in the pull request.
>>
>>
>> --
>> Piotr Dałek
>>
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> --
> Jason
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Any librados C API users out there?

[Jason Dillaman]

It does internally -- which requires the extra copy from C array to a
bufferlist. I had a PR for wrapping the C array into a bufferlist (w/o
the copy), but Sage pointed out a potential issue with such
implementations (which might still be an issue w/ this PR).

[1] https://github.com/yuyuyu101/ceph/commit/794b49b5b860c538a349bdadb16bb6ae97ad9c20

On Wed, Jan 11, 2017 at 12:42 PM, Samuel Just <sjust@redhat.com> wrote:
> Jason: librbd itself uses the librados C++ api though, right?
> -Sam
>
> On Wed, Jan 11, 2017 at 9:37 AM, Jason Dillaman <jdillama@redhat.com> wrote:
>> +1
>>
>> I'd be happy to tweak the internals of librbd to support pass-through
>> of C buffers all the way to librados. librbd clients like QEMU use the
>> C API and this currently results in several extra copies (in librbd
>> and librados).
>>
>> On Wed, Jan 11, 2017 at 11:44 AM, Piotr Dałek <piotr.dalek@corp.ovh.com> wrote:
>>> Hello,
>>>
>>> As the subject says - are here any users/consumers of librados C API? I'm
>>> asking because we're researching if this PR:
>>> https://github.com/ceph/ceph/pull/12216 will be actually beneficial for
>>> larger group of users. This PR adds a bunch of new APIs that perform object
>>> writes without intermediate data copy, which will reduce cpu and memory load
>>> on clients. If you're using librados C API for object writes, feel free to
>>> comment here or in the pull request.
>>>
>>>
>>> --
>>> Piotr Dałek
>>>
>>>
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>> --
>> Jason
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Jason

Re: Any librados C API users out there?

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2655 bytes --]

On Wed, 11 Jan 2017, Jason Dillaman wrote:
> +1
> 
> I'd be happy to tweak the internals of librbd to support pass-through
> of C buffers all the way to librados. librbd clients like QEMU use the
> C API and this currently results in several extra copies (in librbd
> and librados).

+1 from me too.

The caveat is that we have to be very careful with buffers that are 
provided by users.  Currently the userspace messenger code doesn't provide 
a way to manage the provenance of references to the buffer::raw_static 
buffers, which means that even if the write has completed, there may be 
ways for an MOSDOp to still be alive that references that memory.

Either (1) we have to audit the code to be sure that by the time the 
Objecter request completes we know that all messages and their bufferlists 
are cleared (tricky/fragile), or (2) introduce some buffer management 
interface in librados so that the buffer lifecycle is independent of the 
request.  I would prefer (2), but it means the interfaces would be 
something like

 rados_buffer_create(...)
 copy your data into that buffer
 rados_write(...) or whatever
 rados_buffer_release(...)

and then rados can do the proper refcounting and only deallocate the 
memory when all refs have gone away.  Unfortunately, I suspect that there 
is a largish category of users where this isn't sufficient... e.g., if 
some existing C user has its own buffer and it isn't practical to 
allocate/release via rados_buffer_* calls instead of malloc/free (or 
whatever).

Jason, where does librbd fall?

sage


> 
> On Wed, Jan 11, 2017 at 11:44 AM, Piotr Dałek <piotr.dalek-Rm6v+N6rxxBWk0Htik3J/w@public.gmane.org> wrote:
> > Hello,
> >
> > As the subject says - are here any users/consumers of librados C API? I'm
> > asking because we're researching if this PR:
> > https://github.com/ceph/ceph/pull/12216 will be actually beneficial for
> > larger group of users. This PR adds a bunch of new APIs that perform object
> > writes without intermediate data copy, which will reduce cpu and memory load
> > on clients. If you're using librados C API for object writes, feel free to
> > comment here or in the pull request.
> >
> >
> > --
> > Piotr Dałek
> >
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
> -- 
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Any librados C API users out there?

[Jason Dillaman]

On Wed, Jan 11, 2017 at 1:01 PM, Sage Weil <sage-BnTBU8nroG7k1uMJSBkQmQ@public.gmane.org> wrote:
> Jason, where does librbd fall?

Option (2) won't help for users like QEMU unless we can tie the
reference counting back into the AioCompletion (i.e. delay firing
until all references to the memory are released).

-- 
Jason

Re: [ceph-users] Any librados C API users out there?

[Piotr Dałek]

On 01/11/2017 07:01 PM, Sage Weil wrote:
> On Wed, 11 Jan 2017, Jason Dillaman wrote:
>> On Wed, Jan 11, 2017 at 11:44 AM, Piotr Dałek <piotr.dalek@corp.ovh.com> wrote:
>>> As the subject says - are here any users/consumers of librados C API? I'm
>>> asking because we're researching if this PR:
>>> https://github.com/ceph/ceph/pull/12216 will be actually beneficial for
>>> larger group of users. This PR adds a bunch of new APIs that perform object
>>> writes without intermediate data copy, which will reduce cpu and memory load
>>> on clients. If you're using librados C API for object writes, feel free to
>>> comment here or in the pull request.
>> +1
>>
>> I'd be happy to tweak the internals of librbd to support pass-through
>> of C buffers all the way to librados. librbd clients like QEMU use the
>> C API and this currently results in several extra copies (in librbd
>> and librados).
>
> +1 from me too.
>
> The caveat is that we have to be very careful with buffers that are
> provided by users.  Currently the userspace messenger code doesn't provide
> a way to manage the provenance of references to the buffer::raw_static
> buffers, which means that even if the write has completed, there may be
> ways for an MOSDOp to still be alive that references that memory.
>
> Either (1) we have to audit the code to be sure that by the time the
> Objecter request completes we know that all messages and their bufferlists
> are cleared (tricky/fragile), or (2) introduce some buffer management
> interface in librados so that the buffer lifecycle is independent of the
> request.  I would prefer (2), but it means the interfaces would be
> something like
>
>  rados_buffer_create(...)
>  copy your data into that buffer
>  rados_write(...) or whatever
>  rados_buffer_release(...)
>
> and then rados can do the proper refcounting and only deallocate the
> memory when all refs have gone away.  Unfortunately, I suspect that there
> is a largish category of users where this isn't sufficient... e.g., if
> some existing C user has its own buffer and it isn't practical to
> allocate/release via rados_buffer_* calls instead of malloc/free (or
> whatever).

Personally I vote for (1). That way is time consuming, but may help us find other ways to optimize resource consumption of API itself. In 
particular, if librbd, as you all wrote, does several copies of user buffers, then maybe it's worth spending some time on figuring out the 
actual lifetime of buffers? Maybe, for example, buffers in MOSDOp are actually copies of already copied buffers, so the extra copy done by 
librados is unnecessary?
Regardless, the idea behind my PR wasn't to make librados/librbd/libwhatever 100% optimal right away, but to make one step towards it being 
efficient - recduce cpu and memory usage a bit, have it go through testing, if nothing breaks, try harder until we're close to perfection.
even if few percent decrease in memory consumption doesn't look interesting on single-client level, few percents in large scale deployments may 
mean hundreds of gigabytes of memory that could be put to better use.

(2), and particularly the requirement for API changes, along with need to put user data into specific RADOS structure doesn't look good because
- that requires way more changes to user code, and I don't expect any of current user to be interested in that (unless we force it with API change)
- most of devs are simply lazy and instead of using "rados_buffer_create" to put their data directly into them, they'll keep using their own 
buffers and *then* copy them into RADOS buffers, just like we're doing it right now (even if behind the scenes).

-- 
Piotr Dałek
piotr.dalek@corp.ovh.com
https://www.ovh.com/us/

Re: Any librados C API users out there?

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 4895 bytes --]

On Thu, 12 Jan 2017, Piotr Dałek wrote:
> On 01/11/2017 07:01 PM, Sage Weil wrote:
> > On Wed, 11 Jan 2017, Jason Dillaman wrote:
> > > On Wed, Jan 11, 2017 at 11:44 AM, Piotr Dałek <piotr.dalek-Rm6v+N6rxxBWk0Htik3J/w@public.gmane.org>
> > > wrote:
> > > > As the subject says - are here any users/consumers of librados C API?
> > > > I'm
> > > > asking because we're researching if this PR:
> > > > https://github.com/ceph/ceph/pull/12216 will be actually beneficial for
> > > > larger group of users. This PR adds a bunch of new APIs that perform
> > > > object
> > > > writes without intermediate data copy, which will reduce cpu and memory
> > > > load
> > > > on clients. If you're using librados C API for object writes, feel free
> > > > to
> > > > comment here or in the pull request.
> > > +1
> > > 
> > > I'd be happy to tweak the internals of librbd to support pass-through
> > > of C buffers all the way to librados. librbd clients like QEMU use the
> > > C API and this currently results in several extra copies (in librbd
> > > and librados).
> > 
> > +1 from me too.
> > 
> > The caveat is that we have to be very careful with buffers that are
> > provided by users.  Currently the userspace messenger code doesn't provide
> > a way to manage the provenance of references to the buffer::raw_static
> > buffers, which means that even if the write has completed, there may be
> > ways for an MOSDOp to still be alive that references that memory.
> > 
> > Either (1) we have to audit the code to be sure that by the time the
> > Objecter request completes we know that all messages and their bufferlists
> > are cleared (tricky/fragile), or (2) introduce some buffer management
> > interface in librados so that the buffer lifecycle is independent of the
> > request.  I would prefer (2), but it means the interfaces would be
> > something like
> > 
> >  rados_buffer_create(...)
> >  copy your data into that buffer
> >  rados_write(...) or whatever
> >  rados_buffer_release(...)
> > 
> > and then rados can do the proper refcounting and only deallocate the
> > memory when all refs have gone away.  Unfortunately, I suspect that there
> > is a largish category of users where this isn't sufficient... e.g., if
> > some existing C user has its own buffer and it isn't practical to
> > allocate/release via rados_buffer_* calls instead of malloc/free (or
> > whatever).
> 
> Personally I vote for (1). That way is time consuming, but may help us find
> other ways to optimize resource consumption of API itself. In particular, if
> librbd, as you all wrote, does several copies of user buffers, then maybe it's
> worth spending some time on figuring out the actual lifetime of buffers?
> Maybe, for example, buffers in MOSDOp are actually copies of already copied
> buffers, so the extra copy done by librados is unnecessary?
> Regardless, the idea behind my PR wasn't to make librados/librbd/libwhatever
> 100% optimal right away, but to make one step towards it being efficient -
> recduce cpu and memory usage a bit, have it go through testing, if nothing
> breaks, try harder until we're close to perfection.
> even if few percent decrease in memory consumption doesn't look interesting on
> single-client level, few percents in large scale deployments may mean hundreds
> of gigabytes of memory that could be put to better use.
> 
> (2), and particularly the requirement for API changes, along with need to put
> user data into specific RADOS structure doesn't look good because
> - that requires way more changes to user code, and I don't expect any of
> current user to be interested in that (unless we force it with API change)
> - most of devs are simply lazy and instead of using "rados_buffer_create" to
> put their data directly into them, they'll keep using their own buffers and
> *then* copy them into RADOS buffers, just like we're doing it right now (even
> if behind the scenes).

Yeah, I think you're right.  The bad news is just that (1) is hard.  It's 
going to require a few things in order to address the problem of sending 
multiple MOSDOp requests for the same operation (e.g., after peering).  I 
think it will mean

- The Message superclass is going to require a new Mutex that is taken by 
the messenger implementation while accessing the encoded message payload 
buffer

- It will also need a revoke_buffers() method (that takes the mutex) so 
that Objecter can yank references to buffers for any messages in flight 
that it no longer cares about

- Objecter will need to keep a ref of the in-flight request of record.  If 
it needs to resend to a different OSD, it can use that ref to revoke the 
buffers.

- Once the operation completes, Objecter can drop it's ref to that 
Message.

Until we do that, any zero-copy changes like those in the PR will mostly 
work but trigger use-after-free is various cases where the PG mappings are 
changing.

sage

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Any librados C API users out there?

[Jason Dillaman]

There is option (3) which is to have a new (or modified)
"buffer::create_static" take an optional callback to invoke when the
buffer::raw object is destructed. The raw pointer would be destructed
when the last buffer::ptr / buffer::list containing it is destructed,
so you know it's no longer being referenced.

You could then have the new C API methods that wrap the C buffer in a
bufferlist and set a new flag in the librados::AioCompletion to delay
its completion until after it's both completed and the memory is
released. When the buffer is freed, the callback would unblock the
librados::AioCompltion completion callback.

On Thu, Jan 12, 2017 at 8:48 AM, Sage Weil <sweil@redhat.com> wrote:
> On Thu, 12 Jan 2017, Piotr Dałek wrote:
>> On 01/11/2017 07:01 PM, Sage Weil wrote:
>> > On Wed, 11 Jan 2017, Jason Dillaman wrote:
>> > > On Wed, Jan 11, 2017 at 11:44 AM, Piotr Dałek <piotr.dalek@corp.ovh.com>
>> > > wrote:
>> > > > As the subject says - are here any users/consumers of librados C API?
>> > > > I'm
>> > > > asking because we're researching if this PR:
>> > > > https://github.com/ceph/ceph/pull/12216 will be actually beneficial for
>> > > > larger group of users. This PR adds a bunch of new APIs that perform
>> > > > object
>> > > > writes without intermediate data copy, which will reduce cpu and memory
>> > > > load
>> > > > on clients. If you're using librados C API for object writes, feel free
>> > > > to
>> > > > comment here or in the pull request.
>> > > +1
>> > >
>> > > I'd be happy to tweak the internals of librbd to support pass-through
>> > > of C buffers all the way to librados. librbd clients like QEMU use the
>> > > C API and this currently results in several extra copies (in librbd
>> > > and librados).
>> >
>> > +1 from me too.
>> >
>> > The caveat is that we have to be very careful with buffers that are
>> > provided by users.  Currently the userspace messenger code doesn't provide
>> > a way to manage the provenance of references to the buffer::raw_static
>> > buffers, which means that even if the write has completed, there may be
>> > ways for an MOSDOp to still be alive that references that memory.
>> >
>> > Either (1) we have to audit the code to be sure that by the time the
>> > Objecter request completes we know that all messages and their bufferlists
>> > are cleared (tricky/fragile), or (2) introduce some buffer management
>> > interface in librados so that the buffer lifecycle is independent of the
>> > request.  I would prefer (2), but it means the interfaces would be
>> > something like
>> >
>> >  rados_buffer_create(...)
>> >  copy your data into that buffer
>> >  rados_write(...) or whatever
>> >  rados_buffer_release(...)
>> >
>> > and then rados can do the proper refcounting and only deallocate the
>> > memory when all refs have gone away.  Unfortunately, I suspect that there
>> > is a largish category of users where this isn't sufficient... e.g., if
>> > some existing C user has its own buffer and it isn't practical to
>> > allocate/release via rados_buffer_* calls instead of malloc/free (or
>> > whatever).
>>
>> Personally I vote for (1). That way is time consuming, but may help us find
>> other ways to optimize resource consumption of API itself. In particular, if
>> librbd, as you all wrote, does several copies of user buffers, then maybe it's
>> worth spending some time on figuring out the actual lifetime of buffers?
>> Maybe, for example, buffers in MOSDOp are actually copies of already copied
>> buffers, so the extra copy done by librados is unnecessary?
>> Regardless, the idea behind my PR wasn't to make librados/librbd/libwhatever
>> 100% optimal right away, but to make one step towards it being efficient -
>> recduce cpu and memory usage a bit, have it go through testing, if nothing
>> breaks, try harder until we're close to perfection.
>> even if few percent decrease in memory consumption doesn't look interesting on
>> single-client level, few percents in large scale deployments may mean hundreds
>> of gigabytes of memory that could be put to better use.
>>
>> (2), and particularly the requirement for API changes, along with need to put
>> user data into specific RADOS structure doesn't look good because
>> - that requires way more changes to user code, and I don't expect any of
>> current user to be interested in that (unless we force it with API change)
>> - most of devs are simply lazy and instead of using "rados_buffer_create" to
>> put their data directly into them, they'll keep using their own buffers and
>> *then* copy them into RADOS buffers, just like we're doing it right now (even
>> if behind the scenes).
>
> Yeah, I think you're right.  The bad news is just that (1) is hard.  It's
> going to require a few things in order to address the problem of sending
> multiple MOSDOp requests for the same operation (e.g., after peering).  I
> think it will mean
>
> - The Message superclass is going to require a new Mutex that is taken by
> the messenger implementation while accessing the encoded message payload
> buffer
>
> - It will also need a revoke_buffers() method (that takes the mutex) so
> that Objecter can yank references to buffers for any messages in flight
> that it no longer cares about
>
> - Objecter will need to keep a ref of the in-flight request of record.  If
> it needs to resend to a different OSD, it can use that ref to revoke the
> buffers.
>
> - Once the operation completes, Objecter can drop it's ref to that
> Message.
>
> Until we do that, any zero-copy changes like those in the PR will mostly
> work but trigger use-after-free is various cases where the PG mappings are
> changing.
>
> sage



-- 
Jason
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Any librados C API users out there?

[Gregory Farnum]

On Thu, Jan 12, 2017 at 5:54 AM, Jason Dillaman <jdillama-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> There is option (3) which is to have a new (or modified)
> "buffer::create_static" take an optional callback to invoke when the
> buffer::raw object is destructed. The raw pointer would be destructed
> when the last buffer::ptr / buffer::list containing it is destructed,
> so you know it's no longer being referenced.
>
> You could then have the new C API methods that wrap the C buffer in a
> bufferlist and set a new flag in the librados::AioCompletion to delay
> its completion until after it's both completed and the memory is
> released. When the buffer is freed, the callback would unblock the
> librados::AioCompltion completion callback.

I much prefer an approach like this: it's zero-copy; it's not a lot of
user overhead; but it requires them to explicitly pass memory off to
Ceph and keep it immutable until Ceph is done (at which point they are
told so explicitly). Even if we were very careful about not returning
to users until operations are done, just taking buffers into a
multi-threaded application without having explicit markers about
ownership is a recipe for misuse.
-Greg

Re: Any librados C API users out there?

[Sage Weil]

On Thu, 12 Jan 2017, Gregory Farnum wrote:
> On Thu, Jan 12, 2017 at 5:54 AM, Jason Dillaman <jdillama-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> > There is option (3) which is to have a new (or modified)
> > "buffer::create_static" take an optional callback to invoke when the
> > buffer::raw object is destructed. The raw pointer would be destructed
> > when the last buffer::ptr / buffer::list containing it is destructed,
> > so you know it's no longer being referenced.
> >
> > You could then have the new C API methods that wrap the C buffer in a
> > bufferlist and set a new flag in the librados::AioCompletion to delay
> > its completion until after it's both completed and the memory is
> > released. When the buffer is freed, the callback would unblock the
> > librados::AioCompltion completion callback.
> 
> I much prefer an approach like this: it's zero-copy; it's not a lot of
> user overhead; but it requires them to explicitly pass memory off to
> Ceph and keep it immutable until Ceph is done (at which point they are
> told so explicitly). 

Yeah, this is simpler.  I still feel like we should provide a way to 
revoke buffers, though, because otherwise it's possible for calls to block 
semi-indefinitey if, say, an old MOSDOp is quueed for another OSD and that 
OSD is not reading data off the socket but has not failed (e.g., due to 
it's rx throttling).

sage

> Even if we were very careful about not returning
> to users until operations are done, just taking buffers into a
> multi-threaded application without having explicit markers about
> ownership is a recipe for misuse.

Re: [ceph-users] Any librados C API users out there?

[Yehuda Sadeh-Weinraub]

On Thu, Jan 12, 2017 at 12:08 PM, Sage Weil <sweil@redhat.com> wrote:
> On Thu, 12 Jan 2017, Gregory Farnum wrote:
>> On Thu, Jan 12, 2017 at 5:54 AM, Jason Dillaman <jdillama@redhat.com> wrote:
>> > There is option (3) which is to have a new (or modified)
>> > "buffer::create_static" take an optional callback to invoke when the
>> > buffer::raw object is destructed. The raw pointer would be destructed
>> > when the last buffer::ptr / buffer::list containing it is destructed,
>> > so you know it's no longer being referenced.
>> >
>> > You could then have the new C API methods that wrap the C buffer in a
>> > bufferlist and set a new flag in the librados::AioCompletion to delay
>> > its completion until after it's both completed and the memory is
>> > released. When the buffer is freed, the callback would unblock the
>> > librados::AioCompltion completion callback.
>>
>> I much prefer an approach like this: it's zero-copy; it's not a lot of
>> user overhead; but it requires them to explicitly pass memory off to
>> Ceph and keep it immutable until Ceph is done (at which point they are
>> told so explicitly).
>
> Yeah, this is simpler.  I still feel like we should provide a way to
> revoke buffers, though, because otherwise it's possible for calls to block
> semi-indefinitey if, say, an old MOSDOp is quueed for another OSD and that
> OSD is not reading data off the socket but has not failed (e.g., due to
> it's rx throttling).
>

We need to provide some way to cancel requests (at least from the
client's aspect), that would guarantee that buffers are not going to
be used (and no completion callback is going to be called).

Yehuda

Re: Any librados C API users out there?

[Sage Weil]

On Thu, 12 Jan 2017, Yehuda Sadeh-Weinraub wrote:
> On Thu, Jan 12, 2017 at 12:08 PM, Sage Weil <sweil-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> > On Thu, 12 Jan 2017, Gregory Farnum wrote:
> >> On Thu, Jan 12, 2017 at 5:54 AM, Jason Dillaman <jdillama-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> >> > There is option (3) which is to have a new (or modified)
> >> > "buffer::create_static" take an optional callback to invoke when the
> >> > buffer::raw object is destructed. The raw pointer would be destructed
> >> > when the last buffer::ptr / buffer::list containing it is destructed,
> >> > so you know it's no longer being referenced.
> >> >
> >> > You could then have the new C API methods that wrap the C buffer in a
> >> > bufferlist and set a new flag in the librados::AioCompletion to delay
> >> > its completion until after it's both completed and the memory is
> >> > released. When the buffer is freed, the callback would unblock the
> >> > librados::AioCompltion completion callback.
> >>
> >> I much prefer an approach like this: it's zero-copy; it's not a lot of
> >> user overhead; but it requires them to explicitly pass memory off to
> >> Ceph and keep it immutable until Ceph is done (at which point they are
> >> told so explicitly).
> >
> > Yeah, this is simpler.  I still feel like we should provide a way to
> > revoke buffers, though, because otherwise it's possible for calls to block
> > semi-indefinitey if, say, an old MOSDOp is quueed for another OSD and that
> > OSD is not reading data off the socket but has not failed (e.g., due to
> > it's rx throttling).
> >
> 
> We need to provide some way to cancel requests (at least from the
> client's aspect), that would guarantee that buffers are not going to
> be used (and no completion callback is going to be called).

Yeah.  It's a bit more work but I think this is the best path.

sage

Re: Any librados C API users out there?

[Matt Benjamin]

Hi,

----- Original Message -----
> From: "Yehuda Sadeh-Weinraub" <ysadehwe@redhat.com>
> To: "Sage Weil" <sweil@redhat.com>
> Cc: "Gregory Farnum" <gfarnum@redhat.com>, "Jason Dillaman" <dillaman@redhat.com>, "Piotr Dałek"
> <piotr.dalek@corp.ovh.com>, "ceph-devel" <ceph-devel@vger.kernel.org>, "ceph-users" <ceph-users@lists.ceph.com>
> Sent: Thursday, January 12, 2017 3:22:06 PM
> Subject: Re: [ceph-users] Any librados C API users out there?
> 
> On Thu, Jan 12, 2017 at 12:08 PM, Sage Weil <sweil@redhat.com> wrote:
> > On Thu, 12 Jan 2017, Gregory Farnum wrote:
> >> On Thu, Jan 12, 2017 at 5:54 AM, Jason Dillaman <jdillama@redhat.com>
> >> wrote:
> >> > There is option (3) which is to have a new (or modified)
> >> > "buffer::create_static" take an optional callback to invoke when the
> >> > buffer::raw object is destructed. The raw pointer would be destructed
> >> > when the last buffer::ptr / buffer::list containing it is destructed,
> >> > so you know it's no longer being referenced.
> >> >
> >> > You could then have the new C API methods that wrap the C buffer in a
> >> > bufferlist and set a new flag in the librados::AioCompletion to delay
> >> > its completion until after it's both completed and the memory is
> >> > released. When the buffer is freed, the callback would unblock the
> >> > librados::AioCompltion completion callback.
> >>
> >> I much prefer an approach like this: it's zero-copy; it's not a lot of
> >> user overhead; but it requires them to explicitly pass memory off to
> >> Ceph and keep it immutable until Ceph is done (at which point they are
> >> told so explicitly).
> >
> > Yeah, this is simpler.  I still feel like we should provide a way to
> > revoke buffers, though, because otherwise it's possible for calls to block
> > semi-indefinitey if, say, an old MOSDOp is quueed for another OSD and that
> > OSD is not reading data off the socket but has not failed (e.g., due to
> > it's rx throttling).
> >
> 
> We need to provide some way to cancel requests (at least from the
> client's aspect), that would guarantee that buffers are not going to
> be used (and no completion callback is going to be called).

is the client/consumer cancellation async wrt completion?  a cancellation in that case could ensure that, if it succeeds, those guarantees are met, or else fails (because the callback and completion have raced cancellation)?

Matt

> 
> Yehuda
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Any librados C API users out there?

[Yehuda Sadeh-Weinraub]

On Thu, Jan 12, 2017 at 2:14 PM, Matt Benjamin <mbenjamin@redhat.com> wrote:
> Hi,
>
> ----- Original Message -----
>> From: "Yehuda Sadeh-Weinraub" <ysadehwe@redhat.com>
>> To: "Sage Weil" <sweil@redhat.com>
>> Cc: "Gregory Farnum" <gfarnum@redhat.com>, "Jason Dillaman" <dillaman@redhat.com>, "Piotr Dałek"
>> <piotr.dalek@corp.ovh.com>, "ceph-devel" <ceph-devel@vger.kernel.org>, "ceph-users" <ceph-users@lists.ceph.com>
>> Sent: Thursday, January 12, 2017 3:22:06 PM
>> Subject: Re: [ceph-users] Any librados C API users out there?
>>
>> On Thu, Jan 12, 2017 at 12:08 PM, Sage Weil <sweil@redhat.com> wrote:
>> > On Thu, 12 Jan 2017, Gregory Farnum wrote:
>> >> On Thu, Jan 12, 2017 at 5:54 AM, Jason Dillaman <jdillama@redhat.com>
>> >> wrote:
>> >> > There is option (3) which is to have a new (or modified)
>> >> > "buffer::create_static" take an optional callback to invoke when the
>> >> > buffer::raw object is destructed. The raw pointer would be destructed
>> >> > when the last buffer::ptr / buffer::list containing it is destructed,
>> >> > so you know it's no longer being referenced.
>> >> >
>> >> > You could then have the new C API methods that wrap the C buffer in a
>> >> > bufferlist and set a new flag in the librados::AioCompletion to delay
>> >> > its completion until after it's both completed and the memory is
>> >> > released. When the buffer is freed, the callback would unblock the
>> >> > librados::AioCompltion completion callback.
>> >>
>> >> I much prefer an approach like this: it's zero-copy; it's not a lot of
>> >> user overhead; but it requires them to explicitly pass memory off to
>> >> Ceph and keep it immutable until Ceph is done (at which point they are
>> >> told so explicitly).
>> >
>> > Yeah, this is simpler.  I still feel like we should provide a way to
>> > revoke buffers, though, because otherwise it's possible for calls to block
>> > semi-indefinitey if, say, an old MOSDOp is quueed for another OSD and that
>> > OSD is not reading data off the socket but has not failed (e.g., due to
>> > it's rx throttling).
>> >
>>
>> We need to provide some way to cancel requests (at least from the
>> client's aspect), that would guarantee that buffers are not going to
>> be used (and no completion callback is going to be called).
>
> is the client/consumer cancellation async wrt completion?  a cancellation in that case could ensure that, if it succeeds, those guarantees are met, or else fails (because the callback and completion have raced cancellation)?
>

A cancellation succeeding would mean that no completion callback is
triggering, nor will it trigger. This means that in order for the
cancellation to complete, any ongoing completions will need to finish.
So either it waits on the completion to finish or abort cancellation.

Yehuda