Re: Rados and user-provided buffers

[Rutger ter Borg <rutger@terborg.net>]

On 2013-09-18 22:52, Sage Weil wrote:

> Hmm, looking at the code, I'm surprised that this isn't working.  The C
> aio_read call is just doing
>
>    bufferlist bl;
>    bufferptr bp = buffer::create_static(len, buf);
>    bl.push_back(bp);
>
>    ret = ctx->read(oid, bl, len, off);
>    if (ret >= 0) {
>      if (bl.length() > len)
>        return -ERANGE;
>      if (bl.c_str() != buf)
>        bl.copy(0, bl.length(), buf);


Hey Sage,

thanks for the hints. You're citing the synchronous version of 
rados_read, not rados_aio_read. The difference between rados_read and 
rados_aio_read (the C-versions) is that rados_read uses a bufferlist, 
and rados_aio_read uses an overload with a char* buf. Both are delegated 
to IoCtxImpl. IoCtxImpl has two overloads for aio_read, one accepting a 
bufferlist and a char*, but only one overload for read, accepting a 
bufferlist only.

IoCtxImpl's overloads for aio_read are almost identical, the difference 
is that the buflist overload sets a bufferlist on AioCompletionImpl* c,

c->pbl = pbl;

and the char* buffer-overload sets a buffer

c->buf = buf;

AioCompletionImpl contains multiple data members: a bufferlist (bl), a 
pointer to a bufferlist (pbl), and a pointer to a character array buf.
The following calls (in IoCtxImpl's aio_read overloads) to the objecter 
are identical:

objecter->read(oid, oloc,
          off, len, snapid, &c->bl, 0,
          onack, &c->objver);

Looking at the Objecter read and deeper in the call chain, it seems that 
information about data member pbl in AioCompletionImpl* is lost. The 
objecter only knows about a Context, not about AioCompletionImpl. The 
user-provided buffer is not passed on.

My preliminary conclusion is that my problem is caused by information 
lost in IoCtxImpl's aio_read overloads. Maybe it can be solved by 
modifying IoCtxImpl.cc:

  * removing line 615. Not sure why the AioCompletionImpl needs to know 
anything about buffers?
  * replacing '&c->bl' with 'pbl' on line 619 of IoCtxImpl.cc, making 
the call to the objecter

objecter->read(oid, oloc,
          off, len, snapid, pbl, 0,
          onack, &c->objver);

In that way, the bufferlist is passed through, and not thrown away.

Thanks,

Rutger