From mboxrd@z Thu Jan 1 00:00:00 1970 From: Rutger ter Borg Subject: Re: Rados and user-provided buffers Date: Thu, 19 Sep 2013 11:28:55 +0200 Message-ID: References: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Return-path: Received: from plane.gmane.org ([80.91.229.3]:57821 "EHLO plane.gmane.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751711Ab3ISJ3H (ORCPT ); Thu, 19 Sep 2013 05:29:07 -0400 Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1VMaXo-0005Sm-Tw for ceph-devel@vger.kernel.org; Thu, 19 Sep 2013 11:29:04 +0200 Received: from gl-95017.prolocation.net ([62.204.95.17]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 19 Sep 2013 11:29:04 +0200 Received: from rutger by gl-95017.prolocation.net with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 19 Sep 2013 11:29:04 +0200 In-Reply-To: Sender: ceph-devel-owner@vger.kernel.org List-ID: To: ceph-devel@vger.kernel.org On 2013-09-18 22:52, Sage Weil wrote: > Hmm, looking at the code, I'm surprised that this isn't working. The C > aio_read call is just doing > > bufferlist bl; > bufferptr bp = buffer::create_static(len, buf); > bl.push_back(bp); > > ret = ctx->read(oid, bl, len, off); > if (ret >= 0) { > if (bl.length() > len) > return -ERANGE; > if (bl.c_str() != buf) > bl.copy(0, bl.length(), buf); Hey Sage, thanks for the hints. You're citing the synchronous version of rados_read, not rados_aio_read. The difference between rados_read and rados_aio_read (the C-versions) is that rados_read uses a bufferlist, and rados_aio_read uses an overload with a char* buf. Both are delegated to IoCtxImpl. IoCtxImpl has two overloads for aio_read, one accepting a bufferlist and a char*, but only one overload for read, accepting a bufferlist only. IoCtxImpl's overloads for aio_read are almost identical, the difference is that the buflist overload sets a bufferlist on AioCompletionImpl* c, c->pbl = pbl; and the char* buffer-overload sets a buffer c->buf = buf; AioCompletionImpl contains multiple data members: a bufferlist (bl), a pointer to a bufferlist (pbl), and a pointer to a character array buf. The following calls (in IoCtxImpl's aio_read overloads) to the objecter are identical: objecter->read(oid, oloc, off, len, snapid, &c->bl, 0, onack, &c->objver); Looking at the Objecter read and deeper in the call chain, it seems that information about data member pbl in AioCompletionImpl* is lost. The objecter only knows about a Context, not about AioCompletionImpl. The user-provided buffer is not passed on. My preliminary conclusion is that my problem is caused by information lost in IoCtxImpl's aio_read overloads. Maybe it can be solved by modifying IoCtxImpl.cc: * removing line 615. Not sure why the AioCompletionImpl needs to know anything about buffers? * replacing '&c->bl' with 'pbl' on line 619 of IoCtxImpl.cc, making the call to the objecter objecter->read(oid, oloc, off, len, snapid, pbl, 0, onack, &c->objver); In that way, the bufferlist is passed through, and not thrown away. Thanks, Rutger