[Lustre-devel] client side reply handling

[Lustre-devel] client side reply handling

[Peter Braam]

Hi,

I wonder if LNET doesn't have an atomic operation that unlinks the packet
from the delivery process upon receiving a packet.   Iirc it does.  Why
wouldn't one use that?

Peter
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20091209/27c0c4d8/attachment.htm>

[Lustre-devel] client side reply handling

[Yuriy Umanets]

2009/12/9 Peter Braam <peter.braam@clusterstor.com>

> Hi,
>
> hi Peter,


> I wonder if LNET doesn't have an atomic operation that unlinks the packet
> from the delivery process upon receiving a packet.   Iirc it does.  Why
> wouldn't one use that?
>
That is probably because it still may be in use by upper layers (ptlrpc,
etc), i.e. it is up to upper layer to decide should its buffers be still
linked for access or not.

Thanks.

>
> Peter
>
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel
>
>


-- 
umka
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20091215/fd7d522c/attachment.htm>

[Lustre-devel] client side reply handling

[Peter Braam]

No - buffers can and probably should (because, for example, re-delivery may
leaver buffer state undefined) be unlinked from the match list before
passing the buffer up to any other layer.  Portals/LNET certainly can do
this.

The question is why isn't this done, Eric, Isaac, Andreas?

Peter

On Tue, Dec 15, 2009 at 9:28 AM, Yuriy Umanets <
yuriy.umanets@clusterstor.com> wrote:

>
>
> 2009/12/9 Peter Braam <peter.braam@clusterstor.com>
>
>> Hi,
>>
>> hi Peter,
>
>
>> I wonder if LNET doesn't have an atomic operation that unlinks the packet
>> from the delivery process upon receiving a packet.   Iirc it does.  Why
>> wouldn't one use that?
>>
> That is probably because it still may be in use by upper layers (ptlrpc,
> etc), i.e. it is up to upper layer to decide should its buffers be still
> linked for access or not.
>
> Thanks.
>
>>
>> Peter
>>
>> _______________________________________________
>> Lustre-devel mailing list
>> Lustre-devel at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-devel
>>
>>
>
>
> --
> umka
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20091215/21a6c9bb/attachment.htm>

[Lustre-devel] client side reply handling

[Isaac Huang]

On Tue, Dec 15, 2009 at 07:01:33PM -0700, Peter Braam wrote:
>    No - buffers can and probably should (because, for example, re-delivery
>    may leaver buffer state undefined) be unlinked from the match list
>    before passing the buffer up to any other layer.  Portals/LNET
>    certainly can do this.

Hi Peter, LNet does have a MD option to automatically unlink the
buffer once it's been exhausted. Duplicated delivery shouldn't cause any
problem because MD offset is increased locally (i.e. a dup would go
into a different offset), unless the MD has enabled peers to manage
offset remotely, which is now true for PTLRPC reply buffers (because
now early replies and real reply share a same reply buffer).

Isaac

[Lustre-devel] client side reply handling

[Isaac Huang]

On Tue, Dec 15, 2009 at 09:48:18PM -0500, Isaac Huang wrote:
> On Tue, Dec 15, 2009 at 07:01:33PM -0700, Peter Braam wrote:
> >    No - buffers can and probably should (because, for example, re-delivery
> >    may leaver buffer state undefined) be unlinked from the match list
> >    before passing the buffer up to any other layer.  Portals/LNET
> >    certainly can do this.
> 
> Hi Peter, LNet does have a MD option to automatically unlink the
> buffer once it's been exhausted. Duplicated delivery shouldn't cause any
> problem because MD offset is increased locally (i.e. a dup would go
> into a different offset), unless the MD has enabled peers to manage
> offset remotely, which is now true for PTLRPC reply buffers (because
> now early replies and real reply share a same reply buffer).

There was a recent discussion on this topic, under a similar subject:
http://lists.lustre.org/pipermail/lustre-devel/2009-December/003231.html

Isaac

[Lustre-devel] client side reply handling

[Liang Zhen]

Peter,

It depends on how buffers are attached to portal, ptlrpc has three cases 
of this:
1. If buffer is for incoming request (not for RDMA, so it's not unique 
match)
    the MD is constructed with lnet_md_t::threshold = LNET_MD_THRESH_INF,
    so it will be unlinked only for two cases:
    a) user call LNetMDUnlink explicitly
    b) the buffer is exhausted (lnet_md_exhausted() return 1 in LNet)
    Once LNet get a request for that buffer, it will change offset of 
the buffer so
it wouldn't be overwritten by the next match, it's possible that upper 
layer is handing
the request, and the buffer is still on match list

2. If buffer is for bulk (it's unique)
    the MD is constructed with lnet_md_t::threshold = 1, and
    LNetMDAttach(...LNET_UNLINK...)
    so LNet will auto-unlink the buffer from match list once it got a 
match, I think
this is the case you want

3. If the buffer is for reply (it's unique as well)
    the MD is constructed with lnet_md_t::threshold = LNET_MD_THRESH_INF,
so it's same as 1), the buffer wouldn't be unlinked unless user call  
LNetMDUnlink
or LNetMEUnlink, so safety of reply buffer can be guaranteed only if user is
sure the buffer is unlinked.

Regards
Liang


Peter Braam wrote:
> No - buffers can and probably should (because, for example, 
> re-delivery may leaver buffer state undefined) be unlinked from the 
> match list before passing the buffer up to any other layer. 
>  Portals/LNET certainly can do this.  
>
> The question is why isn't this done, Eric, Isaac, Andreas?
>
> Peter
>
> On Tue, Dec 15, 2009 at 9:28 AM, Yuriy Umanets 
> <yuriy.umanets at clusterstor.com <mailto:yuriy.umanets@clusterstor.com>> 
> wrote:
>
>
>
>     2009/12/9 Peter Braam <peter.braam@clusterstor.com
>     <mailto:peter.braam@clusterstor.com>>
>
>         Hi, 
>
>     hi Peter,
>      
>
>         I wonder if LNET doesn't have an atomic operation that unlinks
>         the packet from the delivery process upon receiving a packet.
>           Iirc it does.  Why wouldn't one use that?
>
>     That is probably because it still may be in use by upper layers
>     (ptlrpc, etc), i.e. it is up to upper layer to decide should its
>     buffers be still linked for access or not.
>
>     Thanks.
>
>
>         Peter
>
>         _______________________________________________
>         Lustre-devel mailing list
>         Lustre-devel at lists.lustre.org
>         <mailto:Lustre-devel@lists.lustre.org>
>         http://lists.lustre.org/mailman/listinfo/lustre-devel
>
>
>
>
>     -- 
>     umka
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel
>