CQEs in unreliable connection

CQEs in unreliable connection

[Anuj Kalia]

Hi.

I had 2 questions related to RDMA operations (reads and writes) over
unreliable connections:

1. Are WQEs issued over an unreliable connection supposed to generate
CQEs? The Infiniband spec says that such WQEs may not generate an
'ACK'. I'm confused if ACK refers to CQE.

2. If the status of the work completion (fetched using ibv_poll_cq())
is IBV_WC_SUCCESS, does it mean that the operation was successful at
the remote host (even though we used an unreliable connection)?

Thanks for your help!

--Anuj
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CQEs in unreliable connection

[Dotan Barak]

Hi Anuj.

On 28/11/2013 14:24, Anuj Kalia wrote:
> Hi.
>
> I had 2 questions related to RDMA operations (reads and writes) over
> unreliable connections:
>
> 1. Are WQEs issued over an unreliable connection supposed to generate
> CQEs? The Infiniband spec says that such WQEs may not generate an
> 'ACK'. I'm confused if ACK refers to CQE.
ACK is an acknowledgment that was sent over the wire that the message 
was received by the remote side.
CQE is a mechanism to indicate that a Work Request isn't outstanding 
anymore.

In RC QP, a CQE with successful status of a Send Request means that an 
ACK was received by the sender.
However, In UC QP, a CQE with successful status of a Send Request means 
that the message was sent;
it isn't indicate that the message was received by the remote side ...
(You will get a successful CQE even if the remote side didn't receive 
this message)

>
> 2. If the status of the work completion (fetched using ibv_poll_cq())
> is IBV_WC_SUCCESS, does it mean that the operation was successful at
> the remote host (even though we used an unreliable connection)?
I answered your answer above ...

If you want some more information, you are welcome to read it in:
http://www.rdmamojo.com/2013/06/01/which-queue-pair-type-to-use/


Thanks
Dotan
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CQEs in unreliable connection

[Anuj Kalia]

Dotan,

Thanks a lot for your reply. All my CQEs had status IBV_WC_SUCCESS
even when the responder was heavily loaded and I was wondering why
nothing was getting dropped...

I read your blog post and it was really helpful for understanding the
queue pair types. I had one small question though.

You say that: "There isn’t any guaranteed that the messages will be
received by the other side: corrupted or out of sequence packets are
silently dropped. If a packet is being dropped, the whole message that
it belongs to will be dropped. In this case, the responder won’t stop,
but continues to receive incoming packets. There isn’t any guarantee
about the packet ordering."

Here's my question: if out of sequence packets are dropped, isn't
there an ordering guarantee? I assume that dropping of out of sequence
packets means that if the responder receives packets {2, 1, 3} (in
this order), then it drops packet number {1} and keeps {2, 3}.

Thanks again!
--Anuj

On Fri, Nov 29, 2013 at 1:19 AM, Dotan Barak <dotanba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Hi Anuj.
>
>
> On 28/11/2013 14:24, Anuj Kalia wrote:
>>
>> Hi.
>>
>> I had 2 questions related to RDMA operations (reads and writes) over
>> unreliable connections:
>>
>> 1. Are WQEs issued over an unreliable connection supposed to generate
>> CQEs? The Infiniband spec says that such WQEs may not generate an
>> 'ACK'. I'm confused if ACK refers to CQE.
>
> ACK is an acknowledgment that was sent over the wire that the message was
> received by the remote side.
> CQE is a mechanism to indicate that a Work Request isn't outstanding
> anymore.
>
> In RC QP, a CQE with successful status of a Send Request means that an ACK
> was received by the sender.
> However, In UC QP, a CQE with successful status of a Send Request means that
> the message was sent;
> it isn't indicate that the message was received by the remote side ...
> (You will get a successful CQE even if the remote side didn't receive this
> message)
>
>
>>
>> 2. If the status of the work completion (fetched using ibv_poll_cq())
>> is IBV_WC_SUCCESS, does it mean that the operation was successful at
>> the remote host (even though we used an unreliable connection)?
>
> I answered your answer above ...
>
> If you want some more information, you are welcome to read it in:
> http://www.rdmamojo.com/2013/06/01/which-queue-pair-type-to-use/
>
>
> Thanks
> Dotan
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CQEs in unreliable connection

[Anuj Kalia]

Hari,

Thanks for your reply. Yes, I'm talking about the Unreliable
Connection transport type

I'm concerned with small messages (128 byte RDMA writes or reads), so
I assume that my messages fit inside one packet.

I'm confused with this: "Receiver detects and drops out of order
packets.". For example, if the packet sequence at the receiver is {P1,
P2, P4, P3, P5}, where the numbers denote PSNs, then what are the
packets that get dropped?

Does P4 get dropped because it is after P2 (and not P3)? Or is P4
preserved (because it's still in the increasing order), and P3 dropped
later?

I guess I'm confused between packets and messages..

--Anuj


On Fri, Nov 29, 2013 at 8:30 AM, Hari Subramoni <subramoni.1-ZbGKxL/pcrQ@public.gmane.org> wrote:
> Hi Anuj,
>
> When you say "unreliable connection", I am assuming that you are still
> talking about the "Unreliable Connection" transport type. Are you?
>
> If so, here are some characteristics of an unreliable connection as stated
> by the IB spec
>
>  - No delivery guarantees to the sender. Receiver may drop messages.
>  - No ordering guarantee. Sender / Receiver cannot rely on msgs arriving in
> order.
>  - Receiver detects and drops out of order packets.
>  - Dropped packets may cause the message to be dropped.
>  - After dropping a packet, receiver resumes with the first packet of a new
> message.
>
> Now, to answer you - There is no ordering guarantee and if one packet in a
> message is lost, the entire message "could" be lost.
>
> Thanks,
> Hari.
>
>
> On Thu, Nov 28, 2013 at 8:37 PM, Anuj Kalia <anujkaliaiitd-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>
>> Dotan,
>>
>> Thanks a lot for your reply. All my CQEs had status IBV_WC_SUCCESS
>> even when the responder was heavily loaded and I was wondering why
>> nothing was getting dropped...
>>
>> I read your blog post and it was really helpful for understanding the
>> queue pair types. I had one small question though.
>>
>> You say that: "There isn’t any guaranteed that the messages will be
>> received by the other side: corrupted or out of sequence packets are
>> silently dropped. If a packet is being dropped, the whole message that
>> it belongs to will be dropped. In this case, the responder won’t stop,
>> but continues to receive incoming packets. There isn’t any guarantee
>> about the packet ordering."
>>
>> Here's my question: if out of sequence packets are dropped, isn't
>> there an ordering guarantee? I assume that dropping of out of sequence
>> packets means that if the responder receives packets {2, 1, 3} (in
>> this order), then it drops packet number {1} and keeps {2, 3}.
>>
>> Thanks again!
>> --Anuj
>>
>> On Fri, Nov 29, 2013 at 1:19 AM, Dotan Barak <dotanba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> > Hi Anuj.
>> >
>> >
>> > On 28/11/2013 14:24, Anuj Kalia wrote:
>> >>
>> >> Hi.
>> >>
>> >> I had 2 questions related to RDMA operations (reads and writes) over
>> >> unreliable connections:
>> >>
>> >> 1. Are WQEs issued over an unreliable connection supposed to generate
>> >> CQEs? The Infiniband spec says that such WQEs may not generate an
>> >> 'ACK'. I'm confused if ACK refers to CQE.
>> >
>> > ACK is an acknowledgment that was sent over the wire that the message
>> > was
>> > received by the remote side.
>> > CQE is a mechanism to indicate that a Work Request isn't outstanding
>> > anymore.
>> >
>> > In RC QP, a CQE with successful status of a Send Request means that an
>> > ACK
>> > was received by the sender.
>> > However, In UC QP, a CQE with successful status of a Send Request means
>> > that
>> > the message was sent;
>> > it isn't indicate that the message was received by the remote side ...
>> > (You will get a successful CQE even if the remote side didn't receive
>> > this
>> > message)
>> >
>> >
>> >>
>> >> 2. If the status of the work completion (fetched using ibv_poll_cq())
>> >> is IBV_WC_SUCCESS, does it mean that the operation was successful at
>> >> the remote host (even though we used an unreliable connection)?
>> >
>> > I answered your answer above ...
>> >
>> > If you want some more information, you are welcome to read it in:
>> > http://www.rdmamojo.com/2013/06/01/which-queue-pair-type-to-use/
>> >
>> >
>> > Thanks
>> > Dotan
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CQEs in unreliable connection

[Dotan Barak]

On 29/11/2013 05:11, Anuj Kalia wrote:
> Hari,
>
> Thanks for your reply. Yes, I'm talking about the Unreliable
> Connection transport type
>
> I'm concerned with small messages (128 byte RDMA writes or reads), so
> I assume that my messages fit inside one packet.
>
> I'm confused with this: "Receiver detects and drops out of order
> packets.". For example, if the packet sequence at the receiver is {P1,
> P2, P4, P3, P5}, where the numbers denote PSNs, then what are the
> packets that get dropped?
>
> Does P4 get dropped because it is after P2 (and not P3)? Or is P4
> preserved (because it's still in the increasing order), and P3 dropped
> later?
>
> I guess I'm confused between packets and messages..

Here are some Quotes form the IB spec:

C9-186: For an HCA responder using Unreliable Connection service, if a
multi-packet message is in progress at the time that an out of order packet
is detected, the current message shall be silently dropped. The responder
then waits for the first packet of a new message. It is possible that the
present packet (the out of order packet) is the first packet of a new 
message.
If so, it shall be treated as a new message.

A “new message” is denoted by an inbound request packet with an Op-
Code in the BTH of “first” or “only”.

“Current message” means all the packets received since the most recently
received “first” or “only” OpCode, excluding the present packet.


 From this, one can understand that the message will be dropped because 
P4 was detected,
while P3 was received.

But just remember, that Out Of Sequence (OOS) shouldn't happen in IB, so 
usually if an OOS packet
was received, this means that a packet was dropped in the way...


I hope that I helped you with this.

Thanks
Dotan

>
> --Anuj
>
>
> On Fri, Nov 29, 2013 at 8:30 AM, Hari Subramoni <subramoni.1-ZbGKxL/pcrQ@public.gmane.org> wrote:
>> Hi Anuj,
>>
>> When you say "unreliable connection", I am assuming that you are still
>> talking about the "Unreliable Connection" transport type. Are you?
>>
>> If so, here are some characteristics of an unreliable connection as stated
>> by the IB spec
>>
>>   - No delivery guarantees to the sender. Receiver may drop messages.
>>   - No ordering guarantee. Sender / Receiver cannot rely on msgs arriving in
>> order.
>>   - Receiver detects and drops out of order packets.
>>   - Dropped packets may cause the message to be dropped.
>>   - After dropping a packet, receiver resumes with the first packet of a new
>> message.
>>
>> Now, to answer you - There is no ordering guarantee and if one packet in a
>> message is lost, the entire message "could" be lost.
>>
>> Thanks,
>> Hari.
>>
>>
>> On Thu, Nov 28, 2013 at 8:37 PM, Anuj Kalia <anujkaliaiitd-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>> Dotan,
>>>
>>> Thanks a lot for your reply. All my CQEs had status IBV_WC_SUCCESS
>>> even when the responder was heavily loaded and I was wondering why
>>> nothing was getting dropped...
>>>
>>> I read your blog post and it was really helpful for understanding the
>>> queue pair types. I had one small question though.
>>>
>>> You say that: "There isn’t any guaranteed that the messages will be
>>> received by the other side: corrupted or out of sequence packets are
>>> silently dropped. If a packet is being dropped, the whole message that
>>> it belongs to will be dropped. In this case, the responder won’t stop,
>>> but continues to receive incoming packets. There isn’t any guarantee
>>> about the packet ordering."
>>>
>>> Here's my question: if out of sequence packets are dropped, isn't
>>> there an ordering guarantee? I assume that dropping of out of sequence
>>> packets means that if the responder receives packets {2, 1, 3} (in
>>> this order), then it drops packet number {1} and keeps {2, 3}.
>>>
>>> Thanks again!
>>> --Anuj
>>>
>>> On Fri, Nov 29, 2013 at 1:19 AM, Dotan Barak <dotanba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>> Hi Anuj.
>>>>
>>>>
>>>> On 28/11/2013 14:24, Anuj Kalia wrote:
>>>>> Hi.
>>>>>
>>>>> I had 2 questions related to RDMA operations (reads and writes) over
>>>>> unreliable connections:
>>>>>
>>>>> 1. Are WQEs issued over an unreliable connection supposed to generate
>>>>> CQEs? The Infiniband spec says that such WQEs may not generate an
>>>>> 'ACK'. I'm confused if ACK refers to CQE.
>>>> ACK is an acknowledgment that was sent over the wire that the message
>>>> was
>>>> received by the remote side.
>>>> CQE is a mechanism to indicate that a Work Request isn't outstanding
>>>> anymore.
>>>>
>>>> In RC QP, a CQE with successful status of a Send Request means that an
>>>> ACK
>>>> was received by the sender.
>>>> However, In UC QP, a CQE with successful status of a Send Request means
>>>> that
>>>> the message was sent;
>>>> it isn't indicate that the message was received by the remote side ...
>>>> (You will get a successful CQE even if the remote side didn't receive
>>>> this
>>>> message)
>>>>
>>>>
>>>>> 2. If the status of the work completion (fetched using ibv_poll_cq())
>>>>> is IBV_WC_SUCCESS, does it mean that the operation was successful at
>>>>> the remote host (even though we used an unreliable connection)?
>>>> I answered your answer above ...
>>>>
>>>> If you want some more information, you are welcome to read it in:
>>>> http://www.rdmamojo.com/2013/06/01/which-queue-pair-type-to-use/
>>>>
>>>>
>>>> Thanks
>>>> Dotan
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CQEs in unreliable connection

[Anuj Kalia]

Dotan,

In the example that I asked about, the 5 packets do not form one
message - they are each individual messages. I assume this in the
following.

I now understand that packet reordering cannot happen in IB - thanks!
Only packet loss can happen. From this, I can conclude the following
(please correct me if I'm wrong somewhere):

"If the requester sends a sequence of single packet messages: {M_1,
M_2, ..., M_n}, then the sequence of packets at the responder will be
{M_i1, M_i2, ..., M_ik}, where i1 < i2 < ... < ik. Which is just
saying that messages can be lost, but not reordered."

Is there a case where messages can be reordered in UC transport? i.e.
if the requester issues an RDMA write to buf[0] followed by an RDMA
write to buf[1], is it possible that at the responder the write to
buf[1] is done before the write to buf[0]? I don't think that this is
possible.

Thanks again. Your comments and blog have been really useful.

--Anuj.

On Fri, Nov 29, 2013 at 12:58 AM, Dotan Barak <dotanba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On 29/11/2013 05:11, Anuj Kalia wrote:
>>
>> Hari,
>>
>> Thanks for your reply. Yes, I'm talking about the Unreliable
>> Connection transport type
>>
>> I'm concerned with small messages (128 byte RDMA writes or reads), so
>> I assume that my messages fit inside one packet.
>>
>> I'm confused with this: "Receiver detects and drops out of order
>> packets.". For example, if the packet sequence at the receiver is {P1,
>> P2, P4, P3, P5}, where the numbers denote PSNs, then what are the
>> packets that get dropped?
>>
>> Does P4 get dropped because it is after P2 (and not P3)? Or is P4
>> preserved (because it's still in the increasing order), and P3 dropped
>> later?
>>
>> I guess I'm confused between packets and messages..
>
>
> Here are some Quotes form the IB spec:
>
> C9-186: For an HCA responder using Unreliable Connection service, if a
> multi-packet message is in progress at the time that an out of order packet
> is detected, the current message shall be silently dropped. The responder
> then waits for the first packet of a new message. It is possible that the
> present packet (the out of order packet) is the first packet of a new
> message.
> If so, it shall be treated as a new message.
>
> A “new message” is denoted by an inbound request packet with an Op-
> Code in the BTH of “first” or “only”.
>
> “Current message” means all the packets received since the most recently
> received “first” or “only” OpCode, excluding the present packet.
>
>
> From this, one can understand that the message will be dropped because P4
> was detected,
> while P3 was received.
>
> But just remember, that Out Of Sequence (OOS) shouldn't happen in IB, so
> usually if an OOS packet
> was received, this means that a packet was dropped in the way...
>
>
> I hope that I helped you with this.
>
> Thanks
> Dotan
>
>
>>
>> --Anuj
>>
>>
>> On Fri, Nov 29, 2013 at 8:30 AM, Hari Subramoni <subramoni.1-ZbGKxL/pcrQ@public.gmane.org>
>> wrote:
>>>
>>> Hi Anuj,
>>>
>>> When you say "unreliable connection", I am assuming that you are still
>>> talking about the "Unreliable Connection" transport type. Are you?
>>>
>>> If so, here are some characteristics of an unreliable connection as
>>> stated
>>> by the IB spec
>>>
>>>   - No delivery guarantees to the sender. Receiver may drop messages.
>>>   - No ordering guarantee. Sender / Receiver cannot rely on msgs arriving
>>> in
>>> order.
>>>   - Receiver detects and drops out of order packets.
>>>   - Dropped packets may cause the message to be dropped.
>>>   - After dropping a packet, receiver resumes with the first packet of a
>>> new
>>> message.
>>>
>>> Now, to answer you - There is no ordering guarantee and if one packet in
>>> a
>>> message is lost, the entire message "could" be lost.
>>>
>>> Thanks,
>>> Hari.
>>>
>>>
>>> On Thu, Nov 28, 2013 at 8:37 PM, Anuj Kalia <anujkaliaiitd-Re5JQEeQqe8@public.gmane.orgm>
>>> wrote:
>>>>
>>>> Dotan,
>>>>
>>>> Thanks a lot for your reply. All my CQEs had status IBV_WC_SUCCESS
>>>> even when the responder was heavily loaded and I was wondering why
>>>> nothing was getting dropped...
>>>>
>>>> I read your blog post and it was really helpful for understanding the
>>>> queue pair types. I had one small question though.
>>>>
>>>> You say that: "There isn’t any guaranteed that the messages will be
>>>> received by the other side: corrupted or out of sequence packets are
>>>> silently dropped. If a packet is being dropped, the whole message that
>>>> it belongs to will be dropped. In this case, the responder won’t stop,
>>>> but continues to receive incoming packets. There isn’t any guarantee
>>>> about the packet ordering."
>>>>
>>>> Here's my question: if out of sequence packets are dropped, isn't
>>>> there an ordering guarantee? I assume that dropping of out of sequence
>>>> packets means that if the responder receives packets {2, 1, 3} (in
>>>> this order), then it drops packet number {1} and keeps {2, 3}.
>>>>
>>>> Thanks again!
>>>> --Anuj
>>>>
>>>> On Fri, Nov 29, 2013 at 1:19 AM, Dotan Barak <dotanba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>>>
>>>>> Hi Anuj.
>>>>>
>>>>>
>>>>> On 28/11/2013 14:24, Anuj Kalia wrote:
>>>>>>
>>>>>> Hi.
>>>>>>
>>>>>> I had 2 questions related to RDMA operations (reads and writes) over
>>>>>> unreliable connections:
>>>>>>
>>>>>> 1. Are WQEs issued over an unreliable connection supposed to generate
>>>>>> CQEs? The Infiniband spec says that such WQEs may not generate an
>>>>>> 'ACK'. I'm confused if ACK refers to CQE.
>>>>>
>>>>> ACK is an acknowledgment that was sent over the wire that the message
>>>>> was
>>>>> received by the remote side.
>>>>> CQE is a mechanism to indicate that a Work Request isn't outstanding
>>>>> anymore.
>>>>>
>>>>> In RC QP, a CQE with successful status of a Send Request means that an
>>>>> ACK
>>>>> was received by the sender.
>>>>> However, In UC QP, a CQE with successful status of a Send Request means
>>>>> that
>>>>> the message was sent;
>>>>> it isn't indicate that the message was received by the remote side ...
>>>>> (You will get a successful CQE even if the remote side didn't receive
>>>>> this
>>>>> message)
>>>>>
>>>>>
>>>>>> 2. If the status of the work completion (fetched using ibv_poll_cq())
>>>>>> is IBV_WC_SUCCESS, does it mean that the operation was successful at
>>>>>> the remote host (even though we used an unreliable connection)?
>>>>>
>>>>> I answered your answer above ...
>>>>>
>>>>> If you want some more information, you are welcome to read it in:
>>>>> http://www.rdmamojo.com/2013/06/01/which-queue-pair-type-to-use/
>>>>>
>>>>>
>>>>> Thanks
>>>>> Dotan
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CQEs in unreliable connection

[Dotan Barak]

On 29/11/2013 16:53, Anuj Kalia wrote:
> Dotan,
>
> In the example that I asked about, the 5 packets do not form one
> message - they are each individual messages. I assume this in the
> following.
>
> I now understand that packet reordering cannot happen in IB - thanks!
> Only packet loss can happen. From this, I can conclude the following
> (please correct me if I'm wrong somewhere):
>
> "If the requester sends a sequence of single packet messages: {M_1,
> M_2, ..., M_n}, then the sequence of packets at the responder will be
> {M_i1, M_i2, ..., M_ik}, where i1 < i2 < ... < ik. Which is just
> saying that messages can be lost, but not reordered."
>
> Is there a case where messages can be reordered in UC transport? i.e.
> if the requester issues an RDMA write to buf[0] followed by an RDMA
> write to buf[1], is it possible that at the responder the write to
> buf[1] is done before the write to buf[0]? I don't think that this is
> possible.
I assume that we are talking about a single QP.

The messages will be sent by the order they are sent:
the message that access buf[0] will be sent (and received by the 
receiver) before the message that access buf[1].

If you are talking about the way that the memory data is being written/seen
in the remote side. I'm throwing a wild guess here, but *maybe* your 
process can see
the new content in first buf[1] before buf[0] depends on the memory 
policy, NUMA location, etc...
(this isn't a zone that I have a lot of knowledge in, but I think that 
this may have affect).

Thanks
Dotan


>
> Thanks again. Your comments and blog have been really useful.
>
> --Anuj.
>
>

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CQEs in unreliable connection

[Anuj Kalia]

Dotan,

Thanks for your reply again.

I just wanted to confirm about the order in which messages are sent
and received. I'm not worried about the memory policy.

--Anuj

On Fri, Nov 29, 2013 at 3:30 PM, Dotan Barak <dotanba-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On 29/11/2013 16:53, Anuj Kalia wrote:
>>
>> Dotan,
>>
>> In the example that I asked about, the 5 packets do not form one
>> message - they are each individual messages. I assume this in the
>> following.
>>
>> I now understand that packet reordering cannot happen in IB - thanks!
>> Only packet loss can happen. From this, I can conclude the following
>> (please correct me if I'm wrong somewhere):
>>
>> "If the requester sends a sequence of single packet messages: {M_1,
>> M_2, ..., M_n}, then the sequence of packets at the responder will be
>> {M_i1, M_i2, ..., M_ik}, where i1 < i2 < ... < ik. Which is just
>> saying that messages can be lost, but not reordered."
>>
>> Is there a case where messages can be reordered in UC transport? i.e.
>> if the requester issues an RDMA write to buf[0] followed by an RDMA
>> write to buf[1], is it possible that at the responder the write to
>> buf[1] is done before the write to buf[0]? I don't think that this is
>> possible.
>
> I assume that we are talking about a single QP.
>
> The messages will be sent by the order they are sent:
> the message that access buf[0] will be sent (and received by the receiver)
> before the message that access buf[1].
>
> If you are talking about the way that the memory data is being written/seen
> in the remote side. I'm throwing a wild guess here, but *maybe* your process
> can see
> the new content in first buf[1] before buf[0] depends on the memory policy,
> NUMA location, etc...
> (this isn't a zone that I have a lot of knowledge in, but I think that this
> may have affect).
>
> Thanks
> Dotan
>
>
>
>>
>> Thanks again. Your comments and blog have been really useful.
>>
>> --Anuj.
>>
>>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: CQEs in unreliable connection

[Dotan Barak]

On 29/11/2013 22:50, Anuj Kalia wrote:
> Dotan,
>
> Thanks for your reply again.
>
> I just wanted to confirm about the order in which messages are sent
> and received. I'm not worried about the memory policy.
>

If you are talking about InfiniBand then this is the case ...

Dotan
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html