Send fail notifications

Send fail notifications

[Jamal Hadi Salim]

Lets say  between server and client i have an MTU of 4K.
I send 16K with partial reliability setting.
If the data gets through the receiver sees (I think
SCTP_PARTIAL_DELIVERY_POINT may change this behavior but it probably
defaults to MTU):

Msg1, 4k, !EOR
Msg2, 4k, !EOR
Msg3, 4k, !EOR
Msg4, 4k, EOR

OTOH, at sender if the send fails because PR timeout on some data
chunk occurs i receive from the kernel not 1 but 4 notifications.
Msg1, 4k, Notification(send failed, ssf_length=4k), EOR
Msg2, 4k, Notification(send failed, ssf_length=4k), EOR
Msg3, 4k, Notification(send failed, ssf_length=4k), EOR
Msg4, 4k, Notification(send failed, ssf_length=4k), EOR
.... some last message here to account for notification overhead ....

I was expecting only the last one to have an EOR or least the
ssf_length to be telling me there's more than 4K. Stoopid question:
Is there some  other field I should be looking at?

The only way i could tell the 5 notifications are from the same
original message is when i provide a sinfo_context at sendmsg() time.


cheers,
jamal

Re: Send fail notifications

[Jamal Hadi Salim]

A little more info:
- When i have MTU > 16K, no problem, 1 notification makes it through 
with an EOR and proper info
EOR.
- ssf_flags are always zero (thats one other place that may have carried
such extra info)
- kernel 3.12.0 (with other non-sctp patches)

cheers,
jamal

On 12/18/13 08:59, Jamal Hadi Salim wrote:
>
> Lets say  between server and client i have an MTU of 4K.
> I send 16K with partial reliability setting.
> If the data gets through the receiver sees (I think
> SCTP_PARTIAL_DELIVERY_POINT may change this behavior but it probably
> defaults to MTU):
>
> Msg1, 4k, !EOR
> Msg2, 4k, !EOR
> Msg3, 4k, !EOR
> Msg4, 4k, EOR
>
> OTOH, at sender if the send fails because PR timeout on some data
> chunk occurs i receive from the kernel not 1 but 4 notifications.
> Msg1, 4k, Notification(send failed, ssf_length=4k), EOR
> Msg2, 4k, Notification(send failed, ssf_length=4k), EOR
> Msg3, 4k, Notification(send failed, ssf_length=4k), EOR
> Msg4, 4k, Notification(send failed, ssf_length=4k), EOR
> .... some last message here to account for notification overhead ....
>
> I was expecting only the last one to have an EOR or least the
> ssf_length to be telling me there's more than 4K. Stoopid question:
> Is there some  other field I should be looking at?
>
> The only way i could tell the 5 notifications are from the same
> original message is when i provide a sinfo_context at sendmsg() time.
>
>
> cheers,
> jamal

Re: Send fail notifications

[Vlad Yasevich]

On 12/18/2013 09:06 AM, Jamal Hadi Salim wrote:
> 
> A little more info:
> - When i have MTU > 16K, no problem, 1 notification makes it through
> with an EOR and proper info
> EOR.
> - ssf_flags are always zero (thats one other place that may have carried
> such extra info)

meaning that chunks was unsent.

> - kernel 3.12.0 (with other non-sctp patches)
> 
> cheers,
> jamal
> 
> On 12/18/13 08:59, Jamal Hadi Salim wrote:
>>
>> Lets say  between server and client i have an MTU of 4K.
>> I send 16K with partial reliability setting.
>> If the data gets through the receiver sees (I think
>> SCTP_PARTIAL_DELIVERY_POINT may change this behavior but it probably
>> defaults to MTU):
>>
>> Msg1, 4k, !EOR
>> Msg2, 4k, !EOR
>> Msg3, 4k, !EOR
>> Msg4, 4k, EOR
>>
>> OTOH, at sender if the send fails because PR timeout on some data
>> chunk occurs i receive from the kernel not 1 but 4 notifications.
>> Msg1, 4k, Notification(send failed, ssf_length=4k), EOR
>> Msg2, 4k, Notification(send failed, ssf_length=4k), EOR
>> Msg3, 4k, Notification(send failed, ssf_length=4k), EOR
>> Msg4, 4k, Notification(send failed, ssf_length=4k), EOR
>> .... some last message here to account for notification overhead ....

You receive one notification per chunk.  This is the way lksctp
implemented failure notification.  Whether this is the right or
wrong way to do it may be up for debate.

As I said in the private thread that started this, current spec
is referring to messages when it talks about SEND_FAILED[_EVENT].
It covers the case where the whole message has failed and when
partial message has failed, but it misses the case where parts
of message failed for different reasons (SENT vs. UNSENT).  In
particular the setting sinfo_flags in the sndrcvinfo structure
aren't covered in this case.

So, even when talking about messages, it's feasible that you would
receive multiple notifications for the same message.  One may
specify things like:
   ssf_flags = SENT
   sinfo.sinfo_flags = DATA_FIRST_FRAG

the other may have:
  ssf_info = UNSENT
  sinfo.sinfo_flags = DATA_LAST_FRAG

The EOR flags would be sent on each notification if was received
by your application in full, since in this case EOR signals the
end of a particular notification, the the end of the user data
message we are reporting about.

>>
>> I was expecting only the last one to have an EOR or least the
>> ssf_length to be telling me there's more than 4K. Stoopid question:
>> Is there some  other field I should be looking at?

You could look at the sinfo_flags.  These flags will mirror the flags
from the chunk thus allowing you to kind-of reassemble the data.
The hitch here is the following text in the spec:
   ssf_info:  This field includes the ancillary data (struct
      sctp_sndrcvinfo) used to send the undelivered message
                     ^^^^^^^^^^^^^^^^^^^^^^^
lksctp follows that by simply copying the user supplied data, thus
missing things like chunk ssn, tsn and other fields that could
be useful in identifying this part of the message.

>>
>> The only way i could tell the 5 notifications are from the same
>> original message is when i provide a sinfo_context at sendmsg() time.
>>

That really is the only reliable way to tell if the parts came from the
same message.
In fact, if you look at the description of sinfo_context in Section
5.3.2, you'll see that this is exactly what it's there for.

-vlad

-vlad

Re: Send fail notifications

[Jamal Hadi Salim]

On 12/18/13 10:20, Vlad Yasevich wrote:
> On 12/18/2013 09:06 AM, Jamal Hadi Salim wrote:

> You receive one notification per chunk.  This is the way lksctp
> implemented failure notification.  Whether this is the right or
> wrong way to do it may be up for debate.
>

Ok, assuming the chunk size is corellated to MTU.
With large enough MTU i do receive the full notification + original
message in one read.

> As I said in the private thread that started this, current spec
> is referring to messages when it talks about SEND_FAILED[_EVENT].
> It covers the case where the whole message has failed and when
> partial message has failed, but it misses the case where parts
> of message failed for different reasons (SENT vs. UNSENT).  In
> particular the setting sinfo_flags in the sndrcvinfo structure
> aren't covered in this case.
>
> So, even when talking about messages, it's feasible that you would
> receive multiple notifications for the same message.  One may
> specify things like:
>     ssf_flags = SENT
>     sinfo.sinfo_flags = DATA_FIRST_FRAG
>
> the other may have:
>    ssf_info = UNSENT
>    sinfo.sinfo_flags = DATA_LAST_FRAG
>
> The EOR flags would be sent on each notification if was received
> by your application in full, since in this case EOR signals the
> end of a particular notification, the the end of the user data
> message we are reporting about.
>
> You could look at the sinfo_flags.  These flags will mirror the flags
> from the chunk thus allowing you to kind-of reassemble the data.

Aha, thanks.
Yes, I think these would good enough - painful but sufficient.
I wasnt paying close attention to the sinfo flags earlier
I still expect to receive the full message (although i have complained
about receiving the full message in the past ;->). I see the flag
starting with "2" middle is "0" and termination is "1". Sample log
for one of the messages that failed on my side is:
------
handle_send_failed: Data(cntx 91267) 1024B (0x2) possibly left
handle_send_failed: Data(cntx 91267) 1024B (0x0) possibly left
handle_send_failed: Data(cntx 91267) 1024B (0x0) possibly left
handle_send_failed: Data(cntx 91267) 1024B (0x0) possibly left
handle_send_failed: Data(cntx 91267) 1024B (0x0) possibly left
handle_send_failed: Data(cntx 91267) 1024B (0x0) possibly left
handle_send_failed: Data(cntx 91267) 1024B (0x0) never left
handle_send_failed: Data(cntx 91267) 1024B (0x0) never left
handle_send_failed: Data(cntx 91267) 1024B (0x0) never left
handle_send_failed: Data(cntx 91267) 1024B (0x0) never left
handle_send_failed: Data(cntx 91267) 1024B (0x0) never left
handle_send_failed: Data(cntx 91267) 1024B (0x0) never left
handle_send_failed: Data(cntx 91267) 1024B (0x0) never left
handle_send_failed: Data(cntx 91267) 1024B (0x0) never left
handle_send_failed: Data(cntx 91267) 1024B (0x0) never left
handle_send_failed: Data(cntx 91267) 1024B (0x0) never left
handle_send_failed: Data(cntx 91267) 892B (0x1) never left
-----


> The hitch here is the following text in the spec:
>     ssf_info:  This field includes the ancillary data (struct
>        sctp_sndrcvinfo) used to send the undelivered message
>                       ^^^^^^^^^^^^^^^^^^^^^^^
> lksctp follows that by simply copying the user supplied data, thus
> missing things like chunk ssn, tsn and other fields that could
> be useful in identifying this part of the message.
>

I am probably fine with what you described above.


> That really is the only reliable way to tell if the parts came from the
> same message.
> In fact, if you look at the description of sinfo_context in Section
> 5.3.2, you'll see that this is exactly what it's there for.
>

I love that feature ;-> So for now i will have to use both.
Note, our current use does not desire to retransmit such
a message - if it didnt make it then it is obsolete. But
we need to keep track which particular message didnt get
through; and for that we only need the first chunk.

cheers,
jamal