Re: receiver window questions

Re: receiver window questions

[Vlad Yasevich]

Hi Michael

Michael Tuexen wrote:
> Hi Vlad,
> 
> we are currently testing the receive behaviour of Linux (and
> other systems).
> 
> We are using Fedora 9, kernel 2.6.25-14.
> 
> The receiver application just opens a 1-to-many style socket
> and sleeps forever, not reading any messages.
> 
> The sender (on a different machine), sends a lot of messages of the
> same size, all on stream 0, ordered.
> 
> When setting the receive buffer space (using the SO_RCVBUF socket option)
> to 10000 we oberserve the following:
> 
> - When sending messages of size 1000 bytes, the receiver SACK the first,
>   announces 9000 bytes windows, SACKs the second announces 8000 bytes
>   and so on. Look fine. The a_rwnd goes down to 0 and discards messages.
>   Everything is fine.
> 
> - When sending messages of size 100 bytes, the receiver SACKs the first
>   messages and reduces the a_rwnd accordingly. Then it looks like the
>   receive buffer grows, because messages are accepted and the a_rwnd does
>   not shrink. Is this intended?

No.  The a_rwnd should go down to 0 as before.

>   We also figured out that after about 670947 messages of size 100 bytes
>   the association is aborted. Is this intended?

This is also not intended.  We'll take a look.

Thanks
-vlad

> 
> - Sending messages of size 10 bytes is similar to 100 byes.
> 
> Best regards
> Michael
> 
> 

Re: receiver window questions

[Michael Tuexen]

Hi Vlad,

Irene has done some more testing.

The ABORT was due to the fact that the testtool was not responding to
HEARTBEATs. She fixed that.

She is still investigating, but the conjecture is that for small  
messages
the receiver accepts floor(2*s/(178+msgsize)) messages, where s is the
receive space set by a socket option. Small means smaller than 180, but
178 could be the real number. Irene will check this tomorrow.
In that case the a_rwnd goes down, and when the above limit is reached,
a_rwnd is not decremented anymore, the messages are not put into the
receive buffer anymore (checked if they are delivered to the  
application),
but the cumtsn is still incremented. So the sender thinks, the  
messages are
successfully transmitted, but the receiver just drops them.

Irene will provide definite test case to reproduce the problem tomorrow.

Best regards
Michael

On May 29, 2008, at 5:06 PM, Vlad Yasevich wrote:

> Hi Michael
>
> Michael Tuexen wrote:
>> Hi Vlad,
>> we are currently testing the receive behaviour of Linux (and
>> other systems).
>> We are using Fedora 9, kernel 2.6.25-14.
>> The receiver application just opens a 1-to-many style socket
>> and sleeps forever, not reading any messages.
>> The sender (on a different machine), sends a lot of messages of the
>> same size, all on stream 0, ordered.
>> When setting the receive buffer space (using the SO_RCVBUF socket  
>> option)
>> to 10000 we oberserve the following:
>> - When sending messages of size 1000 bytes, the receiver SACK the  
>> first,
>>  announces 9000 bytes windows, SACKs the second announces 8000 bytes
>>  and so on. Look fine. The a_rwnd goes down to 0 and discards  
>> messages.
>>  Everything is fine.
>> - When sending messages of size 100 bytes, the receiver SACKs the  
>> first
>>  messages and reduces the a_rwnd accordingly. Then it looks like the
>>  receive buffer grows, because messages are accepted and the a_rwnd  
>> does
>>  not shrink. Is this intended?
>
> No.  The a_rwnd should go down to 0 as before.
>
>>  We also figured out that after about 670947 messages of size 100  
>> bytes
>>  the association is aborted. Is this intended?
>
> This is also not intended.  We'll take a look.
>
> Thanks
> -vlad
>
>> - Sending messages of size 10 bytes is similar to 100 byes.
>> Best regards
>> Michael
>
>

Re: receiver window questions

[Neil Horman]

On Thu, May 29, 2008 at 11:06:11AM -0400, Vlad Yasevich wrote:
> Hi Michael
> 
> Michael Tuexen wrote:
> >Hi Vlad,
> >
> >we are currently testing the receive behaviour of Linux (and
> >other systems).
> >
> >We are using Fedora 9, kernel 2.6.25-14.
> >
> >The receiver application just opens a 1-to-many style socket
> >and sleeps forever, not reading any messages.
> >
> >The sender (on a different machine), sends a lot of messages of the
> >same size, all on stream 0, ordered.
> >
> >When setting the receive buffer space (using the SO_RCVBUF socket option)
> >to 10000 we oberserve the following:
> >
> >- When sending messages of size 1000 bytes, the receiver SACK the first,
> >  announces 9000 bytes windows, SACKs the second announces 8000 bytes
> >  and so on. Look fine. The a_rwnd goes down to 0 and discards messages.
> >  Everything is fine.
> >
> >- When sending messages of size 100 bytes, the receiver SACKs the first
> >  messages and reduces the a_rwnd accordingly. Then it looks like the
> >  receive buffer grows, because messages are accepted and the a_rwnd does
> >  not shrink. Is this intended?
> 
> No.  The a_rwnd should go down to 0 as before.
> 
> >  We also figured out that after about 670947 messages of size 100 bytes
> >  the association is aborted. Is this intended?
> 
> This is also not intended.  We'll take a look.
> 
That does sound odd, yes, a_rwnd should be reduced in response to each received
frame.  do you have tcpdumps of this test available?
Thanks
Neil


-- 
/***************************************************
 *Neil Horman
 *nhorman@tuxdriver.com
 *gpg keyid: 1024D / 0x92A74FA1
 *http://pgp.mit.edu
 ***************************************************/

Re: receiver window questions

[Michael Tuexen]

On May 29, 2008, at 5:25 PM, Neil Horman wrote:

> On Thu, May 29, 2008 at 11:06:11AM -0400, Vlad Yasevich wrote:
>> Hi Michael
>>
>> Michael Tuexen wrote:
>>> Hi Vlad,
>>>
>>> we are currently testing the receive behaviour of Linux (and
>>> other systems).
>>>
>>> We are using Fedora 9, kernel 2.6.25-14.
>>>
>>> The receiver application just opens a 1-to-many style socket
>>> and sleeps forever, not reading any messages.
>>>
>>> The sender (on a different machine), sends a lot of messages of the
>>> same size, all on stream 0, ordered.
>>>
>>> When setting the receive buffer space (using the SO_RCVBUF socket  
>>> option)
>>> to 10000 we oberserve the following:
>>>
>>> - When sending messages of size 1000 bytes, the receiver SACK the  
>>> first,
>>> announces 9000 bytes windows, SACKs the second announces 8000 bytes
>>> and so on. Look fine. The a_rwnd goes down to 0 and discards  
>>> messages.
>>> Everything is fine.
>>>
>>> - When sending messages of size 100 bytes, the receiver SACKs the  
>>> first
>>> messages and reduces the a_rwnd accordingly. Then it looks like the
>>> receive buffer grows, because messages are accepted and the a_rwnd  
>>> does
>>> not shrink. Is this intended?
>>
>> No.  The a_rwnd should go down to 0 as before.
>>
>>> We also figured out that after about 670947 messages of size 100  
>>> bytes
>>> the association is aborted. Is this intended?
>>
>> This is also not intended.  We'll take a look.
>>
> That does sound odd, yes, a_rwnd should be reduced in response to  
> each received
> frame.  do you have tcpdumps of this test available?
> Thanks
> Neil
It looks like that you do not reduce the a_rwnd, because you discard  
the message.
The problem seems to be that you ack it by incrementing the  
cumtsn_ack...

Irene did the tests in the lab. We'll provide traces and a testprogram  
tomorrow.
>
>
>
> -- 
> /***************************************************
> *Neil Horman
> *nhorman@tuxdriver.com
> *gpg keyid: 1024D / 0x92A74FA1
> *http://pgp.mit.edu
> ***************************************************/
>

Re: receiver window questions

[Vlad Yasevich]

Neil Horman wrote:
> On Thu, May 29, 2008 at 11:06:11AM -0400, Vlad Yasevich wrote:
>> Hi Michael
>>
>> Michael Tuexen wrote:
>>> Hi Vlad,
>>>
>>> we are currently testing the receive behaviour of Linux (and
>>> other systems).
>>>
>>> We are using Fedora 9, kernel 2.6.25-14.
>>>
>>> The receiver application just opens a 1-to-many style socket
>>> and sleeps forever, not reading any messages.
>>>
>>> The sender (on a different machine), sends a lot of messages of the
>>> same size, all on stream 0, ordered.
>>>
>>> When setting the receive buffer space (using the SO_RCVBUF socket option)
>>> to 10000 we oberserve the following:
>>>
>>> - When sending messages of size 1000 bytes, the receiver SACK the first,
>>>  announces 9000 bytes windows, SACKs the second announces 8000 bytes
>>>  and so on. Look fine. The a_rwnd goes down to 0 and discards messages.
>>>  Everything is fine.
>>>
>>> - When sending messages of size 100 bytes, the receiver SACKs the first
>>>  messages and reduces the a_rwnd accordingly. Then it looks like the
>>>  receive buffer grows, because messages are accepted and the a_rwnd does
>>>  not shrink. Is this intended?
>> No.  The a_rwnd should go down to 0 as before.
>>
>>>  We also figured out that after about 670947 messages of size 100 bytes
>>>  the association is aborted. Is this intended?
>> This is also not intended.  We'll take a look.
>>
> That does sound odd, yes, a_rwnd should be reduced in response to each received
> frame.  do you have tcpdumps of this test available?
> Thanks
> Neil

I think this is hitting a condition where the receiver buffer is exhausted prior
to rwnd.  We generally mark the TSN as received prior to attempting an internal 
allocation to carry the data.  Thus, if this allocation fails, we'll continue
reporting the tsn as received and move the cum-tsn if appropriate.

We've been trying to figure out what the correct way to solve this condition is
and so far haven't come up with a workable solution.

One approach is to reduce the a_rwnd to 1 when the receive buffer reaches some
consumption threshold.  This would allow 1 packet in flight to allow receive 
buffer to drain.  Another piece of this puzzle is also SWS avoidance.  We 
currently have somewhat poor algorithm that allows a max of 1 MTU in transit.

-vlad

Re: receiver window questions

[Michael Tuexen]

On May 29, 2008, at 5:50 PM, Vlad Yasevich wrote:

> Neil Horman wrote:
>> On Thu, May 29, 2008 at 11:06:11AM -0400, Vlad Yasevich wrote:
>>> Hi Michael
>>>
>>> Michael Tuexen wrote:
>>>> Hi Vlad,
>>>>
>>>> we are currently testing the receive behaviour of Linux (and
>>>> other systems).
>>>>
>>>> We are using Fedora 9, kernel 2.6.25-14.
>>>>
>>>> The receiver application just opens a 1-to-many style socket
>>>> and sleeps forever, not reading any messages.
>>>>
>>>> The sender (on a different machine), sends a lot of messages of the
>>>> same size, all on stream 0, ordered.
>>>>
>>>> When setting the receive buffer space (using the SO_RCVBUF socket  
>>>> option)
>>>> to 10000 we oberserve the following:
>>>>
>>>> - When sending messages of size 1000 bytes, the receiver SACK the  
>>>> first,
>>>> announces 9000 bytes windows, SACKs the second announces 8000 bytes
>>>> and so on. Look fine. The a_rwnd goes down to 0 and discards  
>>>> messages.
>>>> Everything is fine.
>>>>
>>>> - When sending messages of size 100 bytes, the receiver SACKs the  
>>>> first
>>>> messages and reduces the a_rwnd accordingly. Then it looks like the
>>>> receive buffer grows, because messages are accepted and the  
>>>> a_rwnd does
>>>> not shrink. Is this intended?
>>> No.  The a_rwnd should go down to 0 as before.
>>>
>>>> We also figured out that after about 670947 messages of size 100  
>>>> bytes
>>>> the association is aborted. Is this intended?
>>> This is also not intended.  We'll take a look.
>>>
>> That does sound odd, yes, a_rwnd should be reduced in response to  
>> each received
>> frame.  do you have tcpdumps of this test available?
>> Thanks
>> Neil
>
> I think this is hitting a condition where the receiver buffer is  
> exhausted prior
> to rwnd.  We generally mark the TSN as received prior to attempting  
> an internal allocation to carry the data.  Thus, if this allocation  
> fails, we'll continue
> reporting the tsn as received and move the cum-tsn if appropriate.
This would explain the behaviour completely.
>
>
> We've been trying to figure out what the correct way to solve this  
> condition is
> and so far haven't come up with a workable solution.
We are looking at the behavior of different implementations to look at  
interoperability
issues. That is how we came to test this...
>
>
> One approach is to reduce the a_rwnd to 1 when the receive buffer  
> reaches some
> consumption threshold.  This would allow 1 packet in flight to allow  
> receive buffer to drain.  Another piece of this puzzle is also SWS  
> avoidance.  We currently have somewhat poor algorithm that allows a  
> max of 1 MTU in transit.
BSD reduces the a_rwnd to 1 in case of small windows (less than 3000  
bytes) for SWS avoidance,
which you have to do anyway. And it makes sure that it does not report  
a message as received when it
is discarded.
>
>
> -vlad
>

Re: receiver window questions

[Neil Horman]

On Thu, May 29, 2008 at 11:50:56AM -0400, Vlad Yasevich wrote:
> Neil Horman wrote:
> >On Thu, May 29, 2008 at 11:06:11AM -0400, Vlad Yasevich wrote:
> 
> I think this is hitting a condition where the receiver buffer is exhausted 
> prior
> to rwnd.  We generally mark the TSN as received prior to attempting an 
> internal allocation to carry the data.  Thus, if this allocation fails, 
> we'll continue
> reporting the tsn as received and move the cum-tsn if appropriate.
> 
> We've been trying to figure out what the correct way to solve this 
> condition is
> and so far haven't come up with a workable solution.
> 
You're right it does sound like that.  You know, I haven't visited that code
since we rewrote the receive buffer management code to expand according to
available memory with the sk_mem_schedule api.  Do you think this could be as
simple as removing this drop point?

Neil

Re: receiver window questions

[Michael Tuexen]

[-- Attachment #1: Type: text/plain, Size: 545 bytes --]

Dear all,

for message size larger or equal to 176 byte, the behaviour seems
to be OK, for smaller ones there is a problem.

Please find attached two traces. arwnd175.pcap shows the buggy  
behaviour,
arwnd176.pcap shows the acceptable behaviour.

One the receiver side we just opened a listening 1-to-many style socket,
waited until the sender (a testtool) sent all its data and then started
to receive all messages. This is how we figured out that messages were  
lost.

Please let us know if you need more information.

Best regards
Michael


[-- Attachment #2: arwnd175.pcap.gz --]
[-- Type: application/x-gzip, Size: 4421 bytes --]

[-- Attachment #3: arwnd176.pcap.gz --]
[-- Type: application/x-gzip, Size: 4053 bytes --]

[-- Attachment #4: Type: text/plain, Size: 995 bytes --]



On May 30, 2008, at 1:51 AM, Neil Horman wrote:

> On Thu, May 29, 2008 at 11:50:56AM -0400, Vlad Yasevich wrote:
>> Neil Horman wrote:
>>> On Thu, May 29, 2008 at 11:06:11AM -0400, Vlad Yasevich wrote:
>>
>> I think this is hitting a condition where the receiver buffer is  
>> exhausted
>> prior
>> to rwnd.  We generally mark the TSN as received prior to attempting  
>> an
>> internal allocation to carry the data.  Thus, if this allocation  
>> fails,
>> we'll continue
>> reporting the tsn as received and move the cum-tsn if appropriate.
>>
>> We've been trying to figure out what the correct way to solve this
>> condition is
>> and so far haven't come up with a workable solution.
>>
> You're right it does sound like that.  You know, I haven't visited  
> that code
> since we rewrote the receive buffer management code to expand  
> according to
> available memory with the sk_mem_schedule api.  Do you think this  
> could be as
> simple as removing this drop point?
>
> Neil
>
>

Re: receiver window questions

[Vlad Yasevich]

Neil Horman wrote:
> On Thu, May 29, 2008 at 11:50:56AM -0400, Vlad Yasevich wrote:
>> Neil Horman wrote:
>>> On Thu, May 29, 2008 at 11:06:11AM -0400, Vlad Yasevich wrote:
>> I think this is hitting a condition where the receiver buffer is exhausted 
>> prior
>> to rwnd.  We generally mark the TSN as received prior to attempting an 
>> internal allocation to carry the data.  Thus, if this allocation fails, 
>> we'll continue
>> reporting the tsn as received and move the cum-tsn if appropriate.
>>
>> We've been trying to figure out what the correct way to solve this 
>> condition is
>> and so far haven't come up with a workable solution.
>>
> You're right it does sound like that.  You know, I haven't visited that code
> since we rewrote the receive buffer management code to expand according to
> available memory with the sk_mem_schedule api.  Do you think this could be as
> simple as removing this drop point?
> 

The problem is that if the user sets the buffer size, we don't auto-tune it.
This is what happened in this case.  However, even with auto-tunning, this
still shows up with 1 byte data chunks.

So, 2 things need to happen:
	1)  when we drop the chunk due to allocation failure, we have to remove
	    the tsn from the map.  This one is easy

	2)  We need to properly detect SWS.  I haven't looked fully at this, but
             it feels a bit more involved.  Implementing something like what BSD
	    has would also work, i.e. reducing a_rwnd 1 when receive buffer is
	    about to be exhausted.

-vlad

Re: receiver window questions

[Michael Tuexen]

Hi Vlad,

is there any description available how the auto-tuning works?
But even with auto-tuning you still have the a_rwnd you report
to the outside world (in SACKs) and an internal value which
you use for tracking you buffer, right?

Best regards
Michael

On May 30, 2008, at 3:02 PM, Vlad Yasevich wrote:

> Neil Horman wrote:
>> On Thu, May 29, 2008 at 11:50:56AM -0400, Vlad Yasevich wrote:
>>> Neil Horman wrote:
>>>> On Thu, May 29, 2008 at 11:06:11AM -0400, Vlad Yasevich wrote:
>>> I think this is hitting a condition where the receiver buffer is  
>>> exhausted prior
>>> to rwnd.  We generally mark the TSN as received prior to  
>>> attempting an internal allocation to carry the data.  Thus, if  
>>> this allocation fails, we'll continue
>>> reporting the tsn as received and move the cum-tsn if appropriate.
>>>
>>> We've been trying to figure out what the correct way to solve this  
>>> condition is
>>> and so far haven't come up with a workable solution.
>>>
>> You're right it does sound like that.  You know, I haven't visited  
>> that code
>> since we rewrote the receive buffer management code to expand  
>> according to
>> available memory with the sk_mem_schedule api.  Do you think this  
>> could be as
>> simple as removing this drop point?
>
> The problem is that if the user sets the buffer size, we don't auto- 
> tune it.
> This is what happened in this case.  However, even with auto- 
> tunning, this
> still shows up with 1 byte data chunks.
>
> So, 2 things need to happen:
> 	1)  when we drop the chunk due to allocation failure, we have to  
> remove
> 	    the tsn from the map.  This one is easy
>
> 	2)  We need to properly detect SWS.  I haven't looked fully at  
> this, but
>            it feels a bit more involved.  Implementing something  
> like what BSD
> 	    has would also work, i.e. reducing a_rwnd 1 when receive buffer  
> is
> 	    about to be exhausted.
>
> -vlad
>

Re: receiver window questions

[Vlad Yasevich]

Michael Tuexen wrote:
> Hi Vlad,
> 
> is there any description available how the auto-tuning works?

There is no formal writeup.  In short it works the same as TCP.

Essentially, we start out with a set buffer space and calculated
rwnd.  Then as buffer space and window is consumed, we see if
buffer space is consumed faster then rwnd.  If we reach a point
where default buffer space is consumed, but rwnd is not, we
grow the buffer space slowly up to the next threshold, thus allowing
all of rwnd to be consumed.

There is also a system-wide threshold that we look at when growing
the buffer.   So that if the system is experiencing memory pressure
conditions, the growth and allocations will fail.

> But even with auto-tuning you still have the a_rwnd you report
> to the outside world (in SACKs) and an internal value which
> you use for tracking you buffer, right?

Buffer space tracking is separate from rwnd and a_rwnd that we
advertise.

-vlad

> 
> Best regards
> Michael
> 

Re: receiver window questions

[Neil Horman]

On Fri, May 30, 2008 at 05:08:10PM +0200, Michael Tuexen wrote:
> Hi Vlad,
>
> is there any description available how the auto-tuning works?
> But even with auto-tuning you still have the a_rwnd you report
> to the outside world (in SACKs) and an internal value which
> you use for tracking you buffer, right?
>


As Vlad said, there is no formal description available, but if you want a
functional overview, the sk_mem_schedule api is the place to look to understand
how thresholds, and memory pressure detection work

Regards
Neil


> Best regards
> Michael
>
> On May 30, 2008, at 3:02 PM, Vlad Yasevich wrote:
>
>> Neil Horman wrote:
>>> On Thu, May 29, 2008 at 11:50:56AM -0400, Vlad Yasevich wrote:
>>>> Neil Horman wrote:
>>>>> On Thu, May 29, 2008 at 11:06:11AM -0400, Vlad Yasevich wrote:
>>>> I think this is hitting a condition where the receiver buffer is 
>>>> exhausted prior
>>>> to rwnd.  We generally mark the TSN as received prior to attempting an 
>>>> internal allocation to carry the data.  Thus, if this allocation fails, 
>>>> we'll continue
>>>> reporting the tsn as received and move the cum-tsn if appropriate.
>>>>
>>>> We've been trying to figure out what the correct way to solve this 
>>>> condition is
>>>> and so far haven't come up with a workable solution.
>>>>
>>> You're right it does sound like that.  You know, I haven't visited that 
>>> code
>>> since we rewrote the receive buffer management code to expand according 
>>> to
>>> available memory with the sk_mem_schedule api.  Do you think this could 
>>> be as
>>> simple as removing this drop point?
>>
>> The problem is that if the user sets the buffer size, we don't auto-tune 
>> it.
>> This is what happened in this case.  However, even with auto-tunning, this
>> still shows up with 1 byte data chunks.
>>
>> So, 2 things need to happen:
>> 	1)  when we drop the chunk due to allocation failure, we have to remove
>> 	    the tsn from the map.  This one is easy
>>
>> 	2)  We need to properly detect SWS.  I haven't looked fully at this, but
>>            it feels a bit more involved.  Implementing something like what 
>> BSD
>> 	    has would also work, i.e. reducing a_rwnd 1 when receive buffer is
>> 	    about to be exhausted.
>>
>> -vlad
>>

-- 
/****************************************************
 * Neil Horman <nhorman@tuxdriver.com>
 * Software Engineer, Red Hat
 ****************************************************/