TCP not triggering a fast retransmit?

TCP not triggering a fast retransmit?

[Ivan Novick]

[-- Attachment #1: Type: text/plain, Size: 1555 bytes --]

Hello all,

Attached is a packet capture from my application that is running on
RedHat Enterprise Linux 5.4

I am seeing a Retransmission timeout but I was hoping this case would
go into fast retransmit and not RTO.

I am wondering why did the sender not send more data?  If the sender
was to send more data and extend the window then it would seem the
duplicate acks or SACKS should trigger fast retransmit.

The application does not constantly send data, but data should arrive
to be sent before 200 milliseconds that it takes for the RTO.

Is it possible that there was no data to send and the window is not
advanced and the sender is waiting for an ACK.  Then data arrives half
way into the RTO 200 milliseconds while waiting for an outstanding ACK
but the window is not advanced?

You can see right after the RTO and retransmission additional data is
sent, so there is additional data.  In theory that additional data
could be arriving right at the 200 milliseconds point, but we see this
pattern in the dump regularly and I believe data is there before the
end of the 200 milliseconds RTO.

As a related point the advertised window from the receiver seems to be
a constant value of 22060, so either the receiver is handling its data
fast enough to never have to reduce its window... or this number is
really not used to indicate space available currently in the receive
window?

Any feedback to help understand why we are not doing fast retransmit
and or why the sender is not extending the window would be greatly
appreciated.

Cheers,
Ivan Novick

[-- Attachment #2: sender_backoff.cap --]
[-- Type: application/octet-stream, Size: 3076 bytes --]

Re: TCP not triggering a fast retransmit?

[Brian Bloniarz]

On 06/30/2010 02:04 PM, Ivan Novick wrote:
> I am wondering why did the sender not send more data?  If the sender
> was to send more data and extend the window then it would seem the
> duplicate acks or SACKS should trigger fast retransmit.

Hi Ivan,

Like you said, the receive window can't explain why the sender wouldn't
send. The congestion window is another possible reason why the sender
might not send, though it's sorta impossible to tell if that's really
it from a small tcpdump. Maybe investigating the details of congestion
control might yield something?

HTH,
-Brian

Re: TCP not triggering a fast retransmit?

[Mitchell Erblich]

On Jun 30, 2010, at 11:04 AM, Ivan Novick wrote:

> Hello all,
> 
> Attached is a packet capture from my application that is running on
> RedHat Enterprise Linux 5.4
> 
> I am seeing a Retransmission timeout but I was hoping this case would
> go into fast retransmit and not RTO.
> 
> I am wondering why did the sender not send more data?  If the sender
> was to send more data and extend the window then it would seem the
> duplicate acks or SACKS should trigger fast retransmit.
> 
> The application does not constantly send data, but data should arrive
> to be sent before 200 milliseconds that it takes for the RTO.
> 
> Is it possible that there was no data to send and the window is not
> advanced and the sender is waiting for an ACK.  Then data arrives half
> way into the RTO 200 milliseconds while waiting for an outstanding ACK
> but the window is not advanced?
> 
> You can see right after the RTO and retransmission additional data is
> sent, so there is additional data.  In theory that additional data
> could be arriving right at the 200 milliseconds point, but we see this
> pattern in the dump regularly and I believe data is there before the
> end of the 200 milliseconds RTO.
> 
> As a related point the advertised window from the receiver seems to be
> a constant value of 22060, so either the receiver is handling its data
> fast enough to never have to reduce its window... or this number is
> really not used to indicate space available currently in the receive
> window?
> 
> Any feedback to help understand why we are not doing fast retransmit
> and or why the sender is not extending the window would be greatly
> appreciated.
> 
> Cheers,
> Ivan Novick
> <sender_backoff.cap>

Ivan, et al,

Without looking at your dump, not getting DUPAcks then
- their isn't enough in-flight data/ segments to generate enough ACKs (Duplicate ACKs)
assuming you get ACKs with non-increasing SEQs
- or the/some ACKs are being dropped.

Normally to fix this is to set the MTU / MSS size to say 576 at the two
end points if this is re-runnable, OR make it a thin flow

With a real quick look at your dump, 

my first question is why the .1 -> .2 always has a small window of 46
one way data transfer?

Now, taking a look at the dump with a -r ,
What makes you think that DupACKs are being sent?
To then cause a Fast Re-transmit?

Sometimes, adding tracing within the tcps, can identify if.
the tcp flow has periods of idleness,
the tcp flow is/has been application limited versus network limited,
whether the flow is in SS or CA?
CA normally has DELayed ACks, which reduces the number 
    of ACKs to 1/2 or more,
Whether Fast re-xmit is triggered by 2 or 3 DupAcks.
whether any burst avoidance has occured,
etc,


Mitchell Erblich

Re: TCP not triggering a fast retransmit?

[Ivan Novick]

On Wed, Jun 30, 2010 at 1:06 PM, Mitchell Erblich
<erblichs@earthlink.net> wrote:
> On Jun 30, 2010, at 11:04 AM, Ivan Novick wrote:
... snip ...
> Sometimes, adding tracing within the tcps, can identify if.
> the tcp flow has periods of idleness,
> the tcp flow is/has been application limited versus network limited,
> whether the flow is in SS or CA?
> CA normally has DELayed ACks, which reduces the number
>    of ACKs to 1/2 or more,
> Whether Fast re-xmit is triggered by 2 or 3 DupAcks.
> whether any burst avoidance has occured,
> etc,

Hi Mitchell,

When you say adding tracing within the tcps... what method are you
referring to, to add this tracing?  Is it some tool or setting or are
you talking about adding custom debugging output to the kernel?

Cheers,
Ivan

Re: TCP not triggering a fast retransmit?

[Mitchell Erblich]

On Jun 30, 2010, at 1:14 PM, Ivan Novick wrote:

> On Wed, Jun 30, 2010 at 1:06 PM, Mitchell Erblich
> <erblichs@earthlink.net> wrote:
>> On Jun 30, 2010, at 11:04 AM, Ivan Novick wrote:
> ... snip ...
>> Sometimes, adding tracing within the tcps, can identify if.
>> the tcp flow has periods of idleness,
>> the tcp flow is/has been application limited versus network limited,
>> whether the flow is in SS or CA?
>> CA normally has DELayed ACks, which reduces the number
>>    of ACKs to 1/2 or more,
>> Whether Fast re-xmit is triggered by 2 or 3 DupAcks.
>> whether any burst avoidance has occured,
>> etc,
> 
> Hi Mitchell,
> 
> When you say adding tracing within the tcps... what method are you
> referring to, to add this tracing?  Is it some tool or setting or are
> you talking about adding custom debugging output to the kernel?
> 
> Cheers,
> Ivan

Ivan,

Custom debug code that was added into the fast-path
   when extensive tcp flow analysis is desired.
Uses a runtime /proc var to not effect the fast-path
   that by default is dis-abled.

Mitchell Erblich

Re: TCP not triggering a fast retransmit?

[Ben Hutchings]

On Wed, 2010-06-30 at 11:04 -0700, Ivan Novick wrote:
> Hello all,
> 
> Attached is a packet capture from my application that is running on
> RedHat Enterprise Linux 5.4
> 
> I am seeing a Retransmission timeout but I was hoping this case would
> go into fast retransmit and not RTO.
> 
> I am wondering why did the sender not send more data?  If the sender
> was to send more data and extend the window then it would seem the
> duplicate acks or SACKS should trigger fast retransmit.
[...]

In that packet capture I see TCP payload lengths which are 2, 3 and 4
times the usual MSS of 1448 bytes, which implies that GRO or LRO is in
use.  In RHEL 5.4 the TCP stack does not ACK often enough in this case
because it is missing this change:

commit ff9b5e0f08cb650d113eef0c654f931c0a7ae730
Author: Herbert Xu <herbert@gondor.apana.org.au>
Date:   Thu Aug 31 15:11:02 2006 -0700

    [TCP]: Fix rcv mss estimate for LRO

Ben.

-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: TCP not triggering a fast retransmit?

[David Miller]

From: Ben Hutchings <bhutchings@solarflare.com>
Date: Wed, 30 Jun 2010 22:03:49 +0100

> In that packet capture I see TCP payload lengths which are 2, 3 and 4
> times the usual MSS of 1448 bytes, which implies that GRO or LRO is in
> use.  In RHEL 5.4 the TCP stack does not ACK often enough in this case
> because it is missing this change:
> 
> commit ff9b5e0f08cb650d113eef0c654f931c0a7ae730
> Author: Herbert Xu <herbert@gondor.apana.org.au>
> Date:   Thu Aug 31 15:11:02 2006 -0700
> 
>     [TCP]: Fix rcv mss estimate for LRO

It certainly could be, I'll try make sure this gets rectified,
thanks!

Re: TCP not triggering a fast retransmit?

[Mitchell Erblich]

On Jun 30, 2010, at 2:22 PM, David Miller wrote:

> From: Ben Hutchings <bhutchings@solarflare.com>
> Date: Wed, 30 Jun 2010 22:03:49 +0100
> 
>> In that packet capture I see TCP payload lengths which are 2, 3 and 4
>> times the usual MSS of 1448 bytes, which implies that GRO or LRO is in
>> use.  In RHEL 5.4 the TCP stack does not ACK often enough in this case
>> because it is missing this change:
>> 
>> commit ff9b5e0f08cb650d113eef0c654f931c0a7ae730
>> Author: Herbert Xu <herbert@gondor.apana.org.au>
>> Date:   Thu Aug 31 15:11:02 2006 -0700
>> 
>>    [TCP]: Fix rcv mss estimate for LRO
> 
> It certainly could be, I'll try make sure this gets rectified,
> thanks!
> --

Guys,

	I think you suggesting that:

__tcp_ack_snd_chk() within tcp_input.c needs: 
an ABC (Appropriate Byte Counting) Allman type check
where the frame rcv'd computed size is 2x or larger
(and no out of order queue) , then

even if NOT in quickack mode, needs to be ACKed,
with tcp_send_ack()

and if in quickack mode, needs a mss incr seq number of ACKs where
number of ACKs equals the number of the multiple of mss

Note: without incrementing the SEQ between ACKs, it would result in
a DupACK at the other end system,

Correct?

Mitchell Erblich


> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: TCP not triggering a fast retransmit?

[Ivan Novick]

On Wed, Jun 30, 2010 at 2:03 PM, Ben Hutchings
<bhutchings@solarflare.com> wrote:
> On Wed, 2010-06-30 at 11:04 -0700, Ivan Novick wrote:
>> Hello all,
>>
>> Attached is a packet capture from my application that is running on
>> RedHat Enterprise Linux 5.4
>>
>> I am seeing a Retransmission timeout but I was hoping this case would
>> go into fast retransmit and not RTO.
>>
>> I am wondering why did the sender not send more data?  If the sender
>> was to send more data and extend the window then it would seem the
>> duplicate acks or SACKS should trigger fast retransmit.
> [...]
>
> In that packet capture I see TCP payload lengths which are 2, 3 and 4
> times the usual MSS of 1448 bytes, which implies that GRO or LRO is in
> use.  In RHEL 5.4 the TCP stack does not ACK often enough in this case
> because it is missing this change:
>
> commit ff9b5e0f08cb650d113eef0c654f931c0a7ae730
> Author: Herbert Xu <herbert@gondor.apana.org.au>
> Date:   Thu Aug 31 15:11:02 2006 -0700
>
>    [TCP]: Fix rcv mss estimate for LRO

Wow, Thanks! Is this the patch you are talking about:
http://www.mail-archive.com/netdev@vger.kernel.org/msg21151.html  ?

It looks like a one liner.   I could apply it and rebuild my RHEL 5.4
to see if it helps.

Cheers,
Ivan Novick

Re: TCP not triggering a fast retransmit?

[Ben Hutchings]

On Wed, 2010-06-30 at 17:27 -0700, Ivan Novick wrote:
> On Wed, Jun 30, 2010 at 2:03 PM, Ben Hutchings
> <bhutchings@solarflare.com> wrote:
> > On Wed, 2010-06-30 at 11:04 -0700, Ivan Novick wrote:
> >> Hello all,
> >>
> >> Attached is a packet capture from my application that is running on
> >> RedHat Enterprise Linux 5.4
> >>
> >> I am seeing a Retransmission timeout but I was hoping this case would
> >> go into fast retransmit and not RTO.
> >>
> >> I am wondering why did the sender not send more data?  If the sender
> >> was to send more data and extend the window then it would seem the
> >> duplicate acks or SACKS should trigger fast retransmit.
> > [...]
> >
> > In that packet capture I see TCP payload lengths which are 2, 3 and 4
> > times the usual MSS of 1448 bytes, which implies that GRO or LRO is in
> > use.  In RHEL 5.4 the TCP stack does not ACK often enough in this case
> > because it is missing this change:
> >
> > commit ff9b5e0f08cb650d113eef0c654f931c0a7ae730
> > Author: Herbert Xu <herbert@gondor.apana.org.au>
> > Date:   Thu Aug 31 15:11:02 2006 -0700
> >
> >    [TCP]: Fix rcv mss estimate for LRO
> 
> Wow, Thanks! Is this the patch you are talking about:
> http://www.mail-archive.com/netdev@vger.kernel.org/msg21151.html  ?
> 
> It looks like a one liner.   I could apply it and rebuild my RHEL 5.4
> to see if it helps.

Right, that's the same patch.

Ben.

-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.