Miscalculated TCP ACK ?

Miscalculated TCP ACK ?

[Gabriel Beddingfield]

I'm having trouble with a particular server via HTTPS.  It appears
that my local linux machines are sending incorrect ACK.  However, I
don't have enough expertise to know for sure.

Using wireshark, the server sends:

    Transmission Control Protocol, Src Port: https (443), Dst Port:
36015 (36015), Seq: 27658, Ack: 827, Len: 18

Local machine replies:

    Transmission Control Protocol, Src Port: 36015 (36015), Dst Port:
https (443), Seq: 827, Ack: 27677, Len: 0

It appears to me that the ACK is off-by-one (should have been 27676).
The server replies again with retransmissions.  This is with several
kernel versions on local machines, ranging from 2.6.31 to 2.6.38.

Can anyone shed insight into what's happening?  This looks like a bug to me.

Thanks,
Gabriel

Re: Miscalculated TCP ACK ?

[Eric Dumazet]

Le lundi 22 août 2011 à 10:34 -0500, Gabriel Beddingfield a écrit :
> I'm having trouble with a particular server via HTTPS.  It appears
> that my local linux machines are sending incorrect ACK.  However, I
> don't have enough expertise to know for sure.
> 
> Using wireshark, the server sends:
> 
>     Transmission Control Protocol, Src Port: https (443), Dst Port:
> 36015 (36015), Seq: 27658, Ack: 827, Len: 18
> 
> Local machine replies:
> 
>     Transmission Control Protocol, Src Port: 36015 (36015), Dst Port:
> https (443), Seq: 827, Ack: 27677, Len: 0
> 
> It appears to me that the ACK is off-by-one (should have been 27676).
> The server replies again with retransmissions.  This is with several
> kernel versions on local machines, ranging from 2.6.31 to 2.6.38.
> 
> Can anyone shed insight into what's happening?  This looks like a bug to me.

Not a bug if frame carried a FIN

18:32:16.177801 IP 10.150.51.213.52264 > 10.150.45.194.22: Flags [F.], seq 3312, ack 3241, win 412, length 0
18:32:16.178096 IP 10.150.45.194.22 > 10.150.51.213.52264: Flags [F.], seq 3241, ack 3313, win 203, length 0
18:32:16.256949 IP 10.150.51.213.52264 > 10.150.45.194.22: Flags [.], ack 3242, win 412, length 0

Re: Miscalculated TCP ACK ?

[Daniel Baluta]

>> It appears to me that the ACK is off-by-one (should have been 27676).
>> The server replies again with retransmissions.  This is with several
>> kernel versions on local machines, ranging from 2.6.31 to 2.6.38.

> Not a bug if frame carried a FIN

Well then, why do retransmissions occur?
Gabriel could you post the full capture file?

thanks,
Daniel.

Re: Miscalculated TCP ACK ?

[Hagen Paul Pfeifer]

* Gabriel Beddingfield | 2011-08-22 10:34:14 [-0500]:

>I'm having trouble with a particular server via HTTPS.  It appears
>that my local linux machines are sending incorrect ACK.  However, I
>don't have enough expertise to know for sure.
>
>Using wireshark, the server sends:
>
>    Transmission Control Protocol, Src Port: https (443), Dst Port:
>36015 (36015), Seq: 27658, Ack: 827, Len: 18
>
>Local machine replies:
>
>    Transmission Control Protocol, Src Port: 36015 (36015), Dst Port:
>https (443), Seq: 827, Ack: 27677, Len: 0
>
>It appears to me that the ACK is off-by-one (should have been 27676).

No, it is absolutely correct: ACK -> last seen Seq plus one (27658 + 18 + 1):

RFC 793:

    If the ACK control bit is set this field contains the value of the
		next sequence number the sender of the segment is expecting to
		receive.  Once a connection is established this is always sent.


Cheers, Hagen

Re: Miscalculated TCP ACK ?

[Gabriel Beddingfield]

On Mon, Aug 22, 2011 at 12:19 PM, Daniel Baluta <daniel.baluta@gmail.com> wrote:
>>> It appears to me that the ACK is off-by-one (should have been 27676).
>>> The server replies again with retransmissions.  This is with several
>>> kernel versions on local machines, ranging from 2.6.31 to 2.6.38.
>
>> Not a bug if frame carried a FIN

The server packet had the FIN bit set.

> Well then, why do retransmissions occur?
> Gabriel could you post the full capture file?

Sure: http://gabe.is-a-geek.org/tmp/20110822-skyward-conversation-segment.tcpdump

I clipped the conversation to the vicinity of the problem.

-gabriel

Re: Miscalculated TCP ACK ?

[Eric Dumazet]

Le lundi 22 août 2011 à 13:05 -0500, Gabriel Beddingfield a écrit : 
> On Mon, Aug 22, 2011 at 12:19 PM, Daniel Baluta <daniel.baluta@gmail.com> wrote:
> >>> It appears to me that the ACK is off-by-one (should have been 27676).
> >>> The server replies again with retransmissions.  This is with several
> >>> kernel versions on local machines, ranging from 2.6.31 to 2.6.38.
> >
> >> Not a bug if frame carried a FIN
> 
> The server packet had the FIN bit set.
> 

then its fine.

> > Well then, why do retransmissions occur?
> > Gabriel could you post the full capture file?
> 
> Sure: http://gabe.is-a-geek.org/tmp/20110822-skyward-conversation-segment.tcpdump
> 
> I clipped the conversation to the vicinity of the problem.
> 

This sounds as a TSO bug, maybe driver/NIC related

What is the NIC/driver used on your server ?

try "ethtool -K eth0 tso off" on the server

$ tcpdump -p -n -S -r 20110822-skyward-conversation-segment.tcpdump
reading from file 20110822-skyward-conversation-segment.tcpdump, link-type EN10MB (Ethernet)

Sorry we miss a few frames just before.

14:48:31.368344 IP 50.84.128.30.443 > 192.168.1.25.36015: Flags [.], seq 401237709:401239157, ack 1404739303, win 64334, options [nop,nop,TS val 1663311 ecr 3624365496], length 1448
14:48:31.368410 IP 192.168.1.25.36015 > 50.84.128.30.443: Flags [.], ack 401239157, win 52128, options [nop,nop,TS val 3624365515 ecr 1663311], length 0
14:48:31.368556 IP 50.84.128.30.443 > 192.168.1.25.36015: Flags [.], seq 401239157:401240605, ack 1404739303, win 64334, options [nop,nop,TS val 1663311 ecr 3624365496], length 1448
14:48:31.369019 IP 50.84.128.30.443 > 192.168.1.25.36015: Flags [.], seq 401240605:401242053, ack 1404739303, win 64334, options [nop,nop,TS val 1663311 ecr 3624365496], length 1448
14:48:31.369099 IP 192.168.1.25.36015 > 50.84.128.30.443: Flags [.], ack 401242053, win 57920, options [nop,nop,TS val 3624365515 ecr 1663311], length 0
14:48:31.369357 IP 50.84.128.30.443 > 192.168.1.25.36015: Flags [.], seq 401242053:401243501, ack 1404739303, win 64334, options [nop,nop,TS val 1663311 ecr 3624365496], length 1448
14:48:31.369412 IP 50.84.128.30.443 > 192.168.1.25.36015: Flags [FP.], seq 401243501:401243519, ack 1404739303, win 64334, options [nop,nop,TS val 1663311 ecr 3624365496], length 18

following frames acknowledge all data sent from server and FIN flag.
14:48:31.369442 IP 192.168.1.25.36015 > 50.84.128.30.443: Flags [.], ack 401243520, win 57920, options [nop,nop,TS val 3624365516 ecr 1663311], length 0

Up to there, its OK.

But following frame makes no sense : this data was already acknowledged

14:48:31.373447 IP 50.84.128.30.443 > 192.168.1.25.36015: Flags [.], seq 401236261:401237709, ack 1404739303, win 64334, options [nop,nop,TS val 1663311 ecr 3624365496], length 1448

14:48:31.373532 IP 192.168.1.25.36015 > 50.84.128.30.443: Flags [.], ack 401243520, win 57920, options [nop,nop,TS val 3624365520 ecr 1663311,nop,nop,sack 1 {401236261:401237709}], length 0

client timeouts :
14:48:31.458943 IP 192.168.1.25.36015 > 50.84.128.30.443: Flags [F.], seq 1404739303, ack 401243520, win 57920, options [nop,nop,TS val 3624365605 ecr 1663311], length 0

14:48:31.475204 IP 50.84.128.30.443 > 192.168.1.25.36015: Flags [.], ack 1404739304, win 64334, options [nop,nop,TS val 1663321 ecr 3624365605], length 0
14:48:31.475351 IP 50.84.128.30.443 > 192.168.1.25.36015: Flags [.], ack 1404739304, win 0, length 0
14:48:31.475395 IP 192.168.1.25.36015 > 50.84.128.30.443: Flags [R], seq 1404739304, win 0, length 0

Re: Miscalculated TCP ACK ?

[Gabriel Beddingfield]

On 08/22/2011 01:49 PM, Eric Dumazet wrote:
> Le lundi 22 août 2011 à 13:05 -0500, Gabriel Beddingfield a écrit :
>> On Mon, Aug 22, 2011 at 12:19 PM, Daniel Baluta<daniel.baluta@gmail.com>  wrote:
>>>>> It appears to me that the ACK is off-by-one (should have been 27676).
>>>>> The server replies again with retransmissions.  This is with several
>>>>> kernel versions on local machines, ranging from 2.6.31 to 2.6.38.
>>>
>>>> Not a bug if frame carried a FIN
>>
>> The server packet had the FIN bit set.
>>
>
> then its fine.

Thank you!


>> I clipped the conversation to the vicinity of the problem.
>
> This sounds as a TSO bug, maybe driver/NIC related
>
> What is the NIC/driver used on your server ?

New news:  If I remove my router (Linksys WRTU54G-TM), the problem goes 
away.

Thank you guys for your helpful insights. :-)

-gabriel