Re: TCP keepalives ignored by kernel when the contain timestamps

[Charles Bearden <Charles.F.Bearden@uth.tmc.edu>]

[-- Attachment #1: Type: text/plain, Size: 4550 bytes --]

On 06/10/2011 08:56 AM, Eric Dumazet wrote:
> Le jeudi 09 juin 2011 à 10:26 -0500, Charles Bearden a écrit :
>> I have come across a case that looks like it might be a kernel bug. It appears
>> that tcp keepalives sent by a remote system are ignored when they contain tcp
>> timestamps, but are ACKed when they don't. When they are ignored, the remote
>> system resets the connection after a number of retries.
>>
>> I have replicated this problem on both Ubuntu 10.04 with a 2.6.32-32-server
>> kernel (x86_64) and CentOS 5.6 with a 2.6.18-238.12.1.el5 kernel. I'm sorry that
>> I haven't had a chance to try to replicate the bug with a newer kernel, though a
>> co-worker has looked through changelogs for more recent kernels and didn't find
>> anything that looked relevant.
>>
>>   From either of these hosts I run an application that connects to a remote host
>> for 2-3 minutes, and that for most of that time sends no application data back
>> and forth. After 30 seconds of no data from the Linux host, the remote host
>> sends a garden variety keepalive. When the remote host includes tcp timestamps
>> in the keepalives, they are ignored by the Linux host, and the remote host
>> resets the connection after 10 unACKed keepalives. When timestamps are absent
>> from the keepalives, the Linux host ACKs each one, and all is copacetic.
>>
>> Text output of a tcpdump trace of a connection that fails:
>>     http://pastebin.com/v6CpteJ9
>>
>> Text output of a tcpdump trace of a connection that succeeds:
>>     http://pastebin.com/KVLb3Mzh
>>
>> More details, in case you think they are relevant:
>>
>>     My application creates a JDBC connection to a remote MS SQL Server and
>>     executes a statement that does not return a result set, and so it doesn't
>>     need to pass application data back and forth while it executes. The
>>     statement takes 2 or 3 minutes to complete. I connect to two different
>>     remote hosts: a Win2003 machine, and a Win2008R2 machine. The Win2003
>>     machine doesn't put timestamps in its keep-alives, so the application
>>     completes successfully when connecting to that host. If tcp timestamps
>>     are enabled on the Linux host, the Win2008 host includes them in its
>>     keepalives, and they are unACKed, so the connection is reset; if they
>>     are disabled on the Linux host, the Win2008 host doesn't include them in
>>     the keepalives, and the application completes successfully. I use (as
>>     you might expect) sysctl to disable tcp timestamps on the Linux hosts.
>>
>> I have dumps for all permutations of CentOS/Ubuntu, Win200[38], and +/-
>> timestamps on the Linux side, and I will post them if the developers think that
>> they would be useful.
>
> Hi Charles
>
> I could not reproduce the problem here, even using a quite old kernel as
> receiver (2.6.9)
>
> 15:54:33.566192 IP 192.168.20.108.55926>  192.168.20.124.777: SWE
> 479814493:479814493(0) win 14600<mss 1460,sackOK,timestamp 151666
> 0,nop,wscale 7>
> 15:54:33.566265 IP 192.168.20.124.777>  192.168.20.108.55926: S
> 3714869381:3714869381(0) ack 479814494 win 5792<mss
> 1460,sackOK,timestamp 54553041 151666,nop,wscale 2>
> 15:54:33.566274 IP 192.168.20.108.55926>  192.168.20.124.777: . ack 1
> win 115<nop,nop,timestamp 151666 54553041>
> 15:54:33.566281 IP 192.168.20.108.55926>  192.168.20.124.777: P 1:5(4)
> ack 1 win 115<nop,nop,timestamp 151666 54553041>
> 15:54:33.566351 IP 192.168.20.124.777>  192.168.20.108.55926: . ack 5
> win 1448<nop,nop,timestamp 54553041 151666>
> 15:54:33.566375 IP 192.168.20.124.777>  192.168.20.108.55926: P 1:5(4)
> ack 5 win 1448<nop,nop,timestamp 54553041 151666>
> 15:54:33.566380 IP 192.168.20.108.55926>  192.168.20.124.777: . ack 5
> win 115<nop,nop,timestamp 151666 54553041>
> 15:54:43.577945 IP 192.168.20.108.55926>  192.168.20.124.777: . 4:5(1)
> ack 5 win 115<nop,nop,timestamp 152668 54553041>
> 15:54:43.578012 IP 192.168.20.124.777>  192.168.20.108.55926: . ack 5
> win 1448<nop,nop,timestamp 54563053 152668,nop,nop,sack sack 1 {4:5}>
> 15:54:53.597946 IP 192.168.20.108.55926>  192.168.20.124.777: . 4:5(1)
> ack 5 win 115<nop,nop,timestamp 153670 54563053>
> 15:54:53.598012 IP 192.168.20.124.777>  192.168.20.108.55926: . ack 5
> win 1448<nop,nop,timestamp 54573073 153670,nop,nop,sack sack 1 {4:5}>
>
>
> Are you sure frame tcp checksums are OK when the 'faulty' linux receive
> them ?   (tcpdump -v)

I will check when I get into the office and let you know.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5168 bytes --]