Re: Bad TCP timestamps on non-PC platforms

Re: Bad TCP timestamps on non-PC platforms

[Alex Dubov]

> > 
> > You dont give new informations ;)
> > 

Some additional information (my wireshark was not validating checksums, so
I missed the real culprit).

It so appears, that when timestamps are enabled on my dev board, TCP
checksum in SYN packet turns out to be incorrect. That's why those packets
are dropped by the remote side. When TCP timestamps are disabled, checksums
are correct.

Any ideas where this particular problem may originate?



      

Re: Bad TCP timestamps on non-PC platforms

[Stephen Hemminger]

On Sun, 9 Jan 2011 19:41:13 -0800 (PST)
Alex Dubov <oakad@yahoo.com> wrote:

> > > 
> > > You dont give new informations ;)
> > > 
> 
> Some additional information (my wireshark was not validating checksums, so
> I missed the real culprit).
> 
> It so appears, that when timestamps are enabled on my dev board, TCP
> checksum in SYN packet turns out to be incorrect. That's why those packets
> are dropped by the remote side. When TCP timestamps are disabled, checksums
> are correct.
> 
> Any ideas where this particular problem may originate?

Is there a bug in checksum offload in the driver?
Does the hardware correctly handle checksum of packets that are
unaligned or whose length is an odd number of bytes?

If the hardware can't do checksum correctly, the driver should either
disable checksum offload or in worst case copy the packet to a new
buffer that is in a known safe place.

Re: Bad TCP timestamps on non-PC platforms

[Alex Dubov]

> > Any ideas where this particular problem may
> originate?
> 
> Is there a bug in checksum offload in the driver?
> Does the hardware correctly handle checksum of packets that
> are
> unaligned or whose length is an odd number of bytes?
> 
> If the hardware can't do checksum correctly, the driver
> should either
> disable checksum offload or in worst case copy the packet
> to a new
> buffer that is in a known safe place.

I managed to work around the issue by clearing the NETIF_F_IP_CSUM
feature flag of the gianfar network driver, so it appears the bug is
indeed somewhere there.



      

Re: Bad TCP timestamps on non-PC platforms

[Eric Dumazet]

Le dimanche 09 janvier 2011 à 22:33 -0800, Alex Dubov a écrit :
> > > Any ideas where this particular problem may
> > originate?
> > 
> > Is there a bug in checksum offload in the driver?
> > Does the hardware correctly handle checksum of packets that
> > are
> > unaligned or whose length is an odd number of bytes?
> > 
> > If the hardware can't do checksum correctly, the driver
> > should either
> > disable checksum offload or in worst case copy the packet
> > to a new
> > buffer that is in a known safe place.
> 
> I managed to work around the issue by clearing the NETIF_F_IP_CSUM
> feature flag of the gianfar network driver, so it appears the bug is
> indeed somewhere there.
> 

Its incredible how TCP timestamps can be "usual suspects" ;)

Re: Bad TCP timestamps on non-PC platforms

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Mon, 10 Jan 2011 08:07:56 +0100

> Le dimanche 09 janvier 2011 à 22:33 -0800, Alex Dubov a écrit :
>> I managed to work around the issue by clearing the NETIF_F_IP_CSUM
>> feature flag of the gianfar network driver, so it appears the bug is
>> indeed somewhere there.
>> 
> 
> Its incredible how TCP timestamps can be "usual suspects" ;)

Or RTC chips! :-)

Bad TCP timestamps on non-PC platforms

[Alex Dubov]

Greetings.
I'm dealing with 2.6.37-rc7 kernel on MPC8548 platform.

It so appears, that recent kernels have sysctl_tcp_timestamps set to "1"
by default.

On embedded platforms, where real time clock is initialized lately or
absent outright, this causes TSval field of outgoing TCP packets to be
set to some garbage value, in my case in the vicinity of 0xffffffff. As a
result, other Linux machines silently drop such packets, preventing normal
completion of network boot or any other TCP dependent operation.

Therefore, I feel, two changes are necessary (I can send in a patch):
1. Make sysctl_tcp_timestamps value config-time selectable (it must be
disabled by default on machines without RTC).
2. When re-enabling tcp_timestamps through sysctl, reset the timestamp
counter to the current system clock value.

And an optional, tricky one:
3. Postpone TCP timestamp counter initialization until RTC is actually
available (if RTC is connected to i2c bus, TCP is initialized well ahead
of it).



      

Re: Bad TCP timestamps on non-PC platforms

[Eric Dumazet]

Le mercredi 05 janvier 2011 à 22:55 -0800, Alex Dubov a écrit :
> Greetings.
> I'm dealing with 2.6.37-rc7 kernel on MPC8548 platform.
> 
> It so appears, that recent kernels have sysctl_tcp_timestamps set to "1"
> by default.
> 
> On embedded platforms, where real time clock is initialized lately or
> absent outright, this causes TSval field of outgoing TCP packets to be
> set to some garbage value, in my case in the vicinity of 0xffffffff. As a
> result, other Linux machines silently drop such packets, preventing normal
> completion of network boot or any other TCP dependent operation.
> 
> Therefore, I feel, two changes are necessary (I can send in a patch):
> 1. Make sysctl_tcp_timestamps value config-time selectable (it must be
> disabled by default on machines without RTC).
> 2. When re-enabling tcp_timestamps through sysctl, reset the timestamp
> counter to the current system clock value.
> 
> And an optional, tricky one:
> 3. Postpone TCP timestamp counter initialization until RTC is actually
> available (if RTC is connected to i2c bus, TCP is initialized well ahead
> of it).
> 

I am trying to understand the problem, but I guess you hit something not
related to RTC at all.

#define tcp_time_stamp		((__u32)(jiffies))
#define INITIAL_JIFFIES ((unsigned long)(unsigned int) (-300*HZ))

jiffies can be ZERO 300 seconds after boot.

It should not matter with RFC1323. Some TCP session could in theory be
done without TCP timestamps being activated (because SYN paquet would
carry a ZERO  tsval/tsecr), but this should not prevents normal network
operation.

Are you sure the "other linux machines" dont have
strange /etc/sysctl.conf settings ?

Re: Bad TCP timestamps on non-PC platforms

[Alex Dubov]

Sorry for the awful synopsis of my problem. I never cease to amaze myself
at how bad those usually turn up. :-)

What I really meant to write is:

I have a dev board running 2.6.37-rc7. Normal kernel config, nothing fancy.
Remote machines are just usual linux boxes in constant operation (I tried
several of those).

UDP/DHCP works correctly all the time, so ethernet side is probably ok.

When tcp_timestamps are enabled, SYN packets from dev board just get
ignored by the remote side. I see them arrive in wireshark, but nothing
else happens.

When I disable tcp_timestamps on the dev board everything works.

The problem is reproducible every single time.

The only difference is the "Options" block of the SYN packets.
If timestamps are not really to blame, then it probably window scale
parameters. That's what I see on a usual dropped packet:

Options: (20 bytes)
        Maximum segment size: 1460 bytes
        SACK permitted
        Timestamps: TSval 4294893842, TSecr 0
        NOP
        Window scale: 5 (multiply by 32)



      

Re: Bad TCP timestamps on non-PC platforms

[Eric Dumazet]

Le jeudi 06 janvier 2011 à 17:55 -0800, Alex Dubov a écrit :
> Sorry for the awful synopsis of my problem. I never cease to amaze myself
> at how bad those usually turn up. :-)
> 
> What I really meant to write is:
> 
> I have a dev board running 2.6.37-rc7. Normal kernel config, nothing fancy.
> Remote machines are just usual linux boxes in constant operation (I tried
> several of those).
> 
> UDP/DHCP works correctly all the time, so ethernet side is probably ok.
> 
> When tcp_timestamps are enabled, SYN packets from dev board just get
> ignored by the remote side. I see them arrive in wireshark, but nothing
> else happens.
> 
> When I disable tcp_timestamps on the dev board everything works.
> 
> The problem is reproducible every single time.
> 
> The only difference is the "Options" block of the SYN packets.
> If timestamps are not really to blame, then it probably window scale
> parameters. That's what I see on a usual dropped packet:
> 
> Options: (20 bytes)
>         Maximum segment size: 1460 bytes
>         SACK permitted
>         Timestamps: TSval 4294893842, TSecr 0
>         NOP
>         Window scale: 5 (multiply by 32)
> 
> 
> 
>       


You dont give new informations ;)

I asked if you could give information on the other side : The bug is to
drop this legal packet.

uname -a
sysctl -a | grep tcp

Re: Bad TCP timestamps on non-PC platforms

[Alex Dubov]

> 
> 
> You dont give new informations ;)
> 
> I asked if you could give information on the other side :
> The bug is to
> drop this legal packet.
> 
> uname -a
> sysctl -a | grep tcp
> 

One of the other machines:

Linux sunapp 2.6.31.3 #2 SMP Mon Oct 12 21:32:20 EST 2009 x86_64 Dual-Core AMD Opteron(tm) Processor 8220 AuthenticAMD GNU/Linux

error: permission denied on key 'net.ipv4.route.flush'                          
fs.nfs.nlm_tcpport = 0
fs.nfs.nfs_callback_tcpport = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.tcp_syn_retries = 5
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_max_orphans = 65536
net.ipv4.tcp_max_tw_buckets = 180000
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_orphan_retries = 0
net.ipv4.tcp_fack = 1
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_mem = 3095040      4126720 6190080
net.ipv4.tcp_wmem = 4096        16384   4194304
net.ipv4.tcp_rmem = 4096        87380   4194304
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_adv_win_scale = 2
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_frto = 2
net.ipv4.tcp_frto_response = 0
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_congestion_control = reno
net.ipv4.tcp_abc = 0
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_base_mss = 512
net.ipv4.tcp_workaround_signed_windows = 0
net.ipv4.tcp_dma_copybreak = 4096
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_available_congestion_control = reno
net.ipv4.tcp_allowed_congestion_control = reno
net.ipv4.tcp_max_ssthresh = 0
error: permission denied on key 'net.ipv6.route.flush'
sunrpc.transports = tcp 1048576
sunrpc.tcp_slot_table_entries = 16
sunrpc.tcp_fin_timeout = 15




      

Re: Bad TCP timestamps on non-PC platforms

[David Miller]

From: Alex Dubov <oakad@yahoo.com>
Date: Wed, 5 Jan 2011 22:55:43 -0800 (PST)

> On embedded platforms, where real time clock is initialized lately or
> absent outright, this causes TSval field of outgoing TCP packets to be
> set to some garbage value, in my case in the vicinity of 0xffffffff.

The TCP timestamp is set from "jiffies", not from the RTC clock.