From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Dumazet Subject: Re: TCP one-by-one acking - RFC interpretation question Date: Fri, 6 Apr 2018 09:49:58 -0700 Message-ID: References: <20180406100511.oify5ej2zvxr3isg@unicorn.suse.cz> <65f7ac65-e6d1-11f2-408e-39eb542ec817@gmail.com> <20180406150345.i3t7cu5g6dmoc3io@unicorn.suse.cz> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Cc: netdev@vger.kernel.org, Yuchung Cheng , Neal Cardwell To: Michal Kubecek , Eric Dumazet Return-path: Received: from mail-qt0-f178.google.com ([209.85.216.178]:45971 "EHLO mail-qt0-f178.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750962AbeDFQuC (ORCPT ); Fri, 6 Apr 2018 12:50:02 -0400 Received: by mail-qt0-f178.google.com with SMTP id f8so1789595qtg.12 for ; Fri, 06 Apr 2018 09:50:01 -0700 (PDT) In-Reply-To: <20180406150345.i3t7cu5g6dmoc3io@unicorn.suse.cz> Content-Language: en-US Sender: netdev-owner@vger.kernel.org List-ID: Cc Neal and Yuchung if they missed this thread. On 04/06/2018 08:03 AM, Michal Kubecek wrote: > On Fri, Apr 06, 2018 at 05:01:29AM -0700, Eric Dumazet wrote: >> >> >> On 04/06/2018 03:05 AM, Michal Kubecek wrote: >>> Hello >>> >>> I encountered a strange behaviour of some (non-linux) TCP stack which >>> I believe is incorrect but support engineers from the company producing >>> it claim is OK. >>> >>> Assume a client (sender, Linux 4.4 kernel) sends a stream of MSS sized >>> segments but segments 2, 4 and 6 do not reach the server (receiver): >>> >>> ACK SAK SAK SAK >>> +-------+-------+-------+-------+-------+-------+-------+ >>> | 1 | 2 | 3 | 4 | 5 | 6 | 7 | >>> +-------+-------+-------+-------+-------+-------+-------+ >>> 34273 35701 37129 38557 39985 41413 42841 44269 >>> >>> When segment 2 is retransmitted after RTO timeout, normal response would >>> be ACK-ing segment 3 (38557) with SACK for 5 and 7 (39985-41413 and >>> 42841-44269). >>> >>> However, this server stack responds with two separate ACKs: >>> >>> - ACK 37129, SACK 37129-38557 39985-41413 42841-44269 >>> - ACK 38557, SACK 39985-41413 42841-44269 >> >> Hmmm... Yes this seems very very wrong and lazy. >> >> Have you verified behavior of more recent linux kernel to such threats ? > > No, unfortunately the problem was only encountered by our customer in > production environment (they tried to reproduce in a test lab but no > luck). They are running backups to NFS server and it happens from time > to time (in the order of hours, IIUC). So it would be probably hard to > let them try with more recent kernel. > > On the other hand, they reported that SLE11 clients (kernel 3.0) do not > run into this kind of problem. It was originally reported as a > a regression on migration from SLE11-SP4 (3.0 kernel) to SLE12-SP2 (4.4 > kernel) and the problem was reported as "SLE12-SP2 is ignoring dupacks" > (which seems to be mostly caused by the switch to RACK). > > It also seems that part of the problem is specific packet loss pattern > where at some point, many packets are lost in "every second" pattern. > The customer finally started to investigate this problem and it seems it > has something to do with their bonding setup (they provided no details, > my guess is packets are divided over two paths and one of them fails). > >> packetdrill test would be relatively easy to write. > > I'll try but I have very little experience with writing packetdrill > scripts so it will probably take some time. > >> Regardless of this broken alien stack, we might be able to work around >> this faster than the vendor is able to fix and deploy a new stack. >> >> ( https://en.wikipedia.org/wiki/Robustness_principle ) >> Be conservative in what you do, be liberal in what you accept from >> others... > > I was thinking about this a bit. "Fixing" the acknowledgment number > could do the trick but it doesn't feel correct. We might use the fact > that TSecr of both ACKs above matches TSval of the retransmission which > triggered them so that RTT calculated from timestamp would be the right > one. So perhaps something like "prefer timestamp RTT if measured RTT > seems way too off". But I'm not sure if it couldn't break other use > cases where (high) measured RTT is actually correct, rather than (low) > timestamp RTT. > > Michal Kubecek >