From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from list by lists.gnu.org with archive (Exim 4.71) id 1ZPwUM-0004tr-Ax for mharc-grub-devel@gnu.org; Thu, 13 Aug 2015 13:40:26 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41722) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZPwUJ-0004sb-0n for grub-devel@gnu.org; Thu, 13 Aug 2015 13:40:24 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZPwUF-0002eJ-O2 for grub-devel@gnu.org; Thu, 13 Aug 2015 13:40:22 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:4350 helo=mx0a-00082601.pphosted.com) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZPwUF-0002e9-IH for grub-devel@gnu.org; Thu, 13 Aug 2015 13:40:19 -0400 Received: from pps.filterd (m0004003 [127.0.0.1]) by mx0b-00082601.pphosted.com (8.14.5/8.14.5) with SMTP id t7DHdjER023727; Thu, 13 Aug 2015 10:40:18 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=fb.com; h=message-id : date : from : mime-version : to : cc : subject : references : in-reply-to : content-type : content-transfer-encoding; s=facebook; bh=qa0C6/CJ+qbk8HCfwFnCixdg5Owcaly9O6KVOjxoVlY=; b=dxbT/DJ+PtPRsU/YKiLPahui9f7rMIyXEP6W8xE9QX38zxjwBIcLa5FoKieQ7mMxS4t2 +RVFizEkXH3yBUqOsD8z80vBkkXSKRJYy9XxgFO7PCDqh69ELg3o9Lg3JMY/uvxtjKsn CLQ/YIq436jP44v6+GL76bX62BMkTTOcd5M= Received: from mail.thefacebook.com ([199.201.64.23]) by mx0b-00082601.pphosted.com with ESMTP id 1w8yehr3uq-1 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Thu, 13 Aug 2015 10:40:18 -0700 Received: from localhost.localdomain (192.168.52.123) by mail.TheFacebook.com (192.168.16.18) with Microsoft SMTP Server (TLS) id 14.3.195.1; Thu, 13 Aug 2015 10:40:15 -0700 Message-ID: <55CCD67D.6070401@fb.com> Date: Thu, 13 Aug 2015 13:40:13 -0400 From: Josef Bacik User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Andrei Borzenkov , The development of GNU GRUB Subject: Re: [PATCH] tcp: ack when we get an OOO/lost packet References: <1439392582-3172342-1-git-send-email-jbacik@fb.com> <55CCA2D3.1050306@fb.com> <55CCD037.4040106@gmail.com> In-Reply-To: <55CCD037.4040106@gmail.com> Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [192.168.52.123] X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.14.151, 1.0.33, 0.0.0000 definitions=2015-08-13_08:2015-08-13, 2015-08-13, 1970-01-01 signatures=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x X-Received-From: 67.231.153.30 Cc: kernel-team@fb.com X-BeenThere: grub-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: The development of GNU GRUB List-Id: The development of GNU GRUB List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2015 17:40:24 -0000 On 08/13/2015 01:13 PM, Andrei Borzenkov wrote: > > > On 13.08.2015 16:59, Josef Bacik wrote: >> On 08/13/2015 04:19 AM, Andrei Borzenkov wrote: >>> On Wed, Aug 12, 2015 at 6:16 PM, Josef Bacik wrote: >>>> While adding tcp window scaling support I was finding that I'd get >>>> some packet >>>> loss or reordering when transferring from large distances and grub >>>> would just >>>> timeout. This is because we weren't ack'ing when we got our OOO >>>> packet, so the >>>> sender didn't know it needed to retransmit anything, so eventually it >>>> would fill >>>> the window and stop transmitting, and we'd time out. Fix this by >>>> ACK'ing when >>>> we don't find our next sequence numbered packet. With this fix I no >>>> longer time >>>> out. Thanks, >>> >>> I have a feeling that your description is misleading. Patch simply >>> sends duplicated ACK, but partner does not know what has been received >>> and what has not, so it must wait for ACK timeout anyway before >>> retransmitting. What this patch may fix would be lost ACK packet >>> *from* GRUB, by increasing rate of ACK packets it sends. Do you have >>> packet trace for timeout case, ideally from both sides simultaneously? >>> >> >> The way linux works is that if you get of DUP >> ack's it triggers a retransmit. > > Do you have pointers to documentation and code? The tcp_reordering systctl allows you to set how many DUP acks you get before retransmitting, you can see the comment above the function tcp_time_to_recover in the kernel. With no SACK support we rely on getting a certain number of DUP ACKs before retransmitting, as we could get the out of order packets we want in time and not have to retransmit. > >> I only have traces from the server >> since tcpdump doesn't work in grub (or if it does I don't know how to do >> it). > > GRUB does not have tcpdump, but your switch quite likely has port > mirroring. > Big comapny, big datacenters etc, etc. I'm a file system developer, you are lucky I know how to spell tcpdump to begin with ;). The tcpdump on the server side supports my hypothesis, we send lots and lots of stuff, the grub box starts falling behind in it's ACK responses because it's waiting for the next SEQ packet to come in, it ACK's when it does finally come in with the new next expected SEQ, and this degrades to the point where the sender has maxed out its send window and the grub box either has lost or has yet to receive the next packet it is waiting for and times out. I can say for sure that we aren't getting the next packet we are looking for while getting a bunch of others just from my instrumentation on the grub side, I _can't_ say for sure if it is just simple re-ordering or packet loss somewhere. With this patch we're definitely getting all of the DUP ACK's, at least there doesn't appear to be any missing in the range (like I see DUP ACK #1-#300 all in a row, not missing anybody.) If you want I can change the commit log to say something like "If we get an out of order packet we still need to ACK with the expected SEQ number so the sender knows we haven't received that packet yet and may need a retransmission." To clear up any ambiguity. Thanks, Josef