From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from list by lists.gnu.org with archive (Exim 4.71) id 1ZRlAG-0003v3-PZ for mharc-grub-devel@gnu.org; Tue, 18 Aug 2015 13:59:12 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41605) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZRlAD-0003t3-22 for grub-devel@gnu.org; Tue, 18 Aug 2015 13:59:10 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZRlA8-0000aq-Ub for grub-devel@gnu.org; Tue, 18 Aug 2015 13:59:08 -0400 Received: from mx0a-00082601.pphosted.com ([67.231.145.42]:4342) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZRlA8-0000aO-LJ for grub-devel@gnu.org; Tue, 18 Aug 2015 13:59:04 -0400 Received: from pps.filterd (m0004347 [127.0.0.1]) by m0004347.ppops.net (8.14.5/8.14.5) with SMTP id t7IHvDr1003308; Tue, 18 Aug 2015 10:59:00 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=fb.com; h=message-id : date : from : mime-version : to : cc : subject : references : in-reply-to : content-type : content-transfer-encoding; s=facebook; bh=sCMYPlhZi6itOksLSNCQZvOSnH4kyGxGBYovBy26m30=; b=O5pstsmpVxnI7Dtg2YaibKG8CntO5WrpJviDgK/ylaOhVLt5L8J6TyNhpTPE6qT5bezw qfIzhi9djOUfTmRDskEVt301tYiVif0c1NJyHd8Mliza4oqDbLojvlh9FtqOIYfoxkgC gFv4mEF7WhNgSG0v++Z+H4zaAc9WG8jlCbE= Received: from mail.thefacebook.com ([199.201.64.23]) by m0004347.ppops.net with ESMTP id 1wc8b4g73y-1 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Tue, 18 Aug 2015 10:59:00 -0700 Received: from localhost.localdomain (192.168.52.123) by mail.thefacebook.com (192.168.16.23) with Microsoft SMTP Server (TLS) id 14.3.195.1; Tue, 18 Aug 2015 10:58:57 -0700 Message-ID: <55D37261.9090907@fb.com> Date: Tue, 18 Aug 2015 10:58:57 -0700 From: Josef Bacik User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Andrei Borzenkov Subject: Re: [PATCH] tcp: ack when we get an OOO/lost packet References: <1439392582-3172342-1-git-send-email-jbacik@fb.com> <55CCA2D3.1050306@fb.com> In-Reply-To: Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [192.168.52.123] X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.14.151, 1.0.33, 0.0.0000 definitions=2015-08-18_09:2015-08-18, 2015-08-18, 1970-01-01 signatures=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x X-Received-From: 67.231.145.42 Cc: The development of GNU GRUB , kernel-team@fb.com X-BeenThere: grub-devel@gnu.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: The development of GNU GRUB List-Id: The development of GNU GRUB List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Aug 2015 17:59:10 -0000 On 08/17/2015 05:38 AM, Andrei Borzenkov wrote: > On Thu, Aug 13, 2015 at 4:59 PM, Josef Bacik wrote: >> On 08/13/2015 04:19 AM, Andrei Borzenkov wrote: >>> >>> On Wed, Aug 12, 2015 at 6:16 PM, Josef Bacik wrote: >>>> >>>> While adding tcp window scaling support I was finding that I'd get some >>>> packet >>>> loss or reordering when transferring from large distances and grub would >>>> just >>>> timeout. This is because we weren't ack'ing when we got our OOO packet, >>>> so the >>>> sender didn't know it needed to retransmit anything, so eventually it >>>> would fill >>>> the window and stop transmitting, and we'd time out. Fix this by ACK'ing >>>> when >>>> we don't find our next sequence numbered packet. With this fix I no >>>> longer time >>>> out. Thanks, >>> >>> >>> I have a feeling that your description is misleading. Patch simply >>> sends duplicated ACK, but partner does not know what has been received >>> and what has not, so it must wait for ACK timeout anyway before >>> retransmitting. What this patch may fix would be lost ACK packet >>> *from* GRUB, by increasing rate of ACK packets it sends. Do you have >>> packet trace for timeout case, ideally from both sides simultaneously? >>> >> >> The way linux works is that if you get of DUP ack's it >> triggers a retransmit. I only have traces from the server since tcpdump >> doesn't work in grub (or if it does I don't know how to do it). The server >> is definitely getting all of the ACK's, > (Sorry was traveling for Linux Plumbers.) > In packet trace you sent me there was almost certain ACK loss for the > segment 20801001- 20805881 (frame 19244). Note that here recovery was > rather fast - server started retransmission after ~0.5sec. It is > unlikely lost packet from server - next ACK from GRUB received by > server was for 20803441, which means it actually got at least initial > half of this segment. Unfortunately some packets are missing in > capture (even packets *from* server), which makes it harder to > interpret. After this server went down to 512 segment size and > everything went more or less well, until frame 19949. Here the server > behavior is rather interesting. It starts retransmission with initial > timeout ~6sec, even though it received quite a lot of DUP ACKs; and > doubling it every time until it hits GRUB timeout (~34 seconds). > Yeah that's the normal re-transmission timeout. This tcpdump was on a non-patched grub. We only sent 3 dup acks, we have the dup ack counter stuff set to like 13 or something like that so we have to get a lot before it triggers the dup ack retransmit logic. > Note the difference in behavior between the former and the latter. Did > you try to ask on Linux networking list why they are so different? > I'll run it by our networking guys when they show up. > OTOH GRUB probably times out too early. Initial TCP RFC suggests 5 > minutes general timeout and RFC1122 - at least 100 seconds. It would > be interesting to increase connection timeout to see if it recovers. > You could try bumping GRUB_NET_TRIES to 82 which result in timeout > slightly over 101 sec. > > Also it seems that huge window may aggravate the issue. According to > trace, 10K is enough to fill pipe and you set it to 1M. It would be > interesting to see the same with default windows size. > Oh yeah the problem doesn't happen with a normal window size, it's only with the giant window size. I'm not sure where you are getting the 10k number, believe me if I could have gotten around this by just jacking up the normal window size I would have done it. When I set it to the max (64k I think?) I get a transfer rate of around 200 kb/s, which is not fast enough to pull down our 250mb image. With the 1mb window I get 5.5 mb/s, so there is a real benefit to the giant window. Thanks, Josef