* Fix FRTO+NewReno problem (Was: Re: This has a work around) [not found] ` <4821C37A.7040306@damtek.com> @ 2008-05-07 22:26 ` Ilpo Järvinen 2008-05-08 8:10 ` Fix FRTO+NewReno problem David Miller ` (3 more replies) 0 siblings, 4 replies; 24+ messages in thread From: Ilpo Järvinen @ 2008-05-07 22:26 UTC (permalink / raw) To: Damon L. Chesser; +Cc: Bug 213081, 478062, Netdev, David Miller [-- Attachment #1: Type: TEXT/PLAIN, Size: 8190 bytes --] Added Cc netdev (=linux network developers list). On Wed, 7 May 2008, Damon L. Chesser wrote: > Ilpo Järvinen wrote: > > > > So the problem was 100% reproducable for you with FRTO (I'm still catching > > up the details here :-))? > > > > Yes, 100% reproducible if you turn FRTO = 2. Printing works as it is > supposed to if FRTO = 0 > > > I suspect there's some corner case which might be buggy in kernel, but there > > are other possibilities currently as well. I'd like to get this fixed if > > it's a kernel bug, and think about work-arounds if it seems to be elsewhere > > (end-point or middlebox). It might be that those devices just blantantly > > discard any out-of-order segments, it could explain this kind of phenomena, > > but we'll soon see what's doing > > I also suspect a corner case. I suspect the hardware used has a "narrow > window" in which to allow the network traffic and the *new* kernels for some > reason are outside this window. However, just because FC8, Ubuntu 8.04 and > Debian all see the same thing does not rule out an issue with cupsys. No we're not sending past the window this time, that bug got resolved before 2.6.25 got released (and it wasn't in 2.6.24 at all)... :-) There were couple of interesting aspect in the tcpdump log: - The printer is using advertized window of 1024 whole the time => mss is set to 512 to allow two segments per rtt, we won't get a larger window at all. - The printer wasn't exactly blantantly discarding all received out-of-order segments, it correctly sent dupacks (most of the time), though the rate it's able to receive segments seems quite low, thus there were some losses as well (and therefore "missing" dupACKs too). - FRTOs occur during a previous recovery. After some code reading, I found the causing bug in the kernel: The printer won't negotiate SACK, yet F-RTO is using in couple of places a condition which lacks check for this making F-RTO to select wrongly SACK enabled behavior. ...There are two bugs actually with the same sympthoms, one for non-SACK case and the other for SACK-enabled case. ...The fix below is for non-SACK case. > > I suppose you're willing to test some kernel patch as well if that becomes > > necessary? > > Yup. I found it. I *own* it. This impacts a very small amount of people as > evidenced by the resounding silence of complaints about it (which is shy I > think the hardware has a "small window" to accept packets) I will have to > dust off my kernel patching skills, been a long long time since I had to patch > a kernel. > > Interesting: I have stopped the cups print job, I have reset the printer, > powered off the printerserver device, and re-seated the network cable > (required to clear the stuck tcp packets so that the other packets can get to > the printer), I have stopped cupsysd and netstat still shows an tcp connection > with 192.168.200.150 (printer). I include this info as this might be needed > by you. I suspect that that is not *normal* behavior, but I have never ran > netstat --tcp -c on a print job that failed, so I don't know: This is expected, once you had make the printer unreachabled, TCP will try a number of retransmissions without getting any response from printer before giving up, it would have just taken even longer time to get it cleaned up. The printer on the other hand will lose TCP states when you resetted it and therefore it's able to receive more stuff right away. > Ran the tcpdump for a solid 20 minutes, attached. Thanks, tcpdump was helpful. > Let me know if you need anything else. Could you next try with tcp_frto set to 1, if my theory proves to be correct, it too should be "enough" to fix the problem (in this particular case). Of course you can verify the patch below too if you want to, the patch should allow cups<->printer to work with tcp_frto = 2 too. In case you have problem to apply the patch to the particular version you're want to try with, just send a note about the version number to me so I can adapt the patch for you (space etc. formatting issues may show up because I recently run a code style cleanup on the tcp code). -- i. -- [PATCH] [TCP] FRTO: SACK variant is errorneously used with NewReno Note: there's actually another bug in FRTO's SACK variant, which is the causing failure in NewReno case because of the error that's fixed here. I'll fix the SACK case separately (it's a separate bug really, though related, but in order to fix that I need to audit tp->snd_nxt usage a bit). There were two places where SACK variant of FRTO is getting incorrectly used even if SACK wasn't negotiated by the TCP flow. This leads to incorrect setting of frto_highmark with NewReno if a previous recovery was interrupted by another RTO. An eventual fallback to conventional recovery then incorrectly considers one or couple of segments as forward transmissions though they weren't, which then are not LOST marked during fallback making them "non-retransmittable" until the next RTO. In a bad case, those segments are really lost and are the only one left in the window. Thus TCP needs another RTO to continue. The next FRTO, however, could again repeat the same events making the progress of the TCP flow extremely slow. In order for these events to occur at all, FRTO must occur again in FRTOs step 3 while the key segments must be lost as well, which is not too likely in practice. It seems to most frequently with some small devices such as network printers that *seem* to accept TCP segments only in-order. In cases were key segments weren't lost, things get automatically resolved because those wrongly marked segments don't need to be retransmitted in order to continue. I found a reproducer after digging up relevant reports (few reports in total, none at netdev or lkml I know of), some cases seemed to indicate middlebox issues which seems now to be a false assumption some people had made. Bugzilla #10063 _might_ be related. Damon L. Chesser <damon@damtek.com> had a reproducable case and was kind enough to tcpdump it for me. With the tcpdump log it was quite trivial to figure out. Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> --- net/ipv4/tcp_input.c | 12 +++++++----- 1 files changed, 7 insertions(+), 5 deletions(-) diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 0298f80..5c503e0 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -113,8 +113,6 @@ int sysctl_tcp_abc __read_mostly; #define FLAG_FORWARD_PROGRESS (FLAG_ACKED|FLAG_DATA_SACKED) #define FLAG_ANY_PROGRESS (FLAG_FORWARD_PROGRESS|FLAG_SND_UNA_ADVANCED) -#define IsSackFrto() (sysctl_tcp_frto == 0x2) - #define TCP_REMNANT (TCP_FLAG_FIN|TCP_FLAG_URG|TCP_FLAG_SYN|TCP_FLAG_PSH) #define TCP_HP_BITS (~(TCP_RESERVED_BITS|TCP_FLAG_PSH)) @@ -1685,6 +1683,10 @@ static inline void tcp_reset_reno_sack(struct tcp_sock *tp) tp->sacked_out = 0; } +static int tcp_is_sackfrto(const struct tcp_sock *tp) { + return (sysctl_tcp_frto == 0x2) && !tcp_is_reno(tp); +} + /* F-RTO can only be used if TCP has never retransmitted anything other than * head (SACK enhanced variant from Appendix B of RFC4138 is more robust here) */ @@ -1701,7 +1703,7 @@ int tcp_use_frto(struct sock *sk) if (icsk->icsk_mtup.probe_size) return 0; - if (IsSackFrto()) + if (tcp_is_sackfrto(tp)) return 1; /* Avoid expensive walking of rexmit queue if possible */ @@ -1791,7 +1793,7 @@ void tcp_enter_frto(struct sock *sk) /* Earlier loss recovery underway (see RFC4138; Appendix B). * The last condition is necessary at least in tp->frto_counter case. */ - if (IsSackFrto() && (tp->frto_counter || + if (tcp_is_sackfrto(tp) && (tp->frto_counter || ((1 << icsk->icsk_ca_state) & (TCPF_CA_Recovery|TCPF_CA_Loss))) && after(tp->high_seq, tp->snd_una)) { tp->frto_highmark = tp->high_seq; @@ -3123,7 +3125,7 @@ static int tcp_process_frto(struct sock *sk, int flag) return 1; } - if (!IsSackFrto() || tcp_is_reno(tp)) { + if (!tcp_is_sackfrto(tp)) { /* RFC4138 shortcoming in step 2; should also have case c): * ACK isn't duplicate nor advances window, e.g., opposite dir * data, winupdate -- 1.5.2.2 ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem 2008-05-07 22:26 ` Fix FRTO+NewReno problem (Was: Re: This has a work around) Ilpo Järvinen @ 2008-05-08 8:10 ` David Miller 2008-05-08 16:50 ` Bug#478062: Fix FRTO+NewReno problem (Was: Re: This has a work around) Damon L. Chesser ` (2 subsequent siblings) 3 siblings, 0 replies; 24+ messages in thread From: David Miller @ 2008-05-08 8:10 UTC (permalink / raw) To: ilpo.jarvinen; +Cc: damon, 213081, 478062, netdev From: "Ilpo_Järvinen" <ilpo.jarvinen@helsinki.fi> Date: Thu, 8 May 2008 01:26:59 +0300 (EEST) > [PATCH] [TCP] FRTO: SACK variant is errorneously used with NewReno I applied this with a minor coding style fixup. From: "Ilpo_Järvinen" <ilpo.jarvinen@helsinki.fi> Date: Thu, 8 May 2008 01:26:59 +0300 (EEST) > +static int tcp_is_sackfrto(const struct tcp_sock *tp) { > + return (sysctl_tcp_frto == 0x2) && !tcp_is_reno(tp); > +} > + Should be: static int tcp_is_sackfrto(const struct tcp_sock *tp) { return (sysctl_tcp_frto == 0x2) && !tcp_is_reno(tp); } I will also queue this up to -stable, thanks so much for this bug fix! ^ permalink raw reply [flat|nested] 24+ messages in thread
* Bug#478062: Fix FRTO+NewReno problem (Was: Re: This has a work around) 2008-05-07 22:26 ` Fix FRTO+NewReno problem (Was: Re: This has a work around) Ilpo Järvinen 2008-05-08 8:10 ` Fix FRTO+NewReno problem David Miller @ 2008-05-08 16:50 ` Damon L. Chesser 2008-05-08 17:05 ` Damon L. Chesser 2008-05-08 18:16 ` Damon L. Chesser 3 siblings, 0 replies; 24+ messages in thread From: Damon L. Chesser @ 2008-05-08 16:50 UTC (permalink / raw) To: Ilpo Järvinen; +Cc: Bug 213081, 478062, Netdev, David Miller [-- Attachment #1: Type: text/plain, Size: 925 bytes --] Ilpo Järvinen wrote: Snip > > Could you next try with tcp_frto set to 1, if my theory proves to be > correct, it too should be "enough" to fix the problem (in this > particular case). Of course you can verify the patch below too if you > want to, the patch should allow cups<->printer to work with tcp_frto = > 2 too. In case you have problem to apply the patch to the particular > version you're want to try with, just send a note about the version > number to me so I can adapt the patch for you (space etc. formatting > issues may show up because I recently run a code style cleanup on the > tcp code). > Ilpo, I tried the tcp_frto = 1 and got the same results as = 2. Attached is the output of the tcpdump for frto=1. I might not get to the patch today as I am feeling a bit slow. Thanks for the work! -- Damon L. Chesser damon@damtek.com http://www.linkedin.com/in/dchesser [-- Attachment #2: frtoprob1.txt --] [-- Type: text/plain, Size: 13322 bytes --] 1210263858.691448 IP (tos 0x0, ttl 64, id 59590, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: . 390384065:390384577(512) ack 27168769 win 5840 1210263858.695038 IP (tos 0x0, ttl 100, id 75, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x7df5 (correct), 1:1(0) ack 512 win 1024 1210263858.695231 IP (tos 0x0, ttl 64, id 59591, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: . 1024:1536(512) ack 1 win 5840 1210263859.394452 IP (tos 0x0, ttl 100, id 76, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x7df5 (correct), 1:1(0) ack 512 win 1024 1210263859.394517 IP (tos 0x0, ttl 64, id 59592, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: P 512:1024(512) ack 1 win 5840 1210263859.400855 IP (tos 0x0, ttl 100, id 77, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x7bf5 (correct), 1:1(0) ack 1024 win 1024 1210263859.400880 IP (tos 0x0, ttl 64, id 59593, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: P 1536:2048(512) ack 1 win 5840 1210263860.096485 IP (tos 0x0, ttl 100, id 78, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x7bf5 (correct), 1:1(0) ack 1024 win 1024 1210263979.417094 IP (tos 0x0, ttl 64, id 59594, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: . 1024:1536(512) ack 1 win 5840 1210263979.421796 IP (tos 0x0, ttl 100, id 82, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x79f5 (correct), 1:1(0) ack 1536 win 1024 1210263979.421850 IP (tos 0x0, ttl 64, id 59595, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: . 2048:2560(512) ack 1 win 5840 1210263980.121475 IP (tos 0x0, ttl 100, id 83, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x79f5 (correct), 1:1(0) ack 1536 win 1024 1210263980.121525 IP (tos 0x0, ttl 64, id 59596, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: P 1536:2048(512) ack 1 win 5840 1210263980.127898 IP (tos 0x0, ttl 100, id 84, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x77f5 (correct), 1:1(0) ack 2048 win 1024 1210263980.127915 IP (tos 0x0, ttl 64, id 59597, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: P 2560:3072(512) ack 1 win 5840 1210263980.822754 IP (tos 0x0, ttl 100, id 85, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x77f5 (correct), 1:1(0) ack 2048 win 1024 1210264100.142724 IP (tos 0x0, ttl 64, id 59598, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: . 2048:2560(512) ack 1 win 5840 1210264100.148733 IP (tos 0x0, ttl 100, id 92, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x75f5 (correct), 1:1(0) ack 2560 win 1024 1210264100.148766 IP (tos 0x0, ttl 64, id 59599, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: . 3072:3584(512) ack 1 win 5840 1210264100.843579 IP (tos 0x0, ttl 100, id 93, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x75f5 (correct), 1:1(0) ack 2560 win 1024 1210264100.843629 IP (tos 0x0, ttl 64, id 59600, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: P 2560:3072(512) ack 1 win 5840 1210264100.850144 IP (tos 0x0, ttl 100, id 94, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x73f5 (correct), 1:1(0) ack 3072 win 1024 1210264100.850161 IP (tos 0x0, ttl 64, id 59601, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: P 3584:4096(512) ack 1 win 5840 1210264101.549783 IP (tos 0x0, ttl 100, id 95, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x73f5 (correct), 1:1(0) ack 3072 win 1024 1210264220.864362 IP (tos 0x0, ttl 64, id 59602, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: . 3072:3584(512) ack 1 win 5840 1210264220.870832 IP (tos 0x0, ttl 100, id 99, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x71f5 (correct), 1:1(0) ack 3584 win 1024 1210264220.870874 IP (tos 0x0, ttl 64, id 59603, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: . 4096:4608(512) ack 1 win 5840 1210264221.570637 IP (tos 0x0, ttl 100, id 100, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x71f5 (correct), 1:1(0) ack 3584 win 1024 1210264221.570684 IP (tos 0x0, ttl 64, id 59604, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: P 3584:4096(512) ack 1 win 5840 1210264221.577194 IP (tos 0x0, ttl 100, id 101, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x6ff5 (correct), 1:1(0) ack 4096 win 1024 1210264221.577211 IP (tos 0x0, ttl 64, id 59605, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: P 4608:5120(512) ack 1 win 5840 1210264222.271894 IP (tos 0x0, ttl 100, id 102, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x6ff5 (correct), 1:1(0) ack 4096 win 1024 1210264341.590019 IP (tos 0x0, ttl 64, id 59606, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: . 4096:4608(512) ack 1 win 5840 1210264341.593578 IP (tos 0x0, ttl 100, id 106, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x6df5 (correct), 1:1(0) ack 4608 win 1024 1210264341.593627 IP (tos 0x0, ttl 64, id 59607, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: . 5120:5632(512) ack 1 win 5840 1210264342.292717 IP (tos 0x0, ttl 100, id 110, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x6df5 (correct), 1:1(0) ack 4608 win 1024 1210264342.292767 IP (tos 0x0, ttl 64, id 59608, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: P 4608:5120(512) ack 1 win 5840 1210264342.299168 IP (tos 0x0, ttl 100, id 111, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x6bf5 (correct), 1:1(0) ack 5120 win 1024 1210264342.299188 IP (tos 0x0, ttl 64, id 59609, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: P 5632:6144(512) ack 1 win 5840 1210264342.999071 IP (tos 0x0, ttl 100, id 112, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x6bf5 (correct), 1:1(0) ack 5120 win 1024 1210264462.315638 IP (tos 0x0, ttl 64, id 59610, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: . 5120:5632(512) ack 1 win 5840 1210264462.320063 IP (tos 0x0, ttl 100, id 116, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x69f5 (correct), 1:1(0) ack 5632 win 1024 1210264462.320099 IP (tos 0x0, ttl 64, id 59611, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: . 6144:6656(512) ack 1 win 5840 1210264463.019784 IP (tos 0x0, ttl 100, id 117, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x69f5 (correct), 1:1(0) ack 5632 win 1024 1210264463.019848 IP (tos 0x0, ttl 64, id 59612, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: P 5632:6144(512) ack 1 win 5840 1210264463.026210 IP (tos 0x0, ttl 100, id 118, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x67f5 (correct), 1:1(0) ack 6144 win 1024 1210264463.026256 IP (tos 0x0, ttl 64, id 59613, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: P 6656:7168(512) ack 1 win 5840 1210264463.721025 IP (tos 0x0, ttl 100, id 119, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x67f5 (correct), 1:1(0) ack 6144 win 1024 1210264583.041283 IP (tos 0x0, ttl 64, id 59614, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: . 6144:6656(512) ack 1 win 5840 1210264583.047127 IP (tos 0x0, ttl 100, id 123, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x65f5 (correct), 1:1(0) ack 6656 win 1024 1210264583.047166 IP (tos 0x0, ttl 64, id 59615, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: . 7168:7680(512) ack 1 win 5840 1210264583.741856 IP (tos 0x0, ttl 100, id 124, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x65f5 (correct), 1:1(0) ack 6656 win 1024 1210264583.741920 IP (tos 0x0, ttl 64, id 59616, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: P 6656:7168(512) ack 1 win 5840 1210264583.748296 IP (tos 0x0, ttl 100, id 125, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x63f5 (correct), 1:1(0) ack 7168 win 1024 1210264583.748351 IP (tos 0x0, ttl 64, id 59617, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: P 7680:8192(512) ack 1 win 5840 1210264584.443149 IP (tos 0x0, ttl 100, id 126, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x63f5 (correct), 1:1(0) ack 7168 win 1024 1210264703.766913 IP (tos 0x0, ttl 64, id 59618, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: . 7168:7680(512) ack 1 win 5840 1210264703.770428 IP (tos 0x0, ttl 100, id 133, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x61f5 (correct), 1:1(0) ack 7680 win 1024 1210264703.770462 IP (tos 0x0, ttl 64, id 59619, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: . 8192:8704(512) ack 1 win 5840 1210264704.469029 IP (tos 0x0, ttl 100, id 134, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x61f5 (correct), 1:1(0) ack 7680 win 1024 1210264704.469076 IP (tos 0x0, ttl 64, id 59620, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: P 7680:8192(512) ack 1 win 5840 1210264704.475326 IP (tos 0x0, ttl 100, id 135, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x5ff5 (correct), 1:1(0) ack 8192 win 1024 1210264704.475346 IP (tos 0x0, ttl 64, id 59621, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: P 8704:9216(512) ack 1 win 5840 1210264705.170173 IP (tos 0x0, ttl 100, id 136, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x5ff5 (correct), 1:1(0) ack 8192 win 1024 1210264824.492552 IP (tos 0x0, ttl 64, id 59622, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: . 8192:8704(512) ack 1 win 5840 1210264824.496167 IP (tos 0x0, ttl 100, id 140, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x5df5 (correct), 1:1(0) ack 8704 win 1024 1210264824.496197 IP (tos 0x0, ttl 64, id 59623, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: . 9216:9728(512) ack 1 win 5840 1210264825.191007 IP (tos 0x0, ttl 100, id 141, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x5df5 (correct), 1:1(0) ack 8704 win 1024 1210264825.191069 IP (tos 0x0, ttl 64, id 59624, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: P 8704:9216(512) ack 1 win 5840 1210264825.197574 IP (tos 0x0, ttl 100, id 142, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x5bf5 (correct), 1:1(0) ack 9216 win 1024 1210264825.197623 IP (tos 0x0, ttl 64, id 59625, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.33718 > 192.168.200.150.9100: P 9728:10240(512) ack 1 win 5840 1210264825.897222 IP (tos 0x0, ttl 100, id 143, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.33718: ., cksum 0x5bf5 (correct), 1:1(0) ack 9216 win 1024 ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem (Was: Re: This has a work around) 2008-05-07 22:26 ` Fix FRTO+NewReno problem (Was: Re: This has a work around) Ilpo Järvinen 2008-05-08 8:10 ` Fix FRTO+NewReno problem David Miller 2008-05-08 16:50 ` Bug#478062: Fix FRTO+NewReno problem (Was: Re: This has a work around) Damon L. Chesser @ 2008-05-08 17:05 ` Damon L. Chesser 2008-05-08 18:16 ` Damon L. Chesser 3 siblings, 0 replies; 24+ messages in thread From: Damon L. Chesser @ 2008-05-08 17:05 UTC (permalink / raw) To: Ilpo Järvinen; +Cc: Bug 213081, 478062, Netdev, David Miller Ilpo Järvinen wrote: Snip > > Could you next try with tcp_frto set to 1, if my theory proves to be > correct, it too should be "enough" to fix the problem (in this > particular case). Of course you can verify the patch below too if you > want to, the patch should allow cups<->printer to work with tcp_frto = > 2 too. In case you have problem to apply the patch to the particular > version you're want to try with, just send a note about the version > number to me so I can adapt the patch for you (space etc. formatting > issues may show up because I recently run a code style cleanup on the > tcp code). > Ilpo, and all others, please ignore frtoprob1.txt. I ran the test on the wrong kernel. The tested kernel was 2.6.23.17-amd64 and should have been 2.6.24-1-amd64. I am re-running the test and will forward the new dump. No idea why 2.6.23.17 did not print, it always did before. I am sure I did something stupid, feeling a bit under the weather. Sorry for the confusion. New test comming shortly. -- Damon L. Chesser damon@damtek.com http://www.linkedin.com/in/dchesser ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem (Was: Re: This has a work around) 2008-05-07 22:26 ` Fix FRTO+NewReno problem (Was: Re: This has a work around) Ilpo Järvinen ` (2 preceding siblings ...) 2008-05-08 17:05 ` Damon L. Chesser @ 2008-05-08 18:16 ` Damon L. Chesser 2008-05-08 20:42 ` Ilpo Järvinen 3 siblings, 1 reply; 24+ messages in thread From: Damon L. Chesser @ 2008-05-08 18:16 UTC (permalink / raw) To: Ilpo Järvinen; +Cc: Bug 213081, 478062, Netdev, David Miller [-- Attachment #1: Type: text/plain, Size: 932 bytes --] Ilpo Järvinen wrote: > SNIP > > Could you next try with tcp_frto set to 1, if my theory proves to be > correct, it too should be "enough" to fix the problem (in this > particular case). Of course you can verify the patch below too if you > want to, the patch should allow cups<->printer to work with tcp_frto = > 2 too. In case you have problem to apply the patch to the particular > version you're want to try with, just send a note about the version > number to me so I can adapt the patch for you (space etc. formatting > issues may show up because I recently run a code style cleanup on the > tcp code). Ilpo, reran the print job with the correct kernel (for control reasons) and received the same results: tcp_frto=1 no print. tcp_frto=0 I can print. Attached is the output of tcpdump uname -r = 2.6.24-1-amd64 Thanks for the work! -- Damon L. Chesser damon@damtek.com http://www.linkedin.com/in/dchesser [-- Attachment #2: frtoprob2.txt --] [-- Type: text/plain, Size: 4445 bytes --] 1210269685.442292 IP (tos 0x0, ttl 64, id 65247, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.53527 > 192.168.200.150.9100: P 1730585386:1730585898(512) ack 129179649 win 5840 1210269685.446179 IP (tos 0x0, ttl 100, id 22, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.53527: ., cksum 0x7134 (correct), 1:1(0) ack 512 win 1024 1210269685.446221 IP (tos 0x0, ttl 64, id 65248, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.53527 > 192.168.200.150.9100: P 1024:1536(512) ack 1 win 5840 1210269686.140866 IP (tos 0x0, ttl 100, id 23, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.53527: ., cksum 0x7134 (correct), 1:1(0) ack 512 win 1024 1210269714.626282 IP (tos 0x0, ttl 64, id 65249, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.53527 > 192.168.200.150.9100: . 512:1024(512) ack 1 win 5840 1210269714.632392 IP (tos 0x0, ttl 100, id 24, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.53527: ., cksum 0x6f34 (correct), 1:1(0) ack 1024 win 1024 1210269714.632425 IP (tos 0x0, ttl 64, id 65250, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.53527 > 192.168.200.150.9100: . 1536:2048(512) ack 1 win 5840 1210269715.327141 IP (tos 0x0, ttl 100, id 25, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.53527: ., cksum 0x6f34 (correct), 1:1(0) ack 1024 win 1024 1210269772.998283 IP (tos 0x0, ttl 64, id 65251, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.53527 > 192.168.200.150.9100: P 1024:1536(512) ack 1 win 5840 1210269773.005111 IP (tos 0x0, ttl 100, id 29, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.53527: ., cksum 0x6d34 (correct), 1:1(0) ack 1536 win 1024 1210269773.005144 IP (tos 0x0, ttl 64, id 65252, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.53527 > 192.168.200.150.9100: P 2048:2560(512) ack 1 win 5840 1210269773.704682 IP (tos 0x0, ttl 100, id 30, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.53527: ., cksum 0x6d34 (correct), 1:1(0) ack 1536 win 1024 1210269889.738282 IP (tos 0x0, ttl 64, id 65253, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.53527 > 192.168.200.150.9100: . 1536:2048(512) ack 1 win 5840 1210269889.745981 IP (tos 0x0, ttl 100, id 37, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.53527: ., cksum 0x6b34 (correct), 1:1(0) ack 2048 win 1024 1210269889.746012 IP (tos 0x0, ttl 64, id 65254, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.53527 > 192.168.200.150.9100: . 2560:3072(512) ack 1 win 5840 1210269890.444886 IP (tos 0x0, ttl 100, id 38, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.53527: ., cksum 0x6b34 (correct), 1:1(0) ack 2048 win 1024 1210270009.746305 IP (tos 0x0, ttl 64, id 65255, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.53527 > 192.168.200.150.9100: P 2048:2560(512) ack 1 win 5840 1210270009.752297 IP (tos 0x0, ttl 100, id 42, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.53527: ., cksum 0x6934 (correct), 1:1(0) ack 2560 win 1024 1210270009.752335 IP (tos 0x0, ttl 64, id 65256, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.53527 > 192.168.200.150.9100: P 3072:3584(512) ack 1 win 5840 1210270010.449445 IP (tos 0x0, ttl 100, id 43, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.53527: ., cksum 0x6934 (correct), 1:1(0) ack 2560 win 1024 1210270129.749294 IP (tos 0x0, ttl 64, id 65257, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.53527 > 192.168.200.150.9100: . 2560:3072(512) ack 1 win 5840 1210270129.755196 IP (tos 0x0, ttl 100, id 47, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.53527: ., cksum 0x6734 (correct), 1:1(0) ack 3072 win 1024 1210270129.755227 IP (tos 0x0, ttl 64, id 65258, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.53527 > 192.168.200.150.9100: . 3584:4096(512) ack 1 win 5840 1210270130.453997 IP (tos 0x0, ttl 100, id 48, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.53527: ., cksum 0x6734 (correct), 1:1(0) ack 3072 win 1024 ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem (Was: Re: This has a work around) 2008-05-08 18:16 ` Damon L. Chesser @ 2008-05-08 20:42 ` Ilpo Järvinen 2008-05-12 10:08 ` Ilpo Järvinen 0 siblings, 1 reply; 24+ messages in thread From: Ilpo Järvinen @ 2008-05-08 20:42 UTC (permalink / raw) To: Damon L. Chesser; +Cc: Bug 213081, 478062, Netdev, David Miller [-- Attachment #1: Type: TEXT/PLAIN, Size: 1370 bytes --] On Thu, 8 May 2008, Damon L. Chesser wrote: > Ilpo Järvinen wrote: > > SNIP > > > > > Could you next try with tcp_frto set to 1, if my theory proves to be > > correct, it too should be "enough" to fix the problem (in this particular > > case). Of course you can verify the patch below too if you want to, the > > patch should allow cups<->printer to work with tcp_frto = 2 too. In case you > > have problem to apply the patch to the particular version you're want to try > > with, just send a note about the version number to me so I can adapt the > > patch for you (space etc. formatting issues may show up because I recently > > run a code style cleanup on the tcp code). > > reran the print job with the correct kernel (for control reasons) and received > the same results: tcp_frto=1 no print. tcp_frto=0 I can print. Attached is > the output of tcpdump > > uname -r = 2.6.24-1-amd64 Well, that was a surprise, there must be something else too I didn't yet notice. I don't think it's that necessary for you to test that patch I sent earlier (basically the code paths it would have fixed were already in use with tcp_frto=1). And that patch was "obviously correct" anyway though it wasn't enough to fix this issue. ...I too can probably reproduce this locally with small amount of work because the receiver pattern is dead obvious from the logs. -- i. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem (Was: Re: This has a work around) 2008-05-08 20:42 ` Ilpo Järvinen @ 2008-05-12 10:08 ` Ilpo Järvinen [not found] ` <4828279C.3010102@damtek.com> 0 siblings, 1 reply; 24+ messages in thread From: Ilpo Järvinen @ 2008-05-12 10:08 UTC (permalink / raw) To: Damon L. Chesser, David Miller; +Cc: Bug 213081, 478062, Netdev [-- Attachment #1: Type: TEXT/PLAIN, Size: 3694 bytes --] On Thu, 8 May 2008, Ilpo Järvinen wrote: > On Thu, 8 May 2008, Damon L. Chesser wrote: > > > reran the print job with the correct kernel (for control reasons) and > > received > > the same results: tcp_frto=1 no print. tcp_frto=0 I can print. Attached > > is > > the output of tcpdump > > > > uname -r = 2.6.24-1-amd64 > > Well, that was a surprise, there must be something else too I didn't yet > notice. I don't think it's that necessary for you to test that patch I sent > earlier (basically the code paths it would have fixed were already in use with > tcp_frto=1). And that patch was "obviously correct" anyway though it wasn't > enough to fix this issue. > > ...I too can probably reproduce this locally with small amount of work because > the receiver pattern is dead obvious from the logs. Yes indeed, some hping3 tcl acting as a clone of that network printer did it :-). Below is the 2nd patch (both are necessary). Besides them there's still SACKFRTO snd_nxt != frto_highmark problem remaining but it is a lot less severe and rare than this problem was and I'm still trying to find a simple way to fix it w/o adding another u32 to tcp_sock. I may need to think this retrans_stamp usage more around the rest of TCP code too as it seems to be somewhat suspicious here and there. -- i. ps. ...you could have at least considered reporting upstream a bit earlier if some problem goes away/appears by changing kernel version (especially since you already tried some non-distro kernels and found them non-working), it might help to catch devs attention who hardly hang much around distro bug trackers :-). -- [PATCH] [TCP] FRTO: Fix fallback to conventional recovery It seems that commit 009a2e3e4ec ("[TCP] FRTO: Improve interoperability with other undo_marker users") run into another land-mine which caused fallback to conventional recovery to break: 1. Cumulative ACK arrives after FRTO retransmission 2. tcp_try_to_open sees zero retrans_out, clears retrans_stamp which should be kept like in CA_Loss state it would be 3. undo_marker change allowed tcp_packet_delayed to return true because of the cleared retrans_stamp once FRTO is terminated causing LossUndo to occur, which means all loss markings FRTO made are reverted. This means that the conventional recovery basically recovered one loss per RTO, which is not that efficient. It becomes a serious problem to progress of the flow if many segments were lost or when losses will persist to the FRTO RTTs as well. Retrans_stamp was incorrectly cleared even before that particular change (though it's effect is not often significant). It was quite unobvious that the undo_marker change broken something like this, I had a quite long session to track it down because of the non-intuitiviness of the bug (luckily I had a trivial reproducer at hand and I was also able to learn to use kprobes in the process as well :-)). This together with the NewReno+FRTO fix (62ab22278308a) should finally fix Damon's problems. Compile tested (but I did experiment with a similar fix on a live kernel with systemtap+kprobes). Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> Reported-by: Damon L. Chesser <damon@damtek.com> --- net/ipv4/tcp_input.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 81ece1f..4c2255c 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -2481,7 +2481,7 @@ static void tcp_try_to_open(struct sock *sk, int flag) tcp_verify_left_out(tp); - if (tp->retrans_out == 0) + if (!tp->frto_counter && tp->retrans_out == 0) tp->retrans_stamp = 0; if (flag & FLAG_ECE) -- 1.5.2.2 ^ permalink raw reply related [flat|nested] 24+ messages in thread
[parent not found: <4828279C.3010102@damtek.com>]
* Re: Fix FRTO+NewReno problem (Was: Re: This has a work around) [not found] ` <4828279C.3010102@damtek.com> @ 2008-05-12 11:32 ` Ilpo Järvinen 2008-05-12 11:55 ` Damon L. Chesser 0 siblings, 1 reply; 24+ messages in thread From: Ilpo Järvinen @ 2008-05-12 11:32 UTC (permalink / raw) To: Damon L. Chesser; +Cc: Netdev, David Miller [-- Attachment #1: Type: TEXT/PLAIN, Size: 2467 bytes --] On Mon, 12 May 2008, Damon L. Chesser wrote: > Ilpo Järvinen wrote: > > On Thu, 8 May 2008, Ilpo Järvinen wrote: > > > > > On Thu, 8 May 2008, Damon L. Chesser wrote: > > > > > > > reran the print job with the correct kernel (for control reasons) and > > > > received > > > > the same results: tcp_frto=1 no print. tcp_frto=0 I can print. > > > Attached > > > > is > > > > the output of tcpdump > > > > > uname -r = 2.6.24-1-amd64 > > > > > > Well, that was a surprise, there must be something else too I didn't yet > > > notice. I don't think it's that necessary for you to test that patch I > > > sent > > > earlier (basically the code paths it would have fixed were already in use > > > with > > > tcp_frto=1). And that patch was "obviously correct" anyway though it > > > wasn't > > > enough to fix this issue. > > > > > > ...I too can probably reproduce this locally with small amount of work > > > because > > > the receiver pattern is dead obvious from the logs. > > > > Yes indeed, some hping3 tcl acting as a clone of that network printer did it > > :-). Below is the 2nd patch (both are necessary). Besides them there's still > > SACKFRTO snd_nxt != frto_highmark problem remaining but it is a lot less > > severe and rare than this problem was and I'm still trying to find a simple > > way to fix it w/o adding another u32 to tcp_sock. I may need to think this > > retrans_stamp usage more around the rest of TCP code too as it seems to be > > somewhat suspicious here and there. > > > on the chance I am being dense: apply this and the other one with the patch > command, then compile as normal to taste? Yes, both are necessary to fully fix it. I'd cut unrelated portions out first though to avoid potential problems before running patch. And if you have a linux git tree instead, then git-am could be used as "a replacement" for patch command (though it would work there too) after trimming away things that are before the patch description (before [PATCH] line). ...Then a normal compile & boot & test should make you a happy person :-). If you were quick enough (ie., before DaveM does apply the patch upstream) and bother to report the result back to netdev without dropping Ccs, Dave will add Tested-by: tag with your name in it into the permanent commit message which is cool when somebody starts to figure out based on those tags who has done testing work for kernel... ;-) ...Thanks for your efforts with this problem. -- i. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem (Was: Re: This has a work around) 2008-05-12 11:32 ` Ilpo Järvinen @ 2008-05-12 11:55 ` Damon L. Chesser 2008-05-12 12:07 ` Ilpo Järvinen 0 siblings, 1 reply; 24+ messages in thread From: Damon L. Chesser @ 2008-05-12 11:55 UTC (permalink / raw) To: Ilpo Järvinen; +Cc: Netdev, David Miller Ilpo Järvinen wrote: > Big snip Ilpo, I ran the first patch and received this error: root@dam-main:/usr/src/linux-2.6.24.1# patch -p1 < ../1st_frto_patch.diff patching file net/ipv4/tcp_input.c patch: **** malformed patch at line 17: @@ -1685,6 +1683,10 @@ static inline void tcp_reset_reno_sack(struct tcp_sock *tp) root@dam-main:/usr/src/linux-2.6.24.1# below is the text of the patch.diff I have: net/ipv4/tcp_input.c | 12 +++++++----- 1 files changed, 7 insertions(+), 5 deletions(-) diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 0298f80..5c503e0 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -113,8 +113,6 @@ int sysctl_tcp_abc __read_mostly; #define FLAG_FORWARD_PROGRESS (FLAG_ACKED|FLAG_DATA_SACKED) #define FLAG_ANY_PROGRESS (FLAG_FORWARD_PROGRESS|FLAG_SND_UNA_ADVANCED) -#define IsSackFrto() (sysctl_tcp_frto == 0x2) - #define TCP_REMNANT (TCP_FLAG_FIN|TCP_FLAG_URG|TCP_FLAG_SYN|TCP_FLAG_PSH) #define TCP_HP_BITS (~(TCP_RESERVED_BITS|TCP_FLAG_PSH)) @@ -1685,6 +1683,10 @@ static inline void tcp_reset_reno_sack(struct tcp_sock *tp) tp->sacked_out = 0; } static int tcp_is_sackfrto(const struct tcp_sock *tp) { return (sysctl_tcp_frto == 0x2) && !tcp_is_reno(tp); } /* F-RTO can only be used if TCP has never retransmitted anything other than * head (SACK enhanced variant from Appendix B of RFC4138 is more robust here) */ @@ -1701,7 +1703,7 @@ int tcp_use_frto(struct sock *sk) if (icsk->icsk_mtup.probe_size) return 0; - if (IsSackFrto()) + if (tcp_is_sackfrto(tp)) return 1; /* Avoid expensive walking of rexmit queue if possible */ @@ -1791,7 +1793,7 @@ void tcp_enter_frto(struct sock *sk) /* Earlier loss recovery underway (see RFC4138; Appendix B). * The last condition is necessary at least in tp->frto_counter case. */ - if (IsSackFrto() && (tp->frto_counter || + if (tcp_is_sackfrto(tp) && (tp->frto_counter || ((1 << icsk->icsk_ca_state) & (TCPF_CA_Recovery|TCPF_CA_Loss))) && after(tp->high_seq, tp->snd_una)) { tp->frto_highmark = tp->high_seq; @@ -3123,7 +3125,7 @@ static int tcp_process_frto(struct sock *sk, int flag) return 1; } - if (!IsSackFrto() || tcp_is_reno(tp)) { + if (!tcp_is_sackfrto(tp)) { /* RFC4138 shortcoming in step 2; should also have case c): * ACK isn't duplicate nor advances window, e.g., opposite dir * data, winupdate -- 1.5.2.2 ---------------- and in the name of completeness this is the patch file I have for the 2nd patch (in the event I did not copy all the correct code): diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 81ece1f..4c2255c 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -2481,7 +2481,7 @@ static void tcp_try_to_open(struct sock *sk, int flag) tcp_verify_left_out(tp); - if (tp->retrans_out == 0) + if (!tp->frto_counter && tp->retrans_out == 0) tp->retrans_stamp = 0; if (flag & FLAG_ECE) -- 1.5.2.2 -------------------------- Am I doing something wrong or have I missed something? Patched against a 2.6.24.1 kernel from kernel.org -- Damon L. Chesser damon@damtek.com http://www.linkedin.com/in/dchesser ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem (Was: Re: This has a work around) 2008-05-12 11:55 ` Damon L. Chesser @ 2008-05-12 12:07 ` Ilpo Järvinen 2008-05-12 13:44 ` Damon L. Chesser 0 siblings, 1 reply; 24+ messages in thread From: Ilpo Järvinen @ 2008-05-12 12:07 UTC (permalink / raw) To: Damon L. Chesser; +Cc: Netdev, David Miller [-- Attachment #1: Type: text/plain, Size: 770 bytes --] On Mon, 12 May 2008, Damon L. Chesser wrote: > I ran the first patch and received this error: > > root@dam-main:/usr/src/linux-2.6.24.1# patch -p1 < ../1st_frto_patch.diff > patching file net/ipv4/tcp_input.c > patch: **** malformed patch at line 17: @@ -1685,6 +1683,10 @@ static inline > void tcp_reset_reno_sack(struct tcp_sock *tp) > > root@dam-main:/usr/src/linux-2.6.24.1# > > below is the text of the patch.diff I have: I suppose there was some space-vs-tab or additional-line-break added issue while you saved it. Please try to save from the attached files instead, it should keep the original formatting, they work as is with patch -p1 (no need to cut anything). I tried them into 2.6.24.1 and both of those should apply, though with some fuzz. -- i. [-- Attachment #2: Type: text/plain, Size: 4420 bytes --] From 7f16a60972abdfa0ff8fe9bfb677aab92b714308 Mon Sep 17 00:00:00 2001 From: =?ISO-8859-1?q?Ilpo=20J=E4rvinen?= <ijjarvin@pointhope.localnet.net> Date: Wed, 7 May 2008 21:36:42 +0300 Subject: [PATCH] [TCP] FRTO: SACK variant is errorneously used with NewReno MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit Note: there's actually another bug in FRTO's SACK variant, which is the causing failure in NewReno case because of the error that's fixed here. I'll fix the SACK case separately (it's a separate bug really, though related, but in order to fix that I need to audit tp->snd_nxt usage a bit). There were two places where SACK variant of FRTO is getting incorrectly used even if SACK wasn't negotiated by the TCP flow. This leads to incorrect setting of frto_highmark with NewReno if a previous recovery was interrupted by another RTO. An eventual fallback to conventional recovery then incorrectly considers one or couple of segments as forward transmissions though they weren't, which then are not LOST marked during fallback making them "non-retransmittable" until the next RTO. In a bad case, those segments are really lost and are the only one left in the window. Thus TCP needs another RTO to continue. The next FRTO, however, could again repeat the same events making the progress of the TCP flow extremely slow. In order for these events to occur at all, FRTO must occur again in FRTOs step 3 while the key segments must be lost as well, which is not too likely in practice. It seems to most frequently with some small devices such as network printers that *seem* to accept TCP segments only in-order. In cases were key segments weren't lost, things get automatically resolved because those wrongly marked segments don't need to be retransmitted in order to continue. I found a reproducer after digging up relevant reports (few reports in total, none at netdev or lkml I know of), some cases seemed to indicate middlebox issues which seems now to be a false assumption some people had made. Bugzilla #10063 _might_ be related. Damon L. Chesser <damon@damtek.com> had a reproducable case and was kind enough to tcpdump it for me. With the tcpdump log it was quite trivial to figure out. Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> --- net/ipv4/tcp_input.c | 13 ++++++++----- 1 files changed, 8 insertions(+), 5 deletions(-) diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 0298f80..81ece1f 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -113,8 +113,6 @@ int sysctl_tcp_abc __read_mostly; #define FLAG_FORWARD_PROGRESS (FLAG_ACKED|FLAG_DATA_SACKED) #define FLAG_ANY_PROGRESS (FLAG_FORWARD_PROGRESS|FLAG_SND_UNA_ADVANCED) -#define IsSackFrto() (sysctl_tcp_frto == 0x2) - #define TCP_REMNANT (TCP_FLAG_FIN|TCP_FLAG_URG|TCP_FLAG_SYN|TCP_FLAG_PSH) #define TCP_HP_BITS (~(TCP_RESERVED_BITS|TCP_FLAG_PSH)) @@ -1685,6 +1683,11 @@ static inline void tcp_reset_reno_sack(struct tcp_sock *tp) tp->sacked_out = 0; } +static int tcp_is_sackfrto(const struct tcp_sock *tp) +{ + return (sysctl_tcp_frto == 0x2) && !tcp_is_reno(tp); +} + /* F-RTO can only be used if TCP has never retransmitted anything other than * head (SACK enhanced variant from Appendix B of RFC4138 is more robust here) */ @@ -1701,7 +1704,7 @@ int tcp_use_frto(struct sock *sk) if (icsk->icsk_mtup.probe_size) return 0; - if (IsSackFrto()) + if (tcp_is_sackfrto(tp)) return 1; /* Avoid expensive walking of rexmit queue if possible */ @@ -1791,7 +1794,7 @@ void tcp_enter_frto(struct sock *sk) /* Earlier loss recovery underway (see RFC4138; Appendix B). * The last condition is necessary at least in tp->frto_counter case. */ - if (IsSackFrto() && (tp->frto_counter || + if (tcp_is_sackfrto(tp) && (tp->frto_counter || ((1 << icsk->icsk_ca_state) & (TCPF_CA_Recovery|TCPF_CA_Loss))) && after(tp->high_seq, tp->snd_una)) { tp->frto_highmark = tp->high_seq; @@ -3123,7 +3126,7 @@ static int tcp_process_frto(struct sock *sk, int flag) return 1; } - if (!IsSackFrto() || tcp_is_reno(tp)) { + if (!tcp_is_sackfrto(tp)) { /* RFC4138 shortcoming in step 2; should also have case c): * ACK isn't duplicate nor advances window, e.g., opposite dir * data, winupdate -- 1.5.2.2 [-- Attachment #3: Type: text/plain, Size: 2027 bytes --] From 580ddf96a4c3abf05feb6836c5a66026476ce88d Mon Sep 17 00:00:00 2001 From: =?ISO-8859-1?q?Ilpo=20J=E4rvinen?= <ilpo.jarvinen@helsinki.fi> Date: Sun, 11 May 2008 22:50:13 +0300 Subject: [PATCH] [TCP] FRTO: Fix fallback to conventional recovery MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit It seems that commit 009a2e3e4ec ("[TCP] FRTO: Improve interoperability with other undo_marker users") run into another land-mine which caused fallback to conventional recovery to break: 1. Cumulative ACK arrives after FRTO retransmission 2. tcp_try_to_open sees zero retrans_out, clears retrans_stamp which should be kept like in CA_Loss state it would be 3. undo_marker change allowed tcp_packet_delayed to return true because of the cleared retrans_stamp once FRTO is terminated causing LossUndo to occur, which means all loss markings FRTO made are reverted. This means that the conventional recovery basically recovered one loss per RTT, which is not that efficient. It was quite unobvious that the undo_marker change broken something like this, I had a quite long session to track it down because of the non-intuitiviness of the bug (luckily I had a trivial reproducer at hand and I was also able to learn to use kprobes in the process as well :-)). This together with the NewReno+FRTO fix should finally fix Damon's problems. Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> Reported-by: Damon L. Chesser <damon@damtek.com> --- net/ipv4/tcp_input.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 81ece1f..4c2255c 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -2481,7 +2481,7 @@ static void tcp_try_to_open(struct sock *sk, int flag) tcp_verify_left_out(tp); - if (tp->retrans_out == 0) + if (!tp->frto_counter && tp->retrans_out == 0) tp->retrans_stamp = 0; if (flag & FLAG_ECE) -- 1.5.2.2 ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem (Was: Re: This has a work around) 2008-05-12 12:07 ` Ilpo Järvinen @ 2008-05-12 13:44 ` Damon L. Chesser 2008-05-12 14:35 ` Ilpo Järvinen 0 siblings, 1 reply; 24+ messages in thread From: Damon L. Chesser @ 2008-05-12 13:44 UTC (permalink / raw) To: Ilpo Järvinen; +Cc: Netdev, David Miller [-- Attachment #1: Type: text/plain, Size: 1430 bytes --] Ilpo Järvinen wrote: > On Mon, 12 May 2008, Damon L. Chesser wrote: > > >> I ran the first patch and received this error: >> >> root@dam-main:/usr/src/linux-2.6.24.1# patch -p1 < ../1st_frto_patch.diff >> patching file net/ipv4/tcp_input.c >> patch: **** malformed patch at line 17: @@ -1685,6 +1683,10 @@ static inline >> void tcp_reset_reno_sack(struct tcp_sock *tp) >> >> root@dam-main:/usr/src/linux-2.6.24.1# >> >> below is the text of the patch.diff I have: >> > > I suppose there was some space-vs-tab or additional-line-break added > issue while you saved it. Please try to save from the attached files > instead, it should keep the original formatting, they work as is with > patch -p1 (no need to cut anything). I tried them into 2.6.24.1 and both > of those should apply, though with some fuzz. > > -- > i. Ilpo, I applied the patches in order, no errors on that. I compiled a stock 2.4.24-1 kernel with the patches, I saw no errors there. booted into new kernel, printed with tcp_frto=0. set tcp_frto =2, restarted the network (is this required, or is this a dynamic setting?), printed from OO document. No joy. tcpdump log attached (almost 15 min. worth of data) If you want, I can re-compile and double check for any compilation errors, however, if there were any, it was not sever enough to stop the compilation. -- Damon L. Chesser damon@damtek.com http://www.linkedin.com/in/dchesser [-- Attachment #2: frtoprob4.txt --] [-- Type: text/plain, Size: 35005 bytes --] 1210598636.849859 arp who-has 192.168.200.150 tell 192.168.200.15 1210598636.851863 arp reply 192.168.200.150 is-at 00:0e:3b:00:3a:75 1210598636.851898 IP (tos 0x0, ttl 64, id 23930, offset 0, flags [DF], proto TCP (6), length 60) 192.168.200.15.49841 > 192.168.200.150.9100: S, cksum 0xa9c2 (correct), 834442554:834442554(0) win 5840 <mss 1460,sackOK,timestamp 116800 0,nop,wscale 7> 1210598636.855444 IP (tos 0x0, ttl 100, id 105, offset 0, flags [none], proto TCP (6), length 44) 192.168.200.150.9100 > 192.168.200.15.49841: S, cksum 0x0e99 (correct), 134529024:134529024(0) ack 834442555 win 1024 <mss 1024> 1210598636.855514 IP (tos 0x0, ttl 64, id 23931, offset 0, flags [DF], proto TCP (6), length 40) 192.168.200.15.49841 > 192.168.200.150.9100: ., cksum 0x11d2 (correct), 1:1(0) ack 1 win 5840 1210598637.996507 IP (tos 0x0, ttl 64, id 23932, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 1:513(512) ack 1 win 5840 1210598637.996532 IP (tos 0x0, ttl 64, id 23933, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 513:1025(512) ack 1 win 5840 1210598638.001927 IP (tos 0x0, ttl 100, id 106, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0x22a2 (correct), 1:1(0) ack 513 win 1024 1210598638.001975 IP (tos 0x0, ttl 64, id 23934, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 1025:1537(512) ack 1 win 5840 1210598638.006423 IP (tos 0x0, ttl 100, id 107, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0x20a2 (correct), 1:1(0) ack 1025 win 1024 1210598638.006473 IP (tos 0x0, ttl 64, id 23935, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 1537:2049(512) ack 1 win 5840 1210598638.011175 IP (tos 0x0, ttl 100, id 108, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0x1ea2 (correct), 1:1(0) ack 1537 win 1024 1210598638.011229 IP (tos 0x0, ttl 64, id 23936, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 2049:2561(512) ack 1 win 5840 1210598638.015937 IP (tos 0x0, ttl 100, id 109, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0x1ca2 (correct), 1:1(0) ack 2049 win 1024 1210598638.015965 IP (tos 0x0, ttl 64, id 23937, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 2561:3073(512) ack 1 win 5840 1210598638.022556 IP (tos 0x0, ttl 100, id 110, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0x1aa2 (correct), 1:1(0) ack 2561 win 1024 1210598638.022576 IP (tos 0x0, ttl 64, id 23938, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 3073:3585(512) ack 1 win 5840 1210598638.025960 IP (tos 0x0, ttl 100, id 111, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0x18a2 (correct), 1:1(0) ack 3073 win 1024 1210598638.025984 IP (tos 0x0, ttl 64, id 23939, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 3585:4097(512) ack 1 win 5840 1210598638.030182 IP (tos 0x0, ttl 100, id 112, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0x16a2 (correct), 1:1(0) ack 3585 win 1024 1210598638.030205 IP (tos 0x0, ttl 64, id 23940, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 4097:4609(512) ack 1 win 5840 1210598638.034514 IP (tos 0x0, ttl 100, id 113, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0x14a2 (correct), 1:1(0) ack 4097 win 1024 1210598638.034565 IP (tos 0x0, ttl 64, id 23941, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 4609:5121(512) ack 1 win 5840 1210598638.040023 IP (tos 0x0, ttl 100, id 114, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0x12a2 (correct), 1:1(0) ack 4609 win 1024 1210598638.040071 IP (tos 0x0, ttl 64, id 23942, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 5121:5633(512) ack 1 win 5840 1210598638.044494 IP (tos 0x0, ttl 100, id 115, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0x10a2 (correct), 1:1(0) ack 5121 win 1024 1210598638.044538 IP (tos 0x0, ttl 64, id 23943, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 5633:6145(512) ack 1 win 5840 1210598638.048656 IP (tos 0x0, ttl 100, id 116, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0x0ea2 (correct), 1:1(0) ack 5633 win 1024 1210598638.048683 IP (tos 0x0, ttl 64, id 23944, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 6145:6657(512) ack 1 win 5840 1210598638.052850 IP (tos 0x0, ttl 100, id 117, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0x0ca2 (correct), 1:1(0) ack 6145 win 1024 1210598638.052863 IP (tos 0x0, ttl 64, id 23945, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 6657:7169(512) ack 1 win 5840 1210598638.059816 IP (tos 0x0, ttl 100, id 118, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0x0aa2 (correct), 1:1(0) ack 6657 win 1024 1210598638.059843 IP (tos 0x0, ttl 64, id 23946, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 7169:7681(512) ack 1 win 5840 1210598638.064378 IP (tos 0x0, ttl 100, id 119, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0x08a2 (correct), 1:1(0) ack 7169 win 1024 1210598638.064420 IP (tos 0x0, ttl 64, id 23947, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 7681:8193(512) ack 1 win 5840 1210598638.069040 IP (tos 0x0, ttl 100, id 120, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0x06a2 (correct), 1:1(0) ack 7681 win 1024 1210598638.069090 IP (tos 0x0, ttl 64, id 23948, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 8193:8705(512) ack 1 win 5840 1210598638.073544 IP (tos 0x0, ttl 100, id 121, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0x04a2 (correct), 1:1(0) ack 8193 win 1024 1210598638.073592 IP (tos 0x0, ttl 64, id 23949, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 8705:9217(512) ack 1 win 5840 1210598638.080437 IP (tos 0x0, ttl 100, id 122, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0x02a2 (correct), 1:1(0) ack 8705 win 1024 1210598638.080461 IP (tos 0x0, ttl 64, id 23950, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 9217:9729(512) ack 1 win 5840 1210598638.084244 IP (tos 0x0, ttl 100, id 123, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0x00a2 (correct), 1:1(0) ack 9217 win 1024 1210598638.084258 IP (tos 0x0, ttl 64, id 23951, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 9729:10241(512) ack 1 win 5840 1210598638.087992 IP (tos 0x0, ttl 100, id 124, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xfea1 (correct), 1:1(0) ack 9729 win 1024 1210598638.088014 IP (tos 0x0, ttl 64, id 23952, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 10241:10753(512) ack 1 win 5840 1210598638.092925 IP (tos 0x0, ttl 100, id 125, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xfca1 (correct), 1:1(0) ack 10241 win 1024 1210598638.092943 IP (tos 0x0, ttl 64, id 23953, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 10753:11265(512) ack 1 win 5840 1210598638.095674 IP (tos 0x0, ttl 100, id 126, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xfca1 (correct), 1:1(0) ack 10753 win 512 1210598638.098915 IP (tos 0x0, ttl 100, id 127, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xfaa1 (correct), 1:1(0) ack 11265 win 512 1210598638.098939 IP (tos 0x0, ttl 64, id 23954, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 11265:11777(512) ack 1 win 5840 1210598638.102729 IP (tos 0x0, ttl 100, id 128, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xf8a1 (correct), 1:1(0) ack 11777 win 512 1210598638.102741 IP (tos 0x0, ttl 64, id 23955, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 11777:12289(512) ack 1 win 5840 1210598638.309935 IP (tos 0x0, ttl 64, id 23956, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 11777:12289(512) ack 1 win 5840 1210598638.334007 IP (tos 0x0, ttl 100, id 129, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xf6a1 (correct), 1:1(0) ack 12289 win 512 1210598638.334054 IP (tos 0x0, ttl 64, id 23957, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 12289:12801(512) ack 1 win 5840 1210598638.556548 IP (tos 0x0, ttl 100, id 130, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xf2a1 (correct), 1:1(0) ack 12801 win 1024 1210598638.556597 IP (tos 0x0, ttl 64, id 23958, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 12801:13313(512) ack 1 win 5840 1210598638.556606 IP (tos 0x0, ttl 64, id 23959, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 13313:13825(512) ack 1 win 5840 1210598638.567069 IP (tos 0x0, ttl 100, id 131, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xf0a1 (correct), 1:1(0) ack 13313 win 1024 1210598638.567100 IP (tos 0x0, ttl 64, id 23960, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 13825:14337(512) ack 1 win 5840 1210598638.571060 IP (tos 0x0, ttl 100, id 132, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xeea1 (correct), 1:1(0) ack 13825 win 1024 1210598638.571083 IP (tos 0x0, ttl 64, id 23961, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 14337:14849(512) ack 1 win 5840 1210598638.574242 IP (tos 0x0, ttl 100, id 133, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xeea1 (correct), 1:1(0) ack 14337 win 512 1210598638.578004 IP (tos 0x0, ttl 100, id 134, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xeca1 (correct), 1:1(0) ack 14849 win 512 1210598638.578026 IP (tos 0x0, ttl 64, id 23962, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 14849:15361(512) ack 1 win 5840 1210598638.583733 IP (tos 0x0, ttl 100, id 135, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xeaa1 (correct), 1:1(0) ack 15361 win 512 1210598638.583754 IP (tos 0x0, ttl 64, id 23963, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 15361:15873(512) ack 1 win 5840 1210598638.587994 IP (tos 0x0, ttl 100, id 136, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xe8a1 (correct), 1:1(0) ack 15873 win 512 1210598638.588008 IP (tos 0x0, ttl 64, id 23964, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 15873:16385(512) ack 1 win 5840 1210598638.592083 IP (tos 0x0, ttl 100, id 137, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xe6a1 (correct), 1:1(0) ack 16385 win 512 1210598638.592106 IP (tos 0x0, ttl 64, id 23965, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 16385:16897(512) ack 1 win 5840 1210598638.596289 IP (tos 0x0, ttl 100, id 138, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xe4a1 (correct), 1:1(0) ack 16897 win 512 1210598638.596301 IP (tos 0x0, ttl 64, id 23966, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 16897:17409(512) ack 1 win 5840 1210598638.601744 IP (tos 0x0, ttl 100, id 139, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xe2a1 (correct), 1:1(0) ack 17409 win 512 1210598638.601766 IP (tos 0x0, ttl 64, id 23967, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 17409:17921(512) ack 1 win 5840 1210598638.605859 IP (tos 0x0, ttl 100, id 140, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xe0a1 (correct), 1:1(0) ack 17921 win 512 1210598638.605872 IP (tos 0x0, ttl 64, id 23968, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 17921:18433(512) ack 1 win 5840 1210598638.611944 IP (tos 0x0, ttl 100, id 141, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xdca1 (correct), 1:1(0) ack 18433 win 1024 1210598638.611996 IP (tos 0x0, ttl 64, id 23969, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 18433:18945(512) ack 1 win 5840 1210598638.612006 IP (tos 0x0, ttl 64, id 23970, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 18945:19457(512) ack 1 win 5840 1210598638.626765 IP (tos 0x0, ttl 100, id 142, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xdaa1 (correct), 1:1(0) ack 18945 win 1024 1210598638.626800 IP (tos 0x0, ttl 64, id 23971, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 19457:19969(512) ack 1 win 5840 1210598638.849968 IP (tos 0x0, ttl 64, id 23972, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 18945:19457(512) ack 1 win 5840 1210598638.853907 IP (tos 0x0, ttl 100, id 143, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xd8a1 (correct), 1:1(0) ack 19457 win 1024 1210598638.853937 IP (tos 0x0, ttl 64, id 23973, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 19969:20481(512) ack 1 win 5840 1210598639.301994 IP (tos 0x0, ttl 64, id 23974, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 19457:19969(512) ack 1 win 5840 1210598639.307911 IP (tos 0x0, ttl 100, id 144, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xd6a1 (correct), 1:1(0) ack 19969 win 1024 1210598639.307957 IP (tos 0x0, ttl 64, id 23975, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 20481:20993(512) ack 1 win 5840 1210598640.002669 IP (tos 0x0, ttl 100, id 145, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xd6a1 (correct), 1:1(0) ack 19969 win 1024 1210598640.002718 IP (tos 0x0, ttl 64, id 23976, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 19969:20481(512) ack 1 win 5840 1210598640.009222 IP (tos 0x0, ttl 100, id 146, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xd4a1 (correct), 1:1(0) ack 20481 win 1024 1210598640.009277 IP (tos 0x0, ttl 64, id 23977, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 20993:21505(512) ack 1 win 5840 1210598640.708995 IP (tos 0x0, ttl 100, id 147, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xd4a1 (correct), 1:1(0) ack 20481 win 1024 1210598640.901981 IP (tos 0x0, ttl 64, id 23978, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 20481:20993(512) ack 1 win 5840 1210598640.908117 IP (tos 0x0, ttl 100, id 148, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xd2a1 (correct), 1:1(0) ack 20993 win 1024 1210598640.908175 IP (tos 0x0, ttl 64, id 23979, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 21505:22017(512) ack 1 win 5840 1210598641.602725 IP (tos 0x0, ttl 100, id 149, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xd2a1 (correct), 1:1(0) ack 20993 win 1024 1210598641.602773 IP (tos 0x0, ttl 64, id 23980, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 20993:21505(512) ack 1 win 5840 1210598641.609275 IP (tos 0x0, ttl 100, id 150, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xd0a1 (correct), 1:1(0) ack 21505 win 1024 1210598641.609292 IP (tos 0x0, ttl 64, id 23981, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 22017:22529(512) ack 1 win 5840 1210598642.308931 IP (tos 0x0, ttl 100, id 151, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xd0a1 (correct), 1:1(0) ack 21505 win 1024 1210598643.398246 IP (tos 0x0, ttl 64, id 23982, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 21505:22017(512) ack 1 win 5840 1210598643.402305 IP (tos 0x0, ttl 100, id 152, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xcea1 (correct), 1:1(0) ack 22017 win 1024 1210598643.402359 IP (tos 0x0, ttl 64, id 23983, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 22529:23041(512) ack 1 win 5840 1210598644.101612 IP (tos 0x0, ttl 100, id 153, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xcea1 (correct), 1:1(0) ack 22017 win 1024 1210598644.101661 IP (tos 0x0, ttl 64, id 23984, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 22017:22529(512) ack 1 win 5840 1210598644.108168 IP (tos 0x0, ttl 100, id 154, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xcca1 (correct), 1:1(0) ack 22529 win 1024 1210598644.108222 IP (tos 0x0, ttl 64, id 23985, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 23041:23553(512) ack 1 win 5840 1210598644.802907 IP (tos 0x0, ttl 100, id 155, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xcca1 (correct), 1:1(0) ack 22529 win 1024 1210598647.690507 IP (tos 0x0, ttl 64, id 23986, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 22529:23041(512) ack 1 win 5840 1210598647.694424 IP (tos 0x0, ttl 100, id 159, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xcaa1 (correct), 1:1(0) ack 23041 win 1024 1210598647.694464 IP (tos 0x0, ttl 64, id 23987, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 23553:24065(512) ack 1 win 5840 1210598648.393212 IP (tos 0x0, ttl 100, id 160, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xcaa1 (correct), 1:1(0) ack 23041 win 1024 1210598648.393259 IP (tos 0x0, ttl 64, id 23988, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 23041:23553(512) ack 1 win 5840 1210598648.399741 IP (tos 0x0, ttl 100, id 161, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xc8a1 (correct), 1:1(0) ack 23553 win 1024 1210598648.399777 IP (tos 0x0, ttl 64, id 23989, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 24065:24577(512) ack 1 win 5840 1210598649.094481 IP (tos 0x0, ttl 100, id 162, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xc8a1 (correct), 1:1(0) ack 23553 win 1024 1210598655.566968 IP (tos 0x0, ttl 64, id 23990, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 23553:24065(512) ack 1 win 5840 1210598655.570872 IP (tos 0x0, ttl 100, id 163, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xc6a1 (correct), 1:1(0) ack 24065 win 1024 1210598655.570903 IP (tos 0x0, ttl 64, id 23991, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 24577:25089(512) ack 1 win 5840 1210598656.270124 IP (tos 0x0, ttl 100, id 164, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xc6a1 (correct), 1:1(0) ack 24065 win 1024 1210598656.270168 IP (tos 0x0, ttl 64, id 23992, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 24065:24577(512) ack 1 win 5840 1210598656.276663 IP (tos 0x0, ttl 100, id 165, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xc4a1 (correct), 1:1(0) ack 24577 win 1024 1210598656.276680 IP (tos 0x0, ttl 64, id 23993, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 25089:25601(512) ack 1 win 5840 1210598656.971404 IP (tos 0x0, ttl 100, id 166, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xc4a1 (correct), 1:1(0) ack 24577 win 1024 1210598670.611871 IP (tos 0x0, ttl 64, id 23994, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 24577:25089(512) ack 1 win 5840 1210598670.617139 IP (tos 0x0, ttl 100, id 167, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xc2a1 (correct), 1:1(0) ack 25089 win 1024 1210598670.617188 IP (tos 0x0, ttl 64, id 23995, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 25601:26113(512) ack 1 win 5840 1210598671.312854 IP (tos 0x0, ttl 100, id 168, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xc2a1 (correct), 1:1(0) ack 25089 win 1024 1210598671.312899 IP (tos 0x0, ttl 64, id 23996, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 25089:25601(512) ack 1 win 5840 1210598671.319408 IP (tos 0x0, ttl 100, id 169, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xc0a1 (correct), 1:1(0) ack 25601 win 1024 1210598671.319424 IP (tos 0x0, ttl 64, id 23997, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 26113:26625(512) ack 1 win 5840 1210598672.019199 IP (tos 0x0, ttl 100, id 170, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xc0a1 (correct), 1:1(0) ack 25601 win 1024 1210598699.989635 IP (tos 0x0, ttl 64, id 23998, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 25601:26113(512) ack 1 win 5840 1210598699.993560 IP (tos 0x0, ttl 100, id 171, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xbea1 (correct), 1:1(0) ack 26113 win 1024 1210598699.993590 IP (tos 0x0, ttl 64, id 23999, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 26625:27137(512) ack 1 win 5840 1210598700.692114 IP (tos 0x0, ttl 100, id 172, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xbea1 (correct), 1:1(0) ack 26113 win 1024 1210598700.692161 IP (tos 0x0, ttl 64, id 24000, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 26113:26625(512) ack 1 win 5840 1210598700.698643 IP (tos 0x0, ttl 100, id 173, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xbca1 (correct), 1:1(0) ack 26625 win 1024 1210598700.698660 IP (tos 0x0, ttl 64, id 24001, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 27137:27649(512) ack 1 win 5840 1210598701.398325 IP (tos 0x0, ttl 100, id 174, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xbca1 (correct), 1:1(0) ack 26625 win 1024 1210598758.045115 IP (tos 0x0, ttl 64, id 24002, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 26625:27137(512) ack 1 win 5840 1210598758.049665 IP (tos 0x0, ttl 100, id 178, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xbaa1 (correct), 1:1(0) ack 27137 win 1024 1210598758.049697 IP (tos 0x0, ttl 64, id 24003, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 27649:28161(512) ack 1 win 5840 1210598758.749359 IP (tos 0x0, ttl 100, id 179, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xbaa1 (correct), 1:1(0) ack 27137 win 1024 1210598758.749404 IP (tos 0x0, ttl 64, id 24004, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 27137:27649(512) ack 1 win 5840 1210598758.755872 IP (tos 0x0, ttl 100, id 180, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xb8a1 (correct), 1:1(0) ack 27649 win 1024 1210598758.755889 IP (tos 0x0, ttl 64, id 24005, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 28161:28673(512) ack 1 win 5840 1210598759.450606 IP (tos 0x0, ttl 100, id 181, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xb8a1 (correct), 1:1(0) ack 27649 win 1024 1210598873.448031 IP (tos 0x0, ttl 64, id 24006, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 27649:28161(512) ack 1 win 5840 1210598873.452984 IP (tos 0x0, ttl 100, id 185, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xb6a1 (correct), 1:1(0) ack 28161 win 1024 1210598873.453010 IP (tos 0x0, ttl 64, id 24007, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 28673:29185(512) ack 1 win 5840 1210598874.147736 IP (tos 0x0, ttl 100, id 186, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xb6a1 (correct), 1:1(0) ack 28161 win 1024 1210598874.147781 IP (tos 0x0, ttl 64, id 24008, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 28161:28673(512) ack 1 win 5840 1210598874.154257 IP (tos 0x0, ttl 100, id 187, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xb4a1 (correct), 1:1(0) ack 28673 win 1024 1210598874.154275 IP (tos 0x0, ttl 64, id 24009, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 29185:29697(512) ack 1 win 5840 1210598874.848987 IP (tos 0x0, ttl 100, id 188, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xb4a1 (correct), 1:1(0) ack 28673 win 1024 1210598994.159267 IP (tos 0x0, ttl 64, id 24010, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 28673:29185(512) ack 1 win 5840 1210598994.165203 IP (tos 0x0, ttl 100, id 195, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xb2a1 (correct), 1:1(0) ack 29185 win 1024 1210598994.165234 IP (tos 0x0, ttl 64, id 24011, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 29697:30209(512) ack 1 win 5840 1210598994.864884 IP (tos 0x0, ttl 100, id 196, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xb2a1 (correct), 1:1(0) ack 29185 win 1024 1210598994.864928 IP (tos 0x0, ttl 64, id 24012, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 29185:29697(512) ack 1 win 5840 1210598994.871533 IP (tos 0x0, ttl 100, id 197, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xb0a1 (correct), 1:1(0) ack 29697 win 1024 1210598994.871549 IP (tos 0x0, ttl 64, id 24013, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 30209:30721(512) ack 1 win 5840 1210598995.566158 IP (tos 0x0, ttl 100, id 198, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xb0a1 (correct), 1:1(0) ack 29697 win 1024 1210599114.878503 IP (tos 0x0, ttl 64, id 24014, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 29697:30209(512) ack 1 win 5840 1210599114.882513 IP (tos 0x0, ttl 100, id 202, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xaea1 (correct), 1:1(0) ack 30209 win 1024 1210599114.882546 IP (tos 0x0, ttl 64, id 24015, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 30721:31233(512) ack 1 win 5840 1210599115.577094 IP (tos 0x0, ttl 100, id 203, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xaea1 (correct), 1:1(0) ack 30209 win 1024 1210599115.577141 IP (tos 0x0, ttl 64, id 24016, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 30209:30721(512) ack 1 win 5840 1210599115.583636 IP (tos 0x0, ttl 100, id 204, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xaca1 (correct), 1:1(0) ack 30721 win 1024 1210599115.583653 IP (tos 0x0, ttl 64, id 24017, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 31233:31745(512) ack 1 win 5840 1210599116.278385 IP (tos 0x0, ttl 100, id 205, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xaca1 (correct), 1:1(0) ack 30721 win 1024 1210599235.589761 IP (tos 0x0, ttl 64, id 24018, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 30721:31233(512) ack 1 win 5840 1210599235.594556 IP (tos 0x0, ttl 100, id 212, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xaaa1 (correct), 1:1(0) ack 31233 win 1024 1210599235.594595 IP (tos 0x0, ttl 64, id 24019, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 31745:32257(512) ack 1 win 5840 1210599236.289314 IP (tos 0x0, ttl 100, id 213, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xaaa1 (correct), 1:1(0) ack 31233 win 1024 1210599236.289361 IP (tos 0x0, ttl 64, id 24020, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 31233:31745(512) ack 1 win 5840 1210599236.295847 IP (tos 0x0, ttl 100, id 214, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xa8a1 (correct), 1:1(0) ack 31745 win 1024 1210599236.295865 IP (tos 0x0, ttl 64, id 24021, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 32257:32769(512) ack 1 win 5840 1210599236.990579 IP (tos 0x0, ttl 100, id 215, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xa8a1 (correct), 1:1(0) ack 31745 win 1024 1210599356.300987 IP (tos 0x0, ttl 64, id 24022, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 31745:32257(512) ack 1 win 5840 1210599356.306860 IP (tos 0x0, ttl 100, id 219, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xa6a1 (correct), 1:1(0) ack 32257 win 1024 1210599356.306914 IP (tos 0x0, ttl 64, id 24023, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: . 32769:33281(512) ack 1 win 5840 1210599357.001550 IP (tos 0x0, ttl 100, id 220, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xa6a1 (correct), 1:1(0) ack 32257 win 1024 1210599357.001597 IP (tos 0x0, ttl 64, id 24024, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 32257:32769(512) ack 1 win 5840 1210599357.008325 IP (tos 0x0, ttl 100, id 221, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xa4a1 (correct), 1:1(0) ack 32769 win 1024 1210599357.008344 IP (tos 0x0, ttl 64, id 24025, offset 0, flags [DF], proto TCP (6), length 552) 192.168.200.15.49841 > 192.168.200.150.9100: P 33281:33793(512) ack 1 win 5840 1210599357.702816 IP (tos 0x0, ttl 100, id 222, offset 0, flags [none], proto TCP (6), length 40) 192.168.200.150.9100 > 192.168.200.15.49841: ., cksum 0xa4a1 (correct), 1:1(0) ack 32769 win 1024 ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem (Was: Re: This has a work around) 2008-05-12 13:44 ` Damon L. Chesser @ 2008-05-12 14:35 ` Ilpo Järvinen 2008-05-12 16:40 ` Ilpo Järvinen ` (2 more replies) 0 siblings, 3 replies; 24+ messages in thread From: Ilpo Järvinen @ 2008-05-12 14:35 UTC (permalink / raw) To: Damon L. Chesser; +Cc: Netdev, David Miller [-- Attachment #1: Type: text/plain, Size: 1254 bytes --] On Mon, 12 May 2008, Damon L. Chesser wrote: > I applied the patches in order, no errors on that. I compiled a stock > 2.4.24-1 kernel with the patches, I saw no errors there. > > booted into new kernel, printed with tcp_frto=0. set tcp_frto =2, restarted > the network (is this required, or is this a dynamic setting?), printed from OO > document. No joy. tcpdump log attached (almost 15 min. worth of data) > > If you want, I can re-compile and double check for any compilation errors, > however, if there were any, it was not sever enough to stop the compilation. On the bright side, the FRTO problem that was occuring previously is now fixed but there seems to be very few ways to communicate with that device sanely because it assumes in-order arrival and keeps discarding, as it seems, _all_ other segments... If you could try with this additional work-around attached (keep the fixes there as well). Turn tcp_frto_inorder_workaround sysctl to 1 before testing with FRTO. ...Can you please send a dump about working case too, this seems rather nasty device to work with (tcp_frto = 0 is enough to attain it, no need to have another kernel booted for that) and I'm interested to see what are the loss rates without FRTO... -- i. [-- Attachment #2: Type: text/plain, Size: 3017 bytes --] From 1fbcd93fe81a2276e9df12fe575f33df4dbfb312 Mon Sep 17 00:00:00 2001 From: =?ISO-8859-1?q?Ilpo=20J=E4rvinen?= <ilpo.jarvinen@helsinki.fi> Date: Mon, 12 May 2008 17:24:02 +0300 Subject: [PATCH] [TCP] FRTO: workaround in-order only receivers MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit If receiver consumes segments successfully only in-order, FRTO fallback to conventional recovery produces RTO loop because FRTO's forward transmissions will always get dropped and need to be resent, yet by default they're not marked as lost (which are the only segments we will retransmit in CA_Loss). Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> Reported-by: Damon L. Chesser <damon@damtek.com> --- include/net/tcp.h | 1 + net/ipv4/sysctl_net_ipv4.c | 8 ++++++++ net/ipv4/tcp_input.c | 4 +++- 3 files changed, 12 insertions(+), 1 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index cb5b033..e6e44b8 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -222,6 +222,7 @@ extern int sysctl_tcp_adv_win_scale; extern int sysctl_tcp_tw_reuse; extern int sysctl_tcp_frto; extern int sysctl_tcp_frto_response; +extern int sysctl_tcp_frto_inorder_workaround; extern int sysctl_tcp_low_latency; extern int sysctl_tcp_dma_copybreak; extern int sysctl_tcp_nometrics_save; diff --git a/net/ipv4/sysctl_net_ipv4.c b/net/ipv4/sysctl_net_ipv4.c index bec6fe8..9cbdccf 100644 --- a/net/ipv4/sysctl_net_ipv4.c +++ b/net/ipv4/sysctl_net_ipv4.c @@ -723,6 +723,14 @@ ctl_table ipv4_table[] = { .proc_handler = &proc_dointvec }, { + .ctl_name = CTL_UNNUMBERED, + .procname = "tcp_frto_inorder_workaround", + .data = &sysctl_tcp_frto_inorder_workaround, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = &proc_dointvec + }, + { .ctl_name = NET_TCP_LOW_LATENCY, .procname = "tcp_low_latency", .data = &sysctl_tcp_low_latency, diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index b39f0d8..0292070 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -87,6 +87,7 @@ int sysctl_tcp_rfc1337 __read_mostly; int sysctl_tcp_max_orphans __read_mostly = NR_FILE; int sysctl_tcp_frto __read_mostly = 2; int sysctl_tcp_frto_response __read_mostly; +int sysctl_tcp_frto_inorder_workaround __read_mostly; int sysctl_tcp_nometrics_save __read_mostly; int sysctl_tcp_moderate_rcvbuf __read_mostly = 1; @@ -1729,7 +1730,8 @@ static void tcp_enter_frto_loss(struct sock *sk, int allowed_segments, int flag) /* Don't lost mark skbs that were fwd transmitted after RTO */ if (!(TCP_SKB_CB(skb)->sacked&TCPCB_SACKED_ACKED) && - !after(TCP_SKB_CB(skb)->end_seq, tp->frto_highmark)) { + (sysctl_tcp_frto_inorder_workaround || + !after(TCP_SKB_CB(skb)->end_seq, tp->frto_highmark))) { TCP_SKB_CB(skb)->sacked |= TCPCB_LOST; tp->lost_out += tcp_skb_pcount(skb); } -- 1.5.2.2 ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem (Was: Re: This has a work around) 2008-05-12 14:35 ` Ilpo Järvinen @ 2008-05-12 16:40 ` Ilpo Järvinen 2008-05-12 17:02 ` Damon L. Chesser 2008-05-12 18:39 ` Damon L. Chesser 2 siblings, 0 replies; 24+ messages in thread From: Ilpo Järvinen @ 2008-05-12 16:40 UTC (permalink / raw) To: Damon L. Chesser; +Cc: Netdev [-- Attachment #1: Type: TEXT/PLAIN, Size: 595 bytes --] On Mon, 12 May 2008, Ilpo Järvinen wrote: > On the bright side, the FRTO problem that was occuring previously is now fixed > but there seems to be very few ways to communicate with that device sanely > because it assumes in-order arrival and keeps discarding, as it seems, _all_ > other segments... If you could try with this additional work-around attached > (keep the fixes there as well). Turn tcp_frto_inorder_workaround sysctl to 1 > before testing with FRTO. I missed one thing with the last workaround patch, please use this one instead if you haven't yet have time to try it. -- i. [-- Attachment #2: Type: TEXT/PLAIN, Size: 3378 bytes --] From 9bdbbca4162ba89f0be27a74d69b39af95cf2c99 Mon Sep 17 00:00:00 2001 From: =?ISO-8859-1?q?Ilpo=20J=E4rvinen?= <ilpo.jarvinen@helsinki.fi> Date: Mon, 12 May 2008 17:24:02 +0300 Subject: [PATCH] [TCP] FRTO: workaround in-order only receivers MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit If receiver consumes segments successfully only in-order, FRTO fallback to conventional recovery produces RTO loop because FRTO's forward transmissions will always get dropped and need to be resent, yet by default they're not marked as lost (which are the only segments we will retransmit in CA_Loss). Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> Reported-by: Damon L. Chesser <damon@damtek.com> --- include/net/tcp.h | 1 + net/ipv4/sysctl_net_ipv4.c | 8 ++++++++ net/ipv4/tcp_input.c | 6 +++++- 3 files changed, 14 insertions(+), 1 deletions(-) diff --git a/include/net/tcp.h b/include/net/tcp.h index cb5b033..e6e44b8 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -222,6 +222,7 @@ extern int sysctl_tcp_adv_win_scale; extern int sysctl_tcp_tw_reuse; extern int sysctl_tcp_frto; extern int sysctl_tcp_frto_response; +extern int sysctl_tcp_frto_inorder_workaround; extern int sysctl_tcp_low_latency; extern int sysctl_tcp_dma_copybreak; extern int sysctl_tcp_nometrics_save; diff --git a/net/ipv4/sysctl_net_ipv4.c b/net/ipv4/sysctl_net_ipv4.c index bec6fe8..9cbdccf 100644 --- a/net/ipv4/sysctl_net_ipv4.c +++ b/net/ipv4/sysctl_net_ipv4.c @@ -723,6 +723,14 @@ ctl_table ipv4_table[] = { .proc_handler = &proc_dointvec }, { + .ctl_name = CTL_UNNUMBERED, + .procname = "tcp_frto_inorder_workaround", + .data = &sysctl_tcp_frto_inorder_workaround, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = &proc_dointvec + }, + { .ctl_name = NET_TCP_LOW_LATENCY, .procname = "tcp_low_latency", .data = &sysctl_tcp_low_latency, diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index b39f0d8..fbab150 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -87,6 +87,7 @@ int sysctl_tcp_rfc1337 __read_mostly; int sysctl_tcp_max_orphans __read_mostly = NR_FILE; int sysctl_tcp_frto __read_mostly = 2; int sysctl_tcp_frto_response __read_mostly; +int sysctl_tcp_frto_inorder_workaround __read_mostly; int sysctl_tcp_nometrics_save __read_mostly; int sysctl_tcp_moderate_rcvbuf __read_mostly = 1; @@ -1729,7 +1730,8 @@ static void tcp_enter_frto_loss(struct sock *sk, int allowed_segments, int flag) /* Don't lost mark skbs that were fwd transmitted after RTO */ if (!(TCP_SKB_CB(skb)->sacked&TCPCB_SACKED_ACKED) && - !after(TCP_SKB_CB(skb)->end_seq, tp->frto_highmark)) { + (sysctl_tcp_frto_inorder_workaround || + !after(TCP_SKB_CB(skb)->end_seq, tp->frto_highmark))) { TCP_SKB_CB(skb)->sacked |= TCPCB_LOST; tp->lost_out += tcp_skb_pcount(skb); } @@ -1746,6 +1748,8 @@ static void tcp_enter_frto_loss(struct sock *sk, int allowed_segments, int flag) sysctl_tcp_reordering); tcp_set_ca_state(sk, TCP_CA_Loss); tp->high_seq = tp->frto_highmark; + if (sysctl_tcp_frto_inorder_workaround) + tp->high_seq = tp->snd_nxt; TCP_ECN_queue_cwr(tp); tcp_clear_retrans_hints_partial(tp); -- 1.5.2.2 ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem (Was: Re: This has a work around) 2008-05-12 14:35 ` Ilpo Järvinen 2008-05-12 16:40 ` Ilpo Järvinen @ 2008-05-12 17:02 ` Damon L. Chesser 2008-05-12 18:23 ` Ilpo Järvinen 2008-05-12 18:39 ` Damon L. Chesser 2 siblings, 1 reply; 24+ messages in thread From: Damon L. Chesser @ 2008-05-12 17:02 UTC (permalink / raw) To: Ilpo Järvinen; +Cc: Netdev, David Miller [-- Attachment #1: Type: text/plain, Size: 1633 bytes --] Ilpo Järvinen wrote: > On Mon, 12 May 2008, Damon L. Chesser wrote: > >> I applied the patches in order, no errors on that. I compiled a stock >> 2.4.24-1 kernel with the patches, I saw no errors there. >> >> booted into new kernel, printed with tcp_frto=0. set tcp_frto =2, >> restarted >> the network (is this required, or is this a dynamic setting?), >> printed from OO >> document. No joy. tcpdump log attached (almost 15 min. worth of data) >> >> If you want, I can re-compile and double check for any compilation >> errors, >> however, if there were any, it was not sever enough to stop the >> compilation. > > On the bright side, the FRTO problem that was occuring previously is > now fixed but there seems to be very few ways to communicate with that > device sanely because it assumes in-order arrival and keeps > discarding, as it seems, _all_ other segments... If you could try with > this additional work-around attached (keep the fixes there as well). > Turn tcp_frto_inorder_workaround sysctl to 1 before testing with FRTO. > > ...Can you please send a dump about working case too, this seems > rather nasty device to work with (tcp_frto = 0 is enough to attain it, > no need to have another kernel booted for that) and I'm interested to > see what are the loss rates without FRTO... > > Attached is the compressed "good" test with tcp_frto = 0. Sorry about the size, I did not think to compress the earlier ones. Five test pages printed, no errors. I will get a new kernel compiled shortly, and tested as you requested. -- Damon L. Chesser damon@damtek.com http://www.linkedin.com/in/dchesser [-- Attachment #2: frtoprob5.txt.gz --] [-- Type: application/x-gzip, Size: 193617 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem (Was: Re: This has a work around) 2008-05-12 17:02 ` Damon L. Chesser @ 2008-05-12 18:23 ` Ilpo Järvinen 0 siblings, 0 replies; 24+ messages in thread From: Ilpo Järvinen @ 2008-05-12 18:23 UTC (permalink / raw) To: Damon L. Chesser; +Cc: Netdev [-- Attachment #1: Type: TEXT/PLAIN, Size: 836 bytes --] On Mon, 12 May 2008, Damon L. Chesser wrote: > Ilpo Järvinen wrote: > > > > ...Can you please send a dump about working case too, this seems rather > > nasty device to work with (tcp_frto = 0 is enough to attain it, no need to > > have another kernel booted for that) and I'm interested to see what are the > > loss rates without FRTO... > > > > > Attached is the compressed "good" test with tcp_frto = 0. Sorry about the > size, I did not think to compress the earlier ones. Five test pages printed, > no errors. I will get a new kernel compiled shortly, and tested as you > requested. ...Thanks, I'll have a look. There was no need to compress the earlier ones as they were not that large, but I realized after asking that one might hit netdev size cutter without compression when things worked, thus compression :-). -- i. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem (Was: Re: This has a work around) 2008-05-12 14:35 ` Ilpo Järvinen 2008-05-12 16:40 ` Ilpo Järvinen 2008-05-12 17:02 ` Damon L. Chesser @ 2008-05-12 18:39 ` Damon L. Chesser 2008-05-12 19:12 ` Ilpo Järvinen 2 siblings, 1 reply; 24+ messages in thread From: Damon L. Chesser @ 2008-05-12 18:39 UTC (permalink / raw) To: Ilpo Järvinen; +Cc: Netdev, David Miller [-- Attachment #1: Type: text/plain, Size: 1559 bytes --] Ilpo Järvinen wrote: > On Mon, 12 May 2008, Damon L. Chesser wrote: > >> I applied the patches in order, no errors on that. I compiled a stock >> 2.4.24-1 kernel with the patches, I saw no errors there. >> >> booted into new kernel, printed with tcp_frto=0. set tcp_frto =2, >> restarted >> the network (is this required, or is this a dynamic setting?), >> printed from OO >> document. No joy. tcpdump log attached (almost 15 min. worth of data) >> >> If you want, I can re-compile and double check for any compilation >> errors, >> however, if there were any, it was not sever enough to stop the >> compilation. > > On the bright side, the FRTO problem that was occuring previously is > now fixed but there seems to be very few ways to communicate with that > device sanely because it assumes in-order arrival and keeps > discarding, as it seems, _all_ other segments... If you could try with > this additional work-around attached (keep the fixes there as well). > Turn tcp_frto_inorder_workaround sysctl to 1 before testing with FRTO. > > ...Can you please send a dump about working case too, this seems > rather nasty device to work with (tcp_frto = 0 is enough to attain it, > no need to have another kernel booted for that) and I'm interested to > see what are the loss rates without FRTO... > > New patch added in with the first two, tcp_frto_inorder_workaround =1 test printed 5 pages: This worked. Attached is the output of tcpdump. Need anything else? -- Damon L. Chesser damon@damtek.com http://www.linkedin.com/in/dchesser [-- Attachment #2: frtoprob6.txt.gz --] [-- Type: application/x-gzip, Size: 235644 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem (Was: Re: This has a work around) 2008-05-12 18:39 ` Damon L. Chesser @ 2008-05-12 19:12 ` Ilpo Järvinen 2008-05-12 19:18 ` Damon L. Chesser 2008-05-12 22:48 ` Fix FRTO+NewReno problem David Miller 0 siblings, 2 replies; 24+ messages in thread From: Ilpo Järvinen @ 2008-05-12 19:12 UTC (permalink / raw) To: Damon L. Chesser, David Miller; +Cc: Netdev [-- Attachment #1: Type: TEXT/PLAIN, Size: 2464 bytes --] On Mon, 12 May 2008, Damon L. Chesser wrote: > Ilpo Järvinen wrote: > > On Mon, 12 May 2008, Damon L. Chesser wrote: > > > > > I applied the patches in order, no errors on that. I compiled a stock > > > 2.4.24-1 kernel with the patches, I saw no errors there. > > > > > > booted into new kernel, printed with tcp_frto=0. set tcp_frto =2, > > > restarted > > > the network (is this required, or is this a dynamic setting?), printed > > > from OO > > > document. No joy. tcpdump log attached (almost 15 min. worth of data) > > > > > > If you want, I can re-compile and double check for any compilation errors, > > > however, if there were any, it was not sever enough to stop the > > > compilation. > > > > On the bright side, the FRTO problem that was occuring previously is now > > fixed but there seems to be very few ways to communicate with that device > > sanely because it assumes in-order arrival and keeps discarding, as it > > seems, _all_ other segments... If you could try with this additional > > work-around attached (keep the fixes there as well). Turn > > tcp_frto_inorder_workaround sysctl to 1 before testing with FRTO. > > > > ...Can you please send a dump about working case too, this seems rather > > nasty device to work with (tcp_frto = 0 is enough to attain it, no need to > > have another kernel booted for that) and I'm interested to see what are the > > loss rates without FRTO... > > > > > New patch added in with the first two, tcp_frto_inorder_workaround =1 test > printed 5 pages: This worked. Attached is the output of tcpdump. Need > anything else? Thanks a lot for the testing & all. The picture is clear enough already, so no additional help needed (I haven't yet looked the non-frto dump but I doubt anything earth-shattering turns out, it's mostly interesting for finding out how efficiently such network printer TCP can consume segments it's receiving once FRTO related "fuzzy" ordering effects are removed, for comparison purposes, mostly interesting and that's for hc tcp guy like me :-)). Then one question for DaveM: What I'm not fully sure of, is do we want this workaround to be a sysctl or unconditionally enabled which causes potentially up to two unnecessary retransmissions? With SACK one or both of them will get SACKed before they get retransmitted (both cases have common scenarios). (I made that workaround patch for 2.6.24.1, so YMMV if you just plainly try to apply it to net-2.6). -- i. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem (Was: Re: This has a work around) 2008-05-12 19:12 ` Ilpo Järvinen @ 2008-05-12 19:18 ` Damon L. Chesser 2008-05-12 19:25 ` Ilpo Järvinen 2008-05-12 22:48 ` Fix FRTO+NewReno problem David Miller 1 sibling, 1 reply; 24+ messages in thread From: Damon L. Chesser @ 2008-05-12 19:18 UTC (permalink / raw) To: Ilpo Järvinen; +Cc: David Miller, Netdev Ilpo Järvinen wrote: > On Mon, 12 May 2008, Damon L. Chesser wrote: > >> Ilpo Järvinen wrote: >> > On Mon, 12 May 2008, Damon L. Chesser wrote: >> > >> > > I applied the patches in order, no errors on that. I compiled a >> stock >> > > 2.4.24-1 kernel with the patches, I saw no errors there. >> > > >> > > booted into new kernel, printed with tcp_frto=0. set tcp_frto =2, >> > > restarted >> > > the network (is this required, or is this a dynamic setting?), >> printed >> > > from OO >> > > document. No joy. tcpdump log attached (almost 15 min. worth of >> data) >> > > >> > > If you want, I can re-compile and double check for any >> compilation errors, >> > > however, if there were any, it was not sever enough to stop the >> > > compilation. >> > >> > On the bright side, the FRTO problem that was occuring previously >> is now >> > fixed but there seems to be very few ways to communicate with that >> device >> > sanely because it assumes in-order arrival and keeps discarding, as it >> > seems, _all_ other segments... If you could try with this additional >> > work-around attached (keep the fixes there as well). Turn >> > tcp_frto_inorder_workaround sysctl to 1 before testing with FRTO. >> > >> > ...Can you please send a dump about working case too, this seems >> rather >> > nasty device to work with (tcp_frto = 0 is enough to attain it, no >> need to >> > have another kernel booted for that) and I'm interested to see what >> are the >> > loss rates without FRTO... >> > >> > >> New patch added in with the first two, tcp_frto_inorder_workaround =1 >> test >> printed 5 pages: This worked. Attached is the output of tcpdump. Need >> anything else? > > Thanks a lot for the testing & all. The picture is clear enough > already, so no additional help needed (I haven't yet looked the > non-frto dump but > I doubt anything earth-shattering turns out, it's mostly interesting > for finding out how efficiently such network printer TCP can consume > segments it's receiving once FRTO related "fuzzy" ordering effects are > removed, for comparison purposes, mostly interesting and that's for hc > tcp guy like me :-)). > > > Then one question for DaveM: > > What I'm not fully sure of, is do we want this workaround to be a > sysctl or unconditionally enabled which causes potentially up to two > unnecessary retransmissions? With SACK one or both of them will get > SACKed before they get retransmitted (both cases have common > scenarios). (I made that workaround patch for 2.6.24.1, so YMMV if you > just plainly try to apply it to net-2.6). > One more question from me, what release will we see these updates in, 2.6.26? -- Damon L. Chesser damon@damtek.com http://www.linkedin.com/in/dchesser ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem (Was: Re: This has a work around) 2008-05-12 19:18 ` Damon L. Chesser @ 2008-05-12 19:25 ` Ilpo Järvinen 0 siblings, 0 replies; 24+ messages in thread From: Ilpo Järvinen @ 2008-05-12 19:25 UTC (permalink / raw) To: Damon L. Chesser; +Cc: David Miller, Netdev On Mon, 12 May 2008, Damon L. Chesser wrote: > One more question from me, what release will we see these updates in, 2.6.26? Yes, 2.6.26 will have it. They will also be included to stable-2.6.25, though I currently don't know yet what the minormost number will then be, likely .4 or .5 but like I said I don't know yet :-). I don't know if 2.6.24.y will get some releases, if it does, it might come there as well. ...And about when distros acquire it to their kernel, I don't really have any idea :-). -- i. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem 2008-05-12 19:12 ` Ilpo Järvinen 2008-05-12 19:18 ` Damon L. Chesser @ 2008-05-12 22:48 ` David Miller 2008-05-13 9:42 ` Ilpo Järvinen 1 sibling, 1 reply; 24+ messages in thread From: David Miller @ 2008-05-12 22:48 UTC (permalink / raw) To: ilpo.jarvinen; +Cc: damon, netdev From: "Ilpo_Järvinen" <ilpo.jarvinen@helsinki.fi> Date: Mon, 12 May 2008 22:12:47 +0300 (EEST) > Then one question for DaveM: > > What I'm not fully sure of, is do we want this workaround to be a sysctl > or unconditionally enabled which causes potentially up to two unnecessary > retransmissions? With SACK one or both of them will get SACKed before they > get retransmitted (both cases have common scenarios). (I made that > workaround patch for 2.6.24.1, so YMMV if you just plainly try to apply it > to net-2.6). I think we will have to handle this behavior, unconditionally, all the time. Don't even add the sysctl. If I understand correctly, it is only non-SACK newreno case that can get the unnecessary retransmissions, right? If so, it's not a big deal at all. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem 2008-05-12 22:48 ` Fix FRTO+NewReno problem David Miller @ 2008-05-13 9:42 ` Ilpo Järvinen 2008-05-13 9:49 ` David Miller 0 siblings, 1 reply; 24+ messages in thread From: Ilpo Järvinen @ 2008-05-13 9:42 UTC (permalink / raw) To: David Miller; +Cc: damon, Netdev [-- Attachment #1: Type: TEXT/PLAIN, Size: 1452 bytes --] On Mon, 12 May 2008, David Miller wrote: > From: "Ilpo_Järvinen" <ilpo.jarvinen@helsinki.fi> > Date: Mon, 12 May 2008 22:12:47 +0300 (EEST) > > > Then one question for DaveM: > > > > What I'm not fully sure of, is do we want this workaround to be a sysctl > > or unconditionally enabled which causes potentially up to two unnecessary > > retransmissions? With SACK one or both of them will get SACKed before they > > get retransmitted (both cases have common scenarios). (I made that > > workaround patch for 2.6.24.1, so YMMV if you just plainly try to apply it > > to net-2.6). > > I think we will have to handle this behavior, unconditionally, all > the time. Don't even add the sysctl. Yes, that's what I was thinking in the second option. > If I understand correctly, it is only non-SACK newreno case that can > get the unnecessary retransmissions, right? If so, it's not a big > deal at all. No, also SACK could get at least one quite easily if TCP has a small window, to get two of them with SACK one needs to have a stranger case unless I missed something. If I just remove that forward transmission LOST marking avoidance unconditionally, it fixes that SACKFRTO window of failure too which was a pending thing to fix :-). Btw, the first two fixes (workaround was not included) also fixed bugzilla #10063, I just got a confirmation about that as well. I'll send the two patches still pending separately in a minute. -- i. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem 2008-05-13 9:42 ` Ilpo Järvinen @ 2008-05-13 9:49 ` David Miller 2008-05-13 10:04 ` Ilpo Järvinen 0 siblings, 1 reply; 24+ messages in thread From: David Miller @ 2008-05-13 9:49 UTC (permalink / raw) To: ilpo.jarvinen; +Cc: damon, netdev From: "Ilpo_Järvinen" <ilpo.jarvinen@helsinki.fi> Date: Tue, 13 May 2008 12:42:03 +0300 (EEST) > On Mon, 12 May 2008, David Miller wrote: > > > If I understand correctly, it is only non-SACK newreno case that can > > get the unnecessary retransmissions, right? If so, it's not a big > > deal at all. > > No, also SACK could get at least one quite easily if TCP has a small > window, to get two of them with SACK one needs to have a stranger case > unless I missed something. If I just remove that forward transmission LOST > marking avoidance unconditionally, it fixes that SACKFRTO window of > failure too which was a pending thing to fix :-). Ok. > Btw, the first two fixes (workaround was not included) also fixed bugzilla > #10063, I just got a confirmation about that as well. > > I'll send the two patches still pending separately in a minute. Thank you. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem 2008-05-13 9:49 ` David Miller @ 2008-05-13 10:04 ` Ilpo Järvinen 2008-05-13 10:08 ` David Miller 0 siblings, 1 reply; 24+ messages in thread From: Ilpo Järvinen @ 2008-05-13 10:04 UTC (permalink / raw) To: David Miller; +Cc: Netdev [-- Attachment #1: Type: TEXT/PLAIN, Size: 1258 bytes --] On Tue, 13 May 2008, David Miller wrote: > From: "Ilpo_Järvinen" <ilpo.jarvinen@helsinki.fi> > Date: Tue, 13 May 2008 12:42:03 +0300 (EEST) > > > On Mon, 12 May 2008, David Miller wrote: > > > > > If I understand correctly, it is only non-SACK newreno case that can > > > get the unnecessary retransmissions, right? If so, it's not a big > > > deal at all. > > > > No, also SACK could get at least one quite easily if TCP has a small > > window, to get two of them with SACK one needs to have a stranger case > > unless I missed something. If I just remove that forward transmission LOST > > marking avoidance unconditionally, it fixes that SACKFRTO window of > > failure too which was a pending thing to fix :-). > > Ok. Hmm, my paragraph was likely ambiguous. So unnecessary transmissions can occur with SACK with that change but another problem that would have required a fix went away with that change. > > Btw, the first two fixes (workaround was not included) also fixed bugzilla > > #10063, I just got a confirmation about that as well. > > > > I'll send the two patches still pending separately in a minute. > > Thank you. Forgot to mention earlier, these should go to stable too (but you probably figured that out anyway). -- i. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: Fix FRTO+NewReno problem 2008-05-13 10:04 ` Ilpo Järvinen @ 2008-05-13 10:08 ` David Miller 0 siblings, 0 replies; 24+ messages in thread From: David Miller @ 2008-05-13 10:08 UTC (permalink / raw) To: ilpo.jarvinen; +Cc: netdev From: "Ilpo_Järvinen" <ilpo.jarvinen@helsinki.fi> Date: Tue, 13 May 2008 13:04:04 +0300 (EEST) > Forgot to mention earlier, these should go to stable too (but you > probably figured that out anyway). Yep, I understood that :-) ^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2008-05-13 10:08 UTC | newest]
Thread overview: 24+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <48207F06.50306@damtek.com>
[not found] ` <Pine.LNX.4.64.0805061912350.8965@wrl-59.cs.helsinki.fi>
[not found] ` <4821C37A.7040306@damtek.com>
2008-05-07 22:26 ` Fix FRTO+NewReno problem (Was: Re: This has a work around) Ilpo Järvinen
2008-05-08 8:10 ` Fix FRTO+NewReno problem David Miller
2008-05-08 16:50 ` Bug#478062: Fix FRTO+NewReno problem (Was: Re: This has a work around) Damon L. Chesser
2008-05-08 17:05 ` Damon L. Chesser
2008-05-08 18:16 ` Damon L. Chesser
2008-05-08 20:42 ` Ilpo Järvinen
2008-05-12 10:08 ` Ilpo Järvinen
[not found] ` <4828279C.3010102@damtek.com>
2008-05-12 11:32 ` Ilpo Järvinen
2008-05-12 11:55 ` Damon L. Chesser
2008-05-12 12:07 ` Ilpo Järvinen
2008-05-12 13:44 ` Damon L. Chesser
2008-05-12 14:35 ` Ilpo Järvinen
2008-05-12 16:40 ` Ilpo Järvinen
2008-05-12 17:02 ` Damon L. Chesser
2008-05-12 18:23 ` Ilpo Järvinen
2008-05-12 18:39 ` Damon L. Chesser
2008-05-12 19:12 ` Ilpo Järvinen
2008-05-12 19:18 ` Damon L. Chesser
2008-05-12 19:25 ` Ilpo Järvinen
2008-05-12 22:48 ` Fix FRTO+NewReno problem David Miller
2008-05-13 9:42 ` Ilpo Järvinen
2008-05-13 9:49 ` David Miller
2008-05-13 10:04 ` Ilpo Järvinen
2008-05-13 10:08 ` David Miller
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).