Excessive TCP retransmits over lossless, high latency link

Excessive TCP retransmits over lossless, high latency link

[Jamie Lokier]

[Oops, don't group-reply to the other one which I sent to
vger.rutgers.edu by mistake :-)]

First let me describe the link.

The kernel is standard 2.4.7, i686 on a laptop, all standard settings.

I have been downloading a large email over a "9600 baud" mobile phone
link, over infrared (from phone to computer).  Protocols: PPP over
IRComm over IRDA at 115200 baud.  The modem is in the phone (the phone
responds to AT commands and provides data, over the IRComm protocol).
The phone is a Nokia 6210e.

Packets arrive _very_ slowly -- about 1 packet every 5.2 seconds.
(Showing that "9600 baud" is even worse false advertising than "56k").

The ping time during this download is ~23 seconds.  I've seen it go up
to >60 seconds.  When the link is idle, the ping time oscillates beteen
1 second and 1.5 seconds.  All the responses seem to arrive eventually,
and in order.

(The irdaping time is about 0.2 seconds, when the IRDA link is idle.
When the link is active, irdaping doesn't see any response, except for
once when the call terminated, perhaps due to irdaping.  At least, we
see that the IRDA link time is negligable).

So we can see that the mail download is acting to fill up the router
queues.  Everyone here knows that TCP does tend to fill router queues,
even when that pessimises performance.  Quite why anybody designs
routers with 60 second queues is unclear to me, but they do.

Even with this, I can see that the large mail is downloading far slower
than it ought to.

The appended "tcpdump -i ppp0 -n" trace shows an excessive number of
retransmits from the remote POP server.  The retransmits are triggered
by excessive duplicate ACKs from the local client.  By excessive, I mean
lots of retransmits of the same data.

The interesting thing is that there isn't any evidence of packet loss.

It appears there is a stable cycle of duplicate ACKs and duplicate
retransmits which doesn't go away, even without packet loss.

I think this is fixable problem at the Linux client end.  Linux is
sending many duplicate ACKs in response to data that it has already
received.  Normally, duplicate ACKs are useful to indicate that a packet
is missing and should be retransmitted.  Those ACKs have a sequence
number that is higher than the packet which triggers them.

In this case, there is no missing data, and all the duplicate ACKs do is
cause the server to resend the same data, repeatedly and redundantly.

If this can happen over a low loss but very slow and high latency link,
I guess it can happen over normal links too, but I haven't seen an example.

Is there some /proc/sys setting to fix this, a kernel patch, or is it
perhaps fixed in a newer kernel already?

thanks,
-- Jamie

ps. please don't send me any large emails ATM, thx, they take about 30
minutes per 100k to download :-)


17:39:38.872919 < 194.217.242.22.pop3 > 193.237.65.219.32772: . 4294964376:4294965836(1460) ack 1 win 8760 (DF)
17:39:38.873003 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 1:1(0) ack 1460 win 62780 (DF)
17:39:44.122914 < 194.217.242.22.pop3 > 193.237.65.219.32772: . 4294965836:0(1460) ack 1 win 8760 (DF)
17:39:44.123006 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 1:1(0) ack 1460 win 62780 (DF)
17:39:49.372887 < 194.217.242.22.pop3 > 193.237.65.219.32772: . 0:1460(1460) ack 1 win 8760 (DF)
17:39:49.372986 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 1:1(0) ack 1460 win 62780 (DF)
17:39:54.612626 < 194.217.242.22.pop3 > 193.237.65.219.32772: . 1460:2920(1460) ack 1 win 8760 (DF)
17:39:54.612712 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 1:1(0) ack 2920 win 62780 (DF)
17:39:57.742873 < 194.217.242.22.pop3 > 193.237.65.219.32772: . 2920:3812(892) ack 1 win 8760 (DF)
17:39:57.742962 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 1:1(0) ack 3812 win 62780 (DF)
17:40:03.482921 < 194.217.242.22.pop3 > 193.237.65.219.32772: . 3812:5272(1460) ack 1 win 8760 (DF)
17:40:03.483010 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 1:1(0) ack 5272 win 62780 (DF)
17:40:08.732901 < 194.217.242.22.pop3 > 193.237.65.219.32772: . 1460:2920(1460) ack 1 win 8760 (DF)
17:40:08.732991 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 1:1(0) ack 5272 win 62780 (DF)
17:40:13.972868 < 194.217.242.22.pop3 > 193.237.65.219.32772: . 1460:2920(1460) ack 1 win 8760 (DF)
17:40:13.972954 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 1:1(0) ack 5272 win 62780 (DF)
17:40:19.712883 < 194.217.242.22.pop3 > 193.237.65.219.32772: . 1460:2920(1460) ack 1 win 8760 (DF)
17:40:19.712975 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 1:1(0) ack 5272 win 62780 (DF)
17:40:24.923064 < 194.217.242.22.pop3 > 193.237.65.219.32772: . 1460:2920(1460) ack 1 win 8760 (DF)
17:40:24.923157 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 1:1(0) ack 5272 win 62780 (DF)
17:40:30.102989 < 194.217.242.22.pop3 > 193.237.65.219.32772: . 2920:4380(1460) ack 1 win 8760 (DF)
17:40:30.103082 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 1:1(0) ack 5272 win 62780 (DF)
17:40:35.312885 < 194.217.242.22.pop3 > 193.237.65.219.32772: . 3812:5272(1460) ack 1 win 8760 (DF)
17:40:35.312978 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 1:1(0) ack 5272 win 62780 (DF)
17:40:40.522880 < 194.217.242.22.pop3 > 193.237.65.219.32772: P 5272:6671(1399) ack 1 win 8760 (DF)
17:40:40.522968 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 1:1(0) ack 6671 win 62780 (DF)
17:40:40.632511 > 193.237.65.219.32772 > 194.217.242.22.pop3: P 1:10(9) ack 6671 win 62780 (DF)
17:40:45.301126 < 194.217.242.22.pop3 > 193.237.65.219.32772: P 5272:6671(1399) ack 1 win 8760 (DF)
17:40:45.301236 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 10:10(0) ack 6671 win 62780 (DF)
17:40:50.503077 < 194.217.242.22.pop3 > 193.237.65.219.32772: P 5272:6671(1399) ack 1 win 8760 (DF)
17:40:50.503179 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 10:10(0) ack 6671 win 62780 (DF)
17:40:55.723023 < 194.217.242.22.pop3 > 193.237.65.219.32772: P 5272:6671(1399) ack 1 win 8760 (DF)
17:40:55.723123 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 10:10(0) ack 6671 win 62780 (DF)
17:40:57.402665 > 193.237.65.219.32772 > 194.217.242.22.pop3: P 1:10(9) ack 6671 win 62780 (DF)
17:41:00.522888 < 194.217.242.22.pop3 > 193.237.65.219.32772: P 5272:6671(1399) ack 1 win 8760 (DF)
17:41:00.523004 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 10:10(0) ack 6671 win 62780 (DF)
17:41:00.522899 < 194.217.242.22.pop3 > 193.237.65.219.32772: P 6671:6677(6) ack 10 win 8760 (DF)
17:41:00.523073 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 10:10(0) ack 6677 win 62780 (DF)
17:41:00.523535 > 193.237.65.219.32772 > 194.217.242.22.pop3: P 10:27(17) ack 6677 win 62780 (DF)
17:41:00.542914 < 194.217.242.22.pop3 > 193.237.65.219.32772: P 6671:6677(6) ack 10 win 8760 (DF)
17:41:00.543035 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 27:27(0) ack 6677 win 62780 (DF)
17:41:00.542921 < 194.217.242.22.pop3 > 193.237.65.219.32772: P 6671:6677(6) ack 10 win 8760 (DF)
17:41:00.543091 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 27:27(0) ack 6677 win 62780 (DF)
17:41:00.560282 < 194.217.242.22.pop3 > 193.237.65.219.32772: P 6671:6677(6) ack 10 win 8760 (DF)
17:41:00.560382 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 27:27(0) ack 6677 win 62780 (DF)
17:41:06.792902 < 194.217.242.22.pop3 > 193.237.65.219.32772: P 6677:8137(1460) ack 27 win 8760 (DF)
17:41:06.793004 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 27:27(0) ack 8137 win 62780 (DF)
17:41:08.882933 < 194.217.242.22.pop3 > 193.237.65.219.32772: P 8137:8706(569) ack 27 win 8760 (DF)
17:41:08.883022 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 27:27(0) ack 8706 win 62780 (DF)
17:41:08.904657 > 193.237.65.219.32772 > 194.217.242.22.pop3: P 27:36(9) ack 8706 win 62780 (DF)
17:41:13.592879 < 194.217.242.22.pop3 > 193.237.65.219.32772: P 6677:8137(1460) ack 27 win 8760 (DF)
17:41:13.592967 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 36:36(0) ack 8706 win 62780 (DF)
17:41:18.819646 < 194.217.242.22.pop3 > 193.237.65.219.32772: P 6677:8137(1460) ack 27 win 8760 (DF)
17:41:18.819733 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 36:36(0) ack 8706 win 62780 (DF)
17:41:20.912868 < 194.217.242.22.pop3 > 193.237.65.219.32772: P 8137:8706(569) ack 27 win 8760 (DF)
17:41:20.912983 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 36:36(0) ack 8706 win 62780 (DF)
17:41:20.912875 < 194.217.242.22.pop3 > 193.237.65.219.32772: P 8706:8712(6) ack 36 win 8760 (DF)
17:41:20.913050 > 193.237.65.219.32772 > 194.217.242.22.pop3: . 36:36(0) ack 8712 win 62780 (DF)
17:41:20.912884 < 194.217.242.22.pop3 > 193.237.65.219.32772: P 8706:8712(6) ack 36 win 8760 (DF)

Re: Excessive TCP retransmits over lossless, high latency link

[Andi Kleen]

On Sat, Sep 01, 2001 at 07:17:29PM +0200, Jamie Lokier wrote:
> The appended "tcpdump -i ppp0 -n" trace shows an excessive number of
> retransmits from the remote POP server.  The retransmits are triggered
> by excessive duplicate ACKs from the local client.  By excessive, I mean
> lots of retransmits of the same data.

The duplicate ACKs should not cause any retransmits (unless the sender
is badly broken), because they contain a high enough ACK number.

The problem is really that a TCP sender cannot recover from a too short RTT 
estimate; if the RTT is longer and it doesn't get any acks it'll assume 
packet loss and never has a chance to find out about the longer RTT, because
that only works with new ACKs. 

The standard initial RTT is 3 seconds; but your RTT is 5.2s. 

What you can do is to change the initial RTT on the sender side. On Linux
it can be done with the "irtt" option of route or the equivalent one of
iproute2. Most other OS have a similar way to configure the IRTT. 

-Andi

Re: Excessive TCP retransmits over lossless, high latency link

[kuznet]

Hello!

> The interesting thing is that there isn't any evidence of packet loss.

Why did you disable bith sacks and timestamps? Exactly to get
maximal damage from long delay link?


> Is there some /proc/sys setting to fix this, a kernel patch, or is it
> perhaps fixed in a newer kernel already?

No patches to block send required ACKs exist of course. :-)

All the problem is at sender, it mispredicts rtt.
What OS is sender? If it is linux too, try to use default configuration
not playing with /proc/sys/net/tcp_*, especially with timestamps
and sacks and the situation should rectify.

Also, please, send full (binary!) tcpdump from SYN and to FIN.
Andi says right thing, but I am still puzzled why rtt is miscalculated
it should be estimated correctly.

Well, and if sender is not linux... no ideas.

Alexey

Re: Excessive TCP retransmits over lossless, high latency link

[kuznet]

Hello!

> packet loss and never has a chance to find out about the longer RTT, because
> that only works with new ACKs. 

Karn algo calculates right rtt after log(rtt/3sec) retransmits.
It is not so big number. In this case, it is 1 in fact. The only effect
of irtt setting is avoiding one retransmission.

Alexey

Re: Excessive TCP retransmits over lossless, high latency link

[Jamie Lokier]

Andi Kleen wrote:
> On Sat, Sep 01, 2001 at 07:17:29PM +0200, Jamie Lokier wrote:
> > The appended "tcpdump -i ppp0 -n" trace shows an excessive number of
> > retransmits from the remote POP server.  The retransmits are triggered
> > by excessive duplicate ACKs from the local client.  By excessive, I mean
> > lots of retransmits of the same data.
> 
> The duplicate ACKs should not cause any retransmits (unless the sender
> is badly broken), because they contain a high enough ACK number.

We are receiving frames with sequence number <= the ACK we're sending.

When the remote end receives those ACKs, it is seeing a series of
duplicate ACKs for data it sent about 5 frames ago.  So of course it
retransmits the data starting from 5 frames ago.  As it receives a whole
series of the same duplicate ACK, it retransmits from the same place
each time.

I don't see what is broken about the remote end in this case.

> The problem is really that a TCP sender cannot recover from a too short RTT 
> estimate; if the RTT is longer and it doesn't get any acks it'll assume 
> packet loss and never has a chance to find out about the longer RTT, because
> that only works with new ACKs. 
> 
> The standard initial RTT is 3 seconds; but your RTT is 5.2s. 

Actually, 5.2s is the most common packet interarrival time.  The RTT is
more like 23s!

> What you can do is to change the initial RTT on the sender side. On Linux
> it can be done with the "irtt" option of route or the equivalent one of
> iproute2. Most other OS have a similar way to configure the IRTT. 

Unfortunately I cannot change the IRTT on my ISP's mail server, or for
that matter on anyone's web server.

Are you saying that the IRTT must be larger than the actual RTT, and the
estimates can only go down?  I thought IRTT was a rough guess and it
was expected to go either up or down.

-- Jamie

Re: Excessive TCP retransmits over lossless, high latency link

[Jamie Lokier]

kuznet@ms2.inr.ac.ru wrote:
> > The interesting thing is that there isn't any evidence of packet loss.
> 
> Why did you disable bith sacks and timestamps? Exactly to get
> maximal damage from long delay link?
>
> What OS is sender? If it is linux too, try to use default configuration
> not playing with /proc/sys/net/tcp_*, especially with timestamps
> and sacks and the situation should rectify.

Unfortunately the other machine is my ISP, who I do not control.  I have
no idea what the sender OS is, but it's obviously not a standard recent
Linux judging from the lack of SACK.

> Also, please, send full (binary!) tcpdump from SYN and to FIN.

I have send myself a 30k mail and produced a trace of receiving it.  A
binary trace with "tcpdump -n -w" is attached, and a text version of it
with "tcpdump -r pktlog2" is appended below.

> Well, and if sender is not linux... no ideas.

I am wondering if not sending so many duplicate ACKs would help the
broken sender (I know, that is against RFC793 but hey if it works...).

For the sake of a Linux test, I shall try proxying this stream via a
Linux 2.4.2 box I have on the net, and see if it is an improvement.

-- Jamie

Re: Excessive TCP retransmits over lossless, high latency link

[Neil Spring]

> I don't see what is broken about the remote end in this case.

The remote end is probably not broken, at least in this
case.  This looks like an artifact of a disgustingly
large queue, and a very slow link.  When the time to
transmit tiny packets in the initial handshake is much
smaller than the time to transmit a full size frame,
retransmission timers can get confused.  A complete trace
would settle this.

I strongly recommend setting the mtu of your ppp0 interface
down to 576 (or smaller) to reduce the time it takes to
transfer a full size frame, decrease the likelihood that
frames suffer corruption, and allow acknowledgements more
often than every 5 seconds.  This is a setting in your ppp
configuration, don't do this using ifconfig. 

Don't take my word for it, see RFC1144, section 5.2: a
good MTU is chosen so that a full size frame is transferred
in 200ms.

RFC1144, written by Van Jacobson:

   To illustrate, note that for a 9600 bps line with
   header compression there is essentially no benefit
   in increasing the MTU beyond 200 bytes: If the MTU is
   increased to 576, the average delay increases by 188%
   while throughput only improves by 3% (from 96 to 99%).

Besides, if you do what you propose (not ack old data with the 
next byte expected) you risk stalling the connection entirely.

-neil

Re: Excessive TCP retransmits over lossless, high latency link

[kuznet]

Hello!

> binary trace with "tcpdump -n -w" is attached,

I am afraid you forgot it.


> I am wondering if not sending so many duplicate ACKs would help the
> broken sender (I know, that is against RFC793 but hey if it works...).

If it works, hack kernel not to send dupacks. What is the problem? :-)
Yes, on such links fast retransmit is not useful in any case.

Actually, you can make something from your side: if rtt is so high,
most likely this means excessive queueing. You can fight this
advertising smaller window (f.e. adding option "window N" to route).
If intepacket gap is 5sec and rtt 23, you can divide window by 23/5
and rtt should drop to some more reasonable number. "Theoretical"
bandwidth will reduce, but practical will grow.


> For the sake of a Linux test, I shall try proxying this stream via a
> Linux 2.4.2 box I have on the net, and see if it is an improvement.

Yes, it is intersting.

Alexey

Re: Excessive TCP retransmits over lossless, high latency link

[Alan Cox]

> Packets arrive _very_ slowly -- about 1 packet every 5.2 seconds.
> (Showing that "9600 baud" is even worse false advertising than "56k").

With my 7110 I get close to 8000bps throughput . To get that I ended up
having to fix various flow control setting so the tiny buffers on the 
phone didnt keep overflowing

> The interesting thing is that there isn't any evidence of packet loss.

The modem/phone link does its own retransmit and recovery sequences. That
means that on a very flaky link you _will_ see long pauses with retransmits
at link layer going on

Re: Excessive TCP retransmits over lossless, high latency link

[Jamie Lokier]

[-- Attachment #1: Type: text/plain, Size: 2385 bytes --]

kuznet@ms2.inr.ac.ru wrote:
> > binary trace with "tcpdump -n -w" is attached,
> 
> I am afraid you forgot it.

I have attached various files to this message.  Hopefully the names make
it clear which they are.

> > I am wondering if not sending so many duplicate ACKs would help the
> > broken sender (I know, that is against RFC793 but hey if it works...).
> If it works, hack kernel not to send dupacks. What is the problem? :-)

I hate rebooting :-)

> Yes, on such links fast retransmit is not useful in any case.

Should I turn of /proc/sys/net/ipv4/tcp_fack then?

> Actually, you can make something from your side: if rtt is so high,
> most likely this means excessive queueing.

Yes, definitely.  Btw, I saw a ping round trip time of 162s just now.

> You can fight this
> advertising smaller window (f.e. adding option "window N" to route).
> If intepacket gap is 5sec and rtt 23, you can divide window by 23/5
> and rtt should drop to some more reasonable number. "Theoretical"
> bandwidth will reduce, but practical will grow.
> 
> 
> > For the sake of a Linux test, I shall try proxying this stream via a
> > Linux 2.4.2 box I have on the net, and see if it is an improvement.
> 
> Yes, it is intersting.

I've tried it, with no special settings (mtu or window or sack etc.) at
either end.  Local end is Linux 2.4.7, remote is 2.4.2-2 from Red Hat 7.1.
The remote is proxying a connection to the final mailserver.

I saw very few retransmits in a single message download.  SACK appears
occasionally.  I don't really understand the local reaction to SACK, or
why a SACK option appears in one ACK sent locally and not the following
ACK, even though the SACK mentions data that does not arrive between the
two locally sent ACKs.

The throughput difference was obvious: POP3 negotiation + 30k message +
headers took:

        5 min 31 sec       downloading unknown OS   ->  Linux 2.4.7
        2 min 15 sec       downloading Linux 2.4.2  ->  Linux 2.4.7

Looking at the traces, I expect the improvement on the straight data
part is better than 2:1, because I saw so many retransmits with the
other OS and hardly any with Linux as the server.

If someone would like to do an OS type probe on sdps.demon.co.uk, we
will know which OS should be avoided for this kind of connection ;-)

It is still very slow considering it is advertised as "9600 baud" though
:-(

cheers,
-- Jamie

[-- Attachment #2: pktlog-from-notlinux --]
[-- Type: application/octet-stream, Size: 20267 bytes --]

[-- Attachment #3: pktlog-from-notlinux.txt --]
[-- Type: text/plain, Size: 16849 bytes --]

19:43:06.892991 > 193.237.65.219.32855 > 158.152.1.58.domain: 48716+ A? sdps.demon.co.uk. (34) (DF)
19:43:08.952861 < 158.152.1.58.domain > 193.237.65.219.32855: 48716 5/4/4 CNAME pop3.mail.demon.net., A 194.217.242.59, A 194.217.242.22, (266)
19:43:08.989850 > 193.237.65.219.32855 > 158.152.1.58.domain: 48717+ AAAA? sdps.demon.co.uk. (34) (DF)
19:43:10.525449 < 158.152.1.58.domain > 193.237.65.219.32855: 48717 1/1/0 CNAME pop3.mail.demon.net. (114)
19:43:10.526084 > 193.237.65.219.32855 > 158.152.1.58.domain: 48718+ A? sdps.demon.co.uk. (34) (DF)
19:43:12.572859 < 158.152.1.58.domain > 193.237.65.219.32855: 48718 5/4/4 CNAME pop3.mail.demon.net., A 194.217.242.59, A 194.217.242.22, (266)
19:43:12.573433 > 193.237.65.219.32809 > 194.217.242.59.pop3: S 2296392844:2296392844(0) win 5840 <mss 1460,sackOK,timestamp 1404252 0,nop,wscale 0> (DF)
19:43:13.615650 < 194.217.242.59.pop3 > 193.237.65.219.32809: S 2152317245:2152317245(0) ack 2296392845 win 8760 <mss 1460> (DF)
19:43:13.615773 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 1:1(0) ack 1 win 5840 (DF)
19:43:14.632866 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 1:44(43) ack 1 win 8760 (DF)
19:43:14.632961 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 1:1(0) ack 44 win 5840 (DF)
19:43:14.649264 > 193.237.65.219.32809 > 194.217.242.59.pop3: P 1:25(24) ack 44 win 5840 (DF)
19:43:15.672850 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 44:50(6) ack 25 win 8760 (DF)
19:43:15.685004 > 193.237.65.219.32809 > 194.217.242.59.pop3: P 25:40(15) ack 50 win 5840 (DF)
19:43:16.702917 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 50:50(0) ack 40 win 8760 (DF)
19:43:17.203063 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 50:56(6) ack 40 win 8760 (DF)
19:43:17.242650 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 40:40(0) ack 56 win 5840 (DF)
19:43:20.222880 > 193.237.65.219.32809 > 194.217.242.59.pop3: P 40:46(6) ack 56 win 5840 (DF)
19:43:21.233068 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 56:69(13) ack 46 win 8760 (DF)
19:43:21.233354 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 46:46(0) ack 69 win 5840 (DF)
19:43:21.251867 > 193.237.65.219.32809 > 194.217.242.59.pop3: P 46:52(6) ack 69 win 5840 (DF)
19:43:22.272869 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 69:124(55) ack 52 win 8760 (DF)
19:43:22.308400 > 193.237.65.219.32809 > 194.217.242.59.pop3: P 52:58(6) ack 124 win 5840 (DF)
19:43:23.322929 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 124:151(27) ack 58 win 8760 (DF)
19:43:23.349075 > 193.237.65.219.32809 > 194.217.242.59.pop3: P 58:74(16) ack 151 win 5840 (DF)
19:43:27.102700 > 193.237.65.219.32809 > 194.217.242.59.pop3: P 58:74(16) ack 151 win 5840 (DF)
19:43:29.732891 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 151:1611(1460) ack 74 win 8760 (DF)
19:43:29.733159 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 1611 win 8760 (DF)
19:43:34.982965 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 1611:3071(1460) ack 74 win 8760 (DF)
19:43:35.022675 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 3071 win 11680 (DF)
19:43:40.262962 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 3071:4531(1460) ack 74 win 8760 (DF)
19:43:40.263062 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 4531 win 14600 (DF)
19:43:45.512900 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 4531:5991(1460) ack 74 win 8760 (DF)
19:43:45.512991 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 5991 win 17520 (DF)
19:43:45.512905 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 5991:5991(0) ack 74 win 8760 (DF)
19:43:50.752866 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 151:1611(1460) ack 74 win 8760 (DF)
19:43:50.752954 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 5991 win 17520 (DF)
19:43:55.952900 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 1611:3071(1460) ack 74 win 8760 (DF)
19:43:55.952987 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 5991 win 17520 (DF)
19:44:01.162979 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 3071:4531(1460) ack 74 win 8760 (DF)
19:44:01.163072 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 5991 win 17520 (DF)
19:44:06.382817 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 5991:7451(1460) ack 74 win 8760 (DF)
19:44:06.382909 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 7451 win 20440 (DF)
19:44:11.602458 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 3071:4531(1460) ack 74 win 8760 (DF)
19:44:11.602546 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 7451 win 20440 (DF)
19:44:16.812904 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 4531:5991(1460) ack 74 win 8760 (DF)
19:44:16.812990 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 7451 win 20440 (DF)
19:44:22.002981 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 5991:7451(1460) ack 74 win 8760 (DF)
19:44:22.003076 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 7451 win 20440 (DF)
19:44:25.642892 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 7451:8342(891) ack 74 win 8760 (DF)
19:44:25.642985 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 8342 win 23360 (DF)
19:44:30.852962 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 5991:7451(1460) ack 74 win 8760 (DF)
19:44:30.853052 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 8342 win 23360 (DF)
19:44:36.062951 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 5991:7451(1460) ack 74 win 8760 (DF)
19:44:36.063046 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 8342 win 23360 (DF)
19:44:41.282732 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 5991:7451(1460) ack 74 win 8760 (DF)
19:44:41.282825 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 8342 win 23360 (DF)
19:44:44.402896 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 7451:8342(891) ack 74 win 8760 (DF)
19:44:44.402995 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 8342 win 23360 (DF)
19:44:49.612909 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 8342:9802(1460) ack 74 win 8760 (DF)
19:44:49.612996 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 9802 win 26280 (DF)
19:44:54.832979 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 9802:11262(1460) ack 74 win 8760 (DF)
19:44:54.833070 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 11262 win 29200 (DF)
19:45:00.052742 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 7451:8911(1460) ack 74 win 8760 (DF)
19:45:00.052831 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 11262 win 29200 (DF)
19:45:05.262884 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 7451:8911(1460) ack 74 win 8760 (DF)
19:45:05.262978 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 11262 win 29200 (DF)
19:45:10.482889 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 8342:9802(1460) ack 74 win 8760 (DF)
19:45:10.482979 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 11262 win 29200 (DF)
19:45:15.702883 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 11262:12722(1460) ack 74 win 8760 (DF)
19:45:15.702975 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 12722 win 32120 (DF)
19:45:20.922616 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 8342:9802(1460) ack 74 win 8760 (DF)
19:45:20.922706 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 12722 win 32120 (DF)
19:45:26.132863 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 8342:9802(1460) ack 74 win 8760 (DF)
19:45:26.132958 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 12722 win 32120 (DF)
19:45:31.352814 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 9802:11262(1460) ack 74 win 8760 (DF)
19:45:31.352898 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 12722 win 32120 (DF)
19:45:36.572622 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 12722:14182(1460) ack 74 win 8760 (DF)
19:45:36.572709 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 14182 win 35040 (DF)
19:45:41.772897 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 14182:15642(1460) ack 74 win 8760 (DF)
19:45:41.772985 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 15642 win 37960 (DF)
19:45:46.982994 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 11262:12722(1460) ack 74 win 8760 (DF)
19:45:46.983083 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 15642 win 37960 (DF)
19:45:50.102890 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 15642:16533(891) ack 74 win 8760 (DF)
19:45:50.102989 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 16533 win 37960 (DF)
19:45:55.328662 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 11262:12722(1460) ack 74 win 8760 (DF)
19:45:55.328755 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 16533 win 37960 (DF)
19:46:00.534606 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 12722:14182(1460) ack 74 win 8760 (DF)
19:46:00.534691 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 16533 win 37960 (DF)
19:46:05.777455 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 16533:17993(1460) ack 74 win 8760 (DF)
19:46:05.777543 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 17993 win 40880 (DF)
19:46:11.002902 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 17993:19453(1460) ack 74 win 8760 (DF)
19:46:11.002992 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 19453 win 43800 (DF)
19:46:16.252870 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 12722:14182(1460) ack 74 win 8760 (DF)
19:46:16.252965 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 19453 win 43800 (DF)
19:46:21.482878 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 12722:14182(1460) ack 74 win 8760 (DF)
19:46:21.482967 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 19453 win 43800 (DF)
19:46:26.732886 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 14182:15642(1460) ack 74 win 8760 (DF)
19:46:26.732977 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 19453 win 43800 (DF)
19:46:31.982906 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 15642:17102(1460) ack 74 win 8760 (DF)
19:46:31.982992 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 19453 win 43800 (DF)
19:46:37.234530 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 16533:17993(1460) ack 74 win 8760 (DF)
19:46:37.234617 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 19453 win 43800 (DF)
19:46:42.472760 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 17993:19453(1460) ack 74 win 8760 (DF)
19:46:42.472856 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 19453 win 43800 (DF)
19:46:47.687936 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 19453:20913(1460) ack 74 win 8760 (DF)
19:46:47.688029 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 20913 win 46720 (DF)
19:46:52.932888 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 20913:22373(1460) ack 74 win 8760 (DF)
19:46:52.932976 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 22373 win 49640 (DF)
19:46:58.172866 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 19453:20913(1460) ack 74 win 8760 (DF)
19:46:58.172953 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 22373 win 49640 (DF)
19:47:03.402904 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 22373:23833(1460) ack 74 win 8760 (DF)
19:47:03.402993 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 23833 win 52560 (DF)
19:47:08.652863 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 19453:20913(1460) ack 74 win 8760 (DF)
19:47:08.652952 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 23833 win 52560 (DF)
19:47:14.414511 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 19453:20913(1460) ack 74 win 8760 (DF)
19:47:14.414604 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 23833 win 52560 (DF)
19:47:19.622913 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 20913:22373(1460) ack 74 win 8760 (DF)
19:47:19.623003 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 23833 win 52560 (DF)
19:47:23.282880 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 23833:24725(892) ack 74 win 8760 (DF)
19:47:23.282980 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 24725 win 52560 (DF)
19:47:28.522888 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 22373:23833(1460) ack 74 win 8760 (DF)
19:47:28.522976 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 24725 win 52560 (DF)
19:47:34.272886 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 24725:26185(1460) ack 74 win 8760 (DF)
19:47:34.272981 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 26185 win 55480 (DF)
19:47:40.002881 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 23833:25293(1460) ack 74 win 8760 (DF)
19:47:40.002971 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 26185 win 55480 (DF)
19:47:45.202902 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 26185:27645(1460) ack 74 win 8760 (DF)
19:47:45.202998 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 27645 win 58400 (DF)
19:47:50.394404 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 27645:29105(1460) ack 74 win 8760 (DF)
19:47:50.394500 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 29105 win 61320 (DF)
19:47:56.121837 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 23833:25293(1460) ack 74 win 8760 (DF)
19:47:56.121944 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 29105 win 61320 (DF)
19:48:01.302918 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 24725:26185(1460) ack 74 win 8760 (DF)
19:48:01.303011 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 29105 win 61320 (DF)
19:48:07.022900 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 24725:26185(1460) ack 74 win 8760 (DF)
19:48:07.022992 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 29105 win 61320 (DF)
19:48:12.212914 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 26185:27645(1460) ack 74 win 8760 (DF)
19:48:12.213001 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 29105 win 61320 (DF)
19:48:17.432888 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 27645:29105(1460) ack 74 win 8760 (DF)
19:48:17.432985 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 29105 win 61320 (DF)
19:48:22.642896 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 29105:30565(1460) ack 74 win 8760 (DF)
19:48:22.642984 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 30565 win 62780 (DF)
19:48:24.212871 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 30565:31088(523) ack 74 win 8760 (DF)
19:48:24.212966 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 74:74(0) ack 31088 win 62780 (DF)
19:48:24.245885 > 193.237.65.219.32809 > 194.217.242.59.pop3: P 74:82(8) ack 31088 win 62780 (DF)
19:48:29.452732 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 29105:30565(1460) ack 74 win 8760 (DF)
19:48:29.452827 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 82:82(0) ack 31088 win 62780 (DF)
19:48:31.762704 > 193.237.65.219.32809 > 194.217.242.59.pop3: P 74:82(8) ack 31088 win 62780 (DF)
19:48:34.882885 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 29105:30565(1460) ack 74 win 8760 (DF)
19:48:34.882977 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 82:82(0) ack 31088 win 62780 (DF)
19:48:36.452866 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 30565:31088(523) ack 74 win 8760 (DF)
19:48:36.452978 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 82:82(0) ack 31088 win 62780 (DF)
19:48:36.452873 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 31088:31094(6) ack 82 win 8760 (DF)
19:48:36.453045 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 82:82(0) ack 31094 win 62780 (DF)
19:48:36.464298 > 193.237.65.219.32809 > 194.217.242.59.pop3: P 82:88(6) ack 31094 win 62780 (DF)
19:48:36.474321 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 31088:31094(6) ack 82 win 8760 (DF)
19:48:36.474421 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 88:88(0) ack 31094 win 62780 (DF)
19:48:36.482923 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 31094:31094(0) ack 82 win 8760 (DF)
19:48:36.502875 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 31088:31094(6) ack 82 win 8760 (DF)
19:48:36.502936 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 88:88(0) ack 31094 win 62780 (DF)
19:48:36.502880 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 31088:31094(6) ack 82 win 8760 (DF)
19:48:36.502992 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 88:88(0) ack 31094 win 62780 (DF)
19:48:37.546707 < 194.217.242.59.pop3 > 193.237.65.219.32809: P 31094:31100(6) ack 88 win 8760 (DF)
19:48:37.546812 > 193.237.65.219.32809 > 194.217.242.59.pop3: . 88:88(0) ack 31100 win 62780 (DF)
19:48:37.546714 < 194.217.242.59.pop3 > 193.237.65.219.32809: F 31100:31100(0) ack 88 win 8760 (DF)
19:48:37.556924 > 193.237.65.219.32809 > 194.217.242.59.pop3: F 88:88(0) ack 31101 win 62780 (DF)
19:48:38.571482 < 194.217.242.59.pop3 > 193.237.65.219.32809: . 31101:31101(0) ack 89 win 8760 (DF)

[-- Attachment #4: pktlog-from-linux242 --]
[-- Type: application/octet-stream, Size: 9909 bytes --]

[-- Attachment #5: pktlog-from-linux242.txt --]
[-- Type: text/plain, Size: 11047 bytes --]

20:49:10.535879 > 193.237.65.219.32857 > 158.152.1.58.domain: 4313+ A? sdps.demon.co.uk. (34) (DF)
20:49:12.607387 < 158.152.1.58.domain > 193.237.65.219.32857: 4313 5/4/4 CNAME pop3.mail.demon.net., A 194.217.242.58, A 194.217.242.59, (266)
20:49:13.112862 < 194.217.242.7.50453 > 193.237.65.219.smtp: S 3979673482:3979673482(0) win 8760 <mss 1460> (DF)
20:49:13.114345 > 193.237.65.219 > 194.217.242.7: icmp: 193.237.65.219 tcp port smtp unreachable [tos 0xc0] 
20:49:13.666321 > 193.237.65.219.32833 > 217.33.12.64.4000: S 2192994338:2192994338(0) win 5840 <mss 1460,sackOK,timestamp 1800361 0,nop,wscale 0> (DF)
20:49:14.689811 < 217.33.12.64.4000 > 193.237.65.219.32833: S 1422559437:1422559437(0) ack 2192994339 win 5792 <mss 1460,sackOK,timestamp 18895656 1800361,nop,wscale 0> (DF)
20:49:14.689916 > 193.237.65.219.32833 > 217.33.12.64.4000: . 1:1(0) ack 1 win 5840 <nop,nop,timestamp 1800463 18895656> (DF)
20:49:15.722871 < 217.33.12.64.4000 > 193.237.65.219.32833: P 1:44(43) ack 1 win 5792 <nop,nop,timestamp 18895765 1800463> (DF)
20:49:15.722963 > 193.237.65.219.32833 > 217.33.12.64.4000: . 1:1(0) ack 44 win 5840 <nop,nop,timestamp 1800567 18895765> (DF)
20:49:15.759946 > 193.237.65.219.32833 > 217.33.12.64.4000: P 1:25(24) ack 44 win 5840 <nop,nop,timestamp 1800570 18895765> (DF)
20:49:16.792932 < 217.33.12.64.4000 > 193.237.65.219.32833: . 44:44(0) ack 25 win 5792 <nop,nop,timestamp 18895869 1800570> (DF)
20:49:16.792940 < 217.33.12.64.4000 > 193.237.65.219.32833: P 44:50(6) ack 25 win 5792 <nop,nop,timestamp 18895871 1800570> (DF)
20:49:16.813410 > 193.237.65.219.32833 > 217.33.12.64.4000: P 25:40(15) ack 50 win 5840 <nop,nop,timestamp 1800676 18895871> (DF)
20:49:17.832858 < 217.33.12.64.4000 > 193.237.65.219.32833: . 50:50(0) ack 40 win 5792 <nop,nop,timestamp 18895977 1800676> (DF)
20:49:18.342863 < 217.33.12.64.4000 > 193.237.65.219.32833: P 50:56(6) ack 40 win 5792 <nop,nop,timestamp 18895986 1800676> (DF)
20:49:18.382680 > 193.237.65.219.32833 > 217.33.12.64.4000: . 40:40(0) ack 56 win 5840 <nop,nop,timestamp 1800833 18895986> (DF)
20:49:21.377055 > 193.237.65.219.32833 > 217.33.12.64.4000: P 40:46(6) ack 56 win 5840 <nop,nop,timestamp 1801132 18895986> (DF)
20:49:22.902855 < 217.33.12.64.4000 > 193.237.65.219.32833: . 56:56(0) ack 46 win 5792 <nop,nop,timestamp 18896469 1801132> (DF)
20:49:23.412846 < 217.33.12.64.4000 > 193.237.65.219.32833: P 56:69(13) ack 46 win 5792 <nop,nop,timestamp 18896471 1801132> (DF)
20:49:23.412935 > 193.237.65.219.32833 > 217.33.12.64.4000: . 46:46(0) ack 69 win 5840 <nop,nop,timestamp 1801336 18896471> (DF)
20:49:23.433965 > 193.237.65.219.32833 > 217.33.12.64.4000: P 46:52(6) ack 69 win 5840 <nop,nop,timestamp 1801338 18896471> (DF)
20:49:24.462871 < 217.33.12.64.4000 > 193.237.65.219.32833: P 69:124(55) ack 52 win 5792 <nop,nop,timestamp 18896638 1801338> (DF)
20:49:24.511822 > 193.237.65.219.32833 > 217.33.12.64.4000: . 52:52(0) ack 124 win 5840 <nop,nop,timestamp 1801445 18896638> (DF)
20:49:24.517953 > 193.237.65.219.32833 > 217.33.12.64.4000: P 52:58(6) ack 124 win 5840 <nop,nop,timestamp 1801446 18896638> (DF)
20:49:25.532880 < 217.33.12.64.4000 > 193.237.65.219.32833: P 124:151(27) ack 58 win 5792 <nop,nop,timestamp 18896752 1801446> (DF)
20:49:25.532976 > 193.237.65.219.32833 > 217.33.12.64.4000: . 58:58(0) ack 151 win 5840 <nop,nop,timestamp 1801548 18896752> (DF)
20:49:25.581259 > 193.237.65.219.32833 > 217.33.12.64.4000: P 58:74(16) ack 151 win 5840 <nop,nop,timestamp 1801552 18896752> (DF)
20:49:26.605437 < 217.33.12.64.4000 > 193.237.65.219.32833: . 151:151(0) ack 74 win 5792 <nop,nop,timestamp 18896860 1801552> (DF)
20:49:32.302918 < 217.33.12.64.4000 > 193.237.65.219.32833: . 151:1599(1448) ack 74 win 5792 <nop,nop,timestamp 18896860 1801552> (DF)
20:49:32.303020 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 1599 win 8688 <nop,nop,timestamp 1802225 18896860> (DF)
20:49:32.302923 < 217.33.12.64.4000 > 193.237.65.219.32833: P 1599:1611(12) ack 74 win 5792 <nop,nop,timestamp 18896860 1801552> (DF)
20:49:32.303081 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 1611 win 8688 <nop,nop,timestamp 1802225 18896860> (DF)
20:49:39.072902 < 217.33.12.64.4000 > 193.237.65.219.32833: . 1611:3059(1448) ack 74 win 5792 <nop,nop,timestamp 18897529 1802225> (DF)
20:49:39.073002 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 3059 win 11584 <nop,nop,timestamp 1802902 18897529> (DF)
20:49:44.319061 < 217.33.12.64.4000 > 193.237.65.219.32833: . 3059:4507(1448) ack 74 win 5792 <nop,nop,timestamp 18897529 1802225> (DF)
20:49:44.319159 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 4507 win 14480 <nop,nop,timestamp 1803426 18897529> (DF)
20:49:49.561886 < 217.33.12.64.4000 > 193.237.65.219.32833: . 4507:5955(1448) ack 74 win 5792 <nop,nop,timestamp 18897529 1802225> (DF)
20:49:49.561979 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 5955 win 17376 <nop,nop,timestamp 1803950 18897529> (DF)
20:49:54.823304 < 217.33.12.64.4000 > 193.237.65.219.32833: . 5955:7403(1448) ack 74 win 5792 <nop,nop,timestamp 18897529 1802225> (DF)
20:49:54.823394 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 7403 win 20272 <nop,nop,timestamp 1804477 18897529> (DF)
20:50:00.072892 < 217.33.12.64.4000 > 193.237.65.219.32833: . 7403:8851(1448) ack 74 win 5792 <nop,nop,timestamp 18898211 1802902> (DF)
20:50:00.072990 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 8851 win 23168 <nop,nop,timestamp 1805002 18898211> (DF)
20:50:05.341269 < 217.33.12.64.4000 > 193.237.65.219.32833: . 8851:10299(1448) ack 74 win 5792 <nop,nop,timestamp 18898211 1802902> (DF)
20:50:05.341362 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 10299 win 26064 <nop,nop,timestamp 1805528 18898211> (DF)
20:50:10.602470 < 217.33.12.64.4000 > 193.237.65.219.32833: P 10299:11747(1448) ack 74 win 5792 <nop,nop,timestamp 18898735 1803426> (DF)
20:50:10.602561 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 11747 win 28960 <nop,nop,timestamp 1806054 18898735> (DF)
20:50:14.272884 < 217.33.12.64.4000 > 193.237.65.219.32833: P 11747:12722(975) ack 74 win 5792 <nop,nop,timestamp 18898735 1803426> (DF)
20:50:14.272983 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 12722 win 31856 <nop,nop,timestamp 1806422 18898735> (DF)
20:50:19.523616 < 217.33.12.64.4000 > 193.237.65.219.32833: . 12722:14170(1448) ack 74 win 5792 <nop,nop,timestamp 18899260 1803950> (DF)
20:50:19.523710 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 14170 win 34752 <nop,nop,timestamp 1806947 18899260> (DF)
20:50:24.785028 < 217.33.12.64.4000 > 193.237.65.219.32833: . 14170:15618(1448) ack 74 win 5792 <nop,nop,timestamp 18899260 1803950> (DF)
20:50:24.785119 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 15618 win 37648 <nop,nop,timestamp 1807473 18899260> (DF)
20:50:30.032878 < 217.33.12.64.4000 > 193.237.65.219.32833: . 15618:17066(1448) ack 74 win 5792 <nop,nop,timestamp 18899787 1804477> (DF)
20:50:30.032969 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 17066 win 40544 <nop,nop,timestamp 1807998 18899787> (DF)
20:50:35.302882 < 217.33.12.64.4000 > 193.237.65.219.32833: . 17066:18514(1448) ack 74 win 5792 <nop,nop,timestamp 18899787 1804477> (DF)
20:50:35.302970 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 18514 win 43440 <nop,nop,timestamp 1808525 18899787> (DF)
20:50:40.562865 < 217.33.12.64.4000 > 193.237.65.219.32833: . 18514:19962(1448) ack 74 win 5792 <nop,nop,timestamp 18900315 1805002> (DF)
20:50:40.562955 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 19962 win 46336 <nop,nop,timestamp 1809051 18900315> (DF)
20:50:45.792976 < 217.33.12.64.4000 > 193.237.65.219.32833: . 19962:21410(1448) ack 74 win 5792 <nop,nop,timestamp 18900315 1805002> (DF)
20:50:45.793066 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 21410 win 49232 <nop,nop,timestamp 1809574 18900315> (DF)
20:50:51.552902 < 217.33.12.64.4000 > 193.237.65.219.32833: P 21410:22858(1448) ack 74 win 5792 <nop,nop,timestamp 18900851 1805528> (DF)
20:50:51.552994 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 22858 win 52128 <nop,nop,timestamp 1810150 18900851> (DF)
20:50:56.792977 < 217.33.12.64.4000 > 193.237.65.219.32833: . 22858:24306(1448) ack 74 win 5792 <nop,nop,timestamp 18900851 1805528> (DF)
20:50:56.793066 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 24306 win 55024 <nop,nop,timestamp 1810674 18900851> (DF)
20:51:02.032865 < 217.33.12.64.4000 > 193.237.65.219.32833: . 24306:25754(1448) ack 74 win 5792 <nop,nop,timestamp 18901377 1806054> (DF)
20:51:02.032955 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 25754 win 57920 <nop,nop,timestamp 1811198 18901377> (DF)
20:51:07.242888 < 217.33.12.64.4000 > 193.237.65.219.32833: . 25754:27202(1448) ack 74 win 5792 <nop,nop,timestamp 18901377 1806054> (DF)
20:51:07.242982 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 27202 win 60816 <nop,nop,timestamp 1811719 18901377> (DF)
20:51:12.962860 < 217.33.12.64.4000 > 193.237.65.219.32833: . 27202:28650(1448) ack 74 win 5792 <nop,nop,timestamp 18901743 1806422> (DF)
20:51:12.962946 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 28650 win 63712 <nop,nop,timestamp 1812291 18901743> (DF)
20:51:18.182724 < 217.33.12.64.4000 > 193.237.65.219.32833: . 28650:30098(1448) ack 74 win 5792 <nop,nop,timestamp 18901743 1806422> (DF)
20:51:18.182819 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 30098 win 63712 <nop,nop,timestamp 1812813 18901743> (DF)
20:51:21.822887 < 217.33.12.64.4000 > 193.237.65.219.32833: P 30098:31062(964) ack 74 win 5792 <nop,nop,timestamp 18902269 1806947> (DF)
20:51:21.822974 > 193.237.65.219.32833 > 217.33.12.64.4000: . 74:74(0) ack 31062 win 63712 <nop,nop,timestamp 1813177 18902269> (DF)
20:51:21.890387 > 193.237.65.219.32833 > 217.33.12.64.4000: P 74:82(8) ack 31062 win 63712 <nop,nop,timestamp 1813183 18902269> (DF)
20:51:22.922519 < 217.33.12.64.4000 > 193.237.65.219.32833: . 31062:31062(0) ack 82 win 5792 <nop,nop,timestamp 18908505 1813183> (DF)
20:51:22.922527 < 217.33.12.64.4000 > 193.237.65.219.32833: P 31062:31068(6) ack 82 win 5792 <nop,nop,timestamp 18908507 1813183> (DF)
20:51:22.922648 > 193.237.65.219.32833 > 217.33.12.64.4000: . 82:82(0) ack 31068 win 63712 <nop,nop,timestamp 1813287 18908507> (DF)
20:51:22.948645 > 193.237.65.219.32833 > 217.33.12.64.4000: P 82:88(6) ack 31068 win 63712 <nop,nop,timestamp 1813289 18908507> (DF)
20:51:23.992468 < 217.33.12.64.4000 > 193.237.65.219.32833: P 31068:31074(6) ack 88 win 5792 <nop,nop,timestamp 18908616 1813289> (DF)
20:51:23.992475 < 217.33.12.64.4000 > 193.237.65.219.32833: F 31074:31074(0) ack 88 win 5792 <nop,nop,timestamp 18908616 1813289> (DF)
20:51:23.992729 > 193.237.65.219.32833 > 217.33.12.64.4000: F 88:88(0) ack 31075 win 63712 <nop,nop,timestamp 1813394 18908616> (DF)
20:51:25.012863 < 217.33.12.64.4000 > 193.237.65.219.32833: . 31075:31075(0) ack 89 win 5792 <nop,nop,timestamp 18908718 1813394> (DF)

Re: Excessive TCP retransmits over lossless, high latency link

[Lukas Beeler]

On Sat, Sep 01, 2001 at 09:02:12PM +0100, Jamie Lokier wrote:
> 
> If someone would like to do an OS type probe on sdps.demon.co.uk, we
> will know which OS should be avoided for this kind of connection ;-)
> 

it's running solaris 2.5 or 2.5.1

-- 
Lukas Beeler                        <lukas.beeler@projectdream.org>
GPG Fingerprint: 8030 1C2F 66C5 9D80 AA31  6604 7D4D 0A67 68D8 B67E

Re: Excessive TCP retransmits over lossless, high latency link

[Andi Kleen]

Lukas Beeler <lukas.beeler@projectdream.org> writes:

> On Sat, Sep 01, 2001 at 09:02:12PM +0100, Jamie Lokier wrote:
> > 
> > If someone would like to do an OS type probe on sdps.demon.co.uk, we
> > will know which OS should be avoided for this kind of connection ;-)
> > 
> 
> it's running solaris 2.5 or 2.5.1

Early unpatched Solaris 2.5 has known problems with its rtt algorithm on 
low bandwidth links.

-Andi

Re: Excessive TCP retransmits over lossless, high latency link

[kuznet]

Hello!

> I hate rebooting :-)

Relax. Really, this will not change anything. After more careful look
into the first tcpdump, I did not find any signs of false fast retransmits.

All the problem is due to wrong rtt estimator at sender.


> > Yes, on such links fast retransmit is not useful in any case.
> 
> Should I turn of /proc/sys/net/ipv4/tcp_fack then?

No. Defaults are defaults, because they are the best defaults. :-)


> Yes, definitely.  Btw, I saw a ping round trip time of 162s just now.

I do not understand, do you share this link with someone or
ping over tcp connection?

If the last is true, reduce window to 4K, maximum 8K. Default 64K, combined
with misconfigured queues and/or with broken error correction is disaster.

Actually, you dump shows that window is not open.


> I saw very few retransmits in a single message download.  SACK appears
> occasionally.  I don't really understand the local reaction to SACK, or
> why a SACK option appears in one ACK sent locally and not the following
> ACK, even though the SACK mentions data that does not arrive between the
> two locally sent ACKs.

But I do not see _any_ sacks in your tcpdumps.


> The throughput difference was obvious: POP3 negotiation + 30k message +
> headers took:
> 
>         5 min 31 sec       downloading unknown OS   ->  Linux 2.4.7
>         2 min 15 sec       downloading Linux 2.4.2  ->  Linux 2.4.7

It is dominated by rtt, one rtt per segment. It is very strange
that cwnd does not want to open. Maybe, it is worth to tcpdump at proxy.

Alexey

Re: Excessive TCP retransmits over lossless, high latency link

[Jamie Lokier]

kuznet@ms2.inr.ac.ru wrote:
> > Yes, definitely.  Btw, I saw a ping round trip time of 162s just now.
> 
> I do not understand, do you share this link with someone or
> ping over tcp connection?

I was doing ping with the TCP connection going.  When there is no TCP
connection, ping time is 1-1.5 seconds.

> > I saw very few retransmits in a single message download.  SACK appears
> > occasionally.  I don't really understand the local reaction to SACK, or
> > why a SACK option appears in one ACK sent locally and not the following
> > ACK, even though the SACK mentions data that does not arrive between the
> > two locally sent ACKs.
> 
> But I do not see _any_ sacks in your tcpdumps.

Sorry, you are right.  I did see an sack, but not in this trace.

> > The throughput difference was obvious: POP3 negotiation + 30k message +
> > headers took:
> > 
> >         5 min 31 sec       downloading unknown OS   ->  Linux 2.4.7
> >         2 min 15 sec       downloading Linux 2.4.2  ->  Linux 2.4.7
> 
> It is dominated by rtt, one rtt per segment. It is very strange
> that cwnd does not want to open. Maybe, it is worth to tcpdump at proxy.

Do you mean that you want to see the Linux -> Linux connection, with
tcpdumps at both ends?

-- Jamie

Re: Excessive TCP retransmits over lossless, high latency link

[kuznet]

Hello!

> Do you mean that you want to see the Linux -> Linux connection, with
> tcpdumps at both ends?

At the end of sender. Receiver side is not so interesting.

Well, if you were able to make it with solaris this would be also interesting.
It would be clear how exactly it miscalculates rtt at least. :-)
But Linux is interesting too. Theoretically it should show better times
even with so short transfer. cwnd stays at 1 and this is very strange.

Alexey