Linux TCP in the presence of delays or drops...

Linux TCP in the presence of delays or drops...

[Oumer Teyeb]

Hi all,

I have some questions regarding Linux TCP in the presence of delays or 
packet drops. It is somehow long mail, but the questions are two or 
three, just wanted to provide a detailed information so that the problem 
is clear. thanx for the patience!!

Best regards,
Oumer

Note that for the traces referred here, SACK,timestamps, and FRTO are 
all disabled...

1) packet drops
================
I have a trace where the tcp sender window is flushed and  then the 
connection speed is  changed from 1Mbps to 384kbps...
The trace files from both the client and the server side can be found at
http://kom.aau.dk/~oumer/drop_0_delay_SERVER.dat
http://kom.aau.dk/~oumer/drop_0_delay_CLIENT.dat
and the tcptrace time sequence curve can be found in
http://kom.aau.dk/~oumer/drop_0_delay.ps

as can be seen from the plot and the trace files at around 
17:19:35.705733, the window was flushed (both the sender's and 
receivers), and hence packets with seq  numbers from
1840001135 upto 1840058075  were dropped (39 packets)...and also the ACK 
for 1840001135 was also dropped (from the traces this can be seen as it 
appears
in the client trace but not on the server trace)...
and since there were still packets to be sent the sender keeps sending a 
few more packets
and when  few of them are received (from the client side trace..)

17:19:35.938017 1840059535:1840060995(1460) ack 3059152863 win 5840 (DF)...
17:19:35.938028  ack 1840001135 win 62780 (DF) [tos 0x8]...first ACK 
that is going to be received by the sender
17:19:35.969316  1840060995:1840062455(1460) ack 3059152863 win 5840 (DF)
17:19:35.969325  1840001135 win 62780 (DF) [tos 0x8]....first duplicate ACK
17:19:36.000519  1840062455:1840063915(1460) ack 3059152863 win 5840 (DF)
17:19:36.000528  ack 1840001135 win 62780 (DF) [tos 0x8]... second 
duplicate ACK

when the server gets this 2nd duplicate ACK, it retransmits the packets 
(this is clearly visible from the tcptrace curve.)..eventhough a 3rd 
duplicate ACK soon follows.
so my first question  "why is the second duplicate ACK triggering a 
retransmission?"...

also after that, there are a couple of retransmissions triggerd by the 
reception of the ACK for the new ACKs and at time instant  (server side 
trace)
17:19:36.057149 . 1840001135:1840002595(1460) ack 3059152863 win 5840 
(DF)..first packet retransmitted
17:19:36.085569   ack 1840001135 win 62780 (DF) [tos 0x8] ...this is the 
third duplicate ACK which should have caused the retrans, but lets 
ignore it for now
17:19:36.248599  ack 1840002595 ...retransmitted packet acked
17:19:36.251382  1840002595:1840004055(1460) ack 3059152863 win 5840 
(DF) ... next packet retransmitted
17:19:36.442831  ack 1840004055 win 61320 (DF) [tos 0x8]...2nd packet 
acked also
17:19:36.445625  1840004055:1840005515(1460) ack 3059152863 win 5840 
(DF) .. third packet retransmitted
17:19:36.637224  ack 1840005515 win 61320 (DF) [tos 0x8] ... third 
packet acked
17:19:37.417022  1840005515:1840006975(1460) ack 3059152863 win 5840 
(DF) ... fourth packet retransmitted

As you can see  there is 0.8 second gap between the ack for the 
reception of the ACK for the third packet and the sending of the fourth 
packet...so my second question  "why didnt the sender immediatly
send the fourth packet after the reception of the ack for the third?"
I generated the same scenario 20 times, and the same thing happens in 
all of them...

2)packet delays
===============
in the second scenario, I have a 2 second delay, but no packet 
drops...the downgrade in bandwidth also happens, but the packets in the 
window are buffered for 2 seconds and released...

The trace files from both the client and the server side can be found at....
http://kom.aau.dk/~oumer/delay_0_drop_SERVER.dat
http://kom.aau.dk/~oumer/delay_0_drop_CLIENT.dat
and the tcptrace time sequence curve can be found in
http://kom.aau.dk/~oumer/delay_0_drop.ps

The delay is applied from 17:20:01.066725 to 17:20:03.067022
as can be seen from the traces and plot packets with seq number 
1858561966 to 1858618906 ( a total of 40 packets) were queued at the 
server and one packet from the receiver, which is an ACK for
pkt # 1858560506  ....
at around 17:20:03.15 this ack is received and sender thinks this is the 
result of its retransmission (which actually was dropped, so at this 
point the receiver hasnot got any retransmissions).. and the normal 
retransmission is resumed (as well as sending of some new data, as the 
window allows it) as can be seen from the server side trace upto time 
instant 17:20:04.539682
...at which point we can see that on the client trace the 
retransmissions actually start arriving at the receiver (so far the ACKs 
that were triggering the retransmissions were acks to the reception of 
the original
but delayed packets)...and this duplicate arrivals lead to multiple 
duplicate ACKs... what I dont understand is why this duplicate ACKs 
(there are 40 duplicate ACKs.), no fast retransmission was triggered..
so my third question "Why is it that the duplicate ACKs are not 
tiggerring fast retransmissions?" this creates a 1.3 second gap 
transmission gap...actually this is better than fast retransmission 
because it is not leading to further retransmissions...so is the linux 
TCP so clever that it can figure out the problem without using SACK, 
timestamps or FRTO ? ...or is this a special "feature" :-)....

I have repeated this also twenty times and the traces are similar...




 

Re: Linux TCP in the presence of delays or drops...

[Oumer Teyeb]

Hi,

it would be so great if some of you could spare a few minutes and take a 
look at the traces I provided.....see below for the original postng...I 
just had a couple of things to add which I noticed in linux TCP 
behaviour which I have not seen documented anywhere else (or which I 
might have misread..:-)...and below I have given yet another trace that 
illustrates one of the TCP linux behaviour which I am having trouble 
understanding....

-If multiple timeouts occur for one packet then even if we are using the 
timestamp option or FRTO TCP linux is not able to detect spurious 
retransmissions... and TCP linux is able to detect spurious 
retransmissions only for a single timeout for one packet or fast 
retransmissions that are caused by duplicate ACK reception.....I have 
some traces that show this behaviour, let me know if you are interested.

-In the cases where TCP timestamp or FRTO is not able to detect spurious 
retransmissions, the performance degrades even more than when TCP 
timestamp or FRTO option are not used....

I also have one additional trace that shows the problem with the case of 
an explained pause in the tcp sender during retransmission which I found 
really hard to explain.... it is similar to the case 1) but this time I 
am doing an upgrade instead from a 384kbps connection to 1Mbps 
connection.... the traces and tcptrace time sequence curve can be found 
at...
http://kom.aau.dk/~oumer/drop_0_delay_UPGRADE_SERVER.dat
http://kom.aau.dk/~oumer/drop_0_delay_UPGRADE_CLIENT.dat
and the tcptrace time sequence curve can be found in
http://kom.aau.dk/~oumer/drop_0_delay_UPGRADE.ps

as you can see from the server side trace... (all the packets shown here 
are retransmissions because I flushed the sender's buffer at time 
instant 17:26:24.657)
17:26:26.261972  2267693336:2267694796(1460) ack 3498775069 win 5840 (DF)
17:26:26.319180  . ack 2267694796 win 61320 (DF) [tos 0x8]
17:26:26.321961  2267694796:2267696256(1460) ack 3498775069 win 5840 (DF)
17:26:26.379160  . ack 2267696256 win 61320 (DF) [tos 0x8]
17:26:26.381940 . 2267696256:2267697716(1460) ack 3498775069 win 5840 (DF)
17:26:26.439138  . ack 2267697716 win 61320 (DF) [tos 0x8]
17:26:26.441925  2267697716:2267699176(1460) ack 3498775069 win 5840 (DF)
17:26:26.499144   ack 2267699176 win 61320 (DF) [tos 0x8]
17:26:28.234327  2267699176:2267700636(1460) ack 3498775069 win 5840 (DF)

eventhough the server got an ACK with # ack 2267699176 at timeinstant 
17:26:26.49...it waited till 17:26:28.234 to resend  the packet... which 
is around
1.73 seconds... I have checked with other traces where I introduced 
delay and for the link the first timeout occurs after 1.73 second, which 
seems to be the RTO at that time, and for no apparent reason
TCP is wating for a timeout...  case 1 is quite similar but there the 
retransmissions were triggered by timeout to begin with, here the 
retransmissions are triggered by duplicate ACKs...in the case1 described 
below this abnormal behaviour occured after only a couple of packets 
were retransmitted...here it took quite some retransmissions before the 
same problem happend... any insight into this is greatly appreciated!!

Thanks in advance,
Oumer

Oumer Teyeb wrote:

> Hi all,
>
> I have some questions regarding Linux TCP in the presence of delays or 
> packet drops. It is somehow long mail, but the questions are two or 
> three, just wanted to provide a detailed information so that the 
> problem is clear. thanx for the patience!!
>
> Best regards,
> Oumer
>
> Note that for the traces referred here, SACK,timestamps, and FRTO are 
> all disabled...
>
> 1) packet drops
> ================
> I have a trace where the tcp sender window is flushed and  then the 
> connection speed is  changed from 1Mbps to 384kbps...
> The trace files from both the client and the server side can be found at
> http://kom.aau.dk/~oumer/drop_0_delay_SERVER.dat
> http://kom.aau.dk/~oumer/drop_0_delay_CLIENT.dat
> and the tcptrace time sequence curve can be found in
> http://kom.aau.dk/~oumer/drop_0_delay.ps
>
> as can be seen from the plot and the trace files at around 
> 17:19:35.705733, the window was flushed (both the sender's and 
> receivers), and hence packets with seq  numbers from
> 1840001135 upto 1840058075  were dropped (39 packets)...and also the 
> ACK for 1840001135 was also dropped (from the traces this can be seen 
> as it appears
> in the client trace but not on the server trace)...
> and since there were still packets to be sent the sender keeps sending 
> a few more packets
> and when  few of them are received (from the client side trace..)
>
> 17:19:35.938017 1840059535:1840060995(1460) ack 3059152863 win 5840 
> (DF)...
> 17:19:35.938028  ack 1840001135 win 62780 (DF) [tos 0x8]...first ACK 
> that is going to be received by the sender
> 17:19:35.969316  1840060995:1840062455(1460) ack 3059152863 win 5840 (DF)
> 17:19:35.969325  1840001135 win 62780 (DF) [tos 0x8]....first 
> duplicate ACK
> 17:19:36.000519  1840062455:1840063915(1460) ack 3059152863 win 5840 (DF)
> 17:19:36.000528  ack 1840001135 win 62780 (DF) [tos 0x8]... second 
> duplicate ACK
>
> when the server gets this 2nd duplicate ACK, it retransmits the 
> packets (this is clearly visible from the tcptrace curve.)..eventhough 
> a 3rd duplicate ACK soon follows.
> so my first question  "why is the second duplicate ACK triggering a 
> retransmission?"...
>
> also after that, there are a couple of retransmissions triggerd by the 
> reception of the ACK for the new ACKs and at time instant  (server 
> side trace)
> 17:19:36.057149 . 1840001135:1840002595(1460) ack 3059152863 win 5840 
> (DF)..first packet retransmitted
> 17:19:36.085569   ack 1840001135 win 62780 (DF) [tos 0x8] ...this is 
> the third duplicate ACK which should have caused the retrans, but lets 
> ignore it for now
> 17:19:36.248599  ack 1840002595 ...retransmitted packet acked
> 17:19:36.251382  1840002595:1840004055(1460) ack 3059152863 win 5840 
> (DF) ... next packet retransmitted
> 17:19:36.442831  ack 1840004055 win 61320 (DF) [tos 0x8]...2nd packet 
> acked also
> 17:19:36.445625  1840004055:1840005515(1460) ack 3059152863 win 5840 
> (DF) .. third packet retransmitted
> 17:19:36.637224  ack 1840005515 win 61320 (DF) [tos 0x8] ... third 
> packet acked
> 17:19:37.417022  1840005515:1840006975(1460) ack 3059152863 win 5840 
> (DF) ... fourth packet retransmitted
>
> As you can see  there is 0.8 second gap between the ack for the 
> reception of the ACK for the third packet and the sending of the 
> fourth packet...so my second question  "why didnt the sender immediatly
> send the fourth packet after the reception of the ack for the third?"
> I generated the same scenario 20 times, and the same thing happens in 
> all of them...
>
> 2)packet delays
> ===============
> in the second scenario, I have a 2 second delay, but no packet 
> drops...the downgrade in bandwidth also happens, but the packets in 
> the window are buffered for 2 seconds and released...
>
> The trace files from both the client and the server side can be found 
> at....
> http://kom.aau.dk/~oumer/delay_0_drop_SERVER.dat
> http://kom.aau.dk/~oumer/delay_0_drop_CLIENT.dat
> and the tcptrace time sequence curve can be found in
> http://kom.aau.dk/~oumer/delay_0_drop.ps
>
> The delay is applied from 17:20:01.066725 to 17:20:03.067022
> as can be seen from the traces and plot packets with seq number 
> 1858561966 to 1858618906 ( a total of 40 packets) were queued at the 
> server and one packet from the receiver, which is an ACK for
> pkt # 1858560506  ....
> at around 17:20:03.15 this ack is received and sender thinks this is 
> the result of its retransmission (which actually was dropped, so at 
> this point the receiver hasnot got any retransmissions).. and the 
> normal retransmission is resumed (as well as sending of some new data, 
> as the window allows it) as can be seen from the server side trace 
> upto time instant 17:20:04.539682
> ...at which point we can see that on the client trace the 
> retransmissions actually start arriving at the receiver (so far the 
> ACKs that were triggering the retransmissions were acks to the 
> reception of the original
> but delayed packets)...and this duplicate arrivals lead to multiple 
> duplicate ACKs... what I dont understand is why this duplicate ACKs 
> (there are 40 duplicate ACKs.), no fast retransmission was triggered..
> so my third question "Why is it that the duplicate ACKs are not 
> tiggerring fast retransmissions?" this creates a 1.3 second gap 
> transmission gap...actually this is better than fast retransmission 
> because it is not leading to further retransmissions...so is the linux 
> TCP so clever that it can figure out the problem without using SACK, 
> timestamps or FRTO ? ...or is this a special "feature" :-)....
>
> I have repeated this also twenty times and the traces are similar...
>
>
>
>
>
>
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Linux TCP in the presence of delays or drops...

[David Miller]

From: Oumer Teyeb <oumer@kom.aau.dk>
Date: Mon, 31 Jul 2006 19:49:28 +0200

> it would be so great if some of you could spare a few minutes and take a 
> look at the traces I provided.....see below for the original postng...

If people are too backlogged and busy to reply to your original
posting, you will only ensure that it will take even longer by
bombarding the list with even more information and questions on
top of your original large query.

Just be patient.

Re: Linux TCP in the presence of delays or drops...

[Oumer Teyeb]

Hi David,

My intention when I wrote the second mail was just to provide some more 
examples that further elaborate my first question. But as you noticed, I 
couldnt resist the temptation to slip in a couple of new questions on 
the new post :-(...sorry and will take your advice into consideration on 
my future postings.

Thanks for the tip!!

Regards,
Oumer

David Miller wrote:

>From: Oumer Teyeb <oumer@kom.aau.dk>
>Date: Mon, 31 Jul 2006 19:49:28 +0200
>
>  
>
>>it would be so great if some of you could spare a few minutes and take a 
>>look at the traces I provided.....see below for the original postng...
>>    
>>
>
>If people are too backlogged and busy to reply to your original
>posting, you will only ensure that it will take even longer by
>bombarding the list with even more information and questions on
>top of your original large query.
>
>Just be patient.
>  
>

Re: Linux TCP in the presence of delays or drops...

[Ilpo Järvinen]

On Mon, 31 Jul 2006, Oumer Teyeb wrote:

> -If multiple timeouts occur for one packet then even if we are using the
> timestamp option or FRTO TCP linux is not able to detect spurious
> retransmissions... and TCP linux is able to detect spurious retransmissions
> only for a single timeout for one packet or fast retransmissions that are
> caused by duplicate ACK reception.....I have some traces that show this
> behaviour, let me know if you are interested.

I have come across this same issue. I can confirm that multiple RTOs is 
not handled correctly. But there are some other issues in FRTO as 
well... nothing extremely dangerous though. In one of the cases, the 
current FRTO algorithm could miss real losses, but you luckily need to be 
quite clever to trigger that, and due to very conservative response used 
in case spurious RTO is detected, it has no significant danger in it even 
then. The other flaws cause too conservative behavior.

We have a set of fixes to F-RTO, but part of them have not yet been 
tested. Since the fixes include 4-5 independent changes to handle also 
rare cases, it takes some time to test. Besides, I'll probably have to 
talk with Pasi Sarolahti (author of FRTO) on couple of interpretation 
issues in RFC4138 as soon as his vacation ends (mid August if I remember 
correctly) to verify some of the changes.

I expect that I'll get some actual results after two weeks or so... I case 
you're are in hurry and are interested on the fixes, I could prepare an 
independent patch quite soon and release it (untested) on our projects web 
site (if you are interested, ask off-list so that we don't bother others 
:-)). But the kernel inclusion of the fixes should IMO wait at least until 
I get some decent test cases analyzed, which will take at least two weeks 
or so due to my schedule.

> -In the cases where TCP timestamp or FRTO is not able to detect spurious
> retransmissions, the performance degrades even more than when TCP timestamp
> or FRTO option are not used....

That's one of the FRTO "features", we have a fix (I cannot say about 
timestamps since we've been running our tests without tstamps for years).


-- 
 i.

Re: Linux TCP in the presence of delays or drops...

[Oumer Teyeb]

Thanks Ilpo for the info!

I am trying out the tests now using timestamps only and without FRTO, 
and vice versa and see if there is any change.
Actually, I have also noticed in some of the traces also this behaviour 
of FRTO where it mistook a loss as spurious which leads to further 
performance degradtion. but I was also using timestamps, so I dont know 
if it also occurs without timestamps.  I will try it out and let you 
know. I will send you the traces I just mentioned (FRTO+timestamps 
leading to a loss being mistaken for a spurious one..)..

Regards,
Oumer

Ilpo Järvinen wrote:

>On Mon, 31 Jul 2006, Oumer Teyeb wrote:
>
>  
>
>>-If multiple timeouts occur for one packet then even if we are using the
>>timestamp option or FRTO TCP linux is not able to detect spurious
>>retransmissions... and TCP linux is able to detect spurious retransmissions
>>only for a single timeout for one packet or fast retransmissions that are
>>caused by duplicate ACK reception.....I have some traces that show this
>>behaviour, let me know if you are interested.
>>    
>>
>
>I have come across this same issue. I can confirm that multiple RTOs is 
>not handled correctly. But there are some other issues in FRTO as 
>well... nothing extremely dangerous though. In one of the cases, the 
>current FRTO algorithm could miss real losses, but you luckily need to be 
>quite clever to trigger that, and due to very conservative response used 
>in case spurious RTO is detected, it has no significant danger in it even 
>then. The other flaws cause too conservative behavior.
>  
>

>We have a set of fixes to F-RTO, but part of them have not yet been 
>tested. Since the fixes include 4-5 independent changes to handle also 
>rare cases, it takes some time to test. Besides, I'll probably have to 
>talk with Pasi Sarolahti (author of FRTO) on couple of interpretation 
>issues in RFC4138 as soon as his vacation ends (mid August if I remember 
>correctly) to verify some of the changes.
>
>I expect that I'll get some actual results after two weeks or so... I case 
>you're are in hurry and are interested on the fixes, I could prepare an 
>independent patch quite soon and release it (untested) on our projects web 
>site (if you are interested, ask off-list so that we don't bother others 
>:-)). But the kernel inclusion of the fixes should IMO wait at least until 
>I get some decent test cases analyzed, which will take at least two weeks 
>or so due to my schedule.
>
>  
>

>>-In the cases where TCP timestamp or FRTO is not able to detect spurious
>>retransmissions, the performance degrades even more than when TCP timestamp
>>or FRTO option are not used....
>>    
>>
>
>That's one of the FRTO "features", we have a fix (I cannot say about 
>timestamps since we've been running our tests without tstamps for years).
>  
>