Strange TCP behavior over HSDPA

Strange TCP behavior over HSDPA

[Kristian Evensen]

Hello,

I have a laptop running Ubuntu 9.04 with the 2.6.32.2-kernel and the machine
connects to the internet using HSDPA. When transferring files (from the same
server) using TCP I notice that the connection often experience severe drops
in throughput. Often, it takes over a second for the (fast) retransmission
to arrive.

First I thought this might be due to poor network conditions, but when
testing tunnels (L2TP and PPTP) I noticed that the throughput was more
stable and the average higher. It seems like the retransmission is sent and
arrives quicker than without tunnels. The drops last for a shorter period of
time and does not affect the performance that much. Based on my
understanding of the tunneling protocols, they offer no reliability for data
and according to my ISP UDP and TCP packets are treated equally.

I have performed several experiments (adjusting the MTU, MSS and running the
tests from different places) trying to figure out why tunnels perform better
than without tunnels, but I have not been able to come up with an answer.
The different experiments all give the same result. So my question is, has
anyone experienced anything similar or have an idea as to why the tunnels
perform better?

The HSDPA-modems I have used are the Huawei E220 and Sierra Wireless Aircard
880u. If anyone is interested in looking at the packet dumps, they can be
found here [1] and here [2]. [1] is without tunnels and a typical packet
loss is shown at packet 1705, while [2] is without tunnels and a typical
packet loss is shown at 3381.

Thanks in advance for any help,
Kristian

[1] - http://heim.ifi.uio.no/~kristrev/hsdpa-captures/loss_hsdpa.pcap 
[2] - http://heim.ifi.uio.no/~kristrev/hsdpa-captures/loss_hsdpa_tunnel.pcap

Re: Strange TCP behavior over HSDPA

[Jarek Poplawski]

On 28-12-2009 13:56, Kristian Evensen wrote:
> Hello,
> 
> I have a laptop running Ubuntu 9.04 with the 2.6.32.2-kernel and the machine
> connects to the internet using HSDPA. When transferring files (from the same
> server) using TCP I notice that the connection often experience severe drops
> in throughput. Often, it takes over a second for the (fast) retransmission
> to arrive.
> 
> First I thought this might be due to poor network conditions, but when
> testing tunnels (L2TP and PPTP) I noticed that the throughput was more
> stable and the average higher. It seems like the retransmission is sent and
> arrives quicker than without tunnels. The drops last for a shorter period of
> time and does not affect the performance that much. Based on my
> understanding of the tunneling protocols, they offer no reliability for data
> and according to my ISP UDP and TCP packets are treated equally.
> 
> I have performed several experiments (adjusting the MTU, MSS and running the
> tests from different places) trying to figure out why tunnels perform better
> than without tunnels, but I have not been able to come up with an answer.
> The different experiments all give the same result. So my question is, has
> anyone experienced anything similar or have an idea as to why the tunnels
> perform better?
> 
> The HSDPA-modems I have used are the Huawei E220 and Sierra Wireless Aircard
> 880u. If anyone is interested in looking at the packet dumps, they can be
> found here [1] and here [2]. [1] is without tunnels and a typical packet
> loss is shown at packet 1705, while [2] is without tunnels and a typical
> packet loss is shown at 3381.
> 
> Thanks in advance for any help,
> Kristian
> 
> [1] - http://heim.ifi.uio.no/~kristrev/hsdpa-captures/loss_hsdpa.pcap 
> [2] - http://heim.ifi.uio.no/~kristrev/hsdpa-captures/loss_hsdpa_tunnel.pcap

The most obvious difference seems to be the SACK option used in [2],
so maybe something on the route breaks it (without tunnels).

Jarek P.

Re: Strange TCP behavior over HSDPA

[Kristian Evensen]

Den 29.12.2009 09:30, skrev Jarek Poplawski:
> The most obvious difference seems to be the SACK option used in [2],
> so maybe something on the route breaks it (without tunnels).
>
> Jarek P.
>    
Thank you for your reply. I have tried disabling SACK and then the 
performance when packet loss occured got even worse, and the rest of the 
time it was pretty similar.

After finally getting hold of someone I know at my ISP, I was told that 
they use TCP accelerators in their mobile network. Also, when looking at 
the packet captures from the server side I see that the sequene numbers 
are changed, so there is a machine doing something to the packets along 
the route. What I suspect is that this or these machines somehow breaks 
when a retransmission occurs. I am not quire sure what could cause it 
though, for example all dupAcks generated by the client are recieved by 
the server, but hopefully my ISP will be able to figure it out.

-Kristian

Re: Strange TCP behavior over HSDPA

[Jarek Poplawski]

On Tue, Dec 29, 2009 at 10:59:43AM +0100, Kristian Evensen wrote:
> Den 29.12.2009 09:30, skrev Jarek Poplawski:
>> The most obvious difference seems to be the SACK option used in [2],
>> so maybe something on the route breaks it (without tunnels).
>>
>> Jarek P.
>>    
> Thank you for your reply. I have tried disabling SACK and then the  
> performance when packet loss occured got even worse, and the rest of the  
> time it was pretty similar.

Did you try to turn off TCP window scaling btw? Anyway, under the
tunnel ([2]), when SACK worked, it saved you a lot of retransmits.

Jarek P.

Re: Strange TCP behavior over HSDPA

[Jarek Poplawski]

On Tue, Dec 29, 2009 at 10:57:48AM +0000, Jarek Poplawski wrote:
> Did you try to turn off TCP window scaling btw? Anyway, under the
> tunnel ([2]), when SACK worked, it saved you a lot of retransmits.

Hmm... Actually, after re-checking, there weren't much more of those
retransmits at all. In [1] there was one more packet lost, so it took
a bit longer. In [2] (with SACK) the retransmit started earlier and
the rcv window was unchanged. So, it rather looks like differences
in timing of TCP recovery techniques.

Jarek P.

Re: Strange TCP behavior over HSDPA

[Jarek Poplawski]

On Tue, Dec 29, 2009 at 12:16:24PM +0000, Jarek Poplawski wrote:
> On Tue, Dec 29, 2009 at 10:57:48AM +0000, Jarek Poplawski wrote:
> > Did you try to turn off TCP window scaling btw? Anyway, under the
> > tunnel ([2]), when SACK worked, it saved you a lot of retransmits.
> 
> Hmm... Actually, after re-checking, there weren't much more of those
> retransmits at all. In [1] there was one more packet lost, so it took
> a bit longer. In [2] (with SACK) the retransmit started earlier and
> the rcv window was unchanged. So, it rather looks like differences
> in timing of TCP recovery techniques.

Hmm#2... On the other hand, I can imagine cases with a larger data
loss, where working SACK should really save on retransmits.

Jarek P.

Re: Strange TCP behavior over HSDPA

[Kristian Evensen]

Den 29.12.2009 14:36, skrev Jarek Poplawski:
> On Tue, Dec 29, 2009 at 12:16:24PM +0000, Jarek Poplawski wrote:
>    
>> On Tue, Dec 29, 2009 at 10:57:48AM +0000, Jarek Poplawski wrote:
>>      
>>> Did you try to turn off TCP window scaling btw? Anyway, under the
>>> tunnel ([2]), when SACK worked, it saved you a lot of retransmits.
>>>        
>> Hmm... Actually, after re-checking, there weren't much more of those
>> retransmits at all. In [1] there was one more packet lost, so it took
>> a bit longer. In [2] (with SACK) the retransmit started earlier and
>> the rcv window was unchanged. So, it rather looks like differences
>> in timing of TCP recovery techniques.
>>      
> Hmm#2... On the other hand, I can imagine cases with a larger data
> loss, where working SACK should really save on retransmits.
>
> Jarek P.
>    
Thanks for all your comments. I have not tried to disable window 
scaling, but will try that as soon as possible. I also noticed the 
second packet loss in [1], but I don't think it affected the situation 
to much. In similar packet captures, the transfer without the tunnel has 
only lost one packet and the throughput drop has been just as 
significant as in [1]. It rather seems to be, as you point out, 
differences in timing of the recovery techniques, probably between this 
accelerator and the server.

However, I find it a bit strange the dupAcks are sent back to the 
server. Based on my, I must admit limited, knowledge of accelerators, 
they will buffer and ACK packets if they for example are responsible for 
retransmissions. But again, maybe it uses the dupAcks to tell the server 
to slow down and then simply discards the retransmitted packet.

-Kristian

Re: Strange TCP behavior over HSDPA

[Jarek Poplawski]

On Tue, Dec 29, 2009 at 03:30:28PM +0100, Kristian Evensen wrote:
> Den 29.12.2009 14:36, skrev Jarek Poplawski:
> >On Tue, Dec 29, 2009 at 12:16:24PM +0000, Jarek Poplawski wrote:
> >>On Tue, Dec 29, 2009 at 10:57:48AM +0000, Jarek Poplawski wrote:
> >>>Did you try to turn off TCP window scaling btw? Anyway, under the
> >>>tunnel ([2]), when SACK worked, it saved you a lot of retransmits.
> >>Hmm... Actually, after re-checking, there weren't much more of those
> >>retransmits at all. In [1] there was one more packet lost, so it took
> >>a bit longer. In [2] (with SACK) the retransmit started earlier and
> >>the rcv window was unchanged. So, it rather looks like differences
> >>in timing of TCP recovery techniques.
> >Hmm#2... On the other hand, I can imagine cases with a larger data
> >loss, where working SACK should really save on retransmits.
> >
> >Jarek P.
> Thanks for all your comments. I have not tried to disable window
> scaling, but will try that as soon as possible. I also noticed the
> second packet loss in [1], but I don't think it affected the
> situation to much. In similar packet captures, the transfer without
> the tunnel has only lost one packet and the throughput drop has been
> just as significant as in [1]. It rather seems to be, as you point
> out, differences in timing of the recovery techniques, probably
> between this accelerator and the server.

Then disabling window scaling seems too radical; maybe you should try
with lower tcp_rmem max etc.

> 
> However, I find it a bit strange the dupAcks are sent back to the
> server. Based on my, I must admit limited, knowledge of
> accelerators, they will buffer and ACK packets if they for example
> are responsible for retransmissions. But again, maybe it uses the
> dupAcks to tell the server to slow down and then simply discards the
> retransmitted packet.

I'm not a TCP expert, but IMHO you don't need much knowledge to assess
quality of this accelerator, if you get better results when omitting
it (partly) with a tunnel ;-)

Jarek P.