Sensitivity of TFRC throughput equation wrt to changes of RTT

Sensitivity of TFRC throughput equation wrt to changes of RTT

[Gerrit Renker]

The comment by Dave brought me back to check what a change of
a factor of 10 does to the throughput in TFRC. 

RFC 3448 gives in section 8 the following alternative format
of the throughput equation (which is directly responsible for
the alllowed sending rate X):

              s
      X =  --------
           R * f(p)

This shows that the dependence is reciprocal. Thus using an RTT
which differs by a factor of 10 to account for in-stack processing
results an a throughput reduction of factor 10.

In other words, 90 Mbits/sec becomes 9 Mbits/sec.

Re: Sensitivity of TFRC throughput equation wrt to changes of RTT

[David Miller]

From: Gerrit Renker <gerrit@erg.abdn.ac.uk>
Date: Fri, 13 Apr 2007 13:03:02 +0100

> RFC 3448 gives in section 8 the following alternative format
> of the throughput equation (which is directly responsible for
> the alllowed sending rate X):
> 
>               s
>       X =  --------
>            R * f(p)
> 
> This shows that the dependence is reciprocal. Thus using an RTT
> which differs by a factor of 10 to account for in-stack processing
> results an a throughput reduction of factor 10.
> 
> In other words, 90 Mbits/sec becomes 9 Mbits/sec.

What I'd like to know in all this is why the RTT influences the
sending rate at all in such a manner.  Please teach me :)

If I have a 10gbit pipe all the way to the planet mars I should
still be feeding that pipe at a rate of 10gbit. :)

TCP doesn't have any of these problems, and we use incredibly coarse
timestamping for RTTs.  We get jiffies granularity at best, with many
in-stack delays, and we still send at full line rate over large RTTs.

Re: Sensitivity of TFRC throughput equation wrt to changes of RTT

[Ian McDonald]

On 4/14/07, Gerrit Renker <gerrit@erg.abdn.ac.uk> wrote:
> The comment by Dave brought me back to check what a change of
> a factor of 10 does to the throughput in TFRC.
>
> RFC 3448 gives in section 8 the following alternative format
> of the throughput equation (which is directly responsible for
> the alllowed sending rate X):
>
>               s
>       X =  --------
>            R * f(p)
>
Which comes into play when we have loss. When we have no loss, or a
long period of non-loss we can send as fast as we want according to
the spec. This should normally be the case on LANs where the RTT makes
a far bigger difference.

So you will have this on very low latency, lossly links potentially.
How realistic is this? Another thing is that TFRC is designed to
respond to loss in a different way than TCP (more smoothly). This is
the attraction of TFRC. Remember this is an unreliable protocol so it
is not designed for high speed file transfer etc. Do we actually need
to chase every last bit of performance (I'm not saying we should be
lazy)?

As an aside I use a network testing with two machines at 1 GHz or so
and the receiver used to hit 100% when testing with iperf on a 100
Mbit LAN (TCP uses around 40%). The sender used far less CPU. Funnily
enough under this scenario the rate used to drop.... I shut down a few
other bits and pieces and then got full rate and just under 100% CPU.
Now my machines/NICs aren't particularly powerful and I haven't tested
with latest code (which replaces linked lists with arrays etc) but I
think if we want to ensure high performance on low RTT links we need
to check this also.
-- 
Web: http://wand.net.nz/~iam4/
Blog: http://iansblog.jandi.co.nz
WAND Network Research Group

Re: Sensitivity of TFRC throughput equation wrt to changes of RTT

[Gerrit Renker]

Quoting David Miller:
|  From: Gerrit Renker <gerrit@erg.abdn.ac.uk>
|  Date: Fri, 13 Apr 2007 13:03:02 +0100
|  
|  > RFC 3448 gives in section 8 the following alternative format
|  > of the throughput equation (which is directly responsible for
|  > the alllowed sending rate X):
|  > 
|  >               s
|  >       X =  --------
|  >            R * f(p)
|  > 
|  > This shows that the dependence is reciprocal. Thus using an RTT
|  > which differs by a factor of 10 to account for in-stack processing
|  > results an a throughput reduction of factor 10.
|  > 
|  > In other words, 90 Mbits/sec becomes 9 Mbits/sec.
|  
|  What I'd like to know in all this is why the RTT influences the
|  sending rate at all in such a manner.  Please teach me :)

I wished someone could tell me that. Ian is right, the formula is used
after the first loss, but the idea is that the sender `overshoots' and then
reduces after the first loss due to overestimating the bandwidth.

So it would try to see the pipe as 20Gbit, experience some loss, and then
reduce proportionally to RTT and f(p). 

We have more problems of the same nature as with using interface timestamps.

I am really not sure that CCID3 can be implemented well without a lot of
real-time and system load requirements - if you have any suggestions or
know of similar problem areas, input would be very welcome.

  
|  TCP doesn't have any of these problems, and we use incredibly coarse
|  timestamping for RTTs.  We get jiffies granularity at best, with many
|  in-stack delays, and we still send at full line rate over large RTTs.

Re: Sensitivity of TFRC throughput equation wrt to changes of RTT

[David Miller]

From: "Ian McDonald" <ian.mcdonald@jandi.co.nz>
Date: Sat, 14 Apr 2007 07:52:32 +1200

> Which comes into play when we have loss. When we have no loss, or a
> long period of non-loss we can send as fast as we want according to
> the spec. This should normally be the case on LANs where the RTT makes
> a far bigger difference.

This is correct, also there is another related issue I should
have mentioned.

Congestion control on LANs and very small RTTs arguably does not make
much sense.  Many very smart networking folk have argued this.

The reason is that any congestion you detect on a LAN will go away
long before you can even react to it in a protocol stack.

Alexey Kuznetsov once was trying to come up with a statistical
theoretical model to justify this in some tangible way, but he
never came up with anything concrete.

It's funny but the RTT measurements of TCP have a minimum which
is established by the timer granularity limitations of the original
TCP implementation in BSD.  See the TCP_RTO_MIN clamping we do
in the Linux TCP stack.

Although originally an implementation side-effect, it plays into
the LAN issues I discussed above.

Re: Sensitivity of TFRC throughput equation wrt to changes of RTT

[David Miller]

From: Gerrit Renker <gerrit@erg.abdn.ac.uk>
Date: Fri, 13 Apr 2007 21:27:54 +0100

> I wished someone could tell me that. Ian is right, the formula is used
> after the first loss, but the idea is that the sender `overshoots' and then
> reduces after the first loss due to overestimating the bandwidth.
> 
> So it would try to see the pipe as 20Gbit, experience some loss, and then
> reduce proportionally to RTT and f(p). 
> 
> We have more problems of the same nature as with using interface timestamps.
> 
> I am really not sure that CCID3 can be implemented well without a lot of
> real-time and system load requirements - if you have any suggestions or
> know of similar problem areas, input would be very welcome.

I wonder what a DCCP implementation on old BSD would do with it's
super-coarse timers :-)

Perhaps the algorithm can get tweaked such that, just like TCP, we
lower bound the RTT and for RTTs seen at or below that minimum we do
not do the sender rate capping.

In this sense coarseness of timers actually helps weed out the
noise of scheduling, the queue that exists between the network
link and the driver actually feeding packets to the stack, etc.

Because I sense that even with the global timestamping, the same
problem will reoccur during high forwarding loads where one network
card competes with another in the NAPI poll processing loops.

So my primary suggstion is to accept that there is a limit to
how accurate you can get timestamps, there is also a specific
regime under which fine-RTT measurements matter, and therefore
the algorithm chosen has to match that reality.

Re: Sensitivity of TFRC throughput equation wrt to changes of RTT

[Eddie Kohler]

I can't resist:

David Miller wrote:
>> I am really not sure that CCID3 can be implemented well without a lot of
>> real-time and system load requirements - if you have any suggestions or
>> know of similar problem areas, input would be very welcome.
> 
> I wonder what a DCCP implementation on old BSD would do with it's
> super-coarse timers :-)

Such a DCCP would do exactly what several people have suggested tens of times 
by now, namely send in bursts... which would be fine.  (Or at least better.)

Eddie

Re: Sensitivity of TFRC throughput equation wrt to changes of RTT

[Gerrit Renker]

Many thanks indeed for the input - referred to in the other 
email as well.
  
|  So my primary suggstion is to accept that there is a limit to
|  how accurate you can get timestamps, there is also a specific
|  regime under which fine-RTT measurements matter, and therefore
|  the algorithm chosen has to match that reality.
|  
I think that by building on such realities we stand a better chance
of arriving at a workable solution than by blindly trying to cast
a specification into code.

There is some revision to be done where your input is very helpful. 

I hope that Eddie will incorporate/pick up the point to bypass the congestion
control when RTTs are low. 

Ian and Eddie seem to have differing opinions regarding RTT accuracy
and I think I also need to check. ( If only we could use jiffie-based algorithm, 
that would axe the cost of all these gettimeofdays (they still exist, even 
after retracting the skb_get_timestamp patches). There is this really nice TCP
algorithm which doesn't use timestamps, tcp_rcv_rtt_update() in tcp_input.c . )