[PATCH 07/29] Use skb timestamp for receiver side

[PATCH 07/29] Use skb timestamp for receiver side

[Arnaldo Carvalho de Melo]

 This patch replaces calls to do_gettimeofday at the receiver CCID 3
 with skb timestamps. The skb timestamps are computed earlier in the
 receive path, experimental measurements have shown that up to several
 hundred microseconds can lie between the skb receive timestamp and the
 timestamp taken when CCID 3 receives the packet. This difference has
 a negative impact on RTT estimation (reduced accuracy).

Signed-off-by: Gerrit Renker <gerrit@erg.abdn.ac.uk>
Acked-by: Ian McDonald <ian.mcdonald@jandi.co.nz>
Signed-off-by: Arnaldo Carvalho de Melo <acme@ghostprotocols.net>
---
 net/dccp/ccids/lib/packet_history.h |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/net/dccp/ccids/lib/packet_history.h b/net/dccp/ccids/lib/packet_history.h
index b40b9f9..330b58f 100644
--- a/net/dccp/ccids/lib/packet_history.h
+++ b/net/dccp/ccids/lib/packet_history.h
@@ -151,7 +151,7 @@ static inline struct dccp_rx_hist_entry *
 		entry->dccphrx_ccval = dh->dccph_ccval;
 		entry->dccphrx_type  = dh->dccph_type;
 		entry->dccphrx_ndp   = ndp;
-		do_gettimeofday(&entry->dccphrx_tstamp);
+		skb_get_timestamp(skb, &entry->dccphrx_tstamp);
 	}
 
 	return entry;
-- 
1.5.0.6

Re: [PATCH 07/29] Use skb timestamp for receiver side

[David Miller]

From: Arnaldo Carvalho de Melo <acme@ghostprotocols.net>
Date: Thu, 12 Apr 2007 18:14:54 -0300

>  This patch replaces calls to do_gettimeofday at the receiver CCID 3
>  with skb timestamps. The skb timestamps are computed earlier in the
>  receive path, experimental measurements have shown that up to several
>  hundred microseconds can lie between the skb receive timestamp and the
>  timestamp taken when CCID 3 receives the packet. This difference has
>  a negative impact on RTT estimation (reduced accuracy).
> 
> Signed-off-by: Gerrit Renker <gerrit@erg.abdn.ac.uk>
> Acked-by: Ian McDonald <ian.mcdonald@jandi.co.nz>
> Signed-off-by: Arnaldo Carvalho de Melo <acme@ghostprotocols.net>

And this creates an even deeper dependency on the global packet
timestamping facility.  Those are supposed to be enabled only
in obscure circumstances because timestamping every packet is
incredibly expensive.

Please find a better way.

Re: [PATCH 07/29] Use skb timestamp for receiver side

[Arnaldo Carvalho de Melo]

On 4/12/07, David Miller <davem@davemloft.net> wrote:
> From: Arnaldo Carvalho de Melo <acme@ghostprotocols.net>
> Date: Thu, 12 Apr 2007 18:14:54 -0300
>
> >  This patch replaces calls to do_gettimeofday at the receiver CCID 3
> >  with skb timestamps. The skb timestamps are computed earlier in the
> >  receive path, experimental measurements have shown that up to several
> >  hundred microseconds can lie between the skb receive timestamp and the
> >  timestamp taken when CCID 3 receives the packet. This difference has
> >  a negative impact on RTT estimation (reduced accuracy).
> >
> > Signed-off-by: Gerrit Renker <gerrit@erg.abdn.ac.uk>
> > Acked-by: Ian McDonald <ian.mcdonald@jandi.co.nz>
> > Signed-off-by: Arnaldo Carvalho de Melo <acme@ghostprotocols.net>
>
> And this creates an even deeper dependency on the global packet
> timestamping facility.  Those are supposed to be enabled only
> in obscure circumstances because timestamping every packet is
> incredibly expensive.
>
> Please find a better way.

Well, there seems to be just one better way, to timestamp the packet
as soon as it enters the dccp stack, in dccp_v[46]_rcv, and consider
the time from the driver -> ip v[46] -> dccp_v[46]_ as "in the wire".

- Arnaldo

Re: [PATCH 07/29] Use skb timestamp for receiver side

[Ian McDonald]

On 4/13/07, David Miller <davem@davemloft.net> wrote:
> From: Arnaldo Carvalho de Melo <acme@ghostprotocols.net>
> Date: Thu, 12 Apr 2007 18:14:54 -0300
>
> >  This patch replaces calls to do_gettimeofday at the receiver CCID 3
> >  with skb timestamps. The skb timestamps are computed earlier in the
> >  receive path, experimental measurements have shown that up to several
> >  hundred microseconds can lie between the skb receive timestamp and the
> >  timestamp taken when CCID 3 receives the packet. This difference has
> >  a negative impact on RTT estimation (reduced accuracy).
> >
> > Signed-off-by: Gerrit Renker <gerrit@erg.abdn.ac.uk>
> > Acked-by: Ian McDonald <ian.mcdonald@jandi.co.nz>
> > Signed-off-by: Arnaldo Carvalho de Melo <acme@ghostprotocols.net>
>
> And this creates an even deeper dependency on the global packet
> timestamping facility.  Those are supposed to be enabled only
> in obscure circumstances because timestamping every packet is
> incredibly expensive.
>
> Please find a better way.

I didn't know this time stamping was expensive but I knew the way we
were trying to optimise LAN is wrong. I say LAN because a few
microseconds or even milliseconds difference on a WAN link makes
bugger all difference in throughput. DCCP takes into account operating
system granularity etc and if we are running lossless (i.e. receiver
can cope with receiving link at full speed) then we can transmit at
line speed. I've tested this myself.

This illustrates a trend by you Gerrit to try and micro-optimise what
you perceive as bottlenecks and not realise the impact. You have been
also promoting possible use of hrtimers which would be even worse for
example in far of terms of loading a system.

Gerrit - you send us a missive on how we should humbly accept your
patches and stop arguing with you. Obviously Eddie (co-author of DCCP
spec) and myself (converted DCCP CCID3 to Linux 2.6) aren't good
enough. Maybe now Dave Miller says something you will listen or do you
need Linus Torvalds to tell you to stop being a fucking idiot at
times? Yes you write lots of good code but you write some bad code at
times also. I probably write more bad code than you and less volume of
code but I try to listen. Stop pretending to be a deity.

Ian
-- 
Web: http://wand.net.nz/~iam4/
Blog: http://iansblog.jandi.co.nz
WAND Network Research Group

Re: [PATCH 07/29] Use skb timestamp for receiver side

[David Miller]

From: "Ian McDonald" <ian.mcdonald@jandi.co.nz>
Date: Fri, 13 Apr 2007 09:50:30 +1200

> I didn't know this time stamping was expensive but I knew the way we
> were trying to optimise LAN is wrong. I say LAN because a few
> microseconds or even milliseconds difference on a WAN link makes
> bugger all difference in throughput. DCCP takes into account operating
> system granularity etc and if we are running lossless (i.e. receiver
> can cope with receiving link at full speed) then we can transmit at
> line speed. I've tested this myself.

You want the scheduling delays and other issues that can
delay DCCP input packet processing to get factored into
the RTT, so that the sender will pace properly.

Trying to get perfect timestamping and RTT measurements,
and ignoring scheduling delays, is quite foolhardy and shows
a lack of understanding of how queueing really works.

Re: [PATCH 07/29] Use skb timestamp for receiver side

[Ian McDonald]

On 4/13/07, David Miller <davem@davemloft.net> wrote:
> From: "Ian McDonald" <ian.mcdonald@jandi.co.nz>
> Date: Fri, 13 Apr 2007 09:50:30 +1200
>
> > I didn't know this time stamping was expensive but I knew the way we
> > were trying to optimise LAN is wrong. I say LAN because a few
> > microseconds or even milliseconds difference on a WAN link makes
> > bugger all difference in throughput. DCCP takes into account operating
> > system granularity etc and if we are running lossless (i.e. receiver
> > can cope with receiving link at full speed) then we can transmit at
> > line speed. I've tested this myself.
>
> You want the scheduling delays and other issues that can
> delay DCCP input packet processing to get factored into
> the RTT, so that the sender will pace properly.
>
> Trying to get perfect timestamping and RTT measurements,
> and ignoring scheduling delays, is quite foolhardy and shows
> a lack of understanding of how queueing really works.

Yes exactly and DCCP spec caters for this. Thank you for summing it up
succinctly.

-- 
Web: http://wand.net.nz/~iam4/
Blog: http://iansblog.jandi.co.nz
WAND Network Research Group

Re: [PATCH 07/29] Use skb timestamp for receiver side

[Gerrit Renker]

Cool it Ian. 

I think you are interpreting something into the email I sent which is not in
there. This is exactly the kind of talk I was asking for and I am glad that 
Dave reviewed this.

Please read that email again. 

I am asking for fair technical review here and rather than calling me names you
could put some care into what you write.

If for instance you read the documentation accompanying that patch, there is a 
difference between 100 usec ... 1msec between skb_get_timestamp and taking the timestamp
in the CCID3 receiver. It all adds up.

With regard to hrtimers - no patch was ever submitted, for the same reason that you
are stating. But with the current schedule_timeout, the controllable limit is at about
12Mbits/sec when HZ\x1000. Above that it is no longer controlled, let alone congestion-
controlled.


Quoting Ian McDonald:
|  I didn't know this time stamping was expensive but I knew the way we
|  were trying to optimise LAN is wrong. I say LAN because a few
|  microseconds or even milliseconds difference on a WAN link makes
|  bugger all difference in throughput. DCCP takes into account operating
|  system granularity etc and if we are running lossless (i.e. receiver
|  can cope with receiving link at full speed) then we can transmit at
|  line speed. I've tested this myself.
|  
|  This illustrates a trend by you Gerrit to try and micro-optimise what
|  you perceive as bottlenecks and not realise the impact. You have been
|  also promoting possible use of hrtimers which would be even worse for
|  example in far of terms of loading a system.
| 
|  Gerrit - you send us a missive on how we should humbly accept your
|  patches and stop arguing with you. Obviously Eddie (co-author of DCCP
|  spec) and myself (converted DCCP CCID3 to Linux 2.6) aren't good
|  enough. Maybe now Dave Miller says something you will listen or do you
|  need Linus Torvalds to tell you to stop being a fucking idiot at
|  times? Yes you write lots of good code but you write some bad code at
|  times also. I probably write more bad code than you and less volume of
|  code but I try to listen. Stop pretending to be a deity.
|  

Re: [PATCH 07/29] Use skb timestamp for receiver side

[Gerrit Renker]

Same comment. Please do read the documentation accompanying the patch.

Quoting David Miller:
|  From: "Ian McDonald" <ian.mcdonald@jandi.co.nz>
|  Date: Fri, 13 Apr 2007 09:50:30 +1200
|  
|  > I didn't know this time stamping was expensive but I knew the way we
|  > were trying to optimise LAN is wrong. I say LAN because a few
|  > microseconds or even milliseconds difference on a WAN link makes
|  > bugger all difference in throughput. DCCP takes into account operating
|  > system granularity etc and if we are running lossless (i.e. receiver
|  > can cope with receiving link at full speed) then we can transmit at
|  > line speed. I've tested this myself.
|  
|  You want the scheduling delays and other issues that can
|  delay DCCP input packet processing to get factored into
|  the RTT, so that the sender will pace properly.
|  
|  Trying to get perfect timestamping and RTT measurements,
|  and ignoring scheduling delays, is quite foolhardy and shows
|  a lack of understanding of how queueing really works.
|  
|  

Re: [PATCH 07/29] Use skb timestamp for receiver side

[Gerrit Renker]

Sorry, I misread the email. Please see below.

|  You want the scheduling delays and other issues that can
|  delay DCCP input packet processing to get factored into
|  the RTT, so that the sender will pace properly.
|  
|  Trying to get perfect timestamping and RTT measurements,
|  and ignoring scheduling delays, is quite foolhardy and shows
|  a lack of understanding of how queueing really works.
CCID3 uses the sampled RTT value as input into an equation so any
inaccuracies will multiply. I observed a difference of about 10.

Currently the performance of CCID3 severely sucks and I tried every 
bit to make it better. Obviously this is not the right way.

Your comments are helpful, thank you. 

I will revert all RX/TX timestamps and I think there is no need to
take the timestamp in the dccp receive input function. 

By taking the timestamp in the CCID3 handler, it will include all 
processing overhead, slowing the sending rate further down in proportion 
to that difference.

Re: [PATCH 07/29] Use skb timestamp for receiver side

[David Miller]

From: Gerrit Renker <gerrit@erg.abdn.ac.uk>
Date: Fri, 13 Apr 2007 10:27:16 +0100

> CCID3 uses the sampled RTT value as input into an equation so any
> inaccuracies will multiply. I observed a difference of about 10.

This would indicate a bug in the equation if such small variances
cause it to explode so easily.

RTTs modulate constantly, that's why they are smoothed out (at least
in TCP), to kill the noise and figure out what part of the signal
is real :-)

> By taking the timestamp in the CCID3 handler, it will include all
> processing overhead, slowing the sending rate further down in
> proportion to that difference.

Sarcastic much? :-/

Re: [PATCH 07/29] Use skb timestamp for receiver side

[Ian McDonald]

On 4/13/07, Gerrit Renker <gerrit@erg.abdn.ac.uk> wrote:
> Cool it Ian.
>
> I think you are interpreting something into the email I sent which is not in
> there. This is exactly the kind of talk I was asking for and I am glad that
> Dave reviewed this.

The mistake I made was that I didn't change the subject of the email
as it didn't apply to this patch really. You are correct in that.
>
> Please read that email again.
>
> I am asking for fair technical review here and rather than calling me names you
> could put some care into what you write.

Well I would have much rather discussed in private but a) you've told
Eddie not to reply in public and the way stated seems to imply I
shouldn't either b) you said that we were unreasonably holding up your
patches in public. It warranted a response but I'm done with my name
calling (for now).
>
> If for instance you read the documentation accompanying that patch, there is a
> difference between 100 usec ... 1msec between skb_get_timestamp and taking the timestamp
> in the CCID3 receiver. It all adds up.
>
I think we need to look at whether the protocol is right for LANs but
I think if we carry out Eddie's suggestions it may well be right.
Against earlier code a couple of months ago with a couple of my
patches I was achieving around 90 Mbits/sec on iperf which matched TCP
performance and it was stable. It responded gracefully to loss also.

I have not tested lately and tried to replicate but I know that the
state of tree before current patches couldn't do that any more... As
time permits I will redo the work.

DCCP CCID3 has proven stable in simulations in ns2 I understand and it
has been stable previously as well (though not in Linus' tree ever) so
rather than say all these inaccuracies are causing problems we need to
work on the solutions.

Ian
-- 
Web: http://wand.net.nz/~iam4/
Blog: http://iansblog.jandi.co.nz
WAND Network Research Group

Re: [PATCH 07/29] Use skb timestamp for receiver side

[Gerrit Renker]

Quoting David Miller:
|  > CCID3 uses the sampled RTT value as input into an equation so any
|  > inaccuracies will multiply. I observed a difference of about 10.
|  
|  This would indicate a bug in the equation if such small variances
|  cause it to explode so easily.
|  
|  RTTs modulate constantly, that's why they are smoothed out (at least
|  in TCP), to kill the noise and figure out what part of the signal
|  is real :-)
|  
|  > By taking the timestamp in the CCID3 handler, it will include all
|  > processing overhead, slowing the sending rate further down in
|  > proportion to that difference.
|  
|  Sarcastic much? :-/
Yeah maybe - it is not always easy to admit being wrong. But even so,
I am glad for your comments; all skb_timestamping operations will be
reverted without exception.

Thanks again.

Re: [PATCH 07/29] Use skb timestamp for receiver side

[Gerrit Renker]

|  And this creates an even deeper dependency on the global packet
|  timestamping facility.  Those are supposed to be enabled only
|  in obscure circumstances because timestamping every packet is
|  incredibly expensive.
... these patches will go into the bin as well.

Re: [PATCH 07/29] Use skb timestamp for receiver side

[Gerrit Renker]

Quoting Ian McDonald:
|  Well I would have much rather discussed in private but a) you've told
|  Eddie not to reply in public and the way stated seems to imply I            
                         ^--- u mean `private' ?
|  shouldn't either b) you said that we were unreasonably holding up your
|  patches in public. It warranted a response but I'm done with my name
|  calling (for now).
If it helps to let off steam ... I am not bearing any grudges. It is good to
vent anger (as Metallica would have us believe). The converse is unhealthy
(as Karl A. Menninger `Man against himself' makes clear).

With regards to offline discussions - yes, please, I would indeed much rather 
have them out in the open.


|  > If for instance you read the documentation accompanying that patch, there is a
|  > difference between 100 usec ... 1msec between skb_get_timestamp and taking the timestamp
|  > in the CCID3 receiver. It all adds up.
|  >
|  I think we need to look at whether the protocol is right for LANs but
|  I think if we carry out Eddie's suggestions it may well be right.
|  Against earlier code a couple of months ago with a couple of my
|  patches I was achieving around 90 Mbits/sec on iperf which matched TCP
|  performance and it was stable. It responded gracefully to loss also.
I have two boxes with Gbit cards here and saw speeds up to 750..820 Mbit.
It seems that the computation overhead allows it to reach up to 80..90% of the
link bandwidth (with coarse-granularity problems still unresolved).

  
|  I have not tested lately and tried to replicate but I know that the
|  state of tree before current patches couldn't do that any more... As
|  time permits I will redo the work.
Do you remember when the `bidirectional' patch was reverted? - after that the 
CCID3 sender slowed down again to 75..80 Mbits/sec on a 100Mbit link.
This comes from the processing overhead, and was the original motivation for 
this patch. 

That incidentally was the second `moronic' patch I had submitted (as it limited
connections to being one-directional). Without Andre's help it would probably still 
be in. This really is the point I was trying to make in the email - I think we need to
focus more on the code. Here are two cases where the missing review/input came from outside.

I am not claiming to be any special. My hacking skills are rather moderate (i.e. you
have been warned about them patches :)
  
Thanks.

Re: [PATCH 07/29] Use skb timestamp for receiver side

[Arnaldo Carvalho de Melo]

On 4/13/07, Gerrit Renker <gerrit@erg.abdn.ac.uk> wrote:
> Quoting Ian McDonald:

> |  > If for instance you read the documentation accompanying that patch, there is a
> |  > difference between 100 usec ... 1msec between skb_get_timestamp and taking the timestamp
> |  > in the CCID3 receiver. It all adds up.
> |  >
> |  I think we need to look at whether the protocol is right for LANs but
> |  I think if we carry out Eddie's suggestions it may well be right.
> |  Against earlier code a couple of months ago with a couple of my
> |  patches I was achieving around 90 Mbits/sec on iperf which matched TCP
> |  performance and it was stable. It responded gracefully to loss also.

> I have two boxes with Gbit cards here and saw speeds up to 750..820 Mbit.
> It seems that the computation overhead allows it to reach up to 80..90% of the
> link bandwidth (with coarse-granularity problems still unresolved).
>
>
> |  I have not tested lately and tried to replicate but I know that the
> |  state of tree before current patches couldn't do that any more... As
> |  time permits I will redo the work.

> Do you remember when the `bidirectional' patch was reverted? - after that the
> CCID3 sender slowed down again to 75..80 Mbits/sec on a 100Mbit link.
> This comes from the processing overhead, and was the original motivation for
> this patch.

How many connections? Up to now, when I was more involved in DCCP
development, for the sake of testing the correctness of the protocol I
mostly tested with just a few connections, most of the time with just
one, which is OK while we're not yet feeling so good about the overal
correctness of the implementation, and because I mostly reused the TCP
machinery, but for performance we really have to test with many
connections, and in fact in combination with TCP connections, so that
we can see how DCCP affects overal system performance/stability.

> That incidentally was the second `moronic' patch I had submitted (as it limited
> connections to being one-directional). Without Andre's help it would probably still
> be in. This really is the point I was trying to make in the email - I think we need to
> focus more on the code. Here are two cases where the missing review/input came from outside.
>
> I am not claiming to be any special. My hacking skills are rather moderate (i.e. you
> have been warned about them patches :)

Neither me, and everybody commit mistakes, I should have known better
that net_enable_timestamp was not supposed to be enabled for normal
connections, just for things like tcpdump, etc, but in the end I
pushed for Dave trying to move things a bit forward, my bad, we really
have to take into account decisions we make that affects the rest of
the system :-\

- Arnaldo

Re: [PATCH 07/29] Use skb timestamp for receiver side

[Gerrit Renker]

 |  > Do you remember when the `bidirectional' patch was reverted? - after that the
|  > CCID3 sender slowed down again to 75..80 Mbits/sec on a 100Mbit link.
|  > This comes from the processing overhead, and was the original motivation for
|  > this patch.
|  
|  How many connections? Up to now, when I was more involved in DCCP
|  development, for the sake of testing the correctness of the protocol I
|  mostly tested with just a few connections, most of the time with just
|  one, which is OK while we're not yet feeling so good about the overal
|  correctness of the implementation, and because I mostly reused the TCP
|  machinery, but for performance we really have to test with many
|  connections, and in fact in combination with TCP connections, so that
|  we can see how DCCP affects overal system performance/stability.
This was a single connection, the results that one gets with iperf in byte-blast mode.
I am reasonably positive that this is due to processing overhead and actually have a
concept how to fix this - by not delivering to the RX CCID when a node has set SHUT_RD.
The application can do this via shutdown() when it knows that it is not going to need
any data and will only send (e.g. a streaming audio server).

<snip>
|  but in the end I pushed for Dave trying to move things a bit forward, my bad, we really
|  have to take into account decisions we make that affects the rest of
|  the system :-\
I am sorry for the delay that this has caused, I much appreciate the support in getting
the patches worked through.