Using jiffies for tcp_time_stamp?

Using jiffies for tcp_time_stamp?

[Srećko Jurić-Kavelj]

Hi,

Recently I tackled round trip time estimation of a TCP connection.
After implementing a straight-forward approach (time stamping sending
and receiving of data using clock_gettime) I found this article:
http://linuxgazette.net/136/pfeiffer.html (using getsockopt() to get
struct tcp_info). The tcp_info structure conveniently has a rtt field.

Using the first method I get 1-3 ms RTT, and by using the second I get
>10ms RTT.

By looking at the code it's clear that the time stamping is done with
jiffies, and my kernel has CONFIG_HZ=100.

I understand that this is for performance reasons (and the RTT
smoothing filter is implemented with bit shifting operations), but
would using a more precise time stamp have significant impact on
performance? Since RTT is used to compute RTO, wouldn't there be any
benefits of having more accurate estimate of this value?

Best regards,

Srećko Jurić-Kavelj, dipl.ing. (Ms.E.E)
Research and Teaching Assistant at University of Zagreb
(Faculty of Electrical Engineering and Computing, Department of
Control and Computer Engineering)

E-mail: srecko.juric-kavelj@fer.hr
URL: http://www.fer.hr/srecko.juric-kavelj

Sanctus Hieronymus: "Parce mihi, Domine, quia dalmata sum!"

Re: Using jiffies for tcp_time_stamp?

[David Miller]

Please discuss networking on the networking developer list,
netdev@vger.kernel.org, as most networking developers do
not read linux-kernel

Using jiffies for tcp_time_stamp?

[Srećko Jurić-Kavelj]

Hi,

Recently I tackled round trip time estimation of a TCP connection.
After implementing a straight-forward approach (time stamping sending
and receiving of data using clock_gettime) I found this article:
http://linuxgazette.net/136/pfeiffer.html (using getsockopt() to get
struct tcp_info). The tcp_info structure conveniently has a rtt field.

Using the first method I get 1-3 ms RTT, and by using the second I get
>=10 ms RTT.

By looking at the code it's clear that the time stamping is done with
jiffies, and my kernel has CONFIG_HZ=100.

I understand that this is for performance reasons (and the RTT
smoothing filter is implemented with bit shifting operations), but
would using a more precise time stamp have significant impact on
performance? Since RTT is used to compute RTO, wouldn't there be any
benefits of having more accurate estimate of this value?

Best regards,

Srećko Jurić-Kavelj, dipl.ing. (Ms.E.E)
Research and Teaching Assistant at University of Zagreb
(Faculty of Electrical Engineering and Computing, Department of
Control and Computer Engineering)

E-mail: srecko.juric-kavelj@fer.hr
URL: http://www.fer.hr/srecko.juric-kavelj

Sanctus Hieronymus: "Parce mihi, Domine, quia dalmata sum!"

Re: Using jiffies for tcp_time_stamp?

[Chris Friesen]

On 05/22/2012 11:21 AM, Srećko Jurić-Kavelj wrote:

> By looking at the code it's clear that the time stamping is done with
> jiffies, and my kernel has CONFIG_HZ=100.
>
> I understand that this is for performance reasons (and the RTT
> smoothing filter is implemented with bit shifting operations), but
> would using a more precise time stamp have significant impact on
> performance? Since RTT is used to compute RTO, wouldn't there be any
> benefits of having more accurate estimate of this value?

I don't know if it would make any difference to the tcp algorithms, but 
certainly on some architectures you can get a fast and accurate hardware 
timestamp.

Chris

Re: Using jiffies for tcp_time_stamp?

[Dave Taht]

On Fri, May 25, 2012 at 4:58 PM, Chris Friesen
<chris.friesen@genband.com> wrote:
> On 05/22/2012 11:21 AM, Srećko Jurić-Kavelj wrote:
>
>> By looking at the code it's clear that the time stamping is done with
>> jiffies, and my kernel has CONFIG_HZ=100.
>>
>> I understand that this is for performance reasons (and the RTT
>> smoothing filter is implemented with bit shifting operations), but
>> would using a more precise time stamp have significant impact on
>> performance? Since RTT is used to compute RTO, wouldn't there be any
>> benefits of having more accurate estimate of this value?
>
>
> I don't know if it would make any difference to the tcp algorithms, but
> certainly on some architectures you can get a fast and accurate hardware
> timestamp.

I would be interested in someone doing that experiment in light of the
codel work.
>
> Chris
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: Using jiffies for tcp_time_stamp?

[Srećko Jurić-Kavelj]

On Fri, May 25, 2012 at 6:17 PM, Dave Taht <dave.taht@gmail.com> wrote:
> On Fri, May 25, 2012 at 4:58 PM, Chris Friesen
> <chris.friesen@genband.com> wrote:
>> I don't know if it would make any difference to the tcp algorithms, but
>> certainly on some architectures you can get a fast and accurate hardware
>> timestamp.
>
> I would be interested in someone doing that experiment in light of the
> codel work.

I've looked this up in other implementations, e.g. FreeBSD uses 1ms
granularity no matter what HZ says, NetBSD has 500ms ticks, ...

I guess that granularity also depends on the retransmit timers used. I
didn't make out what's the precision of the timers that Linux uses in
TCP, but I guess it uses high resolution timers? At least on x86?

I've done a simple experiment by repeatedly calling clock_gettime
(from userspace, but I guess it ends up as a vsyscall). I get >17
million calls per second on a Q6600.

--
JKS

Re: Using jiffies for tcp_time_stamp?

[Eric Dumazet]

On Fri, 2012-05-25 at 18:23 +0200, Srećko Jurić-Kavelj wrote:
> On Fri, May 25, 2012 at 6:17 PM, Dave Taht <dave.taht@gmail.com> wrote:
> > On Fri, May 25, 2012 at 4:58 PM, Chris Friesen
> > <chris.friesen@genband.com> wrote:
> >> I don't know if it would make any difference to the tcp algorithms, but
> >> certainly on some architectures you can get a fast and accurate hardware
> >> timestamp.
> >
> > I would be interested in someone doing that experiment in light of the
> > codel work.
> 
> I've looked this up in other implementations, e.g. FreeBSD uses 1ms
> granularity no matter what HZ says, NetBSD has 500ms ticks, ...
> 
> I guess that granularity also depends on the retransmit timers used. I
> didn't make out what's the precision of the timers that Linux uses in
> TCP, but I guess it uses high resolution timers? At least on x86?
> 
> I've done a simple experiment by repeatedly calling clock_gettime
> (from userspace, but I guess it ends up as a vsyscall). I get >17
> million calls per second on a Q6600.

linux TCP uses high precision timestamps (ktime_get_real()) where
needed.

# find net|xargs grep -n TCP_CONG_RTT_STAMP
net/ipv4/tcp_veno.c:205:	.flags		= TCP_CONG_RTT_STAMP,
net/ipv4/tcp_vegas.c:308:	.flags		= TCP_CONG_RTT_STAMP,
net/ipv4/tcp_cubic.c:478:		cubictcp.flags |= TCP_CONG_RTT_STAMP;
net/ipv4/tcp_output.c:815:	if (icsk->icsk_ca_ops->flags & TCP_CONG_RTT_STAMP)
net/ipv4/tcp_lp.c:317:	.flags = TCP_CONG_RTT_STAMP,
net/ipv4/tcp_yeah.c:229:	.flags		= TCP_CONG_RTT_STAMP,
net/ipv4/tcp_illinois.c:326:	.flags		= TCP_CONG_RTT_STAMP,
net/ipv4/tcp_input.c:3496:				if (ca_ops->flags & TCP_CONG_RTT_STAMP &&

Other than that HZ=1000 seems fine.

HZ=100 seems a poor choice, we have NO_HZ since a long time.

Re: Using jiffies for tcp_time_stamp?

[Srećko Jurić-Kavelj]

On Fri, May 25, 2012 at 6:54 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> linux TCP uses high precision timestamps (ktime_get_real()) where
> needed.
>
> # find net|xargs grep -n TCP_CONG_RTT_STAMP
> net/ipv4/tcp_veno.c:205:        .flags          = TCP_CONG_RTT_STAMP,
> net/ipv4/tcp_vegas.c:308:       .flags          = TCP_CONG_RTT_STAMP,
> net/ipv4/tcp_cubic.c:478:               cubictcp.flags |= TCP_CONG_RTT_STAMP;
> net/ipv4/tcp_output.c:815:      if (icsk->icsk_ca_ops->flags & TCP_CONG_RTT_STAMP)
> net/ipv4/tcp_lp.c:317:  .flags = TCP_CONG_RTT_STAMP,
> net/ipv4/tcp_yeah.c:229:        .flags          = TCP_CONG_RTT_STAMP,
> net/ipv4/tcp_illinois.c:326:    .flags          = TCP_CONG_RTT_STAMP,
> net/ipv4/tcp_input.c:3496:                              if (ca_ops->flags & TCP_CONG_RTT_STAMP &&

Didn't know about TCP_CONG_RTT_STAMP.

Thing is, the device I'm connecting to doesn't even support TCP time
stamp option. The returning SYN ACK packet only has maximum segment
size 1460 bytes in options.

From the net/ipv4/tcp_input.c code, RTT is estimated using

#define tcp_time_stamp          ((__u32)(jiffies))

from include/net/tcp.h.

Could ktime_get_real() be used for tcp_time_stamp instead of jiffies?


> Other than that HZ=1000 seems fine.
>
> HZ=100 seems a poor choice, we have NO_HZ since a long time.

I have:
$ grep HZ /boot/config-2.6.32-41-generic
CONFIG_NO_HZ=y
CONFIG_HZ_100=y
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=100
CONFIG_MACHZ_WDT=m

From what I've seen in the code, NO_HZ doesn't make jiffies go away,
it simply doesn't use regular CONFIG_HZ interrupt to update, but
updates them when has an opportunity?

--
JKS

Re: Using jiffies for tcp_time_stamp?

[Eric Dumazet]

On Fri, 2012-05-25 at 20:35 +0200, Srećko Jurić-Kavelj wrote:

> From what I've seen in the code, NO_HZ doesn't make jiffies go away,
> it simply doesn't use regular CONFIG_HZ interrupt to update, but
> updates them when has an opportunity?

HZ=1000 makes jiffies 10 times more precise, and with NO_HZ, generates
no extra timer interrupts.

This also makes timers workload smoothed, instead of spikes.