How much we can trust packet timestamping

How much we can trust packet timestamping

[uaca]

Hi all

IMHO The problem is quite complicated because

+ common hardware is not designed for real time:

	- sends multiple PDUs within one interrupt, and can be delayed
	- Host adapter bus & infraestructure is not designed to garantee latency
  	etc...
     
+ software is also not designed for realtime

	- drivers may timestamp in softirq's
	- irqs has no deterministic latency either
	etc...

So even if do_gettimeofday() has 1/CPUfreq resolution by using TSC register
packet timestamping meassurement is biased, how much?

anybody has studied this? are there reports/doc about this topic?

of course, this can be avoided by using specialiced hardware, but I'm not
interested on that and I would like to know how much I can trust this
timestamps

Any comment would be greatly appreciated

Thanks

	Ulisses

                Debian GNU/Linux: a dream come true
-----------------------------------------------------------------------------
"Computers are useless. They can only give answers."            Pablo Picasso

--->	Visita http://www.valux.org/ para saber acerca de la	<---
--->	Asociación Valenciana de Usuarios de Linux		<---

Re: How much we can trust packet timestamping

[Alan Cox]

On Mon, 2002-12-30 at 11:28, uaca@alumni.uv.es wrote:
> Hi all
> 
> IMHO The problem is quite complicated because
> 
> + common hardware is not designed for real time:
> 
> 	- sends multiple PDUs within one interrupt, and can be delayed
> 	- Host adapter bus & infraestructure is not designed to garantee latency
>   	etc...

The packet can be timestamped by the hardware receiving as well as by
the kernel netif_rx code. This is actually intentional and there is
hardware that supports doing IRQ raise time sampling which the driver
can then use to get very accurate data.

Alan

Re: How much we can trust packet timestamping

[uaca]

On Mon, Dec 30, 2002 at 01:09:03PM +0000, Alan Cox wrote:
> On Mon, 2002-12-30 at 11:28, uaca@alumni.uv.es wrote:
> > Hi all
> > 
> > IMHO The problem is quite complicated because
> > 
> > + common hardware is not designed for real time:
> > 
> > 	- sends multiple PDUs within one interrupt, and can be delayed
> > 	- Host adapter bus & infraestructure is not designed to garantee latency
> >   	etc...
> 
> The packet can be timestamped by the hardware receiving as well as by
> the kernel netif_rx code. This is actually intentional and there is
> hardware that supports doing IRQ raise time sampling which the driver
> can then use to get very accurate data.

Thanks Alan

Anybody know about a Linux driver that supports doing IRQ raise time
sampling? any doc/pointer/suggestion would be greatly appreciated

Thanks in advance

	Ulisses
                Debian GNU/Linux: a dream come true
-----------------------------------------------------------------------------
"Computers are useless. They can only give answers."            Pablo Picasso

--->	Visita http://www.valux.org/ para saber acerca de la	<---
--->	Asociación Valenciana de Usuarios de Linux		<---
 

Re: How much we can trust packet timestamping

[Alan Cox]

On Mon, 2002-12-30 at 13:01, uaca@alumni.uv.es wrote:
> Anybody know about a Linux driver that supports doing IRQ raise time
> sampling? any doc/pointer/suggestion would be greatly appreciated

Caederus has engineers who did the ISA bus hardware that led to the
SIOCGSTAMP facility in the first place (then the tcpdump type folks
decided it was really rather cool so it went generic anyway)

www.caederus.co.uk

I don't know if anyone is doing this in the PCI bus world

Re: How much we can trust packet timestamping

[Werner Almesberger]

Alan Cox wrote:
> The packet can be timestamped by the hardware receiving as well as by
> the kernel netif_rx code. This is actually intentional and there is
> hardware that supports doing IRQ raise time sampling which the driver
> can then use to get very accurate data.

By the way, the group of Darryl Veitch have done some extremely
interesting work with high-resolution timestamps, in particular
using the TSC on recent ia32:

http://www.cubinlab.ee.mu.oz.au/probing/
http://www.cubinlab.ee.mu.oz.au/~darryl/tscclock_final.pdf.gz

One general issue in this area is what we can do with time
sources that aren't system-wide, e.g. NIC-local timers. The
problem is to calibrate them and to synchronize them to
wall-clock time. I think there are basically two possible
approaches:

 1) driver gives time synchronization system (in user-space)
    access to "raw" running timer value. Timestamps are also
    "raw" timer values, plus a time source ID, which can then
    be used to convert the values to wall-clock time.

 2) user space pushes exact time to kernel space, which then
    does all the math. Timestamps are already converted to
    wall-clock time.

2) is essentially what we can do with today's interfaces (an
event notifier would be useful, though). The big drawback is
that non-trivial math would have to be done in kernel space.
1) is much easier on the kernel, but has the issue of
requiring some API to get time values and time source
characteristics (time representation, range, etc.).

I'm leaning towards solution 1), because it keeps things simple
for the kernel. But perhaps the best approach is to simply
implement both, and then compare ...

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: How much we can trust packet timestamping

[Alan Cox]

On Fri, 2003-01-10 at 22:07, Werner Almesberger wrote:
> One general issue in this area is what we can do with time
> sources that aren't system-wide, e.g. NIC-local timers. The
> problem is to calibrate them and to synchronize them to
> wall-clock time. I think there are basically two possible
> approaches:

You run NTP between the host clock and the nic timer. Its
all you really need. In the i2it hardware the NIC clock 
implemented hardware slewing so it could do NTP stuff

Re: How much we can trust packet timestamping

[Werner Almesberger]

Alan Cox wrote:
> You run NTP between the host clock and the nic timer.

While NTP is a good synchronization source (frequently the only
affordable one around), I'm not so sure it's such a good tool
for correcting drift. If you have a look at figure 5 in
http://www.cubinlab.ee.mu.oz.au/~darryl/tscclock_final.pdf.gz
you'll see that NTP uses drift to correct for offset errors, so
using NTP directly doesn't yield a clock that remains stable
unless it's constantly getting corrected by NTP.

What should work better is to use NTP only as a reference for
offset, and then calibrate the hardware clock from that.
Particularly the TSC is very stable, so there isn't much drift
to worry about.

But what I'm after is the interface between kernel and user space,
and any kernel-internal interfaces that may be needed. If people
really want to use NTP directly on hardware clocks, I guess my
approach 1) (export everything to user space, and let user space
worry about the details) would then be the appropriately flexible
choice ?

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/