Re: [Xenomai-core] Synchronising TSC and periodic timer

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1410 bytes --]

Fillod Stephane wrote:
> Jan Kiszka wrote:
>> Gilles Chanteperdrix wrote:
>>> Jan Kiszka wrote:
>>>  > Likely too simple: The periodic IRQ seems to pop up on every CPU
> so that
>>>  > the TSC could be recorded, but will this happen synchronously? At
> least
>>>  > we will see (IRQ) jitters, and those jitters could already create
> in the
>>>  > single-CPU case a non-monotonic clock...
>>>
>>> I do not know how this issue is solved in Linux, but there seem to be
> a
>>> simple solution: before adding the tsc offset to the last tick time,
>>> this tsc offset should be compared with the tick duration in tsc
>>> counts; if it is greater, then replace with the tick duration in tsc.
>>>
>> Hmm, I would rather express it in absolute tsc values, i.e. always save
>> the tuple (absolute_tsc, jiffies):
>>
>> [timer IRQ]
>> new_tsc = read_rsc() - old_tsc;
>> if (new_tsc < old_tsc + period_in_tsc_ticks)
>> 	new_tsc = old_tsc + period_in_tsc;
>> old_tsc = new_tsc;
>>
>> Disclaimer: I haven't thought about potential accuracy side effects of
>> this implementation, e.g. what would happen over the long term if the
>> condition is always fulfilled and executed...
> 
> Aha! the wonderful world of PLL's :-)
> 

Yes! I was waiting for someone to throw this in ;). But do we really
need them, or are there so cheap implementations that the overhead
doesn't matter?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

[Xenomai-core] Synchronising TSC and periodic timer

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 776 bytes --]

Hi,

for those who haven't followed the endless "RTDM and Timer functions"
thread: we are currently discussing a way to provide high-resolution
timestamps in periodic mode for RTDM users. It was suggested to use the
TSC for this, but I noted that this source will not be in sync with the
periodic system timer and may even be out of sync across multiple CPUs.

A straight-forward approach to overcome this might be to record the
current TSC value together with the current jiffies in
xntimer_do_tick_periodic(). This tuple (per CPU) could then be provided
to skin implementers in order to let them offer a high-resolution
timestamp source even in periodic mode. Hmm, sounds too simple actually,
so I'm waiting now for someone pointing out the pitfalls.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1295 bytes --]

Jan Kiszka wrote:
> Hi,
> 
> for those who haven't followed the endless "RTDM and Timer functions"
> thread: we are currently discussing a way to provide high-resolution
> timestamps in periodic mode for RTDM users. It was suggested to use the
> TSC for this, but I noted that this source will not be in sync with the
> periodic system timer and may even be out of sync across multiple CPUs.
> 
> A straight-forward approach to overcome this might be to record the
> current TSC value together with the current jiffies in
> xntimer_do_tick_periodic(). This tuple (per CPU) could then be provided
> to skin implementers in order to let them offer a high-resolution
> timestamp source even in periodic mode. Hmm, sounds too simple actually,
> so I'm waiting now for someone pointing out the pitfalls.

Likely too simple: The periodic IRQ seems to pop up on every CPU so that
the TSC could be recorded, but will this happen synchronously? At least
we will see (IRQ) jitters, and those jitters could already create in the
single-CPU case a non-monotonic clock...

Does anyone ever studied if and how Linux synchronises across CPUs?
There was some activity around the problematic AMD64 multicores, but I
haven't looked at the details and if it's actually solved now.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2031 bytes --]

Anders Blomdell wrote:
> Jan Kiszka wrote:
>> Jan Kiszka wrote:
>>
>>> Hi,
>>>
>>> for those who haven't followed the endless "RTDM and Timer functions"
>>> thread: we are currently discussing a way to provide high-resolution
>>> timestamps in periodic mode for RTDM users. It was suggested to use the
>>> TSC for this, but I noted that this source will not be in sync with the
>>> periodic system timer and may even be out of sync across multiple CPUs.
>>>
>>> A straight-forward approach to overcome this might be to record the
>>> current TSC value together with the current jiffies in
>>> xntimer_do_tick_periodic(). This tuple (per CPU) could then be provided
>>> to skin implementers in order to let them offer a high-resolution
>>> timestamp source even in periodic mode. Hmm, sounds too simple actually,
>>> so I'm waiting now for someone pointing out the pitfalls.
>>
>>
>> Likely too simple: The periodic IRQ seems to pop up on every CPU so that
>> the TSC could be recorded, but will this happen synchronously? At least
>> we will see (IRQ) jitters, and those jitters could already create in the
>> single-CPU case a non-monotonic clock...
> Returning a struct with { jiffies, cpu#, tsc, clockscaling, ... } and
> routines to compare ( lt | equal | gt | unordered ) and calculate
> differences { diff, accuracy }. And then people (of course) will send
> them over the network and compare items emanating from different systems!
> 

Well, I think it's more obvious that timestamps taken on box A will not
necessarily match timestamps of box B. Moreover, tasks on box A will not
that often get migrated automatically to box B, while this can happen
inside SMP boxes if you do not explicitly bind to a certain CPU.

For the local usage I have this scheme in mind:

local_irqs_off();
read_cpu_tsc();
get_last_cpu_tsc_offset();
local_irqs_on();
calc_corrected_time();

So, nothing has to be exported to the skin user.

Or did I misunderstand what you wanted to express?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Philippe Gerum]

Jan Kiszka wrote:
> Jan Kiszka wrote:
> 
>>Hi,
>>
>>for those who haven't followed the endless "RTDM and Timer functions"
>>thread: we are currently discussing a way to provide high-resolution
>>timestamps in periodic mode for RTDM users. It was suggested to use the
>>TSC for this, but I noted that this source will not be in sync with the
>>periodic system timer and may even be out of sync across multiple CPUs.
>>
>>A straight-forward approach to overcome this might be to record the
>>current TSC value together with the current jiffies in
>>xntimer_do_tick_periodic(). This tuple (per CPU) could then be provided
>>to skin implementers in order to let them offer a high-resolution
>>timestamp source even in periodic mode. Hmm, sounds too simple actually,
>>so I'm waiting now for someone pointing out the pitfalls.
> 
> 
> Likely too simple: The periodic IRQ seems to pop up on every CPU so that

Talking about x86, indeed it does, so that each per-CPU scheduler can have its own 
time feed. The basic principle is as follows on Xenomai SMP/x86:

- a spare APIC vector is used to relay the timer ticks to the Xenomai domain, 
grabbing the APIC timer to trigger those ticks (see RTHAL_APIC_TIMER_VECTOR).
- Xenomai timer ticks are programmed to occur on all CPUs, feeding the per-CPU 
schedulers with timing data, so that each CPU can in turn update the global 
scheduling state according to possible wakeups/suspensions.
- when a thread needs to be awaken or suspended on a remote CPU after a local tick 
on the local one, an IPI is sent to invoke the rescheduling procedure on the 
remote CPU (see xnpod_schedule)).

This also applies to Xeno's ia64 port.

> the TSC could be recorded, but will this happen synchronously?

Nope. You could not predict the state of hardware and/or virtual interrupt masking 
on a particular CPU when the IRQ get raised anyway.

  At least
> we will see (IRQ) jitters, and those jitters could already create in the
> single-CPU case a non-monotonic clock...
> 

At worst, you would see an old timestamp from a previous shot while the timer IRQ 
announcing the most accurate one is still outstanding but untaken, but I think 
that you would still have something behaving in a monotonic way though.

> Does anyone ever studied if and how Linux synchronises across CPUs?
> There was some activity around the problematic AMD64 multicores, but I
> haven't looked at the details and if it's actually solved now.
> 

Only once during boot AFAICT, see arch/i386/kernel/smpboot.c. This said, TSC 
synchronization would not work on NUMA boxen.

> Jan
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Xenomai-core mailing list
> Xenomai-core@domain.hid
> https://mail.gna.org/listinfo/xenomai-core


-- 

Philippe.

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Gilles Chanteperdrix]

Philippe Gerum wrote:
 > At worst, you would see an old timestamp from a previous shot while the timer IRQ 
 > announcing the most accurate one is still outstanding but untaken, but I think 
 > that you would still have something behaving in a monotonic way though.
 > 
 > > Does anyone ever studied if and how Linux synchronises across CPUs?
 > > There was some activity around the problematic AMD64 multicores, but I
 > > haven't looked at the details and if it's actually solved now.
 > > 
 > 
 > Only once during boot AFAICT, see arch/i386/kernel/smpboot.c. This said, TSC 
 > synchronization would not work on NUMA boxen.

I think Jan is talking about using TSC to get intra-ticks precise clock,
by adding tsc offsets to the time derived from the clock irqs count.
This would allow, for example, to run the "latency" test with the timer
set in periodic mode.

The issue with non-monotonic values happens if two clock interrupts are
distant from a bit more than one tick, because of the jitter. Reading
the time just before the second irq then yield a greater value than the
one read just after the second irq.

-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Philippe Gerum]

Gilles Chanteperdrix wrote:
> Philippe Gerum wrote:
>  > At worst, you would see an old timestamp from a previous shot while the timer IRQ 
>  > announcing the most accurate one is still outstanding but untaken, but I think 
>  > that you would still have something behaving in a monotonic way though.
>  > 
>  > > Does anyone ever studied if and how Linux synchronises across CPUs?
>  > > There was some activity around the problematic AMD64 multicores, but I
>  > > haven't looked at the details and if it's actually solved now.
>  > > 
>  > 
>  > Only once during boot AFAICT, see arch/i386/kernel/smpboot.c. This said, TSC 
>  > synchronization would not work on NUMA boxen.
> 
> I think Jan is talking about using TSC to get intra-ticks precise clock,
> by adding tsc offsets to the time derived from the clock irqs count.
> This would allow, for example, to run the "latency" test with the timer
> set in periodic mode.
> 
> The issue with non-monotonic values happens if two clock interrupts are
> distant from a bit more than one tick, because of the jitter. Reading
> the time just before the second irq then yield a greater value than the
> one read just after the second irq.
> 

The issue that worries me - provided that we bound the adjustment offset to the 
duration of one tick after some jittery - is that any attempt to get intra-tick 
precision would lead to a possible discrepancy regarding the elapsed time 
according to those two different scales, between the actual count of jiffies 
tracked by the timer ISR on the timekeeper CPU, and the corrected time value 
returned by rtdm_read_clock. And this discrepancy would last for the total 
duration of the jitter. E.g., for a 100 us period, xnpod_get_time() could return 2 
albeit rtdm_read_clock returns 300, instead of 200. Spuriously mixing both units 
in applications would lead to some funky chaos.

-- 

Philippe.

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1145 bytes --]

Philippe Gerum wrote:
> ...
> The issue that worries me - provided that we bound the adjustment offset
> to the duration of one tick after some jittery - is that any attempt to
> get intra-tick precision would lead to a possible discrepancy regarding
> the elapsed time according to those two different scales, between the
> actual count of jiffies tracked by the timer ISR on the timekeeper CPU,
> and the corrected time value returned by rtdm_read_clock. And this
> discrepancy would last for the total duration of the jitter. E.g., for a
> 100 us period, xnpod_get_time() could return 2 albeit rtdm_read_clock
> returns 300, instead of 200. Spuriously mixing both units in
> applications would lead to some funky chaos.
> 

Trying to pick up this thread again, I just tried to understand your
concerns, but failed so far to imagine a concrete scenario. Could you
sketch such a "funky chaotic" situation from the application point of
view? And what would prevent us from improving the accuracy of other
timestamping API functions beyond RTDM as well, e.g. on converting from
ticks to nanos in rt_timer_ticks2ns()?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Philippe Gerum]

Jan Kiszka wrote:
> Philippe Gerum wrote:
> 
>>...
>>The issue that worries me - provided that we bound the adjustment offset
>>to the duration of one tick after some jittery - is that any attempt to
>>get intra-tick precision would lead to a possible discrepancy regarding
>>the elapsed time according to those two different scales, between the
>>actual count of jiffies tracked by the timer ISR on the timekeeper CPU,
>>and the corrected time value returned by rtdm_read_clock. And this
>>discrepancy would last for the total duration of the jitter. E.g., for a
>>100 us period, xnpod_get_time() could return 2 albeit rtdm_read_clock
>>returns 300, instead of 200. Spuriously mixing both units in
>>applications would lead to some funky chaos.
>>
> 
> 
> Trying to pick up this thread again, I just tried to understand your
> concerns, but failed so far to imagine a concrete scenario. Could you
> sketch such a "funky chaotic" situation from the application point of
> view?

Given the description above, just that some skin might return either 
nucleus ticks or corrected timestamps to the applications, which would 
in turn do some arithmetics for converting values they got from the skin 
between both scales internally, and mistakenly use the result while 
assuming that both scales are always in sync. In this situation, and 
during a fraction of the time (i.e. the jitter), both scales might not 
be in sync, and the result would be unusable. This said, this kind of 
issue could be solved by big fat warnings in documentation, explicitely 
saying that conversions between both scales might be meaningless.

And what would prevent us from improving the accuracy of other
> timestamping API functions beyond RTDM as well, e.g. on converting from
> ticks to nanos in rt_timer_ticks2ns()?
>

I don't understand why rt_timer_ticks2ns() should be impacted by such 
extension. This service must keep a constant behaviour, regardless of 
any outstanding timing issue. I mean, 3 ticks from a 1Khz clock rate 
must always return 3,000,000 nanos, unless you stop passing count of 
ticks but fractional/compound values instead.

The bottom-line is that we should not blur the line between periodic and 
aperiodic timing modes, just for getting precise timestamps in the 
former case. Additionally, and x86-wise, when no TSC is available on the 
target system, rt_timer_tsc() already returns a timestamp obtained from 
the 8254's channel #2 we use as a free running counter, which is the 
most precise source we have at hand to do so.

Periodic mode bears its own limitation, which is basically a loss of 
accuracy we trade against a lower overhead (even if that does not mean 
much except perhaps on x86). What we could do is reducing the jittery 
involved in periodic ticks, by always emulating periodic mode over 
aperiodic shots instead of using e.g. the 8254 in PIT mode (and remove 
the need for the double scale on x86, tsc + 8254 channel #1), but not 
change the basic meaning of periodic timing.

But maybe we are still discussing different issues actually, so it would 
be useful that the core issue that triggered the discussion about 
periodic mode precision be exposed again.


-- 

Philippe.

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Rodrigo Rosenfeld Rosas]

Philippe Gerum wrote:
>...
>Given the description above, just that some skin might return either
>nucleus ticks or corrected timestamps to the applications, which would
>in turn do some arithmetics for converting values they got from the skin
>between both scales internally, and mistakenly use the result while
>assuming that both scales are always in sync. In this situation, and
>during a fraction of the time (i.e. the jitter), both scales might not
>be in sync, and the result would be unusable.

But I still can't find a real situation where the user would need these values 
to be in sync...

>...
>But maybe we are still discussing different issues actually, so it would
>be useful that the core issue that triggered the discussion about
>periodic mode precision be exposed again.

The core issue is that:
I think the driver development should be kind of independent from the user 
programs. That said, if the driver needs a precise timestamp, it should be 
able to get it, even if the user is fine with a, say, 100ms periodic tick. If 
the user have a loop with a deadline of 100ms, and if it takes two 
consecutive images for estimating speed in the same loop, the user will need 
to have a higher precision timestamps for the images. So, the driver will 
need a high precision timer reading for making possible to provide those 
timestamps...

I hope that was what you were asking for...

Regards,

Rodrigo.

		
_______________________________________________________
Yahoo! Acesso Grátis - Internet rápida e grátis. Instale o discador agora!
http://br.acesso.yahoo.com

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Philippe Gerum]

Rodrigo Rosenfeld Rosas wrote:
> Philippe Gerum wrote:
> 
>>...
>>Given the description above, just that some skin might return either
>>nucleus ticks or corrected timestamps to the applications, which would
>>in turn do some arithmetics for converting values they got from the skin
>>between both scales internally, and mistakenly use the result while
>>assuming that both scales are always in sync. In this situation, and
>>during a fraction of the time (i.e. the jitter), both scales might not
>>be in sync, and the result would be unusable.
> 
> 
> But I still can't find a real situation where the user would need these values 
> to be in sync...
> 
> 
>>...
>>But maybe we are still discussing different issues actually, so it would
>>be useful that the core issue that triggered the discussion about
>>periodic mode precision be exposed again.
> 
> 
> The core issue is that:
> I think the driver development should be kind of independent from the user 
> programs. That said, if the driver needs a precise timestamp, it should be 
> able to get it, even if the user is fine with a, say, 100ms periodic tick. If 
> the user have a loop with a deadline of 100ms, and if it takes two 
> consecutive images for estimating speed in the same loop, the user will need 
> to have a higher precision timestamps for the images. So, the driver will 
> need a high precision timer reading for making possible to provide those 
> timestamps...
> 

This is exactely what rt_timer_tsc() or clock_gettime(CLOCK_MONOTONIC) 
[or whatever ends up calling xnarch_get_cpu_tsc() from the underlying 
architecture] are there for.

The thing is that rt_timer_read() is expected to return values 
compatible with the timer mode, always. But rt_timer_tsc() is there to 
return the most precise timestamp available from the underlying 
architecture, regardless of the current timing mode. If no TSC exists on 
x86, then it is going to be emulated (using the PIT's channel #2), but 
in any case, you will get a high precision timestamp, up to the best 
precision the architecture can provide, that is.

The key issue is to acknowledge the fact that periodic ticks and precise 
timestamps are two _unrelated_ units. What I'm reluctant to is to try 
finding some artificial binding between both units, because there is 
none (being stable, that is).

In your example above, you would be able to estimate the elapsed time 
using something like rt_timer_tsc(), and convert this to ticks using 
rt_timer_ns2ticks(rt_timer_tsc2ns(timestamp)). The main problem would be 
the rounding here, and work around the lack of precision the periodic 
mode currently exhibits due to the constant delay between timer shots.

I think that you should try convincing Jan that rtdm_clock_tsc() might 
be a good idea to provide, instead of tweaking rtdm_clock_read() in a 
way which changes its underlying logic. ;o)


> I hope that was what you were asking for...
> 
> Regards,
> 
> Rodrigo.
> 
> 		
> _______________________________________________________
> Yahoo! Acesso Grátis - Internet rápida e grátis. Instale o discador agora!
> http://br.acesso.yahoo.com
> 
> 
> _______________________________________________
> Xenomai-core mailing list
> Xenomai-core@domain.hid
> https://mail.gna.org/listinfo/xenomai-core
> 


-- 

Philippe.

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Rodrigo Rosenfeld Rosas]

Em Segunda 20 Mar=E7o 2006 13:51, Philippe Gerum escreveu:

>...
>I think that you should try convincing Jan that rtdm_clock_tsc() might
>be a good idea to provide, instead of tweaking rtdm_clock_read() in a
>way which changes its underlying logic. ;o)

Yes, that is exactly what I want! :)
I don't see any reason for changing rtdm_timer_read() neither. I think that=
=20
the most common usage of high-precision timestamps is for relative time=20
cases. It doesn't need to be in sync with Xenomai's timer... It is best to=
=20
keep things simple.

What do you think Jan?

P.S: Sorry for the last message, Philippe. I didn't see that one at the tim=
e.

		
_______________________________________________________
Yahoo! Acesso Grátis - Internet rápida e grátis. Instale o discador agora!
http://br.acesso.yahoo.com

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2466 bytes --]

Rodrigo Rosenfeld Rosas wrote:
> Em Segunda 20 Março 2006 13:51, Philippe Gerum escreveu:
> 
>> ...
>> I think that you should try convincing Jan that rtdm_clock_tsc() might
>> be a good idea to provide, instead of tweaking rtdm_clock_read() in a
>> way which changes its underlying logic. ;o)
> 
> Yes, that is exactly what I want! :)
> I don't see any reason for changing rtdm_timer_read() neither. I think that 
> the most common usage of high-precision timestamps is for relative time 
> cases. It doesn't need to be in sync with Xenomai's timer... It is best to 
> keep things simple.
> 
> What do you think Jan?

We discussed a lot about how to prevent the user shooting him/herself in
the knee with inter-tick timestamps, but I still think that
rtdm_clock_read_tsc() would even be worse in this regard.

xnarch_get_cpu_tsc() and derived skin services are not supposed to
deliver consistent results across multiple CPUs, are they? While the
user could avoid such scenarios by locking tasks on a specific CPU,
drivers cannot - at least so far. So, to safely introduce such a
low-level service for RTDM, I think we need

A) CPU affinity for RTDM-registered IRQs
B) CPU affinity for RTDM kernel tasks
C) Some well written docs, explaining how to safely use TSCs at driver
level and how to provide them to the user (the latter aspect makes me
worry most)

While A) and B) might be useful for other (though rare) scenarios as
well, C) will still require a very good understanding and interface
design from the driver writer, while I don't see comparable error
dimensions with the improved rtdm_clock_read(). Comparing apples
(rtdm_clock_read()) to oranges (rt_timer_read()), there will be some
error around a tick period. But comparing apple[0]
(rtdm_clock_read_tsc() on CPU#0) to apple[1] (rtdm_clock_read_tsc() on
CPU#1), the error could become *much* larger and the design and
documentation effort to avoid this will be significant.

Ok, as a simple resolution for this problem, I could imagine introducing
a TSC timestamping service to RTDM that always fall back to that level
of accuracy which is guaranteed to be consistent - either because we run
in aperiodic mode, or on uniprocessor, or thanks to some magic
synchronisation between all CPU clocks. This would have to be decided at
build time, in the first version likely by checking for (multiprocessor
|| aperiodic) to switch to xnpod_get_time().

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Rodrigo Rosenfeld Rosas]

Jan Kiszka escreveu:

>We discussed a lot about how to prevent the user shooting him/herself in
>the knee with inter-tick timestamps, but I still think that
>rtdm_clock_read_tsc() would even be worse in this regard.
>  
>
What do you think abou this documentation:
"This function is meant to be used in periodic mode for getting a high 
timestamp, independently from the system timer's tick.
Its return values should not be used mixed with rtdm_clock_read() values 
because they are not syncronised.
Driver authors[developers?] are advised to state this on driver 
documentation for the final users where returning these values to them 
for avoiding confusion.

Note: This function is available for uniprocessor systems only in the 
meantime."

I think it explains and will not make confusion in driver developers... 
If sometime someone give a good solution to the syncronisation problem 
between multiple processors, this can be changed...

 > ...

Rodrigo.

	

	
		
_______________________________________________________ 
Yahoo! doce lar. Faça do Yahoo! sua homepage. 
http://br.yahoo.com/homepageset.html 

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 4980 bytes --]

Philippe Gerum wrote:
> Jan Kiszka wrote:
>> Philippe Gerum wrote:
>>
>>> ...
>>> The issue that worries me - provided that we bound the adjustment offset
>>> to the duration of one tick after some jittery - is that any attempt to
>>> get intra-tick precision would lead to a possible discrepancy regarding
>>> the elapsed time according to those two different scales, between the
>>> actual count of jiffies tracked by the timer ISR on the timekeeper CPU,
>>> and the corrected time value returned by rtdm_read_clock. And this
>>> discrepancy would last for the total duration of the jitter. E.g., for a
>>> 100 us period, xnpod_get_time() could return 2 albeit rtdm_read_clock
>>> returns 300, instead of 200. Spuriously mixing both units in
>>> applications would lead to some funky chaos.
>>>
>>
>>
>> Trying to pick up this thread again, I just tried to understand your
>> concerns, but failed so far to imagine a concrete scenario. Could you
>> sketch such a "funky chaotic" situation from the application point of
>> view?
> 
> Given the description above, just that some skin might return either
> nucleus ticks or corrected timestamps to the applications, which would
> in turn do some arithmetics for converting values they got from the skin
> between both scales internally, and mistakenly use the result while
> assuming that both scales are always in sync. In this situation, and
> during a fraction of the time (i.e. the jitter), both scales might not
> be in sync, and the result would be unusable. This said, this kind of
> issue could be solved by big fat warnings in documentation, explicitely
> saying that conversions between both scales might be meaningless.

So the worst-case is when a user derives some relative times from two
different time sources, one purely tick-base, the other improved by
inter-tick TSC (when available on that arch)?

Let's say the user takes timestamp t1 = rtdm_clock_read() (via some
driver) and a bit later t2 = rt_timer_tick2ns(rt_timer_read()). t1 was
set to the last tick Tn + the number of TSC ticks since then:

    t1 = Tn * tick_period + TSC_offset

With, e.g., Tn=1001, tick_period = 1000 us, and TSC_offset = 589 us:

    t1 = 1001 * 1000 us + 589 us = 1001569 us

As the next tick may have not stroke yet when taking t2, that value
converted to us can be smaller:

    t2 = Tn * tick_period = 1001000

Now the difference between t2 and t1 becomes negative (-589 us),
although the user may expect it t2-t1 >= 0. Is this non-monotony your
concern?


On the other hand, the advantage of TSC-based synchronised inter-tick
timestamps is that you can do things like

    sleep_until(rt_timer_ns2ticks(rtdm_clock_read() + 1000000))

without risking an error beyond +/- 1 tick (+jitter). With current
jiffies vs. TSC in periodic mode, this is not easily possible. You have
to sync in the application, creating another error source when the delay
between acquiring the TSC and sync'ing the TSC on jiffies is too long.

> 
> And what would prevent us from improving the accuracy of other
>> timestamping API functions beyond RTDM as well, e.g. on converting from
>> ticks to nanos in rt_timer_ticks2ns()?
>>
> 
> I don't understand why rt_timer_ticks2ns() should be impacted by such
> extension. This service must keep a constant behaviour, regardless of
> any outstanding timing issue. I mean, 3 ticks from a 1Khz clock rate
> must always return 3,000,000 nanos, unless you stop passing count of
> ticks but fractional/compound values instead.

Forget about this, it was (pre-lunch) nonsense.

> 
> The bottom-line is that we should not blur the line between periodic and
> aperiodic timing modes, just for getting precise timestamps in the
> former case. Additionally, and x86-wise, when no TSC is available on the
> target system, rt_timer_tsc() already returns a timestamp obtained from
> the 8254's channel #2 we use as a free running counter, which is the
> most precise source we have at hand to do so.
> 
> Periodic mode bears its own limitation, which is basically a loss of
> accuracy we trade against a lower overhead (even if that does not mean
> much except perhaps on x86). What we could do is reducing the jittery
> involved in periodic ticks, by always emulating periodic mode over
> aperiodic shots instead of using e.g. the 8254 in PIT mode (and remove
> the need for the double scale on x86, tsc + 8254 channel #1), but not
> change the basic meaning of periodic timing.

Hmm, interesting, and it also reminds of a long pending (slightly OT)
question I have: why not creating the infrastructure (a dedicated
periodic timer) for providing round-robin scheduling even in aperiodic mode?

> 
> But maybe we are still discussing different issues actually, so it would
> be useful that the core issue that triggered the discussion about
> periodic mode precision be exposed again.

Yep, Rodrigo...?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Philippe Gerum]

Jan Kiszka wrote:
> Philippe Gerum wrote:
>>Given the description above, just that some skin might return either
>>nucleus ticks or corrected timestamps to the applications, which would
>>in turn do some arithmetics for converting values they got from the skin
>>between both scales internally, and mistakenly use the result while
>>assuming that both scales are always in sync. In this situation, and
>>during a fraction of the time (i.e. the jitter), both scales might not
>>be in sync, and the result would be unusable. This said, this kind of
>>issue could be solved by big fat warnings in documentation, explicitely
>>saying that conversions between both scales might be meaningless.
> 
> 
> So the worst-case is when a user derives some relative times from two
> different time sources, one purely tick-base, the other improved by
> inter-tick TSC (when available on that arch)?
> 
> Let's say the user takes timestamp t1 = rtdm_clock_read() (via some
> driver) and a bit later t2 = rt_timer_tick2ns(rt_timer_read()). t1 was
> set to the last tick Tn + the number of TSC ticks since then:
> 
>     t1 = Tn * tick_period + TSC_offset
> 
> With, e.g., Tn=1001, tick_period = 1000 us, and TSC_offset = 589 us:
> 
>     t1 = 1001 * 1000 us + 589 us = 1001569 us
> 
> As the next tick may have not stroke yet when taking t2, that value
> converted to us can be smaller:
> 
>     t2 = Tn * tick_period = 1001000
> 
> Now the difference between t2 and t1 becomes negative (-589 us),
> although the user may expect it t2-t1 >= 0. Is this non-monotony your
> concern?
>

No, because both are sourced from xnpod_get_time(). My concern is that
the offset correction is going to cause situations where actual count of
ticks kept by the nucleus might be different from corrected_time /
period, due to the jitter issue, albeit the corrected timestamp is
expected to depend on the count of ticks. So you end up having two time
scales which are expected to be in sync, albeit there might not under
certain conditions.

> 
> On the other hand, the advantage of TSC-based synchronised inter-tick
> timestamps is that you can do things like
> 
>     sleep_until(rt_timer_ns2ticks(rtdm_clock_read() + 1000000))
> 
> without risking an error beyond +/- 1 tick (+jitter). With current
> jiffies vs. TSC in periodic mode, this is not easily possible. You have
> to sync in the application, creating another error source when the delay
> between acquiring the TSC and sync'ing the TSC on jiffies is too long.
> 

The proper way to solve this is rather to emulate the periodic mode over
the oneshot machinery, so that we stop having this +/- 1 tick error
margin. The periodic mode as it is now is purely a x86 legacy; even on
some ppc boards where the auto-reload feature is available from the
decrementer, Xeno doesn't use it.

The more I think of the x86 situation, the more I find it quite silly. I
mean, picking the periodic mode means that 1) all delays can be
expressed as multiples of a given constant interval, 2) the constant
interval must be large enough so that you don't put your board on its
knees, by processing useless ticks most of the time. What one saves here
- using periodic mode - is a couple of outb's per tick on the ISA bus,
since the PIT handles this automatically without software intervention
once set up properly. We already know that the programming overhead
(i.e. introduced by those outb's) is perfectly bearable even for high
frequency sampling like 10Khz loops in aperiodic mode. So why on earth
do we care about saving two outb's and get a lousy timing accuracy in
the same move, for constant interval delays which are necessarily going
to be much larger than those already supported by the aperiodic mode? Er...

This is a shift in the underlying logic of the periodic mode we are
discussing here actually. It used to be a mode where timing accuracy was
only approximate, mostly to deal with timeouts, in the watchdog sense.
Now, it is becoming a way to rely on a constant interval unit, while
still keeping a high timing accuracy. I'm ok with this, since we don't
rely on true PIT (except for x86, which is fixable) when running in
periodic mode, so I see no problem in raising the level of timing
accuracy of such mode. Existing stuff would not break because of such
change, but improve instead for people who care for exact durations in
periodic mode.

> 
>>And what would prevent us from improving the accuracy of other
>>
>>>timestamping API functions beyond RTDM as well, e.g. on converting from
>>>ticks to nanos in rt_timer_ticks2ns()?
>>>
>>
>>I don't understand why rt_timer_ticks2ns() should be impacted by such
>>extension. This service must keep a constant behaviour, regardless of
>>any outstanding timing issue. I mean, 3 ticks from a 1Khz clock rate
>>must always return 3,000,000 nanos, unless you stop passing count of
>>ticks but fractional/compound values instead.
> 
> 
> Forget about this, it was (pre-lunch) nonsense.
> 
> 
>>The bottom-line is that we should not blur the line between periodic and
>>aperiodic timing modes, just for getting precise timestamps in the
>>former case. Additionally, and x86-wise, when no TSC is available on the
>>target system, rt_timer_tsc() already returns a timestamp obtained from
>>the 8254's channel #2 we use as a free running counter, which is the
>>most precise source we have at hand to do so.
>>
>>Periodic mode bears its own limitation, which is basically a loss of
>>accuracy we trade against a lower overhead (even if that does not mean
>>much except perhaps on x86). What we could do is reducing the jittery
>>involved in periodic ticks, by always emulating periodic mode over
>>aperiodic shots instead of using e.g. the 8254 in PIT mode (and remove
>>the need for the double scale on x86, tsc + 8254 channel #1), but not
>>change the basic meaning of periodic timing.
> 
> 
> Hmm, interesting, and it also reminds of a long pending (slightly OT)
> question I have: why not creating the infrastructure (a dedicated
> periodic timer) for providing round-robin scheduling even in aperiodic mode?
> 

To do that, we would need to decouple the timing policy from any
particular scheduling policy. Therefore, we would need to remove the
round-robin management from the timer code, and move it to a particular
scheduling policy implementation. Which in turn would require to
implement a pluggable scheduler infrastructure, in the first place.

> 
>>But maybe we are still discussing different issues actually, so it would
>>be useful that the core issue that triggered the discussion about
>>periodic mode precision be exposed again.
> 
> 
> Yep, Rodrigo...?
> 
> Jan
> 


-- 

Philippe.

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2647 bytes --]

Philippe Gerum wrote:
> Jan Kiszka wrote:
>> ...
>> On the other hand, the advantage of TSC-based synchronised inter-tick
>> timestamps is that you can do things like
>>
>>     sleep_until(rt_timer_ns2ticks(rtdm_clock_read() + 1000000))
>>
>> without risking an error beyond +/- 1 tick (+jitter). With current
>> jiffies vs. TSC in periodic mode, this is not easily possible. You have
>> to sync in the application, creating another error source when the delay
>> between acquiring the TSC and sync'ing the TSC on jiffies is too long.
>>
> 
> The proper way to solve this is rather to emulate the periodic mode over
> the oneshot machinery, so that we stop having this +/- 1 tick error
> margin. The periodic mode as it is now is purely a x86 legacy; even on
> some ppc boards where the auto-reload feature is available from the
> decrementer, Xeno doesn't use it.
> 
> The more I think of the x86 situation, the more I find it quite silly. I
> mean, picking the periodic mode means that 1) all delays can be
> expressed as multiples of a given constant interval, 2) the constant
> interval must be large enough so that you don't put your board on its
> knees, by processing useless ticks most of the time. What one saves here
> - using periodic mode - is a couple of outb's per tick on the ISA bus,
> since the PIT handles this automatically without software intervention
> once set up properly. We already know that the programming overhead
> (i.e. introduced by those outb's) is perfectly bearable even for high
> frequency sampling like 10Khz loops in aperiodic mode. So why on earth
> do we care about saving two outb's and get a lousy timing accuracy in
> the same move, for constant interval delays which are necessarily going
> to be much larger than those already supported by the aperiodic mode? Er...
> 
> This is a shift in the underlying logic of the periodic mode we are
> discussing here actually. It used to be a mode where timing accuracy was
> only approximate, mostly to deal with timeouts, in the watchdog sense.
> Now, it is becoming a way to rely on a constant interval unit, while
> still keeping a high timing accuracy. I'm ok with this, since we don't
> rely on true PIT (except for x86, which is fixable) when running in
> periodic mode, so I see no problem in raising the level of timing
> accuracy of such mode. Existing stuff would not break because of such
> change, but improve instead for people who care for exact durations in
> periodic mode.

Yep, getting rid of as much periodic mode limitations as reasonable in a
transparent way sounds very good to me.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Philippe Gerum]

Rodrigo Rosenfeld Rosas wrote:
> Philippe Gerum wrote:
> 
>>...
>>Given the description above, just that some skin might return either
>>nucleus ticks or corrected timestamps to the applications, which would
>>in turn do some arithmetics for converting values they got from the skin
>>between both scales internally, and mistakenly use the result while
>>assuming that both scales are always in sync. In this situation, and
>>during a fraction of the time (i.e. the jitter), both scales might not
>>be in sync, and the result would be unusable.
> 
> 
> But I still can't find a real situation where the user would need these values 
> to be in sync...

It's not a matter of dealing with users always doing The Right Thing, 
but preferably preventing people from doing the wrong one.

-- 

Philippe.

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Rodrigo Rosenfeld Rosas]

Em Segunda 20 Mar=E7o 2006 12:23, Philippe Gerum escreveu:

>...
>It's not a matter of dealing with users always doing The Right Thing,
>but preferably preventing people from doing the wrong one.

But we then have two problems and there are tradeoffs here. In one hand we=
=20
want to avoid users from making mistakes. In the other hand we want to=20
provide a way for solving the issue that generated this thread.

A solution for the last one would be to have a function that would return a=
=20
high-precision timestamp, not necessarily in sync with xenomai's timer, sin=
ce=20
it would be used for relative time calculations, but should be in sync=20
between multiple CPUs. But this solution would rise the possibility for a=20
user to do a wrong thing. Of course, the functions should be well documente=
d=20
and states the lack of sync if it is the case. So, we have to choose betwee=
n=20
turning it possible to have such design (as the example I have last message=
)=20
or avoid people from doing the wrong thing. I would choose the first case,=
=20
since I think all rt-programmers are smarter (or should be) then the averag=
e=20
programmer. They must have attention when dealing with rt-programming. So,=
=20
reading a documentation and understanding it should not be a hard task for=
=20
them... In the other hand, if the second approach was chosen, a user wantin=
g=20
to use a rt-video interface will be forced to use the aperiodic timer for=20
having reliable timestamps...

Rodrigo.

		
_______________________________________________________
Yahoo! Acesso Grátis - Internet rápida e grátis. Instale o discador agora!
http://br.acesso.yahoo.com

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Philippe Gerum]

Rodrigo Rosenfeld Rosas wrote:

> I would choose the first case, 
> since I think all rt-programmers are smarter (or should be) then the average 
> programmer.

Uh? no, we aren't smarter. Being smarter would require us to stop 
spending countless hours working on ten lines of code, just for the 
purpose of saving a couple of microseconds on the so-called "fast 
path"... :o>

-- 

Philippe.

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Gilles Chanteperdrix]

Jan Kiszka wrote:
 > Likely too simple: The periodic IRQ seems to pop up on every CPU so that
 > the TSC could be recorded, but will this happen synchronously? At least
 > we will see (IRQ) jitters, and those jitters could already create in the
 > single-CPU case a non-monotonic clock...

I do not know how this issue is solved in Linux, but there seem to be a
simple solution: before adding the tsc offset to the last tick time,
this tsc offset should be compared with the tick duration in tsc
counts; if it is greater, then replace with the tick duration in tsc.

-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1092 bytes --]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>  > Likely too simple: The periodic IRQ seems to pop up on every CPU so that
>  > the TSC could be recorded, but will this happen synchronously? At least
>  > we will see (IRQ) jitters, and those jitters could already create in the
>  > single-CPU case a non-monotonic clock...
> 
> I do not know how this issue is solved in Linux, but there seem to be a
> simple solution: before adding the tsc offset to the last tick time,
> this tsc offset should be compared with the tick duration in tsc
> counts; if it is greater, then replace with the tick duration in tsc.
> 

Hmm, I would rather express it in absolute tsc values, i.e. always save
the tuple (absolute_tsc, jiffies):

[timer IRQ]
new_tsc = read_rsc() - old_tsc;
if (new_tsc < old_tsc + period_in_tsc_ticks)
	new_tsc = old_tsc + period_in_tsc;
old_tsc = new_tsc;

Disclaimer: I haven't thought about potential accuracy side effects of
this implementation, e.g. what would happen over the long term if the
condition is always fulfilled and executed...

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Gilles Chanteperdrix]

Jan Kiszka wrote:
 > Gilles Chanteperdrix wrote:
 > > Jan Kiszka wrote:
 > >  > Likely too simple: The periodic IRQ seems to pop up on every CPU so that
 > >  > the TSC could be recorded, but will this happen synchronously? At least
 > >  > we will see (IRQ) jitters, and those jitters could already create in the
 > >  > single-CPU case a non-monotonic clock...
 > > 
 > > I do not know how this issue is solved in Linux, but there seem to be a
 > > simple solution: before adding the tsc offset to the last tick time,
 > > this tsc offset should be compared with the tick duration in tsc
 > > counts; if it is greater, then replace with the tick duration in tsc.
 > > 
 > 
 > Hmm, I would rather express it in absolute tsc values, i.e. always save
 > the tuple (absolute_tsc, jiffies):
 > 
 > [timer IRQ]
 > new_tsc = read_rsc() - old_tsc;
 > if (new_tsc < old_tsc + period_in_tsc_ticks)
 > 	new_tsc = old_tsc + period_in_tsc;
 > old_tsc = new_tsc;
 > 
 > Disclaimer: I haven't thought about potential accuracy side effects of
 > this implementation, e.g. what would happen over the long term if the
 > condition is always fulfilled and executed...

Here is what I meant:

[timer IRQ]
irq_tsc = rdtsc();
irq_jitter_ns = read_8254()

[xnpod_gettime_offset]
offset_ns = tsc2ns(rdtsc() - irq_tsc()) + irq_jitter_ns
if (offset_ns > period_ns)
    offset_ns = period_ns; /* Avoid non monotonic clock. */

-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-core] Synchronising TSC and periodic timer

[Jan Kiszka]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>  > Gilles Chanteperdrix wrote:
>  > > Jan Kiszka wrote:
>  > >  > Likely too simple: The periodic IRQ seems to pop up on every CPU so that
>  > >  > the TSC could be recorded, but will this happen synchronously? At least
>  > >  > we will see (IRQ) jitters, and those jitters could already create in the
>  > >  > single-CPU case a non-monotonic clock...
>  > > 
>  > > I do not know how this issue is solved in Linux, but there seem to be a
>  > > simple solution: before adding the tsc offset to the last tick time,
>  > > this tsc offset should be compared with the tick duration in tsc
>  > > counts; if it is greater, then replace with the tick duration in tsc.
>  > > 
>  > 
>  > Hmm, I would rather express it in absolute tsc values, i.e. always save
>  > the tuple (absolute_tsc, jiffies):
>  > 
>  > [timer IRQ]
>  > new_tsc = read_rsc() - old_tsc;
>  > if (new_tsc < old_tsc + period_in_tsc_ticks)
>  > 	new_tsc = old_tsc + period_in_tsc;
>  > old_tsc = new_tsc;
>  > 
>  > Disclaimer: I haven't thought about potential accuracy side effects of
>  > this implementation, e.g. what would happen over the long term if the
>  > condition is always fulfilled and executed...
> 
> Here is what I meant:
> 
> [timer IRQ]
> irq_tsc = rdtsc();
> irq_jitter_ns = read_8254()
> 
> [xnpod_gettime_offset]
> offset_ns = tsc2ns(rdtsc() - irq_tsc()) + irq_jitter_ns
> if (offset_ns > period_ns)
>     offset_ns = period_ns; /* Avoid non monotonic clock. */
> 

Ah, I see. Hmm, wouldn't this create some "resolution hole" between the
time offset_ns exceeds a precise period and the time the next IRQ
actually strikes? The returned timestamps would then just stick to last
(irq_tsc + irq_jitter_ns) until the next update occurs.

Jan