Re: High resolution timers on AT91SAM926x

Re: High resolution timers on AT91SAM926x

[Bosko Radivojevic]

David,

thank you for the reply. But, I'm getting pretty strange behavior.
I've applied all your patches to 2.6.23.11-rt14 (with official at91
patch for 2.6.23) kernel without any major rejects. Here is a snippet
from the .config:

CONFIG_GENERIC_GPIO=y
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_RWSEM_GENERIC_SPINLOCK=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_AT91_TIMER_HZ=1000
# CONFIG_NO_HZ is not set
CONFIG_HZ=1000

===========================================

$ cat /proc/timer_list
Timer List Version: v0.3
HRTIMER_MAX_CLOCK_BASES: 2
now at 3782337012585 nsecs

cpu: 0
 clock 0:
  .index:      0
  .resolution: 999961 nsecs
  .get_time:   ktime_get_real
  .offset:     0 nsecs
active timers:
 clock 1:
  .index:      1
  .resolution: 999961 nsecs
  .get_time:   ktime_get
  .offset:     0 nsecs
active timers:
 #0: <c3b2de90>, it_real_fn, S:01
 # expires at 4807555046495 nsecs [in 1025218033910 nsecs]
  .expires_next   : 2147483646999999999 nsecs
  .hres_active    : 0
  .nr_events      : 0
  .nohz_mode      : 0
  .idle_tick      : 0 nsecs
  .tick_stopped   : 0
  .idle_jiffies   : 0
  .idle_calls     : 0
  .idle_sleeps    : 0
  .idle_entrytime : 0 nsecs
  .idle_sleeptime : 0 nsecs
  .last_jiffies   : 0
  .next_jiffies   : 0
  .idle_expires   : 0 nsecs
jiffies: 3482489


Tick Device: mode:     0
Clock Event Device: pit
 max_delta_ns:   0
 min_delta_ns:   0
 mult:           26663156
 shift:          32
 mode:           2
 next_event:     0 nsecs
 set_next_event: __init_begin
 set_mode:       pit_clkevt_mode
 event_handler:  tick_handle_periodic

Also, I wrote a small example to test the behaviour (sleeping 1ms):

#define period 500000
#define onesecond 1000000000

 mlockall();
 sp.sched_priority = 90;
 printf ("setscheduler = %d\n", sched_setscheduler (0, SCHED_FIFO, &sp));
 printf ("setparam = %d\n", sched_setparam (0, &sp));

 while (++loop < 30000) {
   clock_gettime (CLOCK_MONOTONIC, &ts);
   if (ts.tv_nsec + period > onesecond) {
     ts2.tv_sec = ts.tv_sec + 1;
     ts2.tv_nsec = (ts.tv_nsec + period) - onesecond;
   } else {
     ts2.tv_sec = ts.tv_sec;
     ts2.tv_nsec = ts.tv_nsec + period;
   }
   clock_nanosleep (CLOCK_MONOTONIC, TIMER_ABSTIME, &ts2, 0);
   clock_gettime (CLOCK_MONOTONIC, &ts3);
   diff = ((ts3.tv_sec - ts.tv_sec)*onesecond+ts3.tv_nsec) - ts.tv_nsec;
   if (!min && !max)
     min = max = diff;
   if (diff > max)
     max = diff;
   if (diff < min)
     min = diff;
   printf ("%ld %ld\n", min, max);
 }

I'm getting this as a result:
1890872 2924656

So, the minimal difference is almost a 2ms, maximum 3ms. I ran this
test for a long period, without any load on the system (except
printing out results through network). Alsto, I'm wondering why
minimal latency is 1890872, it is less then twice 999961 (timer
resolution).

What I am doing wrong? Is there a way to sleep 1ms? I tried to adjust
(/3, /5, etc) pit_cycle in pit_init but all I got was to lock the
system ;)

Thanks!

On Sat, Mar 1, 2008 at 11:57 PM, David Brownell <david-b@pacbell.net> wrote:
> > Is there a way to enable high resolution timers on AT91SAM926x?
>
>  Update PIT to support the clocksource/clockevent framework:
>   http://marc.info/?l=linux-arm-kernel&m=119940724711435&w=2
>
>  Declare timer/counter block platform devices:
>   http://marc.info/?l=linux-arm-kernel&m=120302318811110&w=2
>
>  Use timer/counter blocks for better clocksource and clockevents:
>   http://marc.info/?l=linux-kernel&m=120373043520279&w=2
>   http://marc.info/?l=linux-kernel&m=120373063320487&w=2
>
>  The focus in that last patch is on NO_HZ support -- so the
>  clockevents run at 32 KiHz (about 31 usec precision for HRT)
>  to allow overall HZ to run below 1 where practical.   If you
>  need even higher precision, you could update that clockevent
>  code to use a different base clock.
>
>  Those last two patches are in some MM tree, and Haavard has
>  some updates to then (which don't much affect functionality).
>
>  I understand the upcoming 2.6.24-at91 patch will include the
>  first two patches.
>
>  - Dave
>

Re: High resolution timers on AT91SAM926x

[David Brownell]

On Monday 03 March 2008, Bosko Radivojevic wrote:
> thank you for the reply. But, I'm getting pretty strange behavior.

Presumably you got different results when your Kconfig
enabled a HRT/NO_HZ capable clockevent source, instead
of just the PIT?

CONFIG_ATMEL_TCLIB=y
CONFIG_ATMEL_TCB_CLKSRC=y
CONFIG_ATMEL_TCB_CLKSRC_BLOCK=0

Re: High resolution timers on AT91SAM926x

[Bosko Radivojevic]

David,

I had CONFIG_ATMEL_TCLIB enabled, but not TCB_CLKSRC and
TCB_CLKSRC_BLOCK=0. With all those options, I finally have HRT
functionality. But, strange thing is that jitter of my little example
(get time, sleep 1ms, get time, show the difference) is around 250us.
Maybe this is normal for this architecture?

System is (as noted on rt.wik site) very slow with NO_HZ option enabled.

Thanks again!

On Mon, Mar 3, 2008 at 7:39 PM, David Brownell <david-b@pacbell.net> wrote:
> On Monday 03 March 2008, Bosko Radivojevic wrote:
>  > thank you for the reply. But, I'm getting pretty strange behavior.
>
>  Presumably you got different results when your Kconfig
>  enabled a HRT/NO_HZ capable clockevent source, instead
>  of just the PIT?
>
>  CONFIG_ATMEL_TCLIB=y
>
>
> CONFIG_ATMEL_TCB_CLKSRC=y
>  CONFIG_ATMEL_TCB_CLKSRC_BLOCK=0
>

Re: High resolution timers on AT91SAM926x

[David Brownell]

On Monday 03 March 2008, Bosko Radivojevic wrote:
> I had CONFIG_ATMEL_TCLIB enabled, but not TCB_CLKSRC and
> TCB_CLKSRC_BLOCK=0. With all those options, I finally have HRT
> functionality. But, strange thing is that jitter of my little example
> (get time, sleep 1ms, get time, show the difference) is around 250us.
> Maybe this is normal for this architecture?

I have no idea why that would be.  Maybe you can find out.  :)


> System is (as noted on rt.wik site) very slow with NO_HZ option enabled.

It doesn't exactly say *why* or compare to other arm926ejs
chips ... 

Could the 64-bit math (ktime stuff) be a factor there?  Your
board probably only runs at 95 (or-so) bogomips after all.

And is that just a NO_HZ issue, or generically an issue when
oneshot timer modes are in heavy use?

- Dave
 

Re: High resolution timers on AT91SAM926x

[Remy Bohmer]

Hello Bosko/Dave,

> On Monday 03 March 2008, Bosko Radivojevic wrote:
>
> > I had CONFIG_ATMEL_TCLIB enabled, but not TCB_CLKSRC and
>  > TCB_CLKSRC_BLOCK=0. With all those options, I finally have HRT
>  > functionality. But, strange thing is that jitter of my little example
>  > (get time, sleep 1ms, get time, show the difference) is around 250us.
>  > Maybe this is normal for this architecture?
> I have no idea why that would be.  Maybe you can find out.  :)

These are normal figures for this core, on preempt-rt.
You are talking about jitter on timers. While on preempt-rt the worst
case latency of scheduling a RT-thread is about 300us, 250us is thus
quite normal, and actually quite good... (the kernel-mainline average
latency is better, but worst-case is unbound)
(Note: The 300 us seem to be caused by something in the networking
layer, without network I noticed worst case latencies about 150us, but
NO guarantee here)

But, besides the worst-case schedule latency; the interrupt handler is
shared with the handler of the DBGU. There are patches available to
split the interrupt handler, this will make this behavior somewhat
better, but do not expect miracles here.

I stopped using HRT on these cores and preempt-RT for quite some time
now (several months), because the smaller timer granularity will allow
applications to make timers elapse on sub-millisec intervals, where
each interval generates a interrupt handling cycle of about 50 -
>100us, especially if different applications use periodic timers which
are out of sync.
This will become visible as extreme CPU-load figures on applications
with 1 ms timers.
13-15% CPU load on a application with just 1 single 1 ms timer is
quite normal...

FWIW: For getting the most accurate realtime response for periodic
timers, I used a TC block to generate a periodic interrupt, that only
needs to wake up a RT-thread. This way the OS-timer framework can be
used for non-RT stuff, and/or slow timers. This is much less heavy on
this core, and gives much better/deterministic RT results.
(This is not what I prefer to do, but these cores are just not that
fast as multi-gigahertz X86 PCs ;-)) )


Kind Regards,

Remy

>
>
>
>  > System is (as noted on rt.wik site) very slow with NO_HZ option enabled.
>
>
> It doesn't exactly say *why* or compare to other arm926ejs
>  chips ...
>
>  Could the 64-bit math (ktime stuff) be a factor there?  Your
>  board probably only runs at 95 (or-so) bogomips after all.
>
>  And is that just a NO_HZ issue, or generically an issue when
>  oneshot timer modes are in heavy use?
>
>  - Dave
>
>
>  --
>  To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>
> the body of a message to majordomo@vger.kernel.org
>  More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: High resolution timers on AT91SAM926x

[Wolfgang Grandegger]

Remy Bohmer wrote:
> Hello Bosko/Dave,
> 
>> On Monday 03 March 2008, Bosko Radivojevic wrote:
>>
>>> I had CONFIG_ATMEL_TCLIB enabled, but not TCB_CLKSRC and
>>  > TCB_CLKSRC_BLOCK=0. With all those options, I finally have HRT
>>  > functionality. But, strange thing is that jitter of my little example
>>  > (get time, sleep 1ms, get time, show the difference) is around 250us.
>>  > Maybe this is normal for this architecture?
>> I have no idea why that would be.  Maybe you can find out.  :)
> 
> These are normal figures for this core, on preempt-rt.
> You are talking about jitter on timers. While on preempt-rt the worst
> case latency of scheduling a RT-thread is about 300us, 250us is thus
> quite normal, and actually quite good... (the kernel-mainline average
> latency is better, but worst-case is unbound)
> (Note: The 300 us seem to be caused by something in the networking
> layer, without network I noticed worst case latencies about 150us, but
> NO guarantee here)

This confirms our observation with PowerPC MPC5200 boards (running at
400 MHz). Without networking, the worst case latency measured with
cyclictest does not exceed 100 us. With networking, and especially with
the NFS file-system, latencies go up to 170 us. I also measured external
interrupt latencies with an external log system using GPIO pins but was
unable to achieve latencies below 1ms. This needs more debugging.

Wolfgang.

Re: High resolution timers on AT91SAM926x

[Bosko Radivojevic]

Hello Remy,

I tried today Xenomai (2.6.20.21 kernel, that is the last supported by
I-Pipe patch), the same example program (recompiled with Xenomai
libpthread_rt). Results are a little bit better (worst-case latency is
around 150us). Using Xenomai in native skin (as kernel module)
produces much better results (worst-case latency is around 50us).
Currently, I am looking for the best way to "transfer" application to
user land/posix.

>  FWIW: For getting the most accurate realtime response for periodic
>  timers, I used a TC block to generate a periodic interrupt, that only
>  needs to wake up a RT-thread. This way the OS-timer framework can be
>  used for non-RT stuff, and/or slow timers. This is much less heavy on
>  this core, and gives much better/deterministic RT results.

May you give me a bit more details on this? Thanks :)

Cheers!

Re: High resolution timers on AT91SAM926x

[Daniel Walker]

On Tue, 2008-03-04 at 23:38 +0100, Bosko Radivojevic wrote:
> Hello Remy,
> 
> I tried today Xenomai (2.6.20.21 kernel, that is the last supported by
> I-Pipe patch), the same example program (recompiled with Xenomai
> libpthread_rt). Results are a little bit better (worst-case latency is
> around 150us). Using Xenomai in native skin (as kernel module)
> produces much better results (worst-case latency is around 50us).
> Currently, I am looking for the best way to "transfer" application to
> user land/posix.

Maybe you could attach your test application? and specifics on how your
running it?

Daniel