Re: Minimum timing resolution in Ubuntu/Linaro on the PandaBoard ES

Re: Minimum timing resolution in Ubuntu/Linaro on the PandaBoard ES

[Dmitry Antipov]

On 02/07/2012 02:43 PM, Andrew Richardson wrote:

> I'm experiencing what appears to be a minimum clock resolution issue
> in using clock_gettime() on a PandaBoard ES running ubuntu.

Do you have CONFIG_OMAP_32K_TIMER enabled in your kernel?
Look at 'dmesg | grep clock' and check for the following:

...
OMAP clockevent source: GPTIMER1 at 32768 Hz
sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 131071999ms
...

Most probably this is the answer - by default, recent OMAPs are configured
to use less-accurate, but more energy-saving timer (32KHz) in favor of
MPU timer.

Disable CONFIG_OMAP_32K_TIMER to switch to MPU timer, and check
'dmesg | grep clock' to see:

...
OMAP clockevent source: GPTIMER1 at 38400000 Hz
OMAP clocksource: GPTIMER2 at 38400000 Hz
sched_clock: 32 bits at 38MHz, resolution 26ns, wraps every 111848ms
...

BTW, I have no ideas why clock_getres(CLOCK_REALTIME,...) returns {0, 1}
regardless of underlying clock source. I expect {0, 30517} for 32K timer
and {0, 26} for MPU timer.

Dmitry

Re: Minimum timing resolution in Ubuntu/Linaro on the PandaBoard ES

[Andrew Richardson]

[-- Attachment #1.1: Type: text/plain, Size: 2983 bytes --]

Ah, very interesting.

> dmesg | grep clock
[    0.000000] OMAP clockevent source: GPTIMER1 at 32768 Hz
[    0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 131071999ms
[    0.309448] omap_hwmod: l4_div_ck: missing clockdomain for l4_div_ck.
[    0.716979] Skipping twl internal clock init and using bootloader value (unknown osc rate)
[    1.001129] Switching to clocksource 32k_counter
[    6.907501] twl_rtc twl_rtc: setting system clock to 2000-01-01 00:00:01 UTC (946684801)

Do you recommend using "Get linaro image tools: method 2 (source code)" ( http://releases.linaro.org/12.01/ubuntu/leb-panda/ ) and building the kernel myself? I think we're, for the most part, unconcerned with power usage for our application (we're robotics researchers and, I believe, computation is only a small fraction of the power draw when compared to the motors).

It seems to me that we would want to disable some of the power-saving changes that have been made, such as this timer, and possibly configure other settings like cache behavior, though I have no idea how they're currently set. I have a bunch of docs from ARM on power and cache config, but I haven't messed around with them as I'm not sure where to start. My best guess is that I would have to rebuild the kernel to start handling that configuration myself. Is that true?

Some people ( http://groups.google.com/group/pandaboard/browse_thread/thread/a18fa3514d130720 ) have mentioned enabling line fill and prefetching to speed up memcpy operations, which also seems useful. Is this also a kernel-level setting?

If you think that's the right route, I would appreciate advice on where, within the build process, I need to start changing things to get the settings I want.

Many thanks,
Andrew



On Feb 8, 2012, at 12:21 AM, Dmitry Antipov wrote:

> On 02/07/2012 02:43 PM, Andrew Richardson wrote:
> 
>> I'm experiencing what appears to be a minimum clock resolution issue
>> in using clock_gettime() on a PandaBoard ES running ubuntu.
> 
> Do you have CONFIG_OMAP_32K_TIMER enabled in your kernel?
> Look at 'dmesg | grep clock' and check for the following:
> 
> ...
> OMAP clockevent source: GPTIMER1 at 32768 Hz
> sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 131071999ms
> ...
> 
> Most probably this is the answer - by default, recent OMAPs are configured
> to use less-accurate, but more energy-saving timer (32KHz) in favor of
> MPU timer.
> 
> Disable CONFIG_OMAP_32K_TIMER to switch to MPU timer, and check
> 'dmesg | grep clock' to see:
> 
> ...
> OMAP clockevent source: GPTIMER1 at 38400000 Hz
> OMAP clocksource: GPTIMER2 at 38400000 Hz
> sched_clock: 32 bits at 38MHz, resolution 26ns, wraps every 111848ms
> ...
> 
> BTW, I have no ideas why clock_getres(CLOCK_REALTIME,...) returns {0, 1}
> regardless of underlying clock source. I expect {0, 30517} for 32K timer
> and {0, 26} for MPU timer.
> 
> Dmitry
> 
> 


[-- Attachment #1.2: Type: text/html, Size: 3902 bytes --]

[-- Attachment #2: Type: text/plain, Size: 175 bytes --]

_______________________________________________
linaro-dev mailing list
linaro-dev-cunTk1MwBs8s++Sfvej+rw@public.gmane.org
http://lists.linaro.org/mailman/listinfo/linaro-dev

Re: Minimum timing resolution in Ubuntu/Linaro on the PandaBoard ES

[Dmitry Antipov]

On 02/08/2012 01:32 AM, Andrew Richardson wrote:

> Do you recommend using "Get linaro image tools: method 2 (source code)"
> ( http://releases.linaro.org/12.01/ubuntu/leb-panda/ ) and building the
> kernel myself?

Unfortunately this is the only way. In theory, there are clocksource=
boot option, and sysfs interface under /sys/devices/system/clocksource,
but, IIUC, there is no way to compile the kernel with both 32K and MPU
timers support and then select one of them for the default clock source
at the boot time or when the system is running.

> It seems to me that we would want to disable some of the power-saving
> changes that have been made, such as this timer, and possibly configure
> other settings like cache behavior, though I have no idea
> how they're currently set. I have a bunch of docs from ARM on power and
> cache config, but I haven't messed around with them as I'm not sure where
> to start. My best guess is that I would have to rebuild the kernel to
> start handling that configuration myself. Is that true?

If you're seriously concerned on the optimization for the particular
workload, you definitely should.

> Some people ( http://groups.google.com/group/pandaboard/browse_thread/thread/a18fa3514d130720 )
> have mentioned enabling line fill and prefetching to speed up memcpy
> operations, which also seems  useful. Is this also a kernel-level setting?

Sure. Caching (and it's relationship to real memory speed) is a hard topic.
For the starting point, try:

1. Read http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.faqs/ka13544.html
    (this is for A8, but should be more or less applicable to A9);

2. Run 'dmesg | grep -i cache' and check for the something similar to:

    L310 cache controller enabled
    l2x0: 16 ways, CACHE_ID 0x410000c4, AUX_CTRL 0x7e470000, Cache size: 1048576 B

3. Read http://infocenter.arm.com/help/topic/com.arm.doc.ddi0246f/DDI0246F_l2c310_r3p2_trm.pdf
    and realize the meaning of these AUX_CTRL bits;

4. Read arch/arm/mach-omap2/omap4-common.c and arch/arm/mm/cache-l2x0.c, try to play
    with 'aux_ctrl' bits within omap_l2_cache_init() and check whether it affects your
    workload. Note this may cause kernel crash and/or prevent the system from booting at all.

Dmitry

Re: Minimum timing resolution in Ubuntu/Linaro on the PandaBoard ES

[Andrew Richardson]

On Feb 8, 2012, at 8:55 AM, Dmitry Antipov wrote:

> On 02/08/2012 01:32 AM, Andrew Richardson wrote:
> 
>> Do you recommend using "Get linaro image tools: method 2 (source code)"
>> ( http://releases.linaro.org/12.01/ubuntu/leb-panda/ ) and building the
>> kernel myself?
> 
> Unfortunately this is the only way. In theory, there are clocksource=
> boot option, and sysfs interface under /sys/devices/system/clocksource,
> but, IIUC, there is no way to compile the kernel with both 32K and MPU
> timers support and then select one of them for the default clock source
> at the boot time or when the system is running.
> 
>> It seems to me that we would want to disable some of the power-saving
>> changes that have been made, such as this timer, and possibly configure
>> other settings like cache behavior, though I have no idea
>> how they're currently set. I have a bunch of docs from ARM on power and
>> cache config, but I haven't messed around with them as I'm not sure where
>> to start. My best guess is that I would have to rebuild the kernel to
>> start handling that configuration myself. Is that true?
> 
> If you're seriously concerned on the optimization for the particular
> workload, you definitely should.
> 
>> Some people ( http://groups.google.com/group/pandaboard/browse_thread/thread/a18fa3514d130720 )
>> have mentioned enabling line fill and prefetching to speed up memcpy
>> operations, which also seems  useful. Is this also a kernel-level setting?
> 
> Sure. Caching (and it's relationship to real memory speed) is a hard topic.
> For the starting point, try:
> 
> 1. Read http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.faqs/ka13544.html
>   (this is for A8, but should be more or less applicable to A9);
> 
> 2. Run 'dmesg | grep -i cache' and check for the something similar to:
> 
>   L310 cache controller enabled
>   l2x0: 16 ways, CACHE_ID 0x410000c4, AUX_CTRL 0x7e470000, Cache size: 1048576 B
> 
> 3. Read http://infocenter.arm.com/help/topic/com.arm.doc.ddi0246f/DDI0246F_l2c310_r3p2_trm.pdf
>   and realize the meaning of these AUX_CTRL bits;
> 
> 4. Read arch/arm/mach-omap2/omap4-common.c and arch/arm/mm/cache-l2x0.c, try to play
>   with 'aux_ctrl' bits within omap_l2_cache_init() and check whether it affects your
>   workload. Note this may cause kernel crash and/or prevent the system from booting at all.
> 
> Dmitry
> 
> 

Re: Minimum timing resolution in Ubuntu/Linaro on the PandaBoard ES

[John Stultz]

On Wed, 2012-02-08 at 04:32 -0500, Andrew Richardson wrote:
> Ah, very interesting.

>         > dmesg | grep clock
>         [    0.000000] OMAP clockevent source: GPTIMER1 at 32768 Hz
>         [    0.000000] sched_clock: 32 bits at 32kHz, resolution
>         30517ns, wraps every 131071999ms

Hrm. So 30us is still much smaller then the 2.5ms you were seeing. So
that doesn't fully explain the behavior.

thanks
-john

Re: Minimum timing resolution in Ubuntu/Linaro on the PandaBoard ES

[John Stultz]

On Wed, 2012-02-08 at 04:32 -0500, Andrew Richardson wrote:
> Ah, very interesting.

>         > dmesg | grep clock
>         [    0.000000] OMAP clockevent source: GPTIMER1 at 32768 Hz
>         [    0.000000] sched_clock: 32 bits at 32kHz, resolution
>         30517ns, wraps every 131071999ms

Hrm. So 30us is still much smaller then the 2.5ms you were seeing. So
that doesn't fully explain the behavior.

thanks
-john

RE: Minimum timing resolution in Ubuntu/Linaro on the PandaBoard ES

[Turgis, Frederic]

Hi,

Never had any issue with 32K and gettimeofday() on Panda (but just starting to use clock_gettime()). It was used to timestamp events happening every few ms or 100s of us.

I would advise as a check:
- read clock_gettime()/gettimeofday() and in parallel 32K register (map and read physical address 0x4A304010) to check behaviour.
- There is potential issue (that we have never seen) when reading 32K register. Worked around by calling clock_gettime()/gettimeofday() twice (we never do that and still it works so ...)

We have been doing tests in the past like while(1) {gettimeofday(); printf("time ...")} and it worked correctly, exhibiting the 30.5us accuracy

Regards
Fred



Texas Instruments France SA, 821 Avenue Jack Kilby, 06270 Villeneuve Loubet. 036 420 040 R.C.S Antibes. Capital de EUR 753.920

-----Original Message-----
From: linaro-dev-bounces@lists.linaro.org [mailto:linaro-dev-bounces@lists.linaro.org] On Behalf Of John Stultz
Sent: Wednesday, February 08, 2012 6:09 PM
To: Andrew Richardson
Cc: linux-omap@vger.kernel.org; linaro-dev@lists.linaro.org
Subject: Re: Minimum timing resolution in Ubuntu/Linaro on the PandaBoard ES

On Wed, 2012-02-08 at 04:32 -0500, Andrew Richardson wrote:
> Ah, very interesting.

>         > dmesg | grep clock
>         [    0.000000] OMAP clockevent source: GPTIMER1 at 32768 Hz
>         [    0.000000] sched_clock: 32 bits at 32kHz, resolution
>         30517ns, wraps every 131071999ms

Hrm. So 30us is still much smaller then the 2.5ms you were seeing. So
that doesn't fully explain the behavior.

thanks
-john

Re: Minimum timing resolution in Ubuntu/Linaro on the PandaBoard ES

[John Stultz]

On Tue, 2012-02-07 at 21:21 -0800, Dmitry Antipov wrote:
> BTW, I have no ideas why clock_getres(CLOCK_REALTIME,...) returns {0, 1}
> regardless of underlying clock source. I expect {0, 30517} for 32K timer
> and {0, 26} for MPU timer.

Yea. I had proposed to export the underlying clocksource's resolution
via clock_getres, but it was argued against. The concern is that
applications might not expect clock_getres to change while the
application is running.  Between any clock_getres() call and a time
read, the clocksources could change.

But if someone has a different reading of the posix spec, it might be
good to revisit this.

thanks
-john

Re: Minimum timing resolution in Ubuntu/Linaro on the PandaBoard ES

[Ming Lei]

Hi,

On Wed, Feb 8, 2012 at 1:21 PM, Dmitry Antipov
<dmitry.antipov@linaro.org> wrote:
> On 02/07/2012 02:43 PM, Andrew Richardson wrote:
>
>> I'm experiencing what appears to be a minimum clock resolution issue
>> in using clock_gettime() on a PandaBoard ES running ubuntu.
>
>
> Do you have CONFIG_OMAP_32K_TIMER enabled in your kernel?
> Look at 'dmesg | grep clock' and check for the following:
>
> ...
> OMAP clockevent source: GPTIMER1 at 32768 Hz
> sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 131071999ms
> ...
>
> Most probably this is the answer - by default, recent OMAPs are configured
> to use less-accurate, but more energy-saving timer (32KHz) in favor of
> MPU timer.

Sorry, I have a question about the two kind of timers. No matter
CONFIG_OMAP_32K_TIMER is defined or not, 'twd' interrupt count
is always increased in '/proc/interrupts', and 'gp timer' interrupt count
keeps unchanged, so looks MPU timer is still enabled even
CONFIG_OMAP_32K_TIMER is disabled, isn't it?

After some investigation, I found the change[1] can remove local
timer of 'twd', and tick will be driven by 'gp timer' interrupt, but I am
not sure if it is the right thing to do.

>
> Disable CONFIG_OMAP_32K_TIMER to switch to MPU timer, and check
> 'dmesg | grep clock' to see:
>
> ...
> OMAP clockevent source: GPTIMER1 at 38400000 Hz
> OMAP clocksource: GPTIMER2 at 38400000 Hz
> sched_clock: 32 bits at 38MHz, resolution 26ns, wraps every 111848ms
> ...
>
> BTW, I have no ideas why clock_getres(CLOCK_REALTIME,...) returns {0, 1}
> regardless of underlying clock source. I expect {0, 30517} for 32K timer
> and {0, 26} for MPU timer.
>

[1], 'not select LOCAL_TIMERS for OMAP4 SMP'
diff --git a/arch/arm/mach-omap2/Kconfig b/arch/arm/mach-omap2/Kconfig
index d965da4..0036218 100644
--- a/arch/arm/mach-omap2/Kconfig
+++ b/arch/arm/mach-omap2/Kconfig
@@ -46,7 +46,7 @@ config ARCH_OMAP4
        select CPU_V7
        select ARM_GIC
        select HAVE_SMP
-       select LOCAL_TIMERS if SMP
+       #select LOCAL_TIMERS if SMP
        select PL310_ERRATA_588369
        select PL310_ERRATA_727915
        select ARM_ERRATA_720789


thanks,
-- 
Ming Lei