[Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Hongfei Cheng]

We've been working on porting the I-pipe core to an ARMv8/AArch64
platform (Qualcomm msm8994).

Taking the advice we received from this forum, our approach has been
to add and enable the I-pipe core patches on this new architecture one
(small) step at a time.

The development platform is currently configured and patched as follows:
1). Only 1 CPU core is enabled (via kernel command line).
2). Interrupt controller, GPIO controller, and platform drivers are
patched following the porting guide and the existing ipipe-core code,
along with the relevant discussions on this list.
3). I-pipe spinlocks are applied to the instances where we think
require "ipipe protection", although we are still uncertain if this
work is complete in our code.
4). High resolution counter is not yet supported as we're exploring
different way(s) to implement TSC.
5). CPU_FREQ and CPU_IDLE are *not yet* disabled, mainly due to the
dependency of big.LITTLE architecture vendor-specific code.

Below is what we've observed so far:
1). Hardware timer appears to be running properly with
arch_timer_ack() being called regularly and reliably.
2). I-pipe is indeed grabbing the interrupt (through examining the call stack).
3). The kernel startup process comes to a screeching halt when probing
certain platform drivers, such as ipa or pcie. No kernel oops is
shown. In this kernel state, the *only* task being scheduled is the
watchdog process.

I believe there are "holes", to borrow the term from Gilles, in the
platform drivers. However the few platform drivers which call
generic_handle_irq have all been replaced with
ipipe_handle_demuxed_irq.

My questions are:
1). Has anyone encountered similar experience while porting I-pipe core?
2). Could such behavior be the result of missing or incorrectly
applied I-pipe spinlocks?

I'll put the kernel config, patched code, and startup logs in the
cloud and post the link here.

Thanks!
Hongfei

P.S. Apologies for my top-postings in the past few weeks which I just
noticed (in Gmail client).

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Gilles Chanteperdrix]

On Thu, Apr 30, 2015 at 08:53:10AM -0700, Hongfei Cheng wrote:
> We've been working on porting the I-pipe core to an ARMv8/AArch64
> platform (Qualcomm msm8994).
> 
> Taking the advice we received from this forum, our approach has been
> to add and enable the I-pipe core patches on this new architecture one
> (small) step at a time.
> 
> The development platform is currently configured and patched as follows:
> 1). Only 1 CPU core is enabled (via kernel command line).
> 2). Interrupt controller, GPIO controller, and platform drivers are
> patched following the porting guide and the existing ipipe-core code,
> along with the relevant discussions on this list.
> 3). I-pipe spinlocks are applied to the instances where we think
> require "ipipe protection", although we are still uncertain if this
> work is complete in our code.

Each spinlock you turn into an ipipe spinlock bears a risk of creating
large masking sections, so, this is something you should not do a
lot. The porting guide warns about that. This is something
important. Do only this when it is really needed, and not before
having examined every section where this spinlock is used by Linux
to see if it has no chance of creating a large masking section. If
it does, you will have to find other workarounds.

> 4). High resolution counter is not yet supported as we're exploring
> different way(s) to implement TSC.
> 5). CPU_FREQ and CPU_IDLE are *not yet* disabled, mainly due to the
> dependency of big.LITTLE architecture vendor-specific code.

This is bad. Probably a source of troubles. Disable them.

> 
> Below is what we've observed so far:
> 1). Hardware timer appears to be running properly with
> arch_timer_ack() being called regularly and reliably.
> 2). I-pipe is indeed grabbing the interrupt (through examining the call stack).
> 3). The kernel startup process comes to a screeching halt when probing
> certain platform drivers, such as ipa or pcie. No kernel oops is
> shown. In this kernel state, the *only* task being scheduled is the
> watchdog process.
> 
> I believe there are "holes", to borrow the term from Gilles, in the
> platform drivers. However the few platform drivers which call
> generic_handle_irq have all been replaced with
> ipipe_handle_demuxed_irq.

This looks wrong. Only when the irq is demuxed should you use
ipipe_handle_demuxed_irq. If the parent irq is a plain Linux irq,
there is no reason to replace generic_handle_irq with
ipipe_handle_demuxed_irq. In the GPIO case, the demux function is
run as an interrupt ack callback, so it runs "ahead of the
pipeline", and is not executed in Linux domain so can not use
generic_handle_irq.

> 
> My questions are:
> 1). Has anyone encountered similar experience while porting I-pipe
> core?

Porting the I-pipe code to a new ARM board has always been full of
surprises for me, it has always required to understand some new
problems, and avoid them. I have always found the I-pipe tracer to
be a great help in that case. Of course, enabling the I-pipe debug
options is a must, also (though it can create double faults
conditions when something goes really wrong).


> 2). Could such behavior be the result of missing or incorrectly
> applied I-pipe spinlocks?

No

> 
> I'll put the kernel config, patched code, and startup logs in the
> cloud and post the link here.

I do not think it is necessary.

-- 
					    Gilles.

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Hongfei Cheng]

On Thu, Apr 30, 2015 at 9:17 AM, Gilles Chanteperdrix
<gilles.chanteperdrix@xenomai.org> wrote:
> On Thu, Apr 30, 2015 at 08:53:10AM -0700, Hongfei Cheng wrote:
>> We've been working on porting the I-pipe core to an ARMv8/AArch64
>> platform (Qualcomm msm8994).
>>
>> Taking the advice we received from this forum, our approach has been
>> to add and enable the I-pipe core patches on this new architecture one
>> (small) step at a time.
>>
>> The development platform is currently configured and patched as follows:
>> 1). Only 1 CPU core is enabled (via kernel command line).
>> 2). Interrupt controller, GPIO controller, and platform drivers are
>> patched following the porting guide and the existing ipipe-core code,
>> along with the relevant discussions on this list.
>> 3). I-pipe spinlocks are applied to the instances where we think
>> require "ipipe protection", although we are still uncertain if this
>> work is complete in our code.
>
> Each spinlock you turn into an ipipe spinlock bears a risk of creating
> large masking sections, so, this is something you should not do a
> lot. The porting guide warns about that. This is something
> important. Do only this when it is really needed, and not before
> having examined every section where this spinlock is used by Linux
> to see if it has no chance of creating a large masking section. If
> it does, you will have to find other workarounds.
>
>> 4). High resolution counter is not yet supported as we're exploring
>> different way(s) to implement TSC.
>> 5). CPU_FREQ and CPU_IDLE are *not yet* disabled, mainly due to the
>> dependency of big.LITTLE architecture vendor-specific code.
>
> This is bad. Probably a source of troubles. Disable them.
>
>>
>> Below is what we've observed so far:
>> 1). Hardware timer appears to be running properly with
>> arch_timer_ack() being called regularly and reliably.
>> 2). I-pipe is indeed grabbing the interrupt (through examining the call stack).
>> 3). The kernel startup process comes to a screeching halt when probing
>> certain platform drivers, such as ipa or pcie. No kernel oops is
>> shown. In this kernel state, the *only* task being scheduled is the
>> watchdog process.
>>
>> I believe there are "holes", to borrow the term from Gilles, in the
>> platform drivers. However the few platform drivers which call
>> generic_handle_irq have all been replaced with
>> ipipe_handle_demuxed_irq.
>
> This looks wrong. Only when the irq is demuxed should you use
> ipipe_handle_demuxed_irq. If the parent irq is a plain Linux irq,
> there is no reason to replace generic_handle_irq with
> ipipe_handle_demuxed_irq. In the GPIO case, the demux function is
> run as an interrupt ack callback, so it runs "ahead of the
> pipeline", and is not executed in Linux domain so can not use
> generic_handle_irq.
>
>>
>> My questions are:
>> 1). Has anyone encountered similar experience while porting I-pipe
>> core?
>
> Porting the I-pipe code to a new ARM board has always been full of
> surprises for me, it has always required to understand some new
> problems, and avoid them. I have always found the I-pipe tracer to
> be a great help in that case. Of course, enabling the I-pipe debug
> options is a must, also (though it can create double faults
> conditions when something goes really wrong).
>
>
>> 2). Could such behavior be the result of missing or incorrectly
>> applied I-pipe spinlocks?
>
> No
>
>>
>> I'll put the kernel config, patched code, and startup logs in the
>> cloud and post the link here.
>
> I do not think it is necessary.
>
> --
>                                             Gilles.

Thank you, Gilles, for your always prompt and insightful response.
We'll re-examine the problematic places in the code, based on your
input and comments.

By the way, Gilles, have you been able to squeeze supporting
I-pipe/Xenomai on ARMv8/HiKey-board into your busy schedule?

Hongfei

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Gilles Chanteperdrix]

On Thu, Apr 30, 2015 at 11:12:58AM -0700, Hongfei Cheng wrote:
> On Thu, Apr 30, 2015 at 9:17 AM, Gilles Chanteperdrix
> <gilles.chanteperdrix@xenomai.org> wrote:
> > On Thu, Apr 30, 2015 at 08:53:10AM -0700, Hongfei Cheng wrote:
> >> We've been working on porting the I-pipe core to an ARMv8/AArch64
> >> platform (Qualcomm msm8994).
> >>
> >> Taking the advice we received from this forum, our approach has been
> >> to add and enable the I-pipe core patches on this new architecture one
> >> (small) step at a time.
> >>
> >> The development platform is currently configured and patched as follows:
> >> 1). Only 1 CPU core is enabled (via kernel command line).
> >> 2). Interrupt controller, GPIO controller, and platform drivers are
> >> patched following the porting guide and the existing ipipe-core code,
> >> along with the relevant discussions on this list.
> >> 3). I-pipe spinlocks are applied to the instances where we think
> >> require "ipipe protection", although we are still uncertain if this
> >> work is complete in our code.
> >
> > Each spinlock you turn into an ipipe spinlock bears a risk of creating
> > large masking sections, so, this is something you should not do a
> > lot. The porting guide warns about that. This is something
> > important. Do only this when it is really needed, and not before
> > having examined every section where this spinlock is used by Linux
> > to see if it has no chance of creating a large masking section. If
> > it does, you will have to find other workarounds.
> >
> >> 4). High resolution counter is not yet supported as we're exploring
> >> different way(s) to implement TSC.
> >> 5). CPU_FREQ and CPU_IDLE are *not yet* disabled, mainly due to the
> >> dependency of big.LITTLE architecture vendor-specific code.
> >
> > This is bad. Probably a source of troubles. Disable them.
> >
> >>
> >> Below is what we've observed so far:
> >> 1). Hardware timer appears to be running properly with
> >> arch_timer_ack() being called regularly and reliably.
> >> 2). I-pipe is indeed grabbing the interrupt (through examining the call stack).
> >> 3). The kernel startup process comes to a screeching halt when probing
> >> certain platform drivers, such as ipa or pcie. No kernel oops is
> >> shown. In this kernel state, the *only* task being scheduled is the
> >> watchdog process.
> >>
> >> I believe there are "holes", to borrow the term from Gilles, in the
> >> platform drivers. However the few platform drivers which call
> >> generic_handle_irq have all been replaced with
> >> ipipe_handle_demuxed_irq.
> >
> > This looks wrong. Only when the irq is demuxed should you use
> > ipipe_handle_demuxed_irq. If the parent irq is a plain Linux irq,
> > there is no reason to replace generic_handle_irq with
> > ipipe_handle_demuxed_irq. In the GPIO case, the demux function is
> > run as an interrupt ack callback, so it runs "ahead of the
> > pipeline", and is not executed in Linux domain so can not use
> > generic_handle_irq.
> >
> >>
> >> My questions are:
> >> 1). Has anyone encountered similar experience while porting I-pipe
> >> core?
> >
> > Porting the I-pipe code to a new ARM board has always been full of
> > surprises for me, it has always required to understand some new
> > problems, and avoid them. I have always found the I-pipe tracer to
> > be a great help in that case. Of course, enabling the I-pipe debug
> > options is a must, also (though it can create double faults
> > conditions when something goes really wrong).
> >
> >
> >> 2). Could such behavior be the result of missing or incorrectly
> >> applied I-pipe spinlocks?
> >
> > No
> >
> >>
> >> I'll put the kernel config, patched code, and startup logs in the
> >> cloud and post the link here.
> >
> > I do not think it is necessary.
> >
> > --
> >                                             Gilles.
> 
> Thank you, Gilles, for your always prompt and insightful response.
> We'll re-examine the problematic places in the code, based on your
> input and comments.
> 
> By the way, Gilles, have you been able to squeeze supporting
> I-pipe/Xenomai on ARMv8/HiKey-board into your busy schedule?

Not yet, but this should not be long now.

-- 
					    Gilles.

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Hongfei Cheng]

On Thu, Apr 30, 2015 at 11:14 AM, Gilles Chanteperdrix
<gilles.chanteperdrix@xenomai.org> wrote:
> On Thu, Apr 30, 2015 at 11:12:58AM -0700, Hongfei Cheng wrote:
>> On Thu, Apr 30, 2015 at 9:17 AM, Gilles Chanteperdrix
>> <gilles.chanteperdrix@xenomai.org> wrote:
>> > On Thu, Apr 30, 2015 at 08:53:10AM -0700, Hongfei Cheng wrote:
>> >> We've been working on porting the I-pipe core to an ARMv8/AArch64
>> >> platform (Qualcomm msm8994).
>> >>
>> >> Taking the advice we received from this forum, our approach has been
>> >> to add and enable the I-pipe core patches on this new architecture one
>> >> (small) step at a time.
>> >>
>> >> The development platform is currently configured and patched as follows:
>> >> 1). Only 1 CPU core is enabled (via kernel command line).
>> >> 2). Interrupt controller, GPIO controller, and platform drivers are
>> >> patched following the porting guide and the existing ipipe-core code,
>> >> along with the relevant discussions on this list.
>> >> 3). I-pipe spinlocks are applied to the instances where we think
>> >> require "ipipe protection", although we are still uncertain if this
>> >> work is complete in our code.
>> >
>> > Each spinlock you turn into an ipipe spinlock bears a risk of creating
>> > large masking sections, so, this is something you should not do a
>> > lot. The porting guide warns about that. This is something
>> > important. Do only this when it is really needed, and not before
>> > having examined every section where this spinlock is used by Linux
>> > to see if it has no chance of creating a large masking section. If
>> > it does, you will have to find other workarounds.
>> >
>> >> 4). High resolution counter is not yet supported as we're exploring
>> >> different way(s) to implement TSC.
>> >> 5). CPU_FREQ and CPU_IDLE are *not yet* disabled, mainly due to the
>> >> dependency of big.LITTLE architecture vendor-specific code.
>> >
>> > This is bad. Probably a source of troubles. Disable them.
>> >
>> >>
>> >> Below is what we've observed so far:
>> >> 1). Hardware timer appears to be running properly with
>> >> arch_timer_ack() being called regularly and reliably.
>> >> 2). I-pipe is indeed grabbing the interrupt (through examining the call stack).
>> >> 3). The kernel startup process comes to a screeching halt when probing
>> >> certain platform drivers, such as ipa or pcie. No kernel oops is
>> >> shown. In this kernel state, the *only* task being scheduled is the
>> >> watchdog process.
>> >>
>> >> I believe there are "holes", to borrow the term from Gilles, in the
>> >> platform drivers. However the few platform drivers which call
>> >> generic_handle_irq have all been replaced with
>> >> ipipe_handle_demuxed_irq.
>> >
>> > This looks wrong. Only when the irq is demuxed should you use
>> > ipipe_handle_demuxed_irq. If the parent irq is a plain Linux irq,
>> > there is no reason to replace generic_handle_irq with
>> > ipipe_handle_demuxed_irq. In the GPIO case, the demux function is
>> > run as an interrupt ack callback, so it runs "ahead of the
>> > pipeline", and is not executed in Linux domain so can not use
>> > generic_handle_irq.
>> >
>> >>
>> >> My questions are:
>> >> 1). Has anyone encountered similar experience while porting I-pipe
>> >> core?
>> >
>> > Porting the I-pipe code to a new ARM board has always been full of
>> > surprises for me, it has always required to understand some new
>> > problems, and avoid them. I have always found the I-pipe tracer to
>> > be a great help in that case. Of course, enabling the I-pipe debug
>> > options is a must, also (though it can create double faults
>> > conditions when something goes really wrong).
>> >
>> >
>> >> 2). Could such behavior be the result of missing or incorrectly
>> >> applied I-pipe spinlocks?
>> >
>> > No
>> >
>> >>
>> >> I'll put the kernel config, patched code, and startup logs in the
>> >> cloud and post the link here.
>> >
>> > I do not think it is necessary.
>> >
>> > --
>> >                                             Gilles.
>>
>> Thank you, Gilles, for your always prompt and insightful response.
>> We'll re-examine the problematic places in the code, based on your
>> input and comments.
>>
>> By the way, Gilles, have you been able to squeeze supporting
>> I-pipe/Xenomai on ARMv8/HiKey-board into your busy schedule?
>
> Not yet, but this should not be long now.
>
> --
>                                             Gilles.

That's great to know!

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Gilles Chanteperdrix]

On Thu, Apr 30, 2015 at 08:53:10AM -0700, Hongfei Cheng wrote:
> 4). High resolution counter is not yet supported as we're exploring
> different way(s) to implement TSC.

Note the implications here: if you say "all armv8 will use
architected timer", we can get away with a unique implementation. If
you want to use another timer, then we will have to implement kernel
helpers with a constant kernel/user ABI, so probably in the vdso.
All this for an improved accuracy that may well not be worth the
trouble.

-- 
					    Gilles.

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Hongfei Cheng]

On Thu, Apr 30, 2015 at 11:31 AM, Gilles Chanteperdrix
<gilles.chanteperdrix@xenomai.org> wrote:
> On Thu, Apr 30, 2015 at 08:53:10AM -0700, Hongfei Cheng wrote:
>> 4). High resolution counter is not yet supported as we're exploring
>> different way(s) to implement TSC.
>
> Note the implications here: if you say "all armv8 will use
> architected timer", we can get away with a unique implementation. If
> you want to use another timer, then we will have to implement kernel
> helpers with a constant kernel/user ABI, so probably in the vdso.
> All this for an improved accuracy that may well not be worth the
> trouble.

Currently we're not concerned about tsc accuracy.

Could you elaborate a bit more on the "architected timer" and the
possible implementation you envision on Cortex-A53/A57?

Is the I-pipe tracer dependent upon a live/running tsc counter?

Thanks again,
Hongfei

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Gilles Chanteperdrix]

On Thu, Apr 30, 2015 at 01:04:07PM -0700, Hongfei Cheng wrote:
> On Thu, Apr 30, 2015 at 11:31 AM, Gilles Chanteperdrix
> <gilles.chanteperdrix@xenomai.org> wrote:
> > On Thu, Apr 30, 2015 at 08:53:10AM -0700, Hongfei Cheng wrote:
> >> 4). High resolution counter is not yet supported as we're exploring
> >> different way(s) to implement TSC.
> >
> > Note the implications here: if you say "all armv8 will use
> > architected timer", we can get away with a unique implementation. If
> > you want to use another timer, then we will have to implement kernel
> > helpers with a constant kernel/user ABI, so probably in the vdso.
> > All this for an improved accuracy that may well not be worth the
> > trouble.
> 
> Currently we're not concerned about tsc accuracy.
> 
> Could you elaborate a bit more on the "architected timer" and the
> possible implementation you envision on Cortex-A53/A57?

"architected timer" is the name of the timers available since cortex
A15 (on A7 also, I guess it allows big/little to run exactly the
same code on both cores), and I thought I had understood it was also
available on armv8. These timers are guaranteed to operate a long
period of time without wrapping (something like tens of years), so
they do not have a resolution as high as they could have (especially
since implementations use less than 32 bits of the higher 32 bits).

> 
> Is the I-pipe tracer dependent upon a live/running tsc counter?

To get timestamped results yes. But you can probably run it with
stubs, the timing information will be unusable that is all. The
frequency can not be null though, so you can set something like 1
MHz and it should work.

-- 
					    Gilles.

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Lennart Sorensen]

On Thu, Apr 30, 2015 at 10:14:09PM +0200, Gilles Chanteperdrix wrote:
> "architected timer" is the name of the timers available since cortex
> A15 (on A7 also, I guess it allows big/little to run exactly the
> same code on both cores), and I thought I had understood it was also
> available on armv8. These timers are guaranteed to operate a long
> period of time without wrapping (something like tens of years), so
> they do not have a resolution as high as they could have (especially
> since implementations use less than 32 bits of the higher 32 bits).

The A15 (and A7, A17, A12, etc) timer is certainly handy.  Some designs
have managed to screw it up though (like the exynos 5420, where samsung
decided they were going to use their own timer design from previous
designs, and not initialize or even run the A15 timer, so the register
is there, the timer is there, but it isn't running and hence you can't
use it, which pretty much means no chance of KVM on the exynos chips
given it assumes the architecture timer can be used by the guest too).

Good thing we use the TI am572x instead, which does have the architecture
timer, although with an errata I had to come up with a software fix for.
The speed is slightly wrong but consistently so.

It sure would be nice if chip makers would NOT try to reinvent the
wheel when the architecture has come up with a standard way to do it
that works well.

> To get timestamped results yes. But you can probably run it with
> stubs, the timing information will be unusable that is all. The
> frequency can not be null though, so you can set something like 1
> MHz and it should work.

-- 
Len Sorensen

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Gilles Chanteperdrix]

On Thu, Apr 30, 2015 at 05:21:51PM -0400, Lennart Sorensen wrote:
> On Thu, Apr 30, 2015 at 10:14:09PM +0200, Gilles Chanteperdrix wrote:
> > "architected timer" is the name of the timers available since cortex
> > A15 (on A7 also, I guess it allows big/little to run exactly the
> > same code on both cores), and I thought I had understood it was also
> > available on armv8. These timers are guaranteed to operate a long
> > period of time without wrapping (something like tens of years), so
> > they do not have a resolution as high as they could have (especially
> > since implementations use less than 32 bits of the higher 32 bits).
> 
> The A15 (and A7, A17, A12, etc) timer is certainly handy.  Some designs
> have managed to screw it up though (like the exynos 5420, where samsung
> decided they were going to use their own timer design from previous
> designs, and not initialize or even run the A15 timer, so the register
> is there, the timer is there, but it isn't running and hence you can't
> use it, which pretty much means no chance of KVM on the exynos chips
> given it assumes the architecture timer can be used by the guest too).
> 
> Good thing we use the TI am572x instead, which does have the architecture
> timer, although with an errata I had to come up with a software fix for.
> The speed is slightly wrong but consistently so.
> 
> It sure would be nice if chip makers would NOT try to reinvent the
> wheel when the architecture has come up with a standard way to do it
> that works well.

The problem I see with the architected timer is the minimum wrap
time imposed by the specification, which forces a low resolution if
the implementer does not want to implement the full 64 bits. So, it
may be a good reason to propose another timer, with say nanosecond
resolution, but without a guaranteed large wrap time, especially
since then implementation can extend it to the full 64 its in
software and have both high resolution and long wrap time.

-- 
					    Gilles.

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Lennart Sorensen]

On Thu, Apr 30, 2015 at 11:27:49PM +0200, Gilles Chanteperdrix wrote:
> The problem I see with the architected timer is the minimum wrap
> time imposed by the specification, which forces a low resolution if
> the implementer does not want to implement the full 64 bits. So, it
> may be a good reason to propose another timer, with say nanosecond
> resolution, but without a guaranteed large wrap time, especially
> since then implementation can extend it to the full 64 its in
> software and have both high resolution and long wrap time.

Well on the TI am572x it is 56 bits at about 6MHz, so it wraps every
2794 seconds according to the boot messages.  Would be nice if it was
64 bits, since that would avoid wrapping for quite a while.  And sure
a timer with 1ns would be great.  This one only gets 162ns.

The exynos5420 has a 32bit timer at 24MHz.  It wraps every 178 seconds.
Good thing we only use that as a build box.

-- 
Len Sorensen

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Gilles Chanteperdrix]

On Fri, May 01, 2015 at 09:45:47AM -0400, Lennart Sorensen wrote:
> On Thu, Apr 30, 2015 at 11:27:49PM +0200, Gilles Chanteperdrix wrote:
> > The problem I see with the architected timer is the minimum wrap
> > time imposed by the specification, which forces a low resolution if
> > the implementer does not want to implement the full 64 bits. So, it
> > may be a good reason to propose another timer, with say nanosecond
> > resolution, but without a guaranteed large wrap time, especially
> > since then implementation can extend it to the full 64 its in
> > software and have both high resolution and long wrap time.
> 
> Well on the TI am572x it is 56 bits at about 6MHz, so it wraps every
> 2794 seconds according to the boot messages.  Would be nice if it was
> 64 bits, since that would avoid wrapping for quite a while.  And sure
> a timer with 1ns would be great.  This one only gets 162ns.

6MHz is still not a lot (even omap3 did better than that).

And no, a A 56 bits counter running at 6MHz does not wrap in 2794
seconds, it wraps in:

(2^56) / 6000000 = 12009599006 s

That is:
12009599006 / (365 * 24 * 3600) = 380 years

So, you are safe, it can be considered to avoid wrapping for a
reasonable amount of time.

Which means really, that I do not understand why the frequency is so
low.

56 bits at 1 GHz would wrap in two years using the same computation,
that clearly would violate ARM specs. But 60 MHz for instance would
wrap in 38 years, that would seem reasonable.

-- 
					    Gilles.

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Lennart Sorensen]

On Fri, May 01, 2015 at 03:56:51PM +0200, Gilles Chanteperdrix wrote:
> 6MHz is still not a lot (even omap3 did better than that).
> 
> And no, a A 56 bits counter running at 6MHz does not wrap in 2794
> seconds, it wraps in:
> 
> (2^56) / 6000000 = 12009599006 s
> 
> That is:
> 12009599006 / (365 * 24 * 3600) = 380 years
> 
> So, you are safe, it can be considered to avoid wrapping for a
> reasonable amount of time.
> 
> Which means really, that I do not understand why the frequency is so
> low.

I hava a suspicition that the 6.144MHz was chosen because it is what
S/PDIF requires to run 48Khz audio.  Some it may have something to do
with multimedia handling.

And in low power mode they run it from 32.768 KHz * 375 / 2.

Sure looks like they could have easily made it faster, although making
it faster wouldn't necesarily make it any more accurate depending on
how it is implemented.  After all if it is just taking a 20MHz system
clock input and incrementing from that, then there isn't much point
running it faster.  If you can run it from the PLL circuit that runs
the CPU core on the other hand, that would be interesting.  Of course
that frequency can scale up and down if cpu frequency scaling is enabled.

> 56 bits at 1 GHz would wrap in two years using the same computation,
> that clearly would violate ARM specs. But 60 MHz for instance would
> wrap in 38 years, that would seem reasonable.

Does the kernel have a bug then:

[    0.000000] OMAP clockevent source: timer1 at 32786 Hz
[    0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 65536000000000ns
[    0.000000] OMAP clocksource: 32k_counter at 32768 Hz
[    0.000335] Architected cp15 timer(s) running at 6.14MHz (phys).
[    0.000370] sched_clock: 56 bits at 6MHz, resolution 162ns, wraps every 2794592043008ns
[    0.000376] Switching to timer-based delay loop
[    0.100554] Calibrating delay loop (skipped), value calculated using timer frequency.. 12.29 BogoMIPS (lpj=6147)

I think 2794592043008ns = 2794.592043008s, and I must admit I assumed
the kernel knew what it was talking about.  It looks like it does not.

I agree that thing isn't going to wrap ever.

I see a few patches have gone in to that code since 3.14, so perhaps
someone has fixed that printout.

-- 
Len Sorensen

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Gilles Chanteperdrix]

On Fri, May 01, 2015 at 10:36:31AM -0400, Lennart Sorensen wrote:
> On Fri, May 01, 2015 at 03:56:51PM +0200, Gilles Chanteperdrix wrote:
> > 6MHz is still not a lot (even omap3 did better than that).
> > 
> > And no, a A 56 bits counter running at 6MHz does not wrap in 2794
> > seconds, it wraps in:
> > 
> > (2^56) / 6000000 = 12009599006 s
> > 
> > That is:
> > 12009599006 / (365 * 24 * 3600) = 380 years
> > 
> > So, you are safe, it can be considered to avoid wrapping for a
> > reasonable amount of time.
> > 
> > Which means really, that I do not understand why the frequency is so
> > low.
> 
> I hava a suspicition that the 6.144MHz was chosen because it is what
> S/PDIF requires to run 48Khz audio.  Some it may have something to do
> with multimedia handling.
> 
> And in low power mode they run it from 32.768 KHz * 375 / 2.
> 
> Sure looks like they could have easily made it faster, although making
> it faster wouldn't necesarily make it any more accurate depending on
> how it is implemented.  After all if it is just taking a 20MHz system
> clock input and incrementing from that, then there isn't much point
> running it faster.  If you can run it from the PLL circuit that runs
> the CPU core on the other hand, that would be interesting.  Of course
> that frequency can scale up and down if cpu frequency scaling is
> enabled.

That is what the cortex A9 did, the twd local timers and the global
timer are derived from the cpu clock. That said, the twd are still
available on A15 I guess, so maybe the global timer is also
available?

> 
> > 56 bits at 1 GHz would wrap in two years using the same computation,
> > that clearly would violate ARM specs. But 60 MHz for instance would
> > wrap in 38 years, that would seem reasonable.
> 
> Does the kernel have a bug then:
> 
> [    0.000000] OMAP clockevent source: timer1 at 32786 Hz
> [    0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 65536000000000ns
> [    0.000000] OMAP clocksource: 32k_counter at 32768 Hz
> [    0.000335] Architected cp15 timer(s) running at 6.14MHz (phys).
> [    0.000370] sched_clock: 56 bits at 6MHz, resolution 162ns, wraps every 2794592043008ns
> [    0.000376] Switching to timer-based delay loop
> [    0.100554] Calibrating delay loop (skipped), value calculated using timer frequency.. 12.29 BogoMIPS (lpj=6147)
> 
> I think 2794592043008ns = 2794.592043008s, and I must admit I assumed
> the kernel knew what it was talking about.  It looks like it does not.

I do not know how that is computed. I do not think I did my
calculations wrong. Maybe the kernel does not use the raw value for
sched_clock, but something transformed with some shifts and
multiplies, and it tells you in how much time that transformed
counter wraps.

-- 
					    Gilles.

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Lennart Sorensen]

On Fri, May 01, 2015 at 04:44:28PM +0200, Gilles Chanteperdrix wrote:
> That is what the cortex A9 did, the twd local timers and the global
> timer are derived from the cpu clock. That said, the twd are still
> available on A15 I guess, so maybe the global timer is also
> available?

Well it looks like the A15 and newer do in fact not have that one.

Looking at the kernel driver for twd, it appears that when the cpu
changes frequency, the timer has to be reprogrammed to match which the
arch timer on the A15 does not because it is a fixed rate.  I suspect
because of the virtualization extensions, having a fixed rate clock that
guests can read and use too was considered important.  Too bad it is
rather low precision.

> I do not know how that is computed. I do not think I did my
> calculations wrong. Maybe the kernel does not use the raw value for
> sched_clock, but something transformed with some shifts and
> multiplies, and it tells you in how much time that transformed
> counter wraps.

There is some talk of a 50% or 12.5% safety margin, but that doesn't
explain it either.  Looks like a bug somehow.

Your calculation looks right, the kernel's looks crazy.

-- 
Len Sorensen

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Lennart Sorensen]

On Fri, May 01, 2015 at 11:33:25AM -0400, Lennart Sorensen wrote:
> There is some talk of a 50% or 12.5% safety margin, but that doesn't
> explain it either.  Looks like a bug somehow.
> 
> Your calculation looks right, the kernel's looks crazy.

Trying to boot 3.14.39 I just saw:

[    0.040557] I-pipe, 6.147 MHz clocksource, wrap in 3000934451555157 ms
[    0.054082] sched_clock: 56 bits at 6MHz, resolution 162ns, wraps every 2794592043008ns

I wonder who could be right... :)

Of course then it gets stuck at
[    0.070615] Switching to timer-based delay loop
[    0.081432] Interrupt pipeline (release #9)

so I better figure out what I did wrong with this new version.  Time to
check the config.  It booted before I added ipipe to it.

-- 
Len Sorensen

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Lennart Sorensen]

On Wed, May 13, 2015 at 01:01:56PM -0400, Lennart Sorensen wrote:
> On Fri, May 01, 2015 at 11:33:25AM -0400, Lennart Sorensen wrote:
> > There is some talk of a 50% or 12.5% safety margin, but that doesn't
> > explain it either.  Looks like a bug somehow.
> > 
> > Your calculation looks right, the kernel's looks crazy.
> 
> Trying to boot 3.14.39 I just saw:
> 
> [    0.040557] I-pipe, 6.147 MHz clocksource, wrap in 3000934451555157 ms
> [    0.054082] sched_clock: 56 bits at 6MHz, resolution 162ns, wraps every 2794592043008ns
> 
> I wonder who could be right... :)
> 
> Of course then it gets stuck at
> [    0.070615] Switching to timer-based delay loop
> [    0.081432] Interrupt pipeline (release #9)
> 
> so I better figure out what I did wrong with this new version.  Time to
> check the config.  It booted before I added ipipe to it.

Oh good, I managed to make it fail without ipipe enabled, so that's good.
Not an ipipe problem I think, unless the ipipe patch is at fault.

-- 
Len Sorensen

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Gilles Chanteperdrix]

On Fri, May 01, 2015 at 09:45:47AM -0400, Lennart Sorensen wrote:
> On Thu, Apr 30, 2015 at 11:27:49PM +0200, Gilles Chanteperdrix wrote:
> The exynos5420 has a 32bit timer at 24MHz.  It wraps every 178 seconds.
> Good thing we only use that as a build box.

Xenomai extends counters to 64 bits in software. So, if that counter
was used as a tsc by Xenomai, it would wrap in :

(1 << 64) / 24000000 = 768614336404 s

That is:

768614336404 / (365 * 24 * 3600) = 24372 years.

-- 
					    Gilles.

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Gilles Chanteperdrix]

On Thu, Apr 30, 2015 at 05:21:51PM -0400, Lennart Sorensen wrote:
> On Thu, Apr 30, 2015 at 10:14:09PM +0200, Gilles Chanteperdrix wrote:
> > "architected timer" is the name of the timers available since cortex
> > A15 (on A7 also, I guess it allows big/little to run exactly the
> > same code on both cores), and I thought I had understood it was also
> > available on armv8. These timers are guaranteed to operate a long
> > period of time without wrapping (something like tens of years), so
> > they do not have a resolution as high as they could have (especially
> > since implementations use less than 32 bits of the higher 32 bits).
> 
> The A15 (and A7, A17, A12, etc) timer is certainly handy.  Some designs
> have managed to screw it up though (like the exynos 5420, where samsung
> decided they were going to use their own timer design from previous
> designs, and not initialize or even run the A15 timer, so the register
> is there, the timer is there, but it isn't running and hence you can't
> use it, which pretty much means no chance of KVM on the exynos chips
> given it assumes the architecture timer can be used by the guest too).
> 
> Good thing we use the TI am572x instead, which does have the architecture
> timer, although with an errata I had to come up with a software fix for.
> The speed is slightly wrong but consistently so.

I had tested architected timers on omap5432, and I believe the
counter frequency was something like 5 MHz, and a high latency, this
looks like a serious regression when omap4 had the global timer
running at 500 MHz and a 22ns latency in user-space (including the
time to jump to the kernel helper), which is better than the latency
I have on an atom N230 where reading the tsc is a dedicated
instruction (like architected timers actually).

-- 
					    Gilles.

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Lennart Sorensen]

On Thu, Apr 30, 2015 at 11:34:04PM +0200, Gilles Chanteperdrix wrote:
> I had tested architected timers on omap5432, and I believe the
> counter frequency was something like 5 MHz, and a high latency, this
> looks like a serious regression when omap4 had the global timer
> running at 500 MHz and a 22ns latency in user-space (including the
> time to jump to the kernel helper), which is better than the latency
> I have on an atom N230 where reading the tsc is a dedicated
> instruction (like architected timers actually).

Well the omap5 uses the A15 architecture timer.  The omap4 does not
since the A9 didn't have one.

Now they do have lots of other timers in the chip that can be programmed.
I wonder if that could be used to make something much better.

I am pretty sure the omap5432 uses 6.144 MHz just like the am572x was
supposed to (if it wasn't for the errata that makes it 6.147541 MHz instead).

Looking at the manual for the am572x it looks like the general purpose
timers are only 32 bit and can run at either 32.768 KHz or system clock
(usually 20 MHz).  That's not much help.

-- 
Len Sorensen

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Gilles Chanteperdrix]

On Fri, May 01, 2015 at 10:20:42AM -0400, Lennart Sorensen wrote:
> On Thu, Apr 30, 2015 at 11:34:04PM +0200, Gilles Chanteperdrix wrote:
> > I had tested architected timers on omap5432, and I believe the
> > counter frequency was something like 5 MHz, and a high latency, this
> > looks like a serious regression when omap4 had the global timer
> > running at 500 MHz and a 22ns latency in user-space (including the
> > time to jump to the kernel helper), which is better than the latency
> > I have on an atom N230 where reading the tsc is a dedicated
> > instruction (like architected timers actually).
> 
> Well the omap5 uses the A15 architecture timer.  The omap4 does not
> since the A9 didn't have one.

The point is: the cortex A9 global timer runs at half the processor
frequency, that is 500 MHz for a 1GHz processor: that gives a much
better resolution than a timer with a frequency lower than 10 MHz
you get with a cortex A15. So since the cortex A15 is newer than the
cortex A9, it could be expected to do better or at least be
equivalent, rather than do much worse.

> 
> Now they do have lots of other timers in the chip that can be programmed.
> I wonder if that could be used to make something much better.
> 
> I am pretty sure the omap5432 uses 6.144 MHz just like the am572x was
> supposed to (if it wasn't for the errata that makes it 6.147541
> MHz instead).

Yes, 5MHZ, 6 MHZ, that is the same bad order of magnitude.

> 
> Looking at the manual for the am572x it looks like the general purpose
> timers are only 32 bit and can run at either 32.768 KHz or system clock
> (usually 20 MHz).  That's not much help.

Once again: Xenomai extends 32 bits counter to 64 bits in software.
So, a 32 bits counter running at 20 MHz is still better than the
architected timer running at 6MHz. 6 MHz is so low that by reading
twice the counter in less than 166ns (which is really really doable
on these processors), you could get twice the same value. On cortex
A9, I could read the counter twice in userspace in 22ns, and get
different values. On cortex A9, anything lower than 50 MHz would
not have been sufficient for user-space.

-- 
					    Gilles.

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Lennart Sorensen]

On Fri, May 01, 2015 at 04:30:45PM +0200, Gilles Chanteperdrix wrote:
> The point is: the cortex A9 global timer runs at half the processor
> frequency, that is 500 MHz for a 1GHz processor: that gives a much
> better resolution than a timer with a frequency lower than 10 MHz
> you get with a cortex A15. So since the cortex A15 is newer than the
> cortex A9, it could be expected to do better or at least be
> equivalent, rather than do much worse.

Did any of the A9 designs allow changing the speed on the fly the way
the A15 and others do?

I certainly like the idea of a 500MHz timer.

> > Now they do have lots of other timers in the chip that can be programmed.
> > I wonder if that could be used to make something much better.
> > 
> > I am pretty sure the omap5432 uses 6.144 MHz just like the am572x was
> > supposed to (if it wasn't for the errata that makes it 6.147541
> > MHz instead).
> 
> Yes, 5MHZ, 6 MHZ, that is the same bad order of magnitude.
> 
> > Looking at the manual for the am572x it looks like the general purpose
> > timers are only 32 bit and can run at either 32.768 KHz or system clock
> > (usually 20 MHz).  That's not much help.
> 
> Once again: Xenomai extends 32 bits counter to 64 bits in software.
> So, a 32 bits counter running at 20 MHz is still better than the
> architected timer running at 6MHz. 6 MHz is so low that by reading
> twice the counter in less than 166ns (which is really really doable
> on these processors), you could get twice the same value. On cortex
> A9, I could read the counter twice in userspace in 22ns, and get
> different values. On cortex A9, anything lower than 50 MHz would
> not have been sufficient for user-space.

Yep, that makes sense.  160 ns is a long time on a 1GHz CPU.

It looks like one of the clock sources for the timers is
./sys_clkin2/abe_dpll_sys_clk_mux/abe_dpll_clk_mux/dpll_abe_ck/dpll_abe_x2_ck/dpll_abe_m2x2_ck/abe_clk/aess_fclk/abe_giclk_div/clk_rate
which is 361267200 Hz (at least right now it is), so that might actually
be able to be used to make a nice counter.  All the other sources are
either external clock input pins, or sysclk1 (20 MHz usually) or sysclk2
(22.5792 MHz).  The 361267200 is 16 * sysclk2.

I wonder how to create a clocksource from a general purpose timer.

-- 
Len Sorensen

Re: [Xenomai] Ipipe-core patched kernel fails to start when certain platform drivers are enabled

[Gilles Chanteperdrix]

On Fri, May 01, 2015 at 11:11:53AM -0400, Lennart Sorensen wrote:
> On Fri, May 01, 2015 at 04:30:45PM +0200, Gilles Chanteperdrix wrote:
> > The point is: the cortex A9 global timer runs at half the processor
> > frequency, that is 500 MHz for a 1GHz processor: that gives a much
> > better resolution than a timer with a frequency lower than 10 MHz
> > you get with a cortex A15. So since the cortex A15 is newer than the
> > cortex A9, it could be expected to do better or at least be
> > equivalent, rather than do much worse.
> 
> Did any of the A9 designs allow changing the speed on the fly the way
> the A15 and others do?
> 
> I certainly like the idea of a 500MHz timer.
> 
> > > Now they do have lots of other timers in the chip that can be programmed.
> > > I wonder if that could be used to make something much better.
> > > 
> > > I am pretty sure the omap5432 uses 6.144 MHz just like the am572x was
> > > supposed to (if it wasn't for the errata that makes it 6.147541
> > > MHz instead).
> > 
> > Yes, 5MHZ, 6 MHZ, that is the same bad order of magnitude.
> > 
> > > Looking at the manual for the am572x it looks like the general purpose
> > > timers are only 32 bit and can run at either 32.768 KHz or system clock
> > > (usually 20 MHz).  That's not much help.
> > 
> > Once again: Xenomai extends 32 bits counter to 64 bits in software.
> > So, a 32 bits counter running at 20 MHz is still better than the
> > architected timer running at 6MHz. 6 MHz is so low that by reading
> > twice the counter in less than 166ns (which is really really doable
> > on these processors), you could get twice the same value. On cortex
> > A9, I could read the counter twice in userspace in 22ns, and get
> > different values. On cortex A9, anything lower than 50 MHz would
> > not have been sufficient for user-space.
> 
> Yep, that makes sense.  160 ns is a long time on a 1GHz CPU.
> 
> It looks like one of the clock sources for the timers is
> ./sys_clkin2/abe_dpll_sys_clk_mux/abe_dpll_clk_mux/dpll_abe_ck/dpll_abe_x2_ck/dpll_abe_m2x2_ck/abe_clk/aess_fclk/abe_giclk_div/clk_rate
> which is 361267200 Hz (at least right now it is), so that might actually
> be able to be used to make a nice counter.  All the other sources are
> either external clock input pins, or sysclk1 (20 MHz usually) or sysclk2
> (22.5792 MHz).  The 361267200 is 16 * sysclk2.
> 
> I wonder how to create a clocksource from a general purpose timer.

OMAP3 uses a GP timer as clocksource, and OMAP4 does when running
only one core. So, if these IPs have not changed, the code is there
to be used. Note however that last time I tried changing the GP
timer used on OMAP4, the timer I used would not start, so a write to
a register to start a clock was probably missing, or not writing to
the right register. Since nobody ever uses another GP timer, this
code is largely untested.

-- 
					    Gilles.