Re: [PATCH v2 RESEND 2/2] ARM: local timers: add timer support using IO mapped register

Re: [PATCH v2 RESEND 2/2] ARM: local timers: add timer support using IO mapped register

[Dave Martin]

[ Note: please aim to CC devicetree-discuss@lists.ozlabs.org with any
patches or bindings relevant to device tree. ]

[ Lorenzo, there's a question for you further down this mail. ]

On Fri, Sep 28, 2012 at 01:28:58PM +0100, Mark Rutland wrote:
> On Tue, Sep 25, 2012 at 08:08:47PM +0100, Rohit Vaswani wrote:
> > Any comments ?
> 
> I have a few questions about the irq side of things.
> 
> > > diff --git a/Documentation/devicetree/bindings/arm/arch_timer.txt b/Documentation/devicetree/bindings/arm/arch_timer.txt
> > > index 52478c8..8e01328 100644
> > > --- a/Documentation/devicetree/bindings/arm/arch_timer.txt
> > > +++ b/Documentation/devicetree/bindings/arm/arch_timer.txt
> > > @@ -7,10 +7,13 @@ The timer is attached to a GIC to deliver its per-processor interrupts.
> > >
> > >   ** Timer node properties:
> > >
> > > -- compatible : Should at least contain "arm,armv7-timer".
> > > +- compatible : Should at least contain "arm,armv7-timer" or
> > > +  "arm,armv7-timer-mem" if using the memory mapped arch timer interface.
> > >
> > > -- interrupts : Interrupt list for secure, non-secure, virtual and
> > > -  hypervisor timers, in that order.
> > > +- interrupts : If using the cp15 interface, the interrupt list for secure,
> > > +  non-secure, virtual and hypervisor timers, in that order.

We should use the correct architectural terms for documenting these.
For example, "non-secure" requires qualification.

If I understand the ARM ARM correctly, the architectural names are

 * Secure PL1 physical
 * Non-secure PL2 physical
 * Non-secure PL1 physical
 * Virtual

[...]

> What privilege level are the interrupts for the memory-mapped interface?
> 
> Could each core have multiple interrupts at different privilege levels?
> 
> I also notice we don't seem to handle the case of systems without security
> extensions, which only have physical and virtual interrupts. If we're adding
> support for different interrupt setups, how do we handle them?

Agreed, some of the timers will be absent depending on the architectural
options implemented by the processor.

Also, timers not visible to / usable by the kernel being booted should
probably left out of the DT, even though the hardware may physically have
them:

For example, a guest should not see the Secure PL1 physical timer or the
Non-secure PL2 physical timer.  A hypervisor or host OS which does not
provide the guest with access to the Non-secure PL1 physical timer (such
as KVM) should leave that out too: there is no exception-free way for
the OS to probe that directly: i.e., from the guest's point of view the
device doesn't exist at all, so it seems wrong for the DT to include them
in this case.


[...]

> > > +  If using the memory mapped interface, list the interrupts for each core,
> > > +  starting with core 0.
> 
> I take it that core 0 means physical cpu 0 (i.e. MPIDR.Aff{2,1,0} == 0)?

Lorenzo, should we have a standard way of referring to CPUs and topology
nodes documented as part of the topology bindings?  We certainly need
rules of some kind, since when the topology is non-trivial there is no
well-defined "first" CPU, nor any single correct order in which to list
CPUs.

The topology may also be sparsely populated (e.g.,
Aff[2,1,0] in { (0,0,0), (0,0,1), (0,1,0), (0,1,1), (0,1,2), (0,1,3) })

It would be bad if different driver bindings end up solving this in
different ways (even non-broken ways)

Cheers
---Dave

[...]

Re: [PATCH v2 RESEND 2/2] ARM: local timers: add timer support using IO mapped register

[Lorenzo Pieralisi]

On Fri, Sep 28, 2012 at 04:57:46PM +0100, Dave Martin wrote:
> [ Note: please aim to CC devicetree-discuss@lists.ozlabs.org with any
> patches or bindings relevant to device tree. ]
> 
> [ Lorenzo, there's a question for you further down this mail. ]

[...]

> > > > +  If using the memory mapped interface, list the interrupts for each core,
> > > > +  starting with core 0.
> > 
> > I take it that core 0 means physical cpu 0 (i.e. MPIDR.Aff{2,1,0} == 0)?
> 
> Lorenzo, should we have a standard way of referring to CPUs and topology
> nodes documented as part of the topology bindings?  We certainly need
> rules of some kind, since when the topology is non-trivial there is no
> well-defined "first" CPU, nor any single correct order in which to list
> CPUs.

I think, and that's just my opinion, that whatever solution we go for to
describe the topology must contain the information needed by all kernel
subsystems to retrieve HW information. I do not think we should document
how devices connect to CPU(s)/Cluster(s) in the topology bindings per-se,
since those are properties that belong to device nodes.

There must be a common way for all devices to link to the topology, though.

The topology must be descriptive enough to cater for all required cases
and that's what Mark with PMU and all of us are trying to come up with, a solid
way to represent with DT the topology of current and future ARM systems.

First idea I implemented and related LAK posting:

http://lists.infradead.org/pipermail/linux-arm-kernel/2012-January/080873.html

Are "cluster" nodes really needed or "cpu" nodes are enough ? I do not
know, let's get this discussion started, that's all I need.

But definitely declaring IRQs in physical CPU id order (and mind, as you say,
physical CPU ids, ie MPIDR, can be sparsely populated) and initializing them
*thinking* the order is the logical one is plainly broken.

> The topology may also be sparsely populated (e.g.,
> Aff[2,1,0] in { (0,0,0), (0,0,1), (0,1,0), (0,1,1), (0,1,2), (0,1,3) })
> 
> It would be bad if different driver bindings end up solving this in
> different ways (even non-broken ways)

Yes, I agree and code that relies on any temporary work-around to tackle
this problem should not be merged before we set in stone proper topology
bindings.

Lorenzo

Re: [PATCH v2 RESEND 2/2] ARM: local timers: add timer support using IO mapped register

[Dave Martin]

On Fri, Sep 28, 2012 at 06:15:53PM +0100, Lorenzo Pieralisi wrote:
> On Fri, Sep 28, 2012 at 04:57:46PM +0100, Dave Martin wrote:
> > [ Note: please aim to CC devicetree-discuss@lists.ozlabs.org with any
> > patches or bindings relevant to device tree. ]
> > 
> > [ Lorenzo, there's a question for you further down this mail. ]
> 
> [...]
> 
> > > > > +  If using the memory mapped interface, list the interrupts for each core,
> > > > > +  starting with core 0.
> > > 
> > > I take it that core 0 means physical cpu 0 (i.e. MPIDR.Aff{2,1,0} == 0)?
> > 
> > Lorenzo, should we have a standard way of referring to CPUs and topology
> > nodes documented as part of the topology bindings?  We certainly need
> > rules of some kind, since when the topology is non-trivial there is no
> > well-defined "first" CPU, nor any single correct order in which to list
> > CPUs.
> 
> I think, and that's just my opinion, that whatever solution we go for to
> describe the topology must contain the information needed by all kernel
> subsystems to retrieve HW information. I do not think we should document
> how devices connect to CPU(s)/Cluster(s) in the topology bindings per-se,
> since those are properties that belong to device nodes.

Well, I guess the other approach is to establish a firm precedent, which
means that we need to watch carefully for people proposing new bindings
which refer to the topology in inconsistent ways.

> 
> There must be a common way for all devices to link to the topology, though.
> 
> The topology must be descriptive enough to cater for all required cases
> and that's what Mark with PMU and all of us are trying to come up with, a solid
> way to represent with DT the topology of current and future ARM systems.
> 
> First idea I implemented and related LAK posting:
> 
> http://lists.infradead.org/pipermail/linux-arm-kernel/2012-January/080873.html
> 
> Are "cluster" nodes really needed or "cpu" nodes are enough ? I do not
> know, let's get this discussion started, that's all I need.

One thing which now occurs to me on this point it that if we want to describe
the CCI properly in the DT (yes) then we need a way to describe the mapping
between clusters and CCI slave ports.  Currently that knowledge just has to
be a hard-coded hack somewhere: it's not probeable at all.

I'm not sure how we do that, or how we describe the cache topology, without
the clusters being explicit in the DT

...unless you already have ideas ?

Cheers
---Dave

> But definitely declaring IRQs in physical CPU id order (and mind, as you say,
> physical CPU ids, ie MPIDR, can be sparsely populated) and initializing them
> *thinking* the order is the logical one is plainly broken.
> 
> > The topology may also be sparsely populated (e.g.,
> > Aff[2,1,0] in { (0,0,0), (0,0,1), (0,1,0), (0,1,1), (0,1,2), (0,1,3) })
> > 
> > It would be bad if different driver bindings end up solving this in
> > different ways (even non-broken ways)
> 
> Yes, I agree and code that relies on any temporary work-around to tackle
> this problem should not be merged before we set in stone proper topology
> bindings.
> 
> Lorenzo
> 

Re: [PATCH v2 RESEND 2/2] ARM: local timers: add timer support using IO mapped register

[Lorenzo Pieralisi]

On Tue, Oct 02, 2012 at 12:27:04PM +0100, Dave Martin wrote:
> On Fri, Sep 28, 2012 at 06:15:53PM +0100, Lorenzo Pieralisi wrote:
> > On Fri, Sep 28, 2012 at 04:57:46PM +0100, Dave Martin wrote:

[...]

> > There must be a common way for all devices to link to the topology, though.
> > 
> > The topology must be descriptive enough to cater for all required cases
> > and that's what Mark with PMU and all of us are trying to come up with, a solid
> > way to represent with DT the topology of current and future ARM systems.
> > 
> > First idea I implemented and related LAK posting:
> > 
> > http://lists.infradead.org/pipermail/linux-arm-kernel/2012-January/080873.html
> > 
> > Are "cluster" nodes really needed or "cpu" nodes are enough ? I do not
> > know, let's get this discussion started, that's all I need.
> 
> One thing which now occurs to me on this point it that if we want to describe
> the CCI properly in the DT (yes) then we need a way to describe the mapping
> between clusters and CCI slave ports.  Currently that knowledge just has to
> be a hard-coded hack somewhere: it's not probeable at all.

That's definitely a good point. We can still define CCI ports as belonging
to a range of CPUs, but that's a bit of a stretch IMHO.

> I'm not sure how we do that, or how we describe the cache topology, without
> the clusters being explicit in the DT
> 
> ...unless you already have ideas ?

Either we define the cluster node explicitly or we can always see it as a
collection of CPUs, ie phandles to "cpu" nodes. That's what the decision
we have to make is all about. I think that describing it explicitly make
sense, but we need to check all possible use cases to see if that's
worthwhile.

Lorenzo

Re: [PATCH v2 RESEND 2/2] ARM: local timers: add timer support using IO mapped register

[Dave Martin]

On Tue, Oct 02, 2012 at 02:44:44PM +0100, Lorenzo Pieralisi wrote:
> On Tue, Oct 02, 2012 at 12:27:04PM +0100, Dave Martin wrote:
> > On Fri, Sep 28, 2012 at 06:15:53PM +0100, Lorenzo Pieralisi wrote:
> > > On Fri, Sep 28, 2012 at 04:57:46PM +0100, Dave Martin wrote:
> 
> [...]
> 
> > > There must be a common way for all devices to link to the topology, though.
> > > 
> > > The topology must be descriptive enough to cater for all required cases
> > > and that's what Mark with PMU and all of us are trying to come up with, a solid
> > > way to represent with DT the topology of current and future ARM systems.
> > > 
> > > First idea I implemented and related LAK posting:
> > > 
> > > http://lists.infradead.org/pipermail/linux-arm-kernel/2012-January/080873.html
> > > 
> > > Are "cluster" nodes really needed or "cpu" nodes are enough ? I do not
> > > know, let's get this discussion started, that's all I need.
> > 
> > One thing which now occurs to me on this point it that if we want to describe
> > the CCI properly in the DT (yes) then we need a way to describe the mapping
> > between clusters and CCI slave ports.  Currently that knowledge just has to
> > be a hard-coded hack somewhere: it's not probeable at all.
> 
> That's definitely a good point. We can still define CCI ports as belonging
> to a range of CPUs, but that's a bit of a stretch IMHO.
> 
> > I'm not sure how we do that, or how we describe the cache topology, without
> > the clusters being explicit in the DT
> > 
> > ...unless you already have ideas ?
> 
> Either we define the cluster node explicitly or we can always see it as a
> collection of CPUs, ie phandles to "cpu" nodes. That's what the decision
> we have to make is all about. I think that describing it explicitly make
> sense, but we need to check all possible use cases to see if that's
> worthwhile.

How is the cache topology described today (forgive my laziness in not
answering this question for myself)?  The issues are somewhat similar.

I still have some misgivings about describing clusters in terms of sets of
CPUs.  For example, when we boot up a cluster, we have to set up ... the
cluster.  This is a distinct thing which we must set up in addition to
any of the actual CPUs.

There is a strict child/parent relationship between clusters and CPUs, so
a tree of nodes does seem the most natural description ... but I'm not
aware of all the background to this discussion.

Cheers
---Dave