PCI bus node location

PCI bus node location

[Rafal Jaworowski]

Hi,
I have a couple of questions regarding host/PCI bridges nodes location:

- What is the reason most of the DTS definitions have the host/PCI  
bridges hanging off the root node, even though they are most often  
really part of the soc?
- Is this is some OF heritage (I couldn't find anything explicit about  
it in the original PCI bindings docs)?
- Is this convention enforced in FDT, or could PCI bus nodes be  
children of the soc node as well?

Rafal

Re: PCI bus node location

[Grant Likely]

On Mon, Nov 9, 2009 at 12:20 PM, Rafal Jaworowski <raj-nYOzD4b6Jr9Wk0Htik3J/w@public.gmane.org> wrote:
> Hi,
> I have a couple of questions regarding host/PCI bridges nodes location:
>
> - What is the reason most of the DTS definitions have the host/PCI bridges
> hanging off the root node, even though they are most often really part of
> the soc?
> - Is this is some OF heritage (I couldn't find anything explicit about it in
> the original PCI bindings docs)?
> - Is this convention enforced in FDT, or could PCI bus nodes be children of
> the soc node as well?

It was a solution to an engineering problem.  The PCI control
registers are indeed within the IMMR region, and when we first started
doing PowerPC FDT board ports, the PCI node was a child of the SoC
node.  However, since PCI is also bridge with its own address space
translation, having it live in the SoC node causes difficulties.
Specifically, all of the entries in the PCI node ranges property would
need similar counterparts in the SoC node ranges property; a scheme
that doesn't reflect well the actual behaviour of the IMMR region.

Two alternate solutions were proposed.  One was to split the PCI node
into a PCI bridge node which describes the translations, and a PCI
control node which describes how to access the PCI bridge registers.
Some sort of linkage (probably a phandle) would be needed to relate
the two.  The second was to simply move the PCI node out to be a
parent of the root.  The second option was the one chosen because it
was the path of least resistance.  It may not be the most 'correct'
solution, but it has worked out quite well in practice.  There is
nothing in the PCI or FDT infrastructure code that enforces this
convention.  In fact, if the appropriate ranges properties were added
back to the IMMR node, then the PCI node could become a child of the
IMMR node without any code changes (but it still wouldn't be a 100%
'correct' description of the hardware).

See also this commit:

commit 1b3c5cdab49a605f0e048e1ccbf4cc61a2626485
Author: Kumar Gala <galak-XVmvHMARGAS8U2dJNN8I7kB+6BGkLq7r@public.gmane.org>
Date:   Wed Sep 12 18:23:46 2007 -0500

    [POWERPC] Move PCI nodes to be sibilings with SOC nodes

    Updated the device trees to have the PCI nodes be at the same level as
    the SOC node.  This is to make it so that the SOC nodes children address
    space is just on chip registers and not other bus memory as well.

    Also, for PCIe nodes added a P2P bridge to handle the virtual P2P bridge
    that exists in the PHB.

    Signed-off-by: Kumar Gala <galak-XVmvHMARGAS8U2dJNN8I7kB+6BGkLq7r@public.gmane.org>

Cheers,
g.

-- 
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.

Re: PCI bus node location

[David Gibson]

On Mon, Nov 09, 2009 at 07:36:57PM -0700, Grant Likely wrote:
> On Mon, Nov 9, 2009 at 12:20 PM, Rafal Jaworowski <raj-nYOzD4b6Jr9Wk0Htik3J/w@public.gmane.org> wrote:
> > Hi,
> > I have a couple of questions regarding host/PCI bridges nodes location:
> >
> > - What is the reason most of the DTS definitions have the host/PCI bridges
> > hanging off the root node, even though they are most often really part of
> > the soc?
> > - Is this is some OF heritage (I couldn't find anything explicit about it in
> > the original PCI bindings docs)?
> > - Is this convention enforced in FDT, or could PCI bus nodes be children of
> > the soc node as well?
> 
> It was a solution to an engineering problem.  The PCI control
> registers are indeed within the IMMR region, and when we first started
> doing PowerPC FDT board ports, the PCI node was a child of the SoC
> node.  However, since PCI is also bridge with its own address space
> translation, having it live in the SoC node causes difficulties.
> Specifically, all of the entries in the PCI node ranges property would
> need similar counterparts in the SoC node ranges property; a scheme
> that doesn't reflect well the actual behaviour of the IMMR region.
> 
> Two alternate solutions were proposed.  One was to split the PCI node
> into a PCI bridge node which describes the translations, and a PCI
> control node which describes how to access the PCI bridge registers.
> Some sort of linkage (probably a phandle) would be needed to relate
> the two.  The second was to simply move the PCI node out to be a
> parent of the root.  The second option was the one chosen because it
> was the path of least resistance.  It may not be the most 'correct'
> solution, but it has worked out quite well in practice.  There is
> nothing in the PCI or FDT infrastructure code that enforces this
> convention.  In fact, if the appropriate ranges properties were added
> back to the IMMR node, then the PCI node could become a child of the
> IMMR node without any code changes (but it still wouldn't be a 100%
> 'correct' description of the hardware).

Right.

Under the new scheme, the "soc" node is really a historical misnomer -
it represents just the things within the IMMR, not everything on the
SoC.  A number of chips also have the localbus controller as a
separate node, likewise within the SoC but not within the IMMR, so not
a child of the soc node.

Note also that 4xx chips, unlike the Freescale ones do have the PCI
host bridge under the plb node (which represents the main bus on the
SoC).

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: PCI bus node location

[Rafal Jaworowski]

On 2009-11-10, at 04:12, David Gibson wrote:

> On Mon, Nov 09, 2009 at 07:36:57PM -0700, Grant Likely wrote:
>> On Mon, Nov 9, 2009 at 12:20 PM, Rafal Jaworowski  
>> <raj-nYOzD4b6Jr9Wk0Htik3J/w@public.gmane.org> wrote:
>>> Hi,
>>> I have a couple of questions regarding host/PCI bridges nodes  
>>> location:
>>>
>>> - What is the reason most of the DTS definitions have the host/PCI  
>>> bridges
>>> hanging off the root node, even though they are most often really  
>>> part of
>>> the soc?
>>> - Is this is some OF heritage (I couldn't find anything explicit  
>>> about it in
>>> the original PCI bindings docs)?
>>> - Is this convention enforced in FDT, or could PCI bus nodes be  
>>> children of
>>> the soc node as well?
>>
>> It was a solution to an engineering problem.  The PCI control
>> registers are indeed within the IMMR region, and when we first  
>> started
>> doing PowerPC FDT board ports, the PCI node was a child of the SoC
>> node.  However, since PCI is also bridge with its own address space
>> translation, having it live in the SoC node causes difficulties.
>> Specifically, all of the entries in the PCI node ranges property  
>> would
>> need similar counterparts in the SoC node ranges property; a scheme
>> that doesn't reflect well the actual behaviour of the IMMR region.
>>
>> Two alternate solutions were proposed.  One was to split the PCI node
>> into a PCI bridge node which describes the translations, and a PCI
>> control node which describes how to access the PCI bridge registers.
>> Some sort of linkage (probably a phandle) would be needed to relate
>> the two.  The second was to simply move the PCI node out to be a
>> parent of the root.  The second option was the one chosen because it
>> was the path of least resistance.  It may not be the most 'correct'
>> solution, but it has worked out quite well in practice.  There is
>> nothing in the PCI or FDT infrastructure code that enforces this
>> convention.  In fact, if the appropriate ranges properties were added
>> back to the IMMR node, then the PCI node could become a child of the
>> IMMR node without any code changes (but it still wouldn't be a 100%
>> 'correct' description of the hardware).
>
> Right.
>
> Under the new scheme, the "soc" node is really a historical misnomer -
> it represents just the things within the IMMR, not everything on the
> SoC.  A number of chips also have the localbus controller as a
> separate node, likewise within the SoC but not within the IMMR, so not
> a child of the soc node.

Hm, how do we know whether something belongs under the IMMR/CCSR node  
or not (even though it physically sits there :-)?

Is the 'soc' node going to be named something less confusing then?

> Note also that 4xx chips, unlike the Freescale ones do have the PCI
> host bridge under the plb node (which represents the main bus on the
> SoC).

Yea, I noticed, which made me even more confused, so thanks again for  
clarifications.

Rafal

Re: PCI bus node location

[David Gibson]

On Tue, Nov 10, 2009 at 05:55:33PM +0100, Rafal Jaworowski wrote:
> On 2009-11-10, at 04:12, David Gibson wrote:
> >On Mon, Nov 09, 2009 at 07:36:57PM -0700, Grant Likely wrote:
> >>On Mon, Nov 9, 2009 at 12:20 PM, Rafal Jaworowski
[snip]
> >Right.
> >
> >Under the new scheme, the "soc" node is really a historical misnomer -
> >it represents just the things within the IMMR, not everything on the
> >SoC.  A number of chips also have the localbus controller as a
> >separate node, likewise within the SoC but not within the IMMR, so not
> >a child of the soc node.
> 
> Hm, how do we know whether something belongs under the IMMR/CCSR
> node or not (even though it physically sits there :-)?

Well, the basic criterion is whether its registers are relocated with
the rest of the IMMR.  Which would cover the PCI control registers, of
course, but they're not there for the reasons given.  Yes, it's kind
of nasty.

> Is the 'soc' node going to be named something less confusing then?

I think it should, but it's difficult to change because u-boot has
dependencies on it.  I think there are a few newer trees where it's
named IMMR.

> >Note also that 4xx chips, unlike the Freescale ones do have the PCI
> >host bridge under the plb node (which represents the main bus on the
> >SoC).
> 
> Yea, I noticed, which made me even more confused, so thanks again
> for clarifications.
> 
> Rafal
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: PCI bus node location

[Rafal Jaworowski]

On 2009-11-11, at 00:44, David Gibson wrote:

> On Tue, Nov 10, 2009 at 05:55:33PM +0100, Rafal Jaworowski wrote:
>> On 2009-11-10, at 04:12, David Gibson wrote:
>>> On Mon, Nov 09, 2009 at 07:36:57PM -0700, Grant Likely wrote:
>>>> On Mon, Nov 9, 2009 at 12:20 PM, Rafal Jaworowski
> [snip]
>>> Right.
>>>
>>> Under the new scheme, the "soc" node is really a historical  
>>> misnomer -
>>> it represents just the things within the IMMR, not everything on the
>>> SoC.  A number of chips also have the localbus controller as a
>>> separate node, likewise within the SoC but not within the IMMR, so  
>>> not
>>> a child of the soc node.
>>
>> Hm, how do we know whether something belongs under the IMMR/CCSR
>> node or not (even though it physically sits there :-)?
>
> Well, the basic criterion is whether its registers are relocated with
> the rest of the IMMR.  Which would cover the PCI control registers, of
> course, but they're not there for the reasons given.  Yes, it's kind
> of nasty.

It seems like the soc/simple-bus node is currently a rather arbitrary  
subset of peripherals available through SOC internal registers range.  
My concern is that without clear grouping criteria this is quite  
confusing.

>> Is the 'soc' node going to be named something less confusing then?
>
> I think it should, but it's difficult to change because u-boot has
> dependencies on it.  I think there are a few newer trees where it's
> named IMMR.

So is device_type = "soc" deprecated now for new trees, and for this  
kind of a bus node the only legit differentiator is the compatible =  
"simple-bus" prop?

BTW: in ePAPR examples there is device_type = "simple-bus" in a 8572  
soc node, is this a bug or a leftover?

Rafal

Re: PCI bus node location

[David Gibson]

On Wed, Nov 11, 2009 at 03:17:52PM +0100, Rafal Jaworowski wrote:
> 
> On 2009-11-11, at 00:44, David Gibson wrote:
> 
> >On Tue, Nov 10, 2009 at 05:55:33PM +0100, Rafal Jaworowski wrote:
> >>On 2009-11-10, at 04:12, David Gibson wrote:
> >>>On Mon, Nov 09, 2009 at 07:36:57PM -0700, Grant Likely wrote:
> >>>>On Mon, Nov 9, 2009 at 12:20 PM, Rafal Jaworowski
> >[snip]
> >>>Right.
> >>>
> >>>Under the new scheme, the "soc" node is really a historical
> >>>misnomer -
> >>>it represents just the things within the IMMR, not everything on the
> >>>SoC.  A number of chips also have the localbus controller as a
> >>>separate node, likewise within the SoC but not within the
> >>>IMMR, so not
> >>>a child of the soc node.
> >>
> >>Hm, how do we know whether something belongs under the IMMR/CCSR
> >>node or not (even though it physically sits there :-)?
> >
> >Well, the basic criterion is whether its registers are relocated with
> >the rest of the IMMR.  Which would cover the PCI control registers, of
> >course, but they're not there for the reasons given.  Yes, it's kind
> >of nasty.
> 
> It seems like the soc/simple-bus node is currently a rather
> arbitrary subset of peripherals available through SOC internal
> registers range. My concern is that without clear grouping criteria
> this is quite confusing.

It's not arbitrary.  It's everything whose registers lie in the
movable IMMR region, except those that have a reason to be outside.
With the reason to be outside being that they control the bridging of
address ranges that are *not* in the IMMR.

> >>Is the 'soc' node going to be named something less confusing then?
> >
> >I think it should, but it's difficult to change because u-boot has
> >dependencies on it.  I think there are a few newer trees where it's
> >named IMMR.
> 
> So is device_type = "soc" deprecated now for new trees, and for this
> kind of a bus node the only legit differentiator is the compatible =
> "simple-bus" prop?

Uh.. well.  It's sort of in that odd half-way house.  It's fairly
widely acknowledged that there are problems with that representation,
but a new better one has not yet been articulated sufficiently clearly
to formally deprecate the old way.  It is after all better to keep
with the old way until we do have a new method that is clearly better,
in order to keep down the number of old deprecated formats we need to
deal with eventually.

> BTW: in ePAPR examples there is device_type = "simple-bus" in a 8572
> soc node, is this a bug or a leftover?

That would be a bug.  I'll add it to my list of things to point out in
the ePAPR revision / errata process.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: PCI bus node location

[Grant Likely]

On Tue, Nov 10, 2009 at 9:55 AM, Rafal Jaworowski <raj-nYOzD4b6Jr9Wk0Htik3J/w@public.gmane.org> wrote:
>
> On 2009-11-10, at 04:12, David Gibson wrote:
>> Under the new scheme, the "soc" node is really a historical misnomer -
>> it represents just the things within the IMMR, not everything on the
>> SoC.  A number of chips also have the localbus controller as a
>> separate node, likewise within the SoC but not within the IMMR, so not
>> a child of the soc node.
>
> Hm, how do we know whether something belongs under the IMMR/CCSR node or not
> (even though it physically sits there :-)?
>
> Is the 'soc' node going to be named something less confusing then?
>
>> Note also that 4xx chips, unlike the Freescale ones do have the PCI
>> host bridge under the plb node (which represents the main bus on the
>> SoC).
>
> Yea, I noticed, which made me even more confused, so thanks again for
> clarifications.

The whole point of the IMMR node is to group things that are
relocatable together.  It was chosen as a matter of convenience, but
it would be absolutely valid to eliminate the IMMR node entirely just
make all the internal devices peers of the root node.  Except for a
few bugs that would probably be exposed, the current kernel would
parse that device tree just fine.

So, while the expressed ideal is to describe the exact bus topology,
the reality is that there are many reasons why the .dts hierarchy will
not match the physical hierarchy exactly (ie. omitting nodes for
transparent bridges which software has no control over, or adding
nodes to group logically related devices together, or as with
Freescale PCI moving the nodes out entirely because adding all the PCI
ranges to the IMMR node seems 'wrong').  However, it still works
because there are well established conventions, and device drivers
generally stick to the published device binding without making
assumptions about the node layout.  As long as common code parses
'reg' and 'ranges' correctly, then the whole system works.

g.

-- 
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.

Re: PCI bus node location

[Rafal Jaworowski]

On 2009-11-10, at 03:36, Grant Likely wrote:

> On Mon, Nov 9, 2009 at 12:20 PM, Rafal Jaworowski <raj-nYOzD4b6Jr9Wk0Htik3J/w@public.gmane.org>  
> wrote:
>> Hi,
>> I have a couple of questions regarding host/PCI bridges nodes  
>> location:
>>
>> - What is the reason most of the DTS definitions have the host/PCI  
>> bridges
>> hanging off the root node, even though they are most often really  
>> part of
>> the soc?
>> - Is this is some OF heritage (I couldn't find anything explicit  
>> about it in
>> the original PCI bindings docs)?
>> - Is this convention enforced in FDT, or could PCI bus nodes be  
>> children of
>> the soc node as well?
>
> It was a solution to an engineering problem.  The PCI control
> registers are indeed within the IMMR region, and when we first started
> doing PowerPC FDT board ports, the PCI node was a child of the SoC
> node.  However, since PCI is also bridge with its own address space
> translation, having it live in the SoC node causes difficulties.
> Specifically, all of the entries in the PCI node ranges property would
> need similar counterparts in the SoC node ranges property; a scheme
> that doesn't reflect well the actual behaviour of the IMMR region.
>
> Two alternate solutions were proposed.  One was to split the PCI node
> into a PCI bridge node which describes the translations, and a PCI
> control node which describes how to access the PCI bridge registers.
> Some sort of linkage (probably a phandle) would be needed to relate
> the two.  The second was to simply move the PCI node out to be a
> parent of the root.  The second option was the one chosen because it
> was the path of least resistance.  It may not be the most 'correct'
> solution, but it has worked out quite well in practice.  There is
> nothing in the PCI or FDT infrastructure code that enforces this
> convention.  In fact, if the appropriate ranges properties were added
> back to the IMMR node, then the PCI node could become a child of the
> IMMR node without any code changes (but it still wouldn't be a 100%
> 'correct' description of the hardware).

Thanks a lot for the historic perspective and explanations. My concern  
with integrating this model into FreeBSD device scheme abstraction was  
that I could not reflect the hierarchy of DT resources directly,  
because two peer bus entities at the same level (soc, pci) would ask  
for overlapping areas to manage (which shouldn't happen); but I  
believe I got an idea how to resolve this.

The current approach seems a bit of a maintenance problem: the PCI  
bridges control reg need to specify the whole address instead of just  
an offset, which is more error prone in case of changes (when a number  
of places need to be adjusted etc.). What would need to be done/ 
extended for the ranges prop you mention to allow for better handling  
cases like this?

Rafal

Re: PCI bus node location

[David Gibson]

On Tue, Nov 10, 2009 at 05:26:47PM +0100, Rafal Jaworowski wrote:
> On 2009-11-10, at 03:36, Grant Likely wrote:
> >On Mon, Nov 9, 2009 at 12:20 PM, Rafal Jaworowski
> ><raj-nYOzD4b6Jr9Wk0Htik3J/w@public.gmane.org> wrote:
[snip]
> Thanks a lot for the historic perspective and explanations. My
> concern with integrating this model into FreeBSD device scheme
> abstraction was that I could not reflect the hierarchy of DT
> resources directly, because two peer bus entities at the same level
> (soc, pci) would ask for overlapping areas to manage (which
> shouldn't happen); but I believe I got an idea how to resolve this.

Well, yeah.  Things shouldn't overlap, but then again, there are
plenty of imperfect device trees out there.  Regardless of this
specific case, this is probably a situation you'll need a hack to deal
with at some point.

> The current approach seems a bit of a maintenance problem: the PCI
> bridges control reg need to specify the whole address instead of
> just an offset, which is more error prone in case of changes (when a

Well, yes.  And worse, it means there's two places that need to be
adjusted rather than one, if the the IMMR is relocated (which it can
be).  But it's a trade-off of this versus the inconvenience of dealing
with separate "control" and "bridge" nodes for the PCI and following
phandles between them.

> number of places need to be adjusted etc.). What would need to be
> done/extended for the ranges prop you mention to allow for better
> handling cases like this?

I don't really understand the question.  As Grant has said the
"correct" approach is to have one node representing the control
registers - located under the IMMR ("soc") node - and another
representing the PCI host bridge itself (which would be in its present
location).  There would need to be phandles linking the two.  It
doesn't really need any extension to the device tree semantics itself
- just a more complex binding for this device.

Bear in mind with all this that we've been working out conventions for
representing various devices as we go along - and in the early days
nearly everyone was pretty inexperienced with device tree design.  A
number of the bindings that have been established have made less than
ideal choices.  We're getting better, but we're going to have to live
with some of those mistakes.

Dealing with badly designed device tree bindings is pretty icky, but
usually the code to handle it can be reasonably well isolated, so it
doesn't infect too much of the codebase.  Just dealing with ugly
representations when parsing, or having some code which applies fixups
to the initially supplied device tree are both feasible approaches.
But we're never going to reach a place where we always get perfect
device trees, so one way or another, you're going to have to deal with
some uglies.  Our view - borne out by experience so far - is that the
device tree representation is still worth it, despite the problems.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: PCI bus node location

[Rafal Jaworowski]

On 2009-11-11, at 01:05, David Gibson wrote:

>> The current approach seems a bit of a maintenance problem: the PCI
>> bridges control reg need to specify the whole address instead of
>> just an offset, which is more error prone in case of changes (when a
>
> Well, yes.  And worse, it means there's two places that need to be
> adjusted rather than one, if the the IMMR is relocated (which it can
> be).  But it's a trade-off of this versus the inconvenience of dealing
> with separate "control" and "bridge" nodes for the PCI and following
> phandles between them.

Would the technique with additional control node and a phandle  
complicate bindings handling much? The clear benefit is the ability to  
truly reflect hierarchy of devices available within IMMR/CCSR block.

>> number of places need to be adjusted etc.). What would need to be
>> done/extended for the ranges prop you mention to allow for better
>> handling cases like this?
>
> I don't really understand the question.  As Grant has said the
> "correct" approach is to have one node representing the control
> registers - located under the IMMR ("soc") node - and another
> representing the PCI host bridge itself (which would be in its present
> location).  There would need to be phandles linking the two.  It
> doesn't really need any extension to the device tree semantics itself
> - just a more complex binding for this device.

Maybe I misunderstood Grant, my impression was that there was possible  
some 'fixing' of ranges properties (which would be alternative to the  
control node approach).

> Bear in mind with all this that we've been working out conventions for
> representing various devices as we go along - and in the early days
> nearly everyone was pretty inexperienced with device tree design.  A
> number of the bindings that have been established have made less than
> ideal choices.  We're getting better, but we're going to have to live
> with some of those mistakes.
>
> Dealing with badly designed device tree bindings is pretty icky, but
> usually the code to handle it can be reasonably well isolated, so it
> doesn't infect too much of the codebase.  Just dealing with ugly
> representations when parsing, or having some code which applies fixups
> to the initially supplied device tree are both feasible approaches.
> But we're never going to reach a place where we always get perfect
> device trees, so one way or another, you're going to have to deal with
> some uglies.  Our view - borne out by experience so far - is that the
> device tree representation is still worth it, despite the problems.

Don't get me wrong -- I'm just trying to understand what is not clear  
to me in the first place. The other aspect though is that if there are  
areas, which could be improved in the design and implementation why  
not do so, especially long term? I realize the already established  
representations cannot be immediately ditched, but switching  
conventions could be applied over time (e.g. using compatible prop in  
a smart way to have code support legacy approach or some time  after  
which there is a complete switchover etc.).

Please note we are targetting ARM (and other arches in the future)  
besides PowerPC, so if there can be any lessons learnt from previous  
encounters I'd rather embrace them.

Rafal

Re: PCI bus node location

[Scott Wood]

Rafal Jaworowski wrote:
> On 2009-11-11, at 01:05, David Gibson wrote:
>> Well, yes.  And worse, it means there's two places that need to be
>> adjusted rather than one, if the the IMMR is relocated (which it can
>> be).  But it's a trade-off of this versus the inconvenience of dealing
>> with separate "control" and "bridge" nodes for the PCI and following
>> phandles between them.
> 
> Would the technique with additional control node and a phandle 
> complicate bindings handling much? The clear benefit is the ability to 
> truly reflect hierarchy of devices available within IMMR/CCSR block.

It's not very complicated at all, just a few extra lines of code to follow the 
link.  I had initially done pq2 PCI that way, but it was NACKed because it was 
different:

http://lists.ozlabs.org/pipermail/linuxppc-dev/2007-August/041671.html

>> I don't really understand the question.  As Grant has said the
>> "correct" approach is to have one node representing the control
>> registers - located under the IMMR ("soc") node - and another
>> representing the PCI host bridge itself (which would be in its present
>> location).  There would need to be phandles linking the two.  It
>> doesn't really need any extension to the device tree semantics itself
>> - just a more complex binding for this device.
> 
> Maybe I misunderstood Grant, my impression was that there was possible 
> some 'fixing' of ranges properties (which would be alternative to the 
> control node approach).

But that would introduce even more dual maintenance.

-Scott

Re: PCI bus node location

[David Gibson]

On Wed, Nov 11, 2009 at 03:16:56PM +0100, Rafal Jaworowski wrote:
> 
> On 2009-11-11, at 01:05, David Gibson wrote:
> 
> >>The current approach seems a bit of a maintenance problem: the PCI
> >>bridges control reg need to specify the whole address instead of
> >>just an offset, which is more error prone in case of changes (when a
> >
> >Well, yes.  And worse, it means there's two places that need to be
> >adjusted rather than one, if the the IMMR is relocated (which it can
> >be).  But it's a trade-off of this versus the inconvenience of dealing
> >with separate "control" and "bridge" nodes for the PCI and following
> >phandles between them.
> 
> Would the technique with additional control node and a phandle
> complicate bindings handling much? The clear benefit is the ability
> to truly reflect hierarchy of devices available within IMMR/CCSR
> block.
> 
> >>number of places need to be adjusted etc.). What would need to be
> >>done/extended for the ranges prop you mention to allow for better
> >>handling cases like this?
> >
> >I don't really understand the question.  As Grant has said the
> >"correct" approach is to have one node representing the control
> >registers - located under the IMMR ("soc") node - and another
> >representing the PCI host bridge itself (which would be in its present
> >location).  There would need to be phandles linking the two.  It
> >doesn't really need any extension to the device tree semantics itself
> >- just a more complex binding for this device.
> 
> Maybe I misunderstood Grant, my impression was that there was
> possible some 'fixing' of ranges properties (which would be
> alternative to the control node approach).

Well.. the other approach would be for the "soc" node to have, in
addition to a range for the IMMR registers, extra ranges for all the
bridged ranges (PCI or localbus, or whatever).  That has its own
problems because it means either we need a complicated encoding, or
it's less obvious which devices have registers in the IMMR space and
which have registers elsewhere.  We need more complex conventions
about which range is which in the soc node's "ranges" property and so
forth.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: PCI bus node location

[Grant Likely]

On Wed, Nov 11, 2009 at 7:03 PM, David Gibson
<david-xT8FGy+AXnRB3Ne2BGzF6laj5H9X9Tb+@public.gmane.org> wrote:
> On Wed, Nov 11, 2009 at 03:16:56PM +0100, Rafal Jaworowski wrote:
>>
>> On 2009-11-11, at 01:05, David Gibson wrote:
>>
>> >>The current approach seems a bit of a maintenance problem: the PCI
>> >>bridges control reg need to specify the whole address instead of
>> >>just an offset, which is more error prone in case of changes (when a
>> >
>> >Well, yes.  And worse, it means there's two places that need to be
>> >adjusted rather than one, if the the IMMR is relocated (which it can
>> >be).  But it's a trade-off of this versus the inconvenience of dealing
>> >with separate "control" and "bridge" nodes for the PCI and following
>> >phandles between them.
>>
>> Would the technique with additional control node and a phandle
>> complicate bindings handling much? The clear benefit is the ability
>> to truly reflect hierarchy of devices available within IMMR/CCSR
>> block.
>>
>> >>number of places need to be adjusted etc.). What would need to be
>> >>done/extended for the ranges prop you mention to allow for better
>> >>handling cases like this?
>> >
>> >I don't really understand the question.  As Grant has said the
>> >"correct" approach is to have one node representing the control
>> >registers - located under the IMMR ("soc") node - and another
>> >representing the PCI host bridge itself (which would be in its present
>> >location).  There would need to be phandles linking the two.  It
>> >doesn't really need any extension to the device tree semantics itself
>> >- just a more complex binding for this device.
>>
>> Maybe I misunderstood Grant, my impression was that there was
>> possible some 'fixing' of ranges properties (which would be
>> alternative to the control node approach).
>
> Well.. the other approach would be for the "soc" node to have, in
> addition to a range for the IMMR registers, extra ranges for all the
> bridged ranges (PCI or localbus, or whatever).  That has its own
> problems because it means either we need a complicated encoding, or
> it's less obvious which devices have registers in the IMMR space and
> which have registers elsewhere.  We need more complex conventions
> about which range is which in the soc node's "ranges" property and so
> forth.

No, the encoding isn't complex at all.  And I think it become very
obvious which range is getting used for by each 'reg' property.  It
just has the ugliness of having to describe PCI BARs in the IMMR node.
 It would look like this (borrowing heavily from the 5200):

immr@f0000000 {
        #size-cells = <1>;
        #address-cells = <1>;
        ranges = <0 f0000000 0x00010000  # 16k Internal register range
                  80000000 80000000 0x40000000>; # 1GB range for PCI windows
        pci@0xd00 {
                #size-cells = <2>;
                #address-cells = <3>;
                reg = <0xd00 0x100>;
                /* 3 range entries; prefetchable, non-prefetchable, & IO. */
                ranges = <0x42000000 0 0x80000000    0x80000000    0 0x20000000
                          0x02000000 0 0xa0000000    0xa0000000    0 0x10000000
                          0x01000000 0 0x00000000    0xb0000000    0
0x01000000>;
        }
}

Cheers,
g.

-- 
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.

Re: PCI bus node location

[Grant Likely]

On Wed, Nov 11, 2009 at 7:16 AM, Rafal Jaworowski <raj-nYOzD4b6Jr9Wk0Htik3J/w@public.gmane.org> wrote:
>
> On 2009-11-11, at 01:05, David Gibson wrote:
>
>>> The current approach seems a bit of a maintenance problem: the PCI
>>> bridges control reg need to specify the whole address instead of
>>> just an offset, which is more error prone in case of changes (when a
>>
>> Well, yes.  And worse, it means there's two places that need to be
>> adjusted rather than one, if the the IMMR is relocated (which it can
>> be).  But it's a trade-off of this versus the inconvenience of dealing
>> with separate "control" and "bridge" nodes for the PCI and following
>> phandles between them.
>
> Would the technique with additional control node and a phandle complicate
> bindings handling much? The clear benefit is the ability to truly reflect
> hierarchy of devices available within IMMR/CCSR block.

No, it wouldn't be particularly more complex.  It would require a
couple of lines of extra code to follow the phandle to obtain some of
the data.

>>> number of places need to be adjusted etc.). What would need to be
>>> done/extended for the ranges prop you mention to allow for better
>>> handling cases like this?
>>
>> I don't really understand the question.  As Grant has said the
>> "correct" approach is to have one node representing the control
>> registers - located under the IMMR ("soc") node - and another
>> representing the PCI host bridge itself (which would be in its present
>> location).  There would need to be phandles linking the two.  It
>> doesn't really need any extension to the device tree semantics itself
>> - just a more complex binding for this device.
>
> Maybe I misunderstood Grant, my impression was that there was possible some
> 'fixing' of ranges properties (which would be alternative to the control
> node approach).

Yes, the PCI node could be moved back into the IMMR node, but, no, it
wouldn't improve the accuracy of the device tree.  It would just be
different.

However, It sounds like we're debating the wrong thing.  Yes, we
should debate how best to describe hardware, but something you said
earlier in this thread raised a warning flag for me:

>>>> Thanks a lot for the historic perspective and explanations.
>>>> My concern with integrating this model into FreeBSD device
>>>> scheme abstraction was that I could not reflect the hierarchy
>>>> of DT resources directly, because two peer bus entities at the
>>>> same level (soc, pci) would ask for overlapping areas to manage
>>>> (which shouldn't happen); but I believe I got an idea how to
>>>> resolve this.

It *sounds* like you may be depending too heavily on the device tree
for making internal FreeBSD decisions about how to manage your
devices.  But there is a fair bit of expressive latitude in device
tree authorship, and you've got no guarantees that you won't have
things like overlapping ranges between nodes, or 'peer' devices having
completely different parent nodes.  It is risky to model internal
kernel implementation details on device tree structure.

For example, in the early days of device tree work, the of_platform
stuff was created for registering of_devices on the of_bus where there
is a 1:1 relationship between device tree nodes and of_device
instances.  However, in hindsight it was a mistake because of_platform
ends up being a clone of the platform bus, but of_drivers and
platform_drivers are not compatible.  What we should do is extract and
adapt the data in the device tree and convert it to a form usable by
the existing kernel infrastructure.

> Please note we are targetting ARM (and other arches in the future) besides
> PowerPC, so if there can be any lessons learnt from previous encounters I'd
> rather embrace them.

Biggest lesson: document and post bindings for new hardware early;
before merging drivers that use them, and definitely before shipping
equipment.  Common errors are caught easily by review.

g.

-- 
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.