[ARM ATTEND] Describing complex, non-probable system topologies

[ARM ATTEND] Describing complex, non-probable system topologies

[Will Deacon]

Hello,

Whilst Linux implements a bunch of different bus types (many of which
are in fact virtual), devices sitting on non-probable, memory mapped
buses inside SoCs typically live on either the platform_bus or the
amba_bus. So far, this has worked out alright; the buses haven't needed
to be visible to software and no additional software control is really
required from the OS. However, as I/O coherency and hardware
virtualisation capabilities start to creep into ARM-based SoCs, Linux
needs to know the topology of the system on which it is running.

Naturally, this would need to be described as a device-tree binding and
communicate:

  - Buses which can be configured as coherent, including which devices
    on those buses can be made coherent.

  - How IOMMUs sit on the bus and interact with masters on that bus (the
    current one-IOMMU-driver-per-bus may not work well for the
    platform_bus).

  - QoS and PM constraints. This isn't really in my area, but we do have
    buses that have these features and expect software to control them.

  - The system topology and linkages between buses and devices.

The last point is increasingly important as various blocks of ARM system
IP start to require knowledge of masters and how things like memory
traffic, DVM messages, interrupts (think MSI) etc are routed between
them in order to configure the system correctly. For example, interfacing
a PCIe device with an SMMU requires knowledge of both the requester id
associated with the device and how that maps to incoming stream ids
(based off the AXI bus id) on the SMMU. Even worse, this mapping is
likely generated dynamically by the host controller, which would need to
know about downstream buses and their SMMUs.

Other than that, I'd be interested in attending since I'm fairly active
on the architectural side of things and keen to follow any discussions
that may impact core architectural code. Previous ARM mini-summits have been
a great success, so I'm really looking forward to this one.

Cheers,

Will

[ARM ATTEND] Describing complex, non-probable system topologies

[Dave Martin]

On Thu, Aug 01, 2013 at 07:35:31PM +0100, Will Deacon wrote:
> Hello,
> 
> Whilst Linux implements a bunch of different bus types (many of which
> are in fact virtual), devices sitting on non-probable, memory mapped
> buses inside SoCs typically live on either the platform_bus or the
> amba_bus. So far, this has worked out alright; the buses haven't needed
> to be visible to software and no additional software control is really
> required from the OS. However, as I/O coherency and hardware
> virtualisation capabilities start to creep into ARM-based SoCs, Linux
> needs to know the topology of the system on which it is running.
> 
> Naturally, this would need to be described as a device-tree binding and
> communicate:
> 
>   - Buses which can be configured as coherent, including which devices
>     on those buses can be made coherent.
> 
>   - How IOMMUs sit on the bus and interact with masters on that bus (the
>     current one-IOMMU-driver-per-bus may not work well for the
>     platform_bus).
> 
>   - QoS and PM constraints. This isn't really in my area, but we do have
>     buses that have these features and expect software to control them.
> 
>   - The system topology and linkages between buses and devices.
> 
> The last point is increasingly important as various blocks of ARM system
> IP start to require knowledge of masters and how things like memory
> traffic, DVM messages, interrupts (think MSI) etc are routed between
> them in order to configure the system correctly. For example, interfacing
> a PCIe device with an SMMU requires knowledge of both the requester id
> associated with the device and how that maps to incoming stream ids
> (based off the AXI bus id) on the SMMU. Even worse, this mapping is
> likely generated dynamically by the host controller, which would need to
> know about downstream buses and their SMMUs.
> 
> Other than that, I'd be interested in attending since I'm fairly active
> on the architectural side of things and keen to follow any discussions
> that may impact core architectural code. Previous ARM mini-summits have been
> a great success, so I'm really looking forward to this one.

+1

This area of discussion has come up a couple of times before as a biggie
looming on the horizon which we'll have to sort out before every SoC
vendor rolls their own solution ... but it's not sorted yet.

Cheers
---Dave

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[David Brown]

On Thu, Aug 01, 2013 at 07:42:52PM +0100, Dave Martin wrote:

>> Whilst Linux implements a bunch of different bus types (many of which
>> are in fact virtual), devices sitting on non-probable, memory mapped
>> buses inside SoCs typically live on either the platform_bus or the
>> amba_bus. So far, this has worked out alright; the buses haven't needed
>> to be visible to software and no additional software control is really
>> required from the OS. However, as I/O coherency and hardware
>> virtualisation capabilities start to creep into ARM-based SoCs, Linux
>> needs to know the topology of the system on which it is running.
>> ...
>
>This area of discussion has come up a couple of times before as a biggie
>looming on the horizon which we'll have to sort out before every SoC
>vendor rolls their own solution ... but it's not sorted yet.

Sadly, it seems that the Qualcomm out-of-tree code has done something
like this, at least sort of.  There is enough of this described in the
DT for the code to use it, but it isn't really done in a way that
would be generalizable.

I wonder how different what we've got is from what other SoCs are
doing.  Is point-to-point DMA being used anywhere else?

David

-- 
sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
hosted by The Linux Foundation

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Greg KH]

On Thu, Aug 01, 2013 at 07:35:31PM +0100, Will Deacon wrote:
> Hello,
> 
> Whilst Linux implements a bunch of different bus types (many of which
> are in fact virtual), devices sitting on non-probable, memory mapped
> buses inside SoCs typically live on either the platform_bus or the
> amba_bus. So far, this has worked out alright; the buses haven't needed
> to be visible to software and no additional software control is really
> required from the OS. However, as I/O coherency and hardware
> virtualisation capabilities start to creep into ARM-based SoCs, Linux
> needs to know the topology of the system on which it is running.
> 
> Naturally, this would need to be described as a device-tree binding and
> communicate:
> 
>   - Buses which can be configured as coherent, including which devices
>     on those buses can be made coherent.
> 
>   - How IOMMUs sit on the bus and interact with masters on that bus (the
>     current one-IOMMU-driver-per-bus may not work well for the
>     platform_bus).

I've been waiting for people to finally run into this one, and realize
that they shouldn't be using "platform_bus" :)

>   - QoS and PM constraints. This isn't really in my area, but we do have
>     buses that have these features and expect software to control them.
> 
>   - The system topology and linkages between buses and devices.

The driver core handles this really well, you just have to create new
busses, and don't rely on the "catch-all" platform_bus.

> The last point is increasingly important as various blocks of ARM system
> IP start to require knowledge of masters and how things like memory
> traffic, DVM messages, interrupts (think MSI) etc are routed between
> them in order to configure the system correctly. For example, interfacing
> a PCIe device with an SMMU requires knowledge of both the requester id
> associated with the device and how that maps to incoming stream ids
> (based off the AXI bus id) on the SMMU. Even worse, this mapping is
> likely generated dynamically by the host controller, which would need to
> know about downstream buses and their SMMUs.

Hm, sounds like an ACPI tree is what you need to be using :)

Seriously, why not use ACPI for stuff like this?  You already are
starting to do that for ARM-based systems, why not just make it the
standard?

thanks,

greg k-h

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Russell King - ARM Linux]

On Fri, Aug 02, 2013 at 03:27:30AM +0800, Greg KH wrote:
> Hm, sounds like an ACPI tree is what you need to be using :)
> 
> Seriously, why not use ACPI for stuff like this?  You already are
> starting to do that for ARM-based systems, why not just make it the
> standard?

Is this in the same spirit as "you should be using DT, DT can describe
everything you need to do.  It's made to describe bindings between
devices!" and here we are, two years down the line, and we apparantly
don't even have stable DT bindings for the ARM architecture because a
lot of the subsystems which SoCs need have taken that long to get
sorted.

The amount of work this has taken so far has been tremendous, and we're
still working out lots of the details.  For instance, in the last six
months, there's been an effort to try and work out how to sanely
describe how a DMA controller is connected to a peripheral in DT.

Maybe some of those experiences can be applied to ACPI - I doubt that
ACPI has the ability to describe everything that we need to with ARM
SoCs, just like DT was missing a whole bunch of established ways to
describe ARM SoCs when we started looking at it.

One advantage we will have though is having gone through the DT pain,
we're now that much more experienced with describing stuff in firmware,
so hopefully we can avoid some of the DT mistakes with ACPI.

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Greg KH]

On Thu, Aug 01, 2013 at 08:39:36PM +0100, Russell King - ARM Linux wrote:
> On Fri, Aug 02, 2013 at 03:27:30AM +0800, Greg KH wrote:
> > Hm, sounds like an ACPI tree is what you need to be using :)
> > 
> > Seriously, why not use ACPI for stuff like this?  You already are
> > starting to do that for ARM-based systems, why not just make it the
> > standard?
> 
> Is this in the same spirit as "you should be using DT, DT can describe
> everything you need to do.  It's made to describe bindings between
> devices!" and here we are, two years down the line, and we apparantly
> don't even have stable DT bindings for the ARM architecture because a
> lot of the subsystems which SoCs need have taken that long to get
> sorted.

No, it's in the same spirit of, "others have already done this before,
why are you thinking you need to reinvent the wheel?"

> The amount of work this has taken so far has been tremendous, and we're
> still working out lots of the details.  For instance, in the last six
> months, there's been an effort to try and work out how to sanely
> describe how a DMA controller is connected to a peripheral in DT.

But if people think that DT should only be used for the "platform_bus",
then they need to change their minds.  That's what I am referring to
here.

The platform bus was created for all of those little "we know where this
device is" devices.  Now that DT is being used to dynamically create
devices and their interconnections, it should not be used, as it starts
to fall apart as Will points out.

I'm not saying move away from DT at all, if it can be used to describe
stuff like this, wonderful.  Just please don't use platform_bus anymore
than you have to.

thanks,

greg k-h

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Russell King - ARM Linux]

On Fri, Aug 02, 2013 at 04:15:39AM +0800, Greg KH wrote:
> I'm not saying move away from DT at all, if it can be used to describe
> stuff like this, wonderful.  Just please don't use platform_bus anymore
> than you have to.

As far as that sentiment goes, it would have been nice if that was made
more vocally ten years ago, because at that time I was the one trying to
encourage people to think about creating appropriate bus types, and what
I was being told was that no, bus types are something which are deprecated
and platform bus is what should be used.

I get sick and tired of being told "this is what you should do" only to
find out years later that the people saying what should be done have
completely changed their minds - or the maintainers have changed and
they have completely the reverse decision.

We are here with platform_bus precisely becuase that's the way people
working on the ARM architecture were told to do.

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Greg KH]

On Thu, Aug 01, 2013 at 09:18:23PM +0100, Russell King - ARM Linux wrote:
> On Fri, Aug 02, 2013 at 04:15:39AM +0800, Greg KH wrote:
> > I'm not saying move away from DT at all, if it can be used to describe
> > stuff like this, wonderful.  Just please don't use platform_bus anymore
> > than you have to.
> 
> As far as that sentiment goes, it would have been nice if that was made
> more vocally ten years ago, because at that time I was the one trying to
> encourage people to think about creating appropriate bus types, and what
> I was being told was that no, bus types are something which are deprecated
> and platform bus is what should be used.

Was that me that said that?  I don't recall it at all, and if I did, I
was flat out wrong.  I've always said that platform_bus is a hack, and
should only be used as a "last resort".  Others have grabbed onto it as
the "only" way to do devices for embedded things because that is what
they were used to.

sorry,

greg k-h

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Russell King - ARM Linux]

On Fri, Aug 02, 2013 at 04:36:31AM +0800, Greg KH wrote:
> On Thu, Aug 01, 2013 at 09:18:23PM +0100, Russell King - ARM Linux wrote:
> > On Fri, Aug 02, 2013 at 04:15:39AM +0800, Greg KH wrote:
> > > I'm not saying move away from DT at all, if it can be used to describe
> > > stuff like this, wonderful.  Just please don't use platform_bus anymore
> > > than you have to.
> > 
> > As far as that sentiment goes, it would have been nice if that was made
> > more vocally ten years ago, because at that time I was the one trying to
> > encourage people to think about creating appropriate bus types, and what
> > I was being told was that no, bus types are something which are deprecated
> > and platform bus is what should be used.
> 
> Was that me that said that?

I don't remember...

> I don't recall it at all, and if I did, I
> was flat out wrong.  I've always said that platform_bus is a hack, and
> should only be used as a "last resort".  Others have grabbed onto it as
> the "only" way to do devices for embedded things because that is what
> they were used to.

Well, I have three bus types to my name: the amba bus type, the ecard
bus type, and the sa1111 companion chip bus type.

I've been under pressure a number of times to convert the amba_bus
implementation to be a platform_bus, and of course I've refused to do
that because I don't see that platform buses provide what's required
there.  Not only that, but people keep using platform devices/driver
when they create a new driver which should be using the amba bus stuff.

Unfortunately, the message that platform devices should be used is soo
ingrained today that it's going to be really difficult to fight it
without basically refusing everything that comes along.

We also have the new problem (as of the adoption of DT on ARM) that it's
also embedded into the DT representation now, because the platform bus
is the "simple-bus" type in DT, and that's where everyone in ARM land
is placing their on-SoC devices.  That happened because DT already had
a way to create platform devices, and as they were already being used
on ARM, it gave a way to have DT create the platform devices without
any additional effort.

Yes, we can change the kernel code, but that now means that rather than
just changing the entirety of ARM, there's also the impact on PowerPC
to think about too with such a change.

Or we have to rip up our existing DT files and start with a different
approach...

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Greg KH]

On Thu, Aug 01, 2013 at 09:45:06PM +0100, Russell King - ARM Linux wrote:
> On Fri, Aug 02, 2013 at 04:36:31AM +0800, Greg KH wrote:
> > On Thu, Aug 01, 2013 at 09:18:23PM +0100, Russell King - ARM Linux wrote:
> > > On Fri, Aug 02, 2013 at 04:15:39AM +0800, Greg KH wrote:
> > > > I'm not saying move away from DT at all, if it can be used to describe
> > > > stuff like this, wonderful.  Just please don't use platform_bus anymore
> > > > than you have to.
> > > 
> > > As far as that sentiment goes, it would have been nice if that was made
> > > more vocally ten years ago, because at that time I was the one trying to
> > > encourage people to think about creating appropriate bus types, and what
> > > I was being told was that no, bus types are something which are deprecated
> > > and platform bus is what should be used.
> > 
> > Was that me that said that?
> 
> I don't remember...
> 
> > I don't recall it at all, and if I did, I
> > was flat out wrong.  I've always said that platform_bus is a hack, and
> > should only be used as a "last resort".  Others have grabbed onto it as
> > the "only" way to do devices for embedded things because that is what
> > they were used to.
> 
> Well, I have three bus types to my name: the amba bus type, the ecard
> bus type, and the sa1111 companion chip bus type.
> 
> I've been under pressure a number of times to convert the amba_bus
> implementation to be a platform_bus, and of course I've refused to do
> that because I don't see that platform buses provide what's required
> there.  Not only that, but people keep using platform devices/driver
> when they create a new driver which should be using the amba bus stuff.

You are correct, you shouldn't be converting amba_bus, it is fine as-is.

> Unfortunately, the message that platform devices should be used is soo
> ingrained today that it's going to be really difficult to fight it
> without basically refusing everything that comes along.

So, what can I do to combat this?  I have no objection to refusing any
new stuff at all, we do it all the time quite successfully :)

> We also have the new problem (as of the adoption of DT on ARM) that it's
> also embedded into the DT representation now, because the platform bus
> is the "simple-bus" type in DT, and that's where everyone in ARM land
> is placing their on-SoC devices.  That happened because DT already had
> a way to create platform devices, and as they were already being used
> on ARM, it gave a way to have DT create the platform devices without
> any additional effort.
> 
> Yes, we can change the kernel code, but that now means that rather than
> just changing the entirety of ARM, there's also the impact on PowerPC
> to think about too with such a change.

Creating a "DT" bus would place us back at the same spot that the
platform_bus code has today.

And the thing is, DT, just like ACPI, doesn't want to be a bus either.
ACPI is having a bunch of rework done in its driver/bus representation
because of mistakes like this that were done in the past.

There is no reason that any devices represented by DT needs to be on the
platform_bus, just like ACPI, so the two should be totally separate.
It's just that new drivers/subsystems that are added, should NOT
abuse the platform_bus code, when they should be creating their own
busses.

> Or we have to rip up our existing DT files and start with a different
> approach...

No, not at all, this shouldn't be needed.

thanks,

greg k-h

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[James Bottomley]

On Fri, 2013-08-02 at 04:36 +0800, Greg KH wrote:
> On Thu, Aug 01, 2013 at 09:18:23PM +0100, Russell King - ARM Linux wrote:
> > On Fri, Aug 02, 2013 at 04:15:39AM +0800, Greg KH wrote:
> > > I'm not saying move away from DT at all, if it can be used to describe
> > > stuff like this, wonderful.  Just please don't use platform_bus anymore
> > > than you have to.
> > 
> > As far as that sentiment goes, it would have been nice if that was made
> > more vocally ten years ago, because at that time I was the one trying to
> > encourage people to think about creating appropriate bus types, and what
> > I was being told was that no, bus types are something which are deprecated
> > and platform bus is what should be used.
> 
> Was that me that said that?  I don't recall it at all, and if I did, I
> was flat out wrong.  I've always said that platform_bus is a hack, and
> should only be used as a "last resort".  Others have grabbed onto it as
> the "only" way to do devices for embedded things because that is what
> they were used to.

I don't think so.

If you recall ancient history, the reason the generic device DMA model
was created was because on parisc the PCI bus isn't the root of the
device tree and we needed some way to express that properly so the
IOMMUs got programmed correctly.  After dma stuff was added to the
generic device, we pushed all the parisc stuff under a platform specific
parisc bus type.  My understanding of platform_bus was that it's always
been for stuff you couldn't probe properly or classify properly.  I can
see why people would think unprobable SoCs should live on platform_bus,
but no-one ever came to us in parisc and told us to stop using our own
internal bus type.

James

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Mark Brown]

On Thu, Aug 01, 2013 at 02:48:50PM -0700, James Bottomley wrote:

> but no-one ever came to us in parisc and told us to stop using our own
> internal bus type.

As far as I recall the times I've seen the discussion it's been in the
form "this bus implementation is just a sed of the platform bus code,
just the platform bus" which is sensible enough.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130802/38da708f/attachment.sig>

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Tony Lindgren]

* Greg KH <greg@kroah.com> [130801 12:33]:
> On Thu, Aug 01, 2013 at 07:35:31PM +0100, Will Deacon wrote:
> > Hello,
> > 
> > Whilst Linux implements a bunch of different bus types (many of which
> > are in fact virtual), devices sitting on non-probable, memory mapped
> > buses inside SoCs typically live on either the platform_bus or the
> > amba_bus. So far, this has worked out alright; the buses haven't needed
> > to be visible to software and no additional software control is really
> > required from the OS. However, as I/O coherency and hardware
> > virtualisation capabilities start to creep into ARM-based SoCs, Linux
> > needs to know the topology of the system on which it is running.
> > 
> > Naturally, this would need to be described as a device-tree binding and
> > communicate:
> > 
> >   - Buses which can be configured as coherent, including which devices
> >     on those buses can be made coherent.
> > 
> >   - How IOMMUs sit on the bus and interact with masters on that bus (the
> >     current one-IOMMU-driver-per-bus may not work well for the
> >     platform_bus).
> 
> I've been waiting for people to finally run into this one, and realize
> that they shouldn't be using "platform_bus" :)
> 
> >   - QoS and PM constraints. This isn't really in my area, but we do have
> >     buses that have these features and expect software to control them.
> > 
> >   - The system topology and linkages between buses and devices.
> 
> The driver core handles this really well, you just have to create new
> busses, and don't rely on the "catch-all" platform_bus.

Hmm do you have some example of a device driver that is generic and
is supported on platform_bus and some other bus?

For example, we have a need for a custom bus to do bus specific reset
and idling of devices for example but all the device drivers are
dt/platform devices.

Regards,

Tony

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Greg KH]

On Fri, Aug 02, 2013 at 02:03:55AM -0700, Tony Lindgren wrote:
> * Greg KH <greg@kroah.com> [130801 12:33]:
> > On Thu, Aug 01, 2013 at 07:35:31PM +0100, Will Deacon wrote:
> > > Hello,
> > > 
> > > Whilst Linux implements a bunch of different bus types (many of which
> > > are in fact virtual), devices sitting on non-probable, memory mapped
> > > buses inside SoCs typically live on either the platform_bus or the
> > > amba_bus. So far, this has worked out alright; the buses haven't needed
> > > to be visible to software and no additional software control is really
> > > required from the OS. However, as I/O coherency and hardware
> > > virtualisation capabilities start to creep into ARM-based SoCs, Linux
> > > needs to know the topology of the system on which it is running.
> > > 
> > > Naturally, this would need to be described as a device-tree binding and
> > > communicate:
> > > 
> > >   - Buses which can be configured as coherent, including which devices
> > >     on those buses can be made coherent.
> > > 
> > >   - How IOMMUs sit on the bus and interact with masters on that bus (the
> > >     current one-IOMMU-driver-per-bus may not work well for the
> > >     platform_bus).
> > 
> > I've been waiting for people to finally run into this one, and realize
> > that they shouldn't be using "platform_bus" :)
> > 
> > >   - QoS and PM constraints. This isn't really in my area, but we do have
> > >     buses that have these features and expect software to control them.
> > > 
> > >   - The system topology and linkages between buses and devices.
> > 
> > The driver core handles this really well, you just have to create new
> > busses, and don't rely on the "catch-all" platform_bus.
> 
> Hmm do you have some example of a device driver that is generic and
> is supported on platform_bus and some other bus?

Take a look at drivers/usb/host/ohci* for one example that I know of,
there are others all through the kernel as well.

greg k-h

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Tony Lindgren]

* Greg KH <greg@kroah.com> [130802 02:37]:
> On Fri, Aug 02, 2013 at 02:03:55AM -0700, Tony Lindgren wrote:
> > * Greg KH <greg@kroah.com> [130801 12:33]:
> > > On Thu, Aug 01, 2013 at 07:35:31PM +0100, Will Deacon wrote:
> > > > Hello,
> > > > 
> > > > Whilst Linux implements a bunch of different bus types (many of which
> > > > are in fact virtual), devices sitting on non-probable, memory mapped
> > > > buses inside SoCs typically live on either the platform_bus or the
> > > > amba_bus. So far, this has worked out alright; the buses haven't needed
> > > > to be visible to software and no additional software control is really
> > > > required from the OS. However, as I/O coherency and hardware
> > > > virtualisation capabilities start to creep into ARM-based SoCs, Linux
> > > > needs to know the topology of the system on which it is running.
> > > > 
> > > > Naturally, this would need to be described as a device-tree binding and
> > > > communicate:
> > > > 
> > > >   - Buses which can be configured as coherent, including which devices
> > > >     on those buses can be made coherent.
> > > > 
> > > >   - How IOMMUs sit on the bus and interact with masters on that bus (the
> > > >     current one-IOMMU-driver-per-bus may not work well for the
> > > >     platform_bus).
> > > 
> > > I've been waiting for people to finally run into this one, and realize
> > > that they shouldn't be using "platform_bus" :)
> > > 
> > > >   - QoS and PM constraints. This isn't really in my area, but we do have
> > > >     buses that have these features and expect software to control them.
> > > > 
> > > >   - The system topology and linkages between buses and devices.
> > > 
> > > The driver core handles this really well, you just have to create new
> > > busses, and don't rely on the "catch-all" platform_bus.
> > 
> > Hmm do you have some example of a device driver that is generic and
> > is supported on platform_bus and some other bus?
> 
> Take a look at drivers/usb/host/ohci* for one example that I know of,
> there are others all through the kernel as well.

Uhh OK so I guess the answer is that the bus glue still needs to
be implemented separately for each driver and there's no generic
way of supporting multiple busses?

Regards,

Tony

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Greg KH]

On Fri, Aug 02, 2013 at 05:34:45AM -0700, Tony Lindgren wrote:
> * Greg KH <greg@kroah.com> [130802 02:37]:
> > On Fri, Aug 02, 2013 at 02:03:55AM -0700, Tony Lindgren wrote:
> > > * Greg KH <greg@kroah.com> [130801 12:33]:
> > > > On Thu, Aug 01, 2013 at 07:35:31PM +0100, Will Deacon wrote:
> > > > > Hello,
> > > > > 
> > > > > Whilst Linux implements a bunch of different bus types (many of which
> > > > > are in fact virtual), devices sitting on non-probable, memory mapped
> > > > > buses inside SoCs typically live on either the platform_bus or the
> > > > > amba_bus. So far, this has worked out alright; the buses haven't needed
> > > > > to be visible to software and no additional software control is really
> > > > > required from the OS. However, as I/O coherency and hardware
> > > > > virtualisation capabilities start to creep into ARM-based SoCs, Linux
> > > > > needs to know the topology of the system on which it is running.
> > > > > 
> > > > > Naturally, this would need to be described as a device-tree binding and
> > > > > communicate:
> > > > > 
> > > > >   - Buses which can be configured as coherent, including which devices
> > > > >     on those buses can be made coherent.
> > > > > 
> > > > >   - How IOMMUs sit on the bus and interact with masters on that bus (the
> > > > >     current one-IOMMU-driver-per-bus may not work well for the
> > > > >     platform_bus).
> > > > 
> > > > I've been waiting for people to finally run into this one, and realize
> > > > that they shouldn't be using "platform_bus" :)
> > > > 
> > > > >   - QoS and PM constraints. This isn't really in my area, but we do have
> > > > >     buses that have these features and expect software to control them.
> > > > > 
> > > > >   - The system topology and linkages between buses and devices.
> > > > 
> > > > The driver core handles this really well, you just have to create new
> > > > busses, and don't rely on the "catch-all" platform_bus.
> > > 
> > > Hmm do you have some example of a device driver that is generic and
> > > is supported on platform_bus and some other bus?
> > 
> > Take a look at drivers/usb/host/ohci* for one example that I know of,
> > there are others all through the kernel as well.
> 
> Uhh OK so I guess the answer is that the bus glue still needs to
> be implemented separately for each driver and there's no generic
> way of supporting multiple busses?

Different busses implies that there are different ways to physically
talk to the device hardware, so of course there is no generic way to
support that.

Unless your subsystem wants to do what we did for USB, and define a
generic way to talk to the hardware in a very abstract way, allowing for
totally different types of physical layers to handle lots of different
logical layer drivers.  But odds are, you don't want to do that.

It should be easy to just abstract out your "this is how I write a byte
to the hardware" logic to handle the different bus types, if you really
are using the same chip/core.  Lots of drivers do this today just fine,
it isn't a big deal.

thanks,

greg k-h

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Dave Martin]

On Fri, Aug 02, 2013 at 10:14:49PM +0800, Greg KH wrote:
> On Fri, Aug 02, 2013 at 05:34:45AM -0700, Tony Lindgren wrote:
> > * Greg KH <greg@kroah.com> [130802 02:37]:
> > > On Fri, Aug 02, 2013 at 02:03:55AM -0700, Tony Lindgren wrote:
> > > > * Greg KH <greg@kroah.com> [130801 12:33]:
> > > > > On Thu, Aug 01, 2013 at 07:35:31PM +0100, Will Deacon wrote:
> > > > > > Hello,
> > > > > > 
> > > > > > Whilst Linux implements a bunch of different bus types (many of which
> > > > > > are in fact virtual), devices sitting on non-probable, memory mapped
> > > > > > buses inside SoCs typically live on either the platform_bus or the
> > > > > > amba_bus. So far, this has worked out alright; the buses haven't needed
> > > > > > to be visible to software and no additional software control is really
> > > > > > required from the OS. However, as I/O coherency and hardware
> > > > > > virtualisation capabilities start to creep into ARM-based SoCs, Linux
> > > > > > needs to know the topology of the system on which it is running.
> > > > > > 
> > > > > > Naturally, this would need to be described as a device-tree binding and
> > > > > > communicate:
> > > > > > 
> > > > > >   - Buses which can be configured as coherent, including which devices
> > > > > >     on those buses can be made coherent.
> > > > > > 
> > > > > >   - How IOMMUs sit on the bus and interact with masters on that bus (the
> > > > > >     current one-IOMMU-driver-per-bus may not work well for the
> > > > > >     platform_bus).
> > > > > 
> > > > > I've been waiting for people to finally run into this one, and realize
> > > > > that they shouldn't be using "platform_bus" :)
> > > > > 
> > > > > >   - QoS and PM constraints. This isn't really in my area, but we do have
> > > > > >     buses that have these features and expect software to control them.
> > > > > > 
> > > > > >   - The system topology and linkages between buses and devices.
> > > > > 
> > > > > The driver core handles this really well, you just have to create new
> > > > > busses, and don't rely on the "catch-all" platform_bus.
> > > > 
> > > > Hmm do you have some example of a device driver that is generic and
> > > > is supported on platform_bus and some other bus?
> > > 
> > > Take a look at drivers/usb/host/ohci* for one example that I know of,
> > > there are others all through the kernel as well.
> > 
> > Uhh OK so I guess the answer is that the bus glue still needs to
> > be implemented separately for each driver and there's no generic
> > way of supporting multiple busses?
> 
> Different busses implies that there are different ways to physically
> talk to the device hardware, so of course there is no generic way to
> support that.
> 
> Unless your subsystem wants to do what we did for USB, and define a
> generic way to talk to the hardware in a very abstract way, allowing for
> totally different types of physical layers to handle lots of different
> logical layer drivers.  But odds are, you don't want to do that.
> 
> It should be easy to just abstract out your "this is how I write a byte
> to the hardware" logic to handle the different bus types, if you really
> are using the same chip/core.  Lots of drivers do this today just fine,
> it isn't a big deal.

I think the focus here is on high-performance memory-mapped buses, since
that's where a lot of the complexity lies in a SoC.  Once the bus is
configured, all buses look much the same at the "how to write a byte"
level.

The pain points are how the buses are configured, what their capabilities
are, and how the view of the system (coherency, and addressing and visibility
of slaves) varies between different bus masters.  These are key things which
may vary per bus.  DT at least has no real way of describing such things
right now.

Unfortunately, hardware vendors have a lot of freedom about the on-SoC
bus architecture, and they do use it.


I think we've mostly escaped from this in the past, since in the past, bus
topologies have tended to be simpler, and the kernel does less active
management of the hardware -- with a bit of luck, the boot ROM or
bootloader sets them up in a sane state, and then Linux can just treat
those buses as generic memory-mapped IO.

But with current and future systems, we're starting to need to care
about dynamic control of bus configuration, particularly for things like
power management and multi-master scenarios.


Cheers
---Dave

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Will Deacon]

On Fri, Aug 02, 2013 at 04:26:48PM +0100, Dave Martin wrote:
> On Fri, Aug 02, 2013 at 10:14:49PM +0800, Greg KH wrote:
> > It should be easy to just abstract out your "this is how I write a byte
> > to the hardware" logic to handle the different bus types, if you really
> > are using the same chip/core.  Lots of drivers do this today just fine,
> > it isn't a big deal.
> 
> I think the focus here is on high-performance memory-mapped buses, since
> that's where a lot of the complexity lies in a SoC.  Once the bus is
> configured, all buses look much the same at the "how to write a byte"
> level.
> 
> The pain points are how the buses are configured, what their capabilities
> are, and how the view of the system (coherency, and addressing and visibility
> of slaves) varies between different bus masters.  These are key things which
> may vary per bus.  DT at least has no real way of describing such things
> right now.

And a simple, concrete example of this is the CCI (Cache Coherent
Interconnect) bus driver that was merged recently (drivers/bus/arm-cci.c).

The CCI supports device-coherency, which the driver exposes via a global
__cci_control_port_by_device symbol. Guess how many callers we have of that
function? Anyway, even if we did have a caller, that function then relies on
CCI-specific properties on the device being made coherent to figure out what
to do, so there's no new bus_type at all -- the linkages are all implicit in
the device-tree binding.

The CCI also has QoS and performance monitoring capabilities exposed in
memory-mapped registers, which aren't touched by the driver (although I have
seen patches for the latter).

Will

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Tony Lindgren]

* Greg KH <greg@kroah.com> [130802 07:20]:
> On Fri, Aug 02, 2013 at 05:34:45AM -0700, Tony Lindgren wrote:
> > * Greg KH <greg@kroah.com> [130802 02:37]:
> > > On Fri, Aug 02, 2013 at 02:03:55AM -0700, Tony Lindgren wrote:
> > > > * Greg KH <greg@kroah.com> [130801 12:33]:
> > > > > 
> > > > > The driver core handles this really well, you just have to create new
> > > > > busses, and don't rely on the "catch-all" platform_bus.
> > > > 
> > > > Hmm do you have some example of a device driver that is generic and
> > > > is supported on platform_bus and some other bus?
> > > 
> > > Take a look at drivers/usb/host/ohci* for one example that I know of,
> > > there are others all through the kernel as well.
> > 
> > Uhh OK so I guess the answer is that the bus glue still needs to
> > be implemented separately for each driver and there's no generic
> > way of supporting multiple busses?
> 
> Different busses implies that there are different ways to physically
> talk to the device hardware, so of course there is no generic way to
> support that.

Right, but in many cases the only difference between SoCs are some bus
specific things like reset and idling of the devices. And using platform
bus allows the driver to stay generic.

> Unless your subsystem wants to do what we did for USB, and define a
> generic way to talk to the hardware in a very abstract way, allowing for
> totally different types of physical layers to handle lots of different
> logical layer drivers.  But odds are, you don't want to do that.
> 
> It should be easy to just abstract out your "this is how I write a byte
> to the hardware" logic to handle the different bus types, if you really
> are using the same chip/core.  Lots of drivers do this today just fine,
> it isn't a big deal.

I agree that makes sense for the driver specific things.

But for things that are completely bus specific for various SoCs, how
would you like to handle those?

For example, we are currently using platform bus and bus notifiers and
then the runtime PM calls from device driver trigger the bus specific
things.

Would you prefer to instead use a custom bus instead of extending
the platform bus for things like that?

Regards,

Tony

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Greg KH]

On Sun, Aug 04, 2013 at 11:55:35PM -0700, Tony Lindgren wrote:
> * Greg KH <greg@kroah.com> [130802 07:20]:
> > On Fri, Aug 02, 2013 at 05:34:45AM -0700, Tony Lindgren wrote:
> > > * Greg KH <greg@kroah.com> [130802 02:37]:
> > > > On Fri, Aug 02, 2013 at 02:03:55AM -0700, Tony Lindgren wrote:
> > > > > * Greg KH <greg@kroah.com> [130801 12:33]:
> > > > > > 
> > > > > > The driver core handles this really well, you just have to create new
> > > > > > busses, and don't rely on the "catch-all" platform_bus.
> > > > > 
> > > > > Hmm do you have some example of a device driver that is generic and
> > > > > is supported on platform_bus and some other bus?
> > > > 
> > > > Take a look at drivers/usb/host/ohci* for one example that I know of,
> > > > there are others all through the kernel as well.
> > > 
> > > Uhh OK so I guess the answer is that the bus glue still needs to
> > > be implemented separately for each driver and there's no generic
> > > way of supporting multiple busses?
> > 
> > Different busses implies that there are different ways to physically
> > talk to the device hardware, so of course there is no generic way to
> > support that.
> 
> Right, but in many cases the only difference between SoCs are some bus
> specific things like reset and idling of the devices. And using platform
> bus allows the driver to stay generic.

Just because it "is there", doesn't mean you should abuse it :)

> > Unless your subsystem wants to do what we did for USB, and define a
> > generic way to talk to the hardware in a very abstract way, allowing for
> > totally different types of physical layers to handle lots of different
> > logical layer drivers.  But odds are, you don't want to do that.
> > 
> > It should be easy to just abstract out your "this is how I write a byte
> > to the hardware" logic to handle the different bus types, if you really
> > are using the same chip/core.  Lots of drivers do this today just fine,
> > it isn't a big deal.
> 
> I agree that makes sense for the driver specific things.
> 
> But for things that are completely bus specific for various SoCs, how
> would you like to handle those?
> 
> For example, we are currently using platform bus and bus notifiers and
> then the runtime PM calls from device driver trigger the bus specific
> things.
> 
> Would you prefer to instead use a custom bus instead of extending
> the platform bus for things like that?

Yes I would.  I would really like to only use the platform bus for very
few things, if any at all.

thanks,

greg k-h

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Tony Lindgren]

* Greg KH <greg@kroah.com> [130805 00:16]:
> On Sun, Aug 04, 2013 at 11:55:35PM -0700, Tony Lindgren wrote:
> > 
> > But for things that are completely bus specific for various SoCs, how
> > would you like to handle those?
> > 
> > For example, we are currently using platform bus and bus notifiers and
> > then the runtime PM calls from device driver trigger the bus specific
> > things.
> > 
> > Would you prefer to instead use a custom bus instead of extending
> > the platform bus for things like that?
> 
> Yes I would.  I would really like to only use the platform bus for very
> few things, if any at all.

OK. How would you prefer to set up things from driver point of view
so the device drivers don't need to care which bus it's connected to?

That is, for the let's say 10 - 15 slightly different types of busses
that are currently handled as platform bus?

Should we have some generic replacement for struct platform_driver
that can bind to let's say SoC specific bus implementation?

Regards,

Tony

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Greg KH]

On Mon, Aug 05, 2013 at 12:37:30AM -0700, Tony Lindgren wrote:
> * Greg KH <greg@kroah.com> [130805 00:16]:
> > On Sun, Aug 04, 2013 at 11:55:35PM -0700, Tony Lindgren wrote:
> > > 
> > > But for things that are completely bus specific for various SoCs, how
> > > would you like to handle those?
> > > 
> > > For example, we are currently using platform bus and bus notifiers and
> > > then the runtime PM calls from device driver trigger the bus specific
> > > things.
> > > 
> > > Would you prefer to instead use a custom bus instead of extending
> > > the platform bus for things like that?
> > 
> > Yes I would.  I would really like to only use the platform bus for very
> > few things, if any at all.
> 
> OK. How would you prefer to set up things from driver point of view
> so the device drivers don't need to care which bus it's connected to?

What do you mean by "don't need to care"?  How are these drivers talking
to the device on the bus in the first place?  If these are different
busses, then they are talked to in different ways, right?

Any specific examples you have to point to in the kernel today?

> That is, for the let's say 10 - 15 slightly different types of busses
> that are currently handled as platform bus?

What makes them "different"?

> Should we have some generic replacement for struct platform_driver
> that can bind to let's say SoC specific bus implementation?

Hm, like perhaps, "struct driver"?  :)

But again, do you have a specific example I can look at to get an idea
of what you are talking about?

thanks,

greg k-h

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Tony Lindgren]

* Greg KH <greg@kroah.com> [130805 01:08]:
> On Mon, Aug 05, 2013 at 12:37:30AM -0700, Tony Lindgren wrote:
> > * Greg KH <greg@kroah.com> [130805 00:16]:
> > > On Sun, Aug 04, 2013 at 11:55:35PM -0700, Tony Lindgren wrote:
> > > > 
> > > > But for things that are completely bus specific for various SoCs, how
> > > > would you like to handle those?
> > > > 
> > > > For example, we are currently using platform bus and bus notifiers and
> > > > then the runtime PM calls from device driver trigger the bus specific
> > > > things.
> > > > 
> > > > Would you prefer to instead use a custom bus instead of extending
> > > > the platform bus for things like that?
> > > 
> > > Yes I would.  I would really like to only use the platform bus for very
> > > few things, if any at all.
> > 
> > OK. How would you prefer to set up things from driver point of view
> > so the device drivers don't need to care which bus it's connected to?
> 
> What do you mean by "don't need to care"?  How are these drivers talking
> to the device on the bus in the first place?  If these are different
> busses, then they are talked to in different ways, right?

Let's assume that just ioremap + read/write is needed.
 
> Any specific examples you have to point to in the kernel today?

The one I'm struggling with is the _omap_device_notifier_call
that's not currently doing much, but we've been trying to find
a clean way to implement runtime PM calls for the bus.

There are other examples doing notifiers with platform_bus, have
not checked but I'm guessing they have similar needs:

$ git grep bus_register_notifier | grep platform_bus_type
arch/arm/mach-exynos/pm_domains.c:      bus_register_notifier(&platform_bus_type, &platform_nb);
arch/arm/mach-highbank/highbank.c:      bus_register_notifier(&platform_bus_type, &highbank_platform_nb);
arch/arm/mach-mvebu/coherency.c:                bus_register_notifier(&platform_bus_type,
arch/arm/mach-omap2/omap_device.c:      bus_register_notifier(&platform_bus_type, &platform_nb);
arch/microblaze/kernel/setup.c: bus_register_notifier(&platform_bus_type, &dflt_plat_bus_notifier);
arch/powerpc/kernel/dma-swiotlb.c:      bus_register_notifier(&platform_bus_type,
arch/powerpc/platforms/cell/beat_iommu.c:       bus_register_notifier(&platform_bus_type, &celleb_of_bus_notifier);
arch/powerpc/platforms/cell/iommu.c:    bus_register_notifier(&platform_bus_type, &cell_of_bus_notifier);
drivers/acpi/acpi_lpss.c:               bus_register_notifier(&platform_bus_type, &acpi_lpss_nb);
drivers/media/platform/soc_camera/sh_mobile_ceu_camera.c:               err = bus_register_notifier(&platform_bus_type, &wait.notifier);
drivers/net/ethernet/ibm/emac/core.c:   bus_register_notifier(&platform_bus_type, &emac_of_bus_notifier);
 
> > That is, for the let's say 10 - 15 slightly different types of busses
> > that are currently handled as platform bus?
> 
> What makes them "different"?

How the power and clock domains are done and how the clocks are gated
or idled. So basically how the devices are reset and idled at the bus
level.
 
> > Should we have some generic replacement for struct platform_driver
> > that can bind to let's say SoC specific bus implementation?
> 
> Hm, like perhaps, "struct driver"?  :)

Sure :)
 
> But again, do you have a specific example I can look at to get an idea
> of what you are talking about?

Hopefully the above gives you some idea what needs to be done
at the bus level. Let's just assume bus specific reset and idle
of devices, that should already get people pretty far with runtime
PM.

Regards,

Tony

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Greg KH]

On Mon, Aug 05, 2013 at 01:21:38AM -0700, Tony Lindgren wrote:
> * Greg KH <greg@kroah.com> [130805 01:08]:
> > On Mon, Aug 05, 2013 at 12:37:30AM -0700, Tony Lindgren wrote:
> > > * Greg KH <greg@kroah.com> [130805 00:16]:
> > > > On Sun, Aug 04, 2013 at 11:55:35PM -0700, Tony Lindgren wrote:
> > > > > 
> > > > > But for things that are completely bus specific for various SoCs, how
> > > > > would you like to handle those?
> > > > > 
> > > > > For example, we are currently using platform bus and bus notifiers and
> > > > > then the runtime PM calls from device driver trigger the bus specific
> > > > > things.
> > > > > 
> > > > > Would you prefer to instead use a custom bus instead of extending
> > > > > the platform bus for things like that?
> > > > 
> > > > Yes I would.  I would really like to only use the platform bus for very
> > > > few things, if any at all.
> > > 
> > > OK. How would you prefer to set up things from driver point of view
> > > so the device drivers don't need to care which bus it's connected to?
> > 
> > What do you mean by "don't need to care"?  How are these drivers talking
> > to the device on the bus in the first place?  If these are different
> > busses, then they are talked to in different ways, right?
> 
> Let's assume that just ioremap + read/write is needed.

That implies that there is no "bus" at all involved here, so what's the
problem?  :)

> > Any specific examples you have to point to in the kernel today?
> 
> The one I'm struggling with is the _omap_device_notifier_call
> that's not currently doing much, but we've been trying to find
> a clean way to implement runtime PM calls for the bus.
> 
> There are other examples doing notifiers with platform_bus, have
> not checked but I'm guessing they have similar needs:
> 
> $ git grep bus_register_notifier | grep platform_bus_type
> arch/arm/mach-exynos/pm_domains.c:      bus_register_notifier(&platform_bus_type, &platform_nb);
> arch/arm/mach-highbank/highbank.c:      bus_register_notifier(&platform_bus_type, &highbank_platform_nb);
> arch/arm/mach-mvebu/coherency.c:                bus_register_notifier(&platform_bus_type,
> arch/arm/mach-omap2/omap_device.c:      bus_register_notifier(&platform_bus_type, &platform_nb);
> arch/microblaze/kernel/setup.c: bus_register_notifier(&platform_bus_type, &dflt_plat_bus_notifier);
> arch/powerpc/kernel/dma-swiotlb.c:      bus_register_notifier(&platform_bus_type,
> arch/powerpc/platforms/cell/beat_iommu.c:       bus_register_notifier(&platform_bus_type, &celleb_of_bus_notifier);
> arch/powerpc/platforms/cell/iommu.c:    bus_register_notifier(&platform_bus_type, &cell_of_bus_notifier);
> drivers/acpi/acpi_lpss.c:               bus_register_notifier(&platform_bus_type, &acpi_lpss_nb);
> drivers/media/platform/soc_camera/sh_mobile_ceu_camera.c:               err = bus_register_notifier(&platform_bus_type, &wait.notifier);
> drivers/net/ethernet/ibm/emac/core.c:   bus_register_notifier(&platform_bus_type, &emac_of_bus_notifier);
>  
> > > That is, for the let's say 10 - 15 slightly different types of busses
> > > that are currently handled as platform bus?
> > 
> > What makes them "different"?
> 
> How the power and clock domains are done and how the clocks are gated
> or idled. So basically how the devices are reset and idled at the bus
> level.

I think of a "bus" as the way that a device talks to the hardware (i.e.
PCI, USB, i2c, spi, etc.)  You are saying a "bus" is something
different, something that is used to control the "raw" devices that just
do iowrites.

Why not just create a bus for your devices, have them register platform
devices (so you can take advantage of the existing drivers) and have
your own "platform bus" of a specific type?  The code to do that would
only need to be created once "per bus", and that way you can handle all
of the needed reset/idle stuff properly for things "attached" to it.

Perhaps we need to get in front of a whiteboard at the ARM mini summit
and hash this all out...

thanks,

greg k-h

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Tony Lindgren]

* Greg KH <greg@kroah.com> [130805 01:56]:
> On Mon, Aug 05, 2013 at 01:21:38AM -0700, Tony Lindgren wrote:
> > * Greg KH <greg@kroah.com> [130805 01:08]:
> > > On Mon, Aug 05, 2013 at 12:37:30AM -0700, Tony Lindgren wrote:
> > > > * Greg KH <greg@kroah.com> [130805 00:16]:
> > > > > On Sun, Aug 04, 2013 at 11:55:35PM -0700, Tony Lindgren wrote:
> > > > > > 
> > > > > > But for things that are completely bus specific for various SoCs, how
> > > > > > would you like to handle those?
> > > > > > 
> > > > > > For example, we are currently using platform bus and bus notifiers and
> > > > > > then the runtime PM calls from device driver trigger the bus specific
> > > > > > things.
> > > > > > 
> > > > > > Would you prefer to instead use a custom bus instead of extending
> > > > > > the platform bus for things like that?
> > > > > 
> > > > > Yes I would.  I would really like to only use the platform bus for very
> > > > > few things, if any at all.
> > > > 
> > > > OK. How would you prefer to set up things from driver point of view
> > > > so the device drivers don't need to care which bus it's connected to?
> > > 
> > > What do you mean by "don't need to care"?  How are these drivers talking
> > > to the device on the bus in the first place?  If these are different
> > > busses, then they are talked to in different ways, right?
> > 
> > Let's assume that just ioremap + read/write is needed.
> 
> That implies that there is no "bus" at all involved here, so what's the
> problem?  :)

Right, it's all mem mapped. But there can be timings and partitioning
of the area that the "bus" needs to set up.
 
> > > Any specific examples you have to point to in the kernel today?
> > 
> > The one I'm struggling with is the _omap_device_notifier_call
> > that's not currently doing much, but we've been trying to find
> > a clean way to implement runtime PM calls for the bus.
> > 
> > There are other examples doing notifiers with platform_bus, have
> > not checked but I'm guessing they have similar needs:
> > 
> > $ git grep bus_register_notifier | grep platform_bus_type
> > arch/arm/mach-exynos/pm_domains.c:      bus_register_notifier(&platform_bus_type, &platform_nb);
> > arch/arm/mach-highbank/highbank.c:      bus_register_notifier(&platform_bus_type, &highbank_platform_nb);
> > arch/arm/mach-mvebu/coherency.c:                bus_register_notifier(&platform_bus_type,
> > arch/arm/mach-omap2/omap_device.c:      bus_register_notifier(&platform_bus_type, &platform_nb);
> > arch/microblaze/kernel/setup.c: bus_register_notifier(&platform_bus_type, &dflt_plat_bus_notifier);
> > arch/powerpc/kernel/dma-swiotlb.c:      bus_register_notifier(&platform_bus_type,
> > arch/powerpc/platforms/cell/beat_iommu.c:       bus_register_notifier(&platform_bus_type, &celleb_of_bus_notifier);
> > arch/powerpc/platforms/cell/iommu.c:    bus_register_notifier(&platform_bus_type, &cell_of_bus_notifier);
> > drivers/acpi/acpi_lpss.c:               bus_register_notifier(&platform_bus_type, &acpi_lpss_nb);
> > drivers/media/platform/soc_camera/sh_mobile_ceu_camera.c:               err = bus_register_notifier(&platform_bus_type, &wait.notifier);
> > drivers/net/ethernet/ibm/emac/core.c:   bus_register_notifier(&platform_bus_type, &emac_of_bus_notifier);
> >  
> > > > That is, for the let's say 10 - 15 slightly different types of busses
> > > > that are currently handled as platform bus?
> > > 
> > > What makes them "different"?
> > 
> > How the power and clock domains are done and how the clocks are gated
> > or idled. So basically how the devices are reset and idled at the bus
> > level.
> 
> I think of a "bus" as the way that a device talks to the hardware (i.e.
> PCI, USB, i2c, spi, etc.)  You are saying a "bus" is something
> different, something that is used to control the "raw" devices that just
> do iowrites.

Yeah. But usually the SoC specific implementation of the above
bus hardware is itself on a mem mapped bus.. That's why we've been using
platform bus.
 
> Why not just create a bus for your devices, have them register platform
> devices (so you can take advantage of the existing drivers) and have
> your own "platform bus" of a specific type?  The code to do that would
> only need to be created once "per bus", and that way you can handle all
> of the needed reset/idle stuff properly for things "attached" to it.

Hmm yes so do you have an example of such a thing?
 
> Perhaps we need to get in front of a whiteboard at the ARM mini summit
> and hash this all out...

Sure that would probably help quite a bit.

Regards,

Tony

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Kevin Hilman]

Greg KH <greg@kroah.com> writes:

> On Mon, Aug 05, 2013 at 01:21:38AM -0700, Tony Lindgren wrote:

[...]

>> 
>> How the power and clock domains are done and how the clocks are gated
>> or idled. So basically how the devices are reset and idled at the bus
>> level.
>
> I think of a "bus" as the way that a device talks to the hardware (i.e.
> PCI, USB, i2c, spi, etc.)  You are saying a "bus" is something
> different, something that is used to control the "raw" devices that just
> do iowrites.
>
> Why not just create a bus for your devices, have them register platform
> devices (so you can take advantage of the existing drivers) and have
> your own "platform bus" of a specific type?  The code to do that would
> only need to be created once "per bus", and that way you can handle all
> of the needed reset/idle stuff properly for things "attached" to it.

Each time we've looked at doing this (when pre-cursors to runtime PM
were being explored, for example), what you end up with is essentially
with a copy of drivers/base/platform.c with a few lines of additional
code.  So each time this has happened, the outcome has been that the
platform_bus has just extended slightly with the new features to avoid
the massive duplication.

This has followed the "extend instead of duplicate" philosophy we
generally follow, but the question now is whether we've finally extended
the platform_bus to it's breaking point.  (I don't think we don't have
to guess what your answer is.) :)

At least for the PM stuff (including reset/idle), I don't think we need
a new bus type.  We now have PM domains which IMO should be used for
this stuff.  PM domains allow arbitrary groupings of (independent of
underlying bus type) that have common PM operations (read struct
dev_pm_ops.)  For these things, it's probabably PM domains (along with
runtime PM/dev_pm_ops) that should be extended for some of the PM
features Tony is referring to.  Then the driver core (not the bus) might
need to grow some new operations to handle new items in dev_pm_ops.

While I think that could address the device/bus specific PM related
operations without the need for a new bus type, it doesn't really solve
the bigger forward-looking features Will is raising.

> Perhaps we need to get in front of a whiteboard at the ARM mini summit
> and hash this all out...

Agreed, this will be a great topic for the ARM mini summit.

Kevin

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Will Deacon]

Hi Greg,

On Thu, Aug 01, 2013 at 08:27:30PM +0100, Greg KH wrote:
> On Thu, Aug 01, 2013 at 07:35:31PM +0100, Will Deacon wrote:
> > Naturally, this would need to be described as a device-tree binding and
> > communicate:
> > 
> >   - Buses which can be configured as coherent, including which devices
> >     on those buses can be made coherent.
> > 
> >   - How IOMMUs sit on the bus and interact with masters on that bus (the
> >     current one-IOMMU-driver-per-bus may not work well for the
> >     platform_bus).
> 
> I've been waiting for people to finally run into this one, and realize
> that they shouldn't be using "platform_bus" :)

But, as pointed out later in this thread, people have been doing the exact
opposite! We can change the mindset by yelling, but if you're writing a new
driver for a peripheral on an ARM SoC, platform_bus is mighty tempting
because you get a bunch of device-tree parsing code for free (see
drivers/of/platform.c).

What's worse is that this nice-and-easy auto-probing doesn't work for nested
device-nodes (i.e. a bunch of device-nodes under a common parent, something
which you might think is pretty common in a `tree') so people shy away from
nesting as a means to group devices too.

> >   - QoS and PM constraints. This isn't really in my area, but we do have
> >     buses that have these features and expect software to control them.
> > 
> >   - The system topology and linkages between buses and devices.
> 
> The driver core handles this really well, you just have to create new
> busses, and don't rely on the "catch-all" platform_bus.

Agreed, it's time that we started to describe these non-probable buses as
separate bus_types, with controller logic for configuring the bus itself
(there are weird-and-wonderful ring-based designs on the horizon which can
require a fair amount of setup).

> > The last point is increasingly important as various blocks of ARM system
> > IP start to require knowledge of masters and how things like memory
> > traffic, DVM messages, interrupts (think MSI) etc are routed between
> > them in order to configure the system correctly. For example, interfacing
> > a PCIe device with an SMMU requires knowledge of both the requester id
> > associated with the device and how that maps to incoming stream ids
> > (based off the AXI bus id) on the SMMU. Even worse, this mapping is
> > likely generated dynamically by the host controller, which would need to
> > know about downstream buses and their SMMUs.
> 
> Hm, sounds like an ACPI tree is what you need to be using :)
> 
> Seriously, why not use ACPI for stuff like this?  You already are
> starting to do that for ARM-based systems, why not just make it the
> standard?

So, like a good proportion of the ARM community, ACPI isn't something I'm
well-versed in. Yes, it's coming, but at the same time it's not going to be
everywhere and we need to continue to support new SoCs using device-tree.
Whilst it might even become a de-factor standard for servers, mobile devices
will likely continue with the bootloaders they currently have. Furthermore,
the mobile space is really the wild-west when it comes to system topology --
exynos SoCs tend to have one IOMMU per device, for example:

http://lists.infradead.org/pipermail/linux-arm-kernel/2013-July/181922.html

On the back of that, how does ACPI describe these relationships? It would
certainly be a good idea to see what's already being done so we don't
reinvent everything again for device-tree.

Will

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Tony Lindgren]

* Will Deacon <will.deacon@arm.com> [130802 05:00]:
> On Thu, Aug 01, 2013 at 08:27:30PM +0100, Greg KH wrote:
> > 
> > The driver core handles this really well, you just have to create new
> > busses, and don't rely on the "catch-all" platform_bus.
> 
> Agreed, it's time that we started to describe these non-probable buses as
> separate bus_types, with controller logic for configuring the bus itself
> (there are weird-and-wonderful ring-based designs on the horizon which can
> require a fair amount of setup).

Yes we somehow need hardware specific buses, but they should appear
generic to the device drivers without having to modify each device
driver using these buses for each bus.

Regards,

Tony

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Greg KH]

On Fri, Aug 02, 2013 at 05:37:52AM -0700, Tony Lindgren wrote:
> * Will Deacon <will.deacon@arm.com> [130802 05:00]:
> > On Thu, Aug 01, 2013 at 08:27:30PM +0100, Greg KH wrote:
> > > 
> > > The driver core handles this really well, you just have to create new
> > > busses, and don't rely on the "catch-all" platform_bus.
> > 
> > Agreed, it's time that we started to describe these non-probable buses as
> > separate bus_types, with controller logic for configuring the bus itself
> > (there are weird-and-wonderful ring-based designs on the horizon which can
> > require a fair amount of setup).
> 
> Yes we somehow need hardware specific buses, but they should appear
> generic to the device drivers without having to modify each device
> driver using these buses for each bus.

I just wrote how to do that, and there are lots of examples of it in the
kernel already from simple, "if this chip then write this way" to
complex, "here's a packet, send it to the hardware" levels.  It all
depends on how much work you want to do.

greg k-h

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Greg KH]

On Fri, Aug 02, 2013 at 12:53:34PM +0100, Will Deacon wrote:
> Hi Greg,
> 
> On Thu, Aug 01, 2013 at 08:27:30PM +0100, Greg KH wrote:
> > On Thu, Aug 01, 2013 at 07:35:31PM +0100, Will Deacon wrote:
> > > Naturally, this would need to be described as a device-tree binding and
> > > communicate:
> > > 
> > >   - Buses which can be configured as coherent, including which devices
> > >     on those buses can be made coherent.
> > > 
> > >   - How IOMMUs sit on the bus and interact with masters on that bus (the
> > >     current one-IOMMU-driver-per-bus may not work well for the
> > >     platform_bus).
> > 
> > I've been waiting for people to finally run into this one, and realize
> > that they shouldn't be using "platform_bus" :)
> 
> But, as pointed out later in this thread, people have been doing the exact
> opposite!

I guess they were wrong :(

I know I mentioned it a few times over the years, but I've been ignoring
ARM for a long time for a variety of reasons.

> We can change the mindset by yelling, but if you're writing a new
> driver for a peripheral on an ARM SoC, platform_bus is mighty tempting
> because you get a bunch of device-tree parsing code for free (see
> drivers/of/platform.c).

I know :(

So, who do I "yell at", and what do I do to make things easier for you
from the driver-core perspective?

> What's worse is that this nice-and-easy auto-probing doesn't work for nested
> device-nodes (i.e. a bunch of device-nodes under a common parent, something
> which you might think is pretty common in a `tree') so people shy away from
> nesting as a means to group devices too.

That's sad.

> > >   - QoS and PM constraints. This isn't really in my area, but we do have
> > >     buses that have these features and expect software to control them.
> > > 
> > >   - The system topology and linkages between buses and devices.
> > 
> > The driver core handles this really well, you just have to create new
> > busses, and don't rely on the "catch-all" platform_bus.
> 
> Agreed, it's time that we started to describe these non-probable buses as
> separate bus_types, with controller logic for configuring the bus itself
> (there are weird-and-wonderful ring-based designs on the horizon which can
> require a fair amount of setup).

I've heard rumors of those for a while now, I'll believe it when I see
them :)

> > > The last point is increasingly important as various blocks of ARM system
> > > IP start to require knowledge of masters and how things like memory
> > > traffic, DVM messages, interrupts (think MSI) etc are routed between
> > > them in order to configure the system correctly. For example, interfacing
> > > a PCIe device with an SMMU requires knowledge of both the requester id
> > > associated with the device and how that maps to incoming stream ids
> > > (based off the AXI bus id) on the SMMU. Even worse, this mapping is
> > > likely generated dynamically by the host controller, which would need to
> > > know about downstream buses and their SMMUs.
> > 
> > Hm, sounds like an ACPI tree is what you need to be using :)
> > 
> > Seriously, why not use ACPI for stuff like this?  You already are
> > starting to do that for ARM-based systems, why not just make it the
> > standard?
> 
> So, like a good proportion of the ARM community, ACPI isn't something I'm
> well-versed in. Yes, it's coming, but at the same time it's not going to be
> everywhere and we need to continue to support new SoCs using device-tree.
> Whilst it might even become a de-factor standard for servers, mobile devices
> will likely continue with the bootloaders they currently have. Furthermore,
> the mobile space is really the wild-west when it comes to system topology --
> exynos SoCs tend to have one IOMMU per device, for example:
> 
> http://lists.infradead.org/pipermail/linux-arm-kernel/2013-July/181922.html
> 
> On the back of that, how does ACPI describe these relationships? It would
> certainly be a good idea to see what's already being done so we don't
> reinvent everything again for device-tree.

I don't recall the specifics of how it does this, but the spec is open
(and bad to read, sorry), and the linux-acpi mailing list is very
welcoming, so I suggest you start there if you have questions.

thanks,

greg k-h

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Will Deacon]

On Fri, Aug 02, 2013 at 03:20:10PM +0100, Greg KH wrote:
> On Fri, Aug 02, 2013 at 12:53:34PM +0100, Will Deacon wrote:
> > We can change the mindset by yelling, but if you're writing a new
> > driver for a peripheral on an ARM SoC, platform_bus is mighty tempting
> > because you get a bunch of device-tree parsing code for free (see
> > drivers/of/platform.c).
> 
> I know :(
> 
> So, who do I "yell at", and what do I do to make things easier for you
> from the driver-core perspective?

I would anticipate most of these drivers going through the arm-soc tree, so
Olof and Kevin would be doing the yelling. You could join in the chorus too!

We basically need reviewers to adopt the position that a new bus should be
considered and dismissed before using the platform_bus, then you can yell
transitively through them.

> > Agreed, it's time that we started to describe these non-probable buses as
> > separate bus_types, with controller logic for configuring the bus itself
> > (there are weird-and-wonderful ring-based designs on the horizon which can
> > require a fair amount of setup).
> 
> I've heard rumors of those for a while now, I'll believe it when I see
> them :)

Oh, they're coming. I'm trying to understand the programmer's model for one
as we speak :) On the plus side, you also get a whole bunch of debug and PMU
data on these things, which is interesting from a kernel perspective.

> > So, like a good proportion of the ARM community, ACPI isn't something I'm
> > well-versed in. Yes, it's coming, but at the same time it's not going to be
> > everywhere and we need to continue to support new SoCs using device-tree.
> > Whilst it might even become a de-factor standard for servers, mobile devices
> > will likely continue with the bootloaders they currently have. Furthermore,
> > the mobile space is really the wild-west when it comes to system topology --
> > exynos SoCs tend to have one IOMMU per device, for example:
> > 
> > http://lists.infradead.org/pipermail/linux-arm-kernel/2013-July/181922.html
> > 
> > On the back of that, how does ACPI describe these relationships? It would
> > certainly be a good idea to see what's already being done so we don't
> > reinvent everything again for device-tree.
> 
> I don't recall the specifics of how it does this, but the spec is open
> (and bad to read, sorry), and the linux-acpi mailing list is very
> welcoming, so I suggest you start there if you have questions.

Understandable, I'll make an effort to RTFM, although I don't think that
mitigates the need for discussion in this area.

Cheers,

Will

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Greg KH]

On Fri, Aug 02, 2013 at 05:09:32PM +0100, Will Deacon wrote:
> On Fri, Aug 02, 2013 at 03:20:10PM +0100, Greg KH wrote:
> > On Fri, Aug 02, 2013 at 12:53:34PM +0100, Will Deacon wrote:
> > > We can change the mindset by yelling, but if you're writing a new
> > > driver for a peripheral on an ARM SoC, platform_bus is mighty tempting
> > > because you get a bunch of device-tree parsing code for free (see
> > > drivers/of/platform.c).
> > 
> > I know :(
> > 
> > So, who do I "yell at", and what do I do to make things easier for you
> > from the driver-core perspective?
> 
> I would anticipate most of these drivers going through the arm-soc tree, so
> Olof and Kevin would be doing the yelling. You could join in the chorus too!
> 
> We basically need reviewers to adopt the position that a new bus should be
> considered and dismissed before using the platform_bus, then you can yell
> transitively through them.

I don't scale if I'm forced to review every driver to ensure that they
shouldn't be using platform device and should be creating their own bus
type.  You can do that, along with the other ARM developers reviewing
these new subsystems and code being added.

thanks,

greg k-h

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Olof Johansson]

On Fri, Aug 2, 2013 at 3:32 PM, Greg KH <greg@kroah.com> wrote:
> On Fri, Aug 02, 2013 at 05:09:32PM +0100, Will Deacon wrote:
>> On Fri, Aug 02, 2013 at 03:20:10PM +0100, Greg KH wrote:
>> > On Fri, Aug 02, 2013 at 12:53:34PM +0100, Will Deacon wrote:
>> > > We can change the mindset by yelling, but if you're writing a new
>> > > driver for a peripheral on an ARM SoC, platform_bus is mighty tempting
>> > > because you get a bunch of device-tree parsing code for free (see
>> > > drivers/of/platform.c).
>> >
>> > I know :(
>> >
>> > So, who do I "yell at", and what do I do to make things easier for you
>> > from the driver-core perspective?
>>
>> I would anticipate most of these drivers going through the arm-soc tree, so
>> Olof and Kevin would be doing the yelling. You could join in the chorus too!
>>
>> We basically need reviewers to adopt the position that a new bus should be
>> considered and dismissed before using the platform_bus, then you can yell
>> transitively through them.
>
> I don't scale if I'm forced to review every driver to ensure that they
> shouldn't be using platform device and should be creating their own bus
> type.  You can do that, along with the other ARM developers reviewing
> these new subsystems and code being added.

For most new platforms, using basic platform_bus devices makes sense
when the setup is simple and just has a few devices. It's only when
they need to add support for {S,IO}MMUs and get the more advanced
functionality going that things get messy and platform_bus stops
working.

So, we have the option of having someone care about the {S,IO}MMU
pieces, and when they spot the attempts to make that work, push them
towards a proper bus architecture. Until then, the simpler solution is
probably a reasonable idea compared to having newcomers worry about
defining new bus architectures and raising the bar for what it takes
to sort out the initial contributions and getting engaged upstream.


-Olof

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Tony Lindgren]

* Olof Johansson <olof@lixom.net> [130802 22:23]:
> On Fri, Aug 2, 2013 at 3:32 PM, Greg KH <greg@kroah.com> wrote:
> > On Fri, Aug 02, 2013 at 05:09:32PM +0100, Will Deacon wrote:
> >> On Fri, Aug 02, 2013 at 03:20:10PM +0100, Greg KH wrote:
> >> > On Fri, Aug 02, 2013 at 12:53:34PM +0100, Will Deacon wrote:
> >> > > We can change the mindset by yelling, but if you're writing a new
> >> > > driver for a peripheral on an ARM SoC, platform_bus is mighty tempting
> >> > > because you get a bunch of device-tree parsing code for free (see
> >> > > drivers/of/platform.c).
> >> >
> >> > I know :(
> >> >
> >> > So, who do I "yell at", and what do I do to make things easier for you
> >> > from the driver-core perspective?
> >>
> >> I would anticipate most of these drivers going through the arm-soc tree, so
> >> Olof and Kevin would be doing the yelling. You could join in the chorus too!
> >>
> >> We basically need reviewers to adopt the position that a new bus should be
> >> considered and dismissed before using the platform_bus, then you can yell
> >> transitively through them.
> >
> > I don't scale if I'm forced to review every driver to ensure that they
> > shouldn't be using platform device and should be creating their own bus
> > type.  You can do that, along with the other ARM developers reviewing
> > these new subsystems and code being added.
> 
> For most new platforms, using basic platform_bus devices makes sense
> when the setup is simple and just has a few devices. It's only when
> they need to add support for {S,IO}MMUs and get the more advanced
> functionality going that things get messy and platform_bus stops
> working.

Yes, and I think we could get some more mileage out of platform bus
by adding some hooks for things like bus specific reset. That's something
that the device drivers don't need to know about.
 
> So, we have the option of having someone care about the {S,IO}MMU
> pieces, and when they spot the attempts to make that work, push them
> towards a proper bus architecture. Until then, the simpler solution is
> probably a reasonable idea compared to having newcomers worry about
> defining new bus architectures and raising the bar for what it takes
> to sort out the initial contributions and getting engaged upstream.

Yes I'd like to avoid having multiple versions of the same driver
to bind to various SoC specific busses that only differ at the bus
level.

Regards,

Tony

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Will Deacon]

On Sat, Aug 03, 2013 at 06:16:57AM +0100, Olof Johansson wrote:
> On Fri, Aug 2, 2013 at 3:32 PM, Greg KH <greg@kroah.com> wrote:
> > I don't scale if I'm forced to review every driver to ensure that they
> > shouldn't be using platform device and should be creating their own bus
> > type.  You can do that, along with the other ARM developers reviewing
> > these new subsystems and code being added.
> 
> For most new platforms, using basic platform_bus devices makes sense
> when the setup is simple and just has a few devices. It's only when
> they need to add support for {S,IO}MMUs and get the more advanced
> functionality going that things get messy and platform_bus stops
> working.

Right, but being more forward-looking, we will need to know this kind of
topological information to do simple things like (coherent) DMA or even
just to generate/route an interrupt.

> So, we have the option of having someone care about the {S,IO}MMU
> pieces, and when they spot the attempts to make that work, push them
> towards a proper bus architecture. Until then, the simpler solution is
> probably a reasonable idea compared to having newcomers worry about
> defining new bus architectures and raising the bar for what it takes
> to sort out the initial contributions and getting engaged upstream.

I take your point, but at the same time we don't want to find ourselves
suddenly landed with having to convert a bunch of drivers to sit on
different buses as an afterthought. There's likely some middle-ground that
is difficult to envisage up-front.

Will

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Hiroshi Doyu]

On Sat, 3 Aug 2013 07:16:57 +0200
Olof Johansson <olof@lixom.net> wrote:

> On Fri, Aug 2, 2013 at 3:32 PM, Greg KH <greg@kroah.com> wrote:
> > On Fri, Aug 02, 2013 at 05:09:32PM +0100, Will Deacon wrote:
> >> On Fri, Aug 02, 2013 at 03:20:10PM +0100, Greg KH wrote:
> >> > On Fri, Aug 02, 2013 at 12:53:34PM +0100, Will Deacon wrote:
> >> > > We can change the mindset by yelling, but if you're writing a new
> >> > > driver for a peripheral on an ARM SoC, platform_bus is mighty tempting
> >> > > because you get a bunch of device-tree parsing code for free (see
> >> > > drivers/of/platform.c).
> >> >
> >> > I know :(
> >> >
> >> > So, who do I "yell at", and what do I do to make things easier for you
> >> > from the driver-core perspective?
> >>
> >> I would anticipate most of these drivers going through the arm-soc tree, so
> >> Olof and Kevin would be doing the yelling. You could join in the chorus too!
> >>
> >> We basically need reviewers to adopt the position that a new bus should be
> >> considered and dismissed before using the platform_bus, then you can yell
> >> transitively through them.
> >
> > I don't scale if I'm forced to review every driver to ensure that they
> > shouldn't be using platform device and should be creating their own bus
> > type.  You can do that, along with the other ARM developers reviewing
> > these new subsystems and code being added.
> 
> For most new platforms, using basic platform_bus devices makes sense
> when the setup is simple and just has a few devices. It's only when
> they need to add support for {S,IO}MMUs and get the more advanced
> functionality going that things get messy and platform_bus stops
> working.
> 
> So, we have the option of having someone care about the {S,IO}MMU
> pieces, and when they spot the attempts to make that work, push them
> towards a proper bus architecture. Until then, the simpler solution is
> probably a reasonable idea compared to having newcomers worry about
> defining new bus architectures and raising the bar for what it takes
> to sort out the initial contributions and getting engaged upstream.

Agree. Tegra has an central IOMMU(SMMU) in memory controller, where
almost all platform devices can depend on IOMMU. Those IOMMU'abilities
are configurable. We have device instanciation order problem since
IOMMU H/W needs to be instanciated earler than any other IOMMU'able
platform devices, but DT doesn't have any framework to force
this instanciation order[1]. One solution could be to use
-EPROBE_DEFER in drivers, but we have too many IOMMU'able devices, and
also we want to allow to boot with/without IOMMU and to keep drivers
as they are regardless of IOMMU enabled/disabled.

[1] https://lists.ozlabs.org/pipermail/devicetree-discuss/2013-June/036542.html

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Will Deacon]

On Fri, Aug 02, 2013 at 11:32:47PM +0100, Greg KH wrote:
> On Fri, Aug 02, 2013 at 05:09:32PM +0100, Will Deacon wrote:
> > I would anticipate most of these drivers going through the arm-soc tree, so
> > Olof and Kevin would be doing the yelling. You could join in the chorus too!
> > 
> > We basically need reviewers to adopt the position that a new bus should be
> > considered and dismissed before using the platform_bus, then you can yell
> > transitively through them.
> 
> I don't scale if I'm forced to review every driver to ensure that they
> shouldn't be using platform device and should be creating their own bus
> type.  You can do that, along with the other ARM developers reviewing
> these new subsystems and code being added.

Of course, I wasn't suggesting that you have to take the frontline for this
(quite the opposite in fact). Once the usual ARM reviewers know which line
to tow, it will start to be enforced by the ARM subtrees automatically.

Cheers,

Will

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[James Bottomley]

On Thu, 2013-08-01 at 19:35 +0100, Will Deacon wrote:
> Hello,
> 
> Whilst Linux implements a bunch of different bus types (many of which
> are in fact virtual), devices sitting on non-probable, memory mapped
> buses inside SoCs typically live on either the platform_bus or the
> amba_bus. So far, this has worked out alright; the buses haven't needed
> to be visible to software and no additional software control is really
> required from the OS. However, as I/O coherency and hardware
> virtualisation capabilities start to creep into ARM-based SoCs, Linux
> needs to know the topology of the system on which it is running.
> 
> Naturally, this would need to be described as a device-tree binding and
> communicate:
> 
>   - Buses which can be configured as coherent, including which devices
>     on those buses can be made coherent.

I don't think coherency is a good bus property:  we have PCI systems
which are incoherent by virtue of the CPU they're attached to rather
than because of any fundamental bus property.

>   - How IOMMUs sit on the bus and interact with masters on that bus (the
>     current one-IOMMU-driver-per-bus may not work well for the
>     platform_bus).

That's really not a good model.  On most parisc system, we have a split
IOMMU model (we mostly have two iommus and the topology determines which
one you have to use), but our two upper iommus are exposed as parisc
devices sitting at the root of the device tree.  When we program the
device for DMA we walk up the bus until we find the correct iommu to
program.

>   - QoS and PM constraints. This isn't really in my area, but we do have
>     buses that have these features and expect software to control them.
> 
>   - The system topology and linkages between buses and devices.

Actually, topology isn't really about linkages between busses and
devices.  Topology is about linkages between devices and devices.  A bus
type is a grouping of a set of devices which share common properties so
if you look at parisc for example, which is closest to ARM, our bus
topology at the top is usually two iommus connected to a set of end
points connected to several parisc specific things, all of which are bus
type parisc.  PCI is the same since PCI bridges are also PCI devices,
you can get device topologies nesting quite deep.

James

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Will Deacon]

Hi James,

On Thu, Aug 01, 2013 at 10:41:49PM +0100, James Bottomley wrote:
> On Thu, 2013-08-01 at 19:35 +0100, Will Deacon wrote:
> > Whilst Linux implements a bunch of different bus types (many of which
> > are in fact virtual), devices sitting on non-probable, memory mapped
> > buses inside SoCs typically live on either the platform_bus or the
> > amba_bus. So far, this has worked out alright; the buses haven't needed
> > to be visible to software and no additional software control is really
> > required from the OS. However, as I/O coherency and hardware
> > virtualisation capabilities start to creep into ARM-based SoCs, Linux
> > needs to know the topology of the system on which it is running.
> > 
> > Naturally, this would need to be described as a device-tree binding and
> > communicate:
> > 
> >   - Buses which can be configured as coherent, including which devices
> >     on those buses can be made coherent.
> 
> I don't think coherency is a good bus property:  we have PCI systems
> which are incoherent by virtue of the CPU they're attached to rather
> than because of any fundamental bus property.

Ok, but this information surely needs to be propagated to the bus driver,
no? I guess I'm asking where all of this information should be correlated so
that devices can be made coherent or not at runtime.

> >   - How IOMMUs sit on the bus and interact with masters on that bus (the
> >     current one-IOMMU-driver-per-bus may not work well for the
> >     platform_bus).
> 
> That's really not a good model.  On most parisc system, we have a split
> IOMMU model (we mostly have two iommus and the topology determines which
> one you have to use), but our two upper iommus are exposed as parisc
> devices sitting at the root of the device tree.  When we program the
> device for DMA we walk up the bus until we find the correct iommu to
> program.

I think this follows on from Greg's points about writing new bus_types. At
the moment, there is no bus topology in Linux for most of the SoCs in the
ARM-space and, as such, sideband information in the device-tree is used to
work out what links to what. This then localises the topology information to
the IOMMU driver and makes it impossible to co-ordinate with other drivers
and subsystems.

An example of this is that we have a bunch of IOMMU drivers for ARM under
drivers/iommu/, each with a different notion of topology. They then register
with the IOMMU framework as per usual, but at the same time call
bus_set_iommu(&platform_bus_type, ...) to receive notifications of devices
being added on the bus. We have private fields in struct dev_archdata so the
IOMMU driver can keep track of where the device lives in the topology (i.e.
which IOMMU it's mastering through). This is fine within the driver, but how
on Earth are users supposed to create mappings without duplicating and/or
spreading the knowledge ingrained in the driver?

To this end, a global arm_iommu_create_mapping symbol was added, which has
exactly *1* caller in the entire kernel, which seems to be... another IOMMU
driver.

> >   - QoS and PM constraints. This isn't really in my area, but we do have
> >     buses that have these features and expect software to control them.
> > 
> >   - The system topology and linkages between buses and devices.
> 
> Actually, topology isn't really about linkages between busses and
> devices.  Topology is about linkages between devices and devices.  A bus
> type is a grouping of a set of devices which share common properties so
> if you look at parisc for example, which is closest to ARM, our bus
> topology at the top is usually two iommus connected to a set of end
> points connected to several parisc specific things, all of which are bus
> type parisc.  PCI is the same since PCI bridges are also PCI devices,
> you can get device topologies nesting quite deep.

Maybe we're discussing across terminology, but the buses I'm referring to
also have a device aspect to them, so there's a bit of a blur here. It's
probably worth making the distinction though, because the interface
controlling a bus is likely to sit on a different bus itself.

Will

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Bjorn Helgaas]

On Thu, Aug 1, 2013 at 12:35 PM, Will Deacon <will.deacon@arm.com> wrote:

> ... For example, interfacing
> a PCIe device with an SMMU requires knowledge of both the requester id
> associated with the device and how that maps to incoming stream ids
> (based off the AXI bus id) on the SMMU.

Tangent: We're trying to hash out a way for IOMMU drivers to use a PCI
interface to discover the requester IDs associated with a device.  We
currently only have one users (intel-iommu.c), and it would be awesome
to have another user to help make things generic.  Please chime in if
you have opinions.

https://lkml.kernel.org/r/20130711210326.1701.56478.stgit at bling.home

[Ksummit-2013-discuss] [ARM ATTEND] Describing complex, non-probable system topologies

[Will Deacon]

On Thu, Aug 01, 2013 at 11:26:30PM +0100, Bjorn Helgaas wrote:
> On Thu, Aug 1, 2013 at 12:35 PM, Will Deacon <will.deacon@arm.com> wrote:
> 
> > ... For example, interfacing
> > a PCIe device with an SMMU requires knowledge of both the requester id
> > associated with the device and how that maps to incoming stream ids
> > (based off the AXI bus id) on the SMMU.
> 
> Tangent: We're trying to hash out a way for IOMMU drivers to use a PCI
> interface to discover the requester IDs associated with a device.  We
> currently only have one users (intel-iommu.c), and it would be awesome
> to have another user to help make things generic.  Please chime in if
> you have opinions.
> 
> https://lkml.kernel.org/r/20130711210326.1701.56478.stgit at bling.home

Ooh, yes, thanks Bjorn. I'll find some time to take a look at that.

Cheers,

Will