Handling of modular boards

Handling of modular boards

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1373 bytes --]

Quite a few reference platforms (including Wolfson ones, which is why
I'm particularly interested) use replaceable modules to allow
configuration changes.  Since we can often identify the configuration at
runtime we should ideally do that but currently there's no infrastructure 
to help with that, generally this seems to be done in arch code for the
machine but this doesn't scale when even the CPU might change and isn't
terribly device tree compatible either.

For reference the code for current Wolfson plugin modules is in
arch/arm/mach-s3c64xx/mach-crag6410-module.c.

The most obvious current fit here is the MFD subsystem but it feels like
we need some slightly different infastructure to what MFD currently
provides.  MFD is really set up to handle platform devices with a core
and linear ranges of resources fanning out from that core since they're
really oriented around chips.  In contrast these boards are more about
remapping random collections of potentially unrelated resources and
instantiating devices on all sorts of buses and share more with board
files.

I'm just starting to put some stuff together for this so I was wondering
if anyone had been thinking about this and had any bright ideas for how
to handle it, and also if people think that MFD is a good fit for this
or if we should split the silicon MFDs from these PCBs.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Handling of modular boards

[Arnd Bergmann]

On Friday 04 May 2012, Mark Brown wrote:
> Quite a few reference platforms (including Wolfson ones, which is why
> I'm particularly interested) use replaceable modules to allow
> configuration changes.  Since we can often identify the configuration at
> runtime we should ideally do that but currently there's no infrastructure 
> to help with that, generally this seems to be done in arch code for the
> machine but this doesn't scale when even the CPU might change and isn't
> terribly device tree compatible either.
> 
> For reference the code for current Wolfson plugin modules is in
> arch/arm/mach-s3c64xx/mach-crag6410-module.c.

Hi Mark,

Thanks for getting the discussion started. I've seen the same issue come
up for arch/arm/mach-ux500/board-mop500*uib.c and for the beaglebone.
I'm sure there are many more, but we should make sure that everyone
of these can live with whatever we come up with.

> The most obvious current fit here is the MFD subsystem but it feels like
> we need some slightly different infastructure to what MFD currently
> provides.  MFD is really set up to handle platform devices with a core
> and linear ranges of resources fanning out from that core since they're
> really oriented around chips.  In contrast these boards are more about
> remapping random collections of potentially unrelated resources and
> instantiating devices on all sorts of buses and share more with board
> files.
> 
> I'm just starting to put some stuff together for this so I was wondering
> if anyone had been thinking about this and had any bright ideas for how
> to handle it, and also if people think that MFD is a good fit for this
> or if we should split the silicon MFDs from these PCBs.

One idea that I've heard before is to put device tree fragments into the
kernel and dynamically add them to the device tree that was passed by the
boot loader whenever we detect the presence of a specific device.
This obviously means it works only for boards using DT for booting, but
it allows us to use some infrastructure that we already have.

Another idea was to put all the possible extensions into the device tree
for a given board and disable them by default, putting it into the
responsibility of the boot loader to enable the one that is actually
being used. This has serious scalibility problems when there are many
possible extensions and also relies more on the boot loader than I would
like.

An intermediate solution that I really like is the ability to
stuff device tree fragments on extension board themselves, but that
can only work for new designs and causes problems when that information
is not actually correct.

	Arnd

Re: Handling of modular boards

[Mark Brown]

On Fri, May 04, 2012 at 07:34:08PM +0000, Arnd Bergmann wrote:

> One idea that I've heard before is to put device tree fragments into the
> kernel and dynamically add them to the device tree that was passed by the
> boot loader whenever we detect the presence of a specific device.
> This obviously means it works only for boards using DT for booting, but
> it allows us to use some infrastructure that we already have.

I think anything that relies on bootloaders (or DT for that matter) is a
bit of a non-starter for my personal use cases.  Even where we're using
DT relying on a sane bootloader seems a bit scary - my personal use
cases would rely on updating this stuff in the field for non-technical
users who would have trouble recovering from issues.

> An intermediate solution that I really like is the ability to
> stuff device tree fragments on extension board themselves, but that
> can only work for new designs and causes problems when that information
> is not actually correct.

I can see the theory, but I can also see some practical concerns.  And
with the boards I'm working with we currently have 8 bits of data so...

Re: Handling of modular boards

[Wolfgang Denk]

Dear Arnd,

In message <201205041934.08830.arnd@arndb.de> you wrote:
>
> One idea that I've heard before is to put device tree fragments into the
> kernel and dynamically add them to the device tree that was passed by the
> boot loader whenever we detect the presence of a specific device.
> This obviously means it works only for boards using DT for booting, but
> it allows us to use some infrastructure that we already have.
> 
> Another idea was to put all the possible extensions into the device tree
> for a given board and disable them by default, putting it into the
> responsibility of the boot loader to enable the one that is actually
> being used. This has serious scalibility problems when there are many
> possible extensions and also relies more on the boot loader than I would
> like.

On the other hand, some of the issues we're trying to solve here
for the kernel are also present in the boot loader, so this needs to
do this anyway - whether by inserting new or modifying (enabling or
disabling) existing properties in the DT is not really relevant here.


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Our business is run on trust.  We trust you will pay in advance.

Re: Handling of modular boards

[Arnd Bergmann]

On Friday 04 May 2012, Wolfgang Denk wrote:
> In message <201205041934.08830.arnd@arndb.de> you wrote:
> >
> > One idea that I've heard before is to put device tree fragments into the
> > kernel and dynamically add them to the device tree that was passed by the
> > boot loader whenever we detect the presence of a specific device.
> > This obviously means it works only for boards using DT for booting, but
> > it allows us to use some infrastructure that we already have.
> > 
> > Another idea was to put all the possible extensions into the device tree
> > for a given board and disable them by default, putting it into the
> > responsibility of the boot loader to enable the one that is actually
> > being used. This has serious scalibility problems when there are many
> > possible extensions and also relies more on the boot loader than I would
> > like.
> 
> On the other hand, some of the issues we're trying to solve here
> for the kernel are also present in the boot loader, so this needs to
> do this anyway - whether by inserting new or modifying (enabling or
> disabling) existing properties in the DT is not really relevant here.

I haven't seen a case where the add-on board is actually required
for booting. What examples are you thinking of?

	Arnd

Re: Handling of modular boards

[Wolfgang Denk]

Dear Arnd,

In message <201205042039.25794.arnd@arndb.de> you wrote:
>
> > On the other hand, some of the issues we're trying to solve here
> > for the kernel are also present in the boot loader, so this needs to
> > do this anyway - whether by inserting new or modifying (enabling or
> > disabling) existing properties in the DT is not really relevant here.
> 
> I haven't seen a case where the add-on board is actually required
> for booting. What examples are you thinking of?

There are systems (and I bet it will be a growing number) where U-Boot
itself uses the DT for configuration.  Also, there are functions that
are needed both by the boot loader and the kernel - for example to
dislay a splash screen the boot loader needs to initialize the
display, so it must be able to detect which type of LCD is attached
(resolution, color-depth, orientation) - the device tree comes in very
handy here.  Why should Linux re-do all such things?

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
I must follow the people.  Am I not their leader? - Benjamin Disraeli

Re: Handling of modular boards

[Arnd Bergmann]

On Friday 04 May 2012, Wolfgang Denk wrote:
> There are systems (and I bet it will be a growing number) where U-Boot
> itself uses the DT for configuration.  Also, there are functions that
> are needed both by the boot loader and the kernel - for example to
> dislay a splash screen the boot loader needs to initialize the
> display, so it must be able to detect which type of LCD is attached
> (resolution, color-depth, orientation) - the device tree comes in very
> handy here.  Why should Linux re-do all such things?

Sure, there are a lot of things that the boot loader can use from the
device tree, but I'm not sure if the LCD panel connection fits into
the same category as the devices that Mark was thinking of.

Anyway, display controllers are definitely something that needs to
be handled in some way, which may or may not be the same way we
handle more complex collections of arbitrary devices.

	Arnd

Re: Handling of modular boards

[Stephen Warren]

On 05/04/2012 03:03 PM, Arnd Bergmann wrote:
> On Friday 04 May 2012, Wolfgang Denk wrote:
>> There are systems (and I bet it will be a growing number) where U-Boot
>> itself uses the DT for configuration.  Also, there are functions that
>> are needed both by the boot loader and the kernel - for example to
>> dislay a splash screen the boot loader needs to initialize the
>> display, so it must be able to detect which type of LCD is attached
>> (resolution, color-depth, orientation) - the device tree comes in very
>> handy here.  Why should Linux re-do all such things?
> 
> Sure, there are a lot of things that the boot loader can use from the
> device tree, but I'm not sure if the LCD panel connection fits into
> the same category as the devices that Mark was thinking of.

A board I have sitting on my desk right now has separate boards for (and
multiple options for each of):

* Motherboard
* CPU+DRAM
* PMU/PMIC
* Display (LCD)
... and many more.

Interaction with the PMU/PMIC is required for at least some of the boot
media options.

Interaction with the display (LCD) while not technically required to
simply boot the kernel is required by desired use-cases, in order to
display a splash screen ASAP during early boot.

Oh, and the motherboard has a gazillion different HW mux options, which
affect, amongst many other things, which SD/eMMC/SDIO ports are usable
on the motherboard, and which are routed to various daughter boards. I'm
not actually 100% sure if the switches controlling those mux settings
are readable from SW. I certainly hope so...

Re: Handling of modular boards

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1460 bytes --]

On Fri, May 04, 2012 at 03:09:19PM -0600, Stephen Warren wrote:
> On 05/04/2012 03:03 PM, Arnd Bergmann wrote:

> > Sure, there are a lot of things that the boot loader can use from the
> > device tree, but I'm not sure if the LCD panel connection fits into
> > the same category as the devices that Mark was thinking of.

> A board I have sitting on my desk right now has separate boards for (and
> multiple options for each of):

> * Motherboard
> * CPU+DRAM
> * PMU/PMIC
> * Display (LCD)
> ... and many more.

> Interaction with the PMU/PMIC is required for at least some of the boot
> media options.

Yeah, similar setup for our boards except the PMICs are soldered down
onto other boards.  We've got a mainboard, three audio boards of various
kinds, a random non-audio components board and a CPU/DRAM board.

There's good solid engineering reasons for doing this.  CPUs and RAMs
tend to be very high density devices with lots of pins and be difficult
enough to route to require large numbers of layers (and ideally you want
the PMIC to be physically close to them since long traces tend to become
electrically interesting for CPU style loads, especially when routed
through connectors) all of which leads to an expensive board which you
pay for by area.  With reference boards with large form factors it's
worth the effort to have a separate, smaller, board manufactured to meet
these requirements - even in very low volumes the cost wins are
noticable.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Handling of modular boards

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 622 bytes --]

On Fri, May 04, 2012 at 10:33:57PM +0200, Wolfgang Denk wrote:

> On the other hand, some of the issues we're trying to solve here
> for the kernel are also present in the boot loader, so this needs to
> do this anyway - whether by inserting new or modifying (enabling or
> disabling) existing properties in the DT is not really relevant here.

FWIW if the bootloader can usefully handle this stuff I think that's a
good approach but there is substantial variation in quality of
implementation between bootloaders and even when the bootloader is a
good one it's not always practical to update it or the data it relies
on.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Handling of modular boards

[Ben Dooks]

On Fri, May 04, 2012 at 11:09:45PM +0100, Mark Brown wrote:
> On Fri, May 04, 2012 at 10:33:57PM +0200, Wolfgang Denk wrote:
> 
> > On the other hand, some of the issues we're trying to solve here
> > for the kernel are also present in the boot loader, so this needs to
> > do this anyway - whether by inserting new or modifying (enabling or
> > disabling) existing properties in the DT is not really relevant here.
> 
> FWIW if the bootloader can usefully handle this stuff I think that's a
> good approach but there is substantial variation in quality of
> implementation between bootloaders and even when the bootloader is a
> good one it's not always practical to update it or the data it relies
> on.

I agree, the list of devices should be in the device-tree handed to
whatever OS is being booted. It isn't a Linux specific problem that
we're looking at here.

Any operating system, pre-OS test suite, etc. is going going to need
this information, and in my view the bootloader should be doing whatever
is needed to create a device-tree to passed through to the next loaded
system.

Also, having the information available to $bootloader means the user
can verify the presence of the peripherals before the OS is loaded.

-- 
Ben Dooks, ben@fluff.org, http://www.fluff.org/ben/

Large Hadron Colada: A large Pina Colada that makes the universe disappear.

Re: Handling of modular boards

[Igor Grinberg]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/05/12 01:09, Mark Brown wrote:
> On Fri, May 04, 2012 at 10:33:57PM +0200, Wolfgang Denk wrote:
> 
>> On the other hand, some of the issues we're trying to solve here
>> for the kernel are also present in the boot loader, so this needs to
>> do this anyway - whether by inserting new or modifying (enabling or
>> disabling) existing properties in the DT is not really relevant here.
> 
> FWIW if the bootloader can usefully handle this stuff I think that's a
> good approach but there is substantial variation in quality of
> implementation between bootloaders and even when the bootloader is a
> good one it's not always practical to update it or the data it relies
> on.

I agree on this (and also with Ben), all our boards/extensions/base boards/etc
can be discovered in the boot loader and we will use the DT to pass
the relevant information on the attached extensions and used base board.

Also, most (if not all) of our boards/extensions have I2C EEPROM which
describes the devices on that board/extension which is useful for
building/extending the DT in the bootloader.

- -- 
Regards,
Igor.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJPq7dXAAoJEBDE8YO64Efag7sQAJ4Ipk/A24p6mO6ROtcd+q2s
gGlEjLHeZ7sNPnwCOCakCgX7+SYXKBv6w5aWKM70eLDUfoVLu9eaUeT4sYd2fGtR
4xFVkcYRenzkuO7MZ3GccAAF4Hg4WwMAn/3StIJU6pyBY7OmOsbqnoTa7PJjKr/J
HwwM8g8eo3q4tg2sB5P922Fol61BK2Wo3S7I9kzHtnwUx9yW3aG2mf/ublfXHUkQ
3RKfXrdREFJL30j9SrJzh6wVqQoE9Y7UrYs1BIiAs9AL/d2+eYgSPNKh/iBw+26f
V+Tr8t9xPyP1b1HGUjLVtGmwWtLncws2kfYnMcjvLKgES967SRbSboqRqwJL95yR
h9xKoVVrqC9lCXZn/uZCHmRaj/7DaPqr4/tbJefJTcU4ErS7bv/QxXARaj41SmS9
sq8A+qefiKaleBfinmiXjoCu8QcUKmfLF8oLt52tylb/4CUnipf4jLbnQ9fdPxGB
gQnrlg6T3WVTR7fzhOsXrgHQvocJa+3/YSYUyWbNvfjJBWFV6ZqERjTQCvhkXivG
bbwBn6tHAol+tRAUfGudlwzcaZZGbJ+vvK/GmSCI+tSfgz++O9uvd0spklatB4ab
NyikQLZBvSufnyWSY6Mp3iShhwgGZosxu9OGgQzclEh1tC8LQEbxOwETaRi/7uGq
PkkXsoPrmBc0C0dO+fIc
=icYW
-----END PGP SIGNATURE-----

Re: Handling of modular boards

[Stephen Warren]

On 05/10/2012 06:40 AM, Igor Grinberg wrote:
> On 05/05/12 01:09, Mark Brown wrote:
>> On Fri, May 04, 2012 at 10:33:57PM +0200, Wolfgang Denk wrote:
> 
>>> On the other hand, some of the issues we're trying to solve
>>> here for the kernel are also present in the boot loader, so
>>> this needs to do this anyway - whether by inserting new or
>>> modifying (enabling or disabling) existing properties in the DT
>>> is not really relevant here.
> 
>> FWIW if the bootloader can usefully handle this stuff I think
>> that's a good approach but there is substantial variation in
>> quality of implementation between bootloaders and even when the
>> bootloader is a good one it's not always practical to update it
>> or the data it relies on.
> 
> I agree on this (and also with Ben), all our boards/extensions/base
> boards/etc can be discovered in the boot loader and we will use the
> DT to pass the relevant information on the attached extensions and
> used base board.
> 
> Also, most (if not all) of our boards/extensions have I2C EEPROM
> which describes the devices on that board/extension which is useful
> for building/extending the DT in the bootloader.

I believe that's true for all/many NVIDIA boards too.

But, duplicating all this in every bootloader might not make sense.
Sure there are some cases where the bootloader needs this information
(e.g. to load the kernel from an SD card that's on 1 of N plugin
boards), but there may be much information the bootloader doesn't care
about.

Would it make sense to write a DT-identifying-and-merging shim that
can be executed by the bootloader, create the final DT, and then jump
to the kernel?

Hmmm. That's probably a bad idea, since then it means needing e.g. I2C
drivers to read the ID EEPROMs, pinmux drivers, ... in the shim.

Perhaps the common shim idea makes sense, but we need a standardized
API it can use that all bootloaders provide for it to operate.

This is beginning to sound a lot like a UEFI byte code module (I
assume; I know little about them)

Re: Handling of modular boards

[Igor Grinberg]

On 05/10/12 19:15, Stephen Warren wrote:
> On 05/10/2012 06:40 AM, Igor Grinberg wrote:
>> On 05/05/12 01:09, Mark Brown wrote:
>>> On Fri, May 04, 2012 at 10:33:57PM +0200, Wolfgang Denk wrote:
>>
>>>> On the other hand, some of the issues we're trying to solve
>>>> here for the kernel are also present in the boot loader, so
>>>> this needs to do this anyway - whether by inserting new or
>>>> modifying (enabling or disabling) existing properties in the DT
>>>> is not really relevant here.
>>
>>> FWIW if the bootloader can usefully handle this stuff I think
>>> that's a good approach but there is substantial variation in
>>> quality of implementation between bootloaders and even when the
>>> bootloader is a good one it's not always practical to update it
>>> or the data it relies on.
>>
>> I agree on this (and also with Ben), all our boards/extensions/base
>> boards/etc can be discovered in the boot loader and we will use the
>> DT to pass the relevant information on the attached extensions and
>> used base board.
>>
>> Also, most (if not all) of our boards/extensions have I2C EEPROM
>> which describes the devices on that board/extension which is useful
>> for building/extending the DT in the bootloader.
> 
> I believe that's true for all/many NVIDIA boards too.
> 
> But, duplicating all this in every bootloader might not make sense.

Yeah, I agree on this, especially when you have multiple bootloaders
for the same board...

On the other hand, that is a common problem of the open source
software - anyone can create a bootloader, publish and use it...

> Sure there are some cases where the bootloader needs this information
> (e.g. to load the kernel from an SD card that's on 1 of N plugin
> boards), but there may be much information the bootloader doesn't care
> about.
> 
> Would it make sense to write a DT-identifying-and-merging shim that
> can be executed by the bootloader, create the final DT, and then jump
> to the kernel?
> 
> Hmmm. That's probably a bad idea, since then it means needing e.g. I2C
> drivers to read the ID EEPROMs, pinmux drivers, ... in the shim.
> 
> Perhaps the common shim idea makes sense, but we need a standardized
> API it can use that all bootloaders provide for it to operate.

Let's see where we're heading:
We created/extended the DT to abstract all the SoC/board specific
(probably also undetectable) stuff away from the kernel and now we will
create a special shi(t)m that will interface the bootloader and create the DT?

I don't think it is a good idea, especially because you still need to change
the bootloader(s) and if you place the I2C/pinmux/gpio drivers inside that
shi(t)m, you duplicate the bootloader, no?

I think we should let the bootloader(s) create/extend the DT and if for some
reason a bootloader wants to separate the DT code into a shim, let it be so.

> 
> This is beginning to sound a lot like a UEFI byte code module (I
> assume; I know little about them)

That's another curse... probably we will not get away from it...

 

-- 
Regards,
Igor.

Re: Handling of modular boards

[Linus Walleij]

On Fri, May 4, 2012 at 9:34 PM, Arnd Bergmann <arnd@arndb.de> wrote:

> Thanks for getting the discussion started. I've seen the same issue come
> up for arch/arm/mach-ux500/board-mop500*uib.c and for the beaglebone.
> I'm sure there are many more, but we should make sure that everyone
> of these can live with whatever we come up with.

The same issue sort of comes up with any system that uses the idiom
to have a few GPIO lines indicate in a binary code what version of the
board we're dealing with and what devices are thus on it, right?

We have this issue for the U9540 reference design and potentially
on the Snowball as well.

Yours,
Linus Walleij

Re: Handling of modular boards

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 767 bytes --]

On Tue, May 08, 2012 at 02:26:54PM +0200, Linus Walleij wrote:
> On Fri, May 4, 2012 at 9:34 PM, Arnd Bergmann <arnd@arndb.de> wrote:

> > Thanks for getting the discussion started. I've seen the same issue come
> > up for arch/arm/mach-ux500/board-mop500*uib.c and for the beaglebone.
> > I'm sure there are many more, but we should make sure that everyone
> > of these can live with whatever we come up with.

> The same issue sort of comes up with any system that uses the idiom
> to have a few GPIO lines indicate in a binary code what version of the
> board we're dealing with and what devices are thus on it, right?

> We have this issue for the U9540 reference design and potentially
> on the Snowball as well.

Yes, I think that's basically the same problem.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Handling of modular boards

[Stephen Warren]

On 05/04/2012 12:58 PM, Mark Brown wrote:
> Quite a few reference platforms (including Wolfson ones, which is why
> I'm particularly interested) use replaceable modules to allow
> configuration changes.  Since we can often identify the configuration at
> runtime we should ideally do that but currently there's no infrastructure 
> to help with that...

So, I'll respond within the context of device tree, although perhaps you
were looking for something more general?

I was just asked basically the same question internally to NVIDIA. One
option that was floated was to store the device tree in chunks and have
the bootloader piece them together. You'd start with the DT for the
basic CPU board, probe what HW was available, and then graft in the
content of additional DT chunks and pass the final result to the kernel.
The advantages here are:

a) The DT is stored in chunks for each plugin board, so there's no bloat
in the DT that gets passed to the kernel; it contains exactly what's on
the board.

b) The kernel doesn't have to do anything much; it gets an exact
description of what's on this particular board configuration and doesn't
even care that it's modular.

c) This is probably pretty easy to implement in the bootloader; I
imagine libfdt already has the ability to graft together or overlay
different chunks of DT, and if not, it's most likely easy enough to add.

Disadvantages are:

a) Relies on the bootloader, so is somewhat out of our control.

b) Doesn't integrate well with hotplug; the DT for the board
configuration is static at boot. What if a board can be unplugged and
another plugged in; a reboot or similar would be needed to adjust the
kernel to this.

Another approach would be to put everything in a single DT, with some
representation of how to identify the child boards, and then have the
kernel only use/parse certain chunks of the DT based on the ID results.
I’m not sure how complex that would be. Perhaps something like:

daughter-board {
    compatible = “daughter-board-mux-eeprom”;
    eeprom = <&i2c_eeprom_node>;
    eeprom-offset = <0x10>; // ID address within EEPOM
    #address-cells = <2>; // # bytes in EEPROM

    board-a {
        compatible = “daughter-board”;
        reg = <0x1234>;
        … nodes for devices on this daughter board
        … or perhaps a list of phandles elsewhere to go activate?
    };

    board-b {
        compatible = “daughter-board”;
        reg = <0x9876>;
        … nodes for devices on this daughter board
    };
};

The complexity here is that all the devices on the daughter board would
end up being on different buses (e.g. 2 different I2C busses, an SPI
bus, even an MMIO bus) so representing this in the daughter board nodes
would be complex. Do we insert a "daughter board mux" onto every single
bus that's routed to the daughter board connectors, or add the ability
to dynamically add nodes into pre-existing busses, e.g. add
/daughter-board/board-a/i2c-0 into /tegra-i2c@xxx/?

One advantage here is that everything is explicitly represented to the
kernel, so it has full knowledge of what's going on. Hotplug could
presumably be integrated pretty easily by making the
daughter-board-mux-eeprom able to react to plug events, and re-probe the
ID EEPROM.

Re: Handling of modular boards

[Wolfgang Denk]

Dear Stephen,

In message <4FA432E9.9050606@wwwdotorg.org> you wrote:
>
> representation of how to identify the child boards, and then have the
> kernel only use/parse certain chunks of the DT based on the ID results.

I expect that this will quickly cause problems, for example in those
many cases where pins and peripheral functions are multiplexed, or
usable for different purposes. I would not want to maintain a DT that
has to describe all combinations used by some boards - in a way that
does not cause conflicts on either of them.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
CONSUMER NOTICE:  Because  of  the  "Uncertainty  Principle,"  It  Is
Impossible  for  the  Consumer  to  Find  Out  at  the Same Time Both
Precisely Where This Product Is and How Fast It Is Moving.

Re: Handling of modular boards

[Stephen Warren]

On 05/04/2012 02:38 PM, Wolfgang Denk wrote:
> Dear Stephen,
> 
> In message <4FA432E9.9050606@wwwdotorg.org> you wrote:
>>
>> representation of how to identify the child boards, and then have the
>> kernel only use/parse certain chunks of the DT based on the ID results.
> 
> I expect that this will quickly cause problems, for example in those
> many cases where pins and peripheral functions are multiplexed, or
> usable for different purposes. I would not want to maintain a DT that
> has to describe all combinations used by some boards - in a way that
> does not cause conflicts on either of them.

With the DT pinctrl bindings, you can define the pinctrl configuration
required to interact with particular child boards in the DT chunk for
that child board. So, I think this would work out fine; you wouldn't
have to represent a huge maze/array of conditional pinctrl settings in
the main board file.

Re: Handling of modular boards

[Mark Brown]

[-- Attachment #1.1: Type: text/plain, Size: 4317 bytes --]

On Fri, May 04, 2012 at 01:50:01PM -0600, Stephen Warren wrote:
> On 05/04/2012 12:58 PM, Mark Brown wrote:
> > Quite a few reference platforms (including Wolfson ones, which is why
> > I'm particularly interested) use replaceable modules to allow
> > configuration changes.  Since we can often identify the configuration at
> > runtime we should ideally do that but currently there's no infrastructure 
> > to help with that...

> So, I'll respond within the context of device tree, although perhaps you
> were looking for something more general?

Like I just said in reply to Arnd I think that anything that relies on
the device tree is rather optimistic.  Device tree isn't even universal
for ARM and there's a huge raft of architectures that have no current
intention to adopt DT at all.  For example I understand that a lot of
the Blackfin boards are modular, though I don't know to what extent they 
can usefully be enumerated from software.

I know DT is a shiny new toy and all that but when developing generic
infrastructure there needs to be an awareness that we can't rely on it.

> I was just asked basically the same question internally to NVIDIA. One
> option that was floated was to store the device tree in chunks and have
> the bootloader piece them together. You'd start with the DT for the
> basic CPU board, probe what HW was available, and then graft in the
> content of additional DT chunks and pass the final result to the kernel.
> The advantages here are:

> a) The DT is stored in chunks for each plugin board, so there's no bloat
> in the DT that gets passed to the kernel; it contains exactly what's on
> the board.

> a) Relies on the bootloader, so is somewhat out of our control.

Yes, this is a crippling issue for my personal usecases.

> b) Doesn't integrate well with hotplug; the DT for the board
> configuration is static at boot. What if a board can be unplugged and
> another plugged in; a reboot or similar would be needed to adjust the
> kernel to this.

This is another issue - a similar set of problems does apply to some PCI
type cards where the PCI device is essentially a bridge to a typical
embedded system - though practically speaking it's much less severe.

> Another approach would be to put everything in a single DT, with some
> representation of how to identify the child boards, and then have the
> kernel only use/parse certain chunks of the DT based on the ID results.
> I’m not sure how complex that would be. Perhaps something like:

This does also have the disadvantage of requiring the device tree for
each CPU to be updated for every single plugin module which could get a
bit tedious (this does also apply to the approach of having the
bootloader create the DT - it scales fine if the CPU is a part of the
base board but if the CPU is one of the swappable modules it's less
clear).  Plus...

> The complexity here is that all the devices on the daughter board would
> end up being on different buses (e.g. 2 different I2C busses, an SPI
> bus, even an MMIO bus) so representing this in the daughter board nodes
> would be complex. Do we insert a "daughter board mux" onto every single
> bus that's routed to the daughter board connectors, or add the ability
> to dynamically add nodes into pre-existing busses, e.g. add
> /daughter-board/board-a/i2c-0 into /tegra-i2c@xxx/?

...there's this, I'm not sure the daughterboard mux is going to fly.
It's requiring each and every bus to understand this concept which seems
like a bit of an imposition to me, especially given that it exists
purely to service the needs of DT.  The idea of having DT blobs injected
for the modules seems easier than this.

I do think we want to be able to write drivers for modules; if we can go
with injecting DT blobs then from a kernel point of view everything is
already sorted so there's nothing to do but this doesn't feel like it
actually resolves the issue, at least for me.  For example, with my
current systems it'd require a DT port for s3c6410 plus the addition of
both DT support and hardware module identification to our current
bootloader and then the ongoing maintainance of the device trees for all
the CPU and board combinations that might exist.  This seems like a lot
of work.

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Handling of modular boards

[Linus Walleij]

On Fri, May 4, 2012 at 10:44 PM, Mark Brown
<broonie@opensource.wolfsonmicro.com> wrote:
> On Fri, May 04, 2012 at 01:50:01PM -0600, Stephen Warren wrote:
>
>> b) Doesn't integrate well with hotplug; the DT for the board
>> configuration is static at boot. What if a board can be unplugged and
>> another plugged in; a reboot or similar would be needed to adjust the
>> kernel to this.
>
> This is another issue - a similar set of problems does apply to some PCI
> type cards where the PCI device is essentially a bridge to a typical
> embedded system - though practically speaking it's much less severe.

I think Alessandro is working on a board like that right now, so looping
in Ale to this discussion to get his attention...

Linus Walleij

Re: Handling of modular boards

[Alessandro Rubini]

Hello.

>> This is another issue - a similar set of problems does apply to some PCI
>> type cards where the PCI device is essentially a bridge to a typical
>> embedded system - though practically speaking it's much less severe.
> 
> I think Alessandro is working on a board like that right now, so looping
> in Ale to this discussion to get his attention...

> I think Alessandro is working on a board like that right now, so looping
> in Ale to this discussion to get his attention...

Yes, that's it. The vendor of my thing has wrapped everything under
pci headers, so the "typical embedded system", which actually is a
demasculated ARM SoC is self-described by PCI (no need for device tree)

A previous poster said:
>>> b) Doesn't integrate well with hotplug; the DT for the board
>>> configuration is static at boot. What if a board can be unplugged and
>>> another plugged in; a reboot or similar would be needed to adjust the
>>> kernel to this.

I think you need some enumeration mechanism in this case. Actually, I
think this will become common in the near future, as you can reprogram
your FPGA devices while the system runs.

The issue is real, and I'm involved in a self-description project; it
allows to use the well-known bus abstraction (with bus controller,
devices, drivers) to ease handling soft cores that may come and go
while the system is alive.

Thank you Linus for involving me, I'll now go to read the whole thread
/alessandro

Re: Handling of modular boards

[Ben Dooks]

On Fri, May 04, 2012 at 01:50:01PM -0600, Stephen Warren wrote:
> On 05/04/2012 12:58 PM, Mark Brown wrote:
> > Quite a few reference platforms (including Wolfson ones, which is why
> > I'm particularly interested) use replaceable modules to allow
> > configuration changes.  Since we can often identify the configuration at
> > runtime we should ideally do that but currently there's no infrastructure 
> > to help with that...
> 
> So, I'll respond within the context of device tree, although perhaps you
> were looking for something more general?
> 
> I was just asked basically the same question internally to NVIDIA. One
> option that was floated was to store the device tree in chunks and have
> the bootloader piece them together. You'd start with the DT for the
> basic CPU board, probe what HW was available, and then graft in the
> content of additional DT chunks and pass the final result to the kernel.
> The advantages here are:
> 
> a) The DT is stored in chunks for each plugin board, so there's no bloat
> in the DT that gets passed to the kernel; it contains exactly what's on
> the board.

Interesting, but how does it sort ofu things like mapping GPIO lines from
the add-in board's view to the rest of the system?


-- 
Ben Dooks, ben@fluff.org, http://www.fluff.org/ben/

Large Hadron Colada: A large Pina Colada that makes the universe disappear.

Re: Handling of modular boards

[Stephen Warren]

On 05/10/2012 04:43 AM, Ben Dooks wrote:
> On Fri, May 04, 2012 at 01:50:01PM -0600, Stephen Warren wrote:
>> On 05/04/2012 12:58 PM, Mark Brown wrote:
>>> Quite a few reference platforms (including Wolfson ones, which is why
>>> I'm particularly interested) use replaceable modules to allow
>>> configuration changes.  Since we can often identify the configuration at
>>> runtime we should ideally do that but currently there's no infrastructure 
>>> to help with that...
>>
>> So, I'll respond within the context of device tree, although perhaps you
>> were looking for something more general?
>>
>> I was just asked basically the same question internally to NVIDIA. One
>> option that was floated was to store the device tree in chunks and have
>> the bootloader piece them together. You'd start with the DT for the
>> basic CPU board, probe what HW was available, and then graft in the
>> content of additional DT chunks and pass the final result to the kernel.
>> The advantages here are:
>>
>> a) The DT is stored in chunks for each plugin board, so there's no bloat
>> in the DT that gets passed to the kernel; it contains exactly what's on
>> the board.
> 
> Interesting, but how does it sort ofu things like mapping GPIO lines from
> the add-in board's view to the rest of the system?

To be fully general, we'd need to have some kind of proxy GPIO object
that always exists on the main board, for the plugin boards to provide
GPIOs to, or consume GPIOs from.

The simple case of a GPIO provider being on the main board and the
consumer being on a plugin board doesn't need this. The case of the GPIO
provider being on a plugin board, and the only GPIO consumer being on
the main board might not need this.

But if the GPIO provider is on one plugin board, and the GPIO consumer
on another, we'd want to have the DT chunks for each plugin board be
completely independent, so you'd need to route everything through
something that always exists, in the motherboard's DT. I haven't really
thought how that would look yet.

I think this is probably true irrespective of whether the bootloader is
merging the DT chunks, or the kernel did it during boot, or any other
way of constructing the final complete DT.

Re: Handling of modular boards

[Russell King - ARM Linux]

On Fri, May 04, 2012 at 07:58:51PM +0100, Mark Brown wrote:
> Quite a few reference platforms (including Wolfson ones, which is why
> I'm particularly interested) use replaceable modules to allow
> configuration changes.  Since we can often identify the configuration at
> runtime we should ideally do that but currently there's no infrastructure 
> to help with that, generally this seems to be done in arch code for the
> machine but this doesn't scale when even the CPU might change and isn't
> terribly device tree compatible either.
> 
> For reference the code for current Wolfson plugin modules is in
> arch/arm/mach-s3c64xx/mach-crag6410-module.c.
> 
> The most obvious current fit here is the MFD subsystem but it feels like
> we need some slightly different infastructure to what MFD currently
> provides.  MFD is really set up to handle platform devices with a core
> and linear ranges of resources fanning out from that core since they're
> really oriented around chips.  In contrast these boards are more about
> remapping random collections of potentially unrelated resources and
> instantiating devices on all sorts of buses and share more with board
> files.
> 
> I'm just starting to put some stuff together for this so I was wondering
> if anyone had been thinking about this and had any bright ideas for how
> to handle it, and also if people think that MFD is a good fit for this
> or if we should split the silicon MFDs from these PCBs.

I don't think its true to say that there's no support for this kind of
thing.

If you're thinking about a motherboard with separate add-on cards, then
you can view the cards as their own separate platform device.  Your
platform device driver would be a "whole board driver" responsible
for creating and registering the specific devices found on the board
in its probe function, and unregistering them in the remove function.

This is exactly how I've now setup the Assabet with Neponset daughter
board - all the Neponset devices sit behind the Neponset platform
device, including things like the SA1111 multifunction chip and SMC91x
network interface.  I can bind and unbind the Neponset device from its
driver and have everything behind that driver appear and disappear
appropriately - which is useful if your modules can be hotplugged.

It also helps to give the right model to the power management support,
because you're automatically arranging the child devices below the
board-level device, which means all the child devices should be
suspended before the board level device, and the board level device
should be resumed before the child devices.

Re: Handling of modular boards

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1834 bytes --]

On Fri, May 04, 2012 at 11:55:14PM +0100, Russell King - ARM Linux wrote:
> On Fri, May 04, 2012 at 07:58:51PM +0100, Mark Brown wrote:

> > I'm just starting to put some stuff together for this so I was wondering
> > if anyone had been thinking about this and had any bright ideas for how
> > to handle it, and also if people think that MFD is a good fit for this
> > or if we should split the silicon MFDs from these PCBs.

> I don't think its true to say that there's no support for this kind of
> thing.

> If you're thinking about a motherboard with separate add-on cards, then
> you can view the cards as their own separate platform device.  Your
> platform device driver would be a "whole board driver" responsible
> for creating and registering the specific devices found on the board
> in its probe function, and unregistering them in the remove function.

Oh, absolutely - there's support there at that level and several boards
doing some or all of this in mainline already.  It's not that you can't
do it, it's that there's a bunch of generic stuff to do with how you map
the resources through to the devices on the modules and describe the
chips that are on the modules for which there's no infrastructure so
everything needs to be hand coded on a per board basis.  The board
identification bits are board specific but the remapping and subdevice
instantiation bits seem like they shouldn't be.

> It also helps to give the right model to the power management support,
> because you're automatically arranging the child devices below the
> board-level device, which means all the child devices should be
> suspended before the board level device, and the board level device
> should be resumed before the child devices.

Yes, I'd anticipate that we'd have a device for the board which should
help with this sort of stuff.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Handling of modular boards

[Russell King - ARM Linux]

On Sat, May 05, 2012 at 12:40:57AM +0100, Mark Brown wrote:
> On Fri, May 04, 2012 at 11:55:14PM +0100, Russell King - ARM Linux wrote:
> > On Fri, May 04, 2012 at 07:58:51PM +0100, Mark Brown wrote:
> 
> > > I'm just starting to put some stuff together for this so I was wondering
> > > if anyone had been thinking about this and had any bright ideas for how
> > > to handle it, and also if people think that MFD is a good fit for this
> > > or if we should split the silicon MFDs from these PCBs.
> 
> > I don't think its true to say that there's no support for this kind of
> > thing.
> 
> > If you're thinking about a motherboard with separate add-on cards, then
> > you can view the cards as their own separate platform device.  Your
> > platform device driver would be a "whole board driver" responsible
> > for creating and registering the specific devices found on the board
> > in its probe function, and unregistering them in the remove function.
> 
> Oh, absolutely - there's support there at that level and several boards
> doing some or all of this in mainline already.  It's not that you can't
> do it, it's that there's a bunch of generic stuff to do with how you map
> the resources through to the devices on the modules and describe the
> chips that are on the modules for which there's no infrastructure so
> everything needs to be hand coded on a per board basis.  The board
> identification bits are board specific but the remapping and subdevice
> instantiation bits seem like they shouldn't be.

How about this - we have struct platform_device_info, which is used to
create platform devices.  We can use this as an array to describe what
platform devices to create in the sub-driver, including what the resources
should be etc.

However, there's a problem with this - what if you need to do some board
level init before hand?  That needs to be handled somehow before these
devices are instantiated.  That could be done via a callback through
platform data.

But... this all seems wrong, because rather than having a driver which
knows about the details of the board, we now have all the details of the
board in question back in platform code which originally declared the
board device.  That's wrong, because a daughter board may be shared
between different platforms, and we don't want multiple copies of that
data all around the place.

I don't think there's an easy or generic solution to this.

The best I can think of is ACPI, where they _do_ handle devices and
such like coming and going dynamically (I know this, it happens to my
serial and printer ports every time I dock and undock the laptop - and
stock Linux doesn't support that.  Which reminds me I need to get those
patches properly prepared and submitted...)  I'd assume that it also
deals with PCI bridges and buses behind that if you have one of those
docking stations too (I don't)...

Re: Handling of modular boards

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1844 bytes --]

On Sat, May 05, 2012 at 12:52:25AM +0100, Russell King - ARM Linux wrote:

> How about this - we have struct platform_device_info, which is used to
> create platform devices.  We can use this as an array to describe what
> platform devices to create in the sub-driver, including what the resources
> should be etc.

We (well, I at least) need to handle devices on other buses like I2C and
SPI too but yes, that's the sort of thing I was looking for.

> However, there's a problem with this - what if you need to do some board
> level init before hand?  That needs to be handled somehow before these
> devices are instantiated.  That could be done via a callback through
> platform data.

> But... this all seems wrong, because rather than having a driver which
> knows about the details of the board, we now have all the details of the
> board in question back in platform code which originally declared the
> board device.  That's wrong, because a daughter board may be shared
> between different platforms, and we don't want multiple copies of that
> data all around the place.

> I don't think there's an easy or generic solution to this.

I think that's OK - if there's any init stuff that needs to be done on a
prior to identification of the board then presumably it's a generic
thing for the motherboard which will apply to any plugin module on that
board and can be done as part of the normal board init.  If the init
needs to be done after identification of the board then probably it
applies to any motherboard the board might be plugged in to so we can
just define callbacks for the plugin module that can be part of the
plugin module description.

Cases that depend on a specific combination will doubtless exist and do
have the problems you describe but are probably less frequent but I
think we can go a long way on the first two.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]