Building kernel for more than one SoC

Building kernel for more than one SoC

[Grant Edwards]

I'm told that it should be possible to build a kernel that will run on
two different SoC chips.  They're closely related (same ARM9 core,
many identical internal peripherals -- AT91SAM9G20 and 'G25), and
would likely have identical external hardware.

In order to handle the internal periphals that differ, it was
recommended that I use loadable modules to keep the kernel size small.
However, my root filesystem is in RAM, so I don't see how loadable
modules helps unless I remove all of the .ko files from the root
filesystem after the kernel has booted.

It seems it would be simpler to just link in all required drivers for
both chips and discard the ones that aren't needed after kernel
initialization. But, I'm not sure if there's a mechanism for doing
that.  I know there's a way to declare a function or data that will be
discarded after kernel init, but is ther a way to that conditionally
depending on probed hardware or the device-tree used at boot-time?

-- 
Grant Edwards               grant.b.edwards        Yow! Let me do my TRIBUTE
                                  at               to FISHNET STOCKINGS ...
                              gmail.com            

Building kernel for more than one SoC

[Thomas Petazzoni]

Dear Grant Edwards,

(Adding Boris and Alexandre in Cc, since they do a lot of AT91 stuff
these days.)

On Thu, 31 Jul 2014 13:59:36 +0000 (UTC), Grant Edwards wrote:

> I'm told that it should be possible to build a kernel that will run on
> two different SoC chips.  They're closely related (same ARM9 core,
> many identical internal peripherals -- AT91SAM9G20 and 'G25), and
> would likely have identical external hardware.
> 
> In order to handle the internal periphals that differ, it was
> recommended that I use loadable modules to keep the kernel size small.
> However, my root filesystem is in RAM, so I don't see how loadable
> modules helps unless I remove all of the .ko files from the root
> filesystem after the kernel has booted.
> 
> It seems it would be simpler to just link in all required drivers for
> both chips and discard the ones that aren't needed after kernel
> initialization. But, I'm not sure if there's a mechanism for doing
> that.  I know there's a way to declare a function or data that will be
> discarded after kernel init, but is ther a way to that conditionally
> depending on probed hardware or the device-tree used at boot-time?

I don't think there's much point in worrying about this: the 9G20 and
9G25 will use identical device drivers for the vast majority of the
hardware blocks of the SoC, so the overhead of having the drivers for
both SoCs inside the same kernel is going to be really low.

Make a test, a build a kernel only for 9G20, another only for 9G25, and
another with both.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Building kernel for more than one SoC

[Grant Edwards]

On 2014-07-31, Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote:

> I don't think there's much point in worrying about this: the 9G20 and
> 9G25 will use identical device drivers for the vast majority of the
> hardware blocks of the SoC,

Really identical drivers, or just compiled from the same source file?
One thing I'm worried about is that they might share a driver that has
code conditionally compiled one way for the G20 and the other for the
G25.  I haven't _found_ any code like that yet, but it would be a pain
to have to deal with something like that...

-- 
Grant Edwards               grant.b.edwards        Yow! I want to dress you
                                  at               up as TALLULAH BANKHEAD and
                              gmail.com            cover you with VASELINE and
                                                   WHEAT THINS ...

Building kernel for more than one SoC

[Russell King - ARM Linux]

On Thu, Jul 31, 2014 at 01:59:36PM +0000, Grant Edwards wrote:
> I'm told that it should be possible to build a kernel that will run on
> two different SoC chips.  They're closely related (same ARM9 core,
> many identical internal peripherals -- AT91SAM9G20 and 'G25), and
> would likely have identical external hardware.
> 
> In order to handle the internal periphals that differ, it was
> recommended that I use loadable modules to keep the kernel size small.
> However, my root filesystem is in RAM, so I don't see how loadable
> modules helps unless I remove all of the .ko files from the root
> filesystem after the kernel has booted.

I think whoever made that recommendation probably isn't aware of the
direction that mainline kernels are heading.

Where we're going with mainline kernels is to have a set of drivers
which work irrespective of the hardware you have.

We've actually had this policy for about the last decade - we've
supported building the same family of SoCs into one single kernel
(identified by their mach-* directory) and expecting that their
drivers will cope with the differences between the SoCs at runtime.
(Note that the CPU itself has never really come into it; we've had
good abstractions for the CPU support since the late 90's, with the
exception of a borderline between the ARMv5 and ARMv6 architectures.)

Whether the Atmel code does that or not, I don't know, but it _should_
do it.

Over the last few years, we've been moving mainline towards even
tighter integration, where it's possible to build completely
dissimilar SoCs together, so ultimately we end up with: legacy kernels,
one multi-platform ARMv5 and earlier kernel, and one multi-platform
ARMv6 and later kernel.

> It seems it would be simpler to just link in all required drivers for
> both chips and discard the ones that aren't needed after kernel
> initialization.

Even better is to abstract the differences and have the same driver
code deal with the different variants internally - the selection of
which variant being controlled via the device tree "compatible"
property, or another appropriate method.

-- 
FTTC broadband for 0.8mile line: currently at 9.5Mbps down 400kbps up
according to speedtest.net.

Building kernel for more than one SoC

[Nicolas Ferre]

On 04/08/2014 22:17, Russell King - ARM Linux :
> On Thu, Jul 31, 2014 at 01:59:36PM +0000, Grant Edwards wrote:
>> I'm told that it should be possible to build a kernel that will run on
>> two different SoC chips.  They're closely related (same ARM9 core,
>> many identical internal peripherals -- AT91SAM9G20 and 'G25), and
>> would likely have identical external hardware.
>>
>> In order to handle the internal periphals that differ, it was
>> recommended that I use loadable modules to keep the kernel size small.
>> However, my root filesystem is in RAM, so I don't see how loadable
>> modules helps unless I remove all of the .ko files from the root
>> filesystem after the kernel has booted.
> 
> I think whoever made that recommendation probably isn't aware of the
> direction that mainline kernels are heading.
> 
> Where we're going with mainline kernels is to have a set of drivers
> which work irrespective of the hardware you have.
> 
> We've actually had this policy for about the last decade - we've
> supported building the same family of SoCs into one single kernel
> (identified by their mach-* directory) and expecting that their
> drivers will cope with the differences between the SoCs at runtime.
> (Note that the CPU itself has never really come into it; we've had
> good abstractions for the CPU support since the late 90's, with the
> exception of a borderline between the ARMv5 and ARMv6 architectures.)
> 
> Whether the Atmel code does that or not, I don't know, but it _should_
> do it.

Absolutely, we do follow this policy for quite some time. So anything
that prevents what Russell describes above should be considered as a bug ;-)

> Over the last few years, we've been moving mainline towards even
> tighter integration, where it's possible to build completely
> dissimilar SoCs together, so ultimately we end up with: legacy kernels,
> one multi-platform ARMv5 and earlier kernel, and one multi-platform
> ARMv6 and later kernel.

We are almost at the point where we can have a multi-platform ARMv6/7
for sama5 SoCs and ARMv5 for our ARM9s.

>> It seems it would be simpler to just link in all required drivers for
>> both chips and discard the ones that aren't needed after kernel
>> initialization.
> 
> Even better is to abstract the differences and have the same driver
> code deal with the different variants internally - the selection of
> which variant being controlled via the device tree "compatible"
> property, or another appropriate method.

Sure. Everything in at91 is converted to device tree those days.

Best regards,
-- 
Nicolas Ferre

Building kernel for more than one SoC

[Grant Edwards]

On 2014-08-11, Nicolas Ferre <nicolas.ferre@atmel.com> wrote:
> On 04/08/2014 22:17, Russell King - ARM Linux :

[...]

>> Even better is to abstract the differences and have the same driver
>> code deal with the different variants internally - the selection of
>> which variant being controlled via the device tree "compatible"
>> property, or another appropriate method.
>
> Sure. Everything in at91 is converted to device tree those days.

That's good to hear, thanks.

Now it's up to somebody else to decide if the price difference between
a G20 and G25 is worth the engineering time to upgrade U-Boot and
Linux kernel to versions that know about device trees...

-- 
Grant Edwards               grant.b.edwards        Yow! I smell a RANCID
                                  at               CORN DOG!
                              gmail.com            

Building kernel for more than one SoC

[Robert Nelson]

> Now it's up to somebody else to decide if the price difference between
> a G20 and G25 is worth the engineering time to upgrade U-Boot and
> Linux kernel to versions that know about device trees...

http://cateee.net/lkddb/web-lkddb/ARM_APPENDED_DTB.html

Regards,

-- 
Robert Nelson
http://www.rcn-ee.com/

Building kernel for more than one SoC

[Grant Edwards]

On 2014-08-11, Robert Nelson <robertcnelson@gmail.com> wrote:
>> Now it's up to somebody else to decide if the price difference between
>> a G20 and G25 is worth the engineering time to upgrade U-Boot and
>> Linux kernel to versions that know about device trees...
>
> http://cateee.net/lkddb/web-lkddb/ARM_APPENDED_DTB.html

Interesting.  That would still require modifying U-Boot so that at
run-time it detects the SoC type and appends the proper DTB to the
kernel image, but it that may be less work than "real" DTB support in
U-Boot.

-- 
Grant Edwards               grant.b.edwards        Yow! Now KEN and BARBIE
                                  at               are PERMANENTLY ADDICTED to
                              gmail.com            MIND-ALTERING DRUGS ...

Building kernel for more than one SoC

[Russell King - ARM Linux]

On Mon, Aug 11, 2014 at 08:43:35PM +0000, Grant Edwards wrote:
> On 2014-08-11, Robert Nelson <robertcnelson@gmail.com> wrote:
> >> Now it's up to somebody else to decide if the price difference between
> >> a G20 and G25 is worth the engineering time to upgrade U-Boot and
> >> Linux kernel to versions that know about device trees...
> >
> > http://cateee.net/lkddb/web-lkddb/ARM_APPENDED_DTB.html
> 
> Interesting.  That would still require modifying U-Boot so that at
> run-time it detects the SoC type and appends the proper DTB to the
> kernel image, but it that may be less work than "real" DTB support in
> U-Boot.

The idea of that feature is:

- You take the kernel zImage
- You take the appropriate dtb file
- You concatenate the dtb file into the zImage
- You run mkimage on the resulting combined image to create the special
  uboot format file for uboot to load
- You use it with uboot as you have done in the past with non-DT kernels.

-- 
FTTC broadband for 0.8mile line: currently at 9.5Mbps down 400kbps up
according to speedtest.net.

Building kernel for more than one SoC

[Grant Edwards]

On 2014-08-11, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
> On Mon, Aug 11, 2014 at 08:43:35PM +0000, Grant Edwards wrote:
>> On 2014-08-11, Robert Nelson <robertcnelson@gmail.com> wrote:
>> >> Now it's up to somebody else to decide if the price difference between
>> >> a G20 and G25 is worth the engineering time to upgrade U-Boot and
>> >> Linux kernel to versions that know about device trees...
>> >
>> > http://cateee.net/lkddb/web-lkddb/ARM_APPENDED_DTB.html
>> 
>> Interesting.  That would still require modifying U-Boot so that at
>> run-time it detects the SoC type and appends the proper DTB to the
>> kernel image, but it that may be less work than "real" DTB support in
>> U-Boot.
>
> The idea of that feature is:
>
> - You take the kernel zImage
> - You take the appropriate dtb file
> - You concatenate the dtb file into the zImage
> - You run mkimage on the resulting combined image to create the special
>   uboot format file for uboot to load

The problem is now you've got a kernel image that won't run on both
the '9g20 and the '9g25.  The requirement is to have a kernel image
that will run on either.

> - You use it with uboot as you have done in the past with non-DT
>   kernels.

Logistically, there's little difference between that and compiling the
kernel twice.  It's more elegant than compiling the kernel twice, but 
in the end it requires the maintenance of two separate kernel images
and some way for customers to figure out which one they should
download (and no matter what you do, when given a choice between two
files, they will download and attempt to install the wrong one more
than half of the time).

-- 
Grant Edwards               grant.b.edwards        Yow! As President I have
                                  at               to go vacuum my coin
                              gmail.com            collection!

Building kernel for more than one SoC

[Robert Nelson]

On Mon, Aug 11, 2014 at 4:15 PM, Grant Edwards
<grant.b.edwards@gmail.com> wrote:
> On 2014-08-11, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
>> On Mon, Aug 11, 2014 at 08:43:35PM +0000, Grant Edwards wrote:
>>> On 2014-08-11, Robert Nelson <robertcnelson@gmail.com> wrote:
>>> >> Now it's up to somebody else to decide if the price difference between
>>> >> a G20 and G25 is worth the engineering time to upgrade U-Boot and
>>> >> Linux kernel to versions that know about device trees...
>>> >
>>> > http://cateee.net/lkddb/web-lkddb/ARM_APPENDED_DTB.html
>>>
>>> Interesting.  That would still require modifying U-Boot so that at
>>> run-time it detects the SoC type and appends the proper DTB to the
>>> kernel image, but it that may be less work than "real" DTB support in
>>> U-Boot.
>>
>> The idea of that feature is:
>>
>> - You take the kernel zImage
>> - You take the appropriate dtb file
>> - You concatenate the dtb file into the zImage
>> - You run mkimage on the resulting combined image to create the special
>>   uboot format file for uboot to load
>
> The problem is now you've got a kernel image that won't run on both
> the '9g20 and the '9g25.  The requirement is to have a kernel image
> that will run on either.

Then, upgrade your u-boot to mainline, use the dtb, etc.  You have a
lot of excuses. ;)

Regards,

-- 
Robert Nelson
http://www.rcn-ee.com/

Building kernel for more than one SoC

[Grant Edwards]

On 2014-08-11, Robert Nelson <robertcnelson@gmail.com> wrote:
> On Mon, Aug 11, 2014 at 4:15 PM, Grant Edwards
><grant.b.edwards@gmail.com> wrote:
>> On 2014-08-11, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
>>> On Mon, Aug 11, 2014 at 08:43:35PM +0000, Grant Edwards wrote:
>>>> On 2014-08-11, Robert Nelson <robertcnelson@gmail.com> wrote:
>>>> >> Now it's up to somebody else to decide if the price difference between
>>>> >> a G20 and G25 is worth the engineering time to upgrade U-Boot and
>>>> >> Linux kernel to versions that know about device trees...
>>>> >
>>>> > http://cateee.net/lkddb/web-lkddb/ARM_APPENDED_DTB.html
>>>>
>>>> Interesting.  That would still require modifying U-Boot so that at
>>>> run-time it detects the SoC type and appends the proper DTB to the
>>>> kernel image, but it that may be less work than "real" DTB support in
>>>> U-Boot.
>>>
>>> The idea of that feature is:
>>>
>>> - You take the kernel zImage
>>> - You take the appropriate dtb file
>>> - You concatenate the dtb file into the zImage
>>> - You run mkimage on the resulting combined image to create the special
>>>   uboot format file for uboot to load
>>
>> The problem is now you've got a kernel image that won't run on both
>> the '9g20 and the '9g25.  The requirement is to have a kernel image
>> that will run on either.
>
> Then, upgrade your u-boot to mainline, use the dtb, etc.  You have a
> lot of excuses. ;)

Yep. Unfortunately, excuses aren't the problem.  Cost (mostly
opportunity cost) is the problem...

-- 
Grant Edwards               grant.b.edwards        Yow! Is it NOUVELLE
                                  at               CUISINE when 3 olives are
                              gmail.com            struggling with a scallop
                                                   in a plate of SAUCE MORNAY?

Building kernel for more than one SoC

[Russell King - ARM Linux]

On Mon, Aug 11, 2014 at 09:15:22PM +0000, Grant Edwards wrote:
> On 2014-08-11, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
> > On Mon, Aug 11, 2014 at 08:43:35PM +0000, Grant Edwards wrote:
> >> On 2014-08-11, Robert Nelson <robertcnelson@gmail.com> wrote:
> >> >> Now it's up to somebody else to decide if the price difference between
> >> >> a G20 and G25 is worth the engineering time to upgrade U-Boot and
> >> >> Linux kernel to versions that know about device trees...
> >> >
> >> > http://cateee.net/lkddb/web-lkddb/ARM_APPENDED_DTB.html
> >> 
> >> Interesting.  That would still require modifying U-Boot so that at
> >> run-time it detects the SoC type and appends the proper DTB to the
> >> kernel image, but it that may be less work than "real" DTB support in
> >> U-Boot.
> >
> > The idea of that feature is:
> >
> > - You take the kernel zImage
> > - You take the appropriate dtb file
> > - You concatenate the dtb file into the zImage
> > - You run mkimage on the resulting combined image to create the special
> >   uboot format file for uboot to load
> 
> The problem is now you've got a kernel image that won't run on both
> the '9g20 and the '9g25.  The requirement is to have a kernel image
> that will run on either.

It depends what you call a kernel image.

As far as I'm concerned (and as I've been concerned from day one of uboot
coming into ARM), the kernel image is the zImage, not the crap that uboot
decides to dictate that you must provide for it to use.

I've been pretty clear over the years that I utterly despise uboot's
custom format - and you're starting to find out why.  Welcome to the
inflexibility has caused. :)

> > - You use it with uboot as you have done in the past with non-DT
> >   kernels.
> 
> Logistically, there's little difference between that and compiling the
> kernel twice.  It's more elegant than compiling the kernel twice, but 
> in the end it requires the maintenance of two separate kernel images
> and some way for customers to figure out which one they should
> download (and no matter what you do, when given a choice between two
> files, they will download and attempt to install the wrong one more
> than half of the time).

While you have a point there, that's a choice of how you do your kernel
upgrades.

If you supply a zImage, all the dtbs, a script which does the
programming of the kernel onto the target, and a copy of mkimage, then
you can do all the steps I've highlighted above on the target - without
the customer even having to know what platform they're on, because your
script can work it out.

If you don't have a shell on the target, then write a C program to do it
instead.

There's plenty of workarounds possible for the old uboot dilemma...

-- 
FTTC broadband for 0.8mile line: currently at 9.5Mbps down 400kbps up
according to speedtest.net.

Building kernel for more than one SoC

[Grant Edwards]

On 2014-08-11, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
>
>> The problem is now you've got a kernel image that won't run on both
>> the '9g20 and the '9g25.  The requirement is to have a kernel image
>> that will run on either.
>
> It depends what you call a kernel image.
>
> As far as I'm concerned (and as I've been concerned from day one of uboot
> coming into ARM), the kernel image is the zImage, not the crap that uboot
> decides to dictate that you must provide for it to use.
>
> I've been pretty clear over the years that I utterly despise uboot's
> custom format - and you're starting to find out why.  Welcome to the
> inflexibility has caused. :)

Yea, I've got my own issues with U-Boot, but that's a whole 'nother
thread.  In the end, it was a lot less painful to put up with U-Boot's
issues than it was to write/port something else.

> While you have a point there, that's a choice of how you do your
> kernel upgrades.
>
> If you supply a zImage, all the dtbs, a script which does the
> programming of the kernel onto the target, and a copy of mkimage,
> then you can do all the steps I've highlighted above on the target -
> without the customer even having to know what platform they're on,
> because your script can work it out.

Good point.

> There's plenty of workarounds possible for the old uboot dilemma...

Definitely.  It all comes to do trying to figure out when the
work-arounds add up to more work than doing it the "right" way and
upgrading everything.

-- 
Grant Edwards               grant.b.edwards        Yow! All of life is a blur
                                  at               of Republicans and meat!
                              gmail.com            

Building kernel for more than one SoC

[Tomasz Figa]

On 12.08.2014 01:02, Grant Edwards wrote:
> On 2014-08-11, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
>>
>>> The problem is now you've got a kernel image that won't run on both
>>> the '9g20 and the '9g25.  The requirement is to have a kernel image
>>> that will run on either.
>>
>> It depends what you call a kernel image.
>>
>> As far as I'm concerned (and as I've been concerned from day one of uboot
>> coming into ARM), the kernel image is the zImage, not the crap that uboot
>> decides to dictate that you must provide for it to use.
>>
>> I've been pretty clear over the years that I utterly despise uboot's
>> custom format - and you're starting to find out why.  Welcome to the
>> inflexibility has caused. :)
> 
> Yea, I've got my own issues with U-Boot, but that's a whole 'nother
> thread.  In the end, it was a lot less painful to put up with U-Boot's
> issues than it was to write/port something else.
> 
>> While you have a point there, that's a choice of how you do your
>> kernel upgrades.
>>
>> If you supply a zImage, all the dtbs, a script which does the
>> programming of the kernel onto the target, and a copy of mkimage,
>> then you can do all the steps I've highlighted above on the target -
>> without the customer even having to know what platform they're on,
>> because your script can work it out.
> 
> Good point.
> 
>> There's plenty of workarounds possible for the old uboot dilemma...
> 
> Definitely.  It all comes to do trying to figure out when the
> work-arounds add up to more work than doing it the "right" way and
> upgrading everything.
> 

Is this maybe what you're looking for?

https://github.com/zonque/pxa-impedance-matcher

Best regards,
Tomasz

Building kernel for more than one SoC

[Olof Johansson]

On Mon, Aug 11, 2014 at 3:43 PM, Russell King - ARM Linux
<linux@arm.linux.org.uk> wrote:
> On Mon, Aug 11, 2014 at 09:15:22PM +0000, Grant Edwards wrote:
>> On 2014-08-11, Russell King - ARM Linux <linux@arm.linux.org.uk> wrote:
>> > On Mon, Aug 11, 2014 at 08:43:35PM +0000, Grant Edwards wrote:
>> >> On 2014-08-11, Robert Nelson <robertcnelson@gmail.com> wrote:
>> >> >> Now it's up to somebody else to decide if the price difference between
>> >> >> a G20 and G25 is worth the engineering time to upgrade U-Boot and
>> >> >> Linux kernel to versions that know about device trees...
>> >> >
>> >> > http://cateee.net/lkddb/web-lkddb/ARM_APPENDED_DTB.html
>> >>
>> >> Interesting.  That would still require modifying U-Boot so that at
>> >> run-time it detects the SoC type and appends the proper DTB to the
>> >> kernel image, but it that may be less work than "real" DTB support in
>> >> U-Boot.
>> >
>> > The idea of that feature is:
>> >
>> > - You take the kernel zImage
>> > - You take the appropriate dtb file
>> > - You concatenate the dtb file into the zImage
>> > - You run mkimage on the resulting combined image to create the special
>> >   uboot format file for uboot to load
>>
>> The problem is now you've got a kernel image that won't run on both
>> the '9g20 and the '9g25.  The requirement is to have a kernel image
>> that will run on either.
>
> It depends what you call a kernel image.
>
> As far as I'm concerned (and as I've been concerned from day one of uboot
> coming into ARM), the kernel image is the zImage, not the crap that uboot
> decides to dictate that you must provide for it to use.
>
> I've been pretty clear over the years that I utterly despise uboot's
> custom format - and you're starting to find out why.  Welcome to the
> inflexibility has caused. :)

Note that u-boot grew a "bootz" command in recent year(s). It'll boot
just a raw zImage instead of the mkimage-wrapped version.

Of course, not all u-boots have it due to age of the vendor fork. I
also have a board that tries to boot a zImage with the 'bootm' command
instead. Yes, you heard that right. Sigh.


-Olof