Appended devicetree with Torvalds' HEAD

Appended devicetree with Torvalds' HEAD

[Linus Walleij]

Hi, I'm new to device tree work but I'm told to
help out.

Does anyone happen to run a mainline kernel with
zImage-appended device tree?

Or:

Are there out-of-tree patches required to do this?
(I guess so.)

And:

Do they actually work?

I'm currently living under the assumption that
absolutely noone is actually using appended device
trees, and that everyone doing device tree work get
their device trees from their U-Boot. I wonder if this
is a correct observation.

Yours,
Linus Walleij

Re: Appended devicetree with Torvalds' HEAD

[Shawn Guo]

On Tue, Sep 06, 2011 at 11:35:53AM +0200, Linus Walleij wrote:
> Hi, I'm new to device tree work but I'm told to
> help out.
> 
> Does anyone happen to run a mainline kernel with
> zImage-appended device tree?
> 
> Or:
> 
> Are there out-of-tree patches required to do this?
> (I guess so.)
> 
http://thread.gmane.org/gmane.linux.drivers.devicetree/6044/match=allow+dtb+appended+arm+zImage+atag+conversion

> And:
> 
> Do they actually work?
> 
Yes, with a small fix on patch #4 given by David Brown in the thread.

Regards,
Shawn

Re: Appended devicetree with Torvalds' HEAD

[Nicolas Pitre]

On Tue, 6 Sep 2011, Linus Walleij wrote:

> Hi, I'm new to device tree work but I'm told to
> help out.
> 
> Does anyone happen to run a mainline kernel with
> zImage-appended device tree?
> 
> Or:
> 
> Are there out-of-tree patches required to do this?
> (I guess so.)
> 
> And:
> 
> Do they actually work?

They sure do, (or so I hope).

My latest patchset is here:

  git://git.linaro.org/people/nico/linux zImage_DTB_append

I'm not fully satisfied to send them upstream yet though.


Nicolas

Re: Appended devicetree with Torvalds' HEAD

[Linus Walleij]

On Tue, Sep 6, 2011 at 7:52 PM, Nicolas Pitre <nicolas.pitre-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org> wrote:

> They sure do, (or so I hope).
> My latest patchset is here:

OK I have cloned your git and already made some minor adjustments,
will return with more details when/if I get this going.

Thanks!
Linus Walleij

Re: Appended devicetree with Torvalds' HEAD

[Linus Walleij]

On Tue, Sep 6, 2011 at 9:24 PM, Linus Walleij <linus.walleij-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org> wrote:
> On Tue, Sep 6, 2011 at 7:52 PM, Nicolas Pitre <nicolas.pitre-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org> wrote:
>
>> They sure do, (or so I hope).
>> My latest patchset is here:
>
> OK I have cloned your git and already made some minor adjustments,
> will return with more details when/if I get this going.

Hm it doesn't boot, if I configure out the DT support it
works fine. (On v3.1-rc4 and ux500) How typical :-/

I tried to regression-test it on a legacy system but then
I ran into a problem that may be serious: that platform does
not use a compressed zImage, it users the plain vmlinux
converted to binary with objcopy.

Is there a particular reason why we want to append the
device tree to the zImage rather than to vmlinux?

Yours,
Linus Walleij

Re: Appended devicetree with Torvalds' HEAD

[Nicolas Pitre]

On Thu, 8 Sep 2011, Linus Walleij wrote:

> On Tue, Sep 6, 2011 at 9:24 PM, Linus Walleij <linus.walleij-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org> wrote:
> > On Tue, Sep 6, 2011 at 7:52 PM, Nicolas Pitre <nicolas.pitre-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org> wrote:
> >
> >> They sure do, (or so I hope).
> >> My latest patchset is here:
> >
> > OK I have cloned your git and already made some minor adjustments,
> > will return with more details when/if I get this going.
> 
> Hm it doesn't boot, if I configure out the DT support it
> works fine. (On v3.1-rc4 and ux500) How typical :-/
> 
> I tried to regression-test it on a legacy system but then
> I ran into a problem that may be serious: that platform does
> not use a compressed zImage, it users the plain vmlinux
> converted to binary with objcopy.
> 
> Is there a particular reason why we want to append the
> device tree to the zImage rather than to vmlinux?

Absolutely.

The ultimate goal of DT is to separate the board configuration data from 
the kernel image.  This is why we expect all bootloaders to provide this 
DTB eventually, so that the kernel could be upgraded at will, and even 
booted on future boards unchanged.

So this ability to simply append the DTB to a zImage is just a 
convenience to accommodate legacy bootloaders.  It should _not_ become 
the primary way of providing a DTB to the kernel.  At least by appending 
it to zImage it can be done independently from the kernel build process 
and performed right before the kernel is stored on the boot media.


Nicolas

Re: Appended devicetree with Torvalds' HEAD

[Linus Walleij]

On Thu, Sep 8, 2011 at 6:35 PM, Nicolas Pitre <nicolas.pitre-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org> wrote:
> On Thu, 8 Sep 2011, Linus Walleij wrote:
>>
>> Is there a particular reason why we want to append the
>> device tree to the zImage rather than to vmlinux?
>
> So this ability to simply append the DTB to a zImage is just a
> convenience to accommodate legacy bootloaders.  It should _not_ become
> the primary way of providing a DTB to the kernel.

I understand this, however this one legacy bootloader for U300
does not support compressed images, just plain vmlinux.

I intended to use it for some cross-testing of the append patches
since they wouldn't work for me on other hardware.

I'll just drop that attempt, no big deal.

Thanks,
Linus Walleij

Re: Appended devicetree with Torvalds' HEAD

[David Brown]

On Tue, Sep 06, 2011 at 01:52:43PM -0400, Nicolas Pitre wrote:

> They sure do, (or so I hope).
> 
> My latest patchset is here:
> 
>   git://git.linaro.org/people/nico/linux zImage_DTB_append
> 
> I'm not fully satisfied to send them upstream yet though.

I'll give these a try, once I'm back from LPC and let you know if they
still work.

David

-- 
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.

Re: Appended devicetree with Torvalds' HEAD

[Linus Walleij]

On Tue, Sep 6, 2011 at 7:52 PM, Nicolas Pitre <nicolas.pitre-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org> wrote:
> On Tue, 6 Sep 2011, Linus Walleij wrote:
>
>> Does anyone happen to run a mainline kernel with
>> zImage-appended device tree?
>>
>> Or:
>>
>> Are there out-of-tree patches required to do this?
>> (I guess so.)
>>
>> And:
>>
>> Do they actually work?
>
> They sure do, (or so I hope).
>
> My latest patchset is here:

I've tested this a bit now with U8500 and RealView
PB1176.

I'm using the clean patches on a clean tree, i.e.
just relying on the code already in mainline.

We add this to the MACHINE_START()

static const char * realview_dt_board_compat[] = {
       "arm,realview-pb1176",
       NULL
};

.dt_compat      = realview_dt_board_compat,

On kernel 3.1-rc4 the system hangs in early boot
(early printk does not help). But this is not related to
these patches at all, it happens as soon as you enable
CONFIG_USE_OF. (Any hints on how to fix that?)

So I tested a bit with just a plain v3.0 kernel.

Using the "supplementing DTB with traditional ATAG data"
patch totally hangs my systems, even if I don't enable
CONFIG_ATAG_DTB_COMPAT, so I think that patch
needs some scrutiny.

Removing that patch the system boots, and I can use
CONFIG_USE_OF with CONFIG_PROC_DEVICETREE
to inspect an empty tree in /proc/device-tree

If I then try to enable these patches it dies
by not finding any "compatible" property. (But it boots,
yay!)

Error: unrecognized/unsupported device tree compatible list:
[ ]
Available machine support:
ID (hex)	NAME
ffffffff	ARM-RealView PB1176
Please check your kernel config and/or bootloader.

(I know the MACHINE_TYPE isn't sane.)

Anyway, so the
prop = of_get_flat_dt_prop(dt_root, "compatible", &size);
returns NULL.

So it appears that the appended device tree does not
show up.

I don't quite know how to debug this further...
Given that the appended approach is generally
frowned upon maybe we are better off trying to fix
our U-Boot, getting a combo that is known to work
with device tree, than trying to get appended device
trees to work?

Yours,
Linus Walleij

Re: Appended devicetree with Torvalds' HEAD

[Nicolas Pitre]

On Mon, 12 Sep 2011, Linus Walleij wrote:

> So it appears that the appended device tree does not
> show up.
> 
> I don't quite know how to debug this further...

I'm in the process of refreshing those patches, bringing them forward 
and cleaning up the parts that I dislike in the process.  I will 
certainly look at your failure case.

> Given that the appended approach is generally
> frowned upon maybe we are better off trying to fix
> our U-Boot, getting a combo that is known to work
> with device tree, than trying to get appended device
> trees to work?

It still has to minimally work as some bootloaders out there simply 
can't be updated at all.


Nicolas

Re: Appended devicetree with Torvalds' HEAD

[Nicolas Pitre]

On Mon, 12 Sep 2011, Domenico Andreoli wrote:

> On Mon, Sep 12, 2011 at 09:25:26AM -0400, Nicolas Pitre wrote:
> > On Mon, 12 Sep 2011, Linus Walleij wrote:
> > 
> > > So it appears that the appended device tree does not
> > > show up.
> > > 
> > > I don't quite know how to debug this further...
> > 
> > I'm in the process of refreshing those patches, bringing them forward 
> > and cleaning up the parts that I dislike in the process.  I will 
> > certainly look at your failure case.
> > 
> > > Given that the appended approach is generally
> > > frowned upon maybe we are better off trying to fix
> > > our U-Boot, getting a combo that is known to work
> > > with device tree, than trying to get appended device
> > > trees to work?
> > 
> > It still has to minimally work as some bootloaders out there simply 
> > can't be updated at all.
> 
> Appending dtb to the kernel image solves the case of those boot loaders
> that cannot be updated but also reduces the genericity of the so built
> kernel because you then need a kernel+dtb bundle for each affected
> bootloader+board combination.
> 
> Couldn't this drawback be mitigated allowing the append of multiple dtbs?
> Kernel would not be truly generic but surely more than kernel+single dtb.

No.  A line must be drawn somewhere and as I said I don't want this 
feature to become more handy than it strictly has to be.  If you care so 
deeply about 1) device tree and 2) having a generic kernel supporting 
multiple targets, then you should really think about updating your 
bootloader. Otherwise you should simply make the DTB concatenation at 
the very last minute before storing the result onto your boot medium and 
distribute the kernel and the various DTBs separately otherwise.


Nicolas

Re: Appended devicetree with Torvalds' HEAD

[David Brown]

On Tue, Sep 06, 2011 at 01:52:43PM -0400, Nicolas Pitre wrote:

> My latest patchset is here:
> 
>   git://git.linaro.org/people/nico/linux zImage_DTB_append
> 
> I'm not fully satisfied to send them upstream yet though.

These patches do appear to still work on MSM.

BTW, there are still a couple of trailing whitespace characters in the
patch, though.

David

-- 
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.

Re: Appended devicetree with Torvalds' HEAD

[Linus Walleij]

On Tue, Sep 13, 2011 at 8:42 PM, David Brown <davidb-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org> wrote:
> On Tue, Sep 06, 2011 at 01:52:43PM -0400, Nicolas Pitre wrote:
>
>> My latest patchset is here:
>>
>>   git://git.linaro.org/people/nico/linux zImage_DTB_append
>>
>> I'm not fully satisfied to send them upstream yet though.
>
> These patches do appear to still work on MSM.

Hm I wonder what is wrong on our end, probably our own fault :-/

Arey you encapsulating the zImage+DTB in a uImage like I do
(so as to use with elder U-Boot) or are you using some other
method to get the zImage+DTB into the system?

Yours,
Linus Walleij

Re: Appended devicetree with Torvalds' HEAD

[David Brown]

On Sun, Sep 18, 2011 at 12:34:20PM +0200, Linus Walleij wrote:
> On Tue, Sep 13, 2011 at 8:42 PM, David Brown <davidb-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org> wrote:
> > On Tue, Sep 06, 2011 at 01:52:43PM -0400, Nicolas Pitre wrote:
> >
> >> My latest patchset is here:
> >>
> >>   git://git.linaro.org/people/nico/linux zImage_DTB_append
> >>
> >> I'm not fully satisfied to send them upstream yet though.
> >
> > These patches do appear to still work on MSM.
> 
> Hm I wonder what is wrong on our end, probably our own fault :-/
> 
> Arey you encapsulating the zImage+DTB in a uImage like I do
> (so as to use with elder U-Boot) or are you using some other
> method to get the zImage+DTB into the system?

Our targets use littlekernel (an Android bootloader), so the
zImage+DTB is combined with an initrd using mkbootimage.

The first time I tried the append patches (not this version), I did
end up having to use JTAG to debug it, and even then the problem was
confusing, since a bug was causing it to move all of the code around
in memory, but some was still cached.

Are you getting far enough to print messages with DEBUG_LL?

David

-- 
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.

Re: Appended devicetree with Torvalds' HEAD

[Linus Walleij]

On Sun, Sep 18, 2011 at 5:17 PM, David Brown <davidb-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org> wrote:
> On Sun, Sep 18, 2011 at 12:34:20PM +0200, Linus Walleij wrote:
>> On Tue, Sep 13, 2011 at 8:42 PM, David Brown <davidb-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org> wrote:
>> > On Tue, Sep 06, 2011 at 01:52:43PM -0400, Nicolas Pitre wrote:
>> >
>> >> My latest patchset is here:
>> >>
>> >>   git://git.linaro.org/people/nico/linux zImage_DTB_append
>> >>
>> >> I'm not fully satisfied to send them upstream yet though.
>> >
>> > These patches do appear to still work on MSM.
>>
>> Hm I wonder what is wrong on our end, probably our own fault :-/
>>
>> Arey you encapsulating the zImage+DTB in a uImage like I do
>> (so as to use with elder U-Boot) or are you using some other
>> method to get the zImage+DTB into the system?
>
> Our targets use littlekernel (an Android bootloader), so the
> zImage+DTB is combined with an initrd using mkbootimage.
>
> The first time I tried the append patches (not this version), I did
> end up having to use JTAG to debug it, and even then the problem was
> confusing, since a bug was causing it to move all of the code around
> in memory, but some was still cached.
>
> Are you getting far enough to print messages with DEBUG_LL?

Nope :-(

So it'll be JTAG if I should get further with this.

Linus Walleij

Re: Appended devicetree with Torvalds' HEAD

[Nicolas Pitre]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 880 bytes --]

On Sun, 18 Sep 2011, Linus Walleij wrote:

> On Tue, Sep 13, 2011 at 8:42 PM, David Brown <davidb-sgV2jX0FEOL9JmXXK+q4OQ@public.gmane.org> wrote:
> > On Tue, Sep 06, 2011 at 01:52:43PM -0400, Nicolas Pitre wrote:
> >
> >> My latest patchset is here:
> >>
> >>   git://git.linaro.org/people/nico/linux zImage_DTB_append
> >>
> >> I'm not fully satisfied to send them upstream yet though.
> >
> > These patches do appear to still work on MSM.
> 
> Hm I wonder what is wrong on our end, probably our own fault :-/
> 
> Arey you encapsulating the zImage+DTB in a uImage like I do
> (so as to use with elder U-Boot) or are you using some other
> method to get the zImage+DTB into the system?

What I do to thest it here is along those lines:

	make <whatever_foo_defconfig_you_want>
	make zImage
	cat foobar.dtb >> arch/arm/boot/zImage
	make uImage


Nicolas

[-- Attachment #2: Type: text/plain, Size: 192 bytes --]

_______________________________________________
devicetree-discuss mailing list
devicetree-discuss-uLR06cmDAlY/bJ5BZ2RsiQ@public.gmane.org
https://lists.ozlabs.org/listinfo/devicetree-discuss

Re: Appended devicetree with Torvalds' HEAD

[Linus Walleij]

On Sun, Sep 18, 2011 at 7:31 PM, Nicolas Pitre <nicolas.pitre-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org> wrote:

>> Arey you encapsulating the zImage+DTB in a uImage like I do
>> (so as to use with elder U-Boot) or are you using some other
>> method to get the zImage+DTB into the system?
>
> What I do to thest it here is along those lines:
>
>        make <whatever_foo_defconfig_you_want>
>        make zImage
>        cat foobar.dtb >> arch/arm/boot/zImage
>        make uImage

This is what I do, that patch I made does exactly that while supplying
the necessary -e -a -T arguments etc to mkimage.

Oh well, U-Boot is working for us now I think, so the problem isn't
imminent.

Thanks,
Linus Walleij

Re: Appended devicetree with Torvalds' HEAD

[Linus Walleij]

On Sun, Sep 18, 2011 at 7:31 PM, Nicolas Pitre <nicolas.pitre-QSEj5FYQhm4dnm+yROfE0A@public.gmane.org> wrote:

> What I do to thest it here is along those lines:
>
>        make <whatever_foo_defconfig_you_want>
>        make zImage
>        cat foobar.dtb >> arch/arm/boot/zImage
>        make uImage

BTW this sequence does not work, and that is why I wrote the patch
attaching the DTB from a specified file in the first place.

After you have issued "cat foobar.dtb >> arch/arm/boot/zImage"
and you issue "make uImage" the zImage is rebuilt and
overwrites your zImage, since the timestamp changes.

So they mkimage command creating the uImage has to be
issued outside of the kernel build with

mkimage <various parameters> zImage uImage

Which is very inconvenient, but as I understood it, this was
intended.

Just making sure it is as inconvenient as you'd want it, and not
overly inconvenient...

For testing uImage I'm using my patch, of course, it's the easiest
way to get a uImage with attached device tree out. I guess
most people wanting to use uImages with attached trees will
use that out-of-tree patch too...

Linus Walleij

Re: Appended devicetree with Torvalds' HEAD

[Nicolas Pitre]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1023 bytes --]

On Mon, 19 Sep 2011, Linus Walleij wrote:

> On Sun, Sep 18, 2011 at 7:31 PM, Nicolas Pitre <nicolas.pitre-QSEj5FYQhm5QFI55V6+gNQ@public.gmane.orgg> wrote:
> 
> > What I do to thest it here is along those lines:
> >
> >        make <whatever_foo_defconfig_you_want>
> >        make zImage
> >        cat foobar.dtb >> arch/arm/boot/zImage
> >        make uImage
> 
> BTW this sequence does not work, and that is why I wrote the patch
> attaching the DTB from a specified file in the first place.

I tell you it works.  You need my full patch series (which is in RMK's 
devel-stable now).

> After you have issued "cat foobar.dtb >> arch/arm/boot/zImage"
> and you issue "make uImage" the zImage is rebuilt and
> overwrites your zImage, since the timestamp changes.

The zImage timestamp changes to a later time, so there is no reason for 
it to be rebuilt as its dependencies still are older.  A separate issue 
caused it to be rebuilt all the time, but I included a patch to fix 
that.


Nicolas

[-- Attachment #2: Type: text/plain, Size: 192 bytes --]

_______________________________________________
devicetree-discuss mailing list
devicetree-discuss-uLR06cmDAlY/bJ5BZ2RsiQ@public.gmane.org
https://lists.ozlabs.org/listinfo/devicetree-discuss