[Buildroot] [RFC/PATCH] linux: remove the zImage before rebuild

[Buildroot] [RFC/PATCH] linux: remove the zImage before rebuild

[Guido Martínez]

Currently, if we run "make linux-rebuild", the device tree blob gets
appended twice to the kernel image, rendering the latter append useless.

Fix this by removing the zImage before building, so the DTB append runs
on a fresh kernel image. This is not optimal as the final Linux linking
step is run again, but at least we get sensible results.

Signed-off-by: Guido Mart?nez <guido@vanguardiasur.com.ar>
---
Hi guys,

Has anyone ran into the above problem? It was a small headache for me,
so I wrote this small patch.

It's not great, the right thing to do would be copying the zImage out
of the kernel tree and appending the DTB there, but my attempt at doing
that broke some other builds. This should work every time.


 linux/linux.mk | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/linux/linux.mk b/linux/linux.mk
index 2fb7437..14c2b8e 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -263,6 +263,10 @@ endif
 # Compilation. We make sure the kernel gets rebuilt when the
 # configuration has changed.
 define LINUX_BUILD_CMDS
+	# Remove the previous image, if it exists, to prevent appending
+	# multiple DTBs
+	rm -f $(KERNEL_ARCH_PATH)/boot/zImage
+
 	$(if $(BR2_LINUX_KERNEL_USE_CUSTOM_DTS),
 		cp $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_DTS_PATH)) $(KERNEL_ARCH_PATH)/boot/dts/)
 	$(TARGET_MAKE_ENV) $(MAKE) $(LINUX_MAKE_FLAGS) -C $(@D) $(LINUX_TARGET_NAME)
-- 
2.1.0

[Buildroot] [RFC/PATCH] linux: remove the zImage before rebuild

[Guido Martínez]

Hey Yann,

On Fri, Sep 26, 2014 at 08:22:54PM -0300, Guido Mart?nez wrote:
> Currently, if we run "make linux-rebuild", the device tree blob gets
> appended twice to the kernel image, rendering the latter append useless.

I'm still finding this problem on current BR. I believe it always
happens when one uses an appended DT, but I'm attaching a defconfig that
reproduces it just in case.

You can check that the problem exists by running linux-rebuild and
checking for some DT string.

   $ make
      [...]
   $ strings output/images/uImage | grep am335x-bone
   ti,am335x-bone
   $ make linux-rebuild
      [...]
   $ strings output/images/uImage | grep am335x-bone
   ti,am335x-bone
   ti,am335x-bone

I have a patch that fixes this for the uImage+dt case, but it messes up
some other builds. Likely, to properly fix this, it should be a pretty
invasive change. I think adding a output/build/linux-staging directory
and doing all the DT appends and uImage creation there (without messing
with the kernel tree) would work.

Cheers!

-- 
Guido Mart?nez, VanguardiaSur
www.vanguardiasur.com.ar
-------------- next part --------------
BR2_arm=y
BR2_cortex_a8=y
BR2_LINUX_KERNEL=y
BR2_LINUX_KERNEL_DEFCONFIG="omap2plus"
BR2_LINUX_KERNEL_APPENDED_UIMAGE=y
BR2_LINUX_KERNEL_UIMAGE_LOADADDR="0x80008000"
BR2_LINUX_KERNEL_INTREE_DTS_NAME="am335x-boneblack"

[Buildroot] [RFC/PATCH] linux: remove the zImage before rebuild

[Thomas Petazzoni]

Dear Guido Mart?nez,

On Fri, 26 Sep 2014 20:22:54 -0300, Guido Mart?nez wrote:
> Currently, if we run "make linux-rebuild", the device tree blob gets
> appended twice to the kernel image, rendering the latter append useless.
> 
> Fix this by removing the zImage before building, so the DTB append runs
> on a fresh kernel image. This is not optimal as the final Linux linking
> step is run again, but at least we get sensible results.
> 
> Signed-off-by: Guido Mart?nez <guido@vanguardiasur.com.ar>

First of all, sorry for the slow response. I could indeed reproduce the
problem.

However, I am not entirely happy with the solution you proposed. I'd
prefer to fix the real problem: the fact that we are overwriting the
kernel zImage file.

I believe we should rather concatenate zImage and the DTB into a file
named maybe zImage.appended-dtb or something like that, and install
this file to $(BINARIES_DIR) and $(TARGET_DIR)/boot/ (if requested).

Or, another possibility is to name the file zImage.<name-of-dtb>, which
would later on allow to generate multiple zImage with various appended
DTBs.

Doing this however requires reworking quite a bit the
LINUX_IMAGE_NAME / LINUX_TARGET_NAME logic. Would you be willing to
work on something like this?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [RFC/PATCH] linux: remove the zImage before rebuild

[Guido Martínez]

Hi Thomas!

On Sun, Mar 08, 2015 at 06:47:30PM +0100, Thomas Petazzoni wrote:
> Dear Guido Mart?nez,
> 
> On Fri, 26 Sep 2014 20:22:54 -0300, Guido Mart?nez wrote:
> > Currently, if we run "make linux-rebuild", the device tree blob gets
> > appended twice to the kernel image, rendering the latter append useless.
> > 
> > Fix this by removing the zImage before building, so the DTB append runs
> > on a fresh kernel image. This is not optimal as the final Linux linking
> > step is run again, but at least we get sensible results.
> > 
> > Signed-off-by: Guido Mart?nez <guido@vanguardiasur.com.ar>
> 
> First of all, sorry for the slow response. I could indeed reproduce the
> problem.
No problem at all.

> However, I am not entirely happy with the solution you proposed. I'd
> prefer to fix the real problem: the fact that we are overwriting the
> kernel zImage file.
Yes, I don't like that either.

> I believe we should rather concatenate zImage and the DTB into a file
> named maybe zImage.appended-dtb or something like that, and install
> this file to $(BINARIES_DIR) and $(TARGET_DIR)/boot/ (if requested).
I had done exacly this a while ago, but never finished it since some
configurations failed to build (I don't remeber why exactly).

> Or, another possibility is to name the file zImage.<name-of-dtb>, which
> would later on allow to generate multiple zImage with various appended
> DTBs.
This sounds better! In fact a feature to append different DTBs to the
image would be useful for myself.

> Doing this however requires reworking quite a bit the
> LINUX_IMAGE_NAME / LINUX_TARGET_NAME logic. Would you be willing to
> work on something like this?
Yes, give me a while and I'll try to send a patch.

Cheers!

-- 
Guido Mart?nez, VanguardiaSur
www.vanguardiasur.com.ar

[Buildroot] [RFC/PATCH] linux: remove the zImage before rebuild

[Hollis Blanchard]

Hi Guido, I just ran across this discussion <http://patchwork.ozlabs.org/patch/394002/> in patchwork, and wanted to say I hit the same thing this week (and spent a few hours figuring out what had gone wrong). Has there been any activity on the problem recently?

Hollis Blanchard
Mentor Graphics Emulation Division

On 09/26/2014 04:22 PM, Guido Mart?nez wrote:
> Currently, if we run "make linux-rebuild", the device tree blob gets
> appended twice to the kernel image, rendering the latter append useless.
>
> Fix this by removing the zImage before building, so the DTB append runs
> on a fresh kernel image. This is not optimal as the final Linux linking
> step is run again, but at least we get sensible results.
>
> Signed-off-by: Guido Mart?nez <guido@vanguardiasur.com.ar>
> ---
> Hi guys,
>
> Has anyone ran into the above problem? It was a small headache for me,
> so I wrote this small patch.
>
> It's not great, the right thing to do would be copying the zImage out
> of the kernel tree and appending the DTB there, but my attempt at doing
> that broke some other builds. This should work every time.
>
>
>  linux/linux.mk | 4 ++++
>  1 file changed, 4 insertions(+)
>
> diff --git a/linux/linux.mk b/linux/linux.mk
> index 2fb7437..14c2b8e 100644
> --- a/linux/linux.mk
> +++ b/linux/linux.mk
> @@ -263,6 +263,10 @@ endif
>  # Compilation. We make sure the kernel gets rebuilt when the
>  # configuration has changed.
>  define LINUX_BUILD_CMDS
> +	# Remove the previous image, if it exists, to prevent appending
> +	# multiple DTBs
> +	rm -f $(KERNEL_ARCH_PATH)/boot/zImage
> +
>  	$(if $(BR2_LINUX_KERNEL_USE_CUSTOM_DTS),
>  		cp $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_DTS_PATH)) $(KERNEL_ARCH_PATH)/boot/dts/)
>  	$(TARGET_MAKE_ENV) $(MAKE) $(LINUX_MAKE_FLAGS) -C $(@D) $(LINUX_TARGET_NAME)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20150717/de46db10/attachment.html>

[Buildroot] [RFC/PATCH] linux: remove the zImage before rebuild

[Guido Martínez]

Hi Hollis,

On Fri, Jul 17, 2015 at 09:01:32AM -0700, Hollis Blanchard wrote:
> Hi Guido, I just ran across this discussion
> <http://patchwork.ozlabs.org/patch/394002/> in patchwork, and wanted
> to say I hit the same thing this week (and spent a few hours figuring
> out what had gone wrong). Has there been any activity on the problem
> recently?
No, not really. We thought of changing quite a bit of the linux.mk file
to avoid this problem but I never got around to it :(.

I'll try to work on it in the following days, however feel free to do so
before me and send the patch :).

In any case you can use this workaround in the mean time, I've been
doing exactly that.

Cheers!

> Hollis Blanchard
> Mentor Graphics Emulation Division
> 
> On 09/26/2014 04:22 PM, Guido Mart?nez wrote:
> > Currently, if we run "make linux-rebuild", the device tree blob gets
> > appended twice to the kernel image, rendering the latter append useless.
> >
> > Fix this by removing the zImage before building, so the DTB append runs
> > on a fresh kernel image. This is not optimal as the final Linux linking
> > step is run again, but at least we get sensible results.
> >
> > Signed-off-by: Guido Mart?nez <guido@vanguardiasur.com.ar>
> > ---
> > Hi guys,
> >
> > Has anyone ran into the above problem? It was a small headache for me,
> > so I wrote this small patch.
> >
> > It's not great, the right thing to do would be copying the zImage out
> > of the kernel tree and appending the DTB there, but my attempt at doing
> > that broke some other builds. This should work every time.
> >
> >
> >  linux/linux.mk | 4 ++++
> >  1 file changed, 4 insertions(+)
> >
> > diff --git a/linux/linux.mk b/linux/linux.mk
> > index 2fb7437..14c2b8e 100644
> > --- a/linux/linux.mk
> > +++ b/linux/linux.mk
> > @@ -263,6 +263,10 @@ endif
> >  # Compilation. We make sure the kernel gets rebuilt when the
> >  # configuration has changed.
> >  define LINUX_BUILD_CMDS
> > +	# Remove the previous image, if it exists, to prevent appending
> > +	# multiple DTBs
> > +	rm -f $(KERNEL_ARCH_PATH)/boot/zImage
> > +
> >  	$(if $(BR2_LINUX_KERNEL_USE_CUSTOM_DTS),
> >  		cp $(call qstrip,$(BR2_LINUX_KERNEL_CUSTOM_DTS_PATH)) $(KERNEL_ARCH_PATH)/boot/dts/)
> >  	$(TARGET_MAKE_ENV) $(MAKE) $(LINUX_MAKE_FLAGS) -C $(@D) $(LINUX_TARGET_NAME)
> 

-- 
Guido Mart?nez, VanguardiaSur
www.vanguardiasur.com.ar

[Buildroot] [RFC/PATCH] linux: remove the zImage before rebuild

[Thomas Petazzoni]

Dear Guido Mart?nez,

On Fri, 17 Jul 2015 13:34:42 -0300, Guido Mart?nez wrote:

> On Fri, Jul 17, 2015 at 09:01:32AM -0700, Hollis Blanchard wrote:
> > Hi Guido, I just ran across this discussion
> > <http://patchwork.ozlabs.org/patch/394002/> in patchwork, and wanted
> > to say I hit the same thing this week (and spent a few hours figuring
> > out what had gone wrong). Has there been any activity on the problem
> > recently?
> No, not really. We thought of changing quite a bit of the linux.mk file
> to avoid this problem but I never got around to it :(.
> 
> I'll try to work on it in the following days, however feel free to do so
> before me and send the patch :).


Due to the inactivity on this patch, it was on our TODO list of things
to tackle during the past Hackaton, but in the end we got busy with
other topics so we didn't get around to working on this.

One thing that we should do is to generalize this appended DTB
mechanism to make it possible to generate several appended images.
Currently, since we append to the zImage directly, we cannot generate
several images, each using a different DTB. By generating to
zImage.<dtb-name>, we can generate multiple images (and also avoid the
overwriting problem).

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [RFC/PATCH] linux: remove the zImage before rebuild

[Hollis Blanchard]

On 07/17/2015 10:41 AM, Thomas Petazzoni wrote:
> One thing that we should do is to generalize this appended DTB
> mechanism to make it possible to generate several appended images.
> Currently, since we append to the zImage directly, we cannot generate
> several images, each using a different DTB. By generating to
> zImage.<dtb-name>, we can generate multiple images (and also avoid the
> overwriting problem).

A bit of a tangent now, but I am trying to write a recipe for another 
bootwrapper, which prepends a little code to a zImage/DTB combo. Right 
now, it basically boils down to this:

    define BOOT_WRAPPER_ARM_BUILD_CMDS

         $(MAKE) CROSS_COMPILE=$(TARGET_CROSS) DTBIMAGE=$(BINARIES_DIR)/zImage -C $(@D)

    endef


Practically speaking, I think the bootwrapper is only really useful in 
the BR2_LINUX_KERNEL_APPENDED_ZIMAGE case, and I can enforce that via 
Kconfig dependencies, so I can assume the kernel name is "zImage".

If your idea above is implemented, would I need to duplicate the 
linux.mk logic around  BR2_LINUX_KERNEL_USE_INTREE_DTS, 
BR2_LINUX_KERNEL_INTREE_DTS_NAME, BR2_LINUX_KERNEL_USE_CUSTOM_DTS, and 
BR2_LINUX_KERNEL_CUSTOM_DTS_PATH? Or is there a better way to do this?

Thanks.

Hollis Blanchard
Mentor Graphics Emulation Division
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20150717/ae8ff273/attachment.html>

[Buildroot] [RFC/PATCH] linux: remove the zImage before rebuild

[Thomas Petazzoni]

Hello,

On Fri, 17 Jul 2015 11:49:04 -0700, Hollis Blanchard wrote:

> A bit of a tangent now, but I am trying to write a recipe for another 
> bootwrapper, which prepends a little code to a zImage/DTB combo. Right 
> now, it basically boils down to this:
> 
>     define BOOT_WRAPPER_ARM_BUILD_CMDS
> 
>          $(MAKE) CROSS_COMPILE=$(TARGET_CROSS) DTBIMAGE=$(BINARIES_DIR)/zImage -C $(@D)
> 
>     endef
> 
> 
> Practically speaking, I think the bootwrapper is only really useful in 
> the BR2_LINUX_KERNEL_APPENDED_ZIMAGE case, and I can enforce that via 
> Kconfig dependencies, so I can assume the kernel name is "zImage".
> 
> If your idea above is implemented, would I need to duplicate the 
> linux.mk logic around  BR2_LINUX_KERNEL_USE_INTREE_DTS, 
> BR2_LINUX_KERNEL_INTREE_DTS_NAME, BR2_LINUX_KERNEL_USE_CUSTOM_DTS, and 
> BR2_LINUX_KERNEL_CUSTOM_DTS_PATH? Or is there a better way to do this?

We already have a similar case in the boot-wrapper-aarch64 package. I
am indeed not sure how that would work with appended DTB and multiple
zImage generated. Needs some thought, indeed.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [RFC/PATCH] linux: remove the zImage before rebuild

[Hollis Blanchard]

On 07/17/2015 02:16 PM, Thomas Petazzoni wrote:
> Hello,
>
> On Fri, 17 Jul 2015 11:49:04 -0700, Hollis Blanchard wrote:
>
>> A bit of a tangent now, but I am trying to write a recipe for another
>> bootwrapper, which prepends a little code to a zImage/DTB combo. Right
>> now, it basically boils down to this:
>>
>>      define BOOT_WRAPPER_ARM_BUILD_CMDS
>>
>>           $(MAKE) CROSS_COMPILE=$(TARGET_CROSS) DTBIMAGE=$(BINARIES_DIR)/zImage -C $(@D)
>>
>>      endef
>>
>>
>> Practically speaking, I think the bootwrapper is only really useful in
>> the BR2_LINUX_KERNEL_APPENDED_ZIMAGE case, and I can enforce that via
>> Kconfig dependencies, so I can assume the kernel name is "zImage".
>>
>> If your idea above is implemented, would I need to duplicate the
>> linux.mk logic around  BR2_LINUX_KERNEL_USE_INTREE_DTS,
>> BR2_LINUX_KERNEL_INTREE_DTS_NAME, BR2_LINUX_KERNEL_USE_CUSTOM_DTS, and
>> BR2_LINUX_KERNEL_CUSTOM_DTS_PATH? Or is there a better way to do this?
> We already have a similar case in the boot-wrapper-aarch64 package. I
> am indeed not sure how that would work with appended DTB and multiple
> zImage generated. Needs some thought, indeed.

OK.

It looks like arm64 Linux builds produce only "arch/arm64/boot/Image", 
so the boot-wrapper-aarch64 build system itself (not 
boot-wrapper-aarch64.mk) assumes "Image" is the file name to process.

PowerPC Linux can create "dtbImage.<board>". "dtbImage", even without 
the "board" extension, is a good name for what's going on, and I think 
it would at least partially satisfy your goal... so perhaps linux.mk 
should create that, i.e. "cat zImage dtb > dtbImage"

Hollis Blanchard
Mentor Graphics Emulation Division