Question / issue

Question / issue

[Paul McGougan]

[-- Attachment #1: Type: text/plain, Size: 1949 bytes --]

Hi all.

We are currently using Poky 1.5.0.

We have created our own custom layer for our powerpc-based board.

We are running u-boot as our bootloader and want to use the new FIT (FDT) style kernel/dtb image blob.

To that end, in our custom layer we have a file poky/meta-OURS/recipes-kernel/linux/linux-yocto_3.10.bbappend, and in that file I have a function do_install_append() in which I call u-boot's mkimage passing it the kernel and DTB files that we want stored in the FIT image that we will use to boot from u-boot.

My first question is, is there a better place to be making the FIT image?

As a side-question to that, I was surprised that there isn't native support already for creating this type of u-boot image considering how useful it is, does anyone know if there is a specific reason no one has done this yet?

Secondly, (and this is our main issue) I have found that by adding the do_install_append function, even if it is completely empty, whenever I try to bitbake anything that depends on the kernel, that bitbake always re-runs the kernel install stage, even when there have been no changes. Literally I can run a bitbake, then run the same bitbake command again, and both times the kernel install stage gets run (this is a problem because it takes a long time to run). It appears to be happening because the stampfile is not found every time (the hash appears to be wrong). Is this a bug? Is there a fix or work-around for this problem?

Thanks.

Paul.

Confidentiality Notice:  This message (including attachments) is a private communication solely for use of the intended recipient(s).  
If you are not the intended recipient(s) or believe you received this message in error, notify the sender immediately and then delete this 
message.  Any other use, retention, dissemination or copying is prohibited and may be a violation of law, including the Electronic 
Communication Privacy Act of 1986."

[-- Attachment #2: Type: text/html, Size: 4428 bytes --]

Re: Question / issue

[Bruce Ashfield]

[-- Attachment #1: Type: text/plain, Size: 2903 bytes --]

On 14-05-09 01:14 AM, Paul McGougan wrote:
>
> Hi all.
>
> We are currently using Poky 1.5.0.
>
> We have created our own custom layer for our powerpc-based board.
>
> We are running u-boot as our bootloader and want to use the new FIT 
> (FDT) style kernel/dtb image blob.
>
> To that end, in our custom layer we have a file 
> poky/meta-OURS/recipes-kernel/linux/linux-yocto_3.10.bbappend, and in 
> that file I have a function do_install_append() in which I call 
> u-boot's mkimage passing it the kernel and DTB files that we want 
> stored in the FIT image that we will use to boot from u-boot.
>
> My first question is, is there a better place to be making the FIT image?
>

It depends on if everything you need to construct the FIT image is
in the kernel's build directory, or also available in the sysroot/deploy
directories.

If you need kernel build artifacts, doing it in the bbappend (or a .inc,
.bbclass, etc) of the kernel recipe is the right place.

> As a side-question to that, I was surprised that there isn't native 
> support already for creating this type of u-boot image considering how 
> useful it is, does anyone know if there is a specific reason no one 
> has done this yet?
>

None that I know of (but I haven't checked all the SDK, vendor
and distro layers in the ecosystem).

Either a image_types bbclass or something like the existing linux-dtb.inc
could fill the roll. It just depends on what is needed to build the uImage.

> Secondly, (and this is our main issue) I have found that by adding the 
> do_install_append function, even if it is completely empty, whenever I 
> try to bitbake anything that depends on the kernel, that bitbake 
> always re-runs the kernel install stage, even when there have been no 
> changes. Literally I can run a bitbake, then run the same bitbake 
> command again, and both times the kernel install stage gets run (this 
> is a problem because it takes a long time to run). It appears to be 
> happening because the stampfile is not found every time (the hash 
> appears to be wrong). Is this a bug? Is there a fix or work-around for 
> this problem?
>

In this front, I'm not the best reference. Checking the sstate signature
might help, it should show which variables are being used .. and possibly
invalidating the signature, triggering the steps to run again.

Bruce

> Thanks.
>
> Paul.
>
>
> Confidentiality Notice: This message (including attachments) is a 
> private communication solely for use of the intended recipient(s). If 
> you are not the intended recipient(s) or believe you received this 
> message in error, notify the sender immediately and then delete this 
> message. Any other use, retention, dissemination or copying is 
> prohibited and may be a violation of law, including the Electronic 
> Communication Privacy Act of 1986."   ­­
>
>


[-- Attachment #2: Type: text/html, Size: 6687 bytes --]

Re: Question / issue

[Paul McGougan]

From: Bruce Ashfield [mailto:bruce.ashfield@windriver.com] 
 
> > Secondly, (and this is our main issue) I have found that by adding the 
> > do_install_append function, even if it is completely empty, whenever I 
> > try to bitbake anything that depends on the kernel, that bitbake always 
> > re-runs the kernel install stage, even when there have been no changes. 
> > Literally I can run a bitbake, then run the same bitbake command again, 
> > and both times the kernel install stage gets run (this is a problem 
> > because it takes a long time to run). It appears to be happening because 
> > the stampfile is not found every time (the hash appears to be wrong). 
> > Is this a bug? Is there a fix or work-around for this problem?

> In this front, I'm not the best reference. Checking the sstate signature
> might help, it should show which variables are being used .. and possibly
> invalidating the signature, triggering the steps to run again.

Thanks Bruce.

Just for reference of anyone else who runs into a similar issue our issue was:
1. The do_install_append function was not *really* empty, the content of it 
was commented out.
2. The dependency parser when parsing recipes not only looks for content
changes, but for also for changes to the values of referenced variables, and
unfortunately it still performs variable-value substitution in commented-out
code. In our case, we had the variable $DATETIME in there (but commented
out), this caused the dependency checker to substitute in the current value for
DATETIME which of course changed on each run.

The fix was to add:
do_install[vardepsexclude] += "DATETIME"

Regards,
Paul McGougan

Confidentiality Notice:  This message (including attachments) is a private communication solely for use of the intended recipient(s).  
If you are not the intended recipient(s) or believe you received this message in error, notify the sender immediately and then delete this 
message.  Any other use, retention, dissemination or copying is prohibited and may be a violation of law, including the Electronic 
Communication Privacy Act of 1986."

Re: Question / issue

[Bruce Ashfield]

On 14-05-12 01:56 AM, Paul McGougan wrote:
> From: Bruce Ashfield [mailto:bruce.ashfield@windriver.com]
>
>>> Secondly, (and this is our main issue) I have found that by adding the
>>> do_install_append function, even if it is completely empty, whenever I
>>> try to bitbake anything that depends on the kernel, that bitbake always
>>> re-runs the kernel install stage, even when there have been no changes.
>>> Literally I can run a bitbake, then run the same bitbake command again,
>>> and both times the kernel install stage gets run (this is a problem
>>> because it takes a long time to run). It appears to be happening because
>>> the stampfile is not found every time (the hash appears to be wrong).
>>> Is this a bug? Is there a fix or work-around for this problem?
>
>> In this front, I'm not the best reference. Checking the sstate signature
>> might help, it should show which variables are being used .. and possibly
>> invalidating the signature, triggering the steps to run again.
>
> Thanks Bruce.
>
> Just for reference of anyone else who runs into a similar issue our issue was:
> 1. The do_install_append function was not *really* empty, the content of it
> was commented out.
> 2. The dependency parser when parsing recipes not only looks for content
> changes, but for also for changes to the values of referenced variables, and
> unfortunately it still performs variable-value substitution in commented-out
> code. In our case, we had the variable $DATETIME in there (but commented
> out), this caused the dependency checker to substitute in the current value for
> DATETIME which of course changed on each run.

Aha. That's what I thought it might be (and what I've been hit by
before in the past), sstate signature checking showed the variable
in these other cases .. so I thought I'd suggest it.

Glad to hear you have a working solution.

Cheers,

Bruce

>
> The fix was to add:
> do_install[vardepsexclude] += "DATETIME"
>
> Regards,
> Paul McGougan
>
> Confidentiality Notice:  This message (including attachments) is a private communication solely for use of the intended recipient(s).
> If you are not the intended recipient(s) or believe you received this message in error, notify the sender immediately and then delete this
> message.  Any other use, retention, dissemination or copying is prohibited and may be a violation of law, including the Electronic
> Communication Privacy Act of 1986."
>

Re: Question / issue

[Paul Eggleton]

On Monday 12 May 2014 07:03:55 Bruce Ashfield wrote:
> On 14-05-12 01:56 AM, Paul McGougan wrote:
> > From: Bruce Ashfield [mailto:bruce.ashfield@windriver.com]
> > 
> >>> Secondly, (and this is our main issue) I have found that by adding the
> >>> do_install_append function, even if it is completely empty, whenever I
> >>> try to bitbake anything that depends on the kernel, that bitbake always
> >>> re-runs the kernel install stage, even when there have been no changes.
> >>> Literally I can run a bitbake, then run the same bitbake command again,
> >>> and both times the kernel install stage gets run (this is a problem
> >>> because it takes a long time to run). It appears to be happening because
> >>> the stampfile is not found every time (the hash appears to be wrong).
> >>> Is this a bug? Is there a fix or work-around for this problem?
> >> 
> >> In this front, I'm not the best reference. Checking the sstate signature
> >> might help, it should show which variables are being used .. and possibly
> >> invalidating the signature, triggering the steps to run again.
> > 
> > Thanks Bruce.
> > 
> > Just for reference of anyone else who runs into a similar issue our issue
> > was: 1. The do_install_append function was not *really* empty, the
> > content of it was commented out.
> > 2. The dependency parser when parsing recipes not only looks for content
> > changes, but for also for changes to the values of referenced variables,
> > and unfortunately it still performs variable-value substitution in
> > commented-out code. In our case, we had the variable $DATETIME in there
> > (but commented out), this caused the dependency checker to substitute in
> > the current value for DATETIME which of course changed on each run.
> 
> Aha. That's what I thought it might be (and what I've been hit by
> before in the past), sstate signature checking showed the variable
> in these other cases .. so I thought I'd suggest it.
> 
> Glad to hear you have a working solution.
> 
> > The fix was to add:
> > do_install[vardepsexclude] += "DATETIME"

Er, if DATETIME is in the dependencies for this task and that is a bug in the 
core recipe, we should fix it in the core. The question is how is it getting in 
there?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Question / issue

[David Nyström]

On 2014-05-12 07:56, Paul McGougan wrote:
> From: Bruce Ashfield [mailto:bruce.ashfield@windriver.com]
>   
>>> Secondly, (and this is our main issue) I have found that by adding the
>>> do_install_append function, even if it is completely empty, whenever I
>>> try to bitbake anything that depends on the kernel, that bitbake always
>>> re-runs the kernel install stage, even when there have been no changes.
>>> Literally I can run a bitbake, then run the same bitbake command again,
>>> and both times the kernel install stage gets run (this is a problem
>>> because it takes a long time to run). It appears to be happening because
>>> the stampfile is not found every time (the hash appears to be wrong).
>>> Is this a bug? Is there a fix or work-around for this problem?
>> In this front, I'm not the best reference. Checking the sstate signature
>> might help, it should show which variables are being used .. and possibly
>> invalidating the signature, triggering the steps to run again.
> Thanks Bruce.
>
> Just for reference of anyone else who runs into a similar issue our issue was:
> 1. The do_install_append function was not *really* empty, the content of it
> was commented out.

To me, Thats sound like a bug in the dependency parser.
It should not parse for changes in comments, i.e. in this case 
do_install_append comments.

Or perhaps in variable expansion, which could be avoided in commented 
sections.

Br,
David

Re: Question / issue

[Paul McGougan]

> -----Original Message-----
> From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com]


> Er, if DATETIME is in the dependencies for this task and that is a bug in the core
> recipe, we should fix it in the core. The question is how is it getting in there?
> 

The DATETIME was being introduced in my bbappend, not the core recipe. So I don't think a change is needed to the core recipe.

But see my response to David's email.

Regards,
Paul

Confidentiality Notice:  This message (including attachments) is a private communication solely for use of the intended recipient(s).  
If you are not the intended recipient(s) or believe you received this message in error, notify the sender immediately and then delete this 
message.  Any other use, retention, dissemination or copying is prohibited and may be a violation of law, including the Electronic 
Communication Privacy Act of 1986."

Re: Question / issue

[Paul McGougan]

> -----Original Message-----
> From: David Nyström [mailto:david.c.nystrom@gmail.com]
 
> To me, Thats sound like a bug in the dependency parser.
> It should not parse for changes in comments, i.e. in this case do_install_append
> comments.
> 
> Or perhaps in variable expansion, which could be avoided in commented
> sections.

Hi David.

Yes, I agree that the dependency parser should be altered.

I would suggest your 2nd suggestion would be the most appropriate, i.e. do not perform variable expansion in commented lines.

Regards,
Paul


Confidentiality Notice:  This message (including attachments) is a private communication solely for use of the intended recipient(s).  
If you are not the intended recipient(s) or believe you received this message in error, notify the sender immediately and then delete this 
message.  Any other use, retention, dissemination or copying is prohibited and may be a violation of law, including the Electronic 
Communication Privacy Act of 1986."

Re: Question / issue

[Richard Purdie]

On Mon, 2014-05-12 at 23:31 +0000, Paul McGougan wrote:
> > -----Original Message-----
> > From: David Nyström [mailto:david.c.nystrom@gmail.com]
>  
> > To me, Thats sound like a bug in the dependency parser.
> > It should not parse for changes in comments, i.e. in this case do_install_append
> > comments.
> > 
> > Or perhaps in variable expansion, which could be avoided in commented
> > sections.
>
> Yes, I agree that the dependency parser should be altered.
> 
> I would suggest your 2nd suggestion would be the most appropriate, i.e. do not perform variable expansion in commented lines.

Its easy to think that at a first glance.

What if the variable being expanded expands to something multiline and
some of those lines are not comments though?

Bitbake's behaviour has always been to expand variables first, then
follow with other things like execution. The dependency parsing follows
the behaviour and includes them as dependencies because of the above
multiline issue.

So rightly or wrongly, the code does at least behave consistently at the
moment and its not as easy to change as you'd first think.

I make no claims about whether multiline expansion is a good thing, just
that these issues are there and that they're hard to change as the
impact isn't always as you'd expect. The behaviour has been like this
for a long time too.

Cheers,

Richard

Re: Question / issue

[Stefano Babic]

[-- Attachment #1: Type: text/plain, Size: 1349 bytes --]

Hi Paul,

sorry for late answer - I read only now you are talking about FIT.

On 09/05/2014 16:44, Bruce Ashfield wrote:
>>
>>  
>>
>> My first question is, is there a better place to be making the FIT image?
>>
> 
> It depends on if everything you need to construct the FIT image is
> in the kernel's build directory, or also available in the sysroot/deploy
> directories.
> 
> If you need kernel build artifacts, doing it in the bbappend (or a .inc,
> .bbclass, etc) of the kernel recipe is the right place.
> 

Indeed. This is what I did in a couple of projects. I added an
image_types_itb.bbclass to produce the FIT image. The IMAGE_FIT_ITS
variable contains the filename of the description file to be passed to
the mkimage tool, and can be overwritten by the image receipe.
Compressions are not needed for this image type, and I should be removed.

Is this the right place to post the patch ? Or should be addressed to
meta-oe ?

Best regards,
Stefano

-- 
=====================================================================
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sbabic@denx.de
=====================================================================

[-- Attachment #2: 0001-meta-add-support-for-FIT-images.patch --]
[-- Type: text/x-patch, Size: 1633 bytes --]

From 0cb68067149657079cad88b029f04cddcadf613a Mon Sep 17 00:00:00 2001
From: Stefano Babic <sbabic@denx.de>
Date: Thu, 26 Sep 2013 17:21:00 +0200
Subject: [PATCH] meta: add support for FIT images

Signed-off-by: Stefano Babic <sbabic@denx.de>
---
 meta/classes/image_types_itb.bbclass | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
 create mode 100644 meta/classes/image_types_itb.bbclass

diff --git a/meta/classes/image_types_itb.bbclass b/meta/classes/image_types_itb.bbclass
new file mode 100644
index 0000000..1c43c79
--- /dev/null
+++ b/meta/classes/image_types_itb.bbclass
@@ -0,0 +1,25 @@
+inherit image_types kernel-arch
+
+oe_mkimage () {
+	cp ${IMAGE_FIT_ITS}  ${DEPLOY_DIR_IMAGE}/.
+
+	FIT_ITS=`basename ${IMAGE_FIT_ITS}`
+	mkimage -f ${FIT_ITS} ${DEPLOY_DIR_IMAGE}/$1.itb
+}
+
+COMPRESSIONTYPES += "gz.u-boot bz2.u-boot lzma.u-boot u-boot"
+
+COMPRESS_DEPENDS_u-boot = "u-boot-mkimage-native"
+COMPRESS_CMD_u-boot      = "oe_mkimage ${IMAGE_NAME}.rootfs.${type} none"
+
+COMPRESS_DEPENDS_gz.u-boot = "u-boot-mkimage-native"
+COMPRESS_CMD_gz.u-boot      = "${COMPRESS_CMD_gz}; oe_mkimage ${IMAGE_NAME}.rootfs.${type}.gz gzip"
+
+COMPRESS_DEPENDS_bz2.u-boot = "u-boot-mkimage-native"
+COMPRESS_CMD_bz2.u-boot      = "${COMPRESS_CMD_bz2}; oe_mkimage ${IMAGE_NAME}.rootfs.${type}.bz2 bzip2"
+
+COMPRESS_DEPENDS_lzma.u-boot = "u-boot-mkimage-native"
+COMPRESS_CMD_lzma.u-boot      = "${COMPRESS_CMD_lzma}; oe_mkimage ${IMAGE_NAME}.rootfs.${type}.lzma lzma"
+
+IMAGE_TYPES += "ext2.itb ext2.gz.itb ext2.bz2.itb ext2.lzma.itb ext3.gz.itb ext4.gz.itb"
+
-- 
1.9.1