Image recipes in Yocto 1.4 (dylan-9.0.0)

Image recipes in Yocto 1.4 (dylan-9.0.0)

[Brian Karcz]

[-- Attachment #1: Type: text/plain, Size: 972 bytes --]

Hi,

I apologize in advance if I'm not posting this to correct place or in the correct manner.

I have a question regarding the shared state code optimizations in yocto 1.4. I'm in the process of upgrading one of our projects from Edison (6.0) to Dylan (9.0.0) and am running into an issue with our existing image recipe.

The recipe brings in files from a "files" directory in the image area. It also adds an image preprocess command that takes action on those files in the work area. After reading the version 1.4 migration guidelines and examining both the old and new builds, it looks like the do_unpack task has been optimized out of the way images are built in the new release. The files listed in the SRC_URI variable don't get populated in the work directory, and actions taken in the image preprocess command fail.

Is there a way to stop this optimization and have the image build populate the work directory as it has in the past?

Thank you,
Brian

[-- Attachment #2: Type: text/html, Size: 3795 bytes --]

Re: Image recipes in Yocto 1.4 (dylan-9.0.0)

[Paul Eggleton]

Hi Brian,

On Monday 24 June 2013 11:01:31 Brian Karcz wrote:
> I have a question regarding the shared state code optimizations in yocto
> 1.4. I'm in the process of upgrading one of our projects from Edison (6.0)
> to Dylan (9.0.0) and am running into an issue with our existing image
> recipe.
> 
> The recipe brings in files from a "files" directory in the image area. It
> also adds an image preprocess command that takes action on those files in
> the work area. After reading the version 1.4 migration guidelines and
> examining both the old and new builds, it looks like the do_unpack task has
> been optimized out of the way images are built in the new release. The
> files listed in the SRC_URI variable don't get populated in the work
> directory, and actions taken in the image preprocess command fail.
> 
> Is there a way to stop this optimization and have the image build populate
> the work directory as it has in the past?

You should be able to do this in your image recipe:

python () {
        d.delVarFlag("do_fetch", "noexec")
        d.delVarFlag("do_unpack", "noexec")
}

This isn't related to shared state, btw, just that image.bbclass disables 
these tasks by default as of version 1.2 (denzil).

Cheers,
Paul


-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Image recipes in Yocto 1.4 (dylan-9.0.0)

[Paul Eggleton]

Hi Brian,

Sorry this message got lost in my inbox.

On Monday 24 June 2013 16:22:27 Brian Karcz wrote:
> > Paul Eggleton wrote:
> > > Brian Karcz wrote:
> > > Is there a way to stop this optimization and have the image build 
> > > populate the work directory as it has in the past?
> > 
> > You should be able to do this in your image recipe:
> > 
> > python () {
> >         d.delVarFlag("do_fetch", "noexec")
> >         d.delVarFlag("do_unpack", "noexec") 
> > }
>
> That was what I needed to get my build(s) moving forward. It has brought me
> to a follow-up question. The image in question (a ramdisk image) is being
> built as a dependency of a larger image build. When I rebuild the parent
> image, bitbake believes the child image needs to be rebuilt, but when this
> occurs, do_fetch and do_unpack once again don't get executed and my build
> fails as before when it jumps straight to do_rootfs with an empty work
> directory. Upon seeing this, I attempted to re-build the child image from
> its own recipe, without trying to build the parent, and the same behavior
> occurs.
> 
> I was considering adding the child image to RM_WORK_EXCLUDE in local.conf,
> but that didn't seem intuitive to the problem. 

So you are using rm_work? If you are I don't think RM_WORK_EXCLUDE will really 
help.

Just to confirm, you're not referring to the WORKDIR of one recipe within the 
other are you? Also, how are you setting up the dependency relationship 
between the images?

> I would think bitbake would do nothing since nothing has changed, or the
> required tasks would be executed, but I wouldn't think telling the last
> build to leave the work area would be the fix.
> 
> Do have any thoughts on making this work past a single build?

So I haven't tested this exact configuration; it's possible that bitbake is 
eliminating a dependency based on the task being marked as "noexec", but I 
wouldn't have thought so.

Cheers,
Paul

PS: please keep replies on the mailing list. Thanks.

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Image recipes in Yocto 1.4 (dylan-9.0.0)

[Brian Karcz]

Hi Paul,

Here is a little background of what is going on. I inherited a project that was implemented under and works with Edison. The project has two image recipes, abc-main.bb and abc-ramdisk.bb. The way the project builds under Edison, abc-main.bb has abc-ramdisk listed in the DEPENDS variable. Abc-main then has an image preprocess command that sneakily goes into the DEPLOY_DIR_IMAGE and retrieves the ramdisk image and inserts it into its own IMAGE_ROOTFS.

As I've been working on porting to the Dylan version of the build system, I found the whole issue with do_fetch and do_unpack not being run at all for images. I applied the fix you advised of to add a python function that removes the noexec flag for the two tasks. This apparently only works for one build of the image. I either need to do a "cleanall" on the image or bump the rev to get it to build again.

To simplify things, and remove the dependency between the main and ramdisk images, I started building just the ramdisk directly. Doing this I see the same behavior:
	Bitbake abc-ramdisk -c cleanall
	Bitbake abc-ramdisk			<--- succeeds
	Bitbake abc-ramdisk			<--- fails trying to operate on one the files in the SRC_URI

As I mentioned in the previous post. I'm using the rm_work directive in the local.conf file. Its interesting that the subsequent re-build doesn't determine that the WORKDIR needs to be repopulated. Another interesting note that might be of interest is that our LICENCE is defined to "XYZ" and issues the following warning, "WARNING: abc-ramdisk: No generic license file exists for: XYZ in any provider", in the first build that succeeds. When the second build is run the warning does not appear and the build jumps right to the do_rootfs task where it fails.

Thanks,
Brian 

-----Original Message-----
From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com] 
Sent: Tuesday, July 16, 2013 1:30 PM
To: Brian Karcz
Cc: yocto@yoctoproject.org
Subject: Re: [yocto] Image recipes in Yocto 1.4 (dylan-9.0.0)

Hi Brian,

Sorry this message got lost in my inbox.

On Monday 24 June 2013 16:22:27 Brian Karcz wrote:
> > Paul Eggleton wrote:
> > > Brian Karcz wrote:
> > > Is there a way to stop this optimization and have the image build 
> > > populate the work directory as it has in the past?
> > 
> > You should be able to do this in your image recipe:
> > 
> > python () {
> >         d.delVarFlag("do_fetch", "noexec")
> >         d.delVarFlag("do_unpack", "noexec") }
>
> That was what I needed to get my build(s) moving forward. It has 
> brought me to a follow-up question. The image in question (a ramdisk 
> image) is being built as a dependency of a larger image build. When I 
> rebuild the parent image, bitbake believes the child image needs to be 
> rebuilt, but when this occurs, do_fetch and do_unpack once again don't 
> get executed and my build fails as before when it jumps straight to 
> do_rootfs with an empty work directory. Upon seeing this, I attempted 
> to re-build the child image from its own recipe, without trying to 
> build the parent, and the same behavior occurs.
> 
> I was considering adding the child image to RM_WORK_EXCLUDE in 
> local.conf, but that didn't seem intuitive to the problem.

So you are using rm_work? If you are I don't think RM_WORK_EXCLUDE will really help.

Just to confirm, you're not referring to the WORKDIR of one recipe within the other are you? Also, how are you setting up the dependency relationship between the images?

> I would think bitbake would do nothing since nothing has changed, or 
> the required tasks would be executed, but I wouldn't think telling the 
> last build to leave the work area would be the fix.
> 
> Do have any thoughts on making this work past a single build?

So I haven't tested this exact configuration; it's possible that bitbake is eliminating a dependency based on the task being marked as "noexec", but I wouldn't have thought so.

Cheers,
Paul

PS: please keep replies on the mailing list. Thanks.

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Image recipes in Yocto 1.4 (dylan-9.0.0)

[Paul Eggleton]

On Wednesday 31 July 2013 20:27:16 Brian Karcz wrote:
> Here is a little background of what is going on. I inherited a project that
> was implemented under and works with Edison. The project has two image
> recipes, abc-main.bb and abc-ramdisk.bb. The way the project builds under
> Edison, abc-main.bb has abc-ramdisk listed in the DEPENDS variable.
> Abc-main then has an image preprocess command that sneakily goes into the
> DEPLOY_DIR_IMAGE and retrieves the ramdisk image and inserts it into its
> own IMAGE_ROOTFS.
> 
> As I've been working on porting to the Dylan version of the build system, I
> found the whole issue with do_fetch and do_unpack not being run at all for
> images. I applied the fix you advised of to add a python function that
> removes the noexec flag for the two tasks. This apparently only works for
> one build of the image. I either need to do a "cleanall" on the image or
> bump the rev to get it to build again.

I haven't had time to try to reproduce this here yet, sorry.
 
However, aside from the files in SRC_URI which could presumably be handled in a 
separate recipe, one possible solution for the ramdisk part did occur to me - 
I'd suggest doing the following in the main image:

do_rootfs[depends] += "abc-ramdisk:do_rootfs"

Then add to ROOTFS_POSTPROCESS_COMMAND or IMAGE_POSTPROCESS_COMMAND (whichever 
is appropriate) a call to a shell function that performs whatever needed 
operation on the abc-ramdisk.

> Another interesting note that might be of interest is that our LICENCE is
> defined to "XYZ" and issues the following warning, "WARNING: abc-ramdisk: No
> generic license file exists for: XYZ in any provider", in the first build that
> succeeds.

I'm no expert on this area but to me it doesn't make sense for an image 
containing multiple differently-licensed components to have a declared license 
in the recipe. We tend to keep things simple and just set LICENSE = "MIT" on 
the image recipe (*not* implying anything about the contents of the image), 
with each installed package license and therefore the image's license manifest 
indicating all of the licenses of the components that make up the image. 
Perhaps a LICENSE value of "N/A" for the image would be more appropriate, 
although the system currently doesn't support any special casing of LICENSE 
for image recipes, and if you are pulling in files in SRC_URI in the image 
recipe there might be some validity to having a LICENSE value to apply to 
those.

> When the second build is run the warning does not appear and the build
> jumps right to the do_rootfs task where it fails.

That's because this check is done in do_populate_lic which isn't getting re-
executed on the second time (presumably because nothing that that task depends 
upon has changed).

Cheers,
Paul
 
-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Image recipes in Yocto 1.4 (dylan-9.0.0)

[Brian Karcz]

Hi Paul,

I'll keep the do_rootfs dependency adjustment for the main image on ready. I'd like to try to fix the simple case first of only building the ramdisk image directly then trying to rebuild it. With the latter case fixed, the former might be fine as is.

As a sanity check last night, I blew away the entire build directory and only built the ramdisk image recipe. I wanted to make sure there was nothing left in there from a main image build that might be effecting an explicit ramdisk image build. This overnight build succeeded. When I performed the rebuild this morning, the same behavior occurred of the IMAGE_PREPROCESS_COMMAND not being able to find a file in the WORKDIR. This wasn't surprising, but I wanted to make sure that there wasn't something being left in the build directory from the main image build that was effecting this.

Sorry for the confusion about the license data. I wasn't interested, necessarily, in the content or cause of the warning, but more the task flow of the build that was causing the warning to appear in one build and not the other. I wasn't sure if the do_populate_lic dependency change might trigger any thoughts on the do_fetch dependencies.

One thing I had a question about was in regards to image vs package recipes and the population of the rootfs. Package recipes populate the rootfs by appending/prepending the do_install task and install things from the WORKDIR to the $D destination directory. Our image recipes, as I've inherited them, create and assign a new IMAGE_PREPROCESS_COMMAND, that install files from the WORKDIR to the IMAGE_ROOTFS destination directory. Is it abnormal/non-standard for an image recipe to have its own SRC_URI entries and subsequent WORKDIR population?

I'm wondering if there is a catch-22 going on here regarding the use of RM_WORK and the fact that the checksums for the SRC_URI contents haven't changed. It seems as though the build is determining that do_fetch doesn't need to be run because nothing is changed, like in a package recipe, but doesn't realize that do_rootfs will always be executed for an image recipe build and might depend on those contents. I'm wondering if this worked under Edison because it didn't have the SRC_URI checksums and a different dependency model that forced the do_fetch prior to do_rootfs for image recipe builds.

Thanks,
Brian


-----Original Message-----
From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com] 
Sent: Thursday, August 01, 2013 4:26 AM
To: Brian Karcz
Cc: yocto@yoctoproject.org
Subject: Re: [yocto] Image recipes in Yocto 1.4 (dylan-9.0.0)

On Wednesday 31 July 2013 20:27:16 Brian Karcz wrote:
> Here is a little background of what is going on. I inherited a project 
> that was implemented under and works with Edison. The project has two 
> image recipes, abc-main.bb and abc-ramdisk.bb. The way the project 
> builds under Edison, abc-main.bb has abc-ramdisk listed in the DEPENDS variable.
> Abc-main then has an image preprocess command that sneakily goes into 
> the DEPLOY_DIR_IMAGE and retrieves the ramdisk image and inserts it 
> into its own IMAGE_ROOTFS.
> 
> As I've been working on porting to the Dylan version of the build 
> system, I found the whole issue with do_fetch and do_unpack not being 
> run at all for images. I applied the fix you advised of to add a 
> python function that removes the noexec flag for the two tasks. This 
> apparently only works for one build of the image. I either need to do 
> a "cleanall" on the image or bump the rev to get it to build again.

I haven't had time to try to reproduce this here yet, sorry.
 
However, aside from the files in SRC_URI which could presumably be handled in a separate recipe, one possible solution for the ramdisk part did occur to me - I'd suggest doing the following in the main image:

do_rootfs[depends] += "abc-ramdisk:do_rootfs"

Then add to ROOTFS_POSTPROCESS_COMMAND or IMAGE_POSTPROCESS_COMMAND (whichever is appropriate) a call to a shell function that performs whatever needed operation on the abc-ramdisk.

> Another interesting note that might be of interest is that our LICENCE 
> is defined to "XYZ" and issues the following warning, "WARNING: 
> abc-ramdisk: No generic license file exists for: XYZ in any provider", 
> in the first build that succeeds.

I'm no expert on this area but to me it doesn't make sense for an image containing multiple differently-licensed components to have a declared license in the recipe. We tend to keep things simple and just set LICENSE = "MIT" on the image recipe (*not* implying anything about the contents of the image), with each installed package license and therefore the image's license manifest indicating all of the licenses of the components that make up the image. 
Perhaps a LICENSE value of "N/A" for the image would be more appropriate, although the system currently doesn't support any special casing of LICENSE for image recipes, and if you are pulling in files in SRC_URI in the image recipe there might be some validity to having a LICENSE value to apply to those.

> When the second build is run the warning does not appear and the build 
> jumps right to the do_rootfs task where it fails.

That's because this check is done in do_populate_lic which isn't getting re- executed on the second time (presumably because nothing that that task depends upon has changed).

Cheers,
Paul
 
-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Image recipes in Yocto 1.4 (dylan-9.0.0)

[Paul Eggleton]

On Thursday 01 August 2013 14:00:24 Brian Karcz wrote:
> I'll keep the do_rootfs dependency adjustment for the main image on ready.
> I'd like to try to fix the simple case first of only building the ramdisk
> image directly then trying to rebuild it. With the latter case fixed, the
> former might be fine as is.
> 
> As a sanity check last night, I blew away the entire build directory and
> only built the ramdisk image recipe. I wanted to make sure there was
> nothing left in there from a main image build that might be effecting an
> explicit ramdisk image build. This overnight build succeeded. When I
> performed the rebuild this morning, the same behavior occurred of the
> IMAGE_PREPROCESS_COMMAND not being able to find a file in the WORKDIR.
>
> Sorry for the confusion about the license data. I wasn't interested,
> necessarily, in the content or cause of the warning, but more the task flow
> of the build that was causing the warning to appear in one build and not
> the other. I wasn't sure if the do_populate_lic dependency change might
> trigger any thoughts on the do_fetch dependencies.
> 
> One thing I had a question about was in regards to image vs package recipes
> and the population of the rootfs. Package recipes populate the rootfs by
> appending/prepending the do_install task and install things from the
> WORKDIR to the $D destination directory. 

To be completely accurate, they don't populate the rootfs directly, they 
install files into the ${D} directory (${WORKDIR}/image) during do_install, 
then do_package creates package definitions from this, then do_package_write_* 
create the actual packages based upon the definitions, and then later when the 
image is constructed some subset of those packages may be installed into the 
rootfs.

> Our image recipes, as I've inherited them, create and assign a new
> IMAGE_PREPROCESS_COMMAND, that install files from the WORKDIR to the
> IMAGE_ROOTFS destination directory. Is it abnormal/non-standard for an image
> recipe to have its own SRC_URI entries and subsequent WORKDIR population?

It is considered so, yes. We disabled this functionality by default in the 
denzil release (1.2) on the assumption that image recipes shouldn't really be 
doing this - additional files that need to be pulled in should be pulled in via 
packages created by a separate recipe. It's perfectly fine for the image to be 
modifying or creating small files from scratch particularly if they're 
constructed from dynamic data, but bringing in extra static files should really 
be handled via additional packages.

> I'm wondering if there is a catch-22 going on here regarding the use of
> RM_WORK and the fact that the checksums for the SRC_URI contents haven't
> changed. It seems as though the build is determining that do_fetch doesn't
> need to be run because nothing is changed, like in a package recipe, but
> doesn't realize that do_rootfs will always be executed for an image recipe
> build and might depend on those contents. I'm wondering if this worked
> under Edison because it didn't have the SRC_URI checksums and a different
> dependency model that forced the do_fetch prior to do_rootfs for image
> recipe builds.

Could you run a test for me - if you comment out the setting of noexec on all 
of the tasks at the end image.bbclass does the rebuilding problem go away?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Image recipes in Yocto 1.4 (dylan-9.0.0)

[Brian Karcz]

Hi Paul,

I commented out all the noexec flag settings in image.bbclass and the results are still the same.

I was starting to get the suspicion that all of that content needed to be moved to a separate package recipe. I'll experiment with that and see where I get. Hopefully that, combined with the information about the checksums you gave me on the other thread can get me back to where we are with Edison, but in a more standardized fashion.

Thanks,
Brian


-----Original Message-----
From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com] 
Sent: Thursday, August 01, 2013 10:27 AM
To: Brian Karcz
Cc: yocto@yoctoproject.org
Subject: Re: [yocto] Image recipes in Yocto 1.4 (dylan-9.0.0)

On Thursday 01 August 2013 14:00:24 Brian Karcz wrote:
> I'll keep the do_rootfs dependency adjustment for the main image on ready.
> I'd like to try to fix the simple case first of only building the 
> ramdisk image directly then trying to rebuild it. With the latter case 
> fixed, the former might be fine as is.
> 
> As a sanity check last night, I blew away the entire build directory 
> and only built the ramdisk image recipe. I wanted to make sure there 
> was nothing left in there from a main image build that might be 
> effecting an explicit ramdisk image build. This overnight build 
> succeeded. When I performed the rebuild this morning, the same 
> behavior occurred of the IMAGE_PREPROCESS_COMMAND not being able to find a file in the WORKDIR.
>
> Sorry for the confusion about the license data. I wasn't interested, 
> necessarily, in the content or cause of the warning, but more the task 
> flow of the build that was causing the warning to appear in one build 
> and not the other. I wasn't sure if the do_populate_lic dependency 
> change might trigger any thoughts on the do_fetch dependencies.
> 
> One thing I had a question about was in regards to image vs package 
> recipes and the population of the rootfs. Package recipes populate the 
> rootfs by appending/prepending the do_install task and install things 
> from the WORKDIR to the $D destination directory.

To be completely accurate, they don't populate the rootfs directly, they install files into the ${D} directory (${WORKDIR}/image) during do_install, then do_package creates package definitions from this, then do_package_write_* create the actual packages based upon the definitions, and then later when the image is constructed some subset of those packages may be installed into the rootfs.

> Our image recipes, as I've inherited them, create and assign a new 
> IMAGE_PREPROCESS_COMMAND, that install files from the WORKDIR to the 
> IMAGE_ROOTFS destination directory. Is it abnormal/non-standard for an 
> image recipe to have its own SRC_URI entries and subsequent WORKDIR population?

It is considered so, yes. We disabled this functionality by default in the denzil release (1.2) on the assumption that image recipes shouldn't really be doing this - additional files that need to be pulled in should be pulled in via packages created by a separate recipe. It's perfectly fine for the image to be modifying or creating small files from scratch particularly if they're constructed from dynamic data, but bringing in extra static files should really be handled via additional packages.

> I'm wondering if there is a catch-22 going on here regarding the use 
> of RM_WORK and the fact that the checksums for the SRC_URI contents 
> haven't changed. It seems as though the build is determining that 
> do_fetch doesn't need to be run because nothing is changed, like in a 
> package recipe, but doesn't realize that do_rootfs will always be 
> executed for an image recipe build and might depend on those contents. 
> I'm wondering if this worked under Edison because it didn't have the 
> SRC_URI checksums and a different dependency model that forced the 
> do_fetch prior to do_rootfs for image recipe builds.

Could you run a test for me - if you comment out the setting of noexec on all of the tasks at the end image.bbclass does the rebuilding problem go away?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre