Create more than one image with WIC

Create more than one image with WIC

[Daniel Schultz]

Hi,

I created two kickstart files (am335x-sdimage.wks, am335x-emmc.wks) and 
added them to the local.conf.

When I build the image only the first wks in WKS_FILES will be used by 
WIC and the second will be ignored. Is it possible to build two images 
in one build?

Includes of the wks files in local.conf:
WKS_FILES_ti33x = "am335x-sdimage.wks am335x-emmc.wks "

-- 
Mit freundlichen Grüßen,
With best regards,
   Daniel Schultz

Re: Create more than one image with WIC

[Ed Bartosh]

On Wed, Mar 08, 2017 at 10:44:21AM +0100, Daniel Schultz wrote:
> Hi,
> 
> I created two kickstart files (am335x-sdimage.wks, am335x-emmc.wks)
> and added them to the local.conf.
> 
> When I build the image only the first wks in WKS_FILES will be used
> by WIC and the second will be ignored. Is it possible to build two
> images in one build?
>

I don't think it's possible to build more than one image for the same
type. wic is not an exception here.

> Includes of the wks files in local.conf:
> WKS_FILES_ti33x = "am335x-sdimage.wks am335x-emmc.wks "

WKS_FILES variable is to provide possible wks files to use. First one found will
be used to produce an image.

--
Regards,
Ed

Re: Create more than one image with WIC

[Gary Thomas]

On 2017-03-08 11:57, Ed Bartosh wrote:
> On Wed, Mar 08, 2017 at 10:44:21AM +0100, Daniel Schultz wrote:
>> Hi,
>>
>> I created two kickstart files (am335x-sdimage.wks, am335x-emmc.wks)
>> and added them to the local.conf.
>>
>> When I build the image only the first wks in WKS_FILES will be used
>> by WIC and the second will be ignored. Is it possible to build two
>> images in one build?
>>
>
> I don't think it's possible to build more than one image for the same
> type. wic is not an exception here.
>
>> Includes of the wks files in local.conf:
>> WKS_FILES_ti33x = "am335x-sdimage.wks am335x-emmc.wks "
>
> WKS_FILES variable is to provide possible wks files to use. First one found will
> be used to produce an image.

Would it work to add/define this variable in the corresponding *image*.bb recipe
rather than local.conf?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Create more than one image with WIC

[Ed Bartosh]

On Wed, Mar 08, 2017 at 02:41:10PM +0100, Gary Thomas wrote:
> On 2017-03-08 11:57, Ed Bartosh wrote:
> >On Wed, Mar 08, 2017 at 10:44:21AM +0100, Daniel Schultz wrote:
> >>Hi,
> >>
> >>I created two kickstart files (am335x-sdimage.wks, am335x-emmc.wks)
> >>and added them to the local.conf.
> >>
> >>When I build the image only the first wks in WKS_FILES will be used
> >>by WIC and the second will be ignored. Is it possible to build two
> >>images in one build?
> >>
> >
> >I don't think it's possible to build more than one image for the same
> >type. wic is not an exception here.
> >
> >>Includes of the wks files in local.conf:
> >>WKS_FILES_ti33x = "am335x-sdimage.wks am335x-emmc.wks "
> >
> >WKS_FILES variable is to provide possible wks files to use. First one found will
> >be used to produce an image.
> 
> Would it work to add/define this variable in the corresponding *image*.bb recipe
> rather than local.conf?

Yes, it would if image recipe is unique, i.e. images of the same type
can be produced for different recipes.

WKS_FILES makes sense to set only if there are multiple wks files that
can be used to produce an image. The most obvious use case is using
multiple layers with different wks files in each and specifying all of
them in WKS_FILES.

If this is not the case then setting WKS_FILE is enough.

--
Regards,
Ed

Re: Create more than one image with WIC

[Daniel Schultz]

Am 08.03.2017 um 14:43 schrieb Ed Bartosh:
> On Wed, Mar 08, 2017 at 02:41:10PM +0100, Gary Thomas wrote:
>> On 2017-03-08 11:57, Ed Bartosh wrote:
>>> On Wed, Mar 08, 2017 at 10:44:21AM +0100, Daniel Schultz wrote:
>>>> Hi,
>>>>
>>>> I created two kickstart files (am335x-sdimage.wks, am335x-emmc.wks)
>>>> and added them to the local.conf.
>>>>
>>>> When I build the image only the first wks in WKS_FILES will be used
>>>> by WIC and the second will be ignored. Is it possible to build two
>>>> images in one build?
>>>>
>>>
>>> I don't think it's possible to build more than one image for the same
>>> type. wic is not an exception here.
>>>
>>>> Includes of the wks files in local.conf:
>>>> WKS_FILES_ti33x = "am335x-sdimage.wks am335x-emmc.wks "
>>>
>>> WKS_FILES variable is to provide possible wks files to use. First one found will
>>> be used to produce an image.
>>
>> Would it work to add/define this variable in the corresponding *image*.bb recipe
>> rather than local.conf?
>
> Yes, it would if image recipe is unique, i.e. images of the same type
> can be produced for different recipes.
>
But isn't wic different to other images like e.g. ext? I agree that 
there are other images that need only build once, but wic images can 
contain these and therefore act as a wrapper.

Do I have misunderstood the design of wic?

I mean MMC and EMMC are very similar, but different in the bootloader 
and it would be a overhead to create an extra image for all EMMC images.

-- 
Mit freundlichen Grüßen,
With best regards,
   Daniel Schultz

> WKS_FILES makes sense to set only if there are multiple wks files that
> can be used to produce an image. The most obvious use case is using
> multiple layers with different wks files in each and specifying all of
> them in WKS_FILES.
>
> If this is not the case then setting WKS_FILE is enough.
>
> --
> Regards,
> Ed
>

Re: Create more than one image with WIC

[Ed Bartosh]

On Mon, Mar 13, 2017 at 05:16:31PM +0100, Daniel Schultz wrote:
> 
> 
> Am 08.03.2017 um 14:43 schrieb Ed Bartosh:
> >On Wed, Mar 08, 2017 at 02:41:10PM +0100, Gary Thomas wrote:
> >>On 2017-03-08 11:57, Ed Bartosh wrote:
> >>>On Wed, Mar 08, 2017 at 10:44:21AM +0100, Daniel Schultz wrote:
> >>>>Hi,
> >>>>
> >>>>I created two kickstart files (am335x-sdimage.wks, am335x-emmc.wks)
> >>>>and added them to the local.conf.
> >>>>
> >>>>When I build the image only the first wks in WKS_FILES will be used
> >>>>by WIC and the second will be ignored. Is it possible to build two
> >>>>images in one build?
> >>>>
> >>>
> >>>I don't think it's possible to build more than one image for the same
> >>>type. wic is not an exception here.
> >>>
> >>>>Includes of the wks files in local.conf:
> >>>>WKS_FILES_ti33x = "am335x-sdimage.wks am335x-emmc.wks "
> >>>
> >>>WKS_FILES variable is to provide possible wks files to use. First one found will
> >>>be used to produce an image.
> >>
> >>Would it work to add/define this variable in the corresponding *image*.bb recipe
> >>rather than local.conf?
> >
> >Yes, it would if image recipe is unique, i.e. images of the same type
> >can be produced for different recipes.
> >
> But isn't wic different to other images like e.g. ext? I agree that
> there are other images that need only build once, but wic images can
> contain these and therefore act as a wrapper.
>
Makes sense.

> Do I have misunderstood the design of wic?
>
> I mean MMC and EMMC are very similar, but different in the
> bootloader and it would be a overhead to create an extra image for
> all EMMC images.

You don't need to create more than one rootfs with the same content. You
can refer to the same rootfs in multiple .wks files, but the recipes
have to be unique. In your example MMC and EMMC image recipes can create
directories for boot partition and their .wks files can use those
directories and common root directory produced by another recipe, e.g.
by core-image-minimal.

It's not a big deal to run wic sequentially to produce multiple images,
but it will create more problems than it solves I believe. It will be slower and
will make a mess in image naming in deploy directory.

--
Regards,
Ed

Re: Create more than one image with WIC

[Patrick Ohly]

On Tue, 2017-03-14 at 19:11 +0200, Ed Bartosh wrote:
> On Mon, Mar 13, 2017 at 05:16:31PM +0100, Daniel Schultz wrote:
> > Do I have misunderstood the design of wic?
> >
> > I mean MMC and EMMC are very similar, but different in the
> > bootloader and it would be a overhead to create an extra image for
> > all EMMC images.
> 
> You don't need to create more than one rootfs with the same content. You
> can refer to the same rootfs in multiple .wks files, but the recipes
> have to be unique. In your example MMC and EMMC image recipes can create
> directories for boot partition and their .wks files can use those
> directories and common root directory produced by another recipe, e.g.
> by core-image-minimal.

But beware of interaction with rm_work.bbclass when doing that...

> It's not a big deal to run wic sequentially to produce multiple images,
> but it will create more problems than it solves I believe. It will be slower and
> will make a mess in image naming in deploy directory.

I personally would prefer to support generating more than one wic image
per image recipe. The different output files could be distinguished with
an additional suffix (".efi.wic", ".live.wiC", etc.) where the suffix is
either specified by the WKS file or the configuration where the WKS file
is selected - the latter might be a bit more flexible.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: Create more than one image with WIC

[Ed Bartosh]

On Tue, Mar 14, 2017 at 06:49:45PM +0100, Patrick Ohly wrote:
> 
> > It's not a big deal to run wic sequentially to produce multiple images,
> > but it will create more problems than it solves I believe. It will be slower and
> > will make a mess in image naming in deploy directory.
> 
> I personally would prefer to support generating more than one wic image
> per image recipe. The different output files could be distinguished with
> an additional suffix (".efi.wic", ".live.wiC", etc.) where the suffix is
> either specified by the WKS file or the configuration where the WKS file
> is selected - the latter might be a bit more flexible.
>

From my point of view ths would be against the current design of oe
image building susbsystem. It would be also more complex and slower
unless we'll find a way to dynamically generate multiple image creation
tasks per one recipe and run them in parallel. In this case it will be
even more complex.

--
Regards,
Ed

Re: Create more than one image with WIC

[Patrick Ohly]

On Tue, 2017-03-14 at 20:06 +0200, Ed Bartosh wrote:
> On Tue, Mar 14, 2017 at 06:49:45PM +0100, Patrick Ohly wrote:
> > 
> > > It's not a big deal to run wic sequentially to produce multiple images,
> > > but it will create more problems than it solves I believe. It will be slower and
> > > will make a mess in image naming in deploy directory.
> > 
> > I personally would prefer to support generating more than one wic image
> > per image recipe. The different output files could be distinguished with
> > an additional suffix (".efi.wic", ".live.wiC", etc.) where the suffix is
> > either specified by the WKS file or the configuration where the WKS file
> > is selected - the latter might be a bit more flexible.
> >
> 
> From my point of view ths would be against the current design of oe
> image building susbsystem. It would be also more complex and slower
> unless we'll find a way to dynamically generate multiple image creation
> tasks per one recipe and run them in parallel. In this case it will be
> even more complex.

That's a good point. However, directly sharing the content of the build
directory from different recipes is also against the OE design and
requires special attention, for example to suppressing the do_rootfs
task in one recipe and preventing do_rm_work in the other.

Either way, something that works out of the box might be useful.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: Create more than one image with WIC

[Ed Bartosh]

On Wed, Mar 15, 2017 at 08:24:34AM +0100, Patrick Ohly wrote:
> On Tue, 2017-03-14 at 20:06 +0200, Ed Bartosh wrote:
> > On Tue, Mar 14, 2017 at 06:49:45PM +0100, Patrick Ohly wrote:
> > > 
> > > > It's not a big deal to run wic sequentially to produce multiple images,
> > > > but it will create more problems than it solves I believe. It will be slower and
> > > > will make a mess in image naming in deploy directory.
> > > 
> > > I personally would prefer to support generating more than one wic image
> > > per image recipe. The different output files could be distinguished with
> > > an additional suffix (".efi.wic", ".live.wiC", etc.) where the suffix is
> > > either specified by the WKS file or the configuration where the WKS file
> > > is selected - the latter might be a bit more flexible.
> > >
> > 
> > From my point of view ths would be against the current design of oe
> > image building susbsystem. It would be also more complex and slower
> > unless we'll find a way to dynamically generate multiple image creation
> > tasks per one recipe and run them in parallel. In this case it will be
> > even more complex.
> 
> That's a good point. However, directly sharing the content of the build
> directory from different recipes is also against the OE design and
> requires special attention, for example to suppressing the do_rootfs
> task in one recipe and preventing do_rm_work in the other.
> 
> Either way, something that works out of the box might be useful.

I'm going to address this in near future when I'll be working on
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10073
Feel free to comment on it, btw.

So far the plan is to treat EFI boot partition as a rootfs. wic job
would be to simply put content of multiple rootfs-es into partitions of
the final image. No suppressing of do_rootfs would be needed, I hope.

Regarding do_rm_work. It should not touch rootfs directories, I believe.

--
Regards,
Ed

Re: Create more than one image with WIC

[Patrick Ohly]

On Wed, 2017-03-15 at 14:58 +0200, Ed Bartosh wrote:
> Regarding do_rm_work. It should not touch rootfs directories, I believe.

It does, and it should by default because a rootfs can be quite large.
If it's not going to be reused in another recipe, then it is worthwhile
to remove it.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: Create more than one image with WIC

[Patrick Ohly]

On Wed, 2017-03-15 at 14:39 +0100, Patrick Ohly wrote:
> On Wed, 2017-03-15 at 14:58 +0200, Ed Bartosh wrote:
> > Regarding do_rm_work. It should not touch rootfs directories, I believe.
> 
> It does, and it should by default because a rootfs can be quite large.
> If it's not going to be reused in another recipe, then it is worthwhile
> to remove it.

I should add that RM_WORK_EXCLUDE_ITEMS += "rootfs" can be used in image
recipes which know that their rootfs is going to be needed elsewhere -
it's just not the default.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: Create more than one image with WIC

[Ed Bartosh]

On Wed, Mar 15, 2017 at 02:41:34PM +0100, Patrick Ohly wrote:
> On Wed, 2017-03-15 at 14:39 +0100, Patrick Ohly wrote:
> > On Wed, 2017-03-15 at 14:58 +0200, Ed Bartosh wrote:
> > > Regarding do_rm_work. It should not touch rootfs directories, I believe.
> > 

> It does, and it should by default because a rootfs can be quite large.
> If it's not going to be reused in another recipe, then it is worthwhile
> to remove it.
This is true unless we're going to use wic as a stand-alone tool, which some
people still do.

> I should add that RM_WORK_EXCLUDE_ITEMS += "rootfs" can be used in image
> recipes which know that their rootfs is going to be needed elsewhere -
> it's just not the default.

Isn't rootfs going to be rebuilt if one rootfs recipe depends on another one?

--
Regards,
Ed

Re: Create more than one image with WIC

[Ed Bartosh]

On Wed, Mar 15, 2017 at 04:01:43PM +0200, Ed Bartosh wrote:
> On Wed, Mar 15, 2017 at 02:41:34PM +0100, Patrick Ohly wrote:
> > On Wed, 2017-03-15 at 14:39 +0100, Patrick Ohly wrote:
> > > On Wed, 2017-03-15 at 14:58 +0200, Ed Bartosh wrote:
> > > > Regarding do_rm_work. It should not touch rootfs directories, I believe.
> > > 
> 
> > It does, and it should by default because a rootfs can be quite large.
> > If it's not going to be reused in another recipe, then it is worthwhile
> > to remove it.
> This is true unless we're going to use wic as a stand-alone tool, which some
> people still do.
> 
> > I should add that RM_WORK_EXCLUDE_ITEMS += "rootfs" can be used in image
> > recipes which know that their rootfs is going to be needed elsewhere -
> > it's just not the default.
> 
> Isn't rootfs going to be rebuilt if one rootfs recipe depends on another one?

Here is an example of dependency I'm talking about:
wic-image-minimal can be built just fine with enabled rm_work.
Its .wks uses 2 rootfs: core-image-minimal and wic-image-minimal.

I didn't use any RM_WORK_EXCLUDE_ITEMS. I did specified dependency to
core-image-minimal in wic-image-minimal recipe:

# core-image-minimal is referenced in .wks, so we need its rootfs
# to be ready before our rootfs
do_rootfs[depends] += "core-image-minimal:do_image core-image-minimal:do_rootfs_wicenv"

Am I missing something here?

--
Regards,
Ed

Re: Create more than one image with WIC

[Patrick Ohly]

On Wed, 2017-03-15 at 16:47 +0200, Ed Bartosh wrote:
> On Wed, Mar 15, 2017 at 04:01:43PM +0200, Ed Bartosh wrote:
> > On Wed, Mar 15, 2017 at 02:41:34PM +0100, Patrick Ohly wrote:
> > > On Wed, 2017-03-15 at 14:39 +0100, Patrick Ohly wrote:
> > > > On Wed, 2017-03-15 at 14:58 +0200, Ed Bartosh wrote:
> > > > > Regarding do_rm_work. It should not touch rootfs directories, I believe.
> > > > 
> > 
> > > It does, and it should by default because a rootfs can be quite large.
> > > If it's not going to be reused in another recipe, then it is worthwhile
> > > to remove it.
> > This is true unless we're going to use wic as a stand-alone tool, which some
> > people still do.
> > 
> > > I should add that RM_WORK_EXCLUDE_ITEMS += "rootfs" can be used in image
> > > recipes which know that their rootfs is going to be needed elsewhere -
> > > it's just not the default.
> > 
> > Isn't rootfs going to be rebuilt if one rootfs recipe depends on another one?
> 
> Here is an example of dependency I'm talking about:
> wic-image-minimal can be built just fine with enabled rm_work.
> Its .wks uses 2 rootfs: core-image-minimal and wic-image-minimal.
> 
> I didn't use any RM_WORK_EXCLUDE_ITEMS. I did specified dependency to
> core-image-minimal in wic-image-minimal recipe:
> 
> # core-image-minimal is referenced in .wks, so we need its rootfs
> # to be ready before our rootfs
> do_rootfs[depends] += "core-image-minimal:do_image core-image-minimal:do_rootfs_wicenv"
> 
> Am I missing something here?

I'm pretty sure you have a race condition, you just haven't triggered
the failure case.

Perhaps you were also lucky because my "rm_work.bbclass: re-enable
recursive do_rm_work_all" is pending and thus
core-image-minimal:do_rm_work never ran at all.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: Create more than one image with WIC

[Patrick Ohly]

On Wed, 2017-03-15 at 16:01 +0200, Ed Bartosh wrote:
> On Wed, Mar 15, 2017 at 02:41:34PM +0100, Patrick Ohly wrote:
> > On Wed, 2017-03-15 at 14:39 +0100, Patrick Ohly wrote:
> > > On Wed, 2017-03-15 at 14:58 +0200, Ed Bartosh wrote:
> > > > Regarding do_rm_work. It should not touch rootfs directories, I believe.
> > > 
> 
> > It does, and it should by default because a rootfs can be quite large.
> > If it's not going to be reused in another recipe, then it is worthwhile
> > to remove it.
> This is true unless we're going to use wic as a stand-alone tool, which some
> people still do.
> 
> > I should add that RM_WORK_EXCLUDE_ITEMS += "rootfs" can be used in image
> > recipes which know that their rootfs is going to be needed elsewhere -
> > it's just not the default.
> 
> Isn't rootfs going to be rebuilt if one rootfs recipe depends on another one?

No, that's now how it works. Suppose there's a
image-a:do_image_wic->image-b:do_rootfs task dependency. That dependency
ensures that image-b:do_rootfs runs before image-a:do_image_wic. But it
does not delay image-b:do_rm_work until image-a:do_image_wic is done.

That could be fixed by also adding a
image-b:do_rm_work->image-a:do_image_wic dependency. As I said,
something that works out of the box for this use case would be nice...
we can't expect developers to figure all of this out themselves.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.