When to create a new build directory for yocto?

When to create a new build directory for yocto?

[Qiang Yu]

[-- Attachment #1: Type: text/plain, Size: 1364 bytes --]

Hi all,

I'm maintaining a Yocto build system to produce these outputs:

1. ARM SOC 1
  1). board 1 image
  2). board 2 image
  3). SDK for i686
  4). SDK for x86_64
2. ARM SOC 2
  1). board 3 image
  2). board 4 image
  3). SDK for i686
  4). SDK for x86_64

Now I don't know the right way to create build directories for these
builds. The simplest way
I think is creating one build directory for each of these builds. But it
wast resource for boards
with the same SOC which can share cross toolchain. And ARM SOC 1 and ARM
SOC 2 use
different conf/bblayers.conf to add different BSP layer, I need to modify
it when switch SOC.
So the right way is creating one build directory for each SOC, then bitbake
each output's recipe,
right?

I am also not sure if I can put the SDK and board image in the same build
directory. Because
once I build a board's image after creating its SDK, I add gdbserver to it,
but the build fail without
really building the gdbserver package. So I guess the board image build is
confused by the SDK
build state.

So the conclusion is I need a recommended way to separate output into
different build directories:
1. minimize rebuild shared packages
2. no build state corruption
3. if multi output in the same build dir, no change to conf/ files or an
easy way to switch between

Regards,
Qiang

[-- Attachment #2: Type: text/html, Size: 1624 bytes --]

Re: When to create a new build directory for yocto?

[ChenQi]

[-- Attachment #1: Type: text/plain, Size: 1636 bytes --]

I would suggest to use separate build directories but let these builds 
to share downloads and sstate.

Regards,
Chen Qi

On 11/30/2014 12:49 PM, Qiang Yu wrote:
> Hi all,
>
> I'm maintaining a Yocto build system to produce these outputs:
>
> 1. ARM SOC 1
>   1). board 1 image
>   2). board 2 image
>   3). SDK for i686
>   4). SDK for x86_64
> 2. ARM SOC 2
>   1). board 3 image
>   2). board 4 image
>   3). SDK for i686
>   4). SDK for x86_64
>
> Now I don't know the right way to create build directories for these 
> builds. The simplest way
> I think is creating one build directory for each of these builds. But 
> it wast resource for boards
> with the same SOC which can share cross toolchain. And ARM SOC 1 and 
> ARM SOC 2 use
> different conf/bblayers.conf to add different BSP layer, I need to 
> modify it when switch SOC.
> So the right way is creating one build directory for each SOC, then 
> bitbake each output's recipe,
> right?
>
> I am also not sure if I can put the SDK and board image in the same 
> build directory. Because
> once I build a board's image after creating its SDK, I add gdbserver 
> to it, but the build fail without
> really building the gdbserver package. So I guess the board image 
> build is confused by the SDK
> build state.
>
> So the conclusion is I need a recommended way to separate output into 
> different build directories:
> 1. minimize rebuild shared packages
> 2. no build state corruption
> 3. if multi output in the same build dir, no change to conf/ files or 
> an easy way to switch between
>
> Regards,
> Qiang
>
>
>


[-- Attachment #2: Type: text/html, Size: 2893 bytes --]

Re: When to create a new build directory for yocto?

[Qiang Yu]

[-- Attachment #1: Type: text/plain, Size: 2013 bytes --]

Why share the sstate? Does it contain previously build ipk packages in
other build directories for reuse?

Regards,
Qiang

On Sun, Nov 30, 2014 at 4:03 PM, ChenQi <Qi.Chen@windriver.com> wrote:

>  I would suggest to use separate build directories but let these builds
> to share downloads and sstate.
>
> Regards,
> Chen Qi
>
>
> On 11/30/2014 12:49 PM, Qiang Yu wrote:
>
> Hi all,
>
> I'm maintaining a Yocto build system to produce these outputs:
>
> 1. ARM SOC 1
>   1). board 1 image
>   2). board 2 image
>   3). SDK for i686
>   4). SDK for x86_64
> 2. ARM SOC 2
>   1). board 3 image
>   2). board 4 image
>   3). SDK for i686
>   4). SDK for x86_64
>
> Now I don't know the right way to create build directories for these
> builds. The simplest way
> I think is creating one build directory for each of these builds. But it
> wast resource for boards
> with the same SOC which can share cross toolchain. And ARM SOC 1 and ARM
> SOC 2 use
> different conf/bblayers.conf to add different BSP layer, I need to modify
> it when switch SOC.
> So the right way is creating one build directory for each SOC, then
> bitbake each output's recipe,
> right?
>
> I am also not sure if I can put the SDK and board image in the same build
> directory. Because
> once I build a board's image after creating its SDK, I add gdbserver to
> it, but the build fail without
> really building the gdbserver package. So I guess the board image build is
> confused by the SDK
> build state.
>
>  So the conclusion is I need a recommended way to separate output into
> different build directories:
> 1. minimize rebuild shared packages
> 2. no build state corruption
> 3. if multi output in the same build dir, no change to conf/ files or an
> easy way to switch between
>
>  Regards,
> Qiang
>
>
>
>
>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>
>

[-- Attachment #2: Type: text/html, Size: 3699 bytes --]

Re: When to create a new build directory for yocto?

[Qiang Yu]

[-- Attachment #1: Type: text/plain, Size: 2799 bytes --]

Now after study share state cache in yocto document, I know it can be used
by multi build dir
to share IPK files. But if all the build dir share the same sstate_cache,
why don't they just use
the same build dir? I can only think one reason: needn't change conf/ files
when switching output.
But when in the same SOC, the conf/ also need not be changed, just switch
output recipes.

And if every output has its own build dir with the same sstate_cache, the
build stage is ignored,
but the IPK will have multi copy in multi build dir. This will wast disk
space. So may be this is
not the best way?

Regards,
Qiang


On Sun, Nov 30, 2014 at 10:28 PM, Qiang Yu <yuq825@gmail.com> wrote:

> Why share the sstate? Does it contain previously build ipk packages in
> other build directories for reuse?
>
> Regards,
> Qiang
>
> On Sun, Nov 30, 2014 at 4:03 PM, ChenQi <Qi.Chen@windriver.com> wrote:
>
>>  I would suggest to use separate build directories but let these builds
>> to share downloads and sstate.
>>
>> Regards,
>> Chen Qi
>>
>>
>> On 11/30/2014 12:49 PM, Qiang Yu wrote:
>>
>> Hi all,
>>
>> I'm maintaining a Yocto build system to produce these outputs:
>>
>> 1. ARM SOC 1
>>   1). board 1 image
>>   2). board 2 image
>>   3). SDK for i686
>>   4). SDK for x86_64
>> 2. ARM SOC 2
>>   1). board 3 image
>>   2). board 4 image
>>   3). SDK for i686
>>   4). SDK for x86_64
>>
>> Now I don't know the right way to create build directories for these
>> builds. The simplest way
>> I think is creating one build directory for each of these builds. But it
>> wast resource for boards
>> with the same SOC which can share cross toolchain. And ARM SOC 1 and ARM
>> SOC 2 use
>> different conf/bblayers.conf to add different BSP layer, I need to modify
>> it when switch SOC.
>> So the right way is creating one build directory for each SOC, then
>> bitbake each output's recipe,
>> right?
>>
>> I am also not sure if I can put the SDK and board image in the same build
>> directory. Because
>> once I build a board's image after creating its SDK, I add gdbserver to
>> it, but the build fail without
>> really building the gdbserver package. So I guess the board image build
>> is confused by the SDK
>> build state.
>>
>>  So the conclusion is I need a recommended way to separate output into
>> different build directories:
>> 1. minimize rebuild shared packages
>> 2. no build state corruption
>> 3. if multi output in the same build dir, no change to conf/ files or an
>> easy way to switch between
>>
>>  Regards,
>> Qiang
>>
>>
>>
>>
>>
>> --
>> _______________________________________________
>> yocto mailing list
>> yocto@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/yocto
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 4897 bytes --]

Re: When to create a new build directory for yocto?

[Qiang Yu]

[-- Attachment #1: Type: text/plain, Size: 2581 bytes --]

Hi Chen Qi,

Thanks for your suggestion.

I mean the tmp/deploy/ipk dir will have one copy each build dir. And two
boards of the same SOC will
share most of them, so the disk space of the tmp/deploy/ipk part is wasted.

Regards,
Qiang


On Mon, Dec 1, 2014 at 10:01 AM, ChenQi <Qi.Chen@windriver.com> wrote:

>  Sharing sstate is safe and efficient.
> sstate does not contain any package.
>
> Regards,
> Chen Qi
>
>
> On 11/30/2014 10:28 PM, Qiang Yu wrote:
>
> Why share the sstate? Does it contain previously build ipk packages in
> other build directories for reuse?
>
>  Regards,
> Qiang
>
> On Sun, Nov 30, 2014 at 4:03 PM, ChenQi <Qi.Chen@windriver.com> wrote:
>
>>  I would suggest to use separate build directories but let these builds
>> to share downloads and sstate.
>>
>> Regards,
>> Chen Qi
>>
>>
>> On 11/30/2014 12:49 PM, Qiang Yu wrote:
>>
>> Hi all,
>>
>> I'm maintaining a Yocto build system to produce these outputs:
>>
>> 1. ARM SOC 1
>>   1). board 1 image
>>   2). board 2 image
>>   3). SDK for i686
>>   4). SDK for x86_64
>> 2. ARM SOC 2
>>   1). board 3 image
>>   2). board 4 image
>>   3). SDK for i686
>>   4). SDK for x86_64
>>
>> Now I don't know the right way to create build directories for these
>> builds. The simplest way
>> I think is creating one build directory for each of these builds. But it
>> wast resource for boards
>> with the same SOC which can share cross toolchain. And ARM SOC 1 and ARM
>> SOC 2 use
>> different conf/bblayers.conf to add different BSP layer, I need to modify
>> it when switch SOC.
>> So the right way is creating one build directory for each SOC, then
>> bitbake each output's recipe,
>> right?
>>
>> I am also not sure if I can put the SDK and board image in the same build
>> directory. Because
>> once I build a board's image after creating its SDK, I add gdbserver to
>> it, but the build fail without
>> really building the gdbserver package. So I guess the board image build
>> is confused by the SDK
>> build state.
>>
>>  So the conclusion is I need a recommended way to separate output into
>> different build directories:
>> 1. minimize rebuild shared packages
>> 2. no build state corruption
>> 3. if multi output in the same build dir, no change to conf/ files or an
>> easy way to switch between
>>
>>  Regards,
>> Qiang
>>
>>
>>
>>
>>
>> --
>> _______________________________________________
>> yocto mailing list
>> yocto@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/yocto
>>
>>
>
>

[-- Attachment #2: Type: text/html, Size: 6071 bytes --]

Re: When to create a new build directory for yocto?

[ChenQi]

[-- Attachment #1: Type: text/plain, Size: 3738 bytes --]

Hi Qiang,

It's possible that you are using (or might use) different layer 
configurations for different builds.
One build might have a bbappend file that another build doesn't need.
Also, different builds may have different DISTRO_FEATURES.
Thus, having different deploy directories is better. Otherwise, your 
package feeds might be broken. And you would suffer trying to maintain it.

If space is a concern for you, inherit 'rm_work' in local.conf.

INHERIT += "rm_work"

Best Regards,
Chen Qi

On 12/01/2014 11:01 AM, Qiang Yu wrote:
> Hi Chen Qi,
>
> Thanks for your suggestion.
>
> I mean the tmp/deploy/ipk dir will have one copy each build dir. And 
> two boards of the same SOC will
> share most of them, so the disk space of the tmp/deploy/ipk part is 
> wasted.
>
> Regards,
> Qiang
>
>
> On Mon, Dec 1, 2014 at 10:01 AM, ChenQi <Qi.Chen@windriver.com 
> <mailto:Qi.Chen@windriver.com>> wrote:
>
>     Sharing sstate is safe and efficient.
>     sstate does not contain any package.
>
>     Regards,
>     Chen Qi
>
>
>     On 11/30/2014 10:28 PM, Qiang Yu wrote:
>>     Why share the sstate? Does it contain previously build ipk
>>     packages in other build directories for reuse?
>>
>>     Regards,
>>     Qiang
>>
>>     On Sun, Nov 30, 2014 at 4:03 PM, ChenQi <Qi.Chen@windriver.com
>>     <mailto:Qi.Chen@windriver.com>> wrote:
>>
>>         I would suggest to use separate build directories but let
>>         these builds to share downloads and sstate.
>>
>>         Regards,
>>         Chen Qi
>>
>>
>>         On 11/30/2014 12:49 PM, Qiang Yu wrote:
>>>         Hi all,
>>>
>>>         I'm maintaining a Yocto build system to produce these outputs:
>>>
>>>         1. ARM SOC 1
>>>           1). board 1 image
>>>           2). board 2 image
>>>           3). SDK for i686
>>>           4). SDK for x86_64
>>>         2. ARM SOC 2
>>>           1). board 3 image
>>>           2). board 4 image
>>>           3). SDK for i686
>>>           4). SDK for x86_64
>>>
>>>         Now I don't know the right way to create build directories
>>>         for these builds. The simplest way
>>>         I think is creating one build directory for each of these
>>>         builds. But it wast resource for boards
>>>         with the same SOC which can share cross toolchain. And ARM
>>>         SOC 1 and ARM SOC 2 use
>>>         different conf/bblayers.conf to add different BSP layer, I
>>>         need to modify it when switch SOC.
>>>         So the right way is creating one build directory for
>>>         each SOC, then bitbake each output's recipe,
>>>         right?
>>>
>>>         I am also not sure if I can put the SDK and board image in
>>>         the same build directory. Because
>>>         once I build a board's image after creating its SDK, I add
>>>         gdbserver to it, but the build fail without
>>>         really building the gdbserver package. So I guess the board
>>>         image build is confused by the SDK
>>>         build state.
>>>
>>>         So the conclusion is I need a recommended way to separate
>>>         output into different build directories:
>>>         1. minimize rebuild shared packages
>>>         2. no build state corruption
>>>         3. if multi output in the same build dir, no change to conf/
>>>         files or an easy way to switch between
>>>
>>>         Regards,
>>>         Qiang
>>>
>>>
>>>
>>
>>
>>         --
>>         _______________________________________________
>>         yocto mailing list
>>         yocto@yoctoproject.org <mailto:yocto@yoctoproject.org>
>>         https://lists.yoctoproject.org/listinfo/yocto
>>
>>
>
>


[-- Attachment #2: Type: text/html, Size: 10048 bytes --]

Re: When to create a new build directory for yocto?

[Qiang Yu]

[-- Attachment #1: Type: text/plain, Size: 802 bytes --]

>
>
> It's possible that you are using (or might use) different layer
> configurations for different builds.
> One build might have a bbappend file that another build doesn't need.
>
Yes, you are right. I use different conf/bblayers.conf for different SOC.
But within the same SOC's different output
(board image, SDK), I just change MACHINE and SDKMACHINE. Is it safe to
build all outputs of the same SOC
in one build dir?


> Also, different builds may have different DISTRO_FEATURES.
> Thus, having different deploy directories is better. Otherwise, your
> package feeds might be broken. And you would suffer trying to maintain it.
>
You mean I can't build two output with different DISTRO_FEATURES in the
same build dir.
Any other config var? What about MACHINE and SDKMACHINE?

[-- Attachment #2: Type: text/html, Size: 1473 bytes --]

Re: When to create a new build directory for yocto?

[ChenQi]

[-- Attachment #1: Type: text/plain, Size: 1101 bytes --]

On 12/01/2014 12:45 PM, Qiang Yu wrote:
>
>
>     It's possible that you are using (or might use) different layer
>     configurations for different builds.
>     One build might have a bbappend file that another build doesn't need.
>
> Yes, you are right. I use different conf/bblayers.conf for different 
> SOC. But within the same SOC's different output
> (board image, SDK), I just change MACHINE and SDKMACHINE. Is it safe 
> to build all outputs of the same SOC
> in one build dir?

Yes.

>     Also, different builds may have different DISTRO_FEATURES.
>     Thus, having different deploy directories is better. Otherwise,
>     your package feeds might be broken. And you would suffer trying to
>     maintain it.
>
> You mean I can't build two output with different DISTRO_FEATURES in 
> the same build dir.
> Any other config var? What about MACHINE and SDKMACHINE?

Yocto doesn't ensure that changing distro features in the same build 
directory works.
It might work, but it's also possible that it doesn't.

Changing MACHINE and SDKMACHINE is OK.

//Chen Qi

[-- Attachment #2: Type: text/html, Size: 3013 bytes --]

Re: When to create a new build directory for yocto?

[Qiang Yu]

[-- Attachment #1: Type: text/plain, Size: 1239 bytes --]

What about MACHINE_FEATURES and IMAGE_FEATURES?


On Mon, Dec 1, 2014 at 1:28 PM, ChenQi <Qi.Chen@windriver.com> wrote:

>  On 12/01/2014 12:45 PM, Qiang Yu wrote:
>
>
>> It's possible that you are using (or might use) different layer
>> configurations for different builds.
>> One build might have a bbappend file that another build doesn't need.
>>
> Yes, you are right. I use different conf/bblayers.conf for different SOC.
> But within the same SOC's different output
> (board image, SDK), I just change MACHINE and SDKMACHINE. Is it safe to
> build all outputs of the same SOC
> in one build dir?
>
>
> Yes.
>
>
>
>>  Also, different builds may have different DISTRO_FEATURES.
>> Thus, having different deploy directories is better. Otherwise, your
>> package feeds might be broken. And you would suffer trying to maintain it.
>>
> You mean I can't build two output with different DISTRO_FEATURES in the
> same build dir.
> Any other config var? What about MACHINE and SDKMACHINE?
>
>
>
> Yocto doesn't ensure that changing distro features in the same build
> directory works.
> It might work, but it's also possible that it doesn't.
>
> Changing MACHINE and SDKMACHINE is OK.
>
> //Chen Qi
>

[-- Attachment #2: Type: text/html, Size: 3287 bytes --]

Re: When to create a new build directory for yocto?

[ChenQi]

[-- Attachment #1: Type: text/plain, Size: 1946 bytes --]

On 12/01/2014 03:04 PM, Qiang Yu wrote:
> What about MACHINE_FEATURES and IMAGE_FEATURES?
>

IMAGE_FEATURES is OK.
MACHINE_FEATURES is like DISTRO_FEATURES. Things are not ensured to work 
correctly if you change it in the same build directory.

If I were you, I would jut let builds to share sstate and downloads and 
use different build directories for each combination of 'machine + 
distro + feature'.
E.g.
build-qemux86-64-poky-systemd-pam
build-qemux86-64-poky-systemd
build-qemux86-64-poky-sysvinit-pam
....

This would ensure a clean upgrade path.

//Chen Qi

>
> On Mon, Dec 1, 2014 at 1:28 PM, ChenQi <Qi.Chen@windriver.com 
> <mailto:Qi.Chen@windriver.com>> wrote:
>
>     On 12/01/2014 12:45 PM, Qiang Yu wrote:
>>
>>
>>         It's possible that you are using (or might use) different
>>         layer configurations for different builds.
>>         One build might have a bbappend file that another build
>>         doesn't need.
>>
>>     Yes, you are right. I use different conf/bblayers.conf for
>>     different SOC. But within the same SOC's different output
>>     (board image, SDK), I just change MACHINE and SDKMACHINE. Is it
>>     safe to build all outputs of the same SOC
>>     in one build dir?
>
>     Yes.
>
>>         Also, different builds may have different DISTRO_FEATURES.
>>         Thus, having different deploy directories is better.
>>         Otherwise, your package feeds might be broken. And you would
>>         suffer trying to maintain it.
>>
>>     You mean I can't build two output with different DISTRO_FEATURES
>>     in the same build dir.
>>     Any other config var? What about MACHINE and SDKMACHINE?
>
>     Yocto doesn't ensure that changing distro features in the same
>     build directory works.
>     It might work, but it's also possible that it doesn't.
>
>     Changing MACHINE and SDKMACHINE is OK.
>
>     //Chen Qi
>
>


[-- Attachment #2: Type: text/html, Size: 5550 bytes --]

Re: When to create a new build directory for yocto?

[Qiang Yu]

[-- Attachment #1: Type: text/plain, Size: 1855 bytes --]

Hi Chen Qi,

Thank you very much. Your advice is valuable.

Regards,
Qiang

On Mon, Dec 1, 2014 at 3:31 PM, ChenQi <Qi.Chen@windriver.com> wrote:

>  Hi Qiang,
>
> If you are just doing some development, you could use the same build
> directory for almost all builds with just a few exceptions.
> (After all, you could clean up the tmp dir and do a rebuild any time you
> like ...)
>
> But if you are maintaining products, you'd better separate build
> directories.
>
> Regards,
> Chen Qi
>
>
> On 12/01/2014 03:04 PM, Qiang Yu wrote:
>
> What about MACHINE_FEATURES and IMAGE_FEATURES?
>
>
> On Mon, Dec 1, 2014 at 1:28 PM, ChenQi <Qi.Chen@windriver.com> wrote:
>
>>  On 12/01/2014 12:45 PM, Qiang Yu wrote:
>>
>>
>>> It's possible that you are using (or might use) different layer
>>> configurations for different builds.
>>> One build might have a bbappend file that another build doesn't need.
>>>
>> Yes, you are right. I use different conf/bblayers.conf for different SOC.
>> But within the same SOC's different output
>> (board image, SDK), I just change MACHINE and SDKMACHINE. Is it safe to
>> build all outputs of the same SOC
>> in one build dir?
>>
>>
>>  Yes.
>>
>>
>>
>>>  Also, different builds may have different DISTRO_FEATURES.
>>> Thus, having different deploy directories is better. Otherwise, your
>>> package feeds might be broken. And you would suffer trying to maintain it.
>>>
>> You mean I can't build two output with different DISTRO_FEATURES in the
>> same build dir.
>> Any other config var? What about MACHINE and SDKMACHINE?
>>
>>
>>
>>  Yocto doesn't ensure that changing distro features in the same build
>> directory works.
>> It might work, but it's also possible that it doesn't.
>>
>> Changing MACHINE and SDKMACHINE is OK.
>>
>> //Chen Qi
>>
>
>
>

[-- Attachment #2: Type: text/html, Size: 5436 bytes --]