Usage of yocto on different (production vs debug) scenarios

Usage of yocto on different (production vs debug) scenarios

[Iván Castell]

[-- Attachment #1: Type: text/plain, Size: 1469 bytes --]

Hello forum.

We are trying to use yocto in a continuous integration environment with
different (production vs debug) scenarios.

To setup a given scenario (production vs debug) we are using something like
this:

    $ SCENARIO=debug
    $ MACHINE=<machine> DISTRO=<distro>-${SCENARIO} source
../../build-<machine>-${SCENARIO}
    $ bitbake <image>-${SCENARIO}

So we have different image recipes:

    * image-production.bb
    * image-debug.bb

Different distros:

    * distro-production.conf
    * distro-debug.conf

And different build directories:

    * build-<machine>-production
    * build-<machine>-debug

To optimize space usage and compilation time, we setup a shared sstate
cache and a shared directory for downloads. This seems a good starting
point.

However, things are getting complicated, because there is no way to exclude
some recipes easily. For example, we don't want iptables installed on the
debug image, but dependency chains include iptables by default even when
declaring IMAGE_INSTALL_remove explicitly. In this case we decided checking
SCENARIO inside iptables_%.bbappend to decide what rules are installed on
the image (rules.production vs rules.debug).

But at the end, this method is poisoning all our recipes with that kind of
ugly controls [...]

Do you think this is the right way to manage this? Can you suggest a
better/more convenient way to deal with this?

Thank you in advance! :)

[-- Attachment #2: Type: text/html, Size: 6887 bytes --]

Re: Usage of yocto on different (production vs debug) scenarios

[Uwe Geuder]

Hi!

On Fri, Apr 20, 2018 at 10:59 AM, Iván Castell <icastell@nayarsystems.com> wrote:

> We are trying to use yocto in a continuous integration environment with
> different (production vs debug) scenarios.
> 
> To setup a given scenario (production vs debug) we are using something like
> this:
> 
>     $ SCENARIO=debug
>     $ MACHINE=<machine> DISTRO=<distro>-${SCENARIO} source
> ../../build-<machine>-${SCENARIO}
>     $ bitbake <image>-${SCENARIO}
> 
> So we have different image recipes:
> 
>     * image-production.bb
>     * image-debug.bb
> 
> Different distros:
> 
>     * distro-production.conf
>     * distro-debug.conf
> 
> And different build directories:
> 
>     * build-<machine>-production
>     * build-<machine>-debug
> 
> To optimize space usage and compilation time, we setup a shared sstate
> cache and a shared directory for downloads. This seems a good starting
> point.

Shared download, yes.

But can you share state between distros? Isn't the purpose of distros to
use different options (variable settings) so the state would always be
different?

(Please note: This is really a follow-up question, not me knowing
better. I am just trying to fully understand these concepts)


> However, things are getting complicated, because there is no way to
> exclude some recipes easily. For example, we don't want iptables
> installed on the debug image, but dependency chains include iptables
> by default

Doesn't blacklist do what you want? 

E.g. in your distro-production.conf

PNBLACKLIST[iptables] = "we don't want iptables in product"


Of course if something has a hard dependency on iptables, the something
might need blacklisting instead or too.


Regards,

Uwe Geuder
Neuro Event Labs Oy
Tampere, Finland
uwe.gexder@neuroeventlabs.com (Bot check: fix one obvious typo)




> even when declaring IMAGE_INSTALL_remove explicitly. In this case we
> decided checking _%.bbappend to decide what rules are installed on ion
> vs rules.debug).
> 
> hod is poisoning all our recipes with that kind of
> 
> 
>  right way to manage this? Can you suggest a
> ay to deal with this?
> 
> 

Re: Usage of yocto on different (production vs debug) scenarios

[Alan Martinovic]

Hi,
had a similar question recently (search How did you separate release
and development builds?)

We have the split like this for now
The release build would be something like:

- has all the functions for production
- attack surface is limited (no ssh, only the required apps)
- config files set with real endpoints (logging or crash servers etc.)

But that's not an image you can really develop on given all the constrains.
So there is also a need for a development image:

- inherits the functions available on production image + debug tools
- developer continence over security (debuggers, ssh access etc.)
- configs set not to depend on payed cloud services

Both are built as a same distro, but are different images.

So far it works ok.
However still haven't experienced the need to have different
configurations for the same recipe, that would probably complicate things.


On Fri, Apr 20, 2018 at 12:47 PM, Uwe Geuder <jrswdnan22@snkmail.com> wrote:
> Hi!
>
> On Fri, Apr 20, 2018 at 10:59 AM, Iván Castell <icastell@nayarsystems.com> wrote:
>
>> We are trying to use yocto in a continuous integration environment with
>> different (production vs debug) scenarios.
>>
>> To setup a given scenario (production vs debug) we are using something like
>> this:
>>
>>     $ SCENARIO=debug
>>     $ MACHINE=<machine> DISTRO=<distro>-${SCENARIO} source
>> ../../build-<machine>-${SCENARIO}
>>     $ bitbake <image>-${SCENARIO}
>>
>> So we have different image recipes:
>>
>>     * image-production.bb
>>     * image-debug.bb
>>
>> Different distros:
>>
>>     * distro-production.conf
>>     * distro-debug.conf
>>
>> And different build directories:
>>
>>     * build-<machine>-production
>>     * build-<machine>-debug
>>
>> To optimize space usage and compilation time, we setup a shared sstate
>> cache and a shared directory for downloads. This seems a good starting
>> point.
>
> Shared download, yes.
>
> But can you share state between distros? Isn't the purpose of distros to
> use different options (variable settings) so the state would always be
> different?
>
> (Please note: This is really a follow-up question, not me knowing
> better. I am just trying to fully understand these concepts)
>
>
>> However, things are getting complicated, because there is no way to
>> exclude some recipes easily. For example, we don't want iptables
>> installed on the debug image, but dependency chains include iptables
>> by default
>
> Doesn't blacklist do what you want?
>
> E.g. in your distro-production.conf
>
> PNBLACKLIST[iptables] = "we don't want iptables in product"
>
>
> Of course if something has a hard dependency on iptables, the something
> might need blacklisting instead or too.
>
>
> Regards,
>
> Uwe Geuder
> Neuro Event Labs Oy
> Tampere, Finland
> uwe.gexder@neuroeventlabs.com (Bot check: fix one obvious typo)
>
>
>
>
>> even when declaring IMAGE_INSTALL_remove explicitly. In this case we
>> decided checking _%.bbappend to decide what rules are installed on ion
>> vs rules.debug).
>>
>> hod is poisoning all our recipes with that kind of
>>
>>
>>  right way to manage this? Can you suggest a
>> ay to deal with this?
>>
>>
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Re: Usage of yocto on different (production vs debug) scenarios

[Uwe Geuder]

Following up on my own message...

On Fri, Apr 20, 2018 at 1:47 PM, Uwe Geuder <jrswdnan22snkmail.com> wrote:

> On Fri, Apr 20, 2018 at 10:59 AM, Iván Castell <icastell@nayarsystems.com> wrote:
>
[...]
>> However, things are getting complicated, because there is no way to
>> exclude some recipes easily. For example, we don't want iptables
>> installed on the debug image, but dependency chains include iptables
>> by default
>
> Doesn't blacklist do what you want?
>
> E.g. in your distro-production.conf
>
> PNBLACKLIST[iptables] = "we don't want iptables in product"
>

As recently mentioned in another thread on this list there is also
BBMASK.

https://lists.yoctoproject.org/pipermail/yocto/2018-April/040861.html

I don't know what would be the suitable/most idiomatic use cases for
PNBLACKLIST vs. BBMASK

Regards,

Uwe Geuder
Neuro Event Labs Oy
Tampere, Finland
uwe.gexder@neuroeventlabs.com (Bot check: fix one obvious typo)

Re: Usage of yocto on different (production vs debug) scenarios

[Alex Kiernan]

On Fri, Apr 20, 2018 at 8:59 AM, Iván Castell <icastell@nayarsystems.com> wrote:
>
> Hello forum.
>
> We are trying to use yocto in a continuous integration environment with
> different (production vs debug) scenarios.
>
> To setup a given scenario (production vs debug) we are using something like
> this:
>
>     $ SCENARIO=debug
>     $ MACHINE=<machine> DISTRO=<distro>-${SCENARIO} source
> ../../build-<machine>-${SCENARIO}
>     $ bitbake <image>-${SCENARIO}
>
> So we have different image recipes:
>
>     * image-production.bb
>     * image-debug.bb
>
> Different distros:
>
>     * distro-production.conf
>     * distro-debug.conf
>
> And different build directories:
>
>     * build-<machine>-production
>     * build-<machine>-debug
>
> To optimize space usage and compilation time, we setup a shared sstate cache
> and a shared directory for downloads. This seems a good starting point.
>
> However, things are getting complicated, because there is no way to exclude
> some recipes easily. For example, we don't want iptables installed on the
> debug image, but dependency chains include iptables by default even when
> declaring IMAGE_INSTALL_remove explicitly. In this case we decided checking
> SCENARIO inside iptables_%.bbappend to decide what rules are installed on
> the image (rules.production vs rules.debug).
>
> But at the end, this method is poisoning all our recipes with that kind of
> ugly controls [...]
>
> Do you think this is the right way to manage this? Can you suggest a
> better/more convenient way to deal with this?
>
> Thank you in advance! :)

We're using the IMAGE_MODE stuff:

https://github.com/intel/intel-iot-refkit/blob/master/meta-refkit-core/classes/image-mode.bbclass

and then add things like this to the image:

CORE_IMAGE_EXTRA_INSTALL += "\
    ${@ '' if (d.getVar('IMAGE_MODE') or 'production') == 'production'
else 'packagegroup-core-full-cmdline' } \
"

-- 
Alex Kiernan

Re: Usage of yocto on different (production vs debug) scenarios

[Burton, Ross]

On 20 April 2018 at 11:47, Uwe Geuder <jrswdnan22@snkmail.com> wrote:
> But can you share state between distros? Isn't the purpose of distros to
> use different options (variable settings) so the state would always be
> different?

If the input to the recipe is different then the hashes would be
different so the content won't conflict.  You can definitely share
sstate between DISTROs.

Ross

Re: Usage of yocto on different (production vs debug) scenarios

[Iván Castell]

[-- Attachment #1: Type: text/plain, Size: 3676 bytes --]

Thanks a lot for all your replies. I am working on a solution trying to get
the best option of all your answers. I will come back after deciding my
solution to share it with the community.


Related with that shared state cache, I found some information on a e-book
(search the text on google to find the source):

"Sharing a shared state cache is possible; however, it needs to be
approached with care. Not all changes are detected by the shared state
cache implementation, and when this happens, some or all of the cache needs
to be invalidated. This can cause problems when the state cache is being
shared. The recommendation in this case depends on the use case. Developers
working on Yocto metadata should keep the shared state cache as default,
separated per project. However, validation and testing engineers, kernel
and bootloader developers, and application developers would probably
benefit from a well-maintained shared state cache."


After reading that I'm not sure to use it because I don't understand what
kind of "problems" can be found, and if they can be easily detected or not.


Does some of you have any experience using a shared sstate cache?





2018-04-23 17:10 GMT+02:00 Burton, Ross <ross.burton@intel.com>:

> On 20 April 2018 at 11:47, Uwe Geuder <jrswdnan22@snkmail.com> wrote:
> > But can you share state between distros? Isn't the purpose of distros to
> > use different options (variable settings) so the state would always be
> > different?
>
> If the input to the recipe is different then the hashes would be
> different so the content won't conflict.  You can definitely share
> sstate between DISTROs.
>
> Ross
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>



-- 




*NOTA LEGAL*
Este correo electrónico y, en su caso, cualquier fichero anexo al mismo,
contiene información de carácter confidencial exclusivamente dirigida a su
destinatario y se encuentra protegido por Ley. Cualquier persona distinta
de su destinataria tiene prohibida su reproducción, uso, divulgación, copia
o impresión total o parcial. Si ha recibido este correo electrónico por
error, se ruega lo notifique de inmediato al remitente borrando el mensaje
original juntamente con sus ficheros anexos. Gracias.

De conformidad con lo establecido en la LOPD, NAYAR SYSTEMS SL garantiza la
adopción de las medidas necesarias para asegurar el tratamiento
confidencial de los datos de carácter personal. Así mismo le informamos de
la inclusión de sus datos en un fichero bajo la responsabilidad de NAYAR
SYSTEMS SL, con la finalidad de poder atender los compromisos derivados de
la relación que mantenemos con usted. Si lo desea, puede ejercer sus
derechos de acceso, rectificación, cancelación y oposición mediante un
escrito a la siguiente dirección: info@nayarsystems.com

*LEGAL NOTE*
This email and any attachments to it contains is confidential information
exclusively intended for the recipients. Any divulgation, copy or
distribution to third parties is prohibited without written permission of
NAYAR SYSTEMS SL. If you have received this e-mail in error, please notify
the sender immediately. In accordance with Law 15/1999 of 13 December on
the Protection of Personal Data, the NAYAR SYSTEMS SL guarantees that it
has adopted the necessary measures to ensure the confidential treatment of
personal information. We also inform you that you can exercise your access,
rectification, cancellation and opposition rights by send us a mail to:
info@nayarsystems.com

[-- Attachment #2: Type: text/html, Size: 8561 bytes --]

Re: Usage of yocto on different (production vs debug) scenarios

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 4179 bytes --]

Very curious as to what book said that, because *any* example of that
happening is a bug in the recipe itself.  I wouldn't listen to it: the YP
autobuilder has a shared sstate for three distributions * four
architectures * two libc implementations and doesn't have problems.

Ross

On 23 April 2018 at 16:23, Iván Castell <icastell@nayarsystems.com> wrote:

> Thanks a lot for all your replies. I am working on a solution trying to
> get the best option of all your answers. I will come back after deciding my
> solution to share it with the community.
>
>
> Related with that shared state cache, I found some information on a e-book
> (search the text on google to find the source):
>
> "Sharing a shared state cache is possible; however, it needs to be
> approached with care. Not all changes are detected by the shared state
> cache implementation, and when this happens, some or all of the cache needs
> to be invalidated. This can cause problems when the state cache is being
> shared. The recommendation in this case depends on the use case. Developers
> working on Yocto metadata should keep the shared state cache as default,
> separated per project. However, validation and testing engineers, kernel
> and bootloader developers, and application developers would probably
> benefit from a well-maintained shared state cache."
>
>
> After reading that I'm not sure to use it because I don't understand what
> kind of "problems" can be found, and if they can be easily detected or not.
>
>
> Does some of you have any experience using a shared sstate cache?
>
>
>
>
>
> 2018-04-23 17:10 GMT+02:00 Burton, Ross <ross.burton@intel.com>:
>
>> On 20 April 2018 at 11:47, Uwe Geuder <jrswdnan22@snkmail.com> wrote:
>> > But can you share state between distros? Isn't the purpose of distros to
>> > use different options (variable settings) so the state would always be
>> > different?
>>
>> If the input to the recipe is different then the hashes would be
>> different so the content won't conflict.  You can definitely share
>> sstate between DISTROs.
>>
>> Ross
>> --
>> _______________________________________________
>> yocto mailing list
>> yocto@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/yocto
>>
>
>
>
> --
>
>
>
>
> *NOTA LEGAL*
> Este correo electrónico y, en su caso, cualquier fichero anexo al mismo,
> contiene información de carácter confidencial exclusivamente dirigida a su
> destinatario y se encuentra protegido por Ley. Cualquier persona distinta
> de su destinataria tiene prohibida su reproducción, uso, divulgación, copia
> o impresión total o parcial. Si ha recibido este correo electrónico por
> error, se ruega lo notifique de inmediato al remitente borrando el mensaje
> original juntamente con sus ficheros anexos. Gracias.
>
> De conformidad con lo establecido en la LOPD, NAYAR SYSTEMS SL garantiza
> la adopción de las medidas necesarias para asegurar el tratamiento
> confidencial de los datos de carácter personal. Así mismo le informamos de
> la inclusión de sus datos en un fichero bajo la responsabilidad de NAYAR
> SYSTEMS SL, con la finalidad de poder atender los compromisos derivados de
> la relación que mantenemos con usted. Si lo desea, puede ejercer sus
> derechos de acceso, rectificación, cancelación y oposición mediante un
> escrito a la siguiente dirección: info@nayarsystems.com
>
> *LEGAL NOTE*
> This email and any attachments to it contains is confidential information
> exclusively intended for the recipients. Any divulgation, copy or
> distribution to third parties is prohibited without written permission of
> NAYAR SYSTEMS SL. If you have received this e-mail in error, please notify
> the sender immediately. In accordance with Law 15/1999 of 13 December on
> the Protection of Personal Data, the NAYAR SYSTEMS SL guarantees that it
> has adopted the necessary measures to ensure the confidential treatment of
> personal information. We also inform you that you can exercise your access,
> rectification, cancellation and opposition rights by send us a mail to:
> info@nayarsystems.com
>
>

[-- Attachment #2: Type: text/html, Size: 9536 bytes --]

Re: Usage of yocto on different (production vs debug) scenarios

[Uwe Geuder]

Thanks Ross for your answers.  I'm still working on fully understanding
shared state so I appreciate your help.  Let me follow up on both your
answers in a single message.

On Mon, Apr 23, 2018 at 6:38 PM, Burton, Ross <ross.burton@intel.com> wrote:
> On 23 April 2018 at 16:23, Iván Castell <icastell@nayarsystems.com> wrote:
>>
[...]
>>
>> Related with that shared state cache, I found some information on a e-book
>> (search the text on google to find the source):
>>
>> "Sharing a shared state cache is possible; however, it needs to be
>> approached with care. Not all changes are detected by the shared state cache
>> implementation, and when this happens, some or all of the cache needs to be
>> invalidated. This can cause problems when the state cache is being shared.
[...]
> Very curious as to what book said that, because *any* example of that
> happening is a bug in the recipe itself.  I wouldn't listen to it: the YP
> autobuilder has a shared sstate for three distributions * four architectures
> * two libc implementations and doesn't have problems.
>
> Ross
>

I think when talking about shared state there can be 2 aspects of sharing

1.) sharing over time: Several invocations of bitbake in the same environment
  (DISTRO, MACHINE etc)
2.) sharing over distros etc.

Number 1 is an incremental build (as opposed to clean or full
build).  Without any doubt that is useful (or really mandatory) for
development work.  You build, you make little changes, you build, you
test, you make more changes...  You share state from the previous build,
because making a full clean build every time would lead to completely
inacceptable cycle times.

But when you are "done" with your changes you make an integration build
and to my experience that should be a clean (aka full) build.  You
intentionally do not share any state from a previous build, you don't
make an incremental build, because there is always the risk of broken
recipes.  I mean if there were no broken software, we would all be
unemployed by now...

Let me give a simple example I experienced my self recently:

I added creation of a new file in some task.  In my incremental
developer build it all worked fine.  When I put into a clean integration
build the task failed.  It turned out that the directory I wrote my new
file to in task A was created by a different task B.  In my development
build task B had already been executed long before I even started to
make the recipe modificationto to create a new file, so task A always
succeeded.  In the clean build task A happened to run first and failed.
A simple missing dependency / incorrect task ordering.  With parallel
clean builds you might not always find it, but with incremental builds
it can go undetected forever.

It's not only "some book" that mentions the risk of "bad" state, but
also the Yocto documentaton:

https://www.yoctoproject.org/docs/2.5/mega-manual/mega-manual.html#concepts-tips-and-tricks

(Especially 4.5.4.2. Invalidating Shared State.  Of course the
invalidating is easy, but finding what you need to invalidate is hard.)

Of course the less something is tested the bigger the risk is.  So the
bitbake system should be rather fine, commonly used Poky recipes
probably too.  But when it comes to random layers and our own local
recipes and bbappends I would expect the risk to grow a lot.

So do you really tell us in the message above that above that Yocto
project runs incremental integration builds?

Another way to think about it: Shared state is used to to avoid
rebuilds.  So how could I benefit from having state information in some
SSTATE_DIR already?  I have to build it anyway, because my build area is
empty to begin with.

The 2nd aspect, sharing over distros was covered in the other message.

On Mon, Apr 23, 2018 at 6:10 PM, Burton, Ross <ross.burton@intel.com> wrote:
> On 20 April 2018 at 11:47, Uwe Geuder <jrswdnan22@snkmail.com> wrote:
>> But can you share state between distros? Isn't the purpose of distros to
>> use different options (variable settings) so the state would always be
>> different?
>
> If the input to the recipe is different then the hashes would be
> different so the content won't conflict.  You can definitely share
> sstate between DISTROs.
>
> Ross

Under the assumption that all recepies were perfect (which they probably
aren't in every point of history, see above) I fully believe that the
build system would work perfectly when I share the SSTATE_DIR over
DISTRO borders (or diferent MACHINE settings).  But do I really have any
commmon data that would lead to any saving? As you say the hashes would
be different.  In my thinking all "expensive" state info should be
different, because a different DISTRO (or different MACHINE) would lead
to different hashes for bascially every item.  So in the best case I end
up with several distinct sets of state data stored under the same top
directory, but that doesn't seem to result in any saving.  In the worst
case an item goes into the same file, so I end up with "cache
thrashing".  Build A calculates the state and stores it, build B finds
the hashes not matching and puts its new info there.  Build A finds the
hashes not matching and...

Or does the saving come on the native part of the build? There I would
guess that common state could occur more commonly.

I think it would really help to have a hello world type of Yocto build,
where you could manually read through the sstate data to understand how
it works exactly.  In a real world sized build the amount of data does
not really help to understand the concepts.  Does anybody have a
recommendation what would be the smallest possible Yocto build that
makes some sense?

Regards,

Uwe Geuder
Neuro Event Labs Oy
Tampere, Finland
uwe.gexder@neuroeventlabs.com (Bot check: fix one obvious typo)

Re: Usage of yocto on different (production vs debug) scenarios

[Iván Castell]

[-- Attachment #1: Type: text/plain, Size: 535 bytes --]

That sounds really convincing, I will give it a chance and come back to
describe the hole process after all. Thank you for that valuable
information Mr. Ross

2018-04-23 17:38 GMT+02:00 Burton, Ross <ross.burton@intel.com>:

> Very curious as to what book said that, because *any* example of that
> happening is a bug in the recipe itself.  I wouldn't listen to it: the YP
> autobuilder has a shared sstate for three distributions * four
> architectures * two libc implementations and doesn't have problems.
>
> Ross
>
>

[-- Attachment #2: Type: text/html, Size: 1410 bytes --]

Re: Usage of yocto on different (production vs debug) scenarios

[Iván Castell]

[-- Attachment #1: Type: text/plain, Size: 902 bytes --]

Hello again forum.

I plan using a PR service on different build scenarios (i.e. development,
production).

I think the right way to proceed is running a different PR server instance
listening on a different port for each scenary. So, a PR server instance
listening on port X for development and another PR server instance
listening on port Y for production.

However, a single shared PR server instance listening on a single unique
port Z could also be used.

Do you think it has sense to have a different PR server for each scenary?
Or a single shared PR sever should work fine for all different scenarios?

Thank you in advance! :-)


2018-04-24 8:45 GMT+02:00 Iván Castell <icastell@nayarsystems.com>:

> That sounds really convincing, I will give it a chance and come back to
> describe the hole process after all. Thank you for that valuable
> information Mr. Ross
>
>

[-- Attachment #2: Type: text/html, Size: 3480 bytes --]