Updating Yocto

Updating Yocto

[Claude Bing]

Hello all. My organization has been working with Yocto recently, and we
have noticed that there are often weird errors encountered after
updating revisions within a release branch (e.g., 3.1.3 -> 3.1.5). Is
there an accepted process to merging in upstream changes?  Here is a
distillation of our workflow:

* Yocto releases 3.1.0 / dunfell

* Create a local tracking branch for the Yocto release
  "sample-yocto-dunfell" based on upstream 3.1.0 tag / dunfell branch.
  This is necessary because some of our local tooling is located in the
  top level directory and it is a pain to copy the files each time.

* Create firmware with Yocto

* Yocto releases 3.1.x

* Merge upstream yocto-3.1.x tag into local tracking branch

* Create firmware with Yocto

Sometimes, after this last step, we encounter problems where patches
cannot be applied, or files cannot be found when bitbake tries to build
the recipes. We have tried deleting build/{tmp,sstate-cache} whenever we
merge upstream changes, but random errors still persist.  For each of
these packages, if we run "bitbake -c cleanall", the error goes away.

Most of the time, these recipes have not been extended in our project,
so they are purely meta-oe / meta packages.

Are we doing something wrong, or is there a more acceptable way to
handle updates?

How tightly should meta-openembedded be tied to the core Yocto release?

-- 
Regards,

Claude Bing

Re: [yocto] Updating Yocto

[Mikko Rapeli]

Hi,

On Mon, Feb 22, 2021 at 12:12:21PM -0500, Claude Bing wrote:
> Hello all. My organization has been working with Yocto recently, and we
> have noticed that there are often weird errors encountered after
> updating revisions within a release branch (e.g., 3.1.3 -> 3.1.5). Is
> there an accepted process to merging in upstream changes?  Here is a
> distillation of our workflow:
> 
> * Yocto releases 3.1.0 / dunfell
> 
> * Create a local tracking branch for the Yocto release
>   "sample-yocto-dunfell" based on upstream 3.1.0 tag / dunfell branch.
>   This is necessary because some of our local tooling is located in the
>   top level directory and it is a pain to copy the files each time.
> 
> * Create firmware with Yocto
> 
> * Yocto releases 3.1.x
> 
> * Merge upstream yocto-3.1.x tag into local tracking branch
> 
> * Create firmware with Yocto
> 
> Sometimes, after this last step, we encounter problems where patches
> cannot be applied, or files cannot be found when bitbake tries to build
> the recipes. We have tried deleting build/{tmp,sstate-cache} whenever we
> merge upstream changes, but random errors still persist.  For each of
> these packages, if we run "bitbake -c cleanall", the error goes away.

When both yocto update and your own changes are modifying the same recipes,
then conflicts can occur.

> Most of the time, these recipes have not been extended in our project,
> so they are purely meta-oe / meta packages.

If builds are failing, then your environment is breaking something.
It might even be the BSP layers that you use.

> Are we doing something wrong, or is there a more acceptable way to
> handle updates?
> 
> How tightly should meta-openembedded be tied to the core Yocto release?

I update all open source meta layers for every run of the update and also resolve
any issues by digging into details why the failure happened. I've done dunfell
updates with several BSP layers for multiple arm64 SoC's and have not seen
major issues.

What is useful and one of the best practices, is to clear the bitbake tmp
directory in between builds. Depending on details, there can be some cruft there
which fails when rebuilding.

Cheers,

-Mikko

Re: [yocto] Updating Yocto

[Claude Bing]

On 2/23/21 7:58 AM, Mikko Rapeli wrote:
> Hi,
> 
> On Mon, Feb 22, 2021 at 12:12:21PM -0500, Claude Bing wrote:
>> Hello all. My organization has been working with Yocto recently, and we
>> have noticed that there are often weird errors encountered after
>> updating revisions within a release branch (e.g., 3.1.3 -> 3.1.5). Is
>> there an accepted process to merging in upstream changes?  Here is a
>> distillation of our workflow:
>>
>> * Yocto releases 3.1.0 / dunfell
>>
>> * Create a local tracking branch for the Yocto release
>>   "sample-yocto-dunfell" based on upstream 3.1.0 tag / dunfell branch.
>>   This is necessary because some of our local tooling is located in the
>>   top level directory and it is a pain to copy the files each time.
>>
>> * Create firmware with Yocto
>>
>> * Yocto releases 3.1.x
>>
>> * Merge upstream yocto-3.1.x tag into local tracking branch
>>
>> * Create firmware with Yocto
>>
>> Sometimes, after this last step, we encounter problems where patches
>> cannot be applied, or files cannot be found when bitbake tries to build
>> the recipes. We have tried deleting build/{tmp,sstate-cache} whenever we
>> merge upstream changes, but random errors still persist.  For each of
>> these packages, if we run "bitbake -c cleanall", the error goes away.
> 
> When both yocto update and your own changes are modifying the same recipes,
> then conflicts can occur.>
>> Most of the time, these recipes have not been extended in our project,
>> so they are purely meta-oe / meta packages.
> 
> If builds are failing, then your environment is breaking something.
> It might even be the BSP layers that you use>
>> Are we doing something wrong, or is there a more acceptable way to
>> handle updates?
>>
>> How tightly should meta-openembedded be tied to the core Yocto release?
> 
> I update all open source meta layers for every run of the update and also resolve
> any issues by digging into details why the failure happened. I've done dunfell
> updates with several BSP layers for multiple arm64 SoC's and have not seen
> major issues.
> 
> What is useful and one of the best practices, is to clear the bitbake tmp
> directory in between builds. Depending on details, there can be some cruft there
> which fails when rebuilding.

I mentioned that we remove that directory, and it does indeed solve a
lot of problems.  Not sure that it was clear from the original message,
but whenever we get a random error on the first build after the upgrade
we run a "bitbake -c cleanall" on that package, and everything works
great. The errors do not show up again.

Thanks for the reply.

> 
> Cheers,
> 
> -Mikko
> 
> 
> 
> 
> 

Re: [yocto] Updating Yocto

[Mikko Rapeli]

On Tue, Feb 23, 2021 at 09:24:47AM -0500, Claude Bing wrote:
> I mentioned that we remove that directory, and it does indeed solve a
> lot of problems.  Not sure that it was clear from the original message,
> but whenever we get a random error on the first build after the upgrade
> we run a "bitbake -c cleanall" on that package, and everything works
> great. The errors do not show up again.

Yes, we're talking about the same thing :)

A lot of yocto things don't work well if major changes happen and WORKDIR
isn't completely cleaned in between. With various BSP layers things
can be even worse and they may actively break both work directory and
sstate caches.

For any major changes in open source or other meta layers, I wipe build/tmp
completely to avoid random problems. In our CI this is the default.
Only download cache and sstate mirror are re-used between builds.
After this, I don't see many problems and build failures which are not
real bugs in either our changes or in upstream, e.g. race conditions
in build scripts.

Cheers,

-Mikko