Yocto Project Status 12 September 2023 (WW37)

Yocto Project Status 12 September 2023 (WW37)

[Stephen K Jolley]

[-- Attachment #1: Type: text/plain, Size: 5712 bytes --]

Current Dev Position: YP 4.3 M4 (Feature Freeze)

Next Deadline: 2nd October 2023 YP 4.3 M4 build date

Next Team Meetings:

   -

   Bug Triage meeting Thursday September 14th 7:30 am PDT (
   https://zoom.us/j/454367603?pwd=ZGxoa2ZXL3FkM3Y0bFd5aVpHVVZ6dz09)
   -

   Weekly Project Engineering Sync Tuesday September 12th at 8 am PDT (
   https://zoom.us/j/990892712?pwd=cHU1MjhoM2x6ck81bkcrYjRrcmJsUT09)
   <https://zoom.us/j/990892712>
   -

   Twitch -  See https://www.twitch.tv/theyoctojester


Key Status/Updates:

   -

   We are now at feature freeze for 4.3 and 4.3 M3 is in QA
   -

   The numpy reproducibility issue is not solved unfortunately
   -

   There are continuing intermittent ptest issues for openssh and
   glib-networking amongst other issues
   -

   The testing for qemuppc and qemumips/qemumips64 has been reduced and
   we’re no longer testing core-image-sato-sdk on these platforms.
   -

   SPDX manifests are now enabled by default for OE-Core/nodistro as well
   as poky.
   -

   Patches have been resubmitted to radically alter the do_unpack process
   for license compliance reasons. The code is complex and hard to understand
   and will have a performance impact on builds as well as making things hard
   to debug. The risk of not taking the changes is that for some legal
   departments, the SPDX data isn’t detailed enough. The compromise between
   performance and ease of use vs. legal requirements is a tough one. We don’t
   really want to have two codepaths either. Feedback/review on the series
   welcome.
   -

   We’re happy to be able to announce that some of the work in the RFQ will
   now be progressing, specifically that:
   -

      Marta Rybczynska will be working on the security topic
      -

      Alexander Kanvin will be working on the core workflow topic
      -

      Savoir-faire Linux will be working on the toaster and VSCode topics
      -

      BayLibre will be working on the patchtest and project tooling topics
      -

      Michael Opdenacker (Bootlin) will be working on the binary distro
      topic
      -

      Smile will be working on the meta-openembedded topic area

We’d also note that Tim Orling (Konsulko) will be working on the layer
index.

Ways to contribute:

   -

   As people are likely aware, the project has a number of components which
   are either unmaintained, or have people with little to no time trying to
   keep them alive. These components include: patchtest, layerindex, devtool,
   toaster, wic, oeqa, autobuilder, CROPs containers, pseudo and more. Many
   have open bugs. Help is welcome in trying to better look after these
   components!
   -

   There are bugs identified as possible for newcomers to the project:
   https://wiki.yoctoproject.org/wiki/Newcomers
   -

   There are bugs that are currently unassigned for YP 4.3. See:
   https://wiki.yoctoproject.org/wiki/Bug_Triage#Medium.2B_4.3_Unassigned_Enhancements.2FBugs
   -

   We’d welcome new maintainers for recipes in OE-Core. Please see the list
   at:
   http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/conf/distro/include/maintainers.inc
   and discuss with the existing maintainer, or ask on the OE-Core mailing
   list. We will likely move a chunk of these to “Unassigned” soon to help
   facilitate this.
   -

   Help is very much welcome in trying to resolve our autobuilder
   intermittent issues. You can see the list of failures we’re continuing to
   see by searching for the “AB-INT” tag in bugzilla:
   https://bugzilla.yoctoproject.org/buglist.cgi?quicksearch=AB-INT.
   -

   Help us resolve CVE issues: CVE metrics
   <https://autobuilder.yocto.io/pub/non-release/patchmetrics/>
   -

   We have a growing number of bugs in bugzilla, any help with them is
   appreciated.


YP 4.3 Milestone Dates:

   -

   YP 4.3 M3 is in QA
   -

   YP 4.3 M4 build date  2023/10/02
   -

   YP 4.3 M4 Release date 2023/10/27


Upcoming dot releases:

   -

   YP 3.1.28 build date 2023/09/18
   -

   YP 3.1.28 Release date 2023/09/29
   -

   YP 4.0.13 build date 2023/09/25
   -

   YP 4.0.13 Release date 2023/10/06
   -

   YP 3.1.29 build date 2023/10/30
   -

   YP 3.1.29 Release date 2023/11/10
   -

   YP 4.0.14 build date 2023/11/06
   -

   YP 4.0.14 Release date 2023/11/17
   -

   YP 4.2.4 build date 2023/11/13
   -

   YP 4.2.4 Release date 2023/11/24
   -

   YP 3.1.30 build date 2023/12/11
   -

   YP 3.1.30 Release date 2023/12/22
   -

   YP 4.0.15 build date 2023/12/18
   -

   YP 4.0.15 Release date 2023/12/29


Tracking Metrics:

   -

   WDD 2530 (last week 2500) (
   https://wiki.yoctoproject.org/charts/combo.html)
   -

   OE-Core/Poky Patch Metrics
   -

      Total patches found: 1186 (last week 1185)
      -

      Patches in the Pending State: 255 (22%) [last week 254 (21%)]
      -

   https://autobuilder.yocto.io/pub/non-release/patchmetrics/


The Yocto Project’s technical governance is through its Technical Steering
Committee, more information is available at:

https://wiki.yoctoproject.org/wiki/TSC

The Status reports are now stored on the wiki at:
https://wiki.yoctoproject.org/wiki/Weekly_Status

[If anyone has suggestions for other information you’d like to see on this
weekly status update, let us know!]

Thanks,



*Stephen K. Jolley*

*Yocto Project Program Manager*

(    *Cell*:                (208) 244-4460

* *Email*:                 *s
<stephen.k.jolley@intel.com>jolley.yp.pm@gmail.com <jolley.yp.pm@gmail.com>*

[-- Attachment #2: Type: text/html, Size: 41115 bytes --]

Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Tue, 12 Sept 2023 at 16:44, Stephen Jolley <sjolley.yp.pm@gmail.com> wrote:
> Alexander Kanavin will be working on the core workflow topic

I am now ready to start doing this, but before I do, I'd like to
decompose the subject into manageable tasks with a bit of help from RP
and the community:

https://www.yoctoproject.org/community/yocto-project-engineering-request-for-quotation/

====
Core Workflow – Process Improvements

Background

The project builds everything from source by default. This means it
has a reputation for being slow and heavy. There are ways the project
can accelerate this which means faster workflows and improved
developer experience but these are currently cumbersome to use.

Rationale

The project aims to defragment customised embedded Linux. This is
important as if we succeed at this, it gives benefits to the wider
ecosystem through making it easier to inject security fixes and an
ability to share and collaborate without re-inventing the wheel.

To do this, we need to provide best-in-class support and compete
against binary distributions for usability and speed. One way we can
do this is provide better support for binary artifacts via our sstate
mechanism. We do already have some of this functionality in our
“extensible SDK” or “eSDK”.

Deliverables

Enable a public sstate mirror via a content delivery network (CDN) and
populate using the autobuilder
Ensure CDN sstate is reused under normal use case scenarios,
particularly for slow components like rust-native. Identify any common
sstate mismatch causes. Ensure test cases are added to cover the use
cases and prevent regressions.
Add lock and unlock commands to allow specific components to be locked
down to specific sstate checksums or allow them to vary
Allow switching between eSDK and non-eSDK modes
Add tooling so we can understand why something is rebuilding when it
isn’t expected to.
=======

So: where to start? Do we need to 'design' something, or maybe time
should go directly into addressing specific sore points? All feedback
welcome.

There's also an unfinished patchset for adding bblock/bbunlock, which
I would prefer to *not* 'take over and finish' but rather see the
author get it merged:
https://patchwork.yoctoproject.org/project/oe-core/list/?series=15276

Thanks,
Alex

Re: Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Thu, 2023-09-14 at 13:52 +0200, Alexander Kanavin wrote:
> On Tue, 12 Sept 2023 at 16:44, Stephen Jolley <sjolley.yp.pm@gmail.com> wrote:
> > Alexander Kanavin will be working on the core workflow topic
> 
> I am now ready to start doing this, but before I do, I'd like to
> decompose the subject into manageable tasks with a bit of help from RP
> and the community:
> 
> https://www.yoctoproject.org/community/yocto-project-engineering-request-for-quotation/
> 
> ====
> Core Workflow – Process Improvements
> 
> Background
> 
> The project builds everything from source by default. This means it
> has a reputation for being slow and heavy. There are ways the project
> can accelerate this which means faster workflows and improved
> developer experience but these are currently cumbersome to use.
> 
> Rationale
> 
> The project aims to defragment customised embedded Linux. This is
> important as if we succeed at this, it gives benefits to the wider
> ecosystem through making it easier to inject security fixes and an
> ability to share and collaborate without re-inventing the wheel.
> 
> To do this, we need to provide best-in-class support and compete
> against binary distributions for usability and speed. One way we can
> do this is provide better support for binary artifacts via our sstate
> mechanism. We do already have some of this functionality in our
> “extensible SDK” or “eSDK”.
> 
> Deliverables
> 
> Enable a public sstate mirror via a content delivery network (CDN) and
> populate using the autobuilder
> Ensure CDN sstate is reused under normal use case scenarios,
> particularly for slow components like rust-native. Identify any common
> sstate mismatch causes. Ensure test cases are added to cover the use
> cases and prevent regressions.
> Add lock and unlock commands to allow specific components to be locked
> down to specific sstate checksums or allow them to vary
> Allow switching between eSDK and non-eSDK modes
> Add tooling so we can understand why something is rebuilding when it
> isn’t expected to.
> =======
> 
> So: where to start? Do we need to 'design' something, or maybe time
> should go directly into addressing specific sore points? All feedback
> welcome.
> 
> There's also an unfinished patchset for adding bblock/bbunlock, which
> I would prefer to *not* 'take over and finish' but rather see the
> author get it merged:
> https://patchwork.yoctoproject.org/project/oe-core/list/?series=15276

To start with I'll try and write down and give a random walk through my
thoughts.

Certainly we need to get something like that patchset over the line. I
think it was blocked on a reply to this email:

https://lists.openembedded.org/g/openembedded-core/message/186497

which has suffered whilst I was pulled into the qemuppc mess. I feel
bad for not getting to a reply to that.

There are design elements to this work. We need to work out how we can
make eSDK and "normal" builds more similar and less of an overhead to
switch between one and the other. A "bblock all" command does partly
get you to an eSDK effectively so part of this may be switching eSDK to
use the new lock command. What other differences are there? What other
differences are necessary or make sense for the use cases eSDK was
designed for? How would you turn an existing build into an eSDK like
one? Could you provide a copy of a local build to someone else easily
using something like eSDK's tooling? What does the eSDK look like at
the end of this. One section we don't have good answers to yet is setup
and configuration although I know you've started on some of that.

For the task signatures, we need to think about some questions. If I
make a change locally, can I query how much will rebuild and how much
will be reused? There is bitbake --dry-run but perhaps it is time for a
an option (or dedicated separate command?) to give some statistics
about what bitbake would do? How much sstate would be reused?

That then logically leads into the questions, can we tell what has
changed? Why isn't my sstate being reused? For that we perhaps should
define some existing scenarios where it is currently very difficult to
work this out and then work out how we can report that information to
the user. These could become test cases?

One of the big problems in the past was that we lost much of the hash
information after parsing completed. This meant that if the hashes then
didn't match, we couldn't tell why as the original computation was
lost. I did some work on allowing us to retain more of the information
so that we didn't have to recompute it every time to be able to do
processing with it. I have to admit I've totally lost track of where I
got to with that.

Michael Halstead will be working on setting the CDN up so I'll let him
comment on when we'll have things ready for testing with that. We do
already have sstate shared from the autobuilder so some basic tests to
make sure our "base" shared files do work as expected is something
which can happen there already.

Another interesting question - would the project be interested in
shipping "locked" hashes for a minimal subset of recipes such as rust-
native and maybe the cross compilers to bootstrap and speed people's
builds, assuming they have a fast network? What does the size of those
pieces look like and would it be useful/effective?

Probably more questions than answers here but it hopefully gives a bit
more insight into some of the directions I'm thinking about.

Cheers,

Richard

Re: Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 14 Sept 2023 at 14:56, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> For the task signatures, we need to think about some questions. If I
> make a change locally, can I query how much will rebuild and how much
> will be reused? There is bitbake --dry-run but perhaps it is time for a
> an option (or dedicated separate command?) to give some statistics
> about what bitbake would do? How much sstate would be reused?
>
> That then logically leads into the questions, can we tell what has
> changed? Why isn't my sstate being reused? For that we perhaps should
> define some existing scenarios where it is currently very difficult to
> work this out and then work out how we can report that information to
> the user. These could become test cases?

So I think there are two questions here that the tools should answer:

1. If I would run a build, what would be missing in the cache and need
to be built? The missing cache objects are in a dependency hierarchy,
so only those missing objects with no dependecies on other missing
objects would be printed. That should be comparatively easy to add as
bitbake already does those checks all the time. Is there something
else that's easily done and useful to print?

2. Then there's the question of *why* they are missing, which is
harder to answer. If, say, curl:do_package is not in the cache, then
the tool would have to walk the cache tree (I/O heavy operation as
there is no index), make a list of all curl:do_package objects that
are there, and do a recursive bitbake-diffsig (going up the task tree)
on them vs the one we want. Then print them starting with the newest.
Something like:

Existing cache objects are not suitable because:
<object id 1> was built on <date> and has a mismatching SRCREV
<object id 2> was built on <earlier date> and has a different do_compile()

> One of the big problems in the past was that we lost much of the hash
> information after parsing completed. This meant that if the hashes then
> didn't match, we couldn't tell why as the original computation was
> lost. I did some work on allowing us to retain more of the information
> so that we didn't have to recompute it every time to be able to do
> processing with it. I have to admit I've totally lost track of where I
> got to with that.

Here's an idea I can't get out of my head. Right now, the cache is
simply an amorphous mass of objects, with no information regarding how
they were created. How about storing complete build confgurations as
well into the same directory? There would be a dedicated, separate
area for each configuration that placed objects into the cache,
containing:
- list of layers and revisions
- config template used
- complete content of build/conf
- bitbake invocation (e.g. targets and prefixed variables like MACHINE etc.)
- complete list of sstate objects that were produced as a result, so
they can be checked for existence

This would be written into the cache dir at the very end of the build
when everything else is already there.

Right now, everyone sets up their own builds first, then points
local.conf or site.conf to the cache, and hopes for the best regarding
hit rates. Having stored build configs would allow inverting the
workflow, so that you first ask from the cache what it can provide
(e.g. it can provide mickledore or kirkstone core-image-minimal for
qemux86, and that's exactly what you want as a starting point), then
you use the build config stored in the cache to set up a build, and
run it - and that would guarantee complete sstate reuse and getting to
a functional image as soon as possible. Kind of like binary distro,
but implemented with sstate.

Alex

Re: Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Thu, 2023-09-14 at 20:51 +0200, Alexander Kanavin wrote:
> On Thu, 14 Sept 2023 at 14:56, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > For the task signatures, we need to think about some questions. If I
> > make a change locally, can I query how much will rebuild and how much
> > will be reused? There is bitbake --dry-run but perhaps it is time for a
> > an option (or dedicated separate command?) to give some statistics
> > about what bitbake would do? How much sstate would be reused?
> > 
> > That then logically leads into the questions, can we tell what has
> > changed? Why isn't my sstate being reused? For that we perhaps should
> > define some existing scenarios where it is currently very difficult to
> > work this out and then work out how we can report that information to
> > the user. These could become test cases?
> 
> So I think there are two questions here that the tools should answer:
> 
> 1. If I would run a build, what would be missing in the cache and need
> to be built? The missing cache objects are in a dependency hierarchy,
> so only those missing objects with no dependecies on other missing
> objects would be printed. That should be comparatively easy to add as
> bitbake already does those checks all the time.

Right, what we lack is a way for the user to ask this and see the
result easily. As you say, bitbake can do this already.

>  Is there something else that's easily done and useful to print?

I think there is also the scenario of:
"I've run a build and have an existing TMPDIR and stamp info. I've now
pulled in a change. How much is going to rebuild and more importantly
*why*?"

This is different to a remote sstate situation as you have the stamp
info of the previous build already there to compare against.

> 2. Then there's the question of *why* they are missing, which is
> harder to answer. If, say, curl:do_package is not in the cache, then
> the tool would have to walk the cache tree (I/O heavy operation as
> there is no index), make a list of all curl:do_package objects that
> are there, and do a recursive bitbake-diffsig (going up the task tree)
> on them vs the one we want. Then print them starting with the newest.
> Something like:
> 
> Existing cache objects are not suitable because:
> <object id 1> was built on <date> and has a mismatching SRCREV
> <object id 2> was built on <earlier date> and has a different do_compile()

In theory you can do an:

ls sstate/*/*/sstate:curl:corei7-64-poky-linux:<PV>:<PR>:corei7-64:10:*_package.tar.zst

and get a list of possible objects. Some key information was put into
the cache file names for this reason. Obviously this becomes much
trickier when the sstate is remote over http though.

"newest" is problematic in this context as you really want the closest
match. There is no concept of build date in the cache as it often isn't
relevant (building an old release for example). The only point a
date/time is used is for cleaning out the cache for files which haven't
been accessed in a long time.

The tools are already supposed to support doing this with local file
sstate sources, they just do a bad job at getting the diffs right. One
intent of this work item was to try and understand why they don't work
and address that so at least for filesystem sstate mirrors, you can get
better results. I don't know how we solve the remote http issue as yet.

> > One of the big problems in the past was that we lost much of the hash
> > information after parsing completed. This meant that if the hashes then
> > didn't match, we couldn't tell why as the original computation was
> > lost. I did some work on allowing us to retain more of the information
> > so that we didn't have to recompute it every time to be able to do
> > processing with it. I have to admit I've totally lost track of where I
> > got to with that.
> 
> Here's an idea I can't get out of my head. Right now, the cache is
> simply an amorphous mass of objects, with no information regarding how
> they were created. How about storing complete build confgurations as
> well into the same directory? There would be a dedicated, separate
> area for each configuration that placed objects into the cache,
> containing:
> - list of layers and revisions
> - config template used
> - complete content of build/conf
> - bitbake invocation (e.g. targets and prefixed variables like MACHINE etc.)
> - complete list of sstate objects that were produced as a result, so
> they can be checked for existence
> 
> This would be written into the cache dir at the very end of the build
> when everything else is already there.

I'm not sure this helps as much as you'd like. For example I build
core-image-sato-sdk on the autobuilder and populate this but you want
to build core-image-sato locally. There would be no info here that
would help with information about what core-image-sato would need,
despite the fact we know one is a derivative of the other and there
would be a near full cache for it.

Taking your example above and curl:do_package, how would the list of
information above tell us why it wasn't and what the configuration
difference was? It can tell us a set of different configs which result
in hashes but not how an unknown hash may differ from the existing
configs. The best you could do is iterate the configs and see which one
got you closest to what you have but I suspect that may be even more
expensive that the sstate search mentioned above, indexing of http
aside.

I can see it helping for some limited debug scenarios but the
information you want is really mostly what is already being encoded
into the sstate filename.

Running with that thought, if that information is what we really need,
should be we indexing it and sharing through some kind of hash
equivalence like service to provide the index?


> Right now, everyone sets up their own builds first, then points
> local.conf or site.conf to the cache, and hopes for the best regarding
> hit rates. Having stored build configs would allow inverting the
> workflow, so that you first ask from the cache what it can provide
> (e.g. it can provide mickledore or kirkstone core-image-minimal for
> qemux86, and that's exactly what you want as a starting point), then
> you use the build config stored in the cache to set up a build, and
> run it - and that would guarantee complete sstate reuse and getting to
> a functional image as soon as possible. Kind of like binary distro,
> but implemented with sstate.

Effectively those "configs" are what the eSDK is, you're just proposing
a server side set of them rather than a built copy. In many ways that
could be useful way of possibly rethinking the way the eSDK works.

So in that sense I like the idea, we should just be clear the problem
it solves as it doesn't solve the "why is my build not using the cache"
question specifically but it potentially solves other issues.

Cheers,

Richard

Re: Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 14 Sept 2023 at 21:54, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:

> Effectively those "configs" are what the eSDK is, you're just proposing
> a server side set of them rather than a built copy. In many ways that
> could be useful way of possibly rethinking the way the eSDK works.
>
> So in that sense I like the idea, we should just be clear the problem
> it solves as it doesn't solve the "why is my build not using the cache"
> question specifically but it potentially solves other issues.

Right, I didn't make it clear that this idea is not about 'why sstate
is not being reused', but rather something I thought of to avoid
placing people in that situation to begin with - offering them
'pre-fabricated builds' with guaranteed sstate availability. I have a
hunch there'll be plenty, for whom this is just fine, and eventually
this could be the basis of that oe-setup holy grail :)

I'll start with the simplest enhancement of printing what would be
rebuilt. There's a lot of code I need to understand, so this needs to
be in manageable chunks :)

Alex

Re: Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Julien Stephan]

Hi Alexander,
About bblock tool: I just force pushed my branch on poky-contrib
(https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock),
trying to fix an autobuilder issue reported by Alexandre Belloni.
I am still working on this, and I would be very happy to get this merged :)

Cheers
Julien

Le ven. 15 sept. 2023 à 10:28, Alexander Kanavin
<alex.kanavin@gmail.com> a écrit :
>
> On Thu, 14 Sept 2023 at 21:54, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
>
> > Effectively those "configs" are what the eSDK is, you're just proposing
> > a server side set of them rather than a built copy. In many ways that
> > could be useful way of possibly rethinking the way the eSDK works.
> >
> > So in that sense I like the idea, we should just be clear the problem
> > it solves as it doesn't solve the "why is my build not using the cache"
> > question specifically but it potentially solves other issues.
>
> Right, I didn't make it clear that this idea is not about 'why sstate
> is not being reused', but rather something I thought of to avoid
> placing people in that situation to begin with - offering them
> 'pre-fabricated builds' with guaranteed sstate availability. I have a
> hunch there'll be plenty, for whom this is just fine, and eventually
> this could be the basis of that oe-setup holy grail :)
>
> I'll start with the simplest enhancement of printing what would be
> rebuilt. There's a lot of code I need to understand, so this needs to
> be in manageable chunks :)
>
> Alex

Re: Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1859 bytes --]

That’s great to hear! When you have a new patch set please post it here. I
think there’s also a new reply from RP that you need to consider.

Alex

On Wed 20. Sep 2023 at 16.25, Julien Stephan <jstephan@baylibre.com> wrote:

> Hi Alexander,
> About bblock tool: I just force pushed my branch on poky-contrib
> (https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock),
> trying to fix an autobuilder issue reported by Alexandre Belloni.
> I am still working on this, and I would be very happy to get this merged :)
>
> Cheers
> Julien
>
> Le ven. 15 sept. 2023 à 10:28, Alexander Kanavin
> <alex.kanavin@gmail.com> a écrit :
> >
> > On Thu, 14 Sept 2023 at 21:54, Richard Purdie
> > <richard.purdie@linuxfoundation.org> wrote:
> >
> > > Effectively those "configs" are what the eSDK is, you're just proposing
> > > a server side set of them rather than a built copy. In many ways that
> > > could be useful way of possibly rethinking the way the eSDK works.
> > >
> > > So in that sense I like the idea, we should just be clear the problem
> > > it solves as it doesn't solve the "why is my build not using the cache"
> > > question specifically but it potentially solves other issues.
> >
> > Right, I didn't make it clear that this idea is not about 'why sstate
> > is not being reused', but rather something I thought of to avoid
> > placing people in that situation to begin with - offering them
> > 'pre-fabricated builds' with guaranteed sstate availability. I have a
> > hunch there'll be plenty, for whom this is just fine, and eventually
> > this could be the basis of that oe-setup holy grail :)
> >
> > I'll start with the simplest enhancement of printing what would be
> > rebuilt. There's a lot of code I need to understand, so this needs to
> > be in manageable chunks :)
> >
> > Alex
>

[-- Attachment #2: Type: text/html, Size: 2711 bytes --]

Re: Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Julien Stephan]

Unless I missed something, I think I fixed all comments from RP on the
branch I force pushed :)

Cheers
Julien

Le mer. 20 sept. 2023 à 16:31, Alexander Kanavin
<alex.kanavin@gmail.com> a écrit :
>
> That’s great to hear! When you have a new patch set please post it here. I think there’s also a new reply from RP that you need to consider.
>
> Alex
>
> On Wed 20. Sep 2023 at 16.25, Julien Stephan <jstephan@baylibre.com> wrote:
>>
>> Hi Alexander,
>> About bblock tool: I just force pushed my branch on poky-contrib
>> (https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/bblock),
>> trying to fix an autobuilder issue reported by Alexandre Belloni.
>> I am still working on this, and I would be very happy to get this merged :)
>>
>> Cheers
>> Julien
>>
>> Le ven. 15 sept. 2023 à 10:28, Alexander Kanavin
>> <alex.kanavin@gmail.com> a écrit :
>> >
>> > On Thu, 14 Sept 2023 at 21:54, Richard Purdie
>> > <richard.purdie@linuxfoundation.org> wrote:
>> >
>> > > Effectively those "configs" are what the eSDK is, you're just proposing
>> > > a server side set of them rather than a built copy. In many ways that
>> > > could be useful way of possibly rethinking the way the eSDK works.
>> > >
>> > > So in that sense I like the idea, we should just be clear the problem
>> > > it solves as it doesn't solve the "why is my build not using the cache"
>> > > question specifically but it potentially solves other issues.
>> >
>> > Right, I didn't make it clear that this idea is not about 'why sstate
>> > is not being reused', but rather something I thought of to avoid
>> > placing people in that situation to begin with - offering them
>> > 'pre-fabricated builds' with guaranteed sstate availability. I have a
>> > hunch there'll be plenty, for whom this is just fine, and eventually
>> > this could be the basis of that oe-setup holy grail :)
>> >
>> > I'll start with the simplest enhancement of printing what would be
>> > rebuilt. There's a lot of code I need to understand, so this needs to
>> > be in manageable chunks :)
>> >
>> > Alex

Re: Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 14 Sept 2023 at 21:54, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> The tools are already supposed to support doing this with local file
> sstate sources, they just do a bad job at getting the diffs right. One
> intent of this work item was to try and understand why they don't work
> and address that so at least for filesystem sstate mirrors, you can get
> better results. I don't know how we solve the remote http issue as yet.

I ran a few experiments with bitbake -S printdiff, and so far I
haven't yet seen an unhelpful result. Here's the most impressive one:

$ bitbake libsolv-native (uses cmake-native)
$ git revert <latest cmake version update>
$ rm -rf build/tmp/
$ bitbake -S printdiff libsolv-native
...
The differences between the current build and any cached tasks start
at the following tasks:
/srv/work/alex/poky/meta/recipes-devtools/cmake/cmake-native_3.27.4.bb:do_recipe_qa
NOTE: Writing task signature files
Writing locked sigs to /srv/storage/alex/yocto/build-sstate/locked-sigs.inc

Task cmake-native:do_recipe_qa couldn't be used from the cache because:
  We need hash 5e649a49c4f0de2e62bc8fa4215df1021b9772762065352ef0d204d2d72f4efb,
closest matching task was
85c73eadca06d0e92fcea130ae6e23e902c96314ef1c38c60a14ed3445d24ed7
  basehash changed from
7d4adf817d99893a30a94330803a0eb1c00652ab217c21599944f81f023af6cd to
7e93631376ed159c11460647bf7f4178cb260d44f367d0858c2cd96ae2256b09
  Variable PV value changed from '3.27.5' to '3.27.4'
  Variable SRC_URI[sha256sum] value changed from
'5175e8fe1ca9b1dd09090130db7201968bcce1595971ff9e9998c2f0765004c9' to
'0a905ca8635ca81aa152e123bdde7e54cbe764fdd9a70d62af44cad8b92967af'

That's pretty good, isn't it? It does print both what needs to be
re-run, and *why* as well.

I have no idea yet what kind of magic it does to find the 'closest
matching task' in sstate, but if this breaks down in some other
scenarios, we need to find them to get a starting point for making the
tools better. Ideas? I'm ready to try them :)

Alex

RE: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[chris.laplante]

> On Thu, 14 Sept 2023 at 21:54, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > The tools are already supposed to support doing this with local file
> > sstate sources, they just do a bad job at getting the diffs right. One
> > intent of this work item was to try and understand why they don't work
> > and address that so at least for filesystem sstate mirrors, you can
> > get better results. I don't know how we solve the remote http issue as yet.
> 
> I ran a few experiments with bitbake -S printdiff, and so far I haven't yet seen
> an unhelpful result. Here's the most impressive one:
> 
> $ bitbake libsolv-native (uses cmake-native) $ git revert <latest cmake version
> update> $ rm -rf build/tmp/ $ bitbake -S printdiff libsolv-native ...
> The differences between the current build and any cached tasks start at the
> following tasks:
> /srv/work/alex/poky/meta/recipes-devtools/cmake/cmake-
> native_3.27.4.bb:do_recipe_qa
> NOTE: Writing task signature files
> Writing locked sigs to /srv/storage/alex/yocto/build-sstate/locked-sigs.inc
> 
> Task cmake-native:do_recipe_qa couldn't be used from the cache because:
>   We need hash
> 5e649a49c4f0de2e62bc8fa4215df1021b9772762065352ef0d204d2d72f4efb,
> closest matching task was
> 85c73eadca06d0e92fcea130ae6e23e902c96314ef1c38c60a14ed3445d24ed7
>   basehash changed from
> 7d4adf817d99893a30a94330803a0eb1c00652ab217c21599944f81f023af6cd to
> 7e93631376ed159c11460647bf7f4178cb260d44f367d0858c2cd96ae2256b09
>   Variable PV value changed from '3.27.5' to '3.27.4'
>   Variable SRC_URI[sha256sum] value changed from
> '5175e8fe1ca9b1dd09090130db7201968bcce1595971ff9e9998c2f0765004c9'
> to
> '0a905ca8635ca81aa152e123bdde7e54cbe764fdd9a70d62af44cad8b92967af'
> 
> That's pretty good, isn't it? It does print both what needs to be re-run, and
> *why* as well.
> 
> I have no idea yet what kind of magic it does to find the 'closest matching task'
> in sstate, but if this breaks down in some other scenarios, we need to find them
> to get a starting point for making the tools better. Ideas? I'm ready to try them
> :)

That is very impressive and I'd also love to hear about what heuristics it uses. 

We use Artifactory to host our sstate. (Artifactory doesn’t have specific support for it, it's just basically acting as a generic HTTP server. But it makes things like LDAP easy). I have for a while been thinking of building a tool that tries to find the "closest" sstate on the server and then recursively runs bitbake-diffsig on it. The tool was going to be called 'why-not-sstate'.

Thanks,
Chris

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 21 Sept 2023 at 16:39, chris.laplante@agilent.com
<chris.laplante@agilent.com> wrote:

> That is very impressive and I'd also love to hear about what heuristics it uses.

It's actually rather simple. It uses glob.glob on stamps in tmp/, then
on local sstate to find possible matches, then sorts them by mtime and
takes the most recent. It's what would work most of the time, but we
could add printdiff-all (print difference with all sstate matches) or
printdiff-N (N most recent). It also could abstain from dumping
locked-sigs.inc into cwd with both -S none and -S printdiff, unless
explicitly asked

I just discovered there's also scripts/bitbake-whatchanged (that
hasn't seen activity in years and is neither documented nor tested).
Unsurprisingly then, it doesn't work in the same scenario:

================
alex@Zen2:/srv/storage/alex/yocto/build-sstate$ bitbake-whatchanged
libsolv-native
Figuring out the STAMPS_DIR ...
Generating the new stamps ... (need several minutes)

=== Summary: (0 changed, 0 unchanged)
Newly added: 0
PV changed: 0
PR changed: 0
Dependencies changed: 0

Removing the newly generated stamps dir ...
================

Maybe this is what RP was referring to when he said the tools don't
work properly?


Alex

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Fri, 2023-09-22 at 11:17 +0200, Alexander Kanavin wrote:
> On Thu, 21 Sept 2023 at 16:39, chris.laplante@agilent.com
> <chris.laplante@agilent.com> wrote:
> 
> > That is very impressive and I'd also love to hear about what heuristics it uses.
> 
> It's actually rather simple. It uses glob.glob on stamps in tmp/, then
> on local sstate to find possible matches, then sorts them by mtime and
> takes the most recent. It's what would work most of the time, but we
> could add printdiff-all (print difference with all sstate matches) or
> printdiff-N (N most recent). It also could abstain from dumping
> locked-sigs.inc into cwd with both -S none and -S printdiff, unless
> explicitly asked
> 
> I just discovered there's also scripts/bitbake-whatchanged (that
> hasn't seen activity in years and is neither documented nor tested).
> Unsurprisingly then, it doesn't work in the same scenario:
> 
> ================
> alex@Zen2:/srv/storage/alex/yocto/build-sstate$ bitbake-whatchanged
> libsolv-native
> Figuring out the STAMPS_DIR ...
> Generating the new stamps ... (need several minutes)
> 
> === Summary: (0 changed, 0 unchanged)
> Newly added: 0
> PV changed: 0
> PR changed: 0
> Dependencies changed: 0
> 
> Removing the newly generated stamps dir ...
> ================
> 
> Maybe this is what RP was referring to when he said the tools don't
> work properly?

No, I've believed that should probably be removed. I think there was a
recent change to it.

I think we had a major step change in this functionality working when
this was fixed:

https://git.yoctoproject.org/poky/commit/?id=84a7485025dd4473403b8da36a0c979a3afd5e93

and this test case was added:

https://git.yoctoproject.org/poky/commit/?id=1bdcd76d2968c3cc6ec2815afceba1cf98efd6d5

Things which used to be problematic:

a) changes involving changes to gcc-source since it uses a shared
sources stamps which confused the tools (at least used to). That may
have been before gcc-source became a recipe?
b) changes to a very common component (e.g. autoconf-native's
do_configure) which make it hard to understand where the root cause of
the changes came from
c) changes which affect many recipes at once, e.g. the do_configure
function in base.bbclass

It might be helpful to write test cases for the scenario you showed as
working above and some of the ones I mention above, then we can
document they work and have an easier way to add tests for issues
if/as/when we identify the problematic scenarios in future.

As you mention, it also uses mtime so perhaps issues happen if you run
a different build, then try and go back to the other config? I suspect
once you understand the algorithm the code uses, you can pick holes in
it.

Cheers,

Richard

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Fri, 22 Sept 2023 at 12:42, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:

> Things which used to be problematic:
>
> a) changes involving changes to gcc-source since it uses a shared
> sources stamps which confused the tools (at least used to). That may
> have been before gcc-source became a recipe?
> b) changes to a very common component (e.g. autoconf-native's
> do_configure) which make it hard to understand where the root cause of
> the changes came from
> c) changes which affect many recipes at once, e.g. the do_configure
> function in base.bbclass
>
> It might be helpful to write test cases for the scenario you showed as
> working above and some of the ones I mention above, then we can
> document they work and have an easier way to add tests for issues
> if/as/when we identify the problematic scenarios in future.

I've now written down the tests for these three scenarios and got them
to pass (in oe-selftest too \0/):
https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/sstate-for-all
(check the commit message too)

I am going to look closer at bitbake-whatchanged, what it aims to do
and why it doesn't work. I have a hunch it can produce useful high
level reports, and so shouldn't be simply thrown away. 'bitbake -S
printdiff' is too techy and verbose for some use cases. Maybe we can
fold that functionality into 'bitbake -S whatchanged'.

I'm on holiday next week.

Alex

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Thu, 2023-09-28 at 18:43 +0200, Alexander Kanavin wrote:
> On Fri, 22 Sept 2023 at 12:42, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> 
> > Things which used to be problematic:
> > 
> > a) changes involving changes to gcc-source since it uses a shared
> > sources stamps which confused the tools (at least used to). That may
> > have been before gcc-source became a recipe?
> > b) changes to a very common component (e.g. autoconf-native's
> > do_configure) which make it hard to understand where the root cause of
> > the changes came from
> > c) changes which affect many recipes at once, e.g. the do_configure
> > function in base.bbclass
> > 
> > It might be helpful to write test cases for the scenario you showed as
> > working above and some of the ones I mention above, then we can
> > document they work and have an easier way to add tests for issues
> > if/as/when we identify the problematic scenarios in future.
> 
> I've now written down the tests for these three scenarios and got them
> to pass (in oe-selftest too \0/):
> https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/sstate-for-all
> (check the commit message too)
> 
> I am going to look closer at bitbake-whatchanged, what it aims to do
> and why it doesn't work. I have a hunch it can produce useful high
> level reports, and so shouldn't be simply thrown away. 'bitbake -S
> printdiff' is too techy and verbose for some use cases. Maybe we can
> fold that functionality into 'bitbake -S whatchanged'.

I've wondered if we should split bitbake -S printdiff into a separate
utility? It exists from a time before we had bitbake command APIs.

I'm curious to see what you find with analysis of bitbake-whatchanged.
I'm also somewhat surprised the scenarios you're testing all work!

I'm guess one of the commits I pointed to must have fixed them (the
removal of paths from the sig files)?

Cheers,

Richard

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 28 Sept 2023 at 18:49, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> I've wondered if we should split bitbake -S printdiff into a separate
> utility? It exists from a time before we had bitbake command APIs.

It can also be a simple shell wrapper.

I've separated -S lockedsigs into it's own option as well, so it can
be explicitly requested (just need to fix up a couple of selftests
that relied on -S none to get locked-sigs.inc):
https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/sstate-for-all&id=41e8a8a79bac4e3b9eb74a99e32e0d26ac8af0c5

> I'm curious to see what you find with analysis of bitbake-whatchanged.
> I'm also somewhat surprised the scenarios you're testing all work!

It's not 100% perfect. One out of 2*3=6 scenarios isn't fully
functional (gcc-source:do_preconfigure signature isn't found from
sstate even though it exists there - but it is found if it's in
tmp/stamps/). I left a FIXME in the test.

Alex

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 28 Sept 2023 at 18:49, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:

> I'm curious to see what you find with analysis of bitbake-whatchanged.

I've taken a look a the script. It obtains the current location of
STAMPS_DIR, then runs this:

        # Generate the new stamps dir
        print("Generating the new stamps ... (need several minutes)")
        cmdline = "STAMPS_DIR=%s bitbake -S none %s" % (new_stampsdir,
args.recipe)

Then it walks both trees, matching up file names with a regex:

# Match the stamp's filename
# group(1): PE_PV (may no PE)
# group(2): PR
# group(3): TASK
# group(4): HASH
stamp_re = re.compile("(?P<pv>.*)-(?P<pr>r\d+)\.(?P<task>do_\w+)\.(?P<hash>[^\.]*)")

Then there's some code that finds out what changed in the above
between the two sets.

I don't see a way to make it work: messing about with STAMPS_DIR like
that isn't supported, and will either do nothing, or remove the
original stamps. Also stamp filenames aren't really a 'public API',
are they?

Should the script simply be removed, or is there some better way to
re-implement answering the 'what has changed' question in a way that
doesn't flood the console with task hashes? I'd be glad to get
suggestions for this.


Alex

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Fri, 2023-09-29 at 14:06 +0200, Alexander Kanavin wrote:
> On Thu, 28 Sept 2023 at 18:49, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> 
> > I'm curious to see what you find with analysis of bitbake-whatchanged.
> 
> I've taken a look a the script. It obtains the current location of
> STAMPS_DIR, then runs this:
> 
>         # Generate the new stamps dir
>         print("Generating the new stamps ... (need several minutes)")
>         cmdline = "STAMPS_DIR=%s bitbake -S none %s" % (new_stampsdir,
> args.recipe)
> 
> Then it walks both trees, matching up file names with a regex:
> 
> # Match the stamp's filename
> # group(1): PE_PV (may no PE)
> # group(2): PR
> # group(3): TASK
> # group(4): HASH
> stamp_re = re.compile("(?P<pv>.*)-(?P<pr>r\d+)\.(?P<task>do_\w+)\.(?P<hash>[^\.]*)")
> 
> Then there's some code that finds out what changed in the above
> between the two sets.
> 
> I don't see a way to make it work: messing about with STAMPS_DIR like
> that isn't supported, and will either do nothing, or remove the
> original stamps. Also stamp filenames aren't really a 'public API',
> are they?
> 
> Should the script simply be removed, or is there some better way to
> re-implement answering the 'what has changed' question in a way that
> doesn't flood the console with task hashes? I'd be glad to get
> suggestions for this.

I'd prefer to see some dedicated bitbake API used even if we need to
create/add it. tinfoil and some of the bblock/unlock work shows we can
get stamp data, the question would be how to get it without
"disturbing" the existing build.

By using dedicated API, we'd be able to control the console output.

Cheers,

Richard

Re: [Openembedded-architecture] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Fri, 29 Sept 2023 at 14:27, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:

> I'd prefer to see some dedicated bitbake API used even if we need to
> create/add it. tinfoil and some of the bblock/unlock work shows we can
> get stamp data, the question would be how to get it without
> "disturbing" the existing build.
>
> By using dedicated API, we'd be able to control the console output.

I just noticed that compare_sigfiles() has a 'collapsed' argument,
which, when set to True, omits most or all of the task hash printing,
and with that its output should approximate what bitbake-whatchanged
is aiming to do. This is currently used only by buildhistory, but it
could be used by -S printdiff too, e.g. by having verbose and concise
modes.

I'll run some experiments, and let's see what the overall output looks
like in real scenarios (e.g. 4.3_M3 vs current master).

Alex

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[adrian.freihofer]

Hi Alex, hi Richard

The discussion looks really interesting. I would like to contribute
some comments from the point of view of a rather naive user and try to
understand the workflows for which these improvements would be
beneficial also on a bigger picture.


On Thu, 2023-09-14 at 20:51 +0200, Alexander Kanavin wrote:
> On Thu, 14 Sept 2023 at 14:56, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > For the task signatures, we need to think about some questions. If
> > I
> > make a change locally, can I query how much will rebuild and how
> > much
> > will be reused? There is bitbake --dry-run but perhaps it is time
> > for a
> > an option (or dedicated separate command?) to give some statistics
> > about what bitbake would do? How much sstate would be reused?
> > 
> > That then logically leads into the questions, can we tell what has
> > changed? Why isn't my sstate being reused? For that we perhaps
> > should
> > define some existing scenarios where it is currently very difficult
> > to
> > work this out and then work out how we can report that information
> > to
> > the user. These could become test cases?
> 
> So I think there are two questions here that the tools should answer:
> 
> 1. If I would run a build, what would be missing in the cache and
> need
> to be built? The missing cache objects are in a dependency hierarchy,
> so only those missing objects with no dependecies on other missing
> objects would be printed. That should be comparatively easy to add as
> bitbake already does those checks all the time. Is there something
> else that's easily done and useful to print?
> 
> 2. Then there's the question of *why* they are missing, which is
> harder to answer. If, say, curl:do_package is not in the cache, then
> the tool would have to walk the cache tree (I/O heavy operation as
> there is no index), make a list of all curl:do_package objects that
> are there, and do a recursive bitbake-diffsig (going up the task
> tree)
> on them vs the one we want. Then print them starting with the newest.
> Something like:
> 

We are currently experimenting with replacing the eSDK installer with
the bitbake build environment for our users. Part of this
transformation is, of course, the shared sstate-cache, for which this
discussion seems quite relevant. The workflow we are aiming for is as
follows:

   1. Setup the layers and build config (out of scope here)
   2. Download the sstate for a particular recipe (usually an image
      recipe).
      
      Note: Working with SSTATE_MIRRORS does not work very well because
      bitbake connects way too often to the sstate server. So we
      started to develop a script which downloads the sstate artifacts
      into SSTATE_DIR to get the SDK set up.
      
      Your point 1. is basically what our (a bit hacky) download script
      does in the --dry-run mode: Printing all the required sstate
      artifacts. I think as a first step, it would be very valuable to
      have a function or a tinfoil API in the core that returns a list
      of sstate artifacts for a given recipe.
      
      As a second step a tool or a new feature of an existing tool
      could download the artifacts into SSTATE_DIR. This would also be
      a great successor for the not so much maintained devtool sdk-
      update command.
   3. At this point, the user is basically in the same situation as
      after installing the eSDK installer. devtool and now also bitbake
      are available, the sstate-cache is fully populated.

Maintaining such an SDK and sstate mirror infrastructure brings us to
your point 2. Tooling for maintaining the sstate cache becomes even
more important than it is now. Also, locking the sstate cache and
treating missing artifacts as errors seems to be an important feature. 

But I would consider the locking/unlocking more relevant for testing
rather than for deploying locked SDKs. Unlocked SDKs allow the user to
switch the branches of the layers to commits where the sstate is not
available. If the user is in a full featured bitbake environment rather
than the constrained eSDK installer environment this is perfectly fine.
Bitbake can just compile the missing recipes.
Such an SDK would combine all the advantages of the current eSDK
installer and the much more flexible Bitbake environment. This would
imply that the sstate download script should just try to download
what's available on the mirror and maybe print a warning for artifacts
which cannot be downloaded. But it should not abort with an error for
missing artifacts.

Does that make sense?

Regrads,
Adrian

> Existing cache objects are not suitable because:
> <object id 1> was built on <date> and has a mismatching SRCREV
> <object id 2> was built on <earlier date> and has a different
> do_compile()
> 
> > One of the big problems in the past was that we lost much of the
> > hash
> > information after parsing completed. This meant that if the hashes
> > then
> > didn't match, we couldn't tell why as the original computation was
> > lost. I did some work on allowing us to retain more of the
> > information
> > so that we didn't have to recompute it every time to be able to do
> > processing with it. I have to admit I've totally lost track of
> > where I
> > got to with that.
> 
> Here's an idea I can't get out of my head. Right now, the cache is
> simply an amorphous mass of objects, with no information regarding
> how
> they were created. How about storing complete build confgurations as
> well into the same directory? There would be a dedicated, separate
> area for each configuration that placed objects into the cache,
> containing:
> - list of layers and revisions
> - config template used
> - complete content of build/conf
> - bitbake invocation (e.g. targets and prefixed variables like
> MACHINE etc.)
> - complete list of sstate objects that were produced as a result, so
> they can be checked for existence
> 
> This would be written into the cache dir at the very end of the build
> when everything else is already there.
> 
> Right now, everyone sets up their own builds first, then points
> local.conf or site.conf to the cache, and hopes for the best
> regarding
> hit rates. Having stored build configs would allow inverting the
> workflow, so that you first ask from the cache what it can provide
> (e.g. it can provide mickledore or kirkstone core-image-minimal for
> qemux86, and that's exactly what you want as a starting point), then
> you use the build config stored in the cache to set up a build, and
> run it - and that would guarantee complete sstate reuse and getting
> to
> a functional image as soon as possible. Kind of like binary distro,
> but implemented with sstate.
> 
> Alex
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#187647):
> https://lists.openembedded.org/g/openembedded-core/message/187647
> Mute This Topic: https://lists.openembedded.org/mt/101356420/4454582
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe:
> https://lists.openembedded.org/g/openembedded-core/unsub [
> adrian.freihofer@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Wed, 1 Nov 2023 at 15:18, <adrian.freihofer@gmail.com> wrote:
> We are currently experimenting with replacing the eSDK installer with
> the bitbake build environment for our users. Part of this
> transformation is, of course, the shared sstate-cache, for which this
> discussion seems quite relevant. The workflow we are aiming for is as
> follows:
>
>    1. Setup the layers and build config (out of scope here)
>    2. Download the sstate for a particular recipe (usually an image
>       recipe).

I'm not sure I understand the case for downloading the complete set of
cache objects up front. What's wrong with 'lazy' downloading, e.g.
only when the objects are actually needed in a build?

Even then, does --setscene-only do just that, or am I completely confused?

Alex

Re: [Openembedded-architecture] [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[adrian.freihofer]

On Wed, 2023-11-01 at 16:19 +0100, Alexander Kanavin via
lists.openembedded.org wrote:
> On Wed, 1 Nov 2023 at 15:18, <adrian.freihofer@gmail.com> wrote:
> > We are currently experimenting with replacing the eSDK installer
> > with
> > the bitbake build environment for our users. Part of this
> > transformation is, of course, the shared sstate-cache, for which
> > this
> > discussion seems quite relevant. The workflow we are aiming for is
> > as
> > follows:
> > 
> >    1. Setup the layers and build config (out of scope here)
> >    2. Download the sstate for a particular recipe (usually an image
> >       recipe).
> 
> I'm not sure I understand the case for downloading the complete set
> of
> cache objects up front. What's wrong with 'lazy' downloading, e.g.
> only when the objects are actually needed in a build?
> 
> Even then, does --setscene-only do just that, or am I completely
> confused?

If I remember correctly, when configuring a SSTATE_MIRROR, the server
is queried when a Setcene task is executed. This is not ideal for
several reasons:
 * Bitbake Setcene tasks run in the context of a recipe, which means
   that thousands of connections are opened against the mirror server.
   Many servers treat this behavior as a denial of service attack. With
   a separate download tool, this can be handled with a reasonable
   number of parallel connections. With bitbake's architecture, this is
   a barely solvable issue.
 * Especially with an SDK, setcene tasks can run much more frequently
   than do_fetch tasks. So the same artifact is downloaded multiple
   times, which is not efficient.
 * SDK users may be located in different places in the world. This
   results in poor performance depending on the location.
 * With a separate download, you can download on a fast network and
   then switch to a slower network or even work offline.
 * Of course, supporting downloading in advance does not mean that a
   SSTATE_MIRROR should not be supported.

Adrian

> 
> Alex
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#1821):
> https://lists.openembedded.org/g/openembedded-architecture/message/1821
> Mute This Topic: https://lists.openembedded.org/mt/102320110/3616858
> Group Owner: openembedded-architecture+owner@lists.openembedded.org
> Unsubscribe:
> https://lists.openembedded.org/g/openembedded-architecture/unsub [
> adrian.freihofer@siemens.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[adrian.freihofer]

Hi Alex, hi Richard

After some internal discussions, I would like to clarify my previous
answers on this topic.

 * Usually there are two different workflows
    - application developers: could use an SDK with a locked sstate-cache.
    - Yocto/BSP developers: need an unlocked SDK. They change the recipes.
 * A locked SDK
    - can work with setscene from SSTATE_MIRRORS
    - setscene does caching in the SSTATE_DIR (no issue about that)
    - But network problems can occur during the initial build because
      bitbake executes many independent setscene tasks. Opening so many
      independent connections slows down the build, especially if the
      server treats them as a denial of service attack.
    - The denial of service problem is difficult to solve because each
      setscene task runs in its own bibtake task. Reusing a connection to
      download multiple sstate artifacts seems almost impossible.
      This is much easier to solve with separate sstate download script.
 * An unlocked SDK
    - Tries to download the sstate cache for changed recipes and their
      dependencies, which obviously can't work.
    - The useless download requests slow down the build considerably and
      cause a high load on the servers without any benefit.
    - A script which gets a list of sstate artifacts from bitbake and then
      does a upfront download works much better
       + The script runs only when the user calls it or the SDK gets boot-
         strapped
       + The script uses a reasonable amount of parallel connections which
         are re-used for more then one artifact download
 * Idea for a smart lock/unlock implementation
    - Form a user's perspective a locked vs. an unlocked SDK does not make
      much sense. It makes more sense if the SDK would automatically
      download the sstate-cache if it is expected to be available.
      Lets think about an implementation (which allows to override the
      logic) to switch from automatic to manual mode:
      
      SSTATE_MIRRORS_ENABLED ?= "${is_sstate_mirror_available()}"
      
      In our case the sstate mirror is expected to provide all artifacts
      for tagged commits and for some git branches of the layer
      repositories.
      The sstate is obviousely not usable for a "dirty" git layer
      repository. That's what the is_sstate_mirror_available function
      could check to automatically enable and disable lazy downloads.
      
    - If is_sstate_mirror_available() returns false, it should still be
      possible to initiate a sstate-cache download manually.
      
 * Terminology
    - Older Yocto Releases:
       + eSDK means an installer which provides a different environment with
         different tools
       + The eSDK was static, with a locked sstate cache
       + Was for one MACHINE, for one image...
    - Newer Yocto Releases:
       + The bitbake environment offers all features of the eSDK installer. I
         consider this as already implemented with meta-ide-support and
         build-sysroots.
       + The term eSDK means a replicable bitbake environment. (The
         documentation was recently changed in that sense)
       + The new SDK installer can be generated in different variants similar
         to what was already supported by the eSDK
         installer: https://docs.yoctoproject.org/sdk-manual/appendix-
         customizing.html#customizing-the-extensible-sdk-standalone-
         installer.
          * The lightest variant is just a script that sets up the layers (git
            clone or git checkout) and provides the build config. If
            SSTATE_MIRRORS are configured lazy downloads will just work
            otherwise bitbake will compile everything from scratch.
          * The heaviest variant of the SDK installer includes the layers and
            the sstate-cache. After installing it is_sstate_mirror_available()
            evaluates to True.
* devtool ide
- Tries to bring this ideas further towards IDE configuration
- It supports two modes: No recipe mode is like the old eSDK
environment. The recipe mode goes beyond that. It is based on
devtool modify.
    - Naming: I was thinking about "devtool ide" versus "devtool esdk"
       + Why ide? I want to set up my IDE to work with the Yocto SDK. And
         that means, of course, that I have to bootstrap the SDK.
       + Could also be esdk: I want to bootstrap the eSDK and that should
         also configure my IDE.
 * Latest patch series is here: 
    - https://lists.openembedded.org/g/openembedded-core/message/189899
    - docs: https://lists.yoctoproject.org/g/docs/message/4578

Adrian

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Sat, 2023-11-04 at 11:29 +0100, adrian.freihofer@gmail.com wrote:
> Hi Alex, hi Richard
> 
> After some internal discussions, I would like to clarify my previous
> answers on this topic.
> 
>  * Usually there are two different workflows
>     - application developers: could use an SDK with a locked sstate-cache.
>     - Yocto/BSP developers: need an unlocked SDK. They change the recipes.
>  * A locked SDK
>     - can work with setscene from SSTATE_MIRRORS
>     - setscene does caching in the SSTATE_DIR (no issue about that)
>     - But network problems can occur during the initial build because
>       bitbake executes many independent setscene tasks. Opening so many
>       independent connections slows down the build, especially if the
>       server treats them as a denial of service attack.
>     - The denial of service problem is difficult to solve because each
>       setscene task runs in its own bibtake task. Reusing a connection to
>       download multiple sstate artifacts seems almost impossible.
>       This is much easier to solve with separate sstate download script.

FWIW, we did have a similar issue with do_fetch overloading
servers/proxies/ISPs and added:

do_fetch[number_threads] = "4"

Finding the right place to put a thread limit on overall setscene tasks
is harder but in theory possible. Or perhaps a "network capable tasks"
thread limit?

Is the overload caused by the initial query of sstate presence, or,
does it happen when the setscene tasks themselves run?


>  * An unlocked SDK
>     - Tries to download the sstate cache for changed recipes and their
>       dependencies, which obviously can't work.
>     - The useless download requests slow down the build considerably and
>       cause a high load on the servers without any benefit.

Is this sstate over http(s) or something else? I seem to remember you
mentioning sftp. If this were using sftp, it would be horribly slow as
it was designed for a light overhead "does this exist?" check which
http(s) can manage well.

Recently we've been wondering about teaching the hashequiv server about
"presence", which would then mean the build would only query things
that stood a good chance of existing.

>     - A script which gets a list of sstate artifacts from bitbake and then
>       does a upfront download works much better
>        + The script runs only when the user calls it or the SDK gets boot-
>          strapped
>        + The script uses a reasonable amount of parallel connections which
>          are re-used for more then one artifact download

Explaining to users they need to do X before Y quickly gets tiring,
both for people explaining it and the people doing it trying to
remember. I'd really like to get to a point where the system "does the
right thing" if we can.

I don't believe the problems you describe are insurmountable. If you
are using sftp, that is going to be a big chunk of the problem as the
system assumes something faster is available. Yes, I've taken patches
to make sftp work but it isn't recommended at all. I appreciate there
would be reasons why you use sftp but if it is possible to get a list
of "available sstate" via other means, it would improve things.

>  * Idea for a smart lock/unlock implementation
>     - Form a user's perspective a locked vs. an unlocked SDK does not make
>       much sense. It makes more sense if the SDK would automatically
>       download the sstate-cache if it is expected to be available.
>       Lets think about an implementation (which allows to override the
>       logic) to switch from automatic to manual mode:
>       
>       SSTATE_MIRRORS_ENABLED ?= "${is_sstate_mirror_available()}"

What determines this availability? I worry that is something very
fragile and specific to your use case. It is also not an all or nothing
binary thing.

>       In our case the sstate mirror is expected to provide all artifacts
>       for tagged commits and for some git branches of the layer
>       repositories.
>       The sstate is obviousely not usable for a "dirty" git layer
>       repository.

That isn't correct and isn't going to work. If I make a single change
locally, there is a good chance that 99.9% of the sstate could still be
valid in some cases. Forcing the user through 10 hours of rebuild when
potentially that much was available is a really really bad user
experience.

>  That's what the is_sstate_mirror_available function
>       could check to automatically enable and disable lazy downloads.
>       
>     - If is_sstate_mirror_available() returns false, it should still be
>       possible to initiate a sstate-cache download manually.
>       
>  * Terminology
>     - Older Yocto Releases:
>        + eSDK means an installer which provides a different environment with
>          different tools
>        + The eSDK was static, with a locked sstate cache
>        + Was for one MACHINE, for one image...
>     - Newer Yocto Releases:
>        + The bitbake environment offers all features of the eSDK installer. I
>          consider this as already implemented with meta-ide-support and
>          build-sysroots.

Remember bblock and bbunlock too. These provide a way to fix or unlock
specific sections of the codebase. Usually a developer has a pretty
good idea of which bits they want to allow to change. I don't think
people have yet realised/explored the potential these offer.

Cheers,

Richard

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[adrian.freihofer]

On Sat, 2023-11-04 at 11:09 +0000, Richard Purdie wrote:
> On Sat, 2023-11-04 at 11:29 +0100, adrian.freihofer@gmail.com wrote:
> > Hi Alex, hi Richard
> > 
> > After some internal discussions, I would like to clarify my
> > previous
> > answers on this topic.
> > 
> >  * Usually there are two different workflows
> >     - application developers: could use an SDK with a locked
> > sstate-cache.
> >     - Yocto/BSP developers: need an unlocked SDK. They change the
> > recipes.
> >  * A locked SDK
> >     - can work with setscene from SSTATE_MIRRORS
> >     - setscene does caching in the SSTATE_DIR (no issue about that)
> >     - But network problems can occur during the initial build
> > because
> >       bitbake executes many independent setscene tasks. Opening so
> > many
> >       independent connections slows down the build, especially if
> > the
> >       server treats them as a denial of service attack.
> >     - The denial of service problem is difficult to solve because
> > each
> >       setscene task runs in its own bibtake task. Reusing a
> > connection to
> >       download multiple sstate artifacts seems almost impossible.
> >       This is much easier to solve with separate sstate download
> > script.
> 
> FWIW, we did have a similar issue with do_fetch overloading
> servers/proxies/ISPs and added:
> 
> do_fetch[number_threads] = "4"
> 
> Finding the right place to put a thread limit on overall setscene
> tasks
> is harder but in theory possible. Or perhaps a "network capable
> tasks"
> thread limit?
> 
> Is the overload caused by the initial query of sstate presence, or,
> does it happen when the setscene tasks themselves run?

The most extreme situation is probably bitbake --setscene-only with an
empty TMPDIR. Each of the setscene tasks establishes a new connection.
A server receives so many connections that it treats them as a denial
of service attack by throttling. A separate script would allow the same
connection to be reused to download all the required artifacts.
Limiting the number of threads does not really solve the issue because
there are still the same amount of connections which get quickly
opened.

> 
> 
> >  * An unlocked SDK
> >     - Tries to download the sstate cache for changed recipes and
> > their
> >       dependencies, which obviously can't work.
> >     - The useless download requests slow down the build
> > considerably and
> >       cause a high load on the servers without any benefit.
> 
> Is this sstate over http(s) or something else? I seem to remember you
> mentioning sftp. If this were using sftp, it would be horribly slow
> as
> it was designed for a light overhead "does this exist?" check which
> http(s) can manage well.

Yes, we are evaluating sftp. You are right, it is not optimal from a
performance point of view. For example S3 is much faster. A compromise
is to set up a limited number of parallel sftp connections. This has
worked very well so far.

The question of why we use sftp brings us to a larger topic that is
probably relevant for almost all Yocto users, but not for the Yocto
project itself: Security.

There is usually a git server infrastructure that makes it possible to
protect Git repositories with finely graded access policies. As the
sstate-cache contains the same source code, the protection concept for
the Git repositories must also be applied to the sstate-cache
artifacts.

First of all a user authentication is required for the sstate-mirror.
An obvious idea is to use the same user authentication for the sstate-
cache server as for the Git server. In addition to https, ssh is also
often used for git repositories. SSH even offers some advantages in
terms of user-friendliness and security (if a ssh agent is used). This
consideration finally leads us to use the sftp protocol for the sstate
mirror. This is also relatively easy to administer: Simply copy the
user's public ssh keys from the git server to the sftp server.

If one then wants to scale an sstate-cache server for many different
projects and users, one quickly wishes for an option for authorization
at artifact level. Ideally, the access rights to the source code would
be completely transferred to the associated sstate artifacts. For such
an authorization the ssate mirror server would require the SRC_URI
which was used to compile the sstate artifact. With this information,
it could ask the Git server whether or not a user has access to all
source code repositories to grant or deny access to a particular sstate
artifact. It should not be forgotten that the access rights to the Git
repositories can change.

> 
> Recently we've been wondering about teaching the hashequiv server
> about
> "presence", which would then mean the build would only query things
> that stood a good chance of existing.
> 
Yes, that sound very interesting. There are probably even more such
kind of meta data which could be provided by the hashserver to improve
the management of a shared sstate mirror.

Would it make sense to include e.g. the SRC_URI in the hashserv
database and extend the hashserver's API to also provide meta data e.g.
for the authorization of the sstate-mirror? Or is security and
authorization something which should be handled independently from hash
equivalence?

Another topic where additional meta data about the sstate-cache seams
to be beneficial is sstate-mirror retention. Knowing which artifact was
compiled for which tag or commit of the bitbake layer could help to
wipe out some artifacts which are not needed anymore.

> >     - A script which gets a list of sstate artifacts from bitbake
> > and then
> >       does a upfront download works much better
> >        + The script runs only when the user calls it or the SDK
> > gets boot-
> >          strapped
> >        + The script uses a reasonable amount of parallel
> > connections which
> >          are re-used for more then one artifact download
> 
> Explaining to users they need to do X before Y quickly gets tiring,
> both for people explaining it and the people doing it trying to
> remember. I'd really like to get to a point where the system "does
> the
> right thing" if we can.
> 
> I don't believe the problems you describe are insurmountable. If you
> are using sftp, that is going to be a big chunk of the problem as the
> system assumes something faster is available. Yes, I've taken patches
> to make sftp work but it isn't recommended at all. I appreciate there
> would be reasons why you use sftp but if it is possible to get a list
> of "available sstate" via other means, it would improve things.
> 
> >  * Idea for a smart lock/unlock implementation
> >     - Form a user's perspective a locked vs. an unlocked SDK does
> > not make
> >       much sense. It makes more sense if the SDK would
> > automatically
> >       download the sstate-cache if it is expected to be available.
> >       Lets think about an implementation (which allows to override
> > the
> >       logic) to switch from automatic to manual mode:
> >       
> >       SSTATE_MIRRORS_ENABLED ?= "${is_sstate_mirror_available()}"
> 
> What determines this availability? I worry that is something very
> fragile and specific to your use case. It is also not an all or
> nothing
> binary thing.

It would probably be better to query a harserver if an artifact is
present.
> 
> >       In our case the sstate mirror is expected to provide all
> > artifacts
> >       for tagged commits and for some git branches of the layer
> >       repositories.
> >       The sstate is obviousely not usable for a "dirty" git layer
> >       repository.
> 
> That isn't correct and isn't going to work. If I make a single change
> locally, there is a good chance that 99.9% of the sstate could still
> be
> valid in some cases. Forcing the user through 10 hours of rebuild
> when
> potentially that much was available is a really really bad user
> experience.

Maybe there is a better idea.

> 
> >  That's what the is_sstate_mirror_available function
> >       could check to automatically enable and disable lazy
> > downloads.
> >       
> >     - If is_sstate_mirror_available() returns false, it should
> > still be
> >       possible to initiate a sstate-cache download manually.
> >       
> >  * Terminology
> >     - Older Yocto Releases:
> >        + eSDK means an installer which provides a different
> > environment with
> >          different tools
> >        + The eSDK was static, with a locked sstate cache
> >        + Was for one MACHINE, for one image...
> >     - Newer Yocto Releases:
> >        + The bitbake environment offers all features of the eSDK
> > installer. I
> >          consider this as already implemented with meta-ide-support
> > and
> >          build-sysroots.
> 
> Remember bblock and bbunlock too. These provide a way to fix or
> unlock
> specific sections of the codebase. Usually a developer has a pretty
> good idea of which bits they want to allow to change. I don't think
> people have yet realised/explored the potential these offer.
> 

Yes, I also started thinking about the possibilities we would get for
the SDK if there is a hash-server or an even more generic a meta data
server for the sstate-cache in the middle of the infrastructure
picture. it would probably solve some challenges which I could not find
a solution so far.


Thank you for your response.

Adrian


> Cheers,
> 
> Richard

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Sun, 5 Nov 2023 at 20:43, <adrian.freihofer@gmail.com> wrote:
> Another topic where additional meta data about the sstate-cache seams
> to be beneficial is sstate-mirror retention. Knowing which artifact was
> compiled for which tag or commit of the bitbake layer could help to
> wipe out some artifacts which are not needed anymore.

There should be progress on this particular point soon when I get
oe-replicate-build prototype to function. It also records the list of
needed sstate objects into the replica bundle (in the form of
locked-sigs.inc, which is in itself not included into the build conf),
and if you place that bundle next to sstate, and make a superset of
all the objects in all bundles registered that way, you have a list of
things that can be pruned from the cache without loss of build times.
There could also be more interesting use cases, like querying the
sstate server for available builds with guaranteed sstate coverage
etc.

Alex

Re: [Openembedded-architecture] [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Mark Hatle]

On 11/5/23 1:43 PM, Adrian Freihofer wrote:
> On Sat, 2023-11-04 at 11:09 +0000, Richard Purdie wrote:
>> On Sat, 2023-11-04 at 11:29 +0100, adrian.freihofer@gmail.com wrote:
>>> Hi Alex, hi Richard
>>>
>>> After some internal discussions, I would like to clarify my
>>> previous
>>> answers on this topic.
>>>
>>>   * Usually there are two different workflows
>>>      - application developers: could use an SDK with a locked
>>> sstate-cache.
>>>      - Yocto/BSP developers: need an unlocked SDK. They change the
>>> recipes.
>>>   * A locked SDK
>>>      - can work with setscene from SSTATE_MIRRORS
>>>      - setscene does caching in the SSTATE_DIR (no issue about that)
>>>      - But network problems can occur during the initial build
>>> because
>>>        bitbake executes many independent setscene tasks. Opening so
>>> many
>>>        independent connections slows down the build, especially if
>>> the
>>>        server treats them as a denial of service attack.
>>>      - The denial of service problem is difficult to solve because
>>> each
>>>        setscene task runs in its own bibtake task. Reusing a
>>> connection to
>>>        download multiple sstate artifacts seems almost impossible.
>>>        This is much easier to solve with separate sstate download
>>> script.
>>
>> FWIW, we did have a similar issue with do_fetch overloading
>> servers/proxies/ISPs and added:
>>
>> do_fetch[number_threads] = "4"
>>
>> Finding the right place to put a thread limit on overall setscene
>> tasks
>> is harder but in theory possible. Or perhaps a "network capable
>> tasks"
>> thread limit?
>>
>> Is the overload caused by the initial query of sstate presence, or,
>> does it happen when the setscene tasks themselves run?
> 
> The most extreme situation is probably bitbake --setscene-only with an
> empty TMPDIR. Each of the setscene tasks establishes a new connection.
> A server receives so many connections that it treats them as a denial
> of service attack by throttling. A separate script would allow the same
> connection to be reused to download all the required artifacts.
> Limiting the number of threads does not really solve the issue because
> there are still the same amount of connections which get quickly
> opened.
> 
>>
>>
>>>   * An unlocked SDK
>>>      - Tries to download the sstate cache for changed recipes and
>>> their
>>>        dependencies, which obviously can't work.
>>>      - The useless download requests slow down the build
>>> considerably and
>>>        cause a high load on the servers without any benefit.
>>
>> Is this sstate over http(s) or something else? I seem to remember you
>> mentioning sftp. If this were using sftp, it would be horribly slow
>> as
>> it was designed for a light overhead "does this exist?" check which
>> http(s) can manage well.
> 
> Yes, we are evaluating sftp. You are right, it is not optimal from a
> performance point of view. For example S3 is much faster. A compromise
> is to set up a limited number of parallel sftp connections. This has
> worked very well so far.
> 
> The question of why we use sftp brings us to a larger topic that is
> probably relevant for almost all Yocto users, but not for the Yocto
> project itself: Security.
> 
> There is usually a git server infrastructure that makes it possible to
> protect Git repositories with finely graded access policies. As the
> sstate-cache contains the same source code, the protection concept for
> the Git repositories must also be applied to the sstate-cache
> artifacts.
> 
> First of all a user authentication is required for the sstate-mirror.
> An obvious idea is to use the same user authentication for the sstate-
> cache server as for the Git server. In addition to https, ssh is also
> often used for git repositories. SSH even offers some advantages in
> terms of user-friendliness and security (if a ssh agent is used). This
> consideration finally leads us to use the sftp protocol for the sstate
> mirror. This is also relatively easy to administer: Simply copy the
> user's public ssh keys from the git server to the sftp server.

While being able to support ssh (or a related protocol) is useful, you need to 
also remember that MANY MANY organizations absolutely block SSH access through 
their firewalls.  So _requiring_ sftp would be bad.  Allowing it's usage would 
be good.

As for logging in, https is transport 'security' but not authentication without 
additional helpers.  I think it's absolutely reasonable to say https access 
either needs an external helper for authentication purposes or it's 
un-authenticated.  If you want (internal to the company) then ssh/sftp or 
similar using the ssh-agent (or similar) should be the suggested approach.

We don't want to exclude anyone, but we want to be clear on the limitations 
based on an organization's specific choice.

> If one then wants to scale an sstate-cache server for many different
> projects and users, one quickly wishes for an option for authorization
> at artifact level. Ideally, the access rights to the source code would
> be completely transferred to the associated sstate artifacts. For such
> an authorization the ssate mirror server would require the SRC_URI
> which was used to compile the sstate artifact. With this information,
> it could ask the Git server whether or not a user has access to all
> source code repositories to grant or deny access to a particular sstate
> artifact. It should not be forgotten that the access rights to the Git
> repositories can change.

In my experience you do not use _one_ sstate-cache for multiple projects (at an 
organization level), each project is responsible for it's own cache.  This 
prevents even the possibility that one project could use code not intended for it.

 From a more generic Yocto Project perspective, this means you really want to 
use a hierarchy of sstate-caches.  (Maybe not a true hierarchy.). I.e. I use YP, 
so I get the YP sstate-cache for the base functionality.  I use 
meta-openembedded, so I want the meta-openembedded cache... project A, I want 
the project's cache, OE and YP caches as well..  project B, I want that projects 
cache, OE and YP caches.  Project C?  I might want it's cache, Project A, 
Project B, and OE and YP.   You can see this gets complicated quickly.

If this either isn't inteded or a good idea, then alternatives need to be 
provided for this.  Everyone always ends up with an upstream provider (or 
providers) be it YP, OE, OSVs, ISVs, local company resources, etc.  How do we 
manage this and keep it aligned?

Bring in hash equivalency and PR service and things get complicated.  The 
sstate-cache itself is NOT separable from those services.  There are ways to 
decouple them, but they can be 'extreme'.  I.e. turn off hash-equivalency, no 
need for a hash-equivalency service.   Don't cache the do_package_write* files, 
no PR service....  (but even that isn't fool proof due to git AUTOINC... so you 
end up seeding the AUTOINC with static entries or some other method...)

All of these items need to be dealt with and documented together.  My PERSONAL 
preference, (without knowing any specific implementation details) is that the 
contents of hash-equivalency and PR service is somehow stored with the sstate-cache.

One possible way this could be done..  System starts up, determines it needs 
something it doesn't have, then goes out and checks if an updated index is 
present.  If it is, downloads it adds to it's hash equivalency server.  If no 
index present, it can then look for the file lets say "sstate:....link".  If 
that comes back, we know we have an equivalency, it's downloaded added to the 
local database and then the pointed to file is retrieved..  (.siginfo and 
.tar.xz or whatever).  This would ensure that the index is an optimization, but 
not a requirement and would allow a "live" sstate-cache while losing some 
performance.  (This doesn't negate any of the comments about rights or 
possibility to DoS a server via too many connections!)

Still have to solve the PR service problem, but this could get 'seeded' via the 
associated do_write_package siginfo or similar..  and for the AUTOINC, seed it 
from the siginfo file for a given hash?

Doing something like the above could then allow the order specificed in the 
SSTATE_MIRRORS to be used to truely indicate the order things are resolved and 
loaded.

>>
>> Recently we've been wondering about teaching the hashequiv server
>> about
>> "presence", which would then mean the build would only query things
>> that stood a good chance of existing.
>>
> Yes, that sound very interesting. There are probably even more such
> kind of meta data which could be provided by the hashserver to improve
> the management of a shared sstate mirror.
> 
> Would it make sense to include e.g. the SRC_URI in the hashserv
> database and extend the hashserver's API to also provide meta data e.g.
> for the authorization of the sstate-mirror? Or is security and
> authorization something which should be handled independently from hash
> equivalence?

The more I've thought about this, any sort of query directly to a remote 
hashservice seems more and more problematic..  Local hash database, absolutely 
needed as an optimization.

There is a second problem.  My org for instant, it's easy for me to request 
https server where I can serve files to the public.  But asking for our IT to 
support a hash equivalency (and pr) server?  This will likely take months of 
negotiation, possible security review, mitigation process, etc etc etc.. and no 
guaranty that it will actually get approved.  I expect other people will be in a 
similar situation.

> Another topic where additional meta data about the sstate-cache seams
> to be beneficial is sstate-mirror retention. Knowing which artifact was
> compiled for which tag or commit of the bitbake layer could help to
> wipe out some artifacts which are not needed anymore.
> 
>>>      - A script which gets a list of sstate artifacts from bitbake
>>> and then
>>>        does a upfront download works much better
>>>         + The script runs only when the user calls it or the SDK
>>> gets boot-
>>>           strapped
>>>         + The script uses a reasonable amount of parallel
>>> connections which
>>>           are re-used for more then one artifact download
>>
>> Explaining to users they need to do X before Y quickly gets tiring,
>> both for people explaining it and the people doing it trying to
>> remember. I'd really like to get to a point where the system "does
>> the
>> right thing" if we can.
>>
>> I don't believe the problems you describe are insurmountable. If you
>> are using sftp, that is going to be a big chunk of the problem as the
>> system assumes something faster is available. Yes, I've taken patches
>> to make sftp work but it isn't recommended at all. I appreciate there
>> would be reasons why you use sftp but if it is possible to get a list
>> of "available sstate" via other means, it would improve things.
>>
>>>   * Idea for a smart lock/unlock implementation
>>>      - Form a user's perspective a locked vs. an unlocked SDK does
>>> not make
>>>        much sense. It makes more sense if the SDK would
>>> automatically
>>>        download the sstate-cache if it is expected to be available.
>>>        Lets think about an implementation (which allows to override
>>> the
>>>        logic) to switch from automatic to manual mode:
>>>        
>>>        SSTATE_MIRRORS_ENABLED ?= "${is_sstate_mirror_available()}"
>>
>> What determines this availability? I worry that is something very
>> fragile and specific to your use case. It is also not an all or
>> nothing
>> binary thing.
> 
> It would probably be better to query a harserver if an artifact is
> present.
>>
>>>        In our case the sstate mirror is expected to provide all
>>> artifacts
>>>        for tagged commits and for some git branches of the layer
>>>        repositories.
>>>        The sstate is obviousely not usable for a "dirty" git layer
>>>        repository.
>>
>> That isn't correct and isn't going to work. If I make a single change
>> locally, there is a good chance that 99.9% of the sstate could still
>> be
>> valid in some cases. Forcing the user through 10 hours of rebuild
>> when
>> potentially that much was available is a really really bad user
>> experience.
> 
> Maybe there is a better idea.
> 
>>
>>>   That's what the is_sstate_mirror_available function
>>>        could check to automatically enable and disable lazy
>>> downloads.
>>>        
>>>      - If is_sstate_mirror_available() returns false, it should
>>> still be
>>>        possible to initiate a sstate-cache download manually.
>>>        
>>>   * Terminology
>>>      - Older Yocto Releases:
>>>         + eSDK means an installer which provides a different
>>> environment with
>>>           different tools
>>>         + The eSDK was static, with a locked sstate cache
>>>         + Was for one MACHINE, for one image...
>>>      - Newer Yocto Releases:
>>>         + The bitbake environment offers all features of the eSDK
>>> installer. I
>>>           consider this as already implemented with meta-ide-support
>>> and
>>>           build-sysroots.
>>
>> Remember bblock and bbunlock too. These provide a way to fix or
>> unlock
>> specific sections of the codebase. Usually a developer has a pretty
>> good idea of which bits they want to allow to change. I don't think
>> people have yet realised/explored the potential these offer.
>>
> 
> Yes, I also started thinking about the possibilities we would get for
> the SDK if there is a hash-server or an even more generic a meta data
> server for the sstate-cache in the middle of the infrastructure
> picture. it would probably solve some challenges which I could not find
> a solution so far.

Using the standard download model/approach we already have a "generic" metadata 
server approach (and standard download URI supported by bitbake).  The 
specialized approaches (prserver/hashserver) are where we run into issues 
because it's no longer "generic" and well understood by others.  Need to figure 
out a way for this all to work and allow the most "reasonable" re-use we can.

--Mark

> 
> Thank you for your response.
> 
> Adrian
> 
> 
>> Cheers,
>>
>> Richard
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#1830): https://lists.openembedded.org/g/openembedded-architecture/message/1830
> Mute This Topic: https://lists.openembedded.org/mt/102320110/3616948
> Group Owner: openembedded-architecture+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub [mark.hatle@kernel.crashing.org]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 14 Sept 2023 at 14:56, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> There are design elements to this work. We need to work out how we can
> make eSDK and "normal" builds more similar and less of an overhead to
> switch between one and the other. A "bblock all" command does partly
> get you to an eSDK effectively so part of this may be switching eSDK to
> use the new lock command. What other differences are there? What other
> differences are necessary or make sense for the use cases eSDK was
> designed for? How would you turn an existing build into an eSDK like
> one? Could you provide a copy of a local build to someone else easily
> using something like eSDK's tooling? What does the eSDK look like at
> the end of this. One section we don't have good answers to yet is setup
> and configuration although I know you've started on some of that.

So I see the following differences between esdk and normal modes:

1. Environment and tooling availability.

a) esdk sets a number of variables from its initialization script that
aid with cross-compiling components directly (e.g. the core use case
of SDKs). Normal mode doesn't do that, but recently added
meta-ide-support will generate a similar initialization script that
will set up the same environment from the normal mode. There are tests
and documentation for it.

b) PATH. eSDK has a number of items in PATH that point to various
locations inside tmp/sysroots/, collectively they provide the
cross-toolchain.

eSDK also puts a selection of yocto tools into path - wic, devtool but
not bitbake:

============================
alex@Zen2:~/poky_sdk$ ls -l sysroots/x86_64-pokysdk-linux/usr/bin/
total 48
lrwxrwxrwx 1 alex alex  39 Oct 30 12:52 devtool ->
../../../../layers/poky/scripts/devtool
lrwxrwxrwx 1 alex alex  54 Oct 30 12:52 oe-find-native-sysroot ->
../../../../layers/poky/scripts/oe-find-native-sysroot
lrwxrwxrwx 1 alex alex  42 Oct 30 12:52 recipetool ->
../../../../layers/poky/scripts/recipetool
lrwxrwxrwx 1 alex alex  39 Oct 30 12:52 runqemu ->
../../../../layers/poky/scripts/runqemu
lrwxrwxrwx 1 alex alex  55 Oct 30 12:52 runqemu-addptable2image ->
../../../../layers/poky/scripts/runqemu-addptable2image
lrwxrwxrwx 1 alex alex  53 Oct 30 12:52 runqemu-export-rootfs ->
../../../../layers/poky/scripts/runqemu-export-rootfs
lrwxrwxrwx 1 alex alex  51 Oct 30 12:52 runqemu-extract-sdk ->
../../../../layers/poky/scripts/runqemu-extract-sdk
lrwxrwxrwx 1 alex alex  51 Oct 30 12:52 runqemu-gen-tapdevs ->
../../../../layers/poky/scripts/runqemu-gen-tapdevs
lrwxrwxrwx 1 alex alex  46 Oct 30 12:52 runqemu-ifdown ->
../../../../layers/poky/scripts/runqemu-ifdown
lrwxrwxrwx 1 alex alex  44 Oct 30 12:52 runqemu-ifup ->
../../../../layers/poky/scripts/runqemu-ifup
lrwxrwxrwx 1 alex alex 100 Oct 30 12:52 unfsd ->
../../../../tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/recipe-sysroot-native/usr/bin/unfsd
lrwxrwxrwx 1 alex alex  35 Oct 30 12:52 wic ->
../../../../layers/poky/scripts/wic
==============================

'normal mode' puts bitbake/bin/ and oe-core/scripts in PATH.
Cross-toolchain can be added by the same environment script made by
meta-ide-support as mentioned in 1a.

2. Configuration (e.g. local.conf).

eSDK local.conf is local.conf from the normal mode that was used to
produce eSDK, stripped of all comments, and with a bunch of extra
settings:

============================
INHERIT:remove = "buildhistory icecc"
CONNECTIVITY_CHECK_URIS = ""

SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK = "none"

SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"

BB_HASHCONFIG_IGNORE_VARS:append = " SIGGEN_UNLOCKED_RECIPES"

BB_SETSCENE_ENFORCE_IGNORE_TASKS = "%:* *:do_shared_workdir
*:do_rm_work wic-tools:* *:do_addto_recipe_sysroot"

BUILDCFG_HEADER = ""

METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5"

# Provide a flag to indicate we are in the EXT_SDK Context
WITHIN_EXT_SDK = "1"

SSTATE_MIRRORS += " file://universal/(.*) file://universal-4.9/\1
file://universal-4.9/(.*) file://universal-4.8/\1"

require conf/locked-sigs.inc
require conf/unlocked-sigs.inc
===============================

Devtool also has a special configuration:
=================
alex@Zen2:~/poky_sdk$ cat conf/devtool.conf
[General]
bitbake_subdir = layers/poky/bitbake
init_path = layers/poky/oe-init-build-env
core_meta_subdir = layers/poky/meta

[SDK]
sdk_targets = core-image-minimal
==================
There is currently no tooling to add/remove these extras in either
esdk mode or normal mode as far as I understand. Their individual
purposes and effects are also not exactly clear to me, and need to be
investigated one by one.

3. Setting up a normal mode in a eSDK installation.

This is actually pretty easy: rather than sourcing the sdk environment
script, source the poky/oe-init-build-env:
==============
alex@Zen2:~/poky_sdk$ . layers/poky/oe-init-build-env .

### Shell environment set up for builds. ###
...
===============
Bitbake will then simply run, albeit with all those extra
esdk-specific items in local.conf (which can be easily taken out to
achieve a pristine normal mode). There will be some warnings but
builds will proceed:
===============
WARNING: You are using a local hash equivalence server but have
configured an sstate mirror. This will likely mean no sstate will
match from the mirror. You may wish to disable the hash equivalence
use (BB_HASHSERVE), or use a hash equivalence server alongside the
sstate mirror.
...
WARNING: spirv-headers-native-1_1.3.261.1-r0 do_fetch: Failed to fetch
URL git://github.com/KhronosGroup/SPIRV-Headers;protocol=https;branch=main,
attempting MIRRORS if available
WARNING: libepoxy-1.5.10-r0 do_fetch: Failed to fetch URL
git://github.com/anholt/libepoxy;branch=master;protocol=https,
attempting MIRRORS if available
WARNING: spirv-tools-native-1_1.3.261.1-r0 do_fetch: Failed to fetch
URL git://github.com/KhronosGroup/SPIRV-Tools.git;branch=main;protocol=https,
attempting MIRRORS if available
WARNING: shaderc-native-2023.6-r0 do_fetch: Failed to fetch URL
git://github.com/google/shaderc.git;protocol=https;branch=main,
attempting MIRRORS if available
===============
4. Setting up esdk mode in a normal yocto installation.

This is on one hand easy: cross-toolchain and unified sysroots are
made available for direct component builds by meta-ide-support and
build-sysroots.

On the other hand, setting up the exact same environment as eSDK sets
up isn't currently possible: tooling to replicate all the local.conf
tweaks (that includes locked-sigs items) would be necessary, at a
minimum. A more pedantic approach would also ensure that
- only the 'allowed' set of tools is in PATH, and not everything from
bitbake/bin/ and scripts/
- environment variables and toolchain sysroots match between what eSDK
sets up, and what meta-ide-support sets up

5. In conclusion: to be honest, right now I'm not sure which direction
should be taken to unify and simplify things. Hopefully the above
sparks some ideas; they might pop into my head later as well :-)

Alex

Re: Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Mon, 2023-10-30 at 14:50 +0100, Alexander Kanavin wrote:
> On Thu, 14 Sept 2023 at 14:56, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > There are design elements to this work. We need to work out how we can
> > make eSDK and "normal" builds more similar and less of an overhead to
> > switch between one and the other. A "bblock all" command does partly
> > get you to an eSDK effectively so part of this may be switching eSDK to
> > use the new lock command. What other differences are there? What other
> > differences are necessary or make sense for the use cases eSDK was
> > designed for? How would you turn an existing build into an eSDK like
> > one? Could you provide a copy of a local build to someone else easily
> > using something like eSDK's tooling? What does the eSDK look like at
> > the end of this. One section we don't have good answers to yet is setup
> > and configuration although I know you've started on some of that.
> 
> So I see the following differences between esdk and normal modes:
> 
> 1. Environment and tooling availability.
> 
> a) esdk sets a number of variables from its initialization script that
> aid with cross-compiling components directly (e.g. the core use case
> of SDKs). Normal mode doesn't do that, but recently added
> meta-ide-support will generate a similar initialization script that
> will set up the same environment from the normal mode. There are tests
> and documentation for it.

In that case, this one is something we can document as how to make the
functionality available in the normal build.

> b) PATH. eSDK has a number of items in PATH that point to various
> locations inside tmp/sysroots/, collectively they provide the
> cross-toolchain.
> 
> eSDK also puts a selection of yocto tools into path - wic, devtool but
> not bitbake:
> 
> ============================
> alex@Zen2:~/poky_sdk$ ls -l sysroots/x86_64-pokysdk-linux/usr/bin/
> total 48
> lrwxrwxrwx 1 alex alex  39 Oct 30 12:52 devtool ->
> ../../../../layers/poky/scripts/devtool
> lrwxrwxrwx 1 alex alex  54 Oct 30 12:52 oe-find-native-sysroot ->
> ../../../../layers/poky/scripts/oe-find-native-sysroot
> lrwxrwxrwx 1 alex alex  42 Oct 30 12:52 recipetool ->
> ../../../../layers/poky/scripts/recipetool
> lrwxrwxrwx 1 alex alex  39 Oct 30 12:52 runqemu ->
> ../../../../layers/poky/scripts/runqemu
> lrwxrwxrwx 1 alex alex  55 Oct 30 12:52 runqemu-addptable2image ->
> ../../../../layers/poky/scripts/runqemu-addptable2image
> lrwxrwxrwx 1 alex alex  53 Oct 30 12:52 runqemu-export-rootfs ->
> ../../../../layers/poky/scripts/runqemu-export-rootfs
> lrwxrwxrwx 1 alex alex  51 Oct 30 12:52 runqemu-extract-sdk ->
> ../../../../layers/poky/scripts/runqemu-extract-sdk
> lrwxrwxrwx 1 alex alex  51 Oct 30 12:52 runqemu-gen-tapdevs ->
> ../../../../layers/poky/scripts/runqemu-gen-tapdevs
> lrwxrwxrwx 1 alex alex  46 Oct 30 12:52 runqemu-ifdown ->
> ../../../../layers/poky/scripts/runqemu-ifdown
> lrwxrwxrwx 1 alex alex  44 Oct 30 12:52 runqemu-ifup ->
> ../../../../layers/poky/scripts/runqemu-ifup
> lrwxrwxrwx 1 alex alex 100 Oct 30 12:52 unfsd ->
> ../../../../tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/recipe-sysroot-native/usr/bin/unfsd
> lrwxrwxrwx 1 alex alex  35 Oct 30 12:52 wic ->
> ../../../../layers/poky/scripts/wic
> ==============================
> 
> 'normal mode' puts bitbake/bin/ and oe-core/scripts in PATH.
> Cross-toolchain can be added by the same environment script made by
> meta-ide-support as mentioned in 1a.

Right, so in theory we can change PATH and change this which can also
easily be documented.

> 2. Configuration (e.g. local.conf).
> 
> eSDK local.conf is local.conf from the normal mode that was used to
> produce eSDK, stripped of all comments, and with a bunch of extra
> settings:
> 
> ============================
> INHERIT:remove = "buildhistory icecc"
> CONNECTIVITY_CHECK_URIS = ""
> 
> SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK = "none"
> 
> SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"
> 
> BB_HASHCONFIG_IGNORE_VARS:append = " SIGGEN_UNLOCKED_RECIPES"
> 
> BB_SETSCENE_ENFORCE_IGNORE_TASKS = "%:* *:do_shared_workdir
> *:do_rm_work wic-tools:* *:do_addto_recipe_sysroot"
> 
> BUILDCFG_HEADER = ""
> 
> METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5"
> 
> # Provide a flag to indicate we are in the EXT_SDK Context
> WITHIN_EXT_SDK = "1"
> 
> SSTATE_MIRRORS += " file://universal/(.*) file://universal-4.9/\1
> file://universal-4.9/(.*) file://universal-4.8/\1"
> 

Perhaps some of this should become a generic include file which is
included which would then make it easy to document as how to lock
things down to match it? That would take the config out of the class
file too which is probably nicer.

> require conf/locked-sigs.inc
> require conf/unlocked-sigs.inc
> ===============================
> 
> Devtool also has a special configuration:
> =================
> alex@Zen2:~/poky_sdk$ cat conf/devtool.conf
> [General]
> bitbake_subdir = layers/poky/bitbake
> init_path = layers/poky/oe-init-build-env
> core_meta_subdir = layers/poky/meta


These are likely there to allow devtool to work with the modified files
layout.

We should probably document it but I'm going to propose we "ignore"
layout for the purposes of this so we can just ignore things which are
for layout only.

> [SDK]
> sdk_targets = core-image-minimal

Not quite sure about this one. Maybe something to do with the target
used to generate the eSDK? Not sure what uses it.


> ==================
> There is currently no tooling to add/remove these extras in either
> esdk mode or normal mode as far as I understand. Their individual
> purposes and effects are also not exactly clear to me, and need to be
> investigated one by one.
> 
> 3. Setting up a normal mode in a eSDK installation.
> 
> This is actually pretty easy: rather than sourcing the sdk environment
> script, source the poky/oe-init-build-env:
> ==============
> alex@Zen2:~/poky_sdk$ . layers/poky/oe-init-build-env .
> 
> ### Shell environment set up for builds. ###
> ...
> ===============
> Bitbake will then simply run, albeit with all those extra
> esdk-specific items in local.conf (which can be easily taken out to
> achieve a pristine normal mode). There will be some warnings but
> builds will proceed:
> ===============
> WARNING: You are using a local hash equivalence server but have
> configured an sstate mirror. This will likely mean no sstate will
> match from the mirror. You may wish to disable the hash equivalence
> use (BB_HASHSERVE), or use a hash equivalence server alongside the
> sstate mirror.
> ...
> WARNING: spirv-headers-native-1_1.3.261.1-r0 do_fetch: Failed to fetch
> URL git://github.com/KhronosGroup/SPIRV-Headers;protocol=https;branch=main,
> attempting MIRRORS if available
> WARNING: libepoxy-1.5.10-r0 do_fetch: Failed to fetch URL
> git://github.com/anholt/libepoxy;branch=master;protocol=https,
> attempting MIRRORS if available
> WARNING: spirv-tools-native-1_1.3.261.1-r0 do_fetch: Failed to fetch
> URL git://github.com/KhronosGroup/SPIRV-Tools.git;branch=main;protocol=https,
> attempting MIRRORS if available
> WARNING: shaderc-native-2023.6-r0 do_fetch: Failed to fetch URL
> git://github.com/google/shaderc.git;protocol=https;branch=main,
> attempting MIRRORS if available
> ===============
> 4. Setting up esdk mode in a normal yocto installation.
> 
> This is on one hand easy: cross-toolchain and unified sysroots are
> made available for direct component builds by meta-ide-support and
> build-sysroots.
> 
> On the other hand, setting up the exact same environment as eSDK sets
> up isn't currently possible: tooling to replicate all the local.conf
> tweaks (that includes locked-sigs items) would be necessary, at a
> minimum. A more pedantic approach would also ensure that
> - only the 'allowed' set of tools is in PATH, and not everything from
> bitbake/bin/ and scripts/
> - environment variables and toolchain sysroots match between what eSDK
> sets up, and what meta-ide-support sets up
> 
> 5. In conclusion: to be honest, right now I'm not sure which direction
> should be taken to unify and simplify things. Hopefully the above
> sparks some ideas; they might pop into my head later as well :-)

The biggest 'gap' appears to be the config to go along with the locked
sigs. If we move that to an include, that probably gets us quite close
to them being similar ignoring layout changes for the files and the
differences to the environment?

Thinking out loud, I guess we could have a "filtered" scripts directory
too as part of the normal build, then the eSDK switches PATH from one
to the other?

Cheers,

Richard

Re: Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Mon, 30 Oct 2023 at 15:07, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> > a) esdk sets a number of variables from its initialization script that
> > aid with cross-compiling components directly (e.g. the core use case
> > of SDKs). Normal mode doesn't do that, but recently added
> > meta-ide-support will generate a similar initialization script that
> > will set up the same environment from the normal mode. There are tests
> > and documentation for it.
>
> In that case, this one is something we can document as how to make the
> functionality available in the normal build.

This has been documented in the sdk manual:
https://docs.yoctoproject.org/sdk-manual/extensible.html#two-ways-to-install-the-extensible-sdk
https://docs.yoctoproject.org/sdk-manual/extensible.html#installing-additional-items-into-the-extensible-sdk

Actually, I have on purpose presented both options in the manual as
'esdk', the standalone installer option is simply locked down more
(task signatures and tooling wise) than the direct yocto build.

So here's what could be done:

- esdk tools become symlinks in poky/scripts/esdk-tools/. esdk
environment script puts that in PATH, rather than some custom
esdk-specific location (the code to generate that can then be
dropped).

- esdk tweaks to local.conf move into a dedicated include file, which
can be static and under version control, except for perhaps
METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5" -
I need to check why is it there and how that is used.

Something else perhaps?

Alex

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Mon, 30 Oct 2023 at 16:02, Alexander Kanavin via
lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org>
wrote:
> So here's what could be done:
>
> - esdk tools become symlinks in poky/scripts/esdk-tools/. esdk
> environment script puts that in PATH, rather than some custom
> esdk-specific location (the code to generate that can then be
> dropped).

This is now implemented (needs to be tested on AB).

> - esdk tweaks to local.conf move into a dedicated include file, which
> can be static and under version control, except for perhaps
> METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5" -
> I need to check why is it there and how that is used.

It's actually more complicated. The code to generate esdk-specific
local.conf with all the tweaks has too much dynamic stuff in it which
is subject to what various variables are set to. So I'm thinking of
extracting that to a dedicated function, then attaching a bitbake task
to that function.

Then we can pull all of it together into 'devtool esdk <image>'
command (or similar), which would enter the esdk environment directly
via:
- running 'bitbake <image> meta-ide-support'
- running the above mentioned bitbake local.conf task to generate the
esdk-specific local.conf
- sourcing the environment script produced by meta-ide-support
- rewriting PATH to provide only the curated esdk tools and not
everything plus bitbake.
- writing a custom devtool.conf similar to that of standalone esdk so
that devtool can find bitbake and bitbake can use the esdk-specific
local.conf

And it would be tested in the same way standalone esdks are.

Thoughts? Anything missing from the above list?

Alex

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Tue, 2023-10-31 at 13:08 +0100, Alexander Kanavin wrote:
> On Mon, 30 Oct 2023 at 16:02, Alexander Kanavin via
> lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org>
> wrote:
> > So here's what could be done:
> > 
> > - esdk tools become symlinks in poky/scripts/esdk-tools/. esdk
> > environment script puts that in PATH, rather than some custom
> > esdk-specific location (the code to generate that can then be
> > dropped).
> 
> This is now implemented (needs to be tested on AB).
> 
> > - esdk tweaks to local.conf move into a dedicated include file, which
> > can be static and under version control, except for perhaps
> > METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5" -
> > I need to check why is it there and how that is used.
> 
> It's actually more complicated. The code to generate esdk-specific
> local.conf with all the tweaks has too much dynamic stuff in it which
> is subject to what various variables are set to. So I'm thinking of
> extracting that to a dedicated function, then attaching a bitbake task
> to that function.
> 
> Then we can pull all of it together into 'devtool esdk <image>'
> command (or similar), which would enter the esdk environment directly
> via:
> - running 'bitbake <image> meta-ide-support'
> - running the above mentioned bitbake local.conf task to generate the
> esdk-specific local.conf
> - sourcing the environment script produced by meta-ide-support
> - rewriting PATH to provide only the curated esdk tools and not
> everything plus bitbake.
> - writing a custom devtool.conf similar to that of standalone esdk so
> that devtool can find bitbake and bitbake can use the esdk-specific
> local.conf
> 
> And it would be tested in the same way standalone esdks are.
> 
> Thoughts? Anything missing from the above list?

That sounds like a good way to handle this to me!

Cheers,

Richard

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Tue, 31 Oct 2023 at 13:28, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:

> > Then we can pull all of it together into 'devtool esdk <image>'
> > command (or similar), which would enter the esdk environment directly
> > via:
> > - running 'bitbake <image> meta-ide-support'
> > - running the above mentioned bitbake local.conf task to generate the
> > esdk-specific local.conf
> > - sourcing the environment script produced by meta-ide-support
> > - rewriting PATH to provide only the curated esdk tools and not
> > everything plus bitbake.
> > - writing a custom devtool.conf similar to that of standalone esdk so
> > that devtool can find bitbake and bitbake can use the esdk-specific
> > local.conf
> >
> > And it would be tested in the same way standalone esdks are.
> >
> > Thoughts? Anything missing from the above list?
>
> That sounds like a good way to handle this to me!

A couple followup points:

- copy_buildsystem() in populate_sdk_ext class is overly long at 400
lines and does many different barely related things, writing
local.conf for esdk one of them:

https://git.yoctoproject.org/poky/tree/meta/classes-recipe/populate_sdk_ext.bbclass#n189

The first step would be to structure it into separate functions each
doing one thing and hopefully fitting on a single screen. Then these
functions can be hand-picked into a task designed to provide esdk
things in a yocto build context, and the code would simply be more
readable, as right now I can barely understand all the various things
that function does, and the spaghetti of local variables etc.

- the tool that would set up the esdk environment in a plain yocto
build could be a shell script in scripts/oe-init-esdk-env perhaps,
similar to oe-init-build-env. No need to make it a devtool plugin; it
could use python helpers to program the more tricky bits such as PATH
manipulations etc. So one would first initialize the yocto
environment, then transition to an esdk environment from that (there
would be no way to go back, as one can simply start a new session).

Alex

Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[adrian.freihofer]

Hi Richard, hi Alex

On Mon, 2023-10-30 at 14:07 +0000, Richard Purdie wrote:
> On Mon, 2023-10-30 at 14:50 +0100, Alexander Kanavin wrote:
> > On Thu, 14 Sept 2023 at 14:56, Richard Purdie
> > <richard.purdie@linuxfoundation.org> wrote:
> > > There are design elements to this work. We need to work out how
> > > we can
> > > make eSDK and "normal" builds more similar and less of an
> > > overhead to
> > > switch between one and the other. A "bblock all" command does
> > > partly
> > > get you to an eSDK effectively so part of this may be switching
> > > eSDK to
> > > use the new lock command. What other differences are there? What
> > > other
> > > differences are necessary or make sense for the use cases eSDK
> > > was
> > > designed for? How would you turn an existing build into an eSDK
> > > like
> > > one? Could you provide a copy of a local build to someone else
> > > easily
> > > using something like eSDK's tooling? What does the eSDK look like
> > > at
> > > the end of this. One section we don't have good answers to yet is
> > > setup
> > > and configuration although I know you've started on some of that.
> > 
> > So I see the following differences between esdk and normal modes:
> > 
> > 1. Environment and tooling availability.
> > 
> > a) esdk sets a number of variables from its initialization script
> > that
> > aid with cross-compiling components directly (e.g. the core use
> > case
> > of SDKs). Normal mode doesn't do that, but recently added
> > meta-ide-support will generate a similar initialization script that
> > will set up the same environment from the normal mode. There are
> > tests
> > and documentation for it.
> 
> In that case, this one is something we can document as how to make
> the
> functionality available in the normal build.
> 
> > b) PATH. eSDK has a number of items in PATH that point to various
> > locations inside tmp/sysroots/, collectively they provide the
> > cross-toolchain.
> > 
> > eSDK also puts a selection of yocto tools into path - wic, devtool
> > but
> > not bitbake:
> > 
> > ============================
> > alex@Zen2:~/poky_sdk$ ls -l sysroots/x86_64-pokysdk-linux/usr/bin/
> > total 48
> > lrwxrwxrwx 1 alex alex  39 Oct 30 12:52 devtool ->
> > ../../../../layers/poky/scripts/devtool
> > lrwxrwxrwx 1 alex alex  54 Oct 30 12:52 oe-find-native-sysroot ->
> > ../../../../layers/poky/scripts/oe-find-native-sysroot
> > lrwxrwxrwx 1 alex alex  42 Oct 30 12:52 recipetool ->
> > ../../../../layers/poky/scripts/recipetool
> > lrwxrwxrwx 1 alex alex  39 Oct 30 12:52 runqemu ->
> > ../../../../layers/poky/scripts/runqemu
> > lrwxrwxrwx 1 alex alex  55 Oct 30 12:52 runqemu-addptable2image ->
> > ../../../../layers/poky/scripts/runqemu-addptable2image
> > lrwxrwxrwx 1 alex alex  53 Oct 30 12:52 runqemu-export-rootfs ->
> > ../../../../layers/poky/scripts/runqemu-export-rootfs
> > lrwxrwxrwx 1 alex alex  51 Oct 30 12:52 runqemu-extract-sdk ->
> > ../../../../layers/poky/scripts/runqemu-extract-sdk
> > lrwxrwxrwx 1 alex alex  51 Oct 30 12:52 runqemu-gen-tapdevs ->
> > ../../../../layers/poky/scripts/runqemu-gen-tapdevs
> > lrwxrwxrwx 1 alex alex  46 Oct 30 12:52 runqemu-ifdown ->
> > ../../../../layers/poky/scripts/runqemu-ifdown
> > lrwxrwxrwx 1 alex alex  44 Oct 30 12:52 runqemu-ifup ->
> > ../../../../layers/poky/scripts/runqemu-ifup
> > lrwxrwxrwx 1 alex alex 100 Oct 30 12:52 unfsd ->
> > ../../../../tmp/work/qemuarm64-poky-linux/core-image-
> > minimal/1.0/recipe-sysroot-native/usr/bin/unfsd
> > lrwxrwxrwx 1 alex alex  35 Oct 30 12:52 wic ->
> > ../../../../layers/poky/scripts/wic
> > ==============================
> > 
> > 'normal mode' puts bitbake/bin/ and oe-core/scripts in PATH.
> > Cross-toolchain can be added by the same environment script made by
> > meta-ide-support as mentioned in 1a.
> 
> Right, so in theory we can change PATH and change this which can also
> easily be documented.
> 
> > 2. Configuration (e.g. local.conf).

I think these differences between SDK and bitbake environment are no
longer required and they have been problematic. I would try to make the
bitbake environment usable like the eSDK environment was without trying
to replicate all the details of the eSDK installer such as these
local.conf settings.
> > 
> > eSDK local.conf is local.conf from the normal mode that was used to
> > produce eSDK, stripped of all comments, and with a bunch of extra
> > settings:
> > 
> > ============================
> > INHERIT:remove = "buildhistory icecc"
That's not needed it there is a full bitbake environment.

> > CONNECTIVITY_CHECK_URIS = ""
> > 
> > SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK = "none"
> > 
> > SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"
As already mentioned in my previous mail, a locked SDK is not a must if
the full bitbake environment with shared sstate-cache is available.
Locking might be added later as an optional feature.

> > 
> > BB_HASHCONFIG_IGNORE_VARS:append = " SIGGEN_UNLOCKED_RECIPES"
> > 
> > BB_SETSCENE_ENFORCE_IGNORE_TASKS = "%:* *:do_shared_workdir
> > *:do_rm_work wic-tools:* *:do_addto_recipe_sysroot"
> > 
> > BUILDCFG_HEADER = ""
> > 
> > METADATA_REVISION:poky = "4a1e0b9625729e422fcf24e632ee2a3c79f986d5"
> > 
> > # Provide a flag to indicate we are in the EXT_SDK Context
> > WITHIN_EXT_SDK = "1"
> > 
> > SSTATE_MIRRORS += " file://universal/(.*) file://universal-4.9/\1
> > file://universal-4.9/(.*) file://universal-4.8/\1"
> > 
As already mentioned in my previous mail, the SSTATE_MIRRORS does not
work well. Downloading the sstate-cache to SSTATE_DIR works much
better, at least for us.

> 
> Perhaps some of this should become a generic include file which is
> included which would then make it easy to document as how to lock
> things down to match it? That would take the config out of the class
> file too which is probably nicer.
> 
> > require conf/locked-sigs.inc
> > require conf/unlocked-sigs.inc
> > ===============================
> > 
> > Devtool also has a special configuration:
> > =================
> > alex@Zen2:~/poky_sdk$ cat conf/devtool.conf
> > [General]
> > bitbake_subdir = layers/poky/bitbake
> > init_path = layers/poky/oe-init-build-env
> > core_meta_subdir = layers/poky/meta
> 
The different folder structures of the eSDK and the bitbake environment
cause many issues in the past. We should get rid of that and keep the
layers structure form the bitbake environment. But that's already the
case if the layers are replicated with the bitbake-layers tools from
Alex.

> 
> These are likely there to allow devtool to work with the modified
> files
> layout.
> 
> We should probably document it but I'm going to propose we "ignore"
> layout for the purposes of this so we can just ignore things which
> are
> for layout only.
> 
> > [SDK]
> > sdk_targets = core-image-minimal
> 
> Not quite sure about this one. Maybe something to do with the target
> used to generate the eSDK? Not sure what uses it.

In the eSDK you can do "devtool build" without specifying the image
which should be used. That's probably then taken from this config line.
But also this seams to be pointless in a full bitbake environment.
> 
> 
> > ==================
> > There is currently no tooling to add/remove these extras in either
> > esdk mode or normal mode as far as I understand. Their individual
> > purposes and effects are also not exactly clear to me, and need to
> > be
> > investigated one by one.
> > 
> > 3. Setting up a normal mode in a eSDK installation.
> > 
> > This is actually pretty easy: rather than sourcing the sdk
> > environment
> > script, source the poky/oe-init-build-env:
> > ==============
> > alex@Zen2:~/poky_sdk$ . layers/poky/oe-init-build-env .
> > 
> > ### Shell environment set up for builds. ###
> > ...
> > ===============
> > Bitbake will then simply run, albeit with all those extra
> > esdk-specific items in local.conf (which can be easily taken out to
> > achieve a pristine normal mode). There will be some warnings but
> > builds will proceed:
> > ===============
> > WARNING: You are using a local hash equivalence server but have
> > configured an sstate mirror. This will likely mean no sstate will
> > match from the mirror. You may wish to disable the hash equivalence
> > use (BB_HASHSERVE), or use a hash equivalence server alongside the
> > sstate mirror.
> > ...
> > WARNING: spirv-headers-native-1_1.3.261.1-r0 do_fetch: Failed to
> > fetch
> > URL git://github.com/KhronosGroup/SPIRV-
> > Headers;protocol=https;branch=main,
> > attempting MIRRORS if available
> > WARNING: libepoxy-1.5.10-r0 do_fetch: Failed to fetch URL
> > git://github.com/anholt/libepoxy;branch=master;protocol=https,
> > attempting MIRRORS if available
> > WARNING: spirv-tools-native-1_1.3.261.1-r0 do_fetch: Failed to
> > fetch
> > URL git://github.com/KhronosGroup/SPIRV-
> > Tools.git;branch=main;protocol=https,
> > attempting MIRRORS if available
> > WARNING: shaderc-native-2023.6-r0 do_fetch: Failed to fetch URL
> > git://github.com/google/shaderc.git;protocol=https;branch=main,
> > attempting MIRRORS if available
> > ===============
> > 4. Setting up esdk mode in a normal yocto installation.

That's exactly what my devtool ide plugin does if it gets called
without a recipe. See
https://patchwork.yoctoproject.org/project/oe-core/patch/20231101110129.647878-9-adrian.freihofer@siemens.com/
and search for *Shared sysroots mode*.

> > 
> > This is on one hand easy: cross-toolchain and unified sysroots are
> > made available for direct component builds by meta-ide-support and
> > build-sysroots.
> > 
> > On the other hand, setting up the exact same environment as eSDK
> > sets
> > up isn't currently possible: tooling to replicate all the
> > local.conf
> > tweaks (that includes locked-sigs items) would be necessary, at a
> > minimum. A more pedantic approach would also ensure that
> > - only the 'allowed' set of tools is in PATH, and not everything
> > from
> > bitbake/bin/ and scripts/
> > - environment variables and toolchain sysroots match between what
> > eSDK
> > sets up, and what meta-ide-support sets up
> > 
> > 5. In conclusion: to be honest, right now I'm not sure which
> > direction
> > should be taken to unify and simplify things. Hopefully the above
> > sparks some ideas; they might pop into my head later as well :-)
> 
> The biggest 'gap' appears to be the config to go along with the
> locked
> sigs. If we move that to an include, that probably gets us quite
> close
> to them being similar ignoring layout changes for the files and the
> differences to the environment?
> 
> Thinking out loud, I guess we could have a "filtered" scripts
> directory
> too as part of the normal build, then the eSDK switches PATH from one
> to the other?

Simply speaking, the new populate_sdk_ext implementation could look
like this:

Create a tar file which contains:
- Output of bitbake-layers create-layers-setup
- Output of bitbake-layers save-build-conf
- An sstate download script which can download the artifacts from a
mirror into SSTATE_DIR.
- Optionally: The populated sstate-cache folder.

Generates a script which:
- includes the tar file as the existing installer
- extracts the tar and runs all the scripts from the tar.

This would allow to initially setup the SDK with an installer. It would
also allow to download a newer installer and install it over and
existing setup. It would basically just checkout different commits of
of the layers and update the sstate-cache. But it would also allow to
update the SDK independently from an installer, by just using git to
change the commits of the layers and re-run the sstate-download script
again. This architecture would also allow to switch for example the
MACHINE or compile another image.

And, please have a look at the devtool ide plugin. It was developed
with all that in mind.

Regards,
Adrian

> 
> Cheers,
> 
> Richard
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#61520):
> https://lists.yoctoproject.org/g/yocto/message/61520
> Mute This Topic: https://lists.yoctoproject.org/mt/101356418/4454582
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe:
> https://lists.yoctoproject.org/g/yocto/leave/8893937/4454582/528657958/xyzzy
>  [adrian.freihofer@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Wed, 1 Nov 2023 at 16:45, <adrian.freihofer@gmail.com> wrote:
> I think these differences between SDK and bitbake environment are no
> longer required and they have been problematic. I would try to make the
> bitbake environment usable like the eSDK environment was without trying
> to replicate all the details of the eSDK installer such as these
> local.conf settings.

I have now split up the populate_sdk_ext into separate functions [1]
for better maintainability, and the more I think about what to do
next, the more I agree with Adrian. I just don't see why (in a
standard yocto build) would we want to manipulate PATH to provide a
restricted set of tools, or to create a "local.conf+extra stuff"
(locked signatures, esdk tweaks) environment, when existing local.conf
by itself is already working fine, and full set of tools is better
than a restricted one. If we want to add or remove locked signatures,
this can be done with 'bitbake -s lockedsigs' or bblock for specific
recipes only. And SDK's cross-toolchain is accessible via
meta-ide-support/build-sysroots flow.

[1] https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/sstate-for-all

THAT SAID. I also agree there should be a way to package the 'standard
yocto build' that does not also do 'weird esdk stuff' on top. Just a
tarball that has 'everything', and that you unpack and get a standard
build exactly like the one that was used to produce the tarball.
Something like what Adrian suggested: basically place all information
needed to replicate a build into one place, then either make a tarball
out of it (with sstate for fast builds), or publish it into the
network. We might be able to reuse some of esdk packaging code for
this, or we might not, but I think this is the direction that should
be taken.

Alex

Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[adrian.freihofer]

On Wed, 2023-11-01 at 18:28 +0100, Alexander Kanavin wrote:
> On Wed, 1 Nov 2023 at 16:45, <adrian.freihofer@gmail.com> wrote:
> > I think these differences between SDK and bitbake environment are
> > no
> > longer required and they have been problematic. I would try to make
> > the
> > bitbake environment usable like the eSDK environment was without
> > trying
> > to replicate all the details of the eSDK installer such as these
> > local.conf settings.
> 
> I have now split up the populate_sdk_ext into separate functions [1]
> for better maintainability, and the more I think about what to do
> next, the more I agree with Adrian. I just don't see why (in a
> standard yocto build) would we want to manipulate PATH to provide a
> restricted set of tools, or to create a "local.conf+extra stuff"
> (locked signatures, esdk tweaks) environment, when existing
> local.conf
> by itself is already working fine, and full set of tools is better
> than a restricted one. If we want to add or remove locked signatures,
> this can be done with 'bitbake -s lockedsigs' or bblock for specific
> recipes only. And SDK's cross-toolchain is accessible via
> meta-ide-support/build-sysroots flow.
> 
> [1]
> https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/sstate-for-all

Splitting allows to copy the new function with the + from the patch
into this email and comment it.

+python copy_buildsystem () {
+ import oe.copy_buildsystem
+
+ baseoutpath = d.getVar('SDK_OUTPUT') + '/' + d.getVar('SDKPATH')
+
+ # Determine if we're building a derivative extensible SDK (from
devtool build-sdk)
+ derivative = (d.getVar('SDK_DERIVATIVE') or '') == '1'

What's the advantage of that? There are now two worlds: The bitbake
world and the SDK world which behave similar but not equal. Both need
to be maintained and tested.

We use only bitbake on our CI/CD infrastructure. For users who have
only the eSDK installed, it's hard to understand what the CI does and
even harder to reproduce errors happening on the CI.

We also had some setups with a containerized eSDK on the CI for the
integration of application source code. But this did not work at all.
The SDK container is basically outdated when the first component of the
firmware gets updated. If other components depend on that component an
SDK container update is required. Handling breaking changes easily is a
big advantage of bitbake which gets lost when using any kind of locked
or badly packaged variant of it.

+
+ conf_initpath, conf_bbpath, core_meta_subdir, sdkbblayers =
copy_bitbake_and_layers(d, baseoutpath, derivative)

Changing the directory layout leads to many pitfalls especially if more
layers than just poky are used. This should be replaced by the new
bitbake-layers tools. This means there is only one official way for
setting up bitbake layers and the folder structure gets exactly
replicated.

+
+ write_devtool_config(d, baseoutpath, conf_bbpath, conf_initpath,
core_meta_subdir)

Not sure there is much left when we have only the bitbake world. But
defining some defaults might be still useful.

+
+ write_unlocked_sigs(d, baseoutpath)

Lets turn this more towards a QA check. As a SDK maintainer I would
like to provide SDKs with 100% sstate included. But as a user, if I
have a choice between waiting a few minutes until bitbake compiled some
missing parts or getting an error message telling me I can't get an SDK
now, I'd probably choose compile.
If I remember correctly, with the old eSDK installer this is even
worse. This error happens during the installation which leads to an SDK
in an undefined state. The user must delete it again and fix the
generation of the SDK installer, which might be a very complicated and
time consuming task with the existing tools.

+
+ write_bblayers_conf(d, baseoutpath, sdkbblayers)

Also something which can be replaced by the new bitbake-layers utility.

+
+ uninative_checksum = copy_uninative(d, baseoutpath)

Not sure if this is still needed. With a bitbake environment this just
happens from sstate, I guess. So why doing it differently for the SDK?

+
+ write_local_conf(d, baseoutpath, derivative, core_meta_subdir,
uninative_checksum)

Also something which can be replaced by the new bitbake-layers utility.

+
+ prepare_locked_cache(d, baseoutpath, conf_initpath)

The sstate could be shipped in three different ways:
 * Included in the installer and just extracted into $SSTATE_DIR. This
   is simple but it does not scale at all. If you maintain multiple
   distros and MACHINES and want to have fast update cycles,
   distributing complete sstate archives quickly becomes practically
   impossible, as the same data is packed into several huge archives.
   That leads to issues on the infrastructure side. But also on the
   user's machine having one sstate folder e.g. ~/sstate-cache instead
   of several $TMPDIR/sstate-cache folders is beneficial.
 * No sstate is included, the sstate gets "lazy" fetched from
   SSTATE_MIRROR. Also that looks easy but does not scale very well for
   the SDK use case. bitbake opens a connection for every setscene task
   which is usually handled as a denial of service attack by modern
   infrastructures.
   Fetching in the setscene tasks does not cache the sstate artifact
   locally (not 100% sure here). So whenever the same setscene tasks
   runs again a new download is needed. This is not efficient and also
   not the quickest possible implementation. Local caching is important
   for the sstate.
 * An sstate-download script which downloads the sstate from the
   SSTATE_MIRROR into SSTATE_DIR before bitbake gets started is working
   very well for us. We share the sstate cache via sftp. This allows us
   to use the public keys which we anyway need for git for the
   authentication and authorization. Since the download script opens
   more than one connection for downloading the speed is also very
   nice. We are still evaluating this. The script is not really
   upstreameable. It's very specific to our infrastructure.
   
Probably a bad idea, or something for later: A download script would
have all the logic for finding the required sstate files on the client.
Not sure if also a server side implementation could be interesting. I'm
thinking about a service with access to the shared sstate of a build
infrastructure. The client could ask for an sstate bundle for a given
recipe and the server would pack it on the fly. Similar to what git
does. Intelligence on the server side would also allow to have a fine
grained authorization concept e.g. per sstate artifact.

+
+ write_manifest(d, baseoutpath)

Not sure if this is still needed.

+
+}


> 
> THAT SAID. I also agree there should be a way to package the
> 'standard
> yocto build' that does not also do 'weird esdk stuff' on top. Just a
> tarball that has 'everything', and that you unpack and get a standard
> build exactly like the one that was used to produce the tarball.
> Something like what Adrian suggested: basically place all information
> needed to replicate a build into one place, then either make a
> tarball
> out of it (with sstate for fast builds), or publish it into the
> network. We might be able to reuse some of esdk packaging code for
> this, or we might not, but I think this is the direction that should
> be taken.

Ideally the "new installer" can be extracted/installed over an existing
SDK=bitbake setup. I think the bitbake-layer tool supports this for the
layers. Extracting a full sstate archive into an already populated
folder should also just work, I assume.

Such a design would allow more advance users to just use git and an
editor to change their SDK. But it would also support less experienced
users with a process where they can just download a minimal installer
which brings the SDK=bitbake into a defined state.

Sorry for repeating some parts which we already had in other emails.
But I tried to summarize the lengthy discussion a bit in one place.

Thank you very much!

Adrian

> 
> Alex

Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 2 Nov 2023 at 09:32, <adrian.freihofer@gmail.com> wrote:

> Sorry for repeating some parts which we already had in other emails.
> But I tried to summarize the lengthy discussion a bit in one place.

So here's what I'd like to try:

- write a new populate_build_replica task that writes a few things
under ${WORKDIR}/replica
-- setup-layers json and script
(another option is to copy the layer trees themselves like esdk does,
which is left for maybe later. Completely offline replication is not
an initial goal, and it's good to get to a minimally viable
implementation asap.)
-- meta-build-config layer with the local.conf/bblayer.conf template.
-- sstate cache needed to fulfil the bitbake target that the task is
for (this would reuse code from esdk that does the same as much as
possible)

All of this is then packaged into a self-extracting shell archive that:
- unpacks itself
- fetches layers using the json/script from the unpacked tree
- sets up a build directory using the template from meta-build-config
in the unpacked tree
- tweaks site.conf to point to the prepackaged sstate

And voila! (in theory) This should be the same build as the one that
was produced elsewhere.

Alex

Re: [yocto] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 2 Nov 2023 at 10:02, Alexander Kanavin via
lists.yoctoproject.org <alex.kanavin=gmail.com@lists.yoctoproject.org>
wrote:
> So here's what I'd like to try:
>
> - write a new populate_build_replica task that writes a few things
> under ${WORKDIR}/replica
> -- setup-layers json and script
> (another option is to copy the layer trees themselves like esdk does,
> which is left for maybe later. Completely offline replication is not
> an initial goal, and it's good to get to a minimally viable
> implementation asap.)
> -- meta-build-config layer with the local.conf/bblayer.conf template.
> -- sstate cache needed to fulfil the bitbake target that the task is
> for (this would reuse code from esdk that does the same as much as
> possible)
>
> All of this is then packaged into a self-extracting shell archive that:
> - unpacks itself
> - fetches layers using the json/script from the unpacked tree
> - sets up a build directory using the template from meta-build-config
> in the unpacked tree
> - tweaks site.conf to point to the prepackaged sstate
>
> And voila! (in theory) This should be the same build as the one that
> was produced elsewhere.

I started writing this as a bitbake task, but then quickly hit an
obstacle: bitbake-layers deadlocks when executed in a bitbake task.

At which point it dawned on me: all of the required functions are
available in command-line utility form, and so this should be a script
as well, with as many options as people find useful (as opposed to
bitbake tasks which are more difficult to parametrize).

So:
- bitbake -S lockedsigs to get the list of sstate objects
- gen-lockedsig-cache to populate a newly made sstate cache directory
with them (optional)
- bitbake-layers to save the layer configuration and build configuration

Then either leave all that in an unpacked form, or package it up in a
self-extracting shell archive or plain tarball. It would also contain
a script that would set up a plain bitbake build from what is in the
archive.

This could be poky/scripts/oe-replicate-build ?

Alex

Re: Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 14 Sept 2023 at 13:52, Alexander Kanavin <alex.kanavin@gmail.com> wrote:
>
> On Tue, 12 Sept 2023 at 16:44, Stephen Jolley <sjolley.yp.pm@gmail.com> wrote:
> > Alexander Kanavin will be working on the core workflow topic

I thought I'd write a summary of where we are with these subjects, and
make a plan for what needs to be done still:

1. Tools for figuring out what needs to be rebuilt and tests for them.

There was a large amount of corner cases to chase here, and fixing all
of that took a bit of time. I'm glad to report that all known issues
have been addressed, all patches have landed in master, and hopefully
this topic doesn't need further attention for now.

2. CDN sstate cache mirror and tests for it.

The tests for the mirror have landed as well, and there is an ongoing
issue that they expose: the CDN server isn't 100% robust. Sometimes it
returns a 5xx error, sometimes simply times out without responding.
This happens particularly with large objects, e.g. 50-60 Mb in size.
The problems are tracked in
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15335 - hopefully
Michael Halstead can take a look soon-ish.

3. Build replicator tool.

I've written a proof of concept:
https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/sstate-for-all

There was positive feedback for the idea, so next I'll add tests and
submit it as a proper patch aiming for inclusion.

4. Any other business.

At this point I don't have any other subjects in mind, but if you do,
please write them here. Ideas, or maybe I missed something super
important.

Appendix.

There's also the oe-setup-build tool in development, which isn't
strictly a part of the core workflow topic, but is related, and I'd
like to get it to completion as well:
https://git.yoctoproject.org/poky-contrib/log/?h=akanavin/setup-layers-tweaks
I'll check if I had missed any feedback, and re-submit.

Alex


Alex

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Mon, 22 Jan 2024 at 11:47, Alexander Kanavin via
lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org>
wrote:
> 3. Build replicator tool.
>
> I've written a proof of concept:
> https://git.yoctoproject.org/poky-contrib/commit/?h=akanavin/sstate-for-all
>
> There was positive feedback for the idea, so next I'll add tests and
> submit it as a proper patch aiming for inclusion.

I started working on bringing this to level suitable for proper
submission, but quickly ran into two issues:

1. The bundle should include a local copy of all the layers so it can
be unpacked and set up entirely offline (and not by re-fetching all
the layers like in the proof of concept). This functionality is
available in the esdk task, but needs to be provided in plain yocto
builds as well. The plan is to add a 'local-copy' plugin to
'bitbake-layers create-layers-setup' that would utilize existing code.

2. Something that didn't occur to me until now: simply copying over
sstate to another location doesn't guarantee that it's going to be
used, if that sstate needs to be discovered via hash equivalency
database. There are two options here:

- lock down sstate signatures to what is available (esdks take this
route, and get away with it because of restricted devtool-only
functionality and hidden layers that never change)

- export the hash equivalency database into something human readable,
and placed into the bundle, and then import it back when the bundle is
unpacked and set up. As far as I understand currently there's no code
for this. I'd find it useful in general, as effects of hash
equivalency can be tricky to grasp, and being able to look at the
database in a human readable format goes a long way towards
understanding how it all fits together. There may be other use cases
where you'd want to move the content of the database between build
machines. I'd like to get Joshua's take on this, did I misunderstand
this completely?

Alex

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Mon, 2024-02-05 at 21:35 +0100, Alexander Kanavin wrote:
> 2. Something that didn't occur to me until now: simply copying over
> sstate to another location doesn't guarantee that it's going to be
> used, if that sstate needs to be discovered via hash equivalency
> database. There are two options here:
> 
> - lock down sstate signatures to what is available (esdks take this
> route, and get away with it because of restricted devtool-only
> functionality and hidden layers that never change)
> 
> - export the hash equivalency database into something human readable,
> and placed into the bundle, and then import it back when the bundle is
> unpacked and set up. As far as I understand currently there's no code
> for this. I'd find it useful in general, as effects of hash
> equivalency can be tricky to grasp, and being able to look at the
> database in a human readable format goes a long way towards
> understanding how it all fits together. There may be other use cases
> where you'd want to move the content of the database between build
> machines. I'd like to get Joshua's take on this, did I misunderstand
> this completely?

You don't misunderstand, this is an issue.

The good news is that the hashes needed will be in
cache/bb_unihashes.dat. The eSDK code saves and restores that to ensure
things all work out ok.

It isn't human readable but could be translated easily enough.

Cheers,

Richard

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Mon, 5 Feb 2024 at 22:11, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> > - export the hash equivalency database into something human readable,
> > and placed into the bundle, and then import it back when the bundle is
> > unpacked and set up. As far as I understand currently there's no code
> > for this. I'd find it useful in general, as effects of hash
> > equivalency can be tricky to grasp, and being able to look at the
> > database in a human readable format goes a long way towards
> > understanding how it all fits together. There may be other use cases
> > where you'd want to move the content of the database between build
> > machines. I'd like to get Joshua's take on this, did I misunderstand
> > this completely?
>
> You don't misunderstand, this is an issue.
>
> The good news is that the hashes needed will be in
> cache/bb_unihashes.dat. The eSDK code saves and restores that to ensure
> things all work out ok.
>
> It isn't human readable but could be translated easily enough.

Right, I didn't realize bb_unihashes.dat is simply a python pickle,
and can be dumped to stdout with a three-liner script.

What I don't understand is why for each task, that file seems to
contain only a single pair of equivalence, and only matching the last
build. The complete set seems to be in hashserv.db, but I didn't yet
look into the structure of that or whether it's portable to other
locations as a file copy.

So how do the two files interact? Shouldn't we copy over the full
database rather, and have bitbake re-create the cache on the other
side from that?

Alex

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Thu, 2024-02-08 at 14:35 +0100, Alexander Kanavin wrote:
> On Mon, 5 Feb 2024 at 22:11, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > > - export the hash equivalency database into something human readable,
> > > and placed into the bundle, and then import it back when the bundle is
> > > unpacked and set up. As far as I understand currently there's no code
> > > for this. I'd find it useful in general, as effects of hash
> > > equivalency can be tricky to grasp, and being able to look at the
> > > database in a human readable format goes a long way towards
> > > understanding how it all fits together. There may be other use cases
> > > where you'd want to move the content of the database between build
> > > machines. I'd like to get Joshua's take on this, did I misunderstand
> > > this completely?
> > 
> > You don't misunderstand, this is an issue.
> > 
> > The good news is that the hashes needed will be in
> > cache/bb_unihashes.dat. The eSDK code saves and restores that to ensure
> > things all work out ok.
> > 
> > It isn't human readable but could be translated easily enough.
> 
> Right, I didn't realize bb_unihashes.dat is simply a python pickle,
> and can be dumped to stdout with a three-liner script.
> 
> What I don't understand is why for each task, that file seems to
> contain only a single pair of equivalence, and only matching the last
> build. The complete set seems to be in hashserv.db, but I didn't yet
> look into the structure of that or whether it's portable to other
> locations as a file copy.
> 
> So how do the two files interact? Shouldn't we copy over the full
> database rather, and have bitbake re-create the cache on the other
> side from that?

hashserve.db is a full hash serve database. When we run a build without
specifying a hash serve to use, bitbake will start a local one and this
is the file which backs it. In a build with a remote hashserve, it
won't be present.

the unihashes file is bitbake's internal lookup cache. It exists so
that if the hashserve isn't local, we have a local lookup of our "last"
value.

This means the file is good for the current (last) build not much
beyond that.

I'm not saying any of this entirely solves the problem at hand, just
that some of the data is there.

Cheers,

Richard

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Thu, 8 Feb 2024 at 14:42, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> hashserve.db is a full hash serve database. When we run a build without
> specifying a hash serve to use, bitbake will start a local one and this
> is the file which backs it. In a build with a remote hashserve, it
> won't be present.
>
> the unihashes file is bitbake's internal lookup cache. It exists so
> that if the hashserve isn't local, we have a local lookup of our "last"
> value.
>
> This means the file is good for the current (last) build not much
> beyond that.
>
> I'm not saying any of this entirely solves the problem at hand, just
> that some of the data is there.

So perhaps we need the following interface:

bitbake -S export-hashes targets...

would save all relevant entries for the targets from outhash and
unihash tables into a local file (similar to locked-sigs.inc creation
with -S lockedsigs). This avoids exporting the entire database, which
is both excessive and unnecessary, and at the same time ensures
everything relevant is in the export (merely copying bb_unihashes.dat
does not guarantee that).

Then we'd also need a way to import the data, but I'm not sure where
to put that. 'bitbake -S' is not suitable. A new command to
bitbake-hashclient perhaps? But bitbake-hashclient doesn't discover
where the hash server should be from bitbake config, and won't
autostart it if/when needed.

Alex

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Richard Purdie]

On Tue, 2024-02-13 at 14:25 +0100, Alexander Kanavin wrote:
> On Thu, 8 Feb 2024 at 14:42, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > hashserve.db is a full hash serve database. When we run a build without
> > specifying a hash serve to use, bitbake will start a local one and this
> > is the file which backs it. In a build with a remote hashserve, it
> > won't be present.
> > 
> > the unihashes file is bitbake's internal lookup cache. It exists so
> > that if the hashserve isn't local, we have a local lookup of our "last"
> > value.
> > 
> > This means the file is good for the current (last) build not much
> > beyond that.
> > 
> > I'm not saying any of this entirely solves the problem at hand, just
> > that some of the data is there.
> 
> So perhaps we need the following interface:
> 
> bitbake -S export-hashes targets...
> 
> would save all relevant entries for the targets from outhash and
> unihash tables into a local file (similar to locked-sigs.inc creation
> with -S lockedsigs). This avoids exporting the entire database, which
> is both excessive and unnecessary, and at the same time ensures
> everything relevant is in the export (merely copying bb_unihashes.dat
> does not guarantee that).
> 
> Then we'd also need a way to import the data, but I'm not sure where
> to put that. 'bitbake -S' is not suitable. A new command to
> bitbake-hashclient perhaps? But bitbake-hashclient doesn't discover
> where the hash server should be from bitbake config, and won't
> autostart it if/when needed.

In theory you can start a local hash server and then report all the
hashes to it?

Cheers,

Richard

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Tue, 13 Feb 2024 at 14:44, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> In theory you can start a local hash server and then report all the
> hashes to it?

Yes, but I'd like to replicate what bitbake does: check configuration,
start the local server if needed, and only then do the mass-import
from the file.

Alex

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Joshua Watt]

[-- Attachment #1: Type: text/plain, Size: 1190 bytes --]

On Tue, Feb 13, 2024, 7:05 AM Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> On Tue, 13 Feb 2024 at 14:44, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > In theory you can start a local hash server and then report all the
> > hashes to it?
>
> Yes, but I'd like to replicate what bitbake does: check configuration,
> start the local server if needed, and only then do the mass-import
> from the file.
>

Sorry for chiming in so late; I think you are on the right track though.
Exporting/importing bb_unihashes.dat is the think to do instead t of the
entire database (which you may not even have if it's remote). That should
at least let you reuse any sstate file that is in that file. Importing the
data as the new cache is (probably) fine and the easy first step, but be
aware it means you probably won't be able to find equivalent hashes that
are derived from the imported data, so you'll be looking at more rebuilding
than you might expect if you change something. Reporting the hashes to the
server might be an option also, but I don't know if the cache file has all
the information to do that correctly... Have to look



> Alex
>

[-- Attachment #2: Type: text/html, Size: 1857 bytes --]

Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[Alexander Kanavin]

On Tue, 13 Feb 2024 at 15:29, Joshua Watt <jpewhacker@gmail.com> wrote:
>> Yes, but I'd like to replicate what bitbake does: check configuration,
>> start the local server if needed, and only then do the mass-import
>> from the file.
>
>
> Sorry for chiming in so late; I think you are on the right track though. Exporting/importing bb_unihashes.dat is the think to do instead t of the entire database (which you may not even have if it's remote). That should at least let you reuse any sstate file that is in that file. Importing the data as the new cache is (probably) fine and the easy first step, but be aware it means you probably won't be able to find equivalent hashes that are derived from the imported data, so you'll be looking at more rebuilding than you might expect if you change something. Reporting the hashes to the server might be an option also, but I don't know if the cache file has all the information to do that correctly... Have to look
>

Let's then pick only the low hanging fruit first: add 'bitbake -S
unihash-cache' which reuses existing api and code behind it (that was
written for esdk). 'Importing' it only requires placing the file in
build/cache/ on the other end. The goal is to get to a working build
replication tool that can be tested.

Alex