Re: [OE-core] Core workflow: sstate for all, bblock/bbunlock, tools for why is sstate not being reused?

[adrian.freihofer@gmail.com]

Hi Alex, hi Richard

After some internal discussions, I would like to clarify my previous
answers on this topic.

 * Usually there are two different workflows
    - application developers: could use an SDK with a locked sstate-cache.
    - Yocto/BSP developers: need an unlocked SDK. They change the recipes.
 * A locked SDK
    - can work with setscene from SSTATE_MIRRORS
    - setscene does caching in the SSTATE_DIR (no issue about that)
    - But network problems can occur during the initial build because
      bitbake executes many independent setscene tasks. Opening so many
      independent connections slows down the build, especially if the
      server treats them as a denial of service attack.
    - The denial of service problem is difficult to solve because each
      setscene task runs in its own bibtake task. Reusing a connection to
      download multiple sstate artifacts seems almost impossible.
      This is much easier to solve with separate sstate download script.
 * An unlocked SDK
    - Tries to download the sstate cache for changed recipes and their
      dependencies, which obviously can't work.
    - The useless download requests slow down the build considerably and
      cause a high load on the servers without any benefit.
    - A script which gets a list of sstate artifacts from bitbake and then
      does a upfront download works much better
       + The script runs only when the user calls it or the SDK gets boot-
         strapped
       + The script uses a reasonable amount of parallel connections which
         are re-used for more then one artifact download
 * Idea for a smart lock/unlock implementation
    - Form a user's perspective a locked vs. an unlocked SDK does not make
      much sense. It makes more sense if the SDK would automatically
      download the sstate-cache if it is expected to be available.
      Lets think about an implementation (which allows to override the
      logic) to switch from automatic to manual mode:
      
      SSTATE_MIRRORS_ENABLED ?= "${is_sstate_mirror_available()}"
      
      In our case the sstate mirror is expected to provide all artifacts
      for tagged commits and for some git branches of the layer
      repositories.
      The sstate is obviousely not usable for a "dirty" git layer
      repository. That's what the is_sstate_mirror_available function
      could check to automatically enable and disable lazy downloads.
      
    - If is_sstate_mirror_available() returns false, it should still be
      possible to initiate a sstate-cache download manually.
      
 * Terminology
    - Older Yocto Releases:
       + eSDK means an installer which provides a different environment with
         different tools
       + The eSDK was static, with a locked sstate cache
       + Was for one MACHINE, for one image...
    - Newer Yocto Releases:
       + The bitbake environment offers all features of the eSDK installer. I
         consider this as already implemented with meta-ide-support and
         build-sysroots.
       + The term eSDK means a replicable bitbake environment. (The
         documentation was recently changed in that sense)
       + The new SDK installer can be generated in different variants similar
         to what was already supported by the eSDK
         installer: https://docs.yoctoproject.org/sdk-manual/appendix-
         customizing.html#customizing-the-extensible-sdk-standalone-
         installer.
          * The lightest variant is just a script that sets up the layers (git
            clone or git checkout) and provides the build config. If
            SSTATE_MIRRORS are configured lazy downloads will just work
            otherwise bitbake will compile everything from scratch.
          * The heaviest variant of the SDK installer includes the layers and
            the sstate-cache. After installing it is_sstate_mirror_available()
            evaluates to True.
* devtool ide
- Tries to bring this ideas further towards IDE configuration
- It supports two modes: No recipe mode is like the old eSDK
environment. The recipe mode goes beyond that. It is based on
devtool modify.
    - Naming: I was thinking about "devtool ide" versus "devtool esdk"
       + Why ide? I want to set up my IDE to work with the Yocto SDK. And
         that means, of course, that I have to bootstrap the SDK.
       + Could also be esdk: I want to bootstrap the eSDK and that should
         also configure my IDE.
 * Latest patch series is here: 
    - https://lists.openembedded.org/g/openembedded-core/message/189899
    - docs: https://lists.yoctoproject.org/g/docs/message/4578

Adrian