esdk devtool finish workflow

esdk devtool finish workflow

[Radoslav Pesek]

[-- Attachment #1: Type: text/plain, Size: 2546 bytes --]

Hello,

I'm trying to do the following (after generating and installing my esdk and sourcing the environment script):
* devtool modify recipe
* do some coding and testing
* devtool finish recipe meta-custom -m srcrev

I want just to update SRCREV of the recipe, *not* to create source code patches (that's why i use -m srcrev) and *not* to create bbappend for the recipe, but:
* if meta-custom is a layer under the esdk (esdk/layers/meta-custom), the issue is, it isn't git repo separate from the other layers under esdk/layers (because whole esdk/layers directory is one git repo), so when i commit the changes made by devtool finish (updated SRCREV), i don't make the commit in the original git commit history and am not sure how now to push the changes to the git server
* if meta-custom is git repo somewhere else on the my machine, cloned from the original git repo, devtool finish doesn't just update the recipe (with changed SRCREV), but creates an bbappend, which i don't like cause updating recipe seems to me to be cleaner then having recipe and bbappend

So i'm not sure, how to accomplish this. Maybe i'm just missing something or don't use esdk/devtool as it's intended. So, could somebody, please, explain to me the intended workflow or why what i'm trying to do isn't possible (or how, if it is)? Cause what documentation/howtos regarding devtool finish i came across, either didn't use it within esdk or didn't use the workflow i'm trying to use. I also tried to search some explanation why sdk/layers is one git repo (instead of having the layers under esdk/layers cloned as separate git repos - as it actually works when you use devtool under yocto build system) or why devtool finish does create bbappend (and doesn't simply update the recipe) when the layer you want to update isn't the original layer, where the recipe comes from (which i guess, within esdk, it's esdk/layers/meta-custom in my case) as for example mentioned here: https://www.openembedded.org/pipermail/openembedded-core/2016-July/123873.html, but to no avail.

I didn't try it yet, but the only thing that could probably help is adding my layer (cloned somewhere on my machine) to the esdk's bblayers.conf, but then, i would have the layer twice in the bblayers.conf (the one that was installed with the esdk and the one added manually), or even try to remove the one that was installed with esdk, which i'm not sure whether it would work.

Currently i'm on scarthgap, if that makes any difference.

Thank you for any clarification, Rado.

[-- Attachment #2: Type: text/html, Size: 2829 bytes --]

Re: [yocto] esdk devtool finish workflow

[Alexander Kanavin]

Unfortunately when standalone esdk archives are made, the layers are
copied into the archive with a simple file tree operation, and all git
history and pointers to upstream repos are thrown away.

I would recommend setting up a plain, direct yocto environment
instead. There's an officially supported way to obtain a SDK toolchain
environment in there nowadays, so it's functionally equivalent to esdk
archives, but gives you 'real yocto with bitbake' at the same time,
and devtool will work exactly as it does in esdk:
https://docs.yoctoproject.org/sdk-manual/extensible.html#installing-the-extensible-sdk

Alex

On Fri, 6 Sept 2024 at 12:38, Radoslav Pesek via
lists.yoctoproject.org
<radoslav.pesek=microstep-mis.com@lists.yoctoproject.org> wrote:
>
> Hello,
>
> I'm trying to do the following (after generating and installing my esdk and sourcing the environment script):
> * devtool modify recipe
> * do some coding and testing
> * devtool finish recipe meta-custom -m srcrev
>
> I want just to update SRCREV of the recipe, not to create source code patches (that's why i use -m srcrev) and not to create bbappend for the recipe, but:
> * if meta-custom is a layer under the esdk (esdk/layers/meta-custom), the issue is, it isn't git repo separate from the other layers under esdk/layers (because whole esdk/layers directory is one git repo), so when i commit the changes made by devtool finish (updated SRCREV), i don't make the commit in the original git commit history and am not sure how now to push the changes to the git server
> * if meta-custom is git repo somewhere else on the my machine, cloned from the original git repo, devtool finish doesn't just update the recipe (with changed SRCREV), but creates an bbappend, which i don't like cause updating recipe seems to me to be cleaner then having recipe and bbappend
>
> So i'm not sure, how to accomplish this. Maybe i'm just missing something or don't use esdk/devtool as it's intended. So, could somebody, please, explain to me the intended workflow or why what i'm trying to do isn't possible (or how, if it is)? Cause what documentation/howtos regarding devtool finish i came across, either didn't use it within esdk or didn't use the workflow i'm trying to use. I also tried to search some explanation why sdk/layers is one git repo (instead of having the layers under esdk/layers cloned as separate git repos - as it actually works when you use devtool under yocto build system) or why devtool finish does create bbappend (and doesn't simply update the recipe) when the layer you want to update isn't the original layer, where the recipe comes from (which i guess, within esdk, it's esdk/layers/meta-custom in my case) as for example mentioned here: https://www.openembedded.org/pipermail/openembedded-core/2016-July/123873.html, but to no avail.
>
> I didn't try it yet, but the only thing that could probably help is adding my layer (cloned somewhere on my machine) to the esdk's bblayers.conf, but then, i would have the layer twice in the bblayers.conf (the one that was installed with the esdk and the one added manually), or even try to remove the one that was installed with esdk, which i'm not sure whether it would work.
>
> Currently i'm on scarthgap, if that makes any difference.
>
> Thank you for any clarification, Rado.
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#63793): https://lists.yoctoproject.org/g/yocto/message/63793
> Mute This Topic: https://lists.yoctoproject.org/mt/108301955/1686489
> Group Owner: yocto+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [yocto] esdk devtool finish workflow

[Radoslav Pesek]

[-- Attachment #1: Type: text/plain, Size: 478 bytes --]

Hi Alex,

thank you for the prompt reply. As for me personally, I am, actually, using the yocto environment, as the yocto maintainer within our project, but wanted to create esdk for the developers as i hoped it would be easier for them to work with recipes and for me/us to maintain it all together - as opposed to, for example, each of us having separate yocto environment. My impression (from all the documentation/howtos i found) was that this is the way to go.

Rado.

[-- Attachment #2: Type: text/html, Size: 545 bytes --]

Re: [yocto] esdk devtool finish workflow

[Alexander Kanavin]

On Fri, 6 Sept 2024 at 15:54, Radoslav Pesek via
Lists.Yoctoproject.Org
<radoslav.pesek=microstep-mis.com@lists.yoctoproject.org> wrote:
> thank you for the prompt reply. As for me personally, I am, actually, using the yocto environment, as the yocto maintainer within our project, but wanted to create esdk for the developers as i hoped it would be easier for them to work with recipes and for me/us to maintain it all together - as opposed to, for example, each of us having separate yocto environment. My impression (from all the documentation/howtos i found) was that this is the way to go.

I'd still recommend that you play with the 'direct esdk' workflow. The
challenge for you would be to provide a well functioning sstate
infrastructure, so developers never have to watch never-ending
do_compile on their feeble laptops :)

We're working on a 'single-command' reproducible setup of yocto
(bitbake-setup), but it's currently in review/prototyping. There's
also a prototype for 'oe-replicate-build' which would bundle a 'real
yocto build' into a self-extracting tarball, like the esdk, but
without the 'special' bits that obscure bitbake and make layer
modifications difficult.

Classic esdk bundles will almost certainly not be developed further.

I'm also CCing Adrian as he's in the same boat as you are, and has a
lot more experience with developing real products and giving
developers a VSCode experience that's integrated with yocto. I'm just
living in an open source bubble :)

Alex

Re: [yocto] esdk devtool finish workflow

[Adrian Freihofer]

On Sat, 2024-09-07 at 12:41 +0200, Alexander Kanavin wrote:
> On Fri, 6 Sept 2024 at 15:54, Radoslav Pesek via
> Lists.Yoctoproject.Org
> <radoslav.pesek=microstep-mis.com@lists.yoctoproject.org> wrote:
> > thank you for the prompt reply. As for me personally, I am,
> > actually, using the yocto environment, as the yocto maintainer
> > within our project, but wanted to create esdk for the developers as
> > i hoped it would be easier for them to work with recipes and for
> > me/us to maintain it all together - as opposed to, for example,
> > each of us having separate yocto environment. My impression (from
> > all the documentation/howtos i found) was that this is the way to
> > go.
> 
> I'd still recommend that you play with the 'direct esdk' workflow.
> The
> challenge for you would be to provide a well functioning sstate
> infrastructure, so developers never have to watch never-ending
> do_compile on their feeble laptops :)
> 
> We're working on a 'single-command' reproducible setup of yocto
> (bitbake-setup), but it's currently in review/prototyping. There's
> also a prototype for 'oe-replicate-build' which would bundle a 'real
> yocto build' into a self-extracting tarball, like the esdk, but
> without the 'special' bits that obscure bitbake and make layer
> modifications difficult.
> 
> Classic esdk bundles will almost certainly not be developed further.
> 
> I'm also CCing Adrian as he's in the same boat as you are, and has a
> lot more experience with developing real products and giving
> developers a VSCode experience that's integrated with yocto. I'm just
> living in an open source bubble :)

Hi Radoslav, hi Alex

It is true that we have replaced our eSDK-based setups with Direct SDK-
based setups. A year ago, I had the opportunity to talk about this
topic at Yocto Sumit: file:///home/adrian/Downloads/yocto-summit-
2023.11-devtool-i_S04eDYy.pdf
One year after this presentation, I can summarize that it works much
better than what we had with the static eSDK installers.

We share the layers and the site.conf file with git submodules. There
is also a tool developed by Alex which will help you with that part of
replacing the eSDK installer without dealing with git submodules.

The second thing you need is a shared sstate-cache. This feature is now
available officially in Yocto. We do not use this so far because we do
not have a shared hash equivalence server yet. We use a simple, a bit
hacky script which downloads all sstate-cache artifacts before we call
bitbake.

@Alex: We should discuss this issue in Vienna. I think there has been
great progress in many areas. But I don't think there is yet a complete
concept for operating a complete infrastructure for distributing layers
and ssate-cache, which also provides security and scalability. It's
definitely doable with enough knowledge and maybe some helper scripts
and compromises as we do it internally or with a CDN without security
as the Yocto Project does it.

Regrads,
Adrian


> 
> Alex

Re: [yocto] esdk devtool finish workflow

[Alexander Kanavin]

On Sun, 8 Sept 2024 at 15:24, Adrian Freihofer
<adrian.freihofer@gmail.com> wrote:
> It is true that we have replaced our eSDK-based setups with Direct SDK-
> based setups. A year ago, I had the opportunity to talk about this
> topic at Yocto Sumit: file:///home/adrian/Downloads/yocto-summit-
> 2023.11-devtool-i_S04eDYy.pdf
> One year after this presentation, I can summarize that it works much
> better than what we had with the static eSDK installers.

You also need to provide a link that can be opened :)

> The second thing you need is a shared sstate-cache. This feature is now
> available officially in Yocto. We do not use this so far because we do
> not have a shared hash equivalence server yet. We use a simple, a bit
> hacky script which downloads all sstate-cache artifacts before we call
> bitbake.

You can switch off hash equivalence in local.conf or distro conf or
site conf. Then sstate signature calculations will work 'as they used
to', and you can provide only the sstate, without having to set up a
hashequiv server as well.

The downside is that it needs to be switched off in CI as well, and
that can lengthen CI builds significantly, if they no longer benefit
from hashequiv shortcuts.

> @Alex: We should discuss this issue in Vienna. I think there has been
> great progress in many areas. But I don't think there is yet a complete
> concept for operating a complete infrastructure for distributing layers
> and ssate-cache, which also provides security and scalability. It's
> definitely doable with enough knowledge and maybe some helper scripts
> and compromises as we do it internally or with a CDN without security
> as the Yocto Project does it.

We're working on it. You can find the oe-replicate-build prototype here:
https://git.yoctoproject.org/poky-contrib/log/?h=akanavin/sstate-for-all
(commits 3 to 5 from the top)
and bitbake-setup prototype here:
https://github.com/kanavin/bitbake/commits/akanavin/bitbake-setup/
https://github.com/kanavin/bitbake-setup-configurations

We should certainly discuss in Vienna, but you can also play with
these beforehands.

Alex

Re: [yocto] esdk devtool finish workflow

[Adrian Freihofer]

On Sun, 2024-09-08 at 15:46 +0200, Alexander Kanavin wrote:
> On Sun, 8 Sept 2024 at 15:24, Adrian Freihofer
> <adrian.freihofer@gmail.com> wrote:
> > It is true that we have replaced our eSDK-based setups with Direct
> > SDK-
> > based setups. A year ago, I had the opportunity to talk about this
> > topic at Yocto Sumit: file:///home/adrian/Downloads/yocto-summit-
> > 2023.11-devtool-i_S04eDYy.pdf
> > One year after this presentation, I can summarize that it works
> > much
> > better than what we had with the static eSDK installers.
> 
> You also need to provide a link that can be opened :)

next try :-)
https://summit.yoctoproject.org/media/yocto-project-summit-2023-11/submissions/UZ9BN7/resources/yocto-summit-2023.11-devtool-i_S04eDYy.pdf

> 
> > The second thing you need is a shared sstate-cache. This feature is
> > now
> > available officially in Yocto. We do not use this so far because we
> > do
> > not have a shared hash equivalence server yet. We use a simple, a
> > bit
> > hacky script which downloads all sstate-cache artifacts before we
> > call
> > bitbake.
> 
> You can switch off hash equivalence in local.conf or distro conf or
> site conf. Then sstate signature calculations will work 'as they used
> to', and you can provide only the sstate, without having to set up a
> hashequiv server as well.
> 
> The downside is that it needs to be switched off in CI as well, and
> that can lengthen CI builds significantly, if they no longer benefit
> from hashequiv shortcuts.

Yes, that's how it is. Ideally there would be an easy way to get all
these nice new features together.

> 
> > @Alex: We should discuss this issue in Vienna. I think there has
> > been
> > great progress in many areas. But I don't think there is yet a
> > complete
> > concept for operating a complete infrastructure for distributing
> > layers
> > and ssate-cache, which also provides security and scalability. It's
> > definitely doable with enough knowledge and maybe some helper
> > scripts
> > and compromises as we do it internally or with a CDN without
> > security
> > as the Yocto Project does it.
> 
> We're working on it. You can find the oe-replicate-build prototype
> here:
> https://git.yoctoproject.org/poky-contrib/log/?h=akanavin/sstate-for-all
> (commits 3 to 5 from the top)
> and bitbake-setup prototype here:
> https://github.com/kanavin/bitbake/commits/akanavin/bitbake-setup/
> https://github.com/kanavin/bitbake-setup-configurations
> 
> We should certainly discuss in Vienna, but you can also play with
> these beforehands.

I will have a look at it. It's some time ago when I tried it.

Adrian

> 
> Alex

Re: [yocto] esdk devtool finish workflow

[Radoslav Pesek]

[-- Attachment #1: Type: text/plain, Size: 1604 bytes --]

Hi Alex, Adrian,

thanks for the explanation. I checked the slides, watched the video and tried this workflow (partially - didn't try deploy-target and finish), and it seems to work, but still have some questions:
* Alex, you write above: Classic esdk bundles will almost certainly not be developed further. - so am i getting this right, that there isn't gonna be a way to generate e/sdk installer in the future (and there will be only this workflow available)?
* Adrian, you write above: The second thing you need is a shared sstate-cache. This feature is now available officially in Yocto. - do you mean by this publishing it via web server? So if have it published and it works for standalone esdk, will it also work for this new devtool workflow under normal yocto build environment? Do i need to do anything specific or will it work out of the box when someone else creates their own yocto build enironment?
* in the presentation you talk about cmake and meson; what's the state of support for building recipes using other ways - for example makefiles and python packages (i tried to devtool modify and build a python app and it seems to work)
* are there any plans to support other IDEs - specifically Eclipse? Or is it difficult to add support for them on our own? Or, If i understood correctly - if we use --ide=none, it means devtool just creates some scripts for building/debugging and then we basically can use any IDE, right?
* what's the state of this new devtool workflow at the moment - do you have any idea when it's gonna be officially released?

Thanks a lot for your time :)

[-- Attachment #2: Type: text/html, Size: 1745 bytes --]

Re: [yocto] esdk devtool finish workflow

[Alexander Kanavin]

On Mon, 9 Sept 2024 at 16:24, Radoslav Pesek via
lists.yoctoproject.org
<radoslav.pesek=microstep-mis.com@lists.yoctoproject.org> wrote:
> thanks for the explanation. I checked the slides, watched the video and tried this workflow (partially - didn't try deploy-target and finish), and it seems to work, but still have some questions:
> * Alex, you write above: Classic esdk bundles will almost certainly not be developed further. - so am i getting this right, that there isn't gonna be a way to generate e/sdk installer in the future (and there will be only this workflow available)?

They will be supported and they will work as they do now. What I meant
is that esdk bundles will not gain new functionality or get their
shortcomings addressed.

> * Adrian, you write above: The second thing you need is a shared sstate-cache. This feature is now available officially in Yocto. - do you mean by this publishing it via web server? So if have it published and it works for standalone esdk, will it also work for this new devtool workflow under normal yocto build environment? Do i need to do anything specific or will it work out of the box when someone else creates their own yocto build enironment?

I'm not sure what Adrian meant by 'now available officially'. Sstate
has been supported for a very long time; the basic workflow is that CI
machines all share a r/w NFS directory which they read and write from,
and then this directory is published over http and made available to
developer machines, so their yocto builds can be quickly assembled
from pre-build sstate artefacts. The key part is that developers build
configurations cannot deviate from what CI has built, or yocto will
have to rebuild the changed parts and everything that depends on
those.

Alex

Re: [yocto] esdk devtool finish workflow

[Radoslav Pesek]

[-- Attachment #1: Type: text/plain, Size: 931 bytes --]

Hi, thanks for reply and sorry, been busy with other stuff.

Regarding esdk - what is their purpose now, when there is this new workflow? I mean, in what use cases would you use esdk instead of this new workflow?

For example, i'm now trying to setup ci/cd pipeline for testing using qemu. Is it good idea to run devtool runqemu or just runqemu from esdk? Cause i tried that and it failed - specifically, i run MACHINE=qemuarm devtool runqemu nographic and get "runqemu - ERROR - IMAGE_LINK_NAME wasn't set to find corresponding .qemuboot.conf file" (i get similar error even when just running MACHINE=qemuarm runqemu). MACHINE=qemuarm runqemu command runs fine under normal yocto environment (devtool runqemu subcommand doesn't exist under this environment). Also, tmp/deploy/images/qemuarm/*.qemuboot.conf exist in esdk. Changing MACHINE to runqemu in conf/local.conf doesn't work either. Is this expected?

Thanks, rado.

[-- Attachment #2: Type: text/html, Size: 1045 bytes --]

Re: [yocto] esdk devtool finish workflow

[Alexander Kanavin]

On Mon, 30 Sept 2024 at 14:33, Radoslav Pesek via
Lists.Yoctoproject.Org
<radoslav.pesek=microstep-mis.com@lists.yoctoproject.org> wrote:
> Regarding esdk - what is their purpose now, when there is this new workflow? I mean, in what use cases would you use esdk instead of this new workflow?

There is no purpose, other than compatibility with legacy processes
and workflows that existing yocto users rely on, and esdk's ability to
provide something self-contained that can be put on a usb stick and
unpacked elsewhere. I've prototyped a tool to do this with a regular
yocto builds.

I'd happily mark bundled esdk as a deprecated feature if I could :)

> For example, i'm now trying to setup ci/cd pipeline for testing using qemu. Is it good idea to run devtool runqemu or just runqemu from esdk? Cause i tried that and it failed - specifically, i run MACHINE=qemuarm devtool runqemu nographic and get "runqemu - ERROR - IMAGE_LINK_NAME wasn't set to find corresponding .qemuboot.conf file" (i get similar error even when just running MACHINE=qemuarm runqemu). MACHINE=qemuarm runqemu command runs fine under normal yocto environment (devtool runqemu subcommand doesn't exist under this environment). Also, tmp/deploy/images/qemuarm/*.qemuboot.conf exist in esdk. Changing MACHINE to runqemu in conf/local.conf doesn't work either. Is this expected?

By all means, in CI you should run runqemu directly from a yocto
build. Esdks are not even intended for CI, they're for developers.

Alex