Using devtool without an absolute path to the workspace in bblayers.conf

Using devtool without an absolute path to the workspace in bblayers.conf

[Mike Crowe]

I'm trying to work out how we can make use of devtool to make our lives
easier during development. In general it seems to work very well, but the
way that it modifies bblayers.conf to add an absolute path to the workspace
directory to BBLAYERS is incompatible with that file being held in a Git
repository and shared by multiple users in multiple trees. There's a high
risk that the file will accidentally be committed containing a path that's
only meaningful for a single user in a single source tree. All of our other
paths in bblayers.conf are relative to a variable that contains the path to
the top of the source tree.

I can manually add ${TOPDIR}/workspace to BBLAYERS in bblayers.conf and
commit that, but unfortunately devtool insists on continuing to add the
absolute path since it doesn't know that's the same as ${TOPDIR}/workspace.

I can set "workspace_path = workspace" in devtool.conf to use a relative
path, and that superficially works until externalsrc.bbclass gets upset
that the EXTERNALSRC is not an absolute path.

I've tried teaching devtool to prepend ${TOPDIR} if the workspace directory
is not absolute when writing bblayers.conf, but it looks like I'd also need
to do so in many other places.

My current workaround is just to add ${TOPDIR}/workspace to the committed
bblayers.conf myself and nobble devtool's _enable_workspace_layer with an
early return. This is ugly and we'd have to carry that change around
forever.

Since I'm clearly swimming against the tide I'm left wondering whether I've
missed something. Is there a way to use devtool without having an absolute
path to the workspace in bblayers.conf?

(At the moment I'm still using dunfell, but I looked at current master and
didn't spot anything that looked like it changed this behaviour.)

Thanks.

Mike.

Re: [OE-core] Using devtool without an absolute path to the workspace in bblayers.conf

[Julien Stephan]

Le mer. 18 oct. 2023 à 16:29, Mike Crowe via lists.openembedded.org
<mac=mcrowe.com@lists.openembedded.org> a écrit :
>
> I'm trying to work out how we can make use of devtool to make our lives
> easier during development. In general it seems to work very well, but the
> way that it modifies bblayers.conf to add an absolute path to the workspace
> directory to BBLAYERS is incompatible with that file being held in a Git
> repository and shared by multiple users in multiple trees. There's a high
> risk that the file will accidentally be committed containing a path that's
> only meaningful for a single user in a single source tree. All of our other
> paths in bblayers.conf are relative to a variable that contains the path to
> the top of the source tree.

Hi Mike,

I think it is a bad idea to version files under the build directory.

Maybe you have a specific use case that you can address in a different way?
You need to share bblayers.conf, I can see 2 options here:
- either you want your co-workers to fetch it only once, when setting
up their source tree
- or you need to frequently modify bblayer.conf because you are adding
or removing layers and want your coworkers to be up-to date

In the first case, TEMPLATECONF is the way to go! More information
here; https://docs.yoctoproject.org/ref-manual/variables.html#term-TEMPLATECONF

In the second scenario, if this is something you *really* need, maybe
you can switch to some other tools such as 'kas'
(https://kas.readthedocs.io/en/latest/)  that should be useful here.

Maybe someone else will have another idea?

If you have a particular use case that is not described here, please
add more details :)

Cheers
Julien




>
> I can manually add ${TOPDIR}/workspace to BBLAYERS in bblayers.conf and
> commit that, but unfortunately devtool insists on continuing to add the
> absolute path since it doesn't know that's the same as ${TOPDIR}/workspace.
>
> I can set "workspace_path = workspace" in devtool.conf to use a relative
> path, and that superficially works until externalsrc.bbclass gets upset
> that the EXTERNALSRC is not an absolute path.
>
> I've tried teaching devtool to prepend ${TOPDIR} if the workspace directory
> is not absolute when writing bblayers.conf, but it looks like I'd also need
> to do so in many other places.
>
> My current workaround is just to add ${TOPDIR}/workspace to the committed
> bblayers.conf myself and nobble devtool's _enable_workspace_layer with an
> early return. This is ugly and we'd have to carry that change around
> forever.
>
> Since I'm clearly swimming against the tide I'm left wondering whether I've
> missed something. Is there a way to use devtool without having an absolute
> path to the workspace in bblayers.conf?
>
> (At the moment I'm still using dunfell, but I looked at current master and
> didn't spot anything that looked like it changed this behaviour.)
>
> Thanks.
>
> Mike.
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#189396): https://lists.openembedded.org/g/openembedded-core/message/189396
> Mute This Topic: https://lists.openembedded.org/mt/102039990/4356723
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [jstephan@baylibre.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] Using devtool without an absolute path to the workspace in bblayers.conf

[Mike Crowe]

On Wednesday 18 October 2023 at 17:47:50 +0200, Julien Stephan wrote:
> Le mer. 18 oct. 2023 � 16:29, Mike Crowe via lists.openembedded.org
> <mac=mcrowe.com@lists.openembedded.org> a �crit :
> >
> > I'm trying to work out how we can make use of devtool to make our lives
> > easier during development. In general it seems to work very well, but the
> > way that it modifies bblayers.conf to add an absolute path to the workspace
> > directory to BBLAYERS is incompatible with that file being held in a Git
> > repository and shared by multiple users in multiple trees. There's a high
> > risk that the file will accidentally be committed containing a path that's
> > only meaningful for a single user in a single source tree. All of our other
> > paths in bblayers.conf are relative to a variable that contains the path to
> > the top of the source tree.
> 
> Hi Mike,
> 
> I think it is a bad idea to version files under the build directory.

Hi Julien,

We don't version all files under the build directory, just ones that
control the configuration that we want to share. We've been using OE that
way for about twelve years now and this is the first time we've run into a
problem. That might at least partly be because we haven't taken advantage
of newer features like devtool though! We mostly avoided devtool by putting
the sources that we modify heavily in submodules and used
externalsrc.bbclass (with some local progressively-less-hacky modifications
that I should try to upstream sometime.)

> Maybe you have a specific use case that you can address in a different way?
> You need to share bblayers.conf, I can see 2 options here:
> - either you want your co-workers to fetch it only once, when setting
> up their source tree
> - or you need to frequently modify bblayer.conf because you are adding
> or removing layers and want your coworkers to be up-to date
> 
> In the first case, TEMPLATECONF is the way to go! More information
> here;
> https://docs.yoctoproject.org/ref-manual/variables.html#term-TEMPLATECONF

We do need to modify bblayers.conf from time to time to add and remove
layers.

Using templates might be possible, but it would appear that this would
force developers to manually incorporate changes (or just wipe their
bblayers.conf file) when the template changes. Just keeping bblayers.conf
under version control doesn't suffer from that problem.

> In the second scenario, if this is something you *really* need, maybe
> you can switch to some other tools such as 'kas'
> (https://kas.readthedocs.io/en/latest/)  that should be useful here.

I have looked at kas in the past (and listened to podcasts about it), but
it seemed to solve a different set of problems. I will look at it again.

Thanks for the suggestions.

Mike

Re: [OE-core] Using devtool without an absolute path to the workspace in bblayers.conf

[Alexander Kanavin]

On Thu, 19 Oct 2023 at 14:33, Mike Crowe via lists.openembedded.org
<mac=mcrowe.com@lists.openembedded.org> wrote:
> We do need to modify bblayers.conf from time to time to add and remove
> layers.
>
> Using templates might be possible, but it would appear that this would
> force developers to manually incorporate changes (or just wipe their
> bblayers.conf file) when the template changes. Just keeping bblayers.conf
> under version control doesn't suffer from that problem.

Generally content of build/conf/, or anything else in build/ is not
designed for being under version control. You may be able to get away
with it by coincidence, but it's neither documented nor tested, and so
any issues are your own.

I don't have a quick solution to ensure that templates and
configurations coming from them are strictly in sync, but welcome to
suggestions. People modify local.conf and bblayers.conf all the time
as well for local experiments, you're simply not supposed to lock them
down.

Alex

Alex

Re: [OE-core] Using devtool without an absolute path to the workspace in bblayers.conf

[Mike Crowe]

On Thursday 19 October 2023 at 14:51:56 +0200, Alexander Kanavin wrote:
> On Thu, 19 Oct 2023 at 14:33, Mike Crowe via lists.openembedded.org
> <mac=mcrowe.com@lists.openembedded.org> wrote:
> > We do need to modify bblayers.conf from time to time to add and remove
> > layers.
> >
> > Using templates might be possible, but it would appear that this would
> > force developers to manually incorporate changes (or just wipe their
> > bblayers.conf file) when the template changes. Just keeping bblayers.conf
> > under version control doesn't suffer from that problem.
> 
> Generally content of build/conf/, or anything else in build/ is not
> designed for being under version control. You may be able to get away
> with it by coincidence, but it's neither documented nor tested, and so
> any issues are your own.
> 
> I don't have a quick solution to ensure that templates and
> configurations coming from them are strictly in sync, but welcome to
> suggestions. People modify local.conf and bblayers.conf all the time
> as well for local experiments, you're simply not supposed to lock them
> down.

There's no problem with modifying them for local experiments, even if they
are under version control, but it's clear when those experiments need to be
committed that those files need to be committed too.

However, it's clear that I'm pushing against the tide. I'll stick with the
early-return hack in devtool for the short term to see if we come across
any other reasons to make changes or any other breakage. Once everything is
working I'll investigate our options for not keeping bblayers.conf under
version control.

Thanks for you help.

Mike.

Re: [OE-core] Using devtool without an absolute path to the workspace in bblayers.conf

[Khem Raj]

On Wed, Oct 18, 2023 at 7:29 AM Mike Crowe via lists.openembedded.org
<mac=mcrowe.com@lists.openembedded.org> wrote:
>
> I'm trying to work out how we can make use of devtool to make our lives
> easier during development. In general it seems to work very well, but the
> way that it modifies bblayers.conf to add an absolute path to the workspace
> directory to BBLAYERS is incompatible with that file being held in a Git
> repository and shared by multiple users in multiple trees. There's a high
> risk that the file will accidentally be committed containing a path that's
> only meaningful for a single user in a single source tree. All of our other
> paths in bblayers.conf are relative to a variable that contains the path to
> the top of the source tree.
>
> I can manually add ${TOPDIR}/workspace to BBLAYERS in bblayers.conf and
> commit that, but unfortunately devtool insists on continuing to add the
> absolute path since it doesn't know that's the same as ${TOPDIR}/workspace.
>
> I can set "workspace_path = workspace" in devtool.conf to use a relative
> path, and that superficially works until externalsrc.bbclass gets upset
> that the EXTERNALSRC is not an absolute path.
>
> I've tried teaching devtool to prepend ${TOPDIR} if the workspace directory
> is not absolute when writing bblayers.conf, but it looks like I'd also need
> to do so in many other places.
>
> My current workaround is just to add ${TOPDIR}/workspace to the committed
> bblayers.conf myself and nobble devtool's _enable_workspace_layer with an
> early return. This is ugly and we'd have to carry that change around
> forever.
>
> Since I'm clearly swimming against the tide I'm left wondering whether I've
> missed something. Is there a way to use devtool without having an absolute
> path to the workspace in bblayers.conf?
>
> (At the moment I'm still using dunfell, but I looked at current master and
> didn't spot anything that looked like it changed this behaviour.)

Perhaps an enhancement to not add the path if its already part of BBLAYERS
might do it. You might also print a diagnostics about its pre-existence to make
sure it's not accidental.

>
> Thanks.
>
> Mike.
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#189396): https://lists.openembedded.org/g/openembedded-core/message/189396
> Mute This Topic: https://lists.openembedded.org/mt/102039990/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] Using devtool without an absolute path to the workspace in bblayers.conf

[Mike Crowe]

On Wednesday 18 October 2023 at 09:19:32 -0700, Khem Raj wrote:
> On Wed, Oct 18, 2023 at 7:29 AM Mike Crowe via lists.openembedded.org
> <mac=mcrowe.com@lists.openembedded.org> wrote:
> >
> > I'm trying to work out how we can make use of devtool to make our lives
> > easier during development. In general it seems to work very well, but the
> > way that it modifies bblayers.conf to add an absolute path to the workspace
> > directory to BBLAYERS is incompatible with that file being held in a Git
> > repository and shared by multiple users in multiple trees. There's a high
> > risk that the file will accidentally be committed containing a path that's
> > only meaningful for a single user in a single source tree. All of our other
> > paths in bblayers.conf are relative to a variable that contains the path to
> > the top of the source tree.
> >
> > I can manually add ${TOPDIR}/workspace to BBLAYERS in bblayers.conf and
> > commit that, but unfortunately devtool insists on continuing to add the
> > absolute path since it doesn't know that's the same as ${TOPDIR}/workspace.
> >
> > I can set "workspace_path = workspace" in devtool.conf to use a relative
> > path, and that superficially works until externalsrc.bbclass gets upset
> > that the EXTERNALSRC is not an absolute path.
> >
> > I've tried teaching devtool to prepend ${TOPDIR} if the workspace directory
> > is not absolute when writing bblayers.conf, but it looks like I'd also need
> > to do so in many other places.
> >
> > My current workaround is just to add ${TOPDIR}/workspace to the committed
> > bblayers.conf myself and nobble devtool's _enable_workspace_layer with an
> > early return. This is ugly and we'd have to carry that change around
> > forever.
> >
> > Since I'm clearly swimming against the tide I'm left wondering whether I've
> > missed something. Is there a way to use devtool without having an absolute
> > path to the workspace in bblayers.conf?
> >
> > (At the moment I'm still using dunfell, but I looked at current master and
> > didn't spot anything that looked like it changed this behaviour.)
> 
> Perhaps an enhancement to not add the path if its already part of BBLAYERS
> might do it. You might also print a diagnostics about its pre-existence to make
> sure it's not accidental.

devtool already tries to do that. It doesn't work in this case since the
code compares the absolute path against the paths in bblayers.conf and
isn't in a position to expand variables at that point.

Thanks.

Mike.

Re: [OE-core] Using devtool without an absolute path to the workspace in bblayers.conf

[Khem Raj]

On Wed, Oct 18, 2023 at 9:28 AM Mike Crowe <mac@mcrowe.com> wrote:
>
> On Wednesday 18 October 2023 at 09:19:32 -0700, Khem Raj wrote:
> > On Wed, Oct 18, 2023 at 7:29 AM Mike Crowe via lists.openembedded.org
> > <mac=mcrowe.com@lists.openembedded.org> wrote:
> > >
> > > I'm trying to work out how we can make use of devtool to make our lives
> > > easier during development. In general it seems to work very well, but the
> > > way that it modifies bblayers.conf to add an absolute path to the workspace
> > > directory to BBLAYERS is incompatible with that file being held in a Git
> > > repository and shared by multiple users in multiple trees. There's a high
> > > risk that the file will accidentally be committed containing a path that's
> > > only meaningful for a single user in a single source tree. All of our other
> > > paths in bblayers.conf are relative to a variable that contains the path to
> > > the top of the source tree.
> > >
> > > I can manually add ${TOPDIR}/workspace to BBLAYERS in bblayers.conf and
> > > commit that, but unfortunately devtool insists on continuing to add the
> > > absolute path since it doesn't know that's the same as ${TOPDIR}/workspace.
> > >
> > > I can set "workspace_path = workspace" in devtool.conf to use a relative
> > > path, and that superficially works until externalsrc.bbclass gets upset
> > > that the EXTERNALSRC is not an absolute path.
> > >
> > > I've tried teaching devtool to prepend ${TOPDIR} if the workspace directory
> > > is not absolute when writing bblayers.conf, but it looks like I'd also need
> > > to do so in many other places.
> > >
> > > My current workaround is just to add ${TOPDIR}/workspace to the committed
> > > bblayers.conf myself and nobble devtool's _enable_workspace_layer with an
> > > early return. This is ugly and we'd have to carry that change around
> > > forever.
> > >
> > > Since I'm clearly swimming against the tide I'm left wondering whether I've
> > > missed something. Is there a way to use devtool without having an absolute
> > > path to the workspace in bblayers.conf?
> > >
> > > (At the moment I'm still using dunfell, but I looked at current master and
> > > didn't spot anything that looked like it changed this behaviour.)
> >
> > Perhaps an enhancement to not add the path if its already part of BBLAYERS
> > might do it. You might also print a diagnostics about its pre-existence to make
> > sure it's not accidental.
>
> devtool already tries to do that. It doesn't work in this case since the
> code compares the absolute path against the paths in bblayers.conf and
> isn't in a position to expand variables at that point.

yeah it perhaps has to compare the evaluated paths

>
> Thanks.
>
> Mike.