[PATCH 1/4] devtool: account for sources in UNPACKDIR

[PATCH 1/4] devtool: account for sources in UNPACKDIR

[Alexander Kanavin]

From: Alexander Kanavin <alex@linutronix.de>

There's a couple of assumptions in devtool that sources
are always in a directory under WORKDIR; this is no longer
the case since introduction of UNPACKDIR: some recipes
are starting to use

S = "${UNPACKDIR}/git"

or similar, and so the logic to determine source locations
needs to be tweaked accordingly.

The issue is that oe-core has no concept of 'top level source
path' (S points to where the build is started from, inside that
top level location), and yet devtool needs to know that in order
to move the complete source tree correctly to a workspace (and possibly
other uses).

And so devtool performs convoluted path calculations from WORKDIR and S;
now this has been extended to include UNPACKDIR and became more convoluted
but hopefully it won't get any worse.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/classes/devtool-source.bbclass |  8 ++++++--
 scripts/lib/devtool/ide_sdk.py      |  2 +-
 scripts/lib/devtool/standard.py     | 25 +++++++++++++++++++------
 scripts/lib/devtool/upgrade.py      |  2 +-
 4 files changed, 27 insertions(+), 10 deletions(-)

diff --git a/meta/classes/devtool-source.bbclass b/meta/classes/devtool-source.bbclass
index 9762003ba75..d4e307e886b 100644
--- a/meta/classes/devtool-source.bbclass
+++ b/meta/classes/devtool-source.bbclass
@@ -92,9 +92,13 @@ python devtool_post_unpack() {
             for fname in local_files:
                 f.write('%s\n' % fname)
 
-    if os.path.dirname(srcsubdir) != workdir:
+    if srcsubdir.startswith(unpackdir) and srcsubdir != unpackdir:
+        srcparentdir = unpackdir
+    else:
+        srcparentdir = workdir
+    if os.path.dirname(srcsubdir) != srcparentdir:
         # Handle if S is set to a subdirectory of the source
-        srcsubdir = os.path.join(workdir, os.path.relpath(srcsubdir, workdir).split(os.sep)[0])
+        srcsubdir = os.path.join(srcparentdir, os.path.relpath(srcsubdir, srcparentdir).split(os.sep)[0])
 
     scriptutils.git_convert_standalone_clone(srcsubdir)
 
diff --git a/scripts/lib/devtool/ide_sdk.py b/scripts/lib/devtool/ide_sdk.py
index f8cf65f4a84..e1717bc4ab5 100755
--- a/scripts/lib/devtool/ide_sdk.py
+++ b/scripts/lib/devtool/ide_sdk.py
@@ -334,7 +334,7 @@ class RecipeModified:
         self.srctree = workspace[workspacepn]['srctree']
         # Need to grab this here in case the source is within a subdirectory
         self.real_srctree = get_real_srctree(
-            self.srctree, recipe_d.getVar('S'), recipe_d.getVar('WORKDIR'))
+            self.srctree, recipe_d.getVar('S'), recipe_d.getVar('WORKDIR'), recipe_d.getVar('UNPACKDIR'))
         self.bbappend = workspace[workspacepn]['bbappend']
 
         self.ide_sdk_dir = os.path.join(
diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index cdfdba43eef..9a8f990d84a 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -625,7 +625,12 @@ def _extract_source(srctree, keep_temp, devbranch, sync, config, basepath, works
                 srcsubdir = f.read()
         except FileNotFoundError as e:
             raise DevtoolError('Something went wrong with source extraction - the devtool-source class was not active or did not function correctly:\n%s' % str(e))
-        srcsubdir_rel = os.path.relpath(srcsubdir, os.path.join(tempdir, 'workdir'))
+        unpackdir = d.getVar('UNPACKDIR')
+        if d.getVar('S').startswith(d.getVar('UNPACKDIR')):
+            srcparentdir = os.path.relpath(d.getVar('UNPACKDIR'), d.getVar('WORKDIR'))
+        else:
+            srcparentdir = ''
+        srcsubdir_rel = os.path.relpath(srcsubdir, os.path.join(tempdir, 'workdir', srcparentdir))
 
         # Check if work-shared is empty, if yes
         # find source and copy to work-shared
@@ -742,14 +747,22 @@ def get_staging_kbranch(srcdir):
         staging_kbranch = "".join(branch.split('\n')[0])
     return staging_kbranch
 
-def get_real_srctree(srctree, s, workdir):
+def get_real_srctree(srctree, s, workdir, unpackdir):
     # Check that recipe isn't using a shared workdir
     s = os.path.abspath(s)
     workdir = os.path.abspath(workdir)
-    if s.startswith(workdir) and s != workdir and os.path.dirname(s) != workdir:
+    unpackdir = os.path.abspath(unpackdir)
+
+    if s.startswith(workdir) and s != workdir:
         # Handle if S is set to a subdirectory of the source
-        srcsubdir = os.path.relpath(s, workdir).split(os.sep, 1)[1]
-        srctree = os.path.join(srctree, srcsubdir)
+        if s.startswith(unpackdir) and s != unpackdir:
+            srcparentdir = unpackdir
+        else:
+            srcparentdir = workdir
+
+        if os.path.dirname(s) != srcparentdir:
+            srcsubdir = os.path.relpath(s, srcparentdir).split(os.sep, 1)[1]
+            srctree = os.path.join(srctree, srcsubdir)
     return srctree
 
 def modify(args, config, basepath, workspace):
@@ -907,7 +920,7 @@ def modify(args, config, basepath, workspace):
 
         # Need to grab this here in case the source is within a subdirectory
         srctreebase = srctree
-        srctree = get_real_srctree(srctree, rd.getVar('S'), rd.getVar('WORKDIR'))
+        srctree = get_real_srctree(srctree, rd.getVar('S'), rd.getVar('WORKDIR'), rd.getVar('UNPACKDIR'))
 
         bb.utils.mkdirhier(os.path.dirname(appendfile))
         with open(appendfile, 'w') as f:
diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index 0dace1fb240..e584e03e17c 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -571,7 +571,7 @@ def upgrade(args, config, basepath, workspace):
         else:
             srctree = standard.get_default_srctree(config, pn)
 
-        srctree_s = standard.get_real_srctree(srctree, rd.getVar('S'), rd.getVar('WORKDIR'))
+        srctree_s = standard.get_real_srctree(srctree, rd.getVar('S'), rd.getVar('WORKDIR'), rd.getVar('UNPACKDIR'))
 
         # try to automatically discover latest version and revision if not provided on command line
         if not args.version and not args.srcrev:
-- 
2.39.5

[PATCH 2/4] tcf-agent: update SRC_URI to https://gitlab.eclipse.org

[Alexander Kanavin]

From: Alexander Kanavin <alex@linutronix.de>

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/recipes-devtools/tcf-agent/tcf-agent_git.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/tcf-agent/tcf-agent_git.bb b/meta/recipes-devtools/tcf-agent/tcf-agent_git.bb
index e1ac0a29a7a..235936288ba 100644
--- a/meta/recipes-devtools/tcf-agent/tcf-agent_git.bb
+++ b/meta/recipes-devtools/tcf-agent/tcf-agent_git.bb
@@ -10,7 +10,7 @@ SRCREV = "1f11747e83ebf4f53e8d17f430136f92ec378709"
 PV = "1.8.0+git"
 
 UPSTREAM_CHECK_GITTAGREGEX = "(?P<pver>(\d+(\.\d+)+))"
-SRC_URI = "git://git.eclipse.org/r/tcf/org.eclipse.tcf.agent.git;protocol=https;branch=master \
+SRC_URI = "git://gitlab.eclipse.org/eclipse/tcf/tcf.agent.git;protocol=https;branch=master \
            file://ldflags.patch \
            file://tcf-agent.init \
            file://tcf-agent.service \
-- 
2.39.5

[PATCH 3/4] libarchive: correct upstream version check

[Alexander Kanavin]

From: Alexander Kanavin <alex@linutronix.de>

File index works now:
http://libarchive.org/downloads/

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/recipes-extended/libarchive/libarchive_3.7.9.bb | 2 --
 1 file changed, 2 deletions(-)

diff --git a/meta/recipes-extended/libarchive/libarchive_3.7.9.bb b/meta/recipes-extended/libarchive/libarchive_3.7.9.bb
index 9d134f7d381..d70cdb3d835 100644
--- a/meta/recipes-extended/libarchive/libarchive_3.7.9.bb
+++ b/meta/recipes-extended/libarchive/libarchive_3.7.9.bb
@@ -31,8 +31,6 @@ EXTRA_OECONF += "--enable-largefile --without-iconv"
 
 SRC_URI = "https://libarchive.org/downloads/libarchive-${PV}.tar.gz"
 
-UPSTREAM_CHECK_URI = "http://libarchive.org/"
-
 SRC_URI[sha256sum] = "aa90732c5a6bdda52fda2ad468ac98d75be981c15dde263d7b5cf6af66fd009f"
 
 inherit autotools update-alternatives pkgconfig
-- 
2.39.5

[PATCH 4/4] libunwind: use github-releases class to determine latest versions

[Alexander Kanavin]

From: Alexander Kanavin <alex@linutronix.de>

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/recipes-support/libunwind/libunwind_1.8.1.bb | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-support/libunwind/libunwind_1.8.1.bb b/meta/recipes-support/libunwind/libunwind_1.8.1.bb
index 1885da4d09f..ee4a8401ba6 100644
--- a/meta/recipes-support/libunwind/libunwind_1.8.1.bb
+++ b/meta/recipes-support/libunwind/libunwind_1.8.1.bb
@@ -5,7 +5,8 @@ LICENSE = "MIT"
 LIC_FILES_CHKSUM = "file://COPYING;md5=2d80c8ed4062b8339b715f90fa68cc9f"
 DEPENDS:append:libc-musl = " libucontext"
 
-SRC_URI = "https://github.com/libunwind/libunwind/releases/download/v${PV}/${BP}.tar.gz \
+GITHUB_BASE_URI = "https://github.com/libunwind/libunwind/releases"
+SRC_URI = "${GITHUB_BASE_URI}/download/v${PV}/${BP}.tar.gz \
            file://mips-byte-order.patch \
            file://0001-tests-Garm64-test-sve-signal-check-that-SVE-is-prese.patch \
            file://0002-coredump-use-glibc-or-musl-register-names-as-appropr.patch \
@@ -16,7 +17,7 @@ SRC_URI = "https://github.com/libunwind/libunwind/releases/download/v${PV}/${BP}
 
 SRC_URI[sha256sum] = "ddf0e32dd5fafe5283198d37e4bf9decf7ba1770b6e7e006c33e6df79e6a6157"
 
-inherit autotools multilib_header
+inherit autotools multilib_header github-releases
 
 COMPATIBLE_HOST:riscv32 = "null"
 
-- 
2.39.5

Re: [OE-core] [PATCH 1/4] devtool: account for sources in UNPACKDIR

[Richard Purdie]

On Thu, 2025-04-24 at 13:10 +0200, Alexander Kanavin via lists.openembedded.org wrote:
> From: Alexander Kanavin <alex@linutronix.de>
> 
> There's a couple of assumptions in devtool that sources
> are always in a directory under WORKDIR; this is no longer
> the case since introduction of UNPACKDIR: some recipes
> are starting to use
> 
> S = "${UNPACKDIR}/git"
> 
> or similar, and so the logic to determine source locations
> needs to be tweaked accordingly.
> 
> The issue is that oe-core has no concept of 'top level source
> path' (S points to where the build is started from, inside that
> top level location), and yet devtool needs to know that in order
> to move the complete source tree correctly to a workspace (and possibly
> other uses).
> 
> And so devtool performs convoluted path calculations from WORKDIR and S;
> now this has been extended to include UNPACKDIR and became more convoluted
> but hopefully it won't get any worse.
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  meta/classes/devtool-source.bbclass |  8 ++++++--
>  scripts/lib/devtool/ide_sdk.py      |  2 +-
>  scripts/lib/devtool/standard.py     | 25 +++++++++++++++++++------
>  scripts/lib/devtool/upgrade.py      |  2 +-
>  4 files changed, 27 insertions(+), 10 deletions(-)

UNPACKDIR was meant to help with this. If devtool is changing the
location, can't it just change UNPACKDIR and leave everything else
alone? S will still be under UNPACKDIR/git even after devtool moves it?

Cheers,

Richard

Re: [OE-core] [PATCH 1/4] devtool: account for sources in UNPACKDIR

[Alexander Kanavin]

On Thu, 24 Apr 2025 at 13:32, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:

> UNPACKDIR was meant to help with this. If devtool is changing the
> location, can't it just change UNPACKDIR and leave everything else
> alone? S will still be under UNPACKDIR/git even after devtool moves it?

Some recipes use
S = /path/to/workdir/git/sub/path
and some use
S = /path/to/workdir/unpackdir/git/sub/path

It's that 'git' part in the middle that needs to be extracted from the
final value of S and appended to UNPACKDIR (or WORKDIR), and then the
resulting path is moved by devtool to a workspace with shutil. I
probably don't fully understand the suggestion?

Alex

Re: [OE-core] [PATCH 1/4] devtool: account for sources in UNPACKDIR

[Richard Purdie]

On Thu, 2025-04-24 at 13:44 +0200, Alexander Kanavin wrote:
> On Thu, 24 Apr 2025 at 13:32, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> 
> > UNPACKDIR was meant to help with this. If devtool is changing the
> > location, can't it just change UNPACKDIR and leave everything else
> > alone? S will still be under UNPACKDIR/git even after devtool moves
> > it?
> 
> Some recipes use
> S = /path/to/workdir/git/sub/path
> and some use
> S = /path/to/workdir/unpackdir/git/sub/path
> 
> It's that 'git' part in the middle that needs to be extracted from
> the final value of S and appended to UNPACKDIR (or WORKDIR), and then
> the resulting path is moved by devtool to a workspace with shutil. I
> probably don't fully understand the suggestion?

In the two examples above, what is UNPACKDIR set to?

Cheers,

Richard

Re: [OE-core] [PATCH 1/4] devtool: account for sources in UNPACKDIR

[Richard Purdie]

On Thu, 2025-04-24 at 13:13 +0100, Richard Purdie via lists.openembedded.org wrote:
> On Thu, 2025-04-24 at 13:44 +0200, Alexander Kanavin wrote:
> > On Thu, 24 Apr 2025 at 13:32, Richard Purdie
> > <richard.purdie@linuxfoundation.org> wrote:
> > 
> > > UNPACKDIR was meant to help with this. If devtool is changing the
> > > location, can't it just change UNPACKDIR and leave everything else
> > > alone? S will still be under UNPACKDIR/git even after devtool moves
> > > it?
> > 
> > Some recipes use
> > S = /path/to/workdir/git/sub/path
> > and some use
> > S = /path/to/workdir/unpackdir/git/sub/path
> > 
> > It's that 'git' part in the middle that needs to be extracted from
> > the final value of S and appended to UNPACKDIR (or WORKDIR), and then
> > the resulting path is moved by devtool to a workspace with shutil. I
> > probably don't fully understand the suggestion?
> 
> In the two examples above, what is UNPACKDIR set to?

Sorry, let me be more specific. What are the actual values of S and
WORKDIR for the different cases?

Cheers,

Richard

Re: [OE-core] [PATCH 1/4] devtool: account for sources in UNPACKDIR

[Alexander Kanavin]

On Thu, 24 Apr 2025 at 14:14, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> > In the two examples above, what is UNPACKDIR set to?
>
> Sorry, let me be more specific. What are the actual values of S and
> WORKDIR for the different cases?

Can we step back for a bit? The original error is that 'devtool
upgrade' breaks cryptically on recipes that set:

S = "${UNPACKDIR}/git" (instead of the usual WORKDIR/git).

(everything else being the defaults from oe-core).

Devtool, in this case, calculates the 'top level source' to be the
same as UNPACKDIR, and breakage follows.

An example of such recipe in core is
meta/recipes-devtools/rpm-sequoia/rpm-sequoia_1.7.0.bb

Alex

Re: [OE-core] [PATCH 1/4] devtool: account for sources in UNPACKDIR

[Richard Purdie]

On Thu, 2025-04-24 at 14:41 +0200, Alexander Kanavin wrote:
> On Thu, 24 Apr 2025 at 14:14, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > > In the two examples above, what is UNPACKDIR set to?
> > 
> > Sorry, let me be more specific. What are the actual values of S and
> > WORKDIR for the different cases?
> 
> Can we step back for a bit? The original error is that 'devtool
> upgrade' breaks cryptically on recipes that set:
> 
> S = "${UNPACKDIR}/git" (instead of the usual WORKDIR/git).
> 
> (everything else being the defaults from oe-core).
> 
> Devtool, in this case, calculates the 'top level source' to be the
> same as UNPACKDIR, and breakage follows.
> 
> An example of such recipe in core is
> meta/recipes-devtools/rpm-sequoia/rpm-sequoia_1.7.0.bb

Stepping back further, we had this mess in devtool in the first place
as it could never work out which files in WORKDIR were "sources" which
it needed to handle carefully.

We now define UNPACKDIR as the place we unpack files to. devtool in
theory can therefore just redefine UNPACKDIR and WORKDIR and things
should 'just work'. The relation of the subpaths between the two should
remain the same.

This should therefore make the code in devtool simpler, not more
complex.

Your patch isn't doing that and I'm wondering if there are code paths
in there which are no longer needed if we handle UNPACKDIR correctly/as
intended.

So in summary, something isn't adding up here to me but I'm not yet
sure quite what.

Cheers,

Richard

Re: [OE-core] [PATCH 1/4] devtool: account for sources in UNPACKDIR

[Adrian Freihofer]

[-- Attachment #1: Type: text/plain, Size: 2772 bytes --]

Richard Purdie via lists.openembedded.org <richard.purdie=
linuxfoundation.org@lists.openembedded.org> schrieb am Do., 24. Apr. 2025,
15:50:

> On Thu, 2025-04-24 at 14:41 +0200, Alexander Kanavin wrote:
> > On Thu, 24 Apr 2025 at 14:14, Richard Purdie
> > <richard.purdie@linuxfoundation.org> wrote:
> > > > In the two examples above, what is UNPACKDIR set to?
> > >
> > > Sorry, let me be more specific. What are the actual values of S and
> > > WORKDIR for the different cases?
> >
> > Can we step back for a bit? The original error is that 'devtool
> > upgrade' breaks cryptically on recipes that set:
> >
> > S = "${UNPACKDIR}/git" (instead of the usual WORKDIR/git).
> >
> > (everything else being the defaults from oe-core).
> >
> > Devtool, in this case, calculates the 'top level source' to be the
> > same as UNPACKDIR, and breakage follows.
> >
> > An example of such recipe in core is
> > meta/recipes-devtools/rpm-sequoia/rpm-sequoia_1.7.0.bb
>
> Stepping back further, we had this mess in devtool in the first place
> as it could never work out which files in WORKDIR were "sources" which
> it needed to handle carefully.
>
> We now define UNPACKDIR as the place we unpack files to. devtool in
> theory can therefore just redefine UNPACKDIR and WORKDIR and things
> should 'just work'. The relation of the subpaths between the two should
> remain the same.
>
> This should therefore make the code in devtool simpler, not more
> complex.
>
> Your patch isn't doing that and I'm wondering if there are code paths
> in there which are no longer needed if we handle UNPACKDIR correctly/as
> intended.
>
> So in summary, something isn't adding up here to me but I'm not yet
> sure quite what.
>

At the moment I'm not able to look into the details. But I remember the
extra
complexity in the code of devtool for handling the corner case when
S=WORKDIR
with an extra folder containing sym-links to the real source files. This is
now
removed thanks to UNPACKDIR which no longer allows this. At least for
ide_sdk
this looked like a huge cleanup to me.
Maybe now it is possible to clearly define how devtool uses or maps S,
UNPACKDIR
and WORKDIR in a generic way rather than trying to fix the code in some way
which
just works. Or is there already such a concept behind this patch?

Adrian

>
>
> Cheers,
>
> Richard
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#215373):
> https://lists.openembedded.org/g/openembedded-core/message/215373
> Mute This Topic: https://lists.openembedded.org/mt/112431116/4454582
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> adrian.freihofer@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 4491 bytes --]

Re: [OE-core] [PATCH 1/4] devtool: account for sources in UNPACKDIR

[Alexander Kanavin]

On Thu, 24 Apr 2025 at 19:36, Adrian Freihofer
<adrian.freihofer@gmail.com> wrote:
> Maybe now it is possible to clearly define how devtool uses or maps S, UNPACKDIR
> and WORKDIR in a generic way rather than trying to fix the code in some way which
> just works. Or is there already such a concept behind this patch?

I've just described to RP how devtool figures out what to place into
workspace/sources/. I'm not sure how we could do something different
than what this patch does, without breaking the layout of that
directory, and all tools that expect the current layout with it. We
could, for example, just copy UNPACKDIR into workspace/sources/${PN} -
simple but completely incompatible with what happens now. And it would
introduce ambiguity and inconsistency into where the source tree is in
there for anyone trying to edit source files.

Alex

Re: [OE-core] [PATCH 1/4] devtool: account for sources in UNPACKDIR

[Alexander Kanavin]

On Thu, 24 Apr 2025 at 15:50, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> Stepping back further, we had this mess in devtool in the first place
> as it could never work out which files in WORKDIR were "sources" which
> it needed to handle carefully.
>
> We now define UNPACKDIR as the place we unpack files to. devtool in
> theory can therefore just redefine UNPACKDIR and WORKDIR and things
> should 'just work'. The relation of the subpaths between the two should
> remain the same.

I still struggle to understand this argument. As far as I see, to
determine where is the 'source tree', devtool cannot avoid looking at
both WORKDIR and S (and now, UNPACKDIR too) and then calculate
something reasonable out of both (and with UNPACKDIR in the mix, all
three).

Let me describe this step by step what happens without the patch.

1. devtool manages a 'workspace': a special layer under
build/workspace/. Part of the workspace is sources/: a place where
source trees are placed. When a recipe is taken into workspace, its
source tree is placed under build/workspace/sources/${PN}/.

2. How does devtool determine where to take the source tree from? It
sets WORKDIR to a temporary location under the real oe-core workdir,
and then performs an unpack operation. Then it, somehow, needs to find
the source tree under that temporary WORKDIR. It cannot simply use S:
this can be set to a nested sub-directory deep inside the actual
source tree. It cannot use WORKDIR either: this is not the source tree
in almost all cases (and is now forbidden anyway).

3. Let's say WORKDIR is set to /some/path/to/workdir, and S is then
/some/path/to/workdir/git/some/subpath/. (via S =
"${WORKDIR}/git/some/subpath")

devtool does this:

a) use os.path.relpath on WORKDIR and S to obtain git/some/subpath part
b) then it splits that part into individual directories and take the
first one: 'git'
c) then it's appended to WORKDIR: /some/path/to/workdir/git
d) then devtool uses shutil.move to move/rename that into workspace,
so /some/path/to/workdir/git becomes build/workspace/sources/${PN}

All fine, so far. What happens if the recipe sets S =
"${UNPACKDIR}/git/some/subpath? The same calculation as in point three
will yield UNPACKDIR (incorrect), instead of UNPACKDIR/git (correct).
And this is what the patch is aiming to address: it changes the code
to first check if S is under UNPACKDIR, and if so, performs the
calculation in point three relative to UNPACKDIR. Otherwise, it's
performed relative to WORKDIR, as before.

The argument I can't understand is that devtool can somehow avoid
looking at S altogether, and achieve the right thing by just
redefining UNPACKDIR and WORKDIR and removing the logic described
above, but how? How would we end up with the correct source tree in
the workspace then?

Alex

Re: [OE-core] [PATCH 1/4] devtool: account for sources in UNPACKDIR

[Richard Purdie]

On Thu, 2025-04-24 at 21:36 +0200, Alexander Kanavin wrote:
> On Thu, 24 Apr 2025 at 15:50, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > Stepping back further, we had this mess in devtool in the first
> > place
> > as it could never work out which files in WORKDIR were "sources"
> > which
> > it needed to handle carefully.
> > 
> > We now define UNPACKDIR as the place we unpack files to. devtool in
> > theory can therefore just redefine UNPACKDIR and WORKDIR and things
> > should 'just work'. The relation of the subpaths between the two
> > should
> > remain the same.
> 
> I still struggle to understand this argument. As far as I see, to
> determine where is the 'source tree', devtool cannot avoid looking at
> both WORKDIR and S (and now, UNPACKDIR too) and then calculate
> something reasonable out of both (and with UNPACKDIR in the mix, all
> three).
> 
> Let me describe this step by step what happens without the patch.
> 
> 1. devtool manages a 'workspace': a special layer under
> build/workspace/. Part of the workspace is sources/: a place where
> source trees are placed. When a recipe is taken into workspace, its
> source tree is placed under build/workspace/sources/${PN}/.
> 
> 2. How does devtool determine where to take the source tree from? It
> sets WORKDIR to a temporary location under the real oe-core workdir,
> and then performs an unpack operation.

Why can't it set UNPACKDIR to the the workspace path and then just run
it to directly unpack to where it wants it?

It can then redefine UNPACKDIR to point at the workspace path and
things should "just work"?

>  Then it, somehow, needs to find
> the source tree under that temporary WORKDIR. It cannot simply use S:
> this can be set to a nested sub-directory deep inside the actual
> source tree. It cannot use WORKDIR either: this is not the source
> tree in almost all cases (and is now forbidden anyway).

It can however use UNPACKDIR?
> 
> 3. Let's say WORKDIR is set to /some/path/to/workdir, and S is then
> /some/path/to/workdir/git/some/subpath/. (via S =
> "${WORKDIR}/git/some/subpath")
> 
> devtool does this:
> 
> a) use os.path.relpath on WORKDIR and S to obtain git/some/subpath
> part
> b) then it splits that part into individual directories and take the
> first one: 'git'
> c) then it's appended to WORKDIR: /some/path/to/workdir/git
> d) then devtool uses shutil.move to move/rename that into workspace,
> so /some/path/to/workdir/git becomes build/workspace/sources/${PN}
> 
> All fine, so far. What happens if the recipe sets S =
> "${UNPACKDIR}/git/some/subpath? The same calculation as in point
> three
> will yield UNPACKDIR (incorrect), instead of UNPACKDIR/git (correct).
> And this is what the patch is aiming to address: it changes the code
> to first check if S is under UNPACKDIR, and if so, performs the
> calculation in point three relative to UNPACKDIR. Otherwise, it's
> performed relative to WORKDIR, as before.
> 
> The argument I can't understand is that devtool can somehow avoid
> looking at S altogether, and achieve the right thing by just
> redefining UNPACKDIR and WORKDIR and removing the logic described
> above, but how? How would we end up with the correct source tree in
> the workspace then?

A recipe doesn't have a single source tree but a set of sources, which
is the contents of UNPACKDIR. do_unpack has some horrible magic moving
which moves the sources into the expected place for the magic value of
S (see the SOURCE_BASEDIR mess) and this is perhaps confusing things.

In my mind, I'm thinking we ultimately stop that move and put a symlink
in place instead for compatibility. There were some issues with
autotools and symlinks when I last tried iirc but I'm digressing...

From memory, one of the devtool issues was that it couldn't track and
customise some of the input sources such as standalone files. The
UNPACKDIR changes were intended to give us a path forward so we could
actually handle the full set of sources rather than just the "S"
contents.

So I still believe there is some kind of simplification/improvement we
probably can make here...

Cheers,

Richard

Re: [OE-core] [PATCH 1/4] devtool: account for sources in UNPACKDIR

[Alexander Kanavin]

On Thu, 24 Apr 2025 at 22:34, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> Why can't it set UNPACKDIR to the the workspace path and then just run
> it to directly unpack to where it wants it?
>
> It can then redefine UNPACKDIR to point at the workspace path and
> things should "just work"?

It can. But it will fundamentally change what users are going to get
under workspace/sources/${PN} and not necessarily for the better:

1. Right now, if you 'cd workspace/sources/${PN}', you'll see a source
tree that matches upstream repository (or tarball) exactly, for the
unpacked item that contains ${S} in it. It's obvious what to do: you
can modify files, run bitbake with these modifications, make commits
and devtool will pick them up and integrate into the recipe.

Yes, technically there can be more than one upstream item listed in
SRC_URI (I'm not sure what devtool is going to do when it sees them),
but the vast majority of recipes have just one item, and this keeps
things simple and straightforward for the users (and tools). The cost
of it: devtool is a convoluted mess (and to make matters worse,
without a maintainer); I don't argue with that.

2. What happens if workspace/sources/${PN} becomes UNPACKDIR? When you
go there, you'll see something named 'git/'. Is that where the actual
source is? If the recipe is fetching a tarball, there's no git/, but
rather ${PN}-${PV}/. So there's an extra, potentially confusing step
to get to the source code. It becomes more ambiguous and less
consistent for the users.

3. This will also break all the tools that assume the current
structure, and another reason this change shouldn't be taken lightly.

4. Maybe there's something I missed that could be done, and won't
break the current layout? For example, we can set UNPACKDIR to
workspace/unpack/${PN}, and then use devtool 'logic' to provide a
hardlink into it from workspace/sources/${PN} ? But it would be the
same logic as now with the patch: take original values of WORKDIR,
UNPACKDIR and S, and figure out where in the new UNPACKDIR to point
to.

Alex

Re: [OE-core] [PATCH 1/4] devtool: account for sources in UNPACKDIR

[Richard Purdie]

On Fri, 2025-04-25 at 10:33 +0200, Alexander Kanavin wrote:
> On Thu, 24 Apr 2025 at 22:34, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > Why can't it set UNPACKDIR to the the workspace path and then just
> > run
> > it to directly unpack to where it wants it?
> > 
> > It can then redefine UNPACKDIR to point at the workspace path and
> > things should "just work"?
> 
> It can. But it will fundamentally change what users are going to get
> under workspace/sources/${PN} and not necessarily for the better:
> 
> 1. Right now, if you 'cd workspace/sources/${PN}', you'll see a
> source
> tree that matches upstream repository (or tarball) exactly, for the
> unpacked item that contains ${S} in it. It's obvious what to do: you
> can modify files, run bitbake with these modifications, make commits
> and devtool will pick them up and integrate into the recipe.
> 
> Yes, technically there can be more than one upstream item listed in
> SRC_URI (I'm not sure what devtool is going to do when it sees them),
> but the vast majority of recipes have just one item, and this keeps
> things simple and straightforward for the users (and tools). The cost
> of it: devtool is a convoluted mess (and to make matters worse,
> without a maintainer); I don't argue with that.

There are continual requests to have devtool able to operate on recipes
with multiple SRC_URI entries, be able to see/modify files, support
things like gcc and the kernel which have interesting build layouts and
so on. I thought we'd already added support for some of this which is
probably why the logic is already convoluted. I suspect it will only
get worse.

> 2. What happens if workspace/sources/${PN} becomes UNPACKDIR? When
> you go there, you'll see something named 'git/'. Is that where the
> actual source is? If the recipe is fetching a tarball, there's no
> git/, but rather ${PN}-${PV}/. So there's an extra, potentially
> confusing step to get to the source code. It becomes more ambiguous
> and less consistent for the users.

It does match what happens in a real build in WORKDIR. Where you have a
subdir of a git repo, this is already exposed as you don't get S but
need to find the subdir. My point being it is already probably
inconsistent.

> 
> 3. This will also break all the tools that assume the current
> structure, and another reason this change shouldn't be taken lightly.

Perhaps, or perhaps not. Do we have a lot of tools that wouldn't cope
with this?

> 4. Maybe there's something I missed that could be done, and won't
> break the current layout? For example, we can set UNPACKDIR to
> workspace/unpack/${PN}, and then use devtool 'logic' to provide a
> hardlink into it from workspace/sources/${PN} ? But it would be the
> same logic as now with the patch: take original values of WORKDIR,
> UNPACKDIR and S, and figure out where in the new UNPACKDIR to point
> to.

That 'logic' is effectively what do_unpack is already doing itself to
maintain 'compatibility'. Perhaps we could use the same logic in both
places and be consistent!

I'm torn here. I could take your patch, bury my head in the sand and
pretend everything is fine. We could try and actually improve this and
set things in a better direction for the future, at the expense of
breaking things in the short/medium term. I don't know what to do for
the best here to be honest. Taking the patch is way easier for me I
doubt we'll ever improvements if I don't push back on things at times
:/.

(I'm spelling this out so people understand the dilemma)

Cheers,

Richard

Re: [OE-core] [PATCH 1/4] devtool: account for sources in UNPACKDIR

[Alexander Kanavin]

On Fri, 25 Apr 2025 at 11:31, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> > 3. This will also break all the tools that assume the current
> > structure, and another reason this change shouldn't be taken lightly.
>
> Perhaps, or perhaps not. Do we have a lot of tools that wouldn't cope
> with this?

Likely Visual Studio Code support will have to be fixed? Impact of
changes like this is difficult to predict until they're done. Then
everyone comes out complaining their private scripts broke.

Otherwise, I'm slowly getting convinced that this is the right thing :)T

> That 'logic' is effectively what do_unpack is already doing itself to
> maintain 'compatibility'. Perhaps we could use the same logic in both
> places and be consistent!
>
> I'm torn here. I could take your patch, bury my head in the sand and
> pretend everything is fine. We could try and actually improve this and
> set things in a better direction for the future, at the expense of
> breaking things in the short/medium term. I don't know what to do for
> the best here to be honest. Taking the patch is way easier for me I
> doubt we'll ever improvements if I don't push back on things at times
> :/.

Here's something I want to propose, feel free to shoot it down. Should
we start by just dropping support for this legacy compatibility of
sources in workdir, and S = "${WORKDIR}/something"? So S, when set
that way, becomes a hard recipe_qa error, and sources are always in
UNPACKDIR. Yes, there would be a ton of recipes to be fixed, but the
fix would be largely a search-and-replace. Admittedly, I didn't follow
the arguments around UNPACKDIR closely and it may have been talked
about, but why not just do it?

Then this fix for devtool wouldn't add more complexity, it would just
similarly replace workdir with unpackdir - not making things better
but not making them worse either. On the other hand, if we do nothing,
then devtool currently doesn't work for recipes that set S to
unpackdir, negatively impacting AUH rates for one thing (this is why I
got into it in the first place) :-(

Alex

Re: [OE-core] [PATCH 1/4] devtool: account for sources in UNPACKDIR

[Richard Purdie]

On Fri, 2025-04-25 at 19:29 +0200, Alexander Kanavin wrote:
> On Fri, 25 Apr 2025 at 11:31, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > > 3. This will also break all the tools that assume the current
> > > structure, and another reason this change shouldn't be taken lightly.
> > 
> > Perhaps, or perhaps not. Do we have a lot of tools that wouldn't cope
> > with this?
> 
> Likely Visual Studio Code support will have to be fixed? Impact of
> changes like this is difficult to predict until they're done. Then
> everyone comes out complaining their private scripts broke.
> 
> Otherwise, I'm slowly getting convinced that this is the right thing :)

I think you can see how my thoughts ended up here :)

> 
> > That 'logic' is effectively what do_unpack is already doing itself to
> > maintain 'compatibility'. Perhaps we could use the same logic in both
> > places and be consistent!
> > 
> > I'm torn here. I could take your patch, bury my head in the sand and
> > pretend everything is fine. We could try and actually improve this and
> > set things in a better direction for the future, at the expense of
> > breaking things in the short/medium term. I don't know what to do for
> > the best here to be honest. Taking the patch is way easier for me I
> > doubt we'll ever improvements if I don't push back on things at times
> > :/.
> 
> Here's something I want to propose, feel free to shoot it down. Should
> we start by just dropping support for this legacy compatibility of
> sources in workdir, and S = "${WORKDIR}/something"? So S, when set
> that way, becomes a hard recipe_qa error, and sources are always in
> UNPACKDIR. Yes, there would be a ton of recipes to be fixed, but the
> fix would be largely a search-and-replace. Admittedly, I didn't follow
> the arguments around UNPACKDIR closely and it may have been talked
> about, but why not just do it?

I think this is where I got to last time I looked at this. I ended up
introducing UNPACKDIR and cleaning up some of the stuff and leaving
that workaround in place in do_unpack.

To be honest I'm torn. Part of me wants to do it, part of me worries
about the breakage. We probably would need to prototype some patches,
see how bad it really is.

> Then this fix for devtool wouldn't add more complexity, it would just
> similarly replace workdir with unpackdir - not making things better
> but not making them worse either. On the other hand, if we do nothing,
> then devtool currently doesn't work for recipes that set S to
> unpackdir, negatively impacting AUH rates for one thing (this is why I
> got into it in the first place) :-(

I agree we need to do something, it is just a question of what (and
perhaps when).

Cheers,

Richard