Re: [OE-core] [PATCH 1/4] devtool: account for sources in UNPACKDIR

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Thu, 2025-04-24 at 21:36 +0200, Alexander Kanavin wrote:
> On Thu, 24 Apr 2025 at 15:50, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > Stepping back further, we had this mess in devtool in the first
> > place
> > as it could never work out which files in WORKDIR were "sources"
> > which
> > it needed to handle carefully.
> > 
> > We now define UNPACKDIR as the place we unpack files to. devtool in
> > theory can therefore just redefine UNPACKDIR and WORKDIR and things
> > should 'just work'. The relation of the subpaths between the two
> > should
> > remain the same.
> 
> I still struggle to understand this argument. As far as I see, to
> determine where is the 'source tree', devtool cannot avoid looking at
> both WORKDIR and S (and now, UNPACKDIR too) and then calculate
> something reasonable out of both (and with UNPACKDIR in the mix, all
> three).
> 
> Let me describe this step by step what happens without the patch.
> 
> 1. devtool manages a 'workspace': a special layer under
> build/workspace/. Part of the workspace is sources/: a place where
> source trees are placed. When a recipe is taken into workspace, its
> source tree is placed under build/workspace/sources/${PN}/.
> 
> 2. How does devtool determine where to take the source tree from? It
> sets WORKDIR to a temporary location under the real oe-core workdir,
> and then performs an unpack operation.

Why can't it set UNPACKDIR to the the workspace path and then just run
it to directly unpack to where it wants it?

It can then redefine UNPACKDIR to point at the workspace path and
things should "just work"?

>  Then it, somehow, needs to find
> the source tree under that temporary WORKDIR. It cannot simply use S:
> this can be set to a nested sub-directory deep inside the actual
> source tree. It cannot use WORKDIR either: this is not the source
> tree in almost all cases (and is now forbidden anyway).

It can however use UNPACKDIR?
> 
> 3. Let's say WORKDIR is set to /some/path/to/workdir, and S is then
> /some/path/to/workdir/git/some/subpath/. (via S =
> "${WORKDIR}/git/some/subpath")
> 
> devtool does this:
> 
> a) use os.path.relpath on WORKDIR and S to obtain git/some/subpath
> part
> b) then it splits that part into individual directories and take the
> first one: 'git'
> c) then it's appended to WORKDIR: /some/path/to/workdir/git
> d) then devtool uses shutil.move to move/rename that into workspace,
> so /some/path/to/workdir/git becomes build/workspace/sources/${PN}
> 
> All fine, so far. What happens if the recipe sets S =
> "${UNPACKDIR}/git/some/subpath? The same calculation as in point
> three
> will yield UNPACKDIR (incorrect), instead of UNPACKDIR/git (correct).
> And this is what the patch is aiming to address: it changes the code
> to first check if S is under UNPACKDIR, and if so, performs the
> calculation in point three relative to UNPACKDIR. Otherwise, it's
> performed relative to WORKDIR, as before.
> 
> The argument I can't understand is that devtool can somehow avoid
> looking at S altogether, and achieve the right thing by just
> redefining UNPACKDIR and WORKDIR and removing the logic described
> above, but how? How would we end up with the correct source tree in
> the workspace then?

A recipe doesn't have a single source tree but a set of sources, which
is the contents of UNPACKDIR. do_unpack has some horrible magic moving
which moves the sources into the expected place for the magic value of
S (see the SOURCE_BASEDIR mess) and this is perhaps confusing things.

In my mind, I'm thinking we ultimately stop that move and put a symlink
in place instead for compatibility. There were some issues with
autotools and symlinks when I last tried iirc but I'm digressing...

From memory, one of the devtool issues was that it couldn't track and
customise some of the input sources such as standalone files. The
UNPACKDIR changes were intended to give us a path forward so we could
actually handle the full set of sources rather than just the "S"
contents.

So I still believe there is some kind of simplification/improvement we
probably can make here...

Cheers,

Richard