Re: [OE-core] [PATCH] ovmf-native: remove .pyc files from install

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Mon, 2024-10-14 at 12:34 +0000, Ross Burton via
lists.openembedded.org wrote:
> On 14 Oct 2024, at 13:01, Richard Purdie via lists.openembedded.org
> <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
> > 
> > On Mon, 2024-10-14 at 13:05 +0200, Alexander Kanavin via
> > lists.openembedded.org wrote:
> > > This doesn't quite make sense. Why is the problem specific to
> > > .pyc
> > > files and not other files ovmf installs?
> > 
> > If it helps, I do roughly understand what is happening here and I
> > believe this is the right fix.
> 
> So my understanding is that this is a general problem: if a native
> recipe ships a .py but not .pyc then when that script is used python
> will write a .pyc _into the sysroot_. If we then upgrade that recipe
> and it now ships a .pyc then you get this error, as the sysroot
> construction call wants to create a file that already exists.
> 
> I think for scripts that are ran with python3-native then we should
> be shipping the .pyc files, even in native recipes, because then
> they’re pregenerated and will be used by python.  The fun is when the
> script is ran by the host python, which is of indeterminate version.

For fun, there are things which could be called by either the host
python or python3-native from the sysroot depending on whether that was
added to PATH for other reasons. That complicates things somewhat.

> I wonder if python has a way to control where the files get written,
> so we could put freshly generated files into a separate directory
> that isn’t part of the sysroot?  We could also make the sysroot read-
> only outside of the sysroot manipulation, which would force python to
> do something different too (though I believe this is just “don’t
> write pyc?).
> 
> Note that this problem impacts plenty of other recipes that for
> historical reasons can't ship bytecode into the sysroot (see eg
> python3-setuptools), so I do think we need a general solution and not
> just a creeping and irreversible trend of deleting .pyc from the
> sysroot.

I'm torn. It is a complex problem causing issues for patches which are
otherwise ok and requires a great deal of thought to fix differently.

The performance impact of python generating those files at first run
probably is negligible, compared even to the overhead of extracting
files from sstate.

Is this an issue we want to focus effort on? Are there gains here to be
had?

Cheers,

Richard