[PATCH v3 1/2] binman: Do not pollute source tree when build with `make O=...`

[PATCH v3 1/2] binman: Do not pollute source tree when build with `make O=...`

[Andy Shevchenko]

Importing libraries in Python caches the bytecode by default.
Since we run scripts in source tree it ignores the current directory
settings, which is $(srctree), and creates cache just in the middle
of the source tree. Move cache to the current directory.

Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
---
v3: avoided crash (Simon), preserved tree hierarchy
 tools/binman/main.py | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/tools/binman/main.py b/tools/binman/main.py
index 8c1e478d54ce..d19ded491af7 100755
--- a/tools/binman/main.py
+++ b/tools/binman/main.py
@@ -16,9 +16,20 @@ import sys
 import traceback
 import unittest
 
+# Get the absolute path to this file at run-time
+our_path = os.path.dirname(os.path.realpath(__file__))
+our1_path = os.path.dirname(our_path)
+our2_path = os.path.dirname(our1_path)
+
+#
+# Do not pollute source tree with cache files:
+# https://stackoverflow.com/a/60024195/2511795
+# https://bugs.python.org/issue33499
+#
+sys.pycache_prefix = os.path.relpath(our_path, os.environ.get('srctree', our2_path))
+
 # Bring in the patman and dtoc libraries (but don't override the first path
 # in PYTHONPATH)
-our_path = os.path.dirname(os.path.realpath(__file__))
 sys.path.insert(2, os.path.join(our_path, '..'))
 
 from patman import test_util
-- 
2.33.0

[PATCH v3 2/2] binman: Use less hard coded magic when inserting new PATH

[Andy Shevchenko]

Instead of joining hard coded '..' to the run-time path of the executable,
take just a dirname out of it. Besides that, use $(srctree) where it makes
sense.

Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
---
v3: avoided crash (Simon), reused one level up folder variable
 tools/binman/main.py | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/tools/binman/main.py b/tools/binman/main.py
index d19ded491af7..6f5ee4e21db7 100755
--- a/tools/binman/main.py
+++ b/tools/binman/main.py
@@ -21,24 +21,26 @@ our_path = os.path.dirname(os.path.realpath(__file__))
 our1_path = os.path.dirname(our_path)
 our2_path = os.path.dirname(our1_path)
 
+# Extract $(srctree) from Kbuild environment, or use relative paths below
+srctree = os.environ.get('srctree', our2_path)
+
 #
 # Do not pollute source tree with cache files:
 # https://stackoverflow.com/a/60024195/2511795
 # https://bugs.python.org/issue33499
 #
-sys.pycache_prefix = os.path.relpath(our_path, os.environ.get('srctree', our2_path))
+sys.pycache_prefix = os.path.relpath(our_path, srctree)
 
 # Bring in the patman and dtoc libraries (but don't override the first path
 # in PYTHONPATH)
-sys.path.insert(2, os.path.join(our_path, '..'))
+sys.path.insert(2, our1_path)
 
 from patman import test_util
 
 # Bring in the libfdt module
 sys.path.insert(2, 'scripts/dtc/pylibfdt')
-sys.path.insert(2, os.path.join(our_path, '../../scripts/dtc/pylibfdt'))
-sys.path.insert(2, os.path.join(our_path,
-                '../../build-sandbox_spl/scripts/dtc/pylibfdt'))
+sys.path.insert(2, os.path.join(srctree, 'scripts/dtc/pylibfdt'))
+sys.path.insert(2, os.path.join(srctree, 'build-sandbox_spl/scripts/dtc/pylibfdt'))
 
 # When running under python-coverage on Ubuntu 16.04, the dist-packages
 # directories are dropped from the python path. Add them in so that we can find
-- 
2.33.0

Re: [PATCH v3 2/2] binman: Use less hard coded magic when inserting new PATH

[Simon Glass]

Instead of joining hard coded '..' to the run-time path of the executable,
take just a dirname out of it. Besides that, use $(srctree) where it makes
sense.

Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
---
v3: avoided crash (Simon), reused one level up folder variable
 tools/binman/main.py | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

Applied to u-boot-dm/next, thanks!

Re: [PATCH v3 1/2] binman: Do not pollute source tree when build with `make O=...`

[Simon Glass]

Importing libraries in Python caches the bytecode by default.
Since we run scripts in source tree it ignores the current directory
settings, which is $(srctree), and creates cache just in the middle
of the source tree. Move cache to the current directory.

Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
---
v3: avoided crash (Simon), preserved tree hierarchy
 tools/binman/main.py | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

Applied to u-boot-dm/next, thanks!

Re: [PATCH v3 1/2] binman: Do not pollute source tree when build with `make O=...`

[Simon Glass]

Hi Andy,

On Tue, 14 Dec 2021 at 17:33, Simon Glass <sjg@chromium.org> wrote:
>
> Importing libraries in Python caches the bytecode by default.
> Since we run scripts in source tree it ignores the current directory
> settings, which is $(srctree), and creates cache just in the middle
> of the source tree. Move cache to the current directory.
>
> Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
> ---
> v3: avoided crash (Simon), preserved tree hierarchy
>  tools/binman/main.py | 13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
>
> Applied to u-boot-dm/next, thanks!

I didn't notice this before, but this seems to create files like this:

./tools/binman/usr/lib/python3/dist-packages/elftools/common/construct_utils.cpython-39.pyc

We don't really want to 'recache' the common Python files. Do you
think we should revert this patch, or find another fix?

Regards,
Simon

Re: [PATCH v3 1/2] binman: Do not pollute source tree when build with `make O=...`

[Andy Shevchenko]

On Thu, Dec 23, 2021 at 7:08 AM Simon Glass <sjg@chromium.org> wrote:
> On Tue, 14 Dec 2021 at 17:33, Simon Glass <sjg@chromium.org> wrote:
> >
> > Importing libraries in Python caches the bytecode by default.
> > Since we run scripts in source tree it ignores the current directory
> > settings, which is $(srctree), and creates cache just in the middle
> > of the source tree. Move cache to the current directory.
> >
> > Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
> > ---
> > v3: avoided crash (Simon), preserved tree hierarchy
> >  tools/binman/main.py | 13 ++++++++++++-
> >  1 file changed, 12 insertions(+), 1 deletion(-)
> >
> > Applied to u-boot-dm/next, thanks!
>
> I didn't notice this before, but this seems to create files like this:
>
> ./tools/binman/usr/lib/python3/dist-packages/elftools/common/construct_utils.cpython-39.pyc
>
> We don't really want to 'recache' the common Python files. Do you
> think we should revert this patch, or find another fix?

I'm not sure I understand. efitools is not common, it's a separate
(non-standard) module and it's cached.

-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH v3 1/2] binman: Do not pollute source tree when build with `make O=...`

[Simon Glass]

Hi Andy,

On Thu, 23 Dec 2021 at 02:07, Andy Shevchenko <andy.shevchenko@gmail.com> wrote:
>
> On Thu, Dec 23, 2021 at 7:08 AM Simon Glass <sjg@chromium.org> wrote:
> > On Tue, 14 Dec 2021 at 17:33, Simon Glass <sjg@chromium.org> wrote:
> > >
> > > Importing libraries in Python caches the bytecode by default.
> > > Since we run scripts in source tree it ignores the current directory
> > > settings, which is $(srctree), and creates cache just in the middle
> > > of the source tree. Move cache to the current directory.
> > >
> > > Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
> > > ---
> > > v3: avoided crash (Simon), preserved tree hierarchy
> > >  tools/binman/main.py | 13 ++++++++++++-
> > >  1 file changed, 12 insertions(+), 1 deletion(-)
> > >
> > > Applied to u-boot-dm/next, thanks!
> >
> > I didn't notice this before, but this seems to create files like this:
> >
> > ./tools/binman/usr/lib/python3/dist-packages/elftools/common/construct_utils.cpython-39.pyc
> >
> > We don't really want to 'recache' the common Python files. Do you
> > think we should revert this patch, or find another fix?
>
> I'm not sure I understand. efitools is not common, it's a separate
> (non-standard) module and it's cached.

The problem is that we don't want to create this file in the source directory.

For me this file is in:

/usr/lib/python3/dist-packages/elftools/

but it could be in ~/.local I suppose.

In any case, the cache file should go there, shouldn't it?

Regards,
Simon