Re: [Buildroot] [PATCH 1/1] utils/scanpypi: refactor setuptools handling to not use imp

["Yann E. MORIN" <yann.morin.1998@free.fr>]

James, All,

Thanks for this quick patch following our earlier discussion on IRC! 👍

On 2023-11-30 12:32 -0700, James Hilliard spake thusly:
> The imp module is deprecated as of python verison 3.12.

It's been deprecated for 10 years, and _removed_ in python 3.12.

> Refactor setuptools handling to remove monkeypatching hack and
> instead do pep517 metadata generation and dependency resolution.

This looks unrelated to the imp->importlib transition, no? Qnd thus it
should be in its ownm separate patch.

> Invert setuptools/pyproject fallback ordering so that we try
> parsing pyproject.toml files first.

Ditto, this is yet another unrelated change, which should also be in a
seaprate patch.

> Use a set comprehension instead of a list comprehension for
> generating pkg_req so that we don't get duplicates.

Again, this is yet a change unrelated to the others, so needs to be in a
separate patch too.

Regards,
Yann E. MORIN.

> Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
> ---
>  utils/scanpypi | 92 +++++++++++++++++---------------------------------
>  1 file changed, 31 insertions(+), 61 deletions(-)
> 
> diff --git a/utils/scanpypi b/utils/scanpypi
> index 021c99a172..a1c2c2bd52 100755
> --- a/utils/scanpypi
> +++ b/utils/scanpypi
> @@ -18,8 +18,8 @@ import hashlib
>  import re
>  import textwrap
>  import tempfile
> -import imp
> -from functools import wraps
> +import importlib
> +from setuptools.build_meta import prepare_metadata_for_build_wheel
>  import six.moves.urllib.request
>  import six.moves.urllib.error
>  import six.moves.urllib.parse
> @@ -93,32 +93,6 @@ def toml_load(f):
>          raise ex
>  
>  
> -def setup_decorator(func, method):
> -    """
> -    Decorator for distutils.core.setup and setuptools.setup.
> -    Puts the arguments with which setup is called as a dict
> -    Add key 'method' which should be either 'setuptools' or 'distutils'.
> -
> -    Keyword arguments:
> -    func -- either setuptools.setup or distutils.core.setup
> -    method -- either 'setuptools' or 'distutils'
> -    """
> -
> -    @wraps(func)
> -    def closure(*args, **kwargs):
> -        # Any python packages calls its setup function to be installed.
> -        # Argument 'name' of this setup function is the package's name
> -        BuildrootPackage.setup_args[kwargs['name']] = kwargs
> -        BuildrootPackage.setup_args[kwargs['name']]['method'] = method
> -    return closure
> -
> -# monkey patch
> -import setuptools  # noqa E402
> -setuptools.setup = setup_decorator(setuptools.setup, 'setuptools')
> -import distutils   # noqa E402
> -distutils.core.setup = setup_decorator(setuptools.setup, 'distutils')
> -
> -
>  def find_file_upper_case(filenames, path='./'):
>      """
>      List generator:
> @@ -345,26 +319,18 @@ class BuildrootPackage():
>          """
>          current_dir = os.getcwd()
>          os.chdir(self.tmp_extract)
> -        sys.path.insert(0, self.tmp_extract)
>          try:
> -            s_file, s_path, s_desc = imp.find_module('setup', [self.tmp_extract])
> -            imp.load_module('__main__', s_file, s_path, s_desc)
> -            if self.metadata_name in self.setup_args:
> -                pass
> -            elif self.metadata_name.replace('_', '-') in self.setup_args:
> -                self.metadata_name = self.metadata_name.replace('_', '-')
> -            elif self.metadata_name.replace('-', '_') in self.setup_args:
> -                self.metadata_name = self.metadata_name.replace('-', '_')
> +            metadata = prepare_metadata_for_build_wheel(self.tmp_extract)
>              try:
> -                self.setup_metadata = self.setup_args[self.metadata_name]
> -            except KeyError:
> -                # This means setup was not called
> -                print('ERROR: Could not determine package metadata for {pkg}.\n'
> -                      .format(pkg=self.real_name))
> -                raise
> +                dist = importlib.metadata.Distribution.at(metadata)
> +                self.metadata_name = dist.name
> +                self.setup_metadata = {'method': 'setuptools'}
> +                if dist.requires:
> +                    self.setup_metadata['install_requires'] = dist.requires
> +            finally:
> +                shutil.rmtree(metadata)
>          finally:
>              os.chdir(current_dir)
> -            sys.path.remove(self.tmp_extract)
>  
>      def load_pyproject(self):
>          """
> @@ -372,7 +338,6 @@ class BuildrootPackage():
>          """
>          current_dir = os.getcwd()
>          os.chdir(self.tmp_extract)
> -        sys.path.insert(0, self.tmp_extract)
>          try:
>              pyproject_data = toml_load('pyproject.toml')
>              try:
> @@ -380,20 +345,25 @@ class BuildrootPackage():
>                  self.metadata_name = self.setup_metadata.get('name', self.real_name)
>                  build_system = pyproject_data.get('build-system', {})
>                  build_backend = build_system.get('build-backend', None)
> -                if build_backend and build_backend == 'flit_core.buildapi':
> -                    self.setup_metadata['method'] = 'flit'
> +                if build_backend:
> +                    if build_backend == 'flit_core.buildapi':
> +                        self.setup_metadata['method'] = 'flit'
> +                    elif build_backend == 'setuptools.build_meta':
> +                        raise Exception("handle setuptools")
> +                    else:
> +                        self.setup_metadata['method'] = 'unknown'
>                  elif build_system.get('backend-path', None):
>                      self.setup_metadata['method'] = 'pep517'
>                  else:
> -                    self.setup_metadata['method'] = 'unknown'
> +                    raise Exception("handle setuptools")
>              except KeyError:
>                  print('ERROR: Could not determine package metadata for {pkg}.\n'
>                        .format(pkg=self.real_name))
>                  raise
>          except FileNotFoundError:
>              raise
> -        os.chdir(current_dir)
> -        sys.path.remove(self.tmp_extract)
> +        finally:
> +            os.chdir(current_dir)
>  
>      def get_requirements(self, pkg_folder):
>          """
> @@ -411,8 +381,8 @@ class BuildrootPackage():
>                          for req in self.pkg_req]
>  
>          # get rid of commented lines and also strip the package strings
> -        self.pkg_req = [item.strip() for item in self.pkg_req
> -                        if len(item) > 0 and item[0] != '#']
> +        self.pkg_req = {item.strip() for item in self.pkg_req
> +                        if len(item) > 0 and item[0] != '#'}
>  
>          req_not_found = self.pkg_req
>          self.pkg_req = list(map(pkg_buildroot_name, self.pkg_req))
> @@ -778,15 +748,15 @@ def main():
>  
>              # Loading the package install info from the package
>              try:
> -                package.load_setup()
> -            except ImportError as err:
> -                if 'buildutils' in str(err):
> -                    print('This package needs buildutils')
> -                    continue
> -                else:
> -                    try:
> -                        package.load_pyproject()
> -                    except Exception:
> +                package.load_pyproject()
> +            except Exception:
> +                try:
> +                    package.load_setup()
> +                except ImportError as err:
> +                    if 'buildutils' in str(err):
> +                        print('This package needs buildutils')
> +                        continue
> +                    else:
>                          raise
>              except (AttributeError, KeyError) as error:
>                  print('Error: Could not install package {pkg}: {error}'.format(
> -- 
> 2.34.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot