[RESEND][PATCH 0/7] Devtool/Recipetool: adding pypi support

[RESEND][PATCH 0/7] Devtool/Recipetool: adding pypi support

[Julien Stephan]

Hello all,

This series adds basic support for the pypi class in devtool/recipetool.

The idea is to be able to detect that a package is available on pypi and
in that case inherits from pypi class. This helps to produce cleaner
recipes for pypi packages.

To do this, I am adding a new optional callback "process_url" that
plugin can register to implement some logic on matching url.

By implementing this callback in create_buildsys_python we can match on
pypi URLs directly or on release tarballs hosted on "files.pythonhosted.org".

To create a recipe taking advantage of the pypi class we can use one of the
following new syntax:

* recipetool create https://pypi.org/project/<package>
* recipetool create https://pypi.org/project/<package>/<version>
* recipetool create https://pypi.org/project/<package> --version <version>

or the old syntax:

* recipetool create https://files.pythonhosted.org/packages/<...>

If the URL points to a github URL or a release tarball not coming from
"files.pythonhosted.org", the created recipe is the same as before.
One can also use the newly added "--no-pypi" switch to NOT inherit 
from pypi class on matching URL, to keep legacy behaviour.

This series also contains some bug fixes I found during my testing.

Pushed my dev branch here: https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/devtool-add-pypi-support 

Cheers
Julien

Julien Stephan (7):
  bitbake: utils: remove spaces on empty lines
  recipetool: create_buildsys_python.py: initialize metadata
  recipetool: create: add trailing newlines
  recipetool: create: add new optional process_url callback for plugins
  recipetool: create_buildsys_python: add pypi support
  oeqa/selftest/recipetool: remove spaces on empty lines
  oeqa/selftest/recipetool/devtool: add test for pypi class

 bitbake/lib/bb/utils.py                       |  16 +--
 meta/lib/oeqa/selftest/cases/devtool.py       |   4 +-
 meta/lib/oeqa/selftest/cases/recipetool.py    | 114 +++++++++++++-----
 scripts/lib/devtool/standard.py               |   3 +
 scripts/lib/recipetool/create.py              |  63 ++++++----
 .../lib/recipetool/create_buildsys_python.py  |  73 +++++++++++
 6 files changed, 208 insertions(+), 65 deletions(-)

-- 
2.42.0

[RESEND][PATCH 1/7] bitbake: utils: remove spaces on empty lines

[Julien Stephan]

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 bitbake/lib/bb/utils.py | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index b401fa5ec7a..61ffad92ce3 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -50,7 +50,7 @@ def clean_context():
 
 def get_context():
     return _context
-    
+
 
 def set_context(ctx):
     _context = ctx
@@ -212,8 +212,8 @@ def explode_dep_versions2(s, *, sort=True):
             inversion = True
             # This list is based on behavior and supported comparisons from deb, opkg and rpm.
             #
-            # Even though =<, <<, ==, !=, =>, and >> may not be supported, 
-            # we list each possibly valid item. 
+            # Even though =<, <<, ==, !=, =>, and >> may not be supported,
+            # we list each possibly valid item.
             # The build system is responsible for validation of what it supports.
             if i.startswith(('<=', '=<', '<<', '==', '!=', '>=', '=>', '>>')):
                 lastcmp = i[0:2]
@@ -347,7 +347,7 @@ def _print_exception(t, value, tb, realfile, text, context):
         exception = traceback.format_exception_only(t, value)
         error.append('Error executing a python function in %s:\n' % realfile)
 
-        # Strip 'us' from the stack (better_exec call) unless that was where the 
+        # Strip 'us' from the stack (better_exec call) unless that was where the
         # error came from
         if tb.tb_next is not None:
             tb = tb.tb_next
@@ -746,9 +746,9 @@ def prunedir(topdir, ionice=False):
 # but thats possibly insane and suffixes is probably going to be small
 #
 def prune_suffix(var, suffixes, d):
-    """ 
+    """
     See if var ends with any of the suffixes listed and
-    remove it if found 
+    remove it if found
     """
     for suffix in suffixes:
         if suffix and var.endswith(suffix):
@@ -1001,9 +1001,9 @@ def umask(new_mask):
         os.umask(current_mask)
 
 def to_boolean(string, default=None):
-    """ 
+    """
     Check input string and return boolean value True/False/None
-    depending upon the checks 
+    depending upon the checks
     """
     if not string:
         return default
-- 
2.42.0

[RESEND][PATCH 2/7] recipetool: create_buildsys_python.py: initialize metadata

[Julien Stephan]

In the case pyproject.toml doesn't contains metadatas, the metadata
variable is not initialized and the plugin throws an error and falls back
to another plugin, which is not the desired behaviour. So just ignore
metadata if we don't have them

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 scripts/lib/recipetool/create_buildsys_python.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py
index 9312e4abf13..b620e3271b1 100644
--- a/scripts/lib/recipetool/create_buildsys_python.py
+++ b/scripts/lib/recipetool/create_buildsys_python.py
@@ -726,6 +726,7 @@ class PythonPyprojectTomlRecipeHandler(PythonRecipeHandler):
 
     def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
         info = {}
+        metadata = {}
 
         if 'buildsystem' in handled:
             return False
-- 
2.42.0

[RESEND][PATCH 3/7] recipetool: create: add trailing newlines

[Julien Stephan]

create_recipe() function relies on oe.recipeutils.patch_recipe_lines()
which relies on bb.utils.edit_metadata(). edit_metada expect lines to
have trailing newlines, so add it to each lines before calling
patch_recipe_lines, otherwise edit_metadata will not be able to squash
blank line if there are two consecutive blanks after a removal

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 scripts/lib/recipetool/create.py | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 293198d1c88..f5d541eb6c1 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -873,8 +873,10 @@ def create_recipe(args):
         outlines.append('')
     outlines.extend(lines_after)
 
+    outlines = [ line.rstrip('\n') +"\n" for line in outlines]
+
     if extravalues:
-        _, outlines = oe.recipeutils.patch_recipe_lines(outlines, extravalues, trailing_newline=False)
+        _, outlines = oe.recipeutils.patch_recipe_lines(outlines, extravalues, trailing_newline=True)
 
     if args.extract_to:
         scriptutils.git_convert_standalone_clone(srctree)
@@ -890,7 +892,7 @@ def create_recipe(args):
         log_info_cond('Source extracted to %s' % args.extract_to, args.devtool)
 
     if outfile == '-':
-        sys.stdout.write('\n'.join(outlines) + '\n')
+        sys.stdout.write(''.join(outlines) + '\n')
     else:
         with open(outfile, 'w') as f:
             lastline = None
@@ -898,7 +900,7 @@ def create_recipe(args):
                 if not lastline and not line:
                     # Skip extra blank lines
                     continue
-                f.write('%s\n' % line)
+                f.write('%s' % line)
                 lastline = line
         log_info_cond('Recipe %s has been created; further editing may be required to make it fully functional' % outfile, args.devtool)
         tinfoil.modified_files()
-- 
2.42.0

[RESEND][PATCH 4/7] recipetool: create: add new optional process_url callback for plugins

[Julien Stephan]

Add a new process_url callback that plugins can optionally implement if
they which to handle url.

Plugins can implement this callback for example, to:
* transform the url
* add special variables using extravalues
* add extra classes
* ...

If a plugin handles the url, it must append 'url' to the handled
list and must return the fetchuri

No functional changes expected for plugins non implementing this
optional callback

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 scripts/lib/recipetool/create.py | 54 +++++++++++++++++++-------------
 1 file changed, 32 insertions(+), 22 deletions(-)

diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index f5d541eb6c1..5c5ac7ae403 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -423,6 +423,36 @@ def create_recipe(args):
     storeTagName = ''
     pv_srcpv = False
 
+    handled = []
+    classes = []
+
+    # Find all plugins that want to register handlers
+    logger.debug('Loading recipe handlers')
+    raw_handlers = []
+    for plugin in plugins:
+        if hasattr(plugin, 'register_recipe_handlers'):
+            plugin.register_recipe_handlers(raw_handlers)
+    # Sort handlers by priority
+    handlers = []
+    for i, handler in enumerate(raw_handlers):
+        if isinstance(handler, tuple):
+            handlers.append((handler[0], handler[1], i))
+        else:
+            handlers.append((handler, 0, i))
+    handlers.sort(key=lambda item: (item[1], -item[2]), reverse=True)
+    for handler, priority, _ in handlers:
+        logger.debug('Handler: %s (priority %d)' % (handler.__class__.__name__, priority))
+        setattr(handler, '_devtool', args.devtool)
+    handlers = [item[0] for item in handlers]
+
+    fetchuri = None
+    for handler in handlers:
+        if hasattr(handler, 'process_url'):
+            ret = handler.process_url(args, classes, handled, extravalues)
+            if 'url' in handled and ret:
+                fetchuri = ret
+                break
+
     if os.path.isfile(source):
         source = 'file://%s' % os.path.abspath(source)
 
@@ -431,7 +461,8 @@ def create_recipe(args):
         if re.match(r'https?://github.com/[^/]+/[^/]+/archive/.+(\.tar\..*|\.zip)$', source):
             logger.warning('github archive files are not guaranteed to be stable and may be re-generated over time. If the latter occurs, the checksums will likely change and the recipe will fail at do_fetch. It is recommended that you point to an actual commit or tag in the repository instead (using the repository URL in conjunction with the -S/--srcrev option).')
         # Fetch a URL
-        fetchuri = reformat_git_uri(urldefrag(source)[0])
+        if not fetchuri:
+            fetchuri = reformat_git_uri(urldefrag(source)[0])
         if args.binary:
             # Assume the archive contains the directory structure verbatim
             # so we need to extract to a subdirectory
@@ -638,8 +669,6 @@ def create_recipe(args):
     # We'll come back and replace this later in handle_license_vars()
     lines_before.append('##LICENSE_PLACEHOLDER##')
 
-    handled = []
-    classes = []
 
     # FIXME This is kind of a hack, we probably ought to be using bitbake to do this
     pn = None
@@ -718,25 +747,6 @@ def create_recipe(args):
     if args.npm_dev:
         extravalues['NPM_INSTALL_DEV'] = 1
 
-    # Find all plugins that want to register handlers
-    logger.debug('Loading recipe handlers')
-    raw_handlers = []
-    for plugin in plugins:
-        if hasattr(plugin, 'register_recipe_handlers'):
-            plugin.register_recipe_handlers(raw_handlers)
-    # Sort handlers by priority
-    handlers = []
-    for i, handler in enumerate(raw_handlers):
-        if isinstance(handler, tuple):
-            handlers.append((handler[0], handler[1], i))
-        else:
-            handlers.append((handler, 0, i))
-    handlers.sort(key=lambda item: (item[1], -item[2]), reverse=True)
-    for handler, priority, _ in handlers:
-        logger.debug('Handler: %s (priority %d)' % (handler.__class__.__name__, priority))
-        setattr(handler, '_devtool', args.devtool)
-    handlers = [item[0] for item in handlers]
-
     # Apply the handlers
     if args.binary:
         classes.append('bin_package')
-- 
2.42.0

[RESEND][PATCH 5/7] recipetool: create_buildsys_python: add pypi support

[Julien Stephan]

Today, we can use devtool/recipetool to create recipes for python projects
using the github url or the direct release tarball of the project, but the
create_buildsys_python plugin doesn't support the pypi class, since we cannot
know from the extracted source if the package is available on pypi or not.

By implementing the new optional process_url callback, we can detect
that the url is a pypi one (i.e 'https://pypi.org/project/<package>')
and retrieve the release tarball location.
Also detect if the url points to a release tarball hosted on
"files.pythonhosted.iorg" (i.e https://files.pythonhosted.org/packages/...)

In both cases, adds the pypi class, remove 'S' and 'SRC_URIxxx'
variables from the created recipe as they will be handled by the pypi class
and add the PYPI_PACKAGE variable

This helps to produce cleaner recipes when package is hosted on pypi.

If the url points to a github url or a release tarball not coming from
"files.pythonhosted.org", the created recipe is the same as before.
One can also use the newly added "--no-pypi" switch to NOT inherit
from pypi class on matching url, to keep legacy behaviour.

To create a recipe for a pypi package, one can now use one of the
new following syntax (using recipetool create / devtool add):

* recipetool create https://pypi.org/project/<package>
* recipetool create https://pypi.org/project/<package>/<version>
* recipetool create https://pypi.org/project/<package> --version <version>

or the old syntax:
* recipetool create https://files.pythonhosted.org/packages/<...>

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 scripts/lib/devtool/standard.py               |  3 +
 scripts/lib/recipetool/create.py              |  1 +
 .../lib/recipetool/create_buildsys_python.py  | 72 +++++++++++++++++++
 3 files changed, 76 insertions(+)

diff --git a/scripts/lib/devtool/standard.py b/scripts/lib/devtool/standard.py
index d53fb810071..f18ebaa70d3 100644
--- a/scripts/lib/devtool/standard.py
+++ b/scripts/lib/devtool/standard.py
@@ -147,6 +147,8 @@ def add(args, config, basepath, workspace):
         extracmdopts += ' -a'
     if args.npm_dev:
         extracmdopts += ' --npm-dev'
+    if args.no_pypi:
+        extracmdopts += ' --no-pypi'
     if args.mirrors:
         extracmdopts += ' --mirrors'
     if args.srcrev:
@@ -2260,6 +2262,7 @@ def register_commands(subparsers, context):
     group.add_argument('--no-same-dir', help='Force build in a separate build directory', action="store_true")
     parser_add.add_argument('--fetch', '-f', help='Fetch the specified URI and extract it to create the source tree (deprecated - pass as positional argument instead)', metavar='URI')
     parser_add.add_argument('--npm-dev', help='For npm, also fetch devDependencies', action="store_true")
+    parser_add.add_argument('--no-pypi', help='Do not inherit pypi class', action="store_true")
     parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)')
     parser_add.add_argument('--no-git', '-g', help='If fetching source, do not set up source tree as a git repository', action="store_true")
     group = parser_add.add_mutually_exclusive_group()
diff --git a/scripts/lib/recipetool/create.py b/scripts/lib/recipetool/create.py
index 5c5ac7ae403..963aa91421e 100644
--- a/scripts/lib/recipetool/create.py
+++ b/scripts/lib/recipetool/create.py
@@ -1413,6 +1413,7 @@ def register_commands(subparsers):
     parser_create.add_argument('-B', '--srcbranch', help='Branch in source repository if fetching from an SCM such as git (default master)')
     parser_create.add_argument('--keep-temp', action="store_true", help='Keep temporary directory (for debugging)')
     parser_create.add_argument('--npm-dev', action="store_true", help='For npm, also fetch devDependencies')
+    parser_create.add_argument('--no-pypi', action="store_true", help='Do not inherit pypi class')
     parser_create.add_argument('--devtool', action="store_true", help=argparse.SUPPRESS)
     parser_create.add_argument('--mirrors', action="store_true", help='Enable PREMIRRORS and MIRRORS for source tree fetching (disabled by default).')
     parser_create.set_defaults(func=create_recipe)
diff --git a/scripts/lib/recipetool/create_buildsys_python.py b/scripts/lib/recipetool/create_buildsys_python.py
index b620e3271b1..5e07222ece1 100644
--- a/scripts/lib/recipetool/create_buildsys_python.py
+++ b/scripts/lib/recipetool/create_buildsys_python.py
@@ -18,7 +18,11 @@ import os
 import re
 import sys
 import subprocess
+import json
+import urllib.request
 from recipetool.create import RecipeHandler
+from urllib.parse import urldefrag
+from recipetool.create import determine_from_url
 
 logger = logging.getLogger('recipetool')
 
@@ -111,6 +115,74 @@ class PythonRecipeHandler(RecipeHandler):
     def __init__(self):
         pass
 
+    def process_url(self, args, classes, handled, extravalues):
+        """
+        Convert any pypi url https://pypi.org/project/<package>/<version> into https://files.pythonhosted.org/packages/source/...
+        which corresponds to the archive location, and add pypi class
+        """
+
+        if 'url' in handled:
+            return None
+
+        fetch_uri = None
+        source = args.source
+        required_version = args.version if args.version else None
+        match = re.match(r'https?://pypi.org/project/([^/]+)(?:/([^/]+))?/?$', urldefrag(source)[0])
+        if match:
+            package = match.group(1)
+            version = match.group(2) if match.group(2) else required_version
+
+            json_url = f"https://pypi.org/pypi/%s/json" % package
+            response = urllib.request.urlopen(json_url)
+            if response.status == 200:
+                data = json.loads(response.read())
+                if not version:
+                    # grab latest version
+                    version = data["info"]["version"]
+                pypi_package = data["info"]["name"]
+                for release in reversed(data["releases"][version]):
+                    if release["packagetype"] == "sdist":
+                        fetch_uri = release["url"]
+                        break
+            else:
+                logger.warning("Cannot handle pypi url %s: cannot fetch package information using %s", source, json_url)
+                return None
+        else:
+            match = re.match(r'^https?://files.pythonhosted.org/packages.*/(.*)-.*$', source)
+            if match:
+                fetch_uri = source
+                pypi_package = match.group(1)
+                _, version = determine_from_url(fetch_uri)
+
+        if match and not args.no_pypi:
+            if required_version and version != required_version:
+                raise Exception("Version specified using --version/-V (%s) and version specified in the url (%s) do not match" % (required_version, version))
+            # This is optionnal if BPN looks like "python-<pypi_package>" or "python3-<pypi_package>" (see pypi.bbclass)
+            # but at this point we cannot know because because user can specify the output name of the recipe on the command line
+            extravalues["PYPI_PACKAGE"] = pypi_package
+            # If the tarball extension is not 'tar.gz' (default value in pypi.bblcass) whe should set PYPI_PACKAGE_EXT in the recipe
+            pypi_package_ext = re.match(r'.*%s-%s\.(.*)$' % (pypi_package, version), fetch_uri)
+            if pypi_package_ext:
+                pypi_package_ext = pypi_package_ext.group(1)
+                if pypi_package_ext != "tar.gz":
+                    extravalues["PYPI_PACKAGE_EXT"] = pypi_package_ext
+
+            # Pypi class will handle S and SRC_URIxxx variables, so remove them
+            # TODO: allow oe.recipeutils.patch_recipe_lines() to accept regexp so we can simplify the following to:
+            # extravalues['SRC_URI(?:\[.*?\])?'] = None
+            extravalues['S'] = None
+            extravalues['SRC_URI'] = None
+            extravalues['SRC_URI[md5sum]'] = None
+            extravalues['SRC_URI[sha1sum]'] = None
+            extravalues['SRC_URI[sha256sum]'] = None
+            extravalues['SRC_URI[sha384sum]'] = None
+            extravalues['SRC_URI[sha512sum]'] = None
+
+            classes.append('pypi')
+
+        handled.append('url')
+        return fetch_uri
+
     def handle_classifier_license(self, classifiers, existing_licenses=""):
 
         licenses = []
-- 
2.42.0

[RESEND][PATCH 6/7] oeqa/selftest/recipetool: remove spaces on empty lines

[Julien Stephan]

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 meta/lib/oeqa/selftest/cases/recipetool.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py
index 55cbba9ca74..4bc28a4f2ee 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -853,7 +853,7 @@ class RecipetoolTests(RecipetoolBase):
         self._test_recipe_contents(deps_require_file, checkvars, [])
 
 
-        
+
     def _copy_file_with_cleanup(self, srcfile, basedstdir, *paths):
         dstdir = basedstdir
         self.assertTrue(os.path.exists(dstdir))
-- 
2.42.0

[RESEND][PATCH 7/7] oeqa/selftest/recipetool/devtool: add test for pypi class

[Julien Stephan]

recipetool now supports the pypi class and python recipes can by created
using the new following syntax:

* recipetool create https://pypi.org/project/<package>
* recipetool create https://pypi.org/project/<package>/<version>
* recipetool create https://pypi.org/project/<package> --version <version>

or the old syntax:
* recipetool create https://files.pythonhosted.org/packages/<...>

So add tests for the new syntax and modify old tests

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 meta/lib/oeqa/selftest/cases/devtool.py    |   4 +-
 meta/lib/oeqa/selftest/cases/recipetool.py | 112 +++++++++++++++------
 2 files changed, 85 insertions(+), 31 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py
index b5c488be8e8..3d9a4901ea5 100644
--- a/meta/lib/oeqa/selftest/cases/devtool.py
+++ b/meta/lib/oeqa/selftest/cases/devtool.py
@@ -543,7 +543,7 @@ class DevtoolAddTests(DevtoolBase):
         self.track_for_cleanup(self.workspacedir)
         self.add_command_to_tearDown('bitbake -c cleansstate %s' % testrecipe)
         self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
-        result = runCmd('devtool add %s %s -f %s' % (testrecipe, srcdir, url))
+        result = runCmd('devtool add --no-pypi %s %s -f %s' % (testrecipe, srcdir, url))
         self.assertExists(os.path.join(self.workspacedir, 'conf', 'layer.conf'), 'Workspace directory not created. %s' % result.output)
         self.assertTrue(os.path.isfile(os.path.join(srcdir, 'setup.py')), 'Unable to find setup.py in source directory')
         self.assertTrue(os.path.isdir(os.path.join(srcdir, '.git')), 'git repository for external source tree was not created')
@@ -562,7 +562,7 @@ class DevtoolAddTests(DevtoolBase):
         result = runCmd('devtool reset -n %s' % testrecipe)
         shutil.rmtree(srcdir)
         fakever = '1.9'
-        result = runCmd('devtool add %s %s -f %s -V %s' % (testrecipe, srcdir, url, fakever))
+        result = runCmd('devtool add --no-pypi %s %s -f %s -V %s' % (testrecipe, srcdir, url, fakever))
         self.assertTrue(os.path.isfile(os.path.join(srcdir, 'setup.py')), 'Unable to find setup.py in source directory')
         # Test devtool status
         result = runCmd('devtool status')
diff --git a/meta/lib/oeqa/selftest/cases/recipetool.py b/meta/lib/oeqa/selftest/cases/recipetool.py
index 4bc28a4f2ee..b59e53f5994 100644
--- a/meta/lib/oeqa/selftest/cases/recipetool.py
+++ b/meta/lib/oeqa/selftest/cases/recipetool.py
@@ -457,13 +457,15 @@ class RecipetoolCreateTests(RecipetoolBase):
 
     def test_recipetool_create_python3_setuptools(self):
         # Test creating python3 package from tarball (using setuptools3 class)
+        # Use the --no-pypi switch to avoid creating a pypi enabled recipe and
+        # and check the created recipe as if it was a more general tarball
         temprecipe = os.path.join(self.tempdir, 'recipe')
         os.makedirs(temprecipe)
         pn = 'python-magic'
         pv = '0.4.15'
         recipefile = os.path.join(temprecipe, '%s_%s.bb' % (pn, pv))
         srcuri = 'https://files.pythonhosted.org/packages/84/30/80932401906eaf787f2e9bd86dc458f1d2e75b064b4c187341f29516945c/python-magic-%s.tar.gz' % pv
-        result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri))
+        result = runCmd('recipetool create --no-pypi -o %s %s' % (temprecipe, srcuri))
         self.assertTrue(os.path.isfile(recipefile))
         checkvars = {}
         checkvars['LICENSE'] = set(['MIT'])
@@ -474,6 +476,82 @@ class RecipetoolCreateTests(RecipetoolBase):
         inherits = ['setuptools3']
         self._test_recipe_contents(recipefile, checkvars, inherits)
 
+    def test_recipetool_create_python3_setuptools_pypi_tarball(self):
+        # Test creating python3 package from tarball (using setuptools3 and pypi classes)
+        temprecipe = os.path.join(self.tempdir, 'recipe')
+        os.makedirs(temprecipe)
+        pn = 'python-magic'
+        pv = '0.4.15'
+        recipefile = os.path.join(temprecipe, '%s_%s.bb' % (pn, pv))
+        srcuri = 'https://files.pythonhosted.org/packages/84/30/80932401906eaf787f2e9bd86dc458f1d2e75b064b4c187341f29516945c/python-magic-%s.tar.gz' % pv
+        result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri))
+        self.assertTrue(os.path.isfile(recipefile))
+        checkvars = {}
+        checkvars['LICENSE'] = set(['MIT'])
+        checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=16a934f165e8c3245f241e77d401bb88'
+        checkvars['PYPI_PACKAGE'] = pn
+        inherits = ['setuptools3', 'pypi']
+        self._test_recipe_contents(recipefile, checkvars, inherits)
+
+    def test_recipetool_create_python3_setuptools_pypi(self):
+        # Test creating python3 package from pypi url (using setuptools3 and pypi classes)
+        # Intentionnaly using setuptools3 class here instead of any of the pep517 class
+        # to avoid the toml dependency and allows this test to run on host autobuilders
+        # with older version of python
+        temprecipe = os.path.join(self.tempdir, 'recipe')
+        os.makedirs(temprecipe)
+        pn = 'python-magic'
+        pv = '0.4.15'
+        recipefile = os.path.join(temprecipe, '%s_%s.bb' % (pn, pv))
+        # First specify the required version in the url
+        srcuri = 'https://pypi.org/project/%s/%s' % (pn, pv)
+        runCmd('recipetool create -o %s %s' % (temprecipe, srcuri))
+        self.assertTrue(os.path.isfile(recipefile))
+        checkvars = {}
+        checkvars['LICENSE'] = set(['MIT'])
+        checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=16a934f165e8c3245f241e77d401bb88'
+        checkvars['PYPI_PACKAGE'] = pn
+        inherits = ['setuptools3', "pypi"]
+        self._test_recipe_contents(recipefile, checkvars, inherits)
+
+        # Now specify the version as a recipetool parameter
+        runCmd('rm -rf %s' % recipefile)
+        self.assertFalse(os.path.isfile(recipefile))
+        srcuri = 'https://pypi.org/project/%s' % pn
+        runCmd('recipetool create -o %s %s --version %s' % (temprecipe, srcuri, pv))
+        self.assertTrue(os.path.isfile(recipefile))
+        checkvars = {}
+        checkvars['LICENSE'] = set(['MIT'])
+        checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=16a934f165e8c3245f241e77d401bb88'
+        checkvars['PYPI_PACKAGE'] = pn
+        inherits = ['setuptools3', "pypi"]
+        self._test_recipe_contents(recipefile, checkvars, inherits)
+
+        # Now, try to grab latest version of the package, so we cannot guess the name of the recipe,
+        # unless hardcoding the latest version but it means we will need to update the test for each release,
+        # so use a regexp
+        runCmd('rm -rf %s' % recipefile)
+        self.assertFalse(os.path.isfile(recipefile))
+        recipefile_re = r'%s_(.*)\.bb' % pn
+        result = runCmd('recipetool create -o %s %s' % (temprecipe, srcuri))
+        dirlist = os.listdir(temprecipe)
+        if len(dirlist) > 1:
+            self.fail('recipetool created more than just one file; output:\n%s\ndirlist:\n%s' % (result.output, str(dirlist)))
+        if len(dirlist) < 1 or not os.path.isfile(os.path.join(temprecipe, dirlist[0])):
+            self.fail('recipetool did not create recipe file; output:\n%s\ndirlist:\n%s' % (result.output, str(dirlist)))
+        import re
+        match = re.match(recipefile_re, dirlist[0])
+        self.assertTrue(match)
+        latest_pv = match.group(1)
+        self.assertTrue(latest_pv != pv)
+        recipefile = os.path.join(temprecipe, '%s_%s.bb' % (pn, latest_pv))
+        # Do not check LIC_FILES_CHKSUM here to avoid having updating the test on each release
+        checkvars = {}
+        checkvars['LICENSE'] = set(['MIT'])
+        checkvars['PYPI_PACKAGE'] = pn
+        inherits = ['setuptools3', "pypi"]
+        self._test_recipe_contents(recipefile, checkvars, inherits)
+
     def test_recipetool_create_python3_pep517_setuptools_build_meta(self):
         # This test require python 3.11 or above for the tomllib module
         # or tomli module to be installed
@@ -498,13 +576,7 @@ class RecipetoolCreateTests(RecipetoolBase):
         checkvars['SUMMARY'] = 'A library for working with the color formats defined by HTML and CSS.'
         checkvars['LICENSE'] = set(['BSD-3-Clause'])
         checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=702b1ef12cf66832a88f24c8f2ee9c19'
-        checkvars['SRC_URI'] = 'https://files.pythonhosted.org/packages/a1/fb/f95560c6a5d4469d9c49e24cf1b5d4d21ffab5608251c6020a965fb7791c/webcolors-${PV}.tar.gz'
-        checkvars['SRC_URI[md5sum]'] = 'c9be30c5b0cf1cad32e4cbacbb2229e9'
-        checkvars['SRC_URI[sha1sum]'] = 'c90b84fb65eed9b4c9dea7f08c657bfac0e820a5'
-        checkvars['SRC_URI[sha256sum]'] = 'c225b674c83fa923be93d235330ce0300373d02885cef23238813b0d5668304a'
-        checkvars['SRC_URI[sha384sum]'] = '45652af349660f19f68d01361dd5bda287789e5ea63608f52a8cea526ac04465614db2ea236103fb8456b1fcaea96ed7'
-        checkvars['SRC_URI[sha512sum]'] = '074aaf135ac6b0025b88b731d1d6dfa4c539b4fff7195658cc58a4326bb9f0449a231685d312b4a1ec48ca535a838bfa5c680787fe0e61473a2a092c448937d0'
-        inherits = ['python_setuptools_build_meta']
+        inherits = ['python_setuptools_build_meta', 'pypi']
 
         self._test_recipe_contents(recipefile, checkvars, inherits)
 
@@ -532,13 +604,7 @@ class RecipetoolCreateTests(RecipetoolBase):
         checkvars['SUMMARY'] = 'Simple module to parse ISO 8601 dates'
         checkvars['LICENSE'] = set(['MIT'])
         checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=aab31f2ef7ba214a5a341eaa47a7f367'
-        checkvars['SRC_URI'] = 'https://files.pythonhosted.org/packages/b9/f3/ef59cee614d5e0accf6fd0cbba025b93b272e626ca89fb70a3e9187c5d15/iso8601-${PV}.tar.gz'
-        checkvars['SRC_URI[md5sum]'] = '6e33910eba87066b3be7fcf3d59d16b5'
-        checkvars['SRC_URI[sha1sum]'] = 'efd225b2c9fa7d9e4a1ec6ad94f3295cee982e61'
-        checkvars['SRC_URI[sha256sum]'] = '6b1d3829ee8921c4301998c909f7829fa9ed3cbdac0d3b16af2d743aed1ba8df'
-        checkvars['SRC_URI[sha384sum]'] = '255002433fe65c19adfd6b91494271b613cb25ef6a35ac77436de1e03d60cc07bf89fd716451b917f1435e4384860ef6'
-        checkvars['SRC_URI[sha512sum]'] = 'db57ab2a25ef91e3bc479c8539d27e853cf1fbf60986820b8999ae15d7e566425a1e0cfba47d0f3b23aa703db0576db368e6c110ba2a2f46c9a34e8ee3611fb7'
-        inherits = ['python_poetry_core']
+        inherits = ['python_poetry_core', 'pypi']
 
         self._test_recipe_contents(recipefile, checkvars, inherits)
 
@@ -566,13 +632,7 @@ class RecipetoolCreateTests(RecipetoolBase):
         checkvars['SUMMARY'] = 'Backported and Experimental Type Hints for Python 3.8+'
         checkvars['LICENSE'] = set(['PSF-2.0'])
         checkvars['LIC_FILES_CHKSUM'] = 'file://LICENSE;md5=fcf6b249c2641540219a727f35d8d2c2'
-        checkvars['SRC_URI'] = 'https://files.pythonhosted.org/packages/1f/7a/8b94bb016069caa12fc9f587b28080ac33b4fbb8ca369b98bc0a4828543e/typing_extensions-${PV}.tar.gz'
-        checkvars['SRC_URI[md5sum]'] = '74bafe841fbd1c27324afdeb099babdf'
-        checkvars['SRC_URI[sha1sum]'] = 'f8bed69cbad4a57a1a67bf8a31b62b657b47f7a3'
-        checkvars['SRC_URI[sha256sum]'] = 'df8e4339e9cb77357558cbdbceca33c303714cf861d1eef15e1070055ae8b7ef'
-        checkvars['SRC_URI[sha384sum]'] = '0bd0112234134d965c6884f3c1f95d27b6ae49cfb08108101158e31dff33c2dce729331628b69818850f1acb68f6c8d0'
-        checkvars['SRC_URI[sha512sum]'] = '5fbff10e085fbf3ac2e35d08d913608d8c8bca66903435ede91cdc7776d775689a53d64f5f0615fe687c6c228ac854c8651d99eb1cb96ec61c56b7ca01fdd440'
-        inherits = ['python_flit_core']
+        inherits = ['python_flit_core', 'pypi']
 
         self._test_recipe_contents(recipefile, checkvars, inherits)
 
@@ -601,13 +661,7 @@ class RecipetoolCreateTests(RecipetoolBase):
         checkvars['HOMEPAGE'] = 'https://github.com/python-jsonschema/jsonschema'
         checkvars['LICENSE'] = set(['MIT'])
         checkvars['LIC_FILES_CHKSUM'] = 'file://COPYING;md5=7a60a81c146ec25599a3e1dabb8610a8 file://json/LICENSE;md5=9d4de43111d33570c8fe49b4cb0e01af'
-        checkvars['SRC_URI'] = 'https://files.pythonhosted.org/packages/e4/43/087b24516db11722c8687e0caf0f66c7785c0b1c51b0ab951dfde924e3f5/jsonschema-${PV}.tar.gz'
-        checkvars['SRC_URI[md5sum]'] = '4d6667ce76f820c35082c2d60a4896ab'
-        checkvars['SRC_URI[sha1sum]'] = '9173714cb88964d07f3a3f4fcaaef638b8ceac0c'
-        checkvars['SRC_URI[sha256sum]'] = 'ec84cc37cfa703ef7cd4928db24f9cb31428a5d0fa77747b8b51a847458e0bbf'
-        checkvars['SRC_URI[sha384sum]'] = '7a53181f0e679aa3dc3eb4d05a420877b7b9bff2d02e81f5c289a37ed1127d6c0cca1f5a5f9e4e166f089ab36bcc2be9'
-        checkvars['SRC_URI[sha512sum]'] = '60fa769faf6e3fc2c14eb9acd189c86e9d366b157230a5681d36552af0c159cb1ad33fd920668a36afdab98bc97253f91501704c5c07b5009fdaf9d29b52060d'
-        inherits = ['python_hatchling']
+        inherits = ['python_hatchling', 'pypi']
 
         self._test_recipe_contents(recipefile, checkvars, inherits)
 
-- 
2.42.0

Re: [OE-core] [RESEND][PATCH 1/7] bitbake: utils: remove spaces on empty lines

[Alexandre Belloni]

Please do not send bitbake patches in the same series as oe-core patches
if there is no dependency. Those patches are applied on different
repositories.

On 04/12/2023 16:59:28+0100, Julien Stephan wrote:
> Signed-off-by: Julien Stephan <jstephan@baylibre.com>
> ---
>  bitbake/lib/bb/utils.py | 16 ++++++++--------
>  1 file changed, 8 insertions(+), 8 deletions(-)
> 
> diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
> index b401fa5ec7a..61ffad92ce3 100644
> --- a/bitbake/lib/bb/utils.py
> +++ b/bitbake/lib/bb/utils.py
> @@ -50,7 +50,7 @@ def clean_context():
>  
>  def get_context():
>      return _context
> -    
> +
>  
>  def set_context(ctx):
>      _context = ctx
> @@ -212,8 +212,8 @@ def explode_dep_versions2(s, *, sort=True):
>              inversion = True
>              # This list is based on behavior and supported comparisons from deb, opkg and rpm.
>              #
> -            # Even though =<, <<, ==, !=, =>, and >> may not be supported, 
> -            # we list each possibly valid item. 
> +            # Even though =<, <<, ==, !=, =>, and >> may not be supported,
> +            # we list each possibly valid item.
>              # The build system is responsible for validation of what it supports.
>              if i.startswith(('<=', '=<', '<<', '==', '!=', '>=', '=>', '>>')):
>                  lastcmp = i[0:2]
> @@ -347,7 +347,7 @@ def _print_exception(t, value, tb, realfile, text, context):
>          exception = traceback.format_exception_only(t, value)
>          error.append('Error executing a python function in %s:\n' % realfile)
>  
> -        # Strip 'us' from the stack (better_exec call) unless that was where the 
> +        # Strip 'us' from the stack (better_exec call) unless that was where the
>          # error came from
>          if tb.tb_next is not None:
>              tb = tb.tb_next
> @@ -746,9 +746,9 @@ def prunedir(topdir, ionice=False):
>  # but thats possibly insane and suffixes is probably going to be small
>  #
>  def prune_suffix(var, suffixes, d):
> -    """ 
> +    """
>      See if var ends with any of the suffixes listed and
> -    remove it if found 
> +    remove it if found
>      """
>      for suffix in suffixes:
>          if suffix and var.endswith(suffix):
> @@ -1001,9 +1001,9 @@ def umask(new_mask):
>          os.umask(current_mask)
>  
>  def to_boolean(string, default=None):
> -    """ 
> +    """
>      Check input string and return boolean value True/False/None
> -    depending upon the checks 
> +    depending upon the checks
>      """
>      if not string:
>          return default
> -- 
> 2.42.0
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#191754): https://lists.openembedded.org/g/openembedded-core/message/191754
> Mute This Topic: https://lists.openembedded.org/mt/102972946/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [RESEND][PATCH 1/7] bitbake: utils: remove spaces on empty lines

[Julien Stephan]

Le mar. 5 déc. 2023 à 08:50, Alexandre Belloni
<alexandre.belloni@bootlin.com> a écrit :
>
> Please do not send bitbake patches in the same series as oe-core patches
> if there is no dependency. Those patches are applied on different
> repositories.

Hi Alexandre,

Noted! Do you want me to resend the series without the bitbake patch?

Cheers
Julien
>
> On 04/12/2023 16:59:28+0100, Julien Stephan wrote:
> > Signed-off-by: Julien Stephan <jstephan@baylibre.com>
> > ---
> >  bitbake/lib/bb/utils.py | 16 ++++++++--------
> >  1 file changed, 8 insertions(+), 8 deletions(-)
> >
> > diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
> > index b401fa5ec7a..61ffad92ce3 100644
> > --- a/bitbake/lib/bb/utils.py
> > +++ b/bitbake/lib/bb/utils.py
> > @@ -50,7 +50,7 @@ def clean_context():
> >
> >  def get_context():
> >      return _context
> > -
> > +
> >
> >  def set_context(ctx):
> >      _context = ctx
> > @@ -212,8 +212,8 @@ def explode_dep_versions2(s, *, sort=True):
> >              inversion = True
> >              # This list is based on behavior and supported comparisons from deb, opkg and rpm.
> >              #
> > -            # Even though =<, <<, ==, !=, =>, and >> may not be supported,
> > -            # we list each possibly valid item.
> > +            # Even though =<, <<, ==, !=, =>, and >> may not be supported,
> > +            # we list each possibly valid item.
> >              # The build system is responsible for validation of what it supports.
> >              if i.startswith(('<=', '=<', '<<', '==', '!=', '>=', '=>', '>>')):
> >                  lastcmp = i[0:2]
> > @@ -347,7 +347,7 @@ def _print_exception(t, value, tb, realfile, text, context):
> >          exception = traceback.format_exception_only(t, value)
> >          error.append('Error executing a python function in %s:\n' % realfile)
> >
> > -        # Strip 'us' from the stack (better_exec call) unless that was where the
> > +        # Strip 'us' from the stack (better_exec call) unless that was where the
> >          # error came from
> >          if tb.tb_next is not None:
> >              tb = tb.tb_next
> > @@ -746,9 +746,9 @@ def prunedir(topdir, ionice=False):
> >  # but thats possibly insane and suffixes is probably going to be small
> >  #
> >  def prune_suffix(var, suffixes, d):
> > -    """
> > +    """
> >      See if var ends with any of the suffixes listed and
> > -    remove it if found
> > +    remove it if found
> >      """
> >      for suffix in suffixes:
> >          if suffix and var.endswith(suffix):
> > @@ -1001,9 +1001,9 @@ def umask(new_mask):
> >          os.umask(current_mask)
> >
> >  def to_boolean(string, default=None):
> > -    """
> > +    """
> >      Check input string and return boolean value True/False/None
> > -    depending upon the checks
> > +    depending upon the checks
> >      """
> >      if not string:
> >          return default
> > --
> > 2.42.0
> >
>
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#191754): https://lists.openembedded.org/g/openembedded-core/message/191754
> > Mute This Topic: https://lists.openembedded.org/mt/102972946/3617179
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

Re: [OE-core] [RESEND][PATCH 1/7] bitbake: utils: remove spaces on empty lines

[Alexandre Belloni]

On 05/12/2023 15:57:58+0100, Julien Stephan wrote:
> Le mar. 5 d�c. 2023 � 08:50, Alexandre Belloni
> <alexandre.belloni@bootlin.com> a �crit :
> >
> > Please do not send bitbake patches in the same series as oe-core patches
> > if there is no dependency. Those patches are applied on different
> > repositories.
> 
> Hi Alexandre,
> 
> Noted! Do you want me to resend the series without the bitbake patch?

No but  didn't take the bitbake patch because it doesn't do what the
commit message says. (i.e. it touches non empty lines)

> 
> Cheers
> Julien
> >
> > On 04/12/2023 16:59:28+0100, Julien Stephan wrote:
> > > Signed-off-by: Julien Stephan <jstephan@baylibre.com>
> > > ---
> > >  bitbake/lib/bb/utils.py | 16 ++++++++--------
> > >  1 file changed, 8 insertions(+), 8 deletions(-)
> > >
> > > diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
> > > index b401fa5ec7a..61ffad92ce3 100644
> > > --- a/bitbake/lib/bb/utils.py
> > > +++ b/bitbake/lib/bb/utils.py
> > > @@ -50,7 +50,7 @@ def clean_context():
> > >
> > >  def get_context():
> > >      return _context
> > > -
> > > +
> > >
> > >  def set_context(ctx):
> > >      _context = ctx
> > > @@ -212,8 +212,8 @@ def explode_dep_versions2(s, *, sort=True):
> > >              inversion = True
> > >              # This list is based on behavior and supported comparisons from deb, opkg and rpm.
> > >              #
> > > -            # Even though =<, <<, ==, !=, =>, and >> may not be supported,
> > > -            # we list each possibly valid item.
> > > +            # Even though =<, <<, ==, !=, =>, and >> may not be supported,
> > > +            # we list each possibly valid item.
> > >              # The build system is responsible for validation of what it supports.
> > >              if i.startswith(('<=', '=<', '<<', '==', '!=', '>=', '=>', '>>')):
> > >                  lastcmp = i[0:2]
> > > @@ -347,7 +347,7 @@ def _print_exception(t, value, tb, realfile, text, context):
> > >          exception = traceback.format_exception_only(t, value)
> > >          error.append('Error executing a python function in %s:\n' % realfile)
> > >
> > > -        # Strip 'us' from the stack (better_exec call) unless that was where the
> > > +        # Strip 'us' from the stack (better_exec call) unless that was where the
> > >          # error came from
> > >          if tb.tb_next is not None:
> > >              tb = tb.tb_next
> > > @@ -746,9 +746,9 @@ def prunedir(topdir, ionice=False):
> > >  # but thats possibly insane and suffixes is probably going to be small
> > >  #
> > >  def prune_suffix(var, suffixes, d):
> > > -    """
> > > +    """
> > >      See if var ends with any of the suffixes listed and
> > > -    remove it if found
> > > +    remove it if found
> > >      """
> > >      for suffix in suffixes:
> > >          if suffix and var.endswith(suffix):
> > > @@ -1001,9 +1001,9 @@ def umask(new_mask):
> > >          os.umask(current_mask)
> > >
> > >  def to_boolean(string, default=None):
> > > -    """
> > > +    """
> > >      Check input string and return boolean value True/False/None
> > > -    depending upon the checks
> > > +    depending upon the checks
> > >      """
> > >      if not string:
> > >          return default
> > > --
> > > 2.42.0
> > >
> >
> > >
> > > -=-=-=-=-=-=-=-=-=-=-=-
> > > Links: You receive all messages sent to this group.
> > > View/Reply Online (#191754): https://lists.openembedded.org/g/openembedded-core/message/191754
> > > Mute This Topic: https://lists.openembedded.org/mt/102972946/3617179
> > > Group Owner: openembedded-core+owner@lists.openembedded.org
> > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> > > -=-=-=-=-=-=-=-=-=-=-=-
> > >
> >
> >
> > --
> > Alexandre Belloni, co-owner and COO, Bootlin
> > Embedded Linux and Kernel engineering
> > https://bootlin.com

-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [RESEND][PATCH 1/7] bitbake: utils: remove spaces on empty lines

[Julien Stephan]

Le mar. 5 déc. 2023 à 16:45, Alexandre Belloni
<alexandre.belloni@bootlin.com> a écrit :
>
> On 05/12/2023 15:57:58+0100, Julien Stephan wrote:
> > Le mar. 5 déc. 2023 à 08:50, Alexandre Belloni
> > <alexandre.belloni@bootlin.com> a écrit :
> > >
> > > Please do not send bitbake patches in the same series as oe-core patches
> > > if there is no dependency. Those patches are applied on different
> > > repositories.
> >
> > Hi Alexandre,
> >
> > Noted! Do you want me to resend the series without the bitbake patch?
>
> No but  didn't take the bitbake patch because it doesn't do what the
> commit message says. (i.e. it touches non empty lines)
>

oups, I will update the commit message and send it to the appropriate list.

> >
> > Cheers
> > Julien
> > >
> > > On 04/12/2023 16:59:28+0100, Julien Stephan wrote:
> > > > Signed-off-by: Julien Stephan <jstephan@baylibre.com>
> > > > ---
> > > >  bitbake/lib/bb/utils.py | 16 ++++++++--------
> > > >  1 file changed, 8 insertions(+), 8 deletions(-)
> > > >
> > > > diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
> > > > index b401fa5ec7a..61ffad92ce3 100644
> > > > --- a/bitbake/lib/bb/utils.py
> > > > +++ b/bitbake/lib/bb/utils.py
> > > > @@ -50,7 +50,7 @@ def clean_context():
> > > >
> > > >  def get_context():
> > > >      return _context
> > > > -
> > > > +
> > > >
> > > >  def set_context(ctx):
> > > >      _context = ctx
> > > > @@ -212,8 +212,8 @@ def explode_dep_versions2(s, *, sort=True):
> > > >              inversion = True
> > > >              # This list is based on behavior and supported comparisons from deb, opkg and rpm.
> > > >              #
> > > > -            # Even though =<, <<, ==, !=, =>, and >> may not be supported,
> > > > -            # we list each possibly valid item.
> > > > +            # Even though =<, <<, ==, !=, =>, and >> may not be supported,
> > > > +            # we list each possibly valid item.
> > > >              # The build system is responsible for validation of what it supports.
> > > >              if i.startswith(('<=', '=<', '<<', '==', '!=', '>=', '=>', '>>')):
> > > >                  lastcmp = i[0:2]
> > > > @@ -347,7 +347,7 @@ def _print_exception(t, value, tb, realfile, text, context):
> > > >          exception = traceback.format_exception_only(t, value)
> > > >          error.append('Error executing a python function in %s:\n' % realfile)
> > > >
> > > > -        # Strip 'us' from the stack (better_exec call) unless that was where the
> > > > +        # Strip 'us' from the stack (better_exec call) unless that was where the
> > > >          # error came from
> > > >          if tb.tb_next is not None:
> > > >              tb = tb.tb_next
> > > > @@ -746,9 +746,9 @@ def prunedir(topdir, ionice=False):
> > > >  # but thats possibly insane and suffixes is probably going to be small
> > > >  #
> > > >  def prune_suffix(var, suffixes, d):
> > > > -    """
> > > > +    """
> > > >      See if var ends with any of the suffixes listed and
> > > > -    remove it if found
> > > > +    remove it if found
> > > >      """
> > > >      for suffix in suffixes:
> > > >          if suffix and var.endswith(suffix):
> > > > @@ -1001,9 +1001,9 @@ def umask(new_mask):
> > > >          os.umask(current_mask)
> > > >
> > > >  def to_boolean(string, default=None):
> > > > -    """
> > > > +    """
> > > >      Check input string and return boolean value True/False/None
> > > > -    depending upon the checks
> > > > +    depending upon the checks
> > > >      """
> > > >      if not string:
> > > >          return default
> > > > --
> > > > 2.42.0
> > > >
> > >
> > > >
> > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > > Links: You receive all messages sent to this group.
> > > > View/Reply Online (#191754): https://lists.openembedded.org/g/openembedded-core/message/191754
> > > > Mute This Topic: https://lists.openembedded.org/mt/102972946/3617179
> > > > Group Owner: openembedded-core+owner@lists.openembedded.org
> > > > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> > > > -=-=-=-=-=-=-=-=-=-=-=-
> > > >
> > >
> > >
> > > --
> > > Alexandre Belloni, co-owner and COO, Bootlin
> > > Embedded Linux and Kernel engineering
> > > https://bootlin.com
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

Re: [OE-core] [RESEND][PATCH 1/7] bitbake: utils: remove spaces on empty lines

[Julien Stephan]

Le mar. 5 déc. 2023 à 17:27, Julien Stephan via lists.openembedded.org
<jstephan=baylibre.com@lists.openembedded.org> a écrit :
>
> Le mar. 5 déc. 2023 à 16:45, Alexandre Belloni
> <alexandre.belloni@bootlin.com> a écrit :
> >
> > On 05/12/2023 15:57:58+0100, Julien Stephan wrote:
> > > Le mar. 5 déc. 2023 à 08:50, Alexandre Belloni
> > > <alexandre.belloni@bootlin.com> a écrit :
> > > >
> > > > Please do not send bitbake patches in the same series as oe-core patches
> > > > if there is no dependency. Those patches are applied on different
> > > > repositories.
> > >
> > > Hi Alexandre,
> > >
> > > Noted! Do you want me to resend the series without the bitbake patch?
> >
> > No but  didn't take the bitbake patch because it doesn't do what the
> > commit message says. (i.e. it touches non empty lines)
> >
>
> oups, I will update the commit message and send it to the appropriate list.
>

It has already been merged with the incorrect commit message :( sorry for that

Cheers
Julien
> > >
> > > Cheers
> > > Julien
> > > >
> > > > On 04/12/2023 16:59:28+0100, Julien Stephan wrote:
> > > > > Signed-off-by: Julien Stephan <jstephan@baylibre.com>
> > > > > ---
> > > > >  bitbake/lib/bb/utils.py | 16 ++++++++--------
> > > > >  1 file changed, 8 insertions(+), 8 deletions(-)
> > > > >
> > > > > diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
> > > > > index b401fa5ec7a..61ffad92ce3 100644
> > > > > --- a/bitbake/lib/bb/utils.py
> > > > > +++ b/bitbake/lib/bb/utils.py
> > > > > @@ -50,7 +50,7 @@ def clean_context():
> > > > >
> > > > >  def get_context():
> > > > >      return _context
> > > > > -
> > > > > +
> > > > >
> > > > >  def set_context(ctx):
> > > > >      _context = ctx
> > > > > @@ -212,8 +212,8 @@ def explode_dep_versions2(s, *, sort=True):
> > > > >              inversion = True
> > > > >              # This list is based on behavior and supported comparisons from deb, opkg and rpm.
> > > > >              #
> > > > > -            # Even though =<, <<, ==, !=, =>, and >> may not be supported,
> > > > > -            # we list each possibly valid item.
> > > > > +            # Even though =<, <<, ==, !=, =>, and >> may not be supported,
> > > > > +            # we list each possibly valid item.
> > > > >              # The build system is responsible for validation of what it supports.
> > > > >              if i.startswith(('<=', '=<', '<<', '==', '!=', '>=', '=>', '>>')):
> > > > >                  lastcmp = i[0:2]
> > > > > @@ -347,7 +347,7 @@ def _print_exception(t, value, tb, realfile, text, context):
> > > > >          exception = traceback.format_exception_only(t, value)
> > > > >          error.append('Error executing a python function in %s:\n' % realfile)
> > > > >
> > > > > -        # Strip 'us' from the stack (better_exec call) unless that was where the
> > > > > +        # Strip 'us' from the stack (better_exec call) unless that was where the
> > > > >          # error came from
> > > > >          if tb.tb_next is not None:
> > > > >              tb = tb.tb_next
> > > > > @@ -746,9 +746,9 @@ def prunedir(topdir, ionice=False):
> > > > >  # but thats possibly insane and suffixes is probably going to be small
> > > > >  #
> > > > >  def prune_suffix(var, suffixes, d):
> > > > > -    """
> > > > > +    """
> > > > >      See if var ends with any of the suffixes listed and
> > > > > -    remove it if found
> > > > > +    remove it if found
> > > > >      """
> > > > >      for suffix in suffixes:
> > > > >          if suffix and var.endswith(suffix):
> > > > > @@ -1001,9 +1001,9 @@ def umask(new_mask):
> > > > >          os.umask(current_mask)
> > > > >
> > > > >  def to_boolean(string, default=None):
> > > > > -    """
> > > > > +    """
> > > > >      Check input string and return boolean value True/False/None
> > > > > -    depending upon the checks
> > > > > +    depending upon the checks
> > > > >      """
> > > > >      if not string:
> > > > >          return default
> > > > > --
> > > > > 2.42.0
> > > > >
> > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Alexandre Belloni, co-owner and COO, Bootlin
> > > > Embedded Linux and Kernel engineering
> > > > https://bootlin.com
> >
> > --
> > Alexandre Belloni, co-owner and COO, Bootlin
> > Embedded Linux and Kernel engineering
> > https://bootlin.com
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#191847): https://lists.openembedded.org/g/openembedded-core/message/191847
> Mute This Topic: https://lists.openembedded.org/mt/102972946/4356723
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [jstephan@baylibre.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [RESEND][PATCH 5/7] recipetool: create_buildsys_python: add pypi support

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 9946 bytes --]

On Mon, Dec 4, 2023 at 7:59 AM Julien Stephan <jstephan@baylibre.com> wrote:

> Today, we can use devtool/recipetool to create recipes for python projects
> using the github url or the direct release tarball of the project, but the
> create_buildsys_python plugin doesn't support the pypi class, since we
> cannot
> know from the extracted source if the package is available on pypi or not.
>
> By implementing the new optional process_url callback, we can detect
> that the url is a pypi one (i.e 'https://pypi.org/project/<package>')
> and retrieve the release tarball location.
> Also detect if the url points to a release tarball hosted on
> "files.pythonhosted.iorg" (i.e https://files.pythonhosted.org/packages/..
> .)
>
> In both cases, adds the pypi class, remove 'S' and 'SRC_URIxxx'
> variables from the created recipe as they will be handled by the pypi class
> and add the PYPI_PACKAGE variable
>
> This helps to produce cleaner recipes when package is hosted on pypi.
>
> If the url points to a github url or a release tarball not coming from
> "files.pythonhosted.org", the created recipe is the same as before.
> One can also use the newly added "--no-pypi" switch to NOT inherit
> from pypi class on matching url, to keep legacy behaviour.
>
> To create a recipe for a pypi package, one can now use one of the
> new following syntax (using recipetool create / devtool add):
>
> * recipetool create https://pypi.org/project/<package>
> * recipetool create https://pypi.org/project/<package>/<version>
> * recipetool create https://pypi.org/project/<package> --version <version>
>
> or the old syntax:
> * recipetool create https://files.pythonhosted.org/packages/<...>
>
> Signed-off-by: Julien Stephan <jstephan@baylibre.com>
> ---
>  scripts/lib/devtool/standard.py               |  3 +
>  scripts/lib/recipetool/create.py              |  1 +
>  .../lib/recipetool/create_buildsys_python.py  | 72 +++++++++++++++++++
>  3 files changed, 76 insertions(+)
>
> diff --git a/scripts/lib/devtool/standard.py
> b/scripts/lib/devtool/standard.py
> index d53fb810071..f18ebaa70d3 100644
> --- a/scripts/lib/devtool/standard.py
> +++ b/scripts/lib/devtool/standard.py
> @@ -147,6 +147,8 @@ def add(args, config, basepath, workspace):
>          extracmdopts += ' -a'
>      if args.npm_dev:
>          extracmdopts += ' --npm-dev'
> +    if args.no_pypi:
> +        extracmdopts += ' --no-pypi'
>      if args.mirrors:
>          extracmdopts += ' --mirrors'
>      if args.srcrev:
> @@ -2260,6 +2262,7 @@ def register_commands(subparsers, context):
>      group.add_argument('--no-same-dir', help='Force build in a separate
> build directory', action="store_true")
>      parser_add.add_argument('--fetch', '-f', help='Fetch the specified
> URI and extract it to create the source tree (deprecated - pass as
> positional argument instead)', metavar='URI')
>      parser_add.add_argument('--npm-dev', help='For npm, also fetch
> devDependencies', action="store_true")
> +    parser_add.add_argument('--no-pypi', help='Do not inherit pypi
> class', action="store_true")
>      parser_add.add_argument('--version', '-V', help='Version to use
> within recipe (PV)')
>      parser_add.add_argument('--no-git', '-g', help='If fetching source,
> do not set up source tree as a git repository', action="store_true")
>      group = parser_add.add_mutually_exclusive_group()
> diff --git a/scripts/lib/recipetool/create.py
> b/scripts/lib/recipetool/create.py
> index 5c5ac7ae403..963aa91421e 100644
> --- a/scripts/lib/recipetool/create.py
> +++ b/scripts/lib/recipetool/create.py
> @@ -1413,6 +1413,7 @@ def register_commands(subparsers):
>      parser_create.add_argument('-B', '--srcbranch', help='Branch in
> source repository if fetching from an SCM such as git (default master)')
>      parser_create.add_argument('--keep-temp', action="store_true",
> help='Keep temporary directory (for debugging)')
>      parser_create.add_argument('--npm-dev', action="store_true",
> help='For npm, also fetch devDependencies')
> +    parser_create.add_argument('--no-pypi', action="store_true", help='Do
> not inherit pypi class')
>      parser_create.add_argument('--devtool', action="store_true",
> help=argparse.SUPPRESS)
>      parser_create.add_argument('--mirrors', action="store_true",
> help='Enable PREMIRRORS and MIRRORS for source tree fetching (disabled by
> default).')
>      parser_create.set_defaults(func=create_recipe)
> diff --git a/scripts/lib/recipetool/create_buildsys_python.py
> b/scripts/lib/recipetool/create_buildsys_python.py
> index b620e3271b1..5e07222ece1 100644
> --- a/scripts/lib/recipetool/create_buildsys_python.py
> +++ b/scripts/lib/recipetool/create_buildsys_python.py
> @@ -18,7 +18,11 @@ import os
>  import re
>  import sys
>  import subprocess
> +import json
> +import urllib.request
>  from recipetool.create import RecipeHandler
> +from urllib.parse import urldefrag
> +from recipetool.create import determine_from_url
>
>  logger = logging.getLogger('recipetool')
>
> @@ -111,6 +115,74 @@ class PythonRecipeHandler(RecipeHandler):
>      def __init__(self):
>          pass
>
> +    def process_url(self, args, classes, handled, extravalues):
> +        """
> +        Convert any pypi url https://pypi.org/project/<package>/<version>
> into https://files.pythonhosted.org/packages/source/...
> +        which corresponds to the archive location, and add pypi class
> +        """
> +
> +        if 'url' in handled:
> +            return None
> +
> +        fetch_uri = None
> +        source = args.source
> +        required_version = args.version if args.version else None
> +        match = re.match(r'https?://pypi.org/project/([
> <http://pypi.org/project/(%5B>^/]+)(?:/([^/]+))?/?$',
> urldefrag(source)[0])
> +        if match:
> +            package = match.group(1)
> +            version = match.group(2) if match.group(2) else
> required_version
> +
> +            json_url = f"https://pypi.org/pypi/%s/json" % package
> +            response = urllib.request.urlopen(json_url)
> +            if response.status == 200:
> +                data = json.loads(response.read())
> +                if not version:
> +                    # grab latest version
> +                    version = data["info"]["version"]
> +                pypi_package = data["info"]["name"]
> +                for release in reversed(data["releases"][version]):
> +                    if release["packagetype"] == "sdist":
> +                        fetch_uri = release["url"]
> +                        break
> +            else:
> +                logger.warning("Cannot handle pypi url %s: cannot fetch
> package information using %s", source, json_url)
> +                return None
> +        else:
> +            match = re.match(r'^https?://
> files.pythonhosted.org/packages.*/(.*)-.*$', source)
> +            if match:
> +                fetch_uri = source
> +                pypi_package = match.group(1)
> +                _, version = determine_from_url(fetch_uri)
> +
> +        if match and not args.no_pypi:
> +            if required_version and version != required_version:
> +                raise Exception("Version specified using --version/-V
> (%s) and version specified in the url (%s) do not match" %
> (required_version, version))
> +            # This is optionnal if BPN looks like "python-<pypi_package>"
> or "python3-<pypi_package>" (see pypi.bbclass)
> +            # but at this point we cannot know because because user can
> specify the output name of the recipe on the command line
> +            extravalues["PYPI_PACKAGE"] = pypi_package
> +            # If the tarball extension is not 'tar.gz' (default value in
> pypi.bblcass) whe should set PYPI_PACKAGE_EXT in the recipe
> +            pypi_package_ext = re.match(r'.*%s-%s\.(.*)$' %
> (pypi_package, version), fetch_uri)
> +            if pypi_package_ext:
> +                pypi_package_ext = pypi_package_ext.group(1)
> +                if pypi_package_ext != "tar.gz":
> +                    extravalues["PYPI_PACKAGE_EXT"] = pypi_package_ext
> +
> +            # Pypi class will handle S and SRC_URIxxx variables, so
> remove them
> +            # TODO: allow oe.recipeutils.patch_recipe_lines() to accept
> regexp so we can simplify the following to:
> +            # extravalues['SRC_URI(?:\[.*?\])?'] = None
> +            extravalues['S'] = None
> +            extravalues['SRC_URI'] = None
> +            extravalues['SRC_URI[md5sum]'] = None
> +            extravalues['SRC_URI[sha1sum]'] = None
> +            extravalues['SRC_URI[sha256sum]'] = None
> +            extravalues['SRC_URI[sha384sum]'] = None
> +            extravalues['SRC_URI[sha512sum]'] = None
> +

this was not correct. The pypi.bbclass does not create the
SRC_URI[sha256sum]. It does create the SRC_URI. So new recipes will not
have any check on the validity of the downloaded tarball.

The only test case which checked for this was with —nopypi
https://git.yoctoproject.org/poky-contrib/tree/meta/lib/oeqa/selftest/cases/recipetool.py#n752


> +            classes.append('pypi')
> +
> +        handled.append('url')
> +        return fetch_uri
> +
>      def handle_classifier_license(self, classifiers,
> existing_licenses=""):
>
>          licenses = []
> --
> 2.42.0
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#191759):
> https://lists.openembedded.org/g/openembedded-core/message/191759
> Mute This Topic: https://lists.openembedded.org/mt/102972952/924729
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> ticotimo@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 13662 bytes --]