[PATCH] pypi: Allow override of PyPI archive name

[PATCH] pypi: Allow override of PyPI archive name

[Paul Barker]

Some packages on PyPI don't follow the usual expectations for archive
naming. For example, the archive for asyncio-mqtt 0.10.0 is named
asyncio_mqtt-0.10.0.tar.gz (with an underscore instead of the dash used
in the package name).

To handle these edge cases a new PYPI_ARCHIVE_NAME variable is
introduced. By default this is set to the expected archive name based on
the PyPI package name, version and extension but it can be set to a
different value if needed in a recipe which inherits the pypi class.

Signed-off-by: Paul Barker <paul@pbarker.dev>
---
 meta/classes/pypi.bbclass | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes/pypi.bbclass b/meta/classes/pypi.bbclass
index 272c220bca..9405d58601 100644
--- a/meta/classes/pypi.bbclass
+++ b/meta/classes/pypi.bbclass
@@ -8,12 +8,12 @@ def pypi_package(d):
 
 PYPI_PACKAGE ?= "${@pypi_package(d)}"
 PYPI_PACKAGE_EXT ?= "tar.gz"
+PYPI_ARCHIVE_NAME ?= "${PYPI_PACKAGE}-${PV}.${PYPI_PACKAGE_EXT}"
 
 def pypi_src_uri(d):
     package = d.getVar('PYPI_PACKAGE')
-    package_ext = d.getVar('PYPI_PACKAGE_EXT')
-    pv = d.getVar('PV')
-    return 'https://files.pythonhosted.org/packages/source/%s/%s/%s-%s.%s' % (package[0], package, package, pv, package_ext)
+    archive_name = d.getVar('PYPI_ARCHIVE_NAME')
+    return 'https://files.pythonhosted.org/packages/source/%s/%s/%s' % (package[0], package, archive_name)
 
 PYPI_SRC_URI ?= "${@pypi_src_uri(d)}"
 
-- 
2.31.1

Re: [OE-core] [PATCH] pypi: Allow override of PyPI archive name

[Khem Raj]

On 8/10/21 4:03 AM, Paul Barker wrote:
> Some packages on PyPI don't follow the usual expectations for archive
> naming. For example, the archive for asyncio-mqtt 0.10.0 is named
> asyncio_mqtt-0.10.0.tar.gz (with an underscore instead of the dash used
> in the package name).
> 
> To handle these edge cases a new PYPI_ARCHIVE_NAME variable is
> introduced. By default this is set to the expected archive name based on
> the PyPI package name, version and extension but it can be set to a
> different value if needed in a recipe which inherits the pypi class.
> 

how many such packages are there roughly ? does it deserve to be 
addressed via introducing another variable I wonder.

> Signed-off-by: Paul Barker <paul@pbarker.dev>
> ---
>   meta/classes/pypi.bbclass | 6 +++---
>   1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/classes/pypi.bbclass b/meta/classes/pypi.bbclass
> index 272c220bca..9405d58601 100644
> --- a/meta/classes/pypi.bbclass
> +++ b/meta/classes/pypi.bbclass
> @@ -8,12 +8,12 @@ def pypi_package(d):
>   
>   PYPI_PACKAGE ?= "${@pypi_package(d)}"
>   PYPI_PACKAGE_EXT ?= "tar.gz"
> +PYPI_ARCHIVE_NAME ?= "${PYPI_PACKAGE}-${PV}.${PYPI_PACKAGE_EXT}"
>   
>   def pypi_src_uri(d):
>       package = d.getVar('PYPI_PACKAGE')
> -    package_ext = d.getVar('PYPI_PACKAGE_EXT')
> -    pv = d.getVar('PV')
> -    return 'https://files.pythonhosted.org/packages/source/%s/%s/%s-%s.%s' % (package[0], package, package, pv, package_ext)
> +    archive_name = d.getVar('PYPI_ARCHIVE_NAME')
> +    return 'https://files.pythonhosted.org/packages/source/%s/%s/%s' % (package[0], package, archive_name)
>   
>   PYPI_SRC_URI ?= "${@pypi_src_uri(d)}"
>   
> 
> 
> 
> 
> 

Re: [OE-core] [PATCH] pypi: Allow override of PyPI archive name

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 2240 bytes --]

On Thu, Aug 12, 2021 at 5:58 PM Khem Raj <raj.khem@gmail.com> wrote:

>
>
> On 8/10/21 4:03 AM, Paul Barker wrote:
> > Some packages on PyPI don't follow the usual expectations for archive
> > naming. For example, the archive for asyncio-mqtt 0.10.0 is named
> > asyncio_mqtt-0.10.0.tar.gz (with an underscore instead of the dash used
> > in the package name).
> >
> > To handle these edge cases a new PYPI_ARCHIVE_NAME variable is
> > introduced. By default this is set to the expected archive name based on
> > the PyPI package name, version and extension but it can be set to a
> > different value if needed in a recipe which inherits the pypi class.
> >
>
> how many such packages are there roughly ? does it deserve to be
> addressed via introducing another variable I wonder.
>

There are 29 packages in meta-oe/meta-python that fail to check latest
upstream version and so far (13/29) it has been for very similar dash vs.
underscore in the SRC_URI (from local AUH run). Not sure it is the complete
solution, but it might help. Might need to apply to the upstream regex too
in that case?


> > Signed-off-by: Paul Barker <paul@pbarker.dev>
> > ---
> >   meta/classes/pypi.bbclass | 6 +++---
> >   1 file changed, 3 insertions(+), 3 deletions(-)
> >
> > diff --git a/meta/classes/pypi.bbclass b/meta/classes/pypi.bbclass
> > index 272c220bca..9405d58601 100644
> > --- a/meta/classes/pypi.bbclass
> > +++ b/meta/classes/pypi.bbclass
> > @@ -8,12 +8,12 @@ def pypi_package(d):
> >
> >   PYPI_PACKAGE ?= "${@pypi_package(d)}"
> >   PYPI_PACKAGE_EXT ?= "tar.gz"
> > +PYPI_ARCHIVE_NAME ?= "${PYPI_PACKAGE}-${PV}.${PYPI_PACKAGE_EXT}"
> >
> >   def pypi_src_uri(d):
> >       package = d.getVar('PYPI_PACKAGE')
> > -    package_ext = d.getVar('PYPI_PACKAGE_EXT')
> > -    pv = d.getVar('PV')
> > -    return '
> https://files.pythonhosted.org/packages/source/%s/%s/%s-%s.%s' %
> (package[0], package, package, pv, package_ext)
> > +    archive_name = d.getVar('PYPI_ARCHIVE_NAME')
> > +    return 'https://files.pythonhosted.org/packages/source/%s/%s/%s' %
> (package[0], package, archive_name)
> >
> >   PYPI_SRC_URI ?= "${@pypi_src_uri(d)}"
> >
> >
> >
> >
> >
> >
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 3313 bytes --]

Re: [OE-core] [PATCH] pypi: Allow override of PyPI archive name

[Khem Raj]

On 8/12/21 6:03 PM, Tim Orling wrote:
> 
> 
> On Thu, Aug 12, 2021 at 5:58 PM Khem Raj <raj.khem@gmail.com 
> <mailto:raj.khem@gmail.com>> wrote:
> 
> 
> 
>     On 8/10/21 4:03 AM, Paul Barker wrote:
>      > Some packages on PyPI don't follow the usual expectations for archive
>      > naming. For example, the archive for asyncio-mqtt 0.10.0 is named
>      > asyncio_mqtt-0.10.0.tar.gz (with an underscore instead of the
>     dash used
>      > in the package name).
>      >
>      > To handle these edge cases a new PYPI_ARCHIVE_NAME variable is
>      > introduced. By default this is set to the expected archive name
>     based on
>      > the PyPI package name, version and extension but it can be set to a
>      > different value if needed in a recipe which inherits the pypi class.
>      >
> 
>     how many such packages are there roughly ? does it deserve to be
>     addressed via introducing another variable I wonder.
> 
> 
> There are 29 packages in meta-oe/meta-python that fail to check latest 
> upstream version and so far (13/29) it has been for very similar dash 
> vs. underscore in the SRC_URI (from local AUH run). Not sure it is the 
> complete solution, but it might help. Might need to apply to the 
> upstream regex too in that case?

OK thats some number and yes extending it for upstream check would be an 
improvement.

> 
> 
>      > Signed-off-by: Paul Barker <paul@pbarker.dev
>     <mailto:paul@pbarker.dev>>
>      > ---
>      >   meta/classes/pypi.bbclass | 6 +++---
>      >   1 file changed, 3 insertions(+), 3 deletions(-)
>      >
>      > diff --git a/meta/classes/pypi.bbclass b/meta/classes/pypi.bbclass
>      > index 272c220bca..9405d58601 100644
>      > --- a/meta/classes/pypi.bbclass
>      > +++ b/meta/classes/pypi.bbclass
>      > @@ -8,12 +8,12 @@ def pypi_package(d):
>      >
>      >   PYPI_PACKAGE ?= "${@pypi_package(d)}"
>      >   PYPI_PACKAGE_EXT ?= "tar.gz"
>      > +PYPI_ARCHIVE_NAME ?= "${PYPI_PACKAGE}-${PV}.${PYPI_PACKAGE_EXT}"
>      >
>      >   def pypi_src_uri(d):
>      >       package = d.getVar('PYPI_PACKAGE')
>      > -    package_ext = d.getVar('PYPI_PACKAGE_EXT')
>      > -    pv = d.getVar('PV')
>      > -    return
>     'https://files.pythonhosted.org/packages/source/%s/%s/%s-%s.%s
>     <https://files.pythonhosted.org/packages/source/%s/%s/%s-%s.%s>' %
>     (package[0], package, package, pv, package_ext)
>      > +    archive_name = d.getVar('PYPI_ARCHIVE_NAME')
>      > +    return
>     'https://files.pythonhosted.org/packages/source/%s/%s/%s
>     <https://files.pythonhosted.org/packages/source/%s/%s/%s>' %
>     (package[0], package, archive_name)
>      >
>      >   PYPI_SRC_URI ?= "${@pypi_src_uri(d)}"
>      >
>      >
>      >
>      >
>      >
>      >
> 
>     
>