[Buildroot] [PATCH] scanpypi: rework download_package error handling

[Buildroot] [PATCH] scanpypi: rework download_package error handling

[yegorslists at googlemail.com]

From: Yegor Yefremov <yegorslists@googlemail.com>

Some packages don't provide source archive but only a wheel file. In
this case download variable is not defined. So define this variable at
the very beginning and check whether it is None after searching for
source archives in the metadata.

Bonus: fix PEP8 issue with wrong indentation.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
---
 utils/scanpypi | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/utils/scanpypi b/utils/scanpypi
index 8a2ae00434..12c96b842e 100755
--- a/utils/scanpypi
+++ b/utils/scanpypi
@@ -178,6 +178,7 @@ class BuildrootPackage():
         """
         Download a package using metadata from pypi
         """
+        download = None
         try:
             self.metadata['urls'][0]['filename']
         except IndexError:
@@ -201,7 +202,7 @@ class BuildrootPackage():
                 continue
             try:
                 print('Downloading package {pkg} from {url}...'.format(
-                      pkg=self.real_name, url=download_url['url']))
+                    pkg=self.real_name, url=download_url['url']))
                 download = six.moves.urllib.request.urlopen(download_url['url'])
             except six.moves.urllib.error.HTTPError as http_error:
                 download = http_error
@@ -213,11 +214,14 @@ class BuildrootPackage():
                 self.md5_sum = hashlib.md5(self.as_string).hexdigest()
                 if self.md5_sum == download_url['digests']['md5']:
                     break
-        else:
-            if download.__class__ == six.moves.urllib.error.HTTPError:
-                raise download
-            raise DownloadFailed('Failed to download package {pkg}'
+
+        if download is None:
+            raise DownloadFailed('Failed to download package {pkg}: '
+                                 'No source archive available'
                                  .format(pkg=self.real_name))
+        elif download.__class__ == six.moves.urllib.error.HTTPError:
+            raise download
+
         self.filename = self.used_url['filename']
         self.url = self.used_url['url']
 
-- 
2.17.0

[Buildroot] [PATCH] scanpypi: rework download_package error handling

[Peter Korsgaard]

>>>>> "yegorslists" == yegorslists  <yegorslists@googlemail.com> writes:

 > From: Yegor Yefremov <yegorslists@googlemail.com>
 > Some packages don't provide source archive but only a wheel file. In
 > this case download variable is not defined. So define this variable at
 > the very beginning and check whether it is None after searching for
 > source archives in the metadata.

 > Bonus: fix PEP8 issue with wrong indentation.

 > Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>

Committed, thanks.

Out of interest, are you using python 2.x or 3.x? I just tried scanpypi
for the validators module on my Debian (so python 2.7) laptop:

./utils/scanpypi validators
spdx_lookup module is not installed. This can lead to an inaccurate licence detection. Please install it via
pip install spdx_lookup
buildroot package name for validators: python-validators
Package: python-validators
Fetching package validators
Downloading package validators from https://files.pythonhosted.org/packages/45/7b/5b7b74208a3e0744d1a0efbfb1935fa46fa4cfe58d3d63f17c49c58c429c/validators-0.12.2.tar.gz...
Added packages 'six, decorator' as dependencies of python-validators
Checking if package ./package/python-validators already exists...
Error: Package ./package/python-validators already exists
Do you want to delete existing package ? [y/N]y
Creating ./package/python-validators/python-validators.mk...
Creating ./package/python-validators/python-validators.hash...
Creating ./package/python-validators/Config.in...
Traceback (most recent call last):
  File "./utils/scanpypi", line 720, in <module>
    main()
  File "./utils/scanpypi", line 712, in main
    package.create_config_in()
  File "./utils/scanpypi", line 606, in create_config_in
    config_file.writelines(lines)
TypeError: writelines() argument must be a sequence of strings

Printing out 'lines', it seems to be a mix of normal strings and unicode:

lines: ['config BR2_PACKAGE_PYTHON_VALIDATORS\n', '\tbool "python-validators"\n', '\tselect BR2_PACKAGE_PYTHON_SIX # runtime\n', '\tselect BR2_PACKAGE_PYTHON_DECORATOR # runtime\n', '\thelp\n', u'\t  Python Data Validation for Humans\u2122.\n', '\n', u'\t  https://github.com/kvesteri/validators\n']

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH] scanpypi: rework download_package error handling

[Yegor Yefremov]

On Fri, Jun 15, 2018 at 2:13 PM, Peter Korsgaard <peter@korsgaard.com> wrote:
>>>>>> "yegorslists" == yegorslists  <yegorslists@googlemail.com> writes:
>
>  > From: Yegor Yefremov <yegorslists@googlemail.com>
>  > Some packages don't provide source archive but only a wheel file. In
>  > this case download variable is not defined. So define this variable at
>  > the very beginning and check whether it is None after searching for
>  > source archives in the metadata.
>
>  > Bonus: fix PEP8 issue with wrong indentation.
>
>  > Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
>
> Committed, thanks.
>
> Out of interest, are you using python 2.x or 3.x? I just tried scanpypi
> for the validators module on my Debian (so python 2.7) laptop:
>
> ./utils/scanpypi validators
> spdx_lookup module is not installed. This can lead to an inaccurate licence detection. Please install it via
> pip install spdx_lookup
> buildroot package name for validators: python-validators
> Package: python-validators
> Fetching package validators
> Downloading package validators from https://files.pythonhosted.org/packages/45/7b/5b7b74208a3e0744d1a0efbfb1935fa46fa4cfe58d3d63f17c49c58c429c/validators-0.12.2.tar.gz...
> Added packages 'six, decorator' as dependencies of python-validators
> Checking if package ./package/python-validators already exists...
> Error: Package ./package/python-validators already exists
> Do you want to delete existing package ? [y/N]y
> Creating ./package/python-validators/python-validators.mk...
> Creating ./package/python-validators/python-validators.hash...
> Creating ./package/python-validators/Config.in...
> Traceback (most recent call last):
>   File "./utils/scanpypi", line 720, in <module>
>     main()
>   File "./utils/scanpypi", line 712, in main
>     package.create_config_in()
>   File "./utils/scanpypi", line 606, in create_config_in
>     config_file.writelines(lines)
> TypeError: writelines() argument must be a sequence of strings
>
> Printing out 'lines', it seems to be a mix of normal strings and unicode:
>
> lines: ['config BR2_PACKAGE_PYTHON_VALIDATORS\n', '\tbool "python-validators"\n', '\tselect BR2_PACKAGE_PYTHON_SIX # runtime\n', '\tselect BR2_PACKAGE_PYTHON_DECORATOR # runtime\n', '\thelp\n', u'\t  Python Data Validation for Humans\u2122.\n', '\n', u'\t  https://github.com/kvesteri/validators\n']

I'm still using Python 2 by default. In this case I think trademark
symbol is the problem. That's why it was important to use scanpypi
with Python 3.

Yegor

[Buildroot] [PATCH] scanpypi: rework download_package error handling

[Peter Korsgaard]

>>>>> "Yegor" == Yegor Yefremov <yegorslists@googlemail.com> writes:

Hi,

 >> Printing out 'lines', it seems to be a mix of normal strings and unicode:
 >> 
 >> lines: ['config BR2_PACKAGE_PYTHON_VALIDATORS\n', '\tbool "python-validators"\n', '\tselect BR2_PACKAGE_PYTHON_SIX # runtime\n', '\tselect BR2_PACKAGE_PYTHON_DECORATOR # runtime\n', '\thelp\n', u'\t  Python Data Validation for Humans\u2122.\n', '\n', u'\t  https://github.com/kvesteri/validators\n']

 > I'm still using Python 2 by default. In this case I think trademark
 > symbol is the problem.

Yes. Perhaps we should filter out these unicode characters? We probably
don't really want a TM character in the help text anyway.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH] scanpypi: rework download_package error handling

[Jaap Crezee]

On 06/15/18 14:13, Peter Korsgaard wrote:
> Out of interest, are you using python 2.x or 3.x? I just tried scanpypi
> for the validators module on my Debian (so python 2.7) laptop:

I use python 3(.6) as default python interpreter. The error you posted "sounds" vaguely familliar 
however.

regards,


Jaap