[PATCH] fetch2:init check if path is none before calculating checksums

[PATCH] fetch2:init check if path is none before calculating checksums

[belouargamohamed]

From: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>

Add one more verification that checks if localpath is None, because
we can't compute checksum of a None.

Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
---
 lib/bb/fetch2/__init__.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 770021de16..f6e5d19627 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1402,6 +1402,9 @@ class FetchMethod(object):
         Is localpath something that can be represented by a checksum?
         """
 
+        # We cannot compute checksums for None
+        if urldata.localpath is None:
+            return False
         # We cannot compute checksums for directories
         if os.path.isdir(urldata.localpath):
             return False
-- 
2.25.1

Re: [bitbake-devel] [PATCH] fetch2:init check if path is none before calculating checksums

[Richard Purdie]

On Sat, 2023-07-22 at 01:49 +0200, belouargamohamed@gmail.com wrote:
> From: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
> 
> Add one more verification that checks if localpath is None, because
> we can't compute checksum of a None.
> 
> Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
> ---
>  lib/bb/fetch2/__init__.py | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
> index 770021de16..f6e5d19627 100644
> --- a/lib/bb/fetch2/__init__.py
> +++ b/lib/bb/fetch2/__init__.py
> @@ -1402,6 +1402,9 @@ class FetchMethod(object):
>          Is localpath something that can be represented by a checksum?
>          """
>  
> +        # We cannot compute checksums for None
> +        if urldata.localpath is None:
> +            return False
>          # We cannot compute checksums for directories
>          if os.path.isdir(urldata.localpath):
>              return False

What was the situation which produced this error? I'm wondering if we
need a new test case in bitbake-selftest (in lib/bb/test/fetch.py)?

Cheers,

Richard

Re: [bitbake-devel] [PATCH] fetch2:init check if path is none before calculating checksums

[Frédéric Martinsons]

[-- Attachment #1: Type: text/plain, Size: 1586 bytes --]

Le lun. 24 juil. 2023, 15:51, Richard Purdie <
richard.purdie@linuxfoundation.org> a écrit :

> On Sat, 2023-07-22 at 01:49 +0200, belouargamohamed@gmail.com wrote:
> > From: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
> >
> > Add one more verification that checks if localpath is None, because
> > we can't compute checksum of a None.
> >
> > Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
> > ---
> >  lib/bb/fetch2/__init__.py | 3 +++
> >  1 file changed, 3 insertions(+)
> >
> > diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
> > index 770021de16..f6e5d19627 100644
> > --- a/lib/bb/fetch2/__init__.py
> > +++ b/lib/bb/fetch2/__init__.py
> > @@ -1402,6 +1402,9 @@ class FetchMethod(object):
> >          Is localpath something that can be represented by a checksum?
> >          """
> >
> > +        # We cannot compute checksums for None
> > +        if urldata.localpath is None:
> > +            return False
> >          # We cannot compute checksums for directories
> >          if os.path.isdir(urldata.localpath):
> >              return False
>
> What was the situation which produced this error? I'm wondering if we
> need a new test case in bitbake-selftest (in lib/bb/test/fetch.py)?
>
> Cheers,
>
> Richard
>

A new test for a new condition is always better. Moreover, if None can
happen for urldata.localpath, what about empty string? If it can happen,
you may want to replace the line with a simple "if not urldata.localpath"
which cover both None and empty string.

[-- Attachment #2: Type: text/html, Size: 2712 bytes --]

Re: [PATCH] fetch2:init check if path is none before calculating checksums

[belouargamohamed]

[-- Attachment #1: Type: text/plain, Size: 4806 bytes --]

The error happens when I was testing if npm works better.

I just generated a recipe and a shrinkwrap using the command:
- devtool add "npm://registry.npmjs.org;package=express-ws;version=5.0.2;"
Then I built it with:
- bitbake express-ws
And this error happened:
ERROR: express-ws-5.0.2-r0 do_create_spdx: Error executing a python function in exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_create_spdx(d)
0003:
File: '/home/twigabecomp/yocto/literal/debug/poky/meta/classes/create-spdx-2.2.bbclass', lineno: 598, function: do_create_spdx
0594:
0595:    doc.packages.append(recipe)
0596:    doc.add_relationship(doc, "DESCRIBES", recipe)
0597:
*** 0598:    add_download_packages(d, doc, recipe)
0599:
0600:    if process_sources(d) and include_sources:
0601:        recipe_archive = deploy_dir_spdx / "recipes" / (doc.name + ".tar.zst")
0602:        with optional_tarfile(recipe_archive, archive_sources) as archive:
File: '/home/twigabecomp/yocto/literal/debug/poky/meta/classes/create-spdx-2.2.bbclass', lineno: 438, function: add_download_packages
0434:
0435:            if f.method.supports_srcrev():
0436:                uri = uri + "@" + f.revisions[name]
0437:
*** 0438:            if f.method.supports_checksum(f):
0439:                for checksum_id in CHECKSUM_LIST:
0440:                    if checksum_id.upper() not in oe.spdx.SPDXPackage.ALLOWED_CHECKSUMS:
0441:                        continue
0442:
File: '/home/twigabecomp/yocto/literal/debug/poky/bitbake/lib/bb/fetch2/__init__.py', lineno: 1406, function: supports_checksum
1402:        Is localpath something that can be represented by a checksum?
1403:        """
1404:
1405:        # We cannot compute checksums for directories
*** 1406:        if os.path.isdir(urldata.localpath):
1407:            return False
1408:        return True
1409:
1410:    def recommends_checksum(self, urldata):
File: '/usr/lib/python3.10/genericpath.py', lineno: 42, function: isdir
0038:# can be true for the same path on systems that support symlinks
0039:def isdir(s):
0040:    """Return true if the pathname refers to an existing directory."""
0041:    try:
*** 0042:        st = os.stat(s)
0043:    except (OSError, ValueError):
0044:        return False
0045:    return stat.S_ISDIR(st.st_mode)
0046:
Exception: TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

ERROR: Logfile of failure stored in: /home/twigabecomp/yocto/literal/debug/build/tmp/work/core2-64-poky-linux/express-ws/5.0.2-r0/temp/log.do_create_spdx.3703234
ERROR: Task (/home/twigabecomp/yocto/literal/debug/build/workspace/recipes/express-ws/express-ws_5.0.2.bb:do_create_spdx) failed with exit code '1'

Knowing that urldata = {'donestamp': None, 'needdonestamp': False, 'localfile': '', 'localpath': None, 'lockfile': None, 'mirrortarballs': [], 'basename': None, 'basepath': None, 'type': 'npmsw', 'host': '', 'path': '/home/twigabecomp/yocto/literal/debug/build/workspace/recipes/express-ws/express-ws/npm-shrinkwrap.json', 'user': '', 'pswd': '', 'parm': OrderedDict(), 'date': '20230725', 'url': 'npmsw:///home/twigabecomp/yocto/literal/debug/build/workspace/recipes/express-ws/express-ws/npm-shrinkwrap.json', 'setup': False, 'names': ['default'], 'method': <bb.fetch2.npmsw.NpmShrinkWrap object at 0x7f50d8c9d600>, 'shrinkwrap_file': '/home/twigabecomp/yocto/literal/debug/build/workspace/recipes/express-ws/express-ws/npm-shrinkwrap.json', 'dev': False, 'deps': [{'url': 'https://registry.npmjs.org/ws/-/ws-7.5.9.tgz;downloadfilename=npm2/ws-7.5.9.tgz;sha512sum=17e3fd26297b52248a4a4a692220fde1d374ec0c2f162c6f2c88f53a0d5197d18632e362a499d5f49ce30fa5eeaa601e8acc06bd498d2e3af9705b97c0d4bbed', 'localpath': '/home/twigabecomp/yocto/literal/debug/build/downloads/npm2/ws-7.5.9.tgz', 'extrapaths': ['/home/twigabecomp/yocto/literal/debug/build/downloads/npm2/ws-7.5.9.tgz.resolved'], 'destsuffix': 'node_modules/ws', 'unpack': True}], 'proxy': <bb.fetch2.Fetch object at 0x7f50d30ff250>, 'md5_name': 'md5sum', 'md5_expected': None, 'sha256_name': 'sha256sum', 'sha256_expected': None, 'sha1_name': 'sha1sum', 'sha1_expected': None, 'sha384_name': 'sha384sum', 'sha384_expected': None, 'sha512_name': 'sha512sum', 'sha512_expected': None, 'ignore_checksums': False}

I think this is a known issue of how npm dependencies are represented, because the real localpath(s) is urldata.deps[i].localpath

Mohamed BELOUARGA
Thanks

[-- Attachment #2: Type: text/html, Size: 6746 bytes --]