[bitbake][kirkstone][2.0][PATCH 00/14] Patch review

[bitbake][kirkstone][2.0][PATCH 00/14] Patch review

[Steve Sakoman]

Please review this set of patches for 2.0/kirkstone and have comments back
by end of day Thursday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/4220

The following changes since commit ce9fe70156e8f909a3a81da017b89ea61bc6fe38:

  utils: Pass lock argument in fileslocked (2022-09-02 11:44:47 +0100)

are available in the Git repository at:

  git://git.openembedded.org/bitbake-contrib stable/2.0-nut
  http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/2.0-nut

Gennaro Iorio (1):
  fetch2: gitsm: fix incorrect handling of git submodule relative urls

Jacob Kroon (1):
  bitbake-user-manual: Correct description of the ??= operator

Jose Quaresma (3):
  bitbake: bitbake-user-manual: hashserv can be accessed on a dedicated
    domain
  bb/utils: remove: check the path again the expand python glob
  bb/utils: movefile: use the logger for printing

Michael Opdenacker (3):
  doc: bitbake-user-manual: add explicit target for crates fetcher
  doc: bitbake-user-manual: document npm and npmsw fetchers
  bitbake-user-manual: npm fetcher: improve description of SRC_URI
    format

Mikko Rapeli (1):
  event.py: ignore exceptions from stdout and sterr operations in atexit

Ola x Nilsson (1):
  ConfHandler: Remove lingering close

Richard Purdie (4):
  runqueue: Fix unihash cache mismatch issues
  cooker: Drop sre_constants usage
  ConfHandler/BBHandler: Improve comment error messages and add tests
  fetch2: Ensure directory exists before creating symlink

 .../bitbake-user-manual-fetching.rst          | 78 ++++++++++++++++++-
 .../bitbake-user-manual-metadata.rst          | 55 +++++++++----
 .../bitbake-user-manual-ref-variables.rst     |  2 +-
 lib/bb/cooker.py                              |  5 +-
 lib/bb/event.py                               | 10 ++-
 lib/bb/fetch2/__init__.py                     |  2 +
 lib/bb/fetch2/gitsm.py                        |  2 +-
 lib/bb/parse/parse_py/BBHandler.py            |  4 +-
 lib/bb/parse/parse_py/ConfHandler.py          | 11 ++-
 lib/bb/runqueue.py                            |  3 +
 lib/bb/tests/parse.py                         | 23 ++++++
 lib/bb/utils.py                               | 16 ++--
 12 files changed, 172 insertions(+), 39 deletions(-)

-- 
2.25.1

[bitbake][kirkstone][2.0][PATCH 01/14] runqueue: Fix unihash cache mismatch issues

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Very occasionally we see errors in eSDK testing on the autobuilder where the task
hashes in the eSDK don't match what was just built. I was able to inspect one of
these build directories and noticed that the bb_unihashes.dat file in the eSDK
was zero sized. Whilst inspecting the code to understand the cause, I noticed that
updated hashes are not saved out in subsequent updates of the values in the rehash
process.

Add a missing sync call to ensure this happens.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 7912dabbcf444a3c3d971cca4a944a8b931e301b)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/runqueue.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 7eea72e6..111dc0ee 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2381,6 +2381,9 @@ class RunQueueExecute:
                     self.rqdata.runtaskentries[hashtid].unihash = unihash
                     bb.parse.siggen.set_unihash(hashtid, unihash)
                     toprocess.add(hashtid)
+                if torehash:
+                    # Need to save after set_unihash above
+                    bb.parse.siggen.save_unitaskhashes()
 
         # Work out all tasks which depend upon these
         total = set()
-- 
2.25.1

[bitbake][kirkstone][2.0][PATCH 02/14] cooker: Drop sre_constants usage

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

As reported by Martin Jansa <Martin.Jansa@gmail.com>:

bitbake/lib/bb/cooker.py:16: DeprecationWarning: module 'sre_constants' is deprecated
  import sre_constants

it's deprecated since 3.11 with:

  https://github.com/python/cpython/issues/91308

The correct replacement for our usage is re.error so use that instead.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3c0cd401472ffee06d5a93bdba566cb033851fcf)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/cooker.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 6da9291f..2adf4d29 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -13,7 +13,6 @@ import sys, os, glob, os.path, re, time
 import itertools
 import logging
 import multiprocessing
-import sre_constants
 import threading
 from io import StringIO, UnsupportedOperation
 from contextlib import closing
@@ -1907,7 +1906,7 @@ class CookerCollectFiles(object):
                 try:
                     re.compile(mask)
                     bbmasks.append(mask)
-                except sre_constants.error:
+                except re.error:
                     collectlog.critical("BBMASK contains an invalid regular expression, ignoring: %s" % mask)
 
             # Then validate the combined regular expressions. This should never
@@ -1915,7 +1914,7 @@ class CookerCollectFiles(object):
             bbmask = "|".join(bbmasks)
             try:
                 bbmask_compiled = re.compile(bbmask)
-            except sre_constants.error:
+            except re.error:
                 collectlog.critical("BBMASK is not a valid regular expression, ignoring: %s" % bbmask)
                 bbmask = None
 
-- 
2.25.1

[bitbake][kirkstone][2.0][PATCH 03/14] event.py: ignore exceptions from stdout and sterr operations in atexit

[Steve Sakoman]

From: Mikko Rapeli <mikko.rapeli@bmw.de>

When atexit functions run, stdout and stderr operations may fail, e.g.
when output is piped to less but has been exited by the user.
This removes error print from output of "bitbake -e sqlite3 | less"
if user presses "q" before bitbake has finished processing:

[Errno 32] Broken pipeError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/builder/src/poky/bitbake/lib/bb/event.py", line 135, in print_ui_queue
    sys.stdout.flush()

Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 35167536c163eb6b7653cbcaad9f65b834d3e2f8)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/event.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/lib/bb/event.py b/lib/bb/event.py
index df020551..97668601 100644
--- a/lib/bb/event.py
+++ b/lib/bb/event.py
@@ -132,8 +132,14 @@ def print_ui_queue():
     if not _uiready:
         from bb.msg import BBLogFormatter
         # Flush any existing buffered content
-        sys.stdout.flush()
-        sys.stderr.flush()
+        try:
+            sys.stdout.flush()
+        except:
+            pass
+        try:
+            sys.stderr.flush()
+        except:
+            pass
         stdout = logging.StreamHandler(sys.stdout)
         stderr = logging.StreamHandler(sys.stderr)
         formatter = BBLogFormatter("%(levelname)s: %(message)s")
-- 
2.25.1

[bitbake][kirkstone][2.0][PATCH 04/14] ConfHandler: Remove lingering close

[Steve Sakoman]

From: Ola x Nilsson <ola.x.nilsson@axis.com>

The f.close() statement should have been removed in
459ad524756a3f9b50feeedf31e33502dceae8d5.

Signed-off-by: Ola x Nilsson <ola.x.nilsson@axis.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9fc1bab6b7e3c0fca3ddec4bc8c7763d2aff8bab)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/parse/parse_py/ConfHandler.py | 2 --
 1 file changed, 2 deletions(-)

diff --git a/lib/bb/parse/parse_py/ConfHandler.py b/lib/bb/parse/parse_py/ConfHandler.py
index b895d5b5..810b6011 100644
--- a/lib/bb/parse/parse_py/ConfHandler.py
+++ b/lib/bb/parse/parse_py/ConfHandler.py
@@ -147,8 +147,6 @@ def handle(fn, data, include):
     if oldfile:
         data.setVar('FILE', oldfile)
 
-    f.close()
-
     for f in confFilters:
         f(fn, data)
 
-- 
2.25.1

[bitbake][kirkstone][2.0][PATCH 05/14] bitbake-user-manual: Correct description of the ??= operator

[Steve Sakoman]

From: Jacob Kroon <jacob.kroon@gmail.com>

Stating that the assignment is done at the end of parsing is misleading.
The weak default value is the value which a variable will expand to if no value
has been assigned to it using any of the assignment operators.

Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 8189f58d0449d16f162b6e8d98c4e5edc6bff875)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 .../bitbake-user-manual-metadata.rst          | 55 +++++++++++++------
 1 file changed, 39 insertions(+), 16 deletions(-)

diff --git a/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst b/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
index af4b1358..33782161 100644
--- a/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
+++ b/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
@@ -195,22 +195,45 @@ value. However, if ``A`` is not set, the variable is set to "aval".
 Setting a weak default value (??=)
 ----------------------------------
 
-It is possible to use a "weaker" assignment than in the previous section
-by using the "??=" operator. This assignment behaves identical to "?="
-except that the assignment is made at the end of the parsing process
-rather than immediately. Consequently, when multiple "??=" assignments
-exist, the last one is used. Also, any "=" or "?=" assignment will
-override the value set with "??=". Here is an example::
-
-   A ??= "somevalue"
-   A ??= "someothervalue"
-
-If ``A`` is set before the above statements are
-parsed, the variable retains its value. If ``A`` is not set, the
-variable is set to "someothervalue".
-
-Again, this assignment is a "lazy" or "weak" assignment because it does
-not occur until the end of the parsing process.
+The weak default value of a variable is the value which that variable
+will expand to if no value has been assigned to it via any of the other
+assignment operators. The "??=" operator takes effect immediately, replacing
+any previously defined weak default value. Here is an example::
+
+   W ??= "x"
+   A := "${W}" # Immediate variable expansion
+   W ??= "y"
+   B := "${W}" # Immediate variable expansion
+   W ??= "z"
+   C = "${W}"
+   W ?= "i"
+
+After parsing we will have::
+
+   A = "x"
+   B = "y"
+   C = "i"
+   W = "i"
+
+Appending and prepending non-override style will not substitute the weak
+default value, which means that after parsing::
+
+   W ??= "x"
+   W += "y"
+
+we will have::
+
+   W = " y"
+
+On the other hand, override-style appends/prepends/removes are applied after
+any active weak default value has been substituted::
+
+   W ??= "x"
+   W:append = "y"
+
+After parsing we will have::
+
+   W = "xy"
 
 Immediate variable expansion (:=)
 ---------------------------------
-- 
2.25.1

[bitbake][kirkstone][2.0][PATCH 06/14] doc: bitbake-user-manual: add explicit target for crates fetcher

[Steve Sakoman]

From: Michael Opdenacker <michael.opdenacker@bootlin.com>

Like in other sections describing fetchers

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c9bab35f6aecbf85ee1a19a7b70e15a80b42471f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 doc/bitbake-user-manual/bitbake-user-manual-fetching.rst | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
index f9d9e617..ea8a8aa7 100644
--- a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
+++ b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
@@ -688,6 +688,8 @@ Here is an example URL::
 
 It can also be used when setting mirrors definitions using the :term:`PREMIRRORS` variable.
 
+.. _crate-fetcher:
+
 Crate Fetcher (``crate://``)
 ----------------------------
 
-- 
2.25.1

[bitbake][kirkstone][2.0][PATCH 07/14] doc: bitbake-user-manual: document npm and npmsw fetchers

[Steve Sakoman]

From: Michael Opdenacker <michael.opdenacker@bootlin.com>

This addresses bug [YOCTO #10098]

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit cca7999586317435d79bf53df4359cdd8bfadff4)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 .../bitbake-user-manual-fetching.rst          | 76 ++++++++++++++++++-
 1 file changed, 74 insertions(+), 2 deletions(-)

diff --git a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
index ea8a8aa7..4dfb7889 100644
--- a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
+++ b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
@@ -706,6 +706,80 @@ Here is an example URL::
 
    SRC_URI = "crate://crates.io/glob/0.2.11"
 
+.. _npm-fetcher:
+
+NPM Fetcher (``npm://``)
+------------------------
+
+This submodule fetches source code from an
+`NPM <https://en.wikipedia.org/wiki/Npm_(software)>`__
+Javascript package registry.
+
+The format for the :term:`SRC_URI` setting must be::
+
+   SRC_URI = "npm://some.registry.url;OptionA=xxx;OptionB=xxx;..."
+
+This fetcher supports the following parameters:
+
+-  *"package":* The NPM package name. This is a mandatory parameter.
+
+-  *"version":* The NPM package version. This is a mandatory parameter.
+
+-  *"downloadfilename":* Specifies the filename used when storing the downloaded file.
+
+-  *"destsuffix":* Specifies the directory to use to unpack the package (default: ``npm``).
+
+Note that NPM fetcher only fetches the package source itself. The dependencies
+can be fetched through the `npmsw-fetcher`_.
+
+Here is an example URL with both fetchers::
+
+   SRC_URI = " \
+       npm://registry.npmjs.org/;package=cute-files;version=${PV} \
+       npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
+       "
+
+See :yocto_docs:`Creating Node Package Manager (NPM) Packages
+</dev-manual/common-tasks.html#creating-node-package-manager-npm-packages>`
+in the Yocto Project manual for details about using
+:yocto_docs:`devtool <https://docs.yoctoproject.org/ref-manual/devtool-reference.html>`
+to automatically create a recipe from an NPM URL.
+
+.. _npmsw-fetcher:
+
+NPM shrinkwrap Fetcher (``npmsw://``)
+-------------------------------------
+
+This submodule fetches source code from an
+`NPM shrinkwrap <https://docs.npmjs.com/cli/v8/commands/npm-shrinkwrap>`__
+description file, which lists the dependencies
+of an NPM package while locking their versions.
+
+The format for the :term:`SRC_URI` setting must be::
+
+   SRC_URI = "npmsw://some.registry.url;OptionA=xxx;OptionB=xxx;..."
+
+This fetcher supports the following parameters:
+
+-  *"dev":* Set this parameter to ``1`` to install "devDependencies".
+
+-  *"destsuffix":* Specifies the directory to use to unpack the dependencies
+   (``${S}`` by default).
+
+Note that the shrinkwrap file can also be provided by the recipe for
+the package which has such dependencies, for example::
+
+   SRC_URI = " \
+       npm://registry.npmjs.org/;package=cute-files;version=${PV} \
+       npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
+       "
+
+Such a file can automatically be generated using
+:yocto_docs:`devtool <https://docs.yoctoproject.org/ref-manual/devtool-reference.html>`
+as described in the :yocto_docs:`Creating Node Package Manager (NPM) Packages
+</dev-manual/common-tasks.html#creating-node-package-manager-npm-packages>`
+section of the Yocto Project.
+
 Other Fetchers
 --------------
 
@@ -715,8 +789,6 @@ Fetch submodules also exist for the following:
 
 -  Mercurial (``hg://``)
 
--  npm (``npm://``)
-
 -  OSC (``osc://``)
 
 -  Secure FTP (``sftp://``)
-- 
2.25.1

[bitbake][kirkstone][2.0][PATCH 08/14] fetch2: gitsm: fix incorrect handling of git submodule relative urls

[Steve Sakoman]

From: Gennaro Iorio <gen.iorio92@gmail.com>

As specified by git submodule manual relative urls can start either
with '..' or './', second case was incorrectly managed leading to an
interpretation of urls starting with './' as absoulte urls.

Signed-off-by: Gennaro Iorio <gennaro.iorio@schindler.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 4a0bd3bcd1f7fc25364df8bbf185ff64881c015b)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/fetch2/gitsm.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
index c5c23d52..c1950e48 100644
--- a/lib/bb/fetch2/gitsm.py
+++ b/lib/bb/fetch2/gitsm.py
@@ -88,7 +88,7 @@ class GitSM(Git):
                 subrevision[m] = module_hash.split()[2]
 
                 # Convert relative to absolute uri based on parent uri
-                if uris[m].startswith('..'):
+                if  uris[m].startswith('..') or uris[m].startswith('./'):
                     newud = copy.copy(ud)
                     newud.path = os.path.realpath(os.path.join(newud.path, uris[m]))
                     uris[m] = Git._get_repo_url(self, newud)
-- 
2.25.1

[bitbake][kirkstone][2.0][PATCH 09/14] ConfHandler/BBHandler: Improve comment error messages and add tests

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Currently if you trigger one of the comment errors, the newline characters
are stripped and the line numbers are incorrect. In one case it prints
the empty line which is also unhelpful.

Rework the code around these errors so the line numbers are correct
and the lines in question are more clearly displayed complete with newlines
so the user can more clearly see the error.

I also added a couple of simplistic test cases to ensure that errors
are raised by the two known comment format errors.

[YOCTO #11904]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 712da71b24445c814d79a206ce26188def8fce0a)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/parse/parse_py/BBHandler.py   |  4 ++--
 lib/bb/parse/parse_py/ConfHandler.py |  9 +++++++--
 lib/bb/tests/parse.py                | 23 +++++++++++++++++++++++
 3 files changed, 32 insertions(+), 4 deletions(-)

diff --git a/lib/bb/parse/parse_py/BBHandler.py b/lib/bb/parse/parse_py/BBHandler.py
index ee9bd760..68415735 100644
--- a/lib/bb/parse/parse_py/BBHandler.py
+++ b/lib/bb/parse/parse_py/BBHandler.py
@@ -178,10 +178,10 @@ def feeder(lineno, s, fn, root, statements, eof=False):
 
     if s and s[0] == '#':
         if len(__residue__) != 0 and __residue__[0][0] != "#":
-            bb.fatal("There is a comment on line %s of file %s (%s) which is in the middle of a multiline expression.\nBitbake used to ignore these but no longer does so, please fix your metadata as errors are likely as a result of this change." % (lineno, fn, s))
+            bb.fatal("There is a comment on line %s of file %s:\n'''\n%s\n'''\nwhich is in the middle of a multiline expression. This syntax is invalid, please correct it." % (lineno, fn, s))
 
     if len(__residue__) != 0 and __residue__[0][0] == "#" and (not s or s[0] != "#"):
-        bb.fatal("There is a confusing multiline, partially commented expression on line %s of file %s (%s).\nPlease clarify whether this is all a comment or should be parsed." % (lineno, fn, s))
+        bb.fatal("There is a confusing multiline partially commented expression on line %s of file %s:\n%s\nPlease clarify whether this is all a comment or should be parsed." % (lineno - len(__residue__), fn, "\n".join(__residue__)))
 
     if s and s[-1] == '\\':
         __residue__.append(s[:-1])
diff --git a/lib/bb/parse/parse_py/ConfHandler.py b/lib/bb/parse/parse_py/ConfHandler.py
index 810b6011..451e68dd 100644
--- a/lib/bb/parse/parse_py/ConfHandler.py
+++ b/lib/bb/parse/parse_py/ConfHandler.py
@@ -125,16 +125,21 @@ def handle(fn, data, include):
             s = f.readline()
             if not s:
                 break
+            origlineno = lineno
+            origline = s
             w = s.strip()
             # skip empty lines
             if not w:
                 continue
             s = s.rstrip()
             while s[-1] == '\\':
-                s2 = f.readline().rstrip()
+                line = f.readline()
+                origline += line
+                s2 = line.rstrip()
                 lineno = lineno + 1
                 if (not s2 or s2 and s2[0] != "#") and s[0] == "#" :
-                    bb.fatal("There is a confusing multiline, partially commented expression on line %s of file %s (%s).\nPlease clarify whether this is all a comment or should be parsed." % (lineno, fn, s))
+                    bb.fatal("There is a confusing multiline, partially commented expression starting on line %s of file %s:\n%s\nPlease clarify whether this is all a comment or should be parsed." % (origlineno, fn, origline))
+
                 s = s[:-1] + s2
             # skip comments
             if s[0] == '#':
diff --git a/lib/bb/tests/parse.py b/lib/bb/tests/parse.py
index 2898f9bb..1a3b7493 100644
--- a/lib/bb/tests/parse.py
+++ b/lib/bb/tests/parse.py
@@ -194,3 +194,26 @@ deltask ${EMPTYVAR}
         self.assertTrue('addtask ignored: " do_patch"' in stdout)
         #self.assertTrue('dependent task do_foo for do_patch does not exist' in stdout)
 
+    broken_multiline_comment = """
+# First line of comment \\
+# Second line of comment \\
+
+"""
+    def test_parse_broken_multiline_comment(self):
+        f = self.parsehelper(self.broken_multiline_comment)
+        with self.assertRaises(bb.BBHandledException):
+            d = bb.parse.handle(f.name, self.d)['']
+
+
+    comment_in_var = """
+VAR = " \\
+    SOMEVAL \\
+#   some comment \\
+    SOMEOTHERVAL \\
+"
+"""
+    def test_parse_comment_in_var(self):
+        f = self.parsehelper(self.comment_in_var)
+        with self.assertRaises(bb.BBHandledException):
+            d = bb.parse.handle(f.name, self.d)['']
+
-- 
2.25.1

[bitbake][kirkstone][2.0][PATCH 10/14] fetch2: Ensure directory exists before creating symlink

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

If the mirrors code is trying to create a symlink and the
parent directory doesn't exist, as might be the case for sstate
mirrors where the fetch is into a subdir, it can silently fail.

Ensure the directory exists in this case to avoid issues.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit eff16e474ee7dc49ae433420a4c8d15d3314a618)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/fetch2/__init__.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index ac557176..a3140626 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1097,6 +1097,8 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
 
 def ensure_symlink(target, link_name):
     if not os.path.exists(link_name):
+        dirname = os.path.dirname(link_name)
+        bb.utils.mkdirhier(dirname)
         if os.path.islink(link_name):
             # Broken symbolic link
             os.unlink(link_name)
-- 
2.25.1

[bitbake][kirkstone][2.0][PATCH 11/14] bitbake: bitbake-user-manual: hashserv can be accessed on a dedicated domain

[Steve Sakoman]

From: Jose Quaresma <quaresma.jose@gmail.com>

Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit b171aa45fb8518dcfbba315b303a4fe9bf2180c6)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst
index af4ff980..12aef3cb 100644
--- a/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst
+++ b/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst
@@ -401,7 +401,7 @@ overview of their function and contents.
 
       Example usage::
 
-         BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687"
+         BB_HASHSERVE_UPSTREAM = "hashserv.yocto.io:8687"
 
    :term:`BB_INVALIDCONF`
       Used in combination with the ``ConfigParsed`` event to trigger
-- 
2.25.1

[bitbake][kirkstone][2.0][PATCH 12/14] bb/utils: remove: check the path again the expand python glob

[Steve Sakoman]

From: Jose Quaresma <quaresma.jose@gmail.com>

When we call the remove with recurse=True we first check if the
remove operation is safe in _check_unsafe_delete_path.
But the check is been done on the path instaed of the expanded
python glog.

Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit 7236488b898309ec5f1880936ddae22a28ccf5d3)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/utils.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/bb/utils.py b/lib/bb/utils.py
index 29801765..e92d5feb 100644
--- a/lib/bb/utils.py
+++ b/lib/bb/utils.py
@@ -694,8 +694,8 @@ def remove(path, recurse=False, ionice=False):
         return
     if recurse:
         for name in glob.glob(path):
-            if _check_unsafe_delete_path(path):
-                raise Exception('bb.utils.remove: called with dangerous path "%s" and recurse=True, refusing to delete!' % path)
+            if _check_unsafe_delete_path(name):
+                raise Exception('bb.utils.remove: called with dangerous path "%s" and recurse=True, refusing to delete!' % name)
         # shutil.rmtree(name) would be ideal but its too slow
         cmd = []
         if ionice:
-- 
2.25.1

[bitbake][kirkstone][2.0][PATCH 13/14] bb/utils: movefile: use the logger for printing

[Steve Sakoman]

From: Jose Quaresma <quaresma.jose@gmail.com>

Signed-off-by: Jose Quaresma <jose.quaresma@foundries.io>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
(cherry picked from commit 274b2d7a2fa0b43b0b542cb5471ff832e692ea93)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/utils.py | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/lib/bb/utils.py b/lib/bb/utils.py
index e92d5feb..95b3c898 100644
--- a/lib/bb/utils.py
+++ b/lib/bb/utils.py
@@ -753,7 +753,7 @@ def movefile(src, dest, newmtime = None, sstat = None):
         if not sstat:
             sstat = os.lstat(src)
     except Exception as e:
-        print("movefile: Stating source file failed...", e)
+        logger.warning("movefile: Stating source file failed...", e)
         return None
 
     destexists = 1
@@ -781,7 +781,7 @@ def movefile(src, dest, newmtime = None, sstat = None):
             os.unlink(src)
             return os.lstat(dest)
         except Exception as e:
-            print("movefile: failed to properly create symlink:", dest, "->", target, e)
+            logger.warning("movefile: failed to properly create symlink:", dest, "->", target, e)
             return None
 
     renamefailed = 1
@@ -798,7 +798,7 @@ def movefile(src, dest, newmtime = None, sstat = None):
         except Exception as e:
             if e.errno != errno.EXDEV:
                 # Some random error.
-                print("movefile: Failed to move", src, "to", dest, e)
+                logger.warning("movefile: Failed to move", src, "to", dest, e)
                 return None
             # Invalid cross-device-link 'bind' mounted or actually Cross-Device
 
@@ -810,13 +810,13 @@ def movefile(src, dest, newmtime = None, sstat = None):
                 bb.utils.rename(destpath + "#new", destpath)
                 didcopy = 1
             except Exception as e:
-                print('movefile: copy', src, '->', dest, 'failed.', e)
+                logger.warning('movefile: copy', src, '->', dest, 'failed.', e)
                 return None
         else:
             #we don't yet handle special, so we need to fall back to /bin/mv
             a = getstatusoutput("/bin/mv -f " + "'" + src + "' '" + dest + "'")
             if a[0] != 0:
-                print("movefile: Failed to move special file:" + src + "' to '" + dest + "'", a)
+                logger.warning("movefile: Failed to move special file:" + src + "' to '" + dest + "'", a)
                 return None # failure
         try:
             if didcopy:
@@ -824,7 +824,7 @@ def movefile(src, dest, newmtime = None, sstat = None):
                 os.chmod(destpath, stat.S_IMODE(sstat[stat.ST_MODE])) # Sticky is reset on chown
                 os.unlink(src)
         except Exception as e:
-            print("movefile: Failed to chown/chmod/unlink", dest, e)
+            logger.warning("movefile: Failed to chown/chmod/unlink", dest, e)
             return None
 
     if newmtime:
-- 
2.25.1

[bitbake][kirkstone][2.0][PATCH 14/14] bitbake-user-manual: npm fetcher: improve description of SRC_URI format

[Steve Sakoman]

From: Michael Opdenacker <michael.opdenacker@bootlin.com>

Using the term "Parameter" which is consistent with the
description of SRC_URI parameters in the following text.

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reported-by: Quentin Schulz <foss@0leil.net>
Reviewed-by: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
(cherry picked from commit 87e42f1202162152c779ccc8bbd06f88f0bdab96)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 doc/bitbake-user-manual/bitbake-user-manual-fetching.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
index 4dfb7889..9c269ca8 100644
--- a/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
+++ b/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst
@@ -717,7 +717,7 @@ Javascript package registry.
 
 The format for the :term:`SRC_URI` setting must be::
 
-   SRC_URI = "npm://some.registry.url;OptionA=xxx;OptionB=xxx;..."
+   SRC_URI = "npm://some.registry.url;ParameterA=xxx;ParameterB=xxx;..."
 
 This fetcher supports the following parameters:
 
@@ -757,7 +757,7 @@ of an NPM package while locking their versions.
 
 The format for the :term:`SRC_URI` setting must be::
 
-   SRC_URI = "npmsw://some.registry.url;OptionA=xxx;OptionB=xxx;..."
+   SRC_URI = "npmsw://some.registry.url;ParameterA=xxx;ParameterB=xxx;..."
 
 This fetcher supports the following parameters:
 
-- 
2.25.1